Shopify interviews
Canada
Experiments
“Two marketers can simultaneously test the same audience with no exclusion criteria, violating the assumption that all other variables are controlled.”
Mojan BenhamMarketing Data Science, Shopify
“In this case, two hours of scoping saved at least two days of engineering work, and we learned something useful in the process.”
Ian WhitestoneShopify Engineering author
“Small merchants have it worse: typically they don't have enough traffic for testing to converge at all.”
Javier MorenoShopify Engineering author
Experiment design
“Randomization units should be independent, and if they aren’t, you may not be measuring the true effect of your change.”
Ian WhitestoneShopify Engineering author
Independent randomization units
Decide whether one winner among 20 AI-written headlines should roll out.
Example“If you’re ever unsure about some statistical effect, it’s very quick (and fun) to run a simulation to see how you’d be affected, before jumping to any conclusions.”
Ian WhitestoneShopify Engineering author
Simulate before you conclude
Decide whether one winner among 20 AI-written headlines should roll out.
ExampleWhat AI changes in the work
“A simulated A/B test in minutes instead of weeks, even for stores with zero real traffic.”
Javier MorenoShopify Engineering author
AI-simulated tests in minutes
Decide whether AI share cards that doubled sharing should roll out.
ExampleLast reviewed
