Company deep dives

Shopify interviews

Canada

Experiments

“Two marketers can simultaneously test the same audience with no exclusion criteria, violating the assumption that all other variables are controlled.”

Mojan BenhamMarketing Data Science, Shopify

Shopify Engineering · 2020-10-15

Overlapping tests break the comparison

Decide whether a 10% onboarding lift is ready to ship.

Example

“In this case, two hours of scoping saved at least two days of engineering work, and we learned something useful in the process.”

Ian WhitestoneShopify Engineering author

Shopify Engineering · 2021-06-17

Scope before you build

Decide whether a 10% onboarding lift is ready to ship.

Example

“Small merchants have it worse: typically they don't have enough traffic for testing to converge at all.”

Javier MorenoShopify Engineering author

Shopify Engineering · 2026-02-27

Small stores lack traffic

Decide whether a 10% onboarding lift is ready to ship.

Example

Experiment design

“Randomization units should be independent, and if they aren’t, you may not be measuring the true effect of your change.”

Ian WhitestoneShopify Engineering author

Shopify Engineering · 2021-06-17

Independent randomization units

Decide whether one winner among 20 AI-written headlines should roll out.

Example

“If you’re ever unsure about some statistical effect, it’s very quick (and fun) to run a simulation to see how you’d be affected, before jumping to any conclusions.”

Ian WhitestoneShopify Engineering author

Shopify Engineering · 2021-06-17

Simulate before you conclude

Decide whether one winner among 20 AI-written headlines should roll out.

Example

What AI changes in the work

“A simulated A/B test in minutes instead of weeks, even for stores with zero real traffic.”

Javier MorenoShopify Engineering author

Shopify Engineering · 2026-02-27

AI-simulated tests in minutes

Decide whether AI share cards that doubled sharing should roll out.

Example

Last reviewed