SkillsSkill 1 of 5
Read the test before you ship the winner
Restate a lift as counts, compare the sample with the plan, and call an underpowered result inconclusive.
Worked example
The new onboarding shows a 10% lift. Would you ship it?
A relative lift on a dashboard is the start of the question, not the answer. Check the counts, the plan and the guardrail first.
The question
Kettlewise, a fictional AI meal-planning app, tested a shorter onboarding quiz against the current one. New users joined over two weeks in September; every one of them has now had 30 days. The team stopped the test on day 14, when the dashboard showed a lift.
| Current quiz | Shorter quiz | |
|---|---|---|
| New users | 4,000 | 4,000 |
| Started a free trial | 600 (15.0%) | 660 (16.5%) |
| Paid by day 30 | 240 (6.0%) | 236 (5.9%) |
Invented data. Users were split at random, half to each quiz. Trial starts differ by 1.5 points, 10% in relative terms.
- Decision metric
- Free-trial starts.
- Guardrail
- Paid subscribers by day 30, watched for a meaningful fall.
- Planned size
- About 9,300 new users per quiz, enough to detect a 1.5-point change from 15% with 80% power at p < 0.05.
- What happened
- Stopped at 4,000 per quiz on day 14, after the dashboard showed +10%.
The shorter quiz lifts trial starts by 10%. Recommend shipping it to all new users.
“The shorter onboarding quiz shows 16.5% trial starts against 15.0%, a 10% lift, in a test stopped on day 14. Would you ship it?”
Your turn
Say your answer out loud, or jot notes below. Both work. Or skip to the worked answer.
Not saved by this site. Your browser may restore text when you navigate back.
A strong answer, step by step
Open each step in turn. This is one strong answer, not the only one.
Step 1: Restate the result in counts
What would you do here? Open the move you’d make.
Lead with the 10% lift: it’s a clean number for the room.
Tempting, but it misses something. A relative lift hides how much evidence there is. Here 10% is 1.5 points: 660 trials against 600 from 4,000 users each.
Restate it in counts: 660 trials against 600 from 4,000 each, 1.5 points or 10% relative, then ask whether that beats noise.
The strong move. Counts and absolute points show the size of the evidence before anyone argues about shipping.
Ask for the dashboard’s confidence score and go with it.
Tempting, but it misses something. The dashboard and the AI readout skipped the plan. A score without counts won’t tell you the test stopped at 4,000 users per quiz, under half the 9,300 planned.
See how a strong candidate says it
“It’s 660 trials against 600, from 4,000 users each: 1.5 points, or 10% relative. Before shipping I’d ask whether that gap is bigger than noise.”
Why it works: Counts and absolute points show the size of the evidence; a relative lift hides it.
Step 2: Check it against the plan
What would you do here? Open the move you’d make.
Call it a loss: p ≈ 0.07 misses 0.05, so the shorter quiz doesn’t work.
Tempting, but it misses something. Missing the bar isn’t evidence of no effect. At 4,000 per quiz the test couldn’t reliably detect a 1.5-point change either way.
Call it a win: p ≈ 0.07 is close enough to 0.05 for a quiz change.
Tempting, but it misses something. Moving the bar after seeing the result is how false wins get shipped. The plan set p < 0.05 and about 9,300 users per quiz; neither was met.
Compare with the plan: it needed about 9,300 per quiz and stopped at 4,000. z ≈ 1.84, p ≈ 0.07, so it’s inconclusive.
The strong move. It reads the result against what the test was built to detect, and names the honest outcome.
See how a strong candidate says it
“The plan needed about 9,300 users per quiz, and the test stopped at 4,000 because the dashboard looked good. On these counts z is about 1.84, p about 0.07. That’s inconclusive: not a loss, and not a win.”
Why it works: Stopping on a good day is how false wins get shipped. Inconclusive is an honest result, not a failure.
Step 3: Read the guardrail
What would you do here? Open the move you’d make.
Check paid by day 30: 240 against 236. Extra trials that don’t pay aren’t the goal, but this sample can’t show a paid change.
The strong move. It checks the outcome the business needs without over-reading a flat number on 4,000 users per quiz.
Skip it: the decision metric is trials, and trials went up.
Tempting, but it misses something. The plan named paid by day 30 as the guardrail for a reason. 60 extra trials matter only if some become subscribers; so far it’s 236 against 240.
Treat 236 against 240 as proof the shorter quiz hurts revenue.
Tempting, but it misses something. A gap of four subscribers on 4,000 users each is noise. The guardrail is flat, not falling.
See how a strong candidate says it
“Paid by day 30 is 240 against 236, 6.0% against 5.9%. Extra trials that don’t become subscribers aren’t the win we want, though this sample is far too small to show a paid change either way.”
Why it works: It checks the outcome the business needs without over-reading a flat number on a small sample.
Step 4: Make a scoped decision
What would you do here? Open the move you’d make.
Ship it to everyone and report a 10% lift in trials.
Tempting, but it misses something. It books a lift the test hasn’t shown. If the true effect is zero, the forecast is wrong and the next test starts from a false baseline.
Don’t call it a win. Resume the test to about 9,300 per quiz without peeking; if we must decide now, ship it as neutral and forecast no lift.
The strong move. A decision the team can act on, with a rule for what would change it.
Scrap the shorter quiz and go back to the drawing board.
Tempting, but it misses something. Nothing here says the quiz is worse. Trials are up 1.5 points and paid is flat; the test hasn’t run long enough to tell.
See how a strong candidate says it
“I wouldn’t announce a 10% win. I’d resume the test and run it to the planned 9,300 per quiz without peeking again, then decide on trials with paid by day 30 as the guardrail. If we must decide now, ship the shorter quiz only as a neutral design change and forecast no lift.”
Why it works: A decision the team can act on, with a stated rule for what would change it.
The answer, in short
- Decision
- Don’t ship it as a win. Resume the test and run it to about 9,300 users per quiz, then decide on trial starts, with paid by day 30 as the guardrail.
- Because
- At 4,000 per quiz the 1.5-point gap gives p ≈ 0.07, the test stopped early on a good-looking dashboard, and paid conversion hasn’t moved: 240 against 236.
- What would change it
- A full-size result that clears the plan’s bar with paid conversion holding, or a separate reason, such as lower support cost, to ship the shorter quiz regardless of lift.
Common traps
- Quoting the relative lift without the counts.
- Stopping a test the day the dashboard looks good.
- Reading p ≈ 0.07 as proof the change does nothing.
- Optimizing trial starts while paid conversion stays flat.
- Trusting an AI readout that never looked at the plan.
Check your answer
- Did I restate the result as counts and absolute points?
- Did I compare the sample with the plan before reading p?
- Did I check the guardrail without over-reading it?
- Did I give a decision and what would change it?
Want feedback on how you’d answer? Practice “Read the test before you ship the winner” with an AI interviewer on Socratify.
Last reviewed Nov 9, 2026