SkillsSkill 2 of 5
Treat cheap variants as guesses, not wins
Count your comparisons, set a bar that fits them, and rerun the best variant before you trust its lift.
Worked example
One of 20 AI-written headlines won. Would you roll it out?
When variants cost nothing, the number of comparisons becomes part of the evidence. Count them before you celebrate.
The question
Kettlewise, a fictional AI meal-planning app, asked an AI tool for 20 new paywall headlines and tested them all at once. 42,000 new users who reached the paywall were split evenly: 2,000 saw the current headline and 2,000 saw each new one.
| Reason | headlines | Share |
|---|---|---|
| Beat the current headline at p < 0.05 | 1 | |
| No significant difference | 19 | |
| Total | 20 |
Invented data. Every headline was compared with the same group of 2,000 users. At p < 0.05, about 1 comparison in 20 crosses the line by chance when nothing is different.
| Current headline | Headline 14 | |
|---|---|---|
| Saw the paywall | 2,000 | 2,000 |
| Subscribed | 100 (5.0%) | 132 (6.6%) |
Invented data. A 1.6-point gap, 32% in relative terms. Two-sided p ≈ 0.03 for this one comparison on its own.
Headline 14 (“Dinner, decided.”) is the clear winner: +32% subscriptions. Roll it out to all users.
“Of 20 AI-written paywall headlines, one beat the current headline: 6.6% against 5.0%, p ≈ 0.03. Would you roll it out?”
Your turn
Say your answer out loud, or jot notes below. Both work. Or skip to the worked answer.
Not saved by this site. Your browser may restore text when you navigate back.
A strong answer, step by step
Open each step in turn. This is one strong answer, not the only one.
Step 1: Count the comparisons
What would you do here? Open the move you’d make.
Count the tries: with 20 headlines, about one comes out significant by luck, and there’s about a 40% chance one looks like a win.
The strong move. Volume changes what a single p-value means. One winner in 20 is well within what chance produces.
Headline 14 won at p ≈ 0.03, below 0.05, so it’s a real result.
Tempting, but it misses something. That bar holds for one test. With 20 comparisons against the same control, a result like this is expected even if no headline is better.
Judge the copy: “Dinner, decided.” reads well, so the lift is plausible.
Tempting, but it misses something. A plausible story fits any winner. The question is whether 132 against 100 survives 20 tries, not whether the line is catchy.
See how a strong candidate says it
“Twenty headlines means twenty chances. If none were better, about one in 20 would still come out significant by luck, better or worse: roughly a 64% chance of at least one, and about 40% that at least one looks like a win. One winner is well within what chance produces.”
Why it works: Volume changes what a single p-value means. Cheap variants make this the first question, not a footnote.
Step 2: Use a bar that fits the number of tests
What would you do here? Open the move you’d make.
Lower the bar to p < 0.01 to be safe.
Tempting, but it misses something. Closer, but arbitrary. The bar should follow from the number of tests: 0.05 over 20 is 0.0025, and p ≈ 0.03 fails either way.
Keep p < 0.05; corrections are too strict for product tests.
Tempting, but it misses something. Without one, running 20 tests at 0.05 makes at least one false significant result more likely than not: roughly 64%.
Set a bar for 20 tests: 0.05 ÷ 20 = 0.0025. Headline 14’s p ≈ 0.03 doesn’t clear it.
The strong move. It names a concrete correction instead of a vague worry about luck.
See how a strong candidate says it
“Spread over 20 comparisons, a simple Bonferroni bar is 0.05 divided by 20, or 0.0025. Headline 14’s p of about 0.03 doesn’t clear it.”
Why it works: It names a concrete correction instead of a vague worry about luck.
Step 3: Expect the winner to shrink
What would you do here? Open the move you’d make.
Forecast +32% subscriptions after rollout.
Tempting, but it misses something. The best of 20 results overstates its effect even if it’s real. +32% is an upper-end guess, not a forecast.
Even if headline 14 is better, picking the best of 20 inflates its lift. I’d expect less than +32%.
The strong move. Naming the winner’s curse shows you know why top results shrink after launch.
Average all 20 headlines’ lifts to estimate the effect.
Tempting, but it misses something. That answers a different question, how good AI headlines are on average, not whether headline 14 beats the current one.
See how a strong candidate says it
“Even if headline 14 really is better, picking the best of 20 inflates its measured lift. I wouldn’t forecast +32%; the true effect is probably smaller.”
Why it works: Picking the top result overstates it, the winner’s curse. Naming it is the senior signal.
Step 4: Decide and replicate
What would you do here? Open the move you’d make.
Roll out headline 14 now; it can’t hurt to try.
Tempting, but it misses something. It can. A false win becomes the new control, and the team stops looking for what actually stops people subscribing.
Generate 50 more headlines and test them all to find a bigger winner.
Tempting, but it misses something. More variants on the same traffic means smaller groups and more chance wins. Volume is the problem here, not the fix.
Rerun headline 14 alone against the current one, about 8,200 users each, and draw new ideas from why people don’t subscribe.
The strong move. A replication sized to detect a one-point lift from 5% can confirm or kill the result.
See how a strong candidate says it
“I wouldn’t roll it out on this. I’d rerun headline 14 alone against the current headline, about 8,200 users each, enough to detect a one-point lift from 5%. And I’d draw the next variants from what stops people subscribing, not from volume.”
Why it works: It turns a lucky-looking result into a test that can confirm it, and fixes how ideas are chosen.
The answer, in short
- Decision
- Don’t roll out headline 14 yet. Rerun it alone against the current headline with about 8,200 users each, and treat +32% as an upper-end guess.
- Because
- With 20 comparisons, one result at p ≈ 0.03 is about what chance predicts, and it misses the 0.0025 bar that 20 tests imply.
- What would change it
- A replication that beats the current headline at the planned size, with first-month cancellations no worse.
Common traps
- Celebrating the one winner out of many tries.
- Reading each p-value as if it were the only test.
- Forecasting the best variant’s lift at face value.
- Mistaking test volume for learning.
- Letting the tool that wrote the variants also judge them.
Check your answer
- Did I count how many comparisons were run?
- Did I set a significance bar that fits that number?
- Did I say why the winner’s lift is likely overstated?
- Did I design a replication with a stated size?
Want feedback on how you’d answer? Practice “Treat cheap variants as guesses, not wins” with an AI interviewer on Socratify.
Last reviewed Nov 9, 2026