A/B test readout

How the readout works

1. Rates and lift. Each arm’s rate is conversions ÷ visitors. The absolute lift is the variant’s rate minus the control’s, in percentage points. The relative lift is that difference ÷ the control’s rate.
2. Interval. The difference ± z × √(p₁(1 − p₁) ÷ n₁ + p₂(1 − p₂) ÷ n₂), where z is 1.96 for 95%. If the interval includes zero, the data fit no effect at all.
3. p-value. A two-proportion z-test: pool both arms’ conversions into one rate p, then z = difference ÷ √(p(1 − p)(1 ÷ n₁ + 1 ÷ n₂)). The p-value is the chance of a gap at least this big, in either direction, if the arms were really the same.
4. Sample size. To detect a move from the control’s rate p₁ to p₂ = p₁ × (1 + your minimum lift), each arm needs (z + 0.84)² × (p₁(1 − p₁) + p₂(1 − p₂)) ÷ (p₂ − p₁)² users, for 80% power.

On Kettlewise’s onboarding test, 660 trials from 4,000 against 600 from 4,000 is 16.5% against 15.0%: +1.5 points, a 10% lift. The 95% interval runs from −0.1 to +3.1 points, so it includes zero, and p is about 0.07. Detecting a 10% lift from 15% needs 9,254 users per arm; the test stopped at 4,000. Inconclusive, not a loss.

The arithmetic can’t catch everything:

• Peeking. These numbers assume you decide once, at the planned sample. Checking daily and stopping on a good day makes false wins far more likely.
• Many variants. Each extra arm is another chance of a fluke. With 20 variants at 95%, expect about one to look significant by luck.
• Novelty. A new screen can lift clicks for a week and then fade. Run a full business cycle, and check a later outcome such as paid conversion.
• Small samples. With few conversions the interval is wide, and the normal approximation behind it gets rough. A test that is too small can’t tell you a change does nothing.