SkillsSkill 5 of 5
Measure the whole loop, not the step that moved
Judge a loop by the new active users it produces, not by the step that moved.
Worked example
AI share cards doubled sharing. Would you roll them out?
A loop is a chain. Doubling one link means nothing if another link gets weaker.
The question
Kettlewise, a fictional AI meal-planning app, lets users share a week’s meal plan. A non-user who opens a shared plan can sign up. The team tested AI-written share cards, a generated image and caption, against the current cards, which show the user’s own plan. 10,000 active users were split evenly for four weeks, and every new user has had two weeks to come back.
| Current cards | AI-written cards | |
|---|---|---|
| Active users in the test | 5,000 | 5,000 |
| Shared a plan | 400 (8%) | 800 (16%) |
| Views of shared plans | 2,000 | 4,000 |
| Viewers who signed up | 120 (6.0%) | 100 (2.5%) |
| New users active in week 2 | 54 (45%) | 45 (45%) |
Invented data. Each share was viewed five times on average in both groups. Loop output: 10.8 and 9.0 new active users per 1,000 active users.
AI share cards doubled sharing, from 8% to 16%. Recommend rolling them out to all users.
“AI-written share cards doubled the share rate from 8% to 16%. Would you roll them out?”
Your turn
Say your answer out loud, or jot notes below. Both work. Or skip to the worked answer.
Not saved by this site. Your browser may restore text when you navigate back.
A strong answer, step by step
Open each step in turn. This is one strong answer, not the only one.
Step 1: Follow the loop to the end
What would you do here? Open the move you’d make.
Judge the loop on new active users: 54 from current cards, 45 from AI cards. Shares doubled; sign-ups per view fell from 6.0% to 2.5%.
The strong move. It judges the loop on its output, which is what compounds.
Roll out: the share rate doubled from 8% to 16%.
Tempting, but it misses something. Shares are a step, not the goal. 800 sharers brought 100 sign-ups; 400 brought 120.
Judge it on views: AI cards got 4,000 against 2,000.
Tempting, but it misses something. Views come straight from shares, five per share in both groups, so they doubled too. What matters is who signs up and stays.
See how a strong candidate says it
“The loop exists to bring in new active users, not shares. Current cards produced 54 from 5,000 users; AI cards produced 45. Shares doubled, but sign-ups per view fell from 6.0% to 2.5%.”
Why it works: It judges the loop on its output, which is what compounds.
Step 2: Find where it leaked
What would you do here? Open the move you’d make.
Blame the sign-up page and work on its conversion rate.
Tempting, but it misses something. The page didn’t change between groups; the card did. Sign-ups per view fell from 6.0% to 2.5% only with AI cards.
Blame the sharers: AI-card users must have weaker networks.
Tempting, but it misses something. Users were split at random, so the two groups’ networks should be alike. The difference is what viewers saw.
Find the step that fell: viewers signing up, 100 against 120. My guess is a generic card says less than a real plan, so I’d check what viewers saw.
The strong move. It locates the weak link and labels the explanation as a guess to check.
See how a strong candidate says it
“More people shared, but fewer viewers signed up: 100 against 120. My guess is a generic AI card says less about Kettlewise than a real plan does, so I’d look at what viewers actually saw.”
Why it works: It locates the weak link and labels the explanation as a hypothesis to check.
Step 3: Be honest about the sample
What would you do here? Open the move you’d make.
Declare the AI cards worse: 45 is 17% fewer than 54.
Tempting, but it misses something. On counts this small the gap isn’t significant; 54 against 45 could easily flip on a rerun.
Say what it shows: 54 against 45 doesn’t prove AI cards are worse, but doubling shares didn’t bring more users.
The strong move. It refuses to swap one overclaim for another.
Ignore the result: 5,000 users per group is too few to learn anything.
Tempting, but it misses something. It’s enough to see that shares doubled and sign-ups didn’t follow. It just can’t settle 54 against 45.
See how a strong candidate says it
“54 against 45 is a small difference on small counts; it doesn’t show AI cards are worse. It does show that doubling shares didn’t produce more users.”
Why it works: It refuses to swap one overclaim for another.
Step 4: Make a scoped decision
What would you do here? Open the move you’d make.
Don’t roll out on shares. Test a card built from the user’s real plan with an AI caption, judged on new active users per 1,000 active.
The strong move. It keeps what worked, more sharing, and fixes the measure of success.
Roll out AI cards now and fix sign-ups later.
Tempting, but it misses something. On this test the loop made 9.0 new active users per 1,000 active against 10.8. Shipping first locks in a card that hasn’t earned it.
Drop sharing as a channel: 10.8 per 1,000 is too small to matter.
Tempting, but it misses something. Small isn’t zero, and it compounds. The test is about which card grows the loop, not whether the loop is worth having.
See how a strong candidate says it
“I wouldn’t roll them out on the share rate. I’d test a card built from the user’s real plan with an AI caption, judged on new active users per 1,000 active, and run it long enough to tell the arms apart.”
Why it works: A next step that keeps what worked, sharing, and fixes the measure of success.
The answer, in short
- Decision
- Don’t roll out AI-written cards. Test a card built from the user’s real plan, judged on new active users per 1,000 active users.
- Because
- Sharing doubled, from 400 to 800, but sign-ups fell from 120 to 100 and new active users from 54 to 45: the loop’s output didn’t grow.
- What would change it
- A version that raises new active users per 1,000 active, not just shares, over a run long enough to separate it from noise.
Common traps
- Celebrating the step that moved.
- Ignoring what the person on the other end of a share sees.
- Declaring the AI cards worse on 54 against 45.
- Calling a loop viral without computing its output.
Check your answer
- Did I judge the loop on new active users?
- Did I find which step got weaker?
- Did I avoid over-reading small counts in either direction?
- Did I propose a next test with a clear success measure?
Want feedback on how you’d answer? Practice “Measure the whole loop, not the step that moved” with an AI interviewer on Socratify.
Last reviewed Nov 9, 2026