SkillsSkill 3 of 5
Split retention by cohort before blaming the product
Split a blended rate by cohort and channel, and judge each channel on its cost per retained user.
Worked example
Retention fell after an AI launch. Would you roll it back?
A blended retention rate mixes who you acquired with how well you kept them. Separate the two before you decide anything.
The question
Kettlewise, a fictional AI meal-planning app, launched AI meal plans in February. In April it also started paying TikTok creators to promote the app. It is now July, so both cohorts below have had their full second month. Leadership wants to roll the AI plans back.
| Nov sign-ups | Nov retained | Apr sign-ups | Apr retained | |
|---|---|---|---|---|
| Organic and referral | 3,000 | 1,350 | 3,000 | 1,350 |
| Paid social | 1,000 | 250 | 1,000 | 250 |
| TikTok creators | — | — | 4,000 | 600 |
| All channels | 4,000 | 1,600 | 8,000 | 2,200 |
Invented data. Retained means active 31–60 days after sign-up. Organic 45% and paid social 25% in both cohorts; TikTok 15%. All channels: 40.0% in November, 27.5% in April. AI plans launched between the two cohorts.
- TikTok creators
- $32,000 for 4,000 sign-ups, $8 each.
- Paid social
- $12,000 for 1,000 sign-ups, $12 each.
Invented figures. Organic and referral sign-ups have no direct spend.
- Head of productRetention dropped from 40% to 27.5% after the AI plans shipped. Roll them back.
- Growth marketerTikTok is our cheapest channel per sign-up. Let’s double it.
“Month-2 retention fell from 40% to 27.5% after AI meal plans launched. Would you roll the feature back?”
Your turn
Say your answer out loud, or jot notes below. Both work. Or skip to the worked answer.
Not saved by this site. Your browser may restore text when you navigate back.
A strong answer, step by step
Open each step in turn. This is one strong answer, not the only one.
Step 1: Split the average by cohort and channel
What would you do here? Open the move you’d make.
Roll back the AI plans: retention fell from 40% to 27.5% right after launch.
Tempting, but it misses something. Timing isn’t cause. Within each channel retention didn’t move: organic kept 45% and paid social 25% in both cohorts.
Survey churned April users about the AI plans.
Tempting, but it misses something. Useful later, but it skips the cheapest check. The table already shows organic and paid social users kept the same share.
Split by channel first: organic kept 45% and paid social 25% in both cohorts; April added 4,000 TikTok sign-ups at 15%.
The strong move. A change in mix can move an average when no group changed.
See how a strong candidate says it
“Within each channel nothing moved: organic kept 45% in both cohorts, paid social 25%. The blended rate fell because April added 4,000 TikTok sign-ups who kept 15%.”
Why it works: A change in mix can move an average when no group changed. This is the first check on any blended rate.
Step 2: Look at the absolute numbers
What would you do here? Open the move you’d make.
Focus on the rate; counts just grow with spend.
Tempting, but it misses something. Here the counts tell the story: retained users rose from 1,600 to 2,200 while the rate fell.
Check the counts: retained users rose from 1,600 to 2,200. The rate fell because we bought a different kind of user.
The strong move. It separates acquisition quality from product quality, which have different owners and fixes.
Conclude the AI plans raised retention, since retained users grew by 600.
Tempting, but it misses something. All 600 extra retained users came from TikTok. Organic and paid social retained 1,600 in both cohorts, so the product claim isn’t supported either way.
See how a strong candidate says it
“Retained users rose from 1,600 to 2,200. The rate fell because we acquired a different kind of user, not because the people we already served left faster.”
Why it works: It separates acquisition quality from product quality, which point to different owners and fixes.
Step 3: Judge the new channel on retained users
What would you do here? Open the move you’d make.
Price it per retained user: TikTok is $8 against $12 per sign-up, but about $53 against $48 per retained user.
The strong move. It answers the “double TikTok” claim with the same data.
Double TikTok: at $8 a sign-up it’s the cheapest channel.
Tempting, but it misses something. Cheap sign-ups who leave aren’t cheap. $32,000 bought 600 retained users, about $53 each, more than paid social’s $48.
Cut TikTok entirely: its 15% retention drags the average down.
Tempting, but it misses something. A lower blended rate isn’t a cost. TikTok still added 600 retained users; the questions are their price and whether they hold at month 3.
See how a strong candidate says it
“TikTok is cheaper per sign-up, $8 against $12, but dearer per retained user: $32,000 for 600 is about $53, against $48 for paid social. Doubling it on sign-up cost would buy the wrong thing.”
Why it works: It answers the second claim in the room with the same data instead of ignoring it.
Step 4: Make a scoped decision
What would you do here? Open the move you’d make.
Roll back the AI plans to be safe and watch whether retention recovers.
Tempting, but it misses something. It removes a feature the data doesn’t blame, and the blended rate would stay low as long as TikTok sign-ups keep coming.
Keep the AI plans, report retention by channel cohort, and hold TikTok spend until its month-3 retention is in.
The strong move. A decision on each claim, honest that a before-and-after comparison can’t prove the AI plans helped.
Keep the AI plans and announce they had no effect on retention.
Tempting, but it misses something. Two cohorts months apart can’t show no effect. They show no sign of harm, which is a weaker claim.
See how a strong candidate says it
“I’d keep the AI plans: this data gives no reason to roll them back, and no proof they helped either. I’d report retention by channel cohort, and hold TikTok spend flat until we see whether its April cohort flattens by month 3.”
Why it works: A decision for each claim, honest about what a before-and-after comparison can’t prove.
The answer, in short
- Decision
- Don’t roll back the AI meal plans. Report retention by sign-up channel and cohort, and hold TikTok spend flat until its month-3 retention is in.
- Because
- Organic (45%) and paid social (25%) kept the same share in both cohorts; the fall to 27.5% comes from 4,000 TikTok sign-ups keeping 15%.
- What would change it
- A fall within the same channel after launch, such as organic dropping below 45%, or a TikTok cohort that holds well above 15% after month 2.
Common traps
- Blaming the most recent launch for a blended drop.
- Reading a rate without the counts behind it.
- Judging a channel on cost per sign-up.
- Treating one cohort per period as a trend.
- Claiming the AI plans helped because nothing fell.
Check your answer
- Did I split the rate by cohort and channel before explaining it?
- Did I check absolute retained users as well as the rate?
- Did I judge the channel on cost per retained user?
- Did I say what this comparison can’t prove?
Want feedback on how you’d answer? Practice “Split retention by cohort before blaming the product” with an AI interviewer on Socratify.
Last reviewed Nov 9, 2026