SkillsSkill 4 of 5
Choose the metric that predicts the outcome
An activation metric must predict the outcome you care about. Check that before it becomes a goal.
Worked example
Plan generation jumped to 71%. Should it be the goal?
A metric that AI makes easy to move is not the same as a metric that predicts whether people stay.
The question
Kettlewise, a fictional AI meal-planning app, shipped one-tap AI meal plans in August. The head of product wants “generated a first plan on day 1” to become the onboarding team’s activation metric and goal, because it jumped.
| July (before) | September (after) | |
|---|---|---|
| Signed up | 10,000 | 10,000 |
| Generated a first plan on day 1 | 5,200 (52%) | 7,100 (71%) |
| Cooked a planned recipe in week 1 | 2,600 (26%) | 2,500 (25%) |
| Active in week 4 | 2,080 (20.8%) | 2,000 (20.0%) |
Invented data. Two cohorts a month apart, not a randomized test. Cooking is logged when a user marks a planned recipe as made.
| Users | Active in week 4 | |
|---|---|---|
| Cooked a planned recipe | 2,600 | 1,560 (60%) |
| Generated a plan on day 1, didn’t cook | 2,600 | 390 (15%) |
| No plan on day 1 | 4,800 | 130 (2.7%) |
Invented data. Everyone who cooked had generated a plan on day 1, so the rows add up to 10,000 users and 2,080 active in week 4. Observational: it shows association, not cause.
“One-tap AI plans lifted day-1 plan generation from 52% to 71%. Should “generated a first plan” become the onboarding team’s activation metric?”
Your turn
Say your answer out loud, or jot notes below. Both work. Or skip to the worked answer.
Not saved by this site. Your browser may restore text when you navigate back.
A strong answer, step by step
Open each step in turn. This is one strong answer, not the only one.
Step 1: Say what the metric must predict
What would you do here? Open the move you’d make.
Adopt it: a 19-point jump proves the team can move it.
Tempting, but it misses something. Being easy to move isn’t the test. Week-4 activity went from 20.8% to 20.0% while plan generation jumped.
First name the outcome: being active in week 4. Plan generation rose 19 points; week 4 went from 20.8% to 20.0%.
The strong move. It sets the test every candidate metric must pass.
Pick the metric with the most users in it, since it’s the most stable.
Tempting, but it misses something. Stability doesn’t make a metric useful. It has to predict the outcome, and this one moved while week 4 didn’t.
See how a strong candidate says it
“An activation metric earns its place by predicting the outcome we care about, here being active in week 4. Plan generation rose 19 points, but week-4 activity went from 20.8% to 20.0%.”
Why it works: It sets the test every candidate metric has to pass before arguing about any of them.
Step 2: Find the behavior that separates people who return
What would you do here? Open the move you’d make.
Compare July users: 60% of those who cooked were active in week 4, against 15% of plan-makers who didn’t cook.
The strong move. It finds the step closest to the value users came for.
Make week-4 activity itself the onboarding goal.
Tempting, but it misses something. It’s the outcome, but it arrives a month late. The team needs a week-1 signal that predicts it, and cooking separates 60% from 15%.
Ask users in a survey which feature they value most.
Tempting, but it misses something. What people say isn’t what they do. The logs already show cooking separates people who return: 60% against 15%.
See how a strong candidate says it
“In July, 60% of users who cooked a planned recipe were active in week 4, against 15% of those who generated a plan but didn’t cook. Making a plan barely separates them; cooking does.”
Why it works: It looks for the step closest to the value users came for, not the step the feature made easy.
Step 3: Name the limits
What would you do here? Open the move you’d make.
Treat cooking as the cause: get everyone to cook and week 4 will reach 60%.
Tempting, but it misses something. That’s association. Keen cooks may simply be keen; pushing reluctant users to cook won’t make them return at 60%.
Ignore the July breakdown; correlations prove nothing.
Tempting, but it misses something. Association still shows where to look. 60% against 15% is a big gap; the next step is to test it, not discard it.
Call it association, and confirm it on later cohorts or with a holdout before setting a goal on it.
The strong move. It avoids swapping one unproven metric for another.
See how a strong candidate says it
“That’s association: keen cooks may simply be keen. And it’s two cohorts a month apart, not a test. I’d confirm it on more cohorts, or with a holdout, before setting a goal on it.”
Why it works: It avoids swapping one unproven metric for another.
Step 4: Make a scoped decision
What would you do here? Open the move you’d make.
Adopt plan generation now and revisit it next quarter.
Tempting, but it misses something. Goals shape a quarter of work. A team chasing plan generation can raise it again without moving week 4, as one-tap plans just did.
Propose “cooked a planned recipe in week 1” as the candidate, validated against week 4, and note one-tap plans didn’t move cooking.
The strong move. A specific metric, plus the outcome that keeps it honest.
Roll back one-tap plans, since they didn’t raise week-4 activity.
Tempting, but it misses something. Two cohorts a month apart can’t show the feature hurt. The gap, 2,000 against 2,080 active in week 4, isn’t significant on 10,000 users each.
See how a strong candidate says it
“I wouldn’t adopt “generated a first plan”. I’d propose “cooked a planned recipe in week 1” as the candidate, with week-4 activity as the outcome it must keep predicting. And I’d note one-tap plans didn’t move cooking: 2,500 against 2,600.”
Why it works: A specific recommendation, with the outcome that keeps the metric honest.
The answer, in short
- Decision
- Don’t make “generated a first plan” the goal. Propose “cooked a planned recipe in week 1” as the candidate activation metric, and validate it against week-4 activity.
- Because
- Plan generation jumped from 52% to 71% while week-4 activity slipped from 20.8% to 20.0%; in July, users who cooked returned at 60% and plan-makers who didn’t cook at 15%.
- What would change it
- A holdout or later cohorts showing that more cooking doesn’t raise week-4 activity, or that plan generation does track it.
Common traps
- Picking the metric that just moved.
- Treating a correlation as a lever.
- Comparing two cohorts as if they were a test.
- Setting a goal that AI can raise without users getting value.
Check your answer
- Did I name the outcome the metric must predict?
- Did I check whether the jump reached that outcome?
- Did I separate association from cause?
- Did I propose a specific metric and how to validate it?
Want feedback on how you’d answer? Practice “Choose the metric that predicts the outcome” with an AI interviewer on Socratify.
Last reviewed Nov 9, 2026