Experiments & A/B Testing (step by step)
P1.stats-literacy.02 · Audience: guest, it-ml, language-pro · Prerequisites: Correlation, Causation & Confounding
Take it one idea at a time: why splitting users randomly is what makes a result trustworthy, the common ways experiments quietly mislead you, and how to ask for more evidence when the decision is riskier. Each step has a worked example. No maths required.
One mental model. An experiment earns trust when the only planned difference between two groups is the change you're testing. Split people randomly, decide the rules up front, and wait — then a difference in the outcome can be blamed on the change and nothing else.
An A/B test splits users randomly into a control group (the current experience) and a variant (the change you want to try). Randomization balances all the other factors — known and unknown — across the two groups, so any later difference can be attributed to the change itself.
Worked example — self-selected beta vs a random split
You want to know if a new editor speeds people up. Two ways to run it — only one is trustworthy:
| Approach | Who's in the variant | If the variant looks faster… |
|---|---|---|
| Opt-in beta (users choose) | keen power users who volunteered | maybe the tool — or maybe just keener users |
| Random assignment | a fair cross-section, by chance | it's the tool; groups started comparable |
The opt-in group is faster before you change a thing, because motivated users self-selected in — that's a correlation, not proof. The random split makes the groups a fair like-for-like comparison, which is what turns 'these moved together' into 'this change caused that'.
⚡ Interview Ref — the quick-scan Reference face
In plain language: why splitting users randomly is what makes a result trustworthy, the common ways experiments quietly mislead you, and how to ask for more evidence when the decision is riskier. No maths required.
One mental model. An experiment earns trust when the only planned difference between two groups is the change you're testing. Split people randomly, decide the rules up front, and wait — then a difference in the outcome can be blamed on the change and nothing else.
1 — Why randomization makes A/B results trustworthy (b14)
An A/B test splits users randomly into a control group (the current experience) and a variant (the change you want to try). Everyone is assigned by chance, not by choice.
Why the random split is the whole point: randomization balances all the other factors — known and unknown — across the two groups. Roughly the same mix of new and loyal users, devices, regions, and moods lands on each side.
- The groups start out comparable, so they are a fair like-for-like comparison.
- Any later difference in the outcome can be attributed to the change itself, not to who ended up where.
- This is what turns 'these two things moved together' into 'this change caused that'.
Without the random split you only have a correlation; with it, you have evidence of cause.
2 — Common ways A/B tests go wrong (b15)
Even a well-built test can mislead you. The usual traps:
- Peeking — checking the results over and over and stopping the moment they look significant. Peeking inflates false positives: look often enough and random noise will eventually cross the line by chance.
- Too-small samples / stopping early — too few people to tell a real effect from noise.
- Novelty and seasonality effects — people react to anything new, and holidays, paydays, or weekends distort short windows.
- Ignoring guardrail metrics — a guardrail is a health check (like complaints, refunds, or unsubscribes) you watch so a 'win' on one number isn't quietly harming another.
- Users leaking between groups — the same person sees both versions, or the groups influence each other, blurring the comparison.
The fix is discipline: fix the sample size and duration up front, pre-declare the metric you'll judge on, then wait for the test to finish before you look and decide.
3 — Matching the evidence standard to the stakes (b27)
Not every decision deserves the same burden of proof. Calibrate how much rigour you demand to the consequences of being wrong.
- A low-stakes copy tweak (a button label, a subject line) is cheap to get wrong and easy to reverse — a quick, light test is enough.
- A pricing or safety change is expensive or dangerous to get wrong — it deserves larger samples, longer runs, and a higher bar before you ship.
Ask for strength of evidence proportional to the decision's cost of error.
The senior instinct is not 'always demand maximum proof' nor 'ship on a hunch' — it's matching the evidence standard to the stakes, so effort goes where being wrong hurts most.
Interview one-liners
- An A/B test works because the random split balances everything else, so a difference can be blamed on the change — not on who ended up in each group.
- Avoid peeking: fix the sample size and duration up front, pre-declare the metric, watch your guardrail metrics, and wait.
- Match the evidence standard to the stakes — a copy tweak needs less proof than a pricing or safety change.
📚 Go Further
Plain-language explainers on experiments and trustworthy decisions.
| Type | Resource |
|---|---|
| Book | Trustworthy Online Controlled Experiments — Kohavi, Tang & Xu (2020) |
| Idea | Correlation is not causation — why the random split is what buys you cause |
| In-app | Metrics & evaluation, for choosing the metric before you run the test |
| In-app | Reading Numbers, for signal vs noise when you read a test's result |
Ask the mentor about this module
Ask a question about this content. The mentor explains and grounds its answer in what you are studying; asking is recorded as a learning signal, not a grade.
Where next?
Later in Data & Statistics Literacy