Designing an A/B Test

P8.experimentation.02 · Audience: guest, it-ml, language-pro · Prerequisites: Why Randomize?

LLM grading (this page):Real gradingOff — no LLM calls

Randomization guarantees a fair comparison — but only of a test that was designed to answer something. Before a single user is split, four decisions are already made, and getting any one of them wrong quietly wastes the whole experiment: what you measure, whom you split, how long you run, and how big the test must be to notice the effect at all. This module is about making those four decisions on purpose, before the coin is flipped.

Step 1 / 5 — One metric, decided first

An experiment needs a single primary metric chosen before the data arrives — the one number that decides ship-or-not. Naming it in advance is not bureaucracy; it is the only defence against the human habit of hunting through a dozen metrics after the fact and celebrating whichever one happened to move.

Alongside it sit a few guardrail metrics — things that must not get worse even if the primary metric improves (page load time, error rate, unsubscribe rate). The primary metric says "did it help?"; the guardrails say "at what cost?".

ⓘ Concept: Primary metric and guardrails

1 primary metric (ship decision) + a few guardrails (must-not-harm)

Why it matters — With no pre-declared primary metric, every experiment 'succeeds', because some metric always moves by chance. Deciding the one number that matters — and the handful that must not degrade — before launch is what keeps the test an honest test rather than a fishing expedition. The pitfalls module shows exactly how fishing across many metrics manufactures fake wins.

Prefer a metric close to the value you actually want. "Clicks on the new button" is easy to move and easy to fool yourself with; "did the user complete a purchase" is harder to move and far harder to fake.

✅ Key takeaways

Choose one primary metric — the number that decides ship — before the data arrives, with guardrails for what must not get worse. A pre-declared metric is the defence against celebrating whichever number happened to move.
The unit of assignment must cover the whole experience being measured. Randomize per user by default; drop to a coarse cluster only when users influence each other — and pay for it in lost power.
Power is the chance of detecting a real effect; the MDE is the smallest effect the test can catch. Power rises with sample size and effect size, falls with noise — decide the MDE first, then let it set the sample size.
You can simulate power instead of deriving it: run many synthetic experiments and count how often they reach significance. The panel shows the curve climbing to 80% as the sample grows.
Run whole cycles (usually full weeks) and fix the stop time in advance. Hitting the sample target early is not permission to stop — that is the peeking trap the next track dissects.

Next: P8.experimentation.03 — Reading the Readout

Ask the mentor about this module

Ask a question about this content. The mentor explains and grounds its answer in what you are studying; asking is recorded as a learning signal, not a grade.

Ctrl/Cmd + Enter to send

🎓 Practice ladder

4 graded rungs · ~33 min

Feel the sizing trade-off with your own hands: predict which effect needs the fewest users, simulate a power curve from scratch, invert it to the sample size that reaches 80% power, then reason about the duration trap a weekly cycle sets.

Rung 1 — Bigger lift, smaller sample (explorer)

Loading exercise…

Rung 2 — Simulate the power curve

Loading exercise…

Rung 3 — Find the sample size

Loading exercise…

Rung 4 — The duration trap (senior)

Loading exercise…

Try it yourself

A scratch console for this page's ideas — ungraded, nothing you run here is recorded.

Scratch console

A scratch console with the scientific stack (pandas, numpy, scikit-learn). Runs on the server — no network, resource-limited and measured.

Output appears here.

Where next?

Continue

Reading the Readout →

Experimentation · P8

Later in Experimentation

This module unlocks

The Pitfalls Catalogue

Go up a level