The Pitfalls Catalogue

P8.experimentation.04 · Audience: guest, it-ml, language-pro · Prerequisites: Designing an A/B Test

Reading the Readout Confounders & Causal Diagrams

LLM grading (this page):Real gradingOff — no LLM calls

Randomization makes the comparison fair; the design makes it big enough; honest reading turns it into a decision. And yet the most confident wrong conclusions in industry come from experiments that did all three — then were quietly corrupted by a handful of recurring mistakes. Each one manufactures significance that was never really there. This module is the catalogue: know these five by name and you will catch most of the false wins before they ship.

Step 1 / 5 — Peeking: the moving finish line

The most seductive pitfall. You watch the dashboard daily, and the moment the p-value dips below 0.05 you declare victory and stop. It feels responsible — why waste traffic once you have your answer? It is in fact one of the surest ways to ship pure noise.

ⓘ Concept: Peeking (optional stopping)

fixed sample + one look = 5% false positives; peek daily + stop on first hit ≫ 5%

Why it matters — A p-value wanders as data accumulates. Even when there is no real effect at all, that random walk will dip below 0.05 at some point in a long-enough test — check often enough and you are guaranteed to catch it on a lucky day. Stopping the instant you see significance converts that guaranteed fluke into a 'win'. The false-positive rate of a peeked test is far above the 5% the 0.05 threshold promises.

The fix is the discipline the design module already demanded: fix the sample size and stop time in advance, and read the result once, at the end. If you genuinely need to look early — and sometimes you do — there are sequential testing methods that budget the error across many looks, but the naive "stop when green" is never among them.

🗣️ From a linguist's perspective: Best of one, not best of seven

Judge a translation by one agreed test passage and you get an honest verdict. Let the vendor keep submitting passages until one scores well, then quote that score, and you have measured luck, not skill. Peeking is exactly best-of-seven dressed as best-of-one: the more attempts you allow, the better the fluke you are guaranteed to find.

✅ Key takeaways

Peeking — stopping the moment the p-value dips below 0.05 — turns a guaranteed random fluke into a 'win'. Fix the sample size and stop time in advance and read the result once.
Multiple comparisons guarantee a false winner: twenty innocent tests have a ~64% chance of one crossing 0.05. Correct the threshold (Bonferroni) or, better, pre-declare a single primary metric.
A sample-ratio mismatch means randomization broke — a chi-square check on the split catches it. It usually hides a real bug and invalidates the whole test, so check it before reading the lift; the same drift trips the alarm only once the sample is large.
Novelty effects and seasonality collisions produce effects that are real in the test but do not generalise — the lift fades as novelty wears off or vanishes when the season ends. Run whole cycles and plot the effect over time.
Audit every 'we won' against five checks: peeking, multiple comparisons, SRM, novelty, seasonality. Passing all five is what earns a conclusion; the causal-methods track takes over when randomization is impossible.

Next: P8.causal-methods.01 — Confounders & Causal Diagrams

Ask the mentor about this module

Ask a question about this content. The mentor explains and grounds its answer in what you are studying; asking is recorded as a learning signal, not a grade.

Ctrl/Cmd + Enter to send

🎓 Practice ladder

4 graded rungs · ~39 min

Hunt the pitfalls in the wild: judge whether an early stop was justified, build the sample-ratio-mismatch check by hand, watch multiple comparisons inflate false positives and tame them, then audit a deliberately flawed readout end to end.

Rung 1 — Catch the peeker (explorer)

Loading exercise…

Rung 2 — Sample-ratio mismatch

Loading exercise…

Rung 3 — Multiple-comparison inflation

Loading exercise…

Rung 4 — Audit this readout (senior)

Loading exercise…

Try it yourself

A scratch console for this page's ideas — ungraded, nothing you run here is recorded.

Scratch console

A scratch console with the scientific stack (pandas, numpy, scikit-learn). Runs on the server — no network, resource-limited and measured.

Output appears here.

Where next?

Continue

Confounders & Causal Diagrams →

Causal Methods · P8

Go up a level