Reading the Readout

P8.experimentation.03 · Audience: guest, it-ml, language-pro · Prerequisites: Designing an A/B Test

Designing an A/B Test The Pitfalls Catalogue

The test ran, the coin did its work, and a dashboard now shows a lift: "+2.3%". The hard part is not computing that number — the Statistics pillar already taught the machinery that produces it. The hard part is deciding what the number licenses you to do. A lift is an estimate wrapped in uncertainty, and the same readout can mean ship it, do not ship it, or we still do not know — depending entirely on how you read it. This module is about that reading.

Step 1 / 4 — The lift is an estimate, not a fact

A readout of "+2.3%" is never the truth — it is one noisy draw from a process. Run the identical experiment again next week and you would get +1.8%, or +3.1%. The single number is a point estimate; what matters is the range of true lifts consistent with the data, its confidence interval.

ⓘ Concept: Read the interval, not the point

report: lift ± interval — the width is the message

Why it matters — A +2.3% point estimate with an interval of [+2.0%, +2.6%] is a confident win. The same +2.3% with an interval of [−1.5%, +6.1%] is a shrug — the data is equally consistent with a small loss and a large gain. Two experiments can show the identical headline lift and justify opposite decisions. Reading the point without the interval is how confident mistakes get shipped.

How that interval is actually computed — standard errors, the test statistic, the p-value that goes with it — is machinery this pillar deliberately does not re-derive. It belongs to Inference & Hypothesis Testing in the Statistics pillar, and the A/B specific walkthrough lives in Experiments & A/B Testing. Read those for how the interval is built; read on here for what to do once you have it.

✅ Key takeaways

A lift is a noisy point estimate. Read the confidence interval, not the single number — its width decides the outcome, and two experiments with the same headline lift can justify opposite decisions.
Statistical significance (distinguishable from zero) and practical significance (large enough to matter) are different questions. A decision needs both; a huge sample can make a worthless effect significant.
A flat readout from a well-powered test is a real, valuable finding — the effect is smaller than the MDE you cared about. A flat readout from an underpowered test says nothing: absence of evidence is not evidence of absence.
The interval, compared against zero and the MDE, maps onto ship / too-small / informative-null / inconclusive. 'Ship' is only one of four honest outcomes.
This module frames the decision; the test mechanics — p-values, confidence intervals, the tests themselves — belong to the Statistics pillar and are cross-linked, never re-derived here.

Next: P8.experimentation.04 — The Pitfalls Catalogue

Ask the mentor about this module

Ask a question about this content. The mentor explains and grounds its answer in what you are studying; asking is recorded as a learning signal, not a grade.

Ctrl/Cmd + Enter to send

Try it yourself

A scratch console for this page's ideas — ungraded, nothing you run here is recorded.

Scratch console

A scratch console with the scientific stack (pandas, numpy, scikit-learn). Runs on the server — no network, resource-limited and measured.

Output appears here.

Where next?

Continue

The Pitfalls Catalogue →

Experimentation · P8

Later in Experimentation

The Pitfalls Catalogue

Go up a level