Why Randomize?

P8.experimentation.01 · Audience: guest, it-ml, language-pro

Backtesting & Anomaly Detection Designing an A/B Test

The Machine Learning domain so far has been about prediction: fit a model, evaluate it honestly, forecast the next value. This pillar drops a quieter assumption hiding underneath all of it — that prediction is the goal at all. The moment someone asks did our change actually cause the improvement? the question is no longer "what will happen" but "what would have happened otherwise". Observed correlation, however strong, stops being enough — and the fix, surprisingly, is not a better model but a better experiment.

Step 1 / 5 — A different question

Prediction asks: given what I can see, what is the most likely outcome? Causation asks something the data alone can never show: if I intervene — ship the feature, give the drug, change the price — how much does the outcome change because of my action, and not because of everything else going on?

The gap between the two is the whole of this pillar. A model can predict that people carrying umbrellas are more likely to see rain, and be completely right — yet confiscating umbrellas will not bring the sun out. The umbrella predicts rain without causing it. Confusing the two is not a beginner mistake; it is the single most expensive error in applied data work.

ⓘ Concept: Correlation is not causation

X predicts Y ≠ changing X changes Y

Why it matters — Every decision to act — launch, spend, treat, intervene — is a causal claim in disguise. If you justify the action with a correlation, you are betting that the pattern you measured will survive your interference. Very often it will not, because the pattern was produced by something you did not change and cannot control.

🗣️ From a linguist's perspective: The bilingual-child fallacy

Suppose bilingual children score higher on a certain reasoning test. Does teaching a second language cause sharper reasoning? Perhaps — but bilingual households also tend to differ in income, schooling, and dozens of other things that move the same score. The correlation is real; the causal claim needs far more than the correlation to stand up. A careful linguist and a careful data scientist face the identical trap, in the identical shape.

✅ Key takeaways

This pillar drops the assumption that prediction is the goal. The causal question is not what will happen, but what would have happened otherwise — and correlation, however strong, cannot answer it.
A confounder influences both who gets the treatment and the outcome. It can make a hospital that treats every patient better look worse overall — a raw comparison compares patient mixes, not hospitals.
The counterfactual — what would have happened to this same unit under the other condition — is never observable for an individual. Causal inference retreats to the average effect over a comparable group.
Randomization builds that comparable group by assigning treatment with a blind coin, balancing measured and unmeasured confounders alike in expectation. Any surviving outcome difference must come from the treatment.
Randomization guarantees a fair comparison, not a well-run or well-read one. The rest of this track designs, sizes, reads, and audits the experiment; the causal-methods track handles the case where you cannot randomize at all.

Next: P8.experimentation.02 — Designing an A/B Test

Ask the mentor about this module

Ask a question about this content. The mentor explains and grounds its answer in what you are studying; asking is recorded as a learning signal, not a grade.

Ctrl/Cmd + Enter to send

Try it yourself

A scratch console for this page's ideas — ungraded, nothing you run here is recorded.

Scratch console

A scratch console with the scientific stack (pandas, numpy, scikit-learn). Runs on the server — no network, resource-limited and measured.

Output appears here.

Where next?

Continue

Designing an A/B Test →

Experimentation · P8

Later in Experimentation

This module unlocks

Confounders & Causal Diagrams

Go up a level