P8 · Causal Inference & Experimentation
A/B at scale, uplift, quasi-experiments
Correlation is not the goal here — the counterfactual is: what would have happened otherwise. Randomize when you can, and when you cannot, identify the effect anyway with diagrams, matching and difference-in-differences.
Every model so far answered one kind of question: given what I can see, what will happen? This pillar asks the harder one: what would have happened otherwise? A hospital whose patients die more often may be the best in the region — it takes the sickest cases. A sales bump after a campaign may be the campaign, or may be the season the campaign launched in. Prediction cannot separate these; only a causal question can, and answering it honestly is a different discipline with its own tools.
The first track is the gold standard: the randomized experiment. When you can assign treatment by coin flip, confounders balance out in expectation and the difference you measure is the effect. But running a trustworthy A/B test is deceptively hard — you will design one properly (metric, unit of assignment, power) and then walk the catalogue of ways experiments quietly lie: peeking at results early, testing twenty metrics and celebrating the one that moved, sample-ratio mismatch, novelty effects.
The second track is for when you cannot randomize — which is most of the time. Confounders and the causal diagrams that tame them; matching, to compare like with like; difference-in-differences, which cancels a shared trend to recover an effect from observational data; and uplift, which asks not "did it work?" but "for whom?". These are the methods that turn "correlation is not causation" from a warning into a working craft.