Difference-in-Differences
P8.causal-methods.03 · Audience: guest, it-ml, language-pro · Prerequisites: Confounders & Causal Diagrams
Matching hit a wall: it cannot balance a confounder you never measured. This module climbs the wall a different way. When a policy, a launch, or a price change hits one group and not another, you can compare the two groups' change over time rather than their levels — and if both groups were drifting in parallel beforehand, that comparison cancels every stable difference between them, measured or not. This is difference-in-differences (DiD), the workhorse of quasi-experimental evaluation, and its whole engine is a 2×2 table you can fill in by hand.
A city raises its minimum wage in January. In December its restaurants employed a certain number of people; in March they employ 20 more per chain. Did the wage rise cause +20 jobs? You have two tempting single-difference answers, and both are traps.
Before-vs-after in the treated city alone credits the policy with everything that changed — including the spring hiring season, the wider economy, and every other force that would have moved employment anyway. Treated-vs-control after the change alone credits the policy with every standing difference between the two cities — one has always been bigger, richer, more touristy. Each single difference confounds the treatment with something else.
ⓘ Concept: Why a single difference confounds
Why it matters — Before/after cannot separate the treatment from time itself; treated/control cannot separate the treatment from pre-existing group differences. Every naive policy evaluation is one of these two, and each silently attributes a background trend or a fixed gap to the intervention. DiD's insight is that using both differences lets each one cancel the other's confounder.
Going deeper (technical) — when even a control group won't hold still: instrumental variables, lite
Interview-grade notes for the technical (it-ml) track.
The problem DiD can't touch. DiD leans on a clean before/after and a parallel control. Sometimes you have neither — treatment is taken up by choice, right now, and the choosers differ from the non-choosers on unmeasured things (the classic endogeneity of a self-selected treatment). An instrumental variable (IV) buys causal traction from a different direction: an outside source of variation in treatment that is as-good-as-random.
What makes a valid instrument Z. Three conditions, and the second and third are assumptions you argue, not test:
- Relevance — Z actually shifts treatment:
Z → Tis real and strong. (This one you can check in the data.) - Exclusion — Z affects the outcome only through treatment: no arrow
Z → Yexcept the one that passes through T. On the DAG, Z must have no other path to Y. - As-good-as-random (independence) — Z shares no common cause with Y; it is unconfounded, like a coin flip you found in the wild.
A canonical instrument is an encouragement design: randomly send some people a reminder to take the treatment. The reminder (Z) nudges take-up (relevance), plausibly affects the outcome only by changing whether they took the treatment (exclusion), and was randomized (independence).
The Wald estimator — IV by hand. With a binary instrument, the effect is a ratio of two differences you can compute directly:
effect = (Ȳ | Z=1 − Ȳ | Z=0) / (T̄ | Z=1 − T̄ | Z=0)
= reduced-form effect of Z on Y / first-stage effect of Z on T
The numerator is how much the outcome moved between the encouraged and non-encouraged groups; the denominator is how much treatment uptake moved between them. Dividing rescales the "intention" effect of the nudge into the effect of treatment itself. If a reminder raised uptake by 40 points and lifted the outcome by 8 points, the implied treatment effect is 8 / 0.40 = 20 — the reminder's effect, "grossed up" by how much treatment it actually bought.
The LATE caveat. IV does not recover the average effect over everyone. It recovers the Local Average Treatment Effect — the effect among compliers, the subpopulation whose treatment status the instrument actually moved (the people who took the treatment because they were encouraged). Always-takers and never-takers are invisible to Z, so they contribute nothing to the estimate. Reporting a LATE as if it were the population effect is the standard IV overclaim; a weak instrument (tiny first stage) makes the ratio's denominator small and the estimate wildly unstable — the weak-instrument problem that closes most IV interview questions.
Ask the mentor about this module
Ask a question about this content. The mentor explains and grounds its answer in what you are studying; asking is recorded as a learning signal, not a grade.
🎓 Practice ladder
3 graded rungs · ~27 minNow run the method yourself. Each rung is a three-panel workspace: instructions on the left, a code editor in the middle, output and test results on the right. Run checks the visible tests; Submit grades against hidden edge cases. The explorer rung predicts the DiD from four cell means; the middle rung computes it from a bundled panel; the senior rung injects a pre-trend into the control series and measures how badly it biases the naive estimate — proving the parallel-trends assumption by sabotaging it.
Rung 1 — The 2x2 table (explorer)
Loading exercise…
Rung 2 — Compute DiD from the panel
Loading exercise…
Rung 3 — Break parallel trends (senior)
Loading exercise…
Try it yourself
A scratch console for this page's ideas — ungraded, nothing you run here is recorded.
Scratch console
A scratch console with the scientific stack (pandas, numpy, scikit-learn). Runs on the server — no network, resource-limited and measured.
Output appears here.
Where next?
Later in Causal Methods