Causal Inference & Experiments

Interview-depth causal reasoning for the ML track: conditional independence and d-separation, confounding and Simpson's paradox, trustworthy A/B testing, and identification beyond RCTs — with the untestable assumption behind every method.

1 — Conditional independence, colliders & d-separation (s7)

Conditional independence X ⊥ Y ∣ Z means that, once Z is known, X carries no extra information about Y: P(X, Y ∣ Z) = P(X ∣ Z) · P(Y ∣ Z). A DAG encodes such independencies through three canonical structures:

• Chain (mediator) X → Z → Y: X and Y are associated, but X ⊥ Y ∣ Z — conditioning on the mediator Z blocks the path.
• Fork (common cause / confounder) X ← Z → Y: again X ⊥ Y ∣ Z — conditioning on the confounder removes the spurious association it created.
• Collider (common effect) X → Z ← Y: here X and Y are marginally independent, but conditioning on the collider Z (or on any descendant of it) opens the path and induces a spurious association.

Key trap. Conditioning on a collider does the opposite of conditioning on a confounder: it creates dependence rather than removing it — selection / collider bias, also known as Berkson's paradox (e.g. among hospitalised patients two unrelated diseases appear negatively correlated because admission is the collider you conditioned on).

d-separation is the graphical rule that reads all these independencies off a DAG at once: a path is blocked if it contains a chain or fork whose middle node is in Z, or a collider whose node (and all descendants) are not in Z. If every path between X and Y is blocked given Z, then X ⊥ Y ∣ Z.

2 — Confounding and Simpson's paradox (s17)

A confounder is a variable that influences both the treatment and the outcome. It opens a back-door path T ← C → Y, so the naive association between T and Y mixes the causal effect with the confounder's — biasing the estimate. You de-confound by:

• conditioning / stratification or regression adjustment on the confounder,
• matching (compare treated and control units with similar covariates), or
• design — randomisation, which severs C → T so no confounder can bias T.

Simpson's paradox is confounding made vivid: an association that holds within every subgroup reverses when the subgroups are pooled, because an imbalanced confounder dominates the aggregate (the classic Berkeley admissions and kidney-stone examples). The direction of the "true" effect is not a statistical question — the causal graph decides which table to trust.

The corollary — "don't control everything." Adjusting for more variables is not always safer: you must never adjust for a collider or a mediator. Conditioning on a collider opens a spurious path (Section 1); conditioning on a mediator removes part of the very effect you want to measure. Control the variables the causal graph identifies as back-door blockers (the confounders), and leave the others alone. Simpson's paradox warns that pooling or splitting on the wrong variable can flip your conclusion entirely.

3 — Trustworthy A/B testing (s18)

Randomisation is the whole point: assigning treatment at random makes it independent of all confounders (observed and unobserved), so the simple difference in group means is an unbiased estimate of the average causal effect — no adjustment needed.

Getting an A/B test right is mostly about the traps around that clean idea:

• Power / sample-size up front. Fix the minimum detectable effect, α, and target power (1 − β, usually 80%) before launch; n ∝ σ²/δ². Under-powered tests mostly produce noise.
• Don't peek. Repeatedly checking significance and stopping at the first "p < 0.05" inflates the false-positive rate far above α. Use a fixed horizon decided in advance, or proper sequential / always-valid testing (group-sequential boundaries, mSPRT).
• Variance reduction. CUPED (regress out a pre-experiment covariate) and stratification / blocking shrink σ, buying power at fixed n.
• Validity checks. Watch guardrail metrics (don't win one metric by wrecking another), run an A/A test and sample-ratio-mismatch check for assignment bugs, account for novelty / primacy effects over time, and respect SUTVA — no interference between units (network effects, marketplace cannibalisation) or the independence assumption breaks.

4 — Causal inference beyond RCTs (s34)

When randomisation is impossible or unethical, we identify causal effects from observational data — but only under an untestable identifying assumption. Each classic design trades a different such assumption:

• Propensity-score methods — model the treatment probability e(x) = P(T = 1 ∣ X = x), then match or weight (IPW) on it to balance covariates. Valid under ignorability / unconfoundedness (no unobserved confounders) plus overlap / positivity (0 < e(x) < 1 for all x). The propensity score collapses many covariates into one balancing scalar — but it can do nothing about confounders you never measured.
• Instrumental variables — find an instrument Z that affects the outcome only through the treatment. Valid under relevance (Z moves T) and the exclusion restriction (Z has no other path to Y, and no confounder of Z–Y). Instrumental variables recover a causal effect (the LATE for compliers) even with unobserved confounding — the price is the exclusion restriction, which the data cannot verify.
• Difference-in-differences — compare the before/after change in a treated group against a control group's change, cancelling fixed differences. Difference-in-differences is valid under the parallel-trends assumption: absent treatment, the two groups would have moved in parallel.
• Regression discontinuity — when treatment switches at a cutoff in a running variable, compare units just above vs just below. Valid under continuity of potential outcomes at the cutoff, so units near the threshold are "as good as randomised" — but the estimate is local to the cutoff.

Interview one-liners

• Confounder vs collider: condition on a confounder to remove spurious association; condition on a collider and you create it (Berkson).
• d-separation reads conditional independencies off the DAG — block chains/forks by their middle node, and don't condition on colliders.
• Simpson's paradox: a within-group association flips on pooling — the causal graph, not the numbers, says which table to believe.
• A/B testing: randomisation kills confounding, but peeking inflates false positives and SUTVA/interference can invalidate the difference in means.
• Beyond RCTs: propensity scores need unconfoundedness+overlap, instrumental variables need the exclusion restriction, difference-in-differences needs parallel trends, RDD needs continuity — each an untestable trade.

Rigorous references for causal inference and experimentation.