Experimentation
A track of P8 · Causal Inference & Experimentation.
Randomize, power, run, read: designing A/B tests and dodging the pitfalls that quietly fabricate significance.
A randomized experiment is the closest thing data science has to a clean answer. Flip a coin for each user, show the treatment to one group, and any difference in outcomes is caused by the treatment — because randomization balances every confounder, known and unknown, in expectation. This track is about earning that clean answer, and about all the ways a careless experiment throws it away.
You will design an A/B test the way a trustworthy one is actually built: choosing the metric, the unit of assignment and the duration, and sizing it with power and the minimum detectable effect — simulated, not derived, so you watch power climb with sample size on the panel. Then the pitfalls catalogue: peeking at a running test until it crosses significance, testing many metrics and reporting the lucky one, sample-ratio mismatch that signals a broken assignment, and novelty effects that fade. Reading the readout — the uncertainty on the lift, practical versus statistical significance, and why a flat result is a real answer — leans on the hypothesis-testing machinery P1 owns; this track supplies the design and the judgement.