Applied Case — The Leakage & Bias-Variance Clinic

P5.applied-cases.04 · Audience: guest, it-ml, language-pro · Prerequisites: Applied Case — The Honest Tabular Model, End to End

Applied Case — The Leaderboard Challenge The Perceptron

LLM grading (this page):Real gradingOff — no LLM calls

The end-to-end case built the discipline; this clinic shows what its absence looks like — from the inside. Three notebooks, each scoring far too well, each broken in a different way. The diagnosis pattern is always the same: a score that good is a symptom. Your job in each is to name the leak, fix it, and reproduce the honest number.

Step 1 / 5 — Why great scores should scare you

Every leak has the same anatomy: information that will not exist at prediction time is present at training time. The model is not wrong to use it — gradient descent has no ethics — the evaluation is wrong to reward it. And because the leak inflates precisely the number you use to judge the model, the better the score looks, the harder the production collapse.

Leak	Mechanism	This clinic's numbers
Target leak	A feature caused by the outcome	0.95 → honest 0.80
Duplicate leak	Twin rows straddle the split	0.80 → honest 0.61
Temporal leak	Random split hands the model the future	0.84 → honest 0.73

In bias-variance language (Model Evaluation & Selection): a leak fakes a variance reduction — the model looks like it generalises because the "unseen" data secretly is not. The honest fix usually raises measured error while lowering deployed error. That trade is always worth taking, and explaining why is a favourite senior-interview probe.

📚 Go Further

Type	Resource
In-app	P5.applied-cases.01 — The Honest Tabular Model — the discipline whose absence this clinic dissects
In-app	P5.ml-methods.05 — Time Series & Data Issues — drift, regimes and time-respecting evaluation
In-app	P5.ml-methods.06 — Model Evaluation, Selection & Interpretability — leakage, generalisation and the bias-variance frame
External	scikit-learn user guide — Common pitfalls and recommended practices: data leakage
External	Kapoor & Narayanan (2023), Leakage and the reproducibility crisis in ML-based science — the taxonomy this clinic teaches, measured across 294 published papers

Ask the mentor about this module

Ask a question about this content. The mentor explains and grounds its answer in what you are studying; asking is recorded as a learning signal, not a grade.

Ctrl/Cmd + Enter to send

🎓 Practice ladder

4 graded rungs · ~68 min

Four broken notebooks, one per trap — each with the seeded dataset, the inflated score reproduced in front of you, and hidden tests that only pass when the leak is found, fixed, and the honest number lands in the reference band. The traps run in-browser (Pyodide) — instant, no kernel queue. The senior rung hides the subtlest species: every step looks correct, yet pure noise scores 71%.

Rung 1 — The 0.99 notebook: find the target leak

Loading exercise…

Rung 2 — Twins in both splits: duplicate leakage

Loading exercise…

Rung 3 — Tomorrow in today's training set: temporal leakage

Loading exercise…

Senior rung — The leak the clinic has not shown you

Loading exercise…

✅ Key takeaways

Every leak is the same disease: information unavailable at prediction time was available at training time — and the evaluation rewarded it.
Target leak: ask of every column, could this exist before the outcome? (0.95 → honest 0.80 once the refund column goes.)
Duplicate leak: dedupe before splitting, and split by entity when rows share one (0.80 → honest 0.61 — memorisation credit, refunded).
Temporal leak: when rows have a time order, the split is a wall in time — in CV and tuning too (0.84 → honest 0.73).
A fixed leak lowers the measured score and raises the deployed one; reaching for a bigger model to win the points back is buying variance with borrowed honesty.
The meta-rule: count the decisions your evaluation number has already made — a metric that picked features, model, threshold and stopping point has confessed under torture (s52's p-hacking lesson, in ML clothes).

Next: P5.applied-cases.02 — Clustering & Segmentation — drops the labels entirely; then P5.applied-cases.03 turns this pillar's discipline into a leaderboard challenge. Or step across to P13 — ML in Production — where case 01's reference model gets packaged, served and drift-monitored.

Try it yourself

A scratch console for this page's ideas — ungraded, nothing you run here is recorded.

Scratch console

A scratch console with the scientific stack (pandas, numpy, scikit-learn). Runs on the server — no network, resource-limited and measured.

Output appears here.

Where next?

Continue

The Perceptron →

Neural-Network Foundations · P6

Go up a level