Applied Case — The Leaderboard Challenge

P5.applied-cases.03 · Audience: guest, it-ml, language-pro · Prerequisites: Applied Case — The Honest Tabular Model, End to End

Applied Case — Clustering & Segmentation Applied Case — The Leakage & Bias-Variance Clinic

LLM grading (this page):Real gradingOff — no LLM calls

Every metric this pillar taught you was computed on data you could see. This case takes that away too — in the other direction from the clustering case: the labels exist, but the rows that grade you are locked on the organizer's side. Welcome to the competition format: public training data, hidden holdout, one number on a score board.

Step 1 / 4 — The format — public train, hidden holdout

A leaderboard task splits the world in two. The public partition mounts read-only in your kernel at/data/p5-case03/public/train.csv — 26,048 census rows with the income label, 24.1 % of them in the positive >50K class, real missing values in the work columns. The holdout partition (6,513 rows, same class mix, drawn by a stratified split the organizer seeded once and will never re-draw) exists only on the server. Run never mounts it. Only the hidden grading suite, on Submit, can touch it.

Partition	Rows	Positive rate	Who can read it
Public train	26,048	24.08 %	You, freely
Hidden holdout	6,513	24.07 %	The hidden suite, on Submit

Why so strict? Because every look you get at the holdout leaks into your choices — the exact disease the leakage clinic dissected, institutionalised away. A test set you can query at will stops being a forecast and becomes a validation set you are overfitting politely.

Kaggle competitions show a public leaderboard during the contest and a private one at the end — and the final ranking often reshuffles. What is the mechanism, and what does this case's design do about it?

📚 Go Further

Type	Resource
In-app	P5.applied-cases.01 — The Honest Tabular Model — the same dataset, walked end-to-end; its pipeline is this competition's baseline
In-app	P5.applied-cases.04 — The Leakage & Bias-Variance Clinic — why the holdout is locked away: every leak from that catalogue is a leaderboard exploit here
In-app	P5.evaluation.01 — Evaluation Metrics — ROC-AUC's construction, and why it is the imbalance-safe competition metric
External	scikit-learn user guide — HistGradientBoostingClassifier: the model family behind the reference solution's 0.9313
External	Kaggle's public vs private leaderboard documentation — the shake-up phenomenon this case's zero-feedback design removes

Ask the mentor about this module

Ask a question about this content. The mentor explains and grounds its answer in what you are studying; asking is recorded as a learning signal, not a grade.

Ctrl/Cmd + Enter to send

🎓 Practice ladder

3 graded rungs · ~75 min

Three kernel rungs on the real competition split — real files under /data/p5-case03/public, resource-metered grading. Build the local-validation harness, beat your own baseline with a full pipeline, then the finale: an open-ended leaderboard project where the hidden suite scores you on 6,513 rows you will never see, keeps your best, and a passing submission earns a design review against the rubric.

Rung 1 — Read the competition: public train, private validation

Loading exercise…

Rung 2 — Beat your baseline on your own validation split

Loading exercise…

Rung 3 — Leaderboard finale: beat 0.9098 on the hidden holdout (Project)

Loading exercise…

✅ Key takeaways

A leaderboard task is evaluation discipline made physical: the holdout cannot leak into your choices because you cannot read it.
The metric fits the data, not the habit: at 24 % positives, ROC-AUC compares probability rankings where accuracy rewards the base rate.
Your local validation split is the only scoreboard you can query freely — build it stratified and seeded, and trust it to forecast the holdout.
When local and holdout scores diverge, the divergence is a diagnosis: validation overfitting or a split that does not resemble the holdout.
Spend submissions like the scarce resource a real test set is — iterate locally, submit on evidence.
The pass floor is a checkpoint; the baseline is the game. Beating 0.9098 took a model-family change (trees over logistic), not another hyperparameter nudge.

Next: The P5 applied-cases track is complete — all four cases live. Next stop: P13 — ML in Production, where case 01's reference artefact gets packaged, served and drift-monitored.

Try it yourself

A scratch console for this page's ideas — ungraded, nothing you run here is recorded.

Scratch console

A scratch console with the scientific stack (pandas, numpy, scikit-learn). Runs on the server — no network, resource-limited and measured.

Output appears here.

Where next?

Continue

Applied Case — The Leakage & Bias-Variance Clinic →

Applied Cases · P5

Later in Applied Cases

Applied Case — The Leakage & Bias-Variance Clinic

Go up a level