Monitoring & drift

P13.operations.02 · Audience: guest, it-ml, language-pro · Prerequisites: Serving & scaling

Serving & scaling Applied LLMOps at scale

LLM grading (this page):Real gradingOff — no LLM calls

A model that passed every offline gate still decays in production — not because the code changed, but because the world did. Monitoring is the discipline of noticing that the live inputs (and outputs) no longer look like the data the model was trained and evaluated on, and turning that observation into a decision. P5.ml-methods.07 named drift as step 1 of the production loop; this module makes it measurable: PSI, the two-sample KS test, and the decision policy that converts a drift score into no-shift / monitor / retrain.

Step 1 / 4 — Three kinds of drift

"Drift" is not one thing. Production systems fail in distinct ways, and the fix differs for each — so the first job of a monitor is to name which distribution moved. Write the model as learning P(y | x) over inputs drawn from P(x).

Data drift (covariate shift) — P(x) changes while P(y | x) holds. A new hospital onboards, so the age mix of incoming requests shifts. The learned mapping is still correct; you are just seeing inputs the model saw rarely in training.
Concept drift — P(y | x) itself changes. The same input now has a different correct answer: a fraud pattern that used to be benign is now malicious. This is the dangerous one — the model is now wrong, not just under-sampled.
Label shift (prior shift) — P(y) changes while P(x | y) holds. The base rate of the positive class moves (an outbreak triples disease prevalence), which miscalibrates any threshold tuned to the old prior.

ⓘ Concept: Why you monitor inputs, not just accuracy

Ground-truth labels arrive late (a booking is confirmed days later) or never. Input drift is observable immediately — you have the features the moment a request lands. So monitoring watches the leading signal (P(x)) as an early warning, and confirms with the lagging signal (measured accuracy) once labels land.

This is the same measurement machinery as the offline eval harness in P13.ml-lifecycle.01, run at a different point in the lifecycle: the harness scores a candidate before ship; monitoring scores the live system after ship. A monitored drift is precisely the trigger to re-open the eval loop — refresh the eval set, re-run the gate, retrain. Offline and online are one loop, not two systems.

Ask the mentor about this module

Ask a question about this content. The mentor explains and grounds its answer in what you are studying; asking is recorded as a learning signal, not a grade.

Ctrl/Cmd + Enter to send

🎓 Practice ladder

3 graded rungs · ~34 min

Build the drift detector yourself, bottom-up: the Population Stability Index from raw bins, a binning-free KS drift test on top of SciPy, and the worst-feature action policy that turns those scores into a no-shift / monitor / retrain decision. Local-first — the kernel has NumPy and SciPy but no live feature store, so you implement the detector; a real monitor wires it to a windowed stream.

Rung 1 — Population Stability Index from scratch

Loading exercise…

Rung 2 — a two-sample KS drift test

Loading exercise…

Rung 3 — the drift-to-action policy (senior)

Loading exercise…

⚡ Interview Ref — the quick-scan Reference face

Three drifts: data/covariate (P(x) moves, mapping still right) · concept (P(y|x) moves, model now wrong) · label/prior (P(y) moves, thresholds miscalibrated). Monitor inputs because labels lag or never arrive.
PSI = Σ (a% − e%)·ln(a%/e%) over frozen reference bins. Bands: <0.1 stable · 0.1–0.2 moderate · ≥0.2 significant. Clamp fractions with an epsilon so empty bins don't blow up ln.
KS (two-sample) = max gap D between empirical CDFs + a p-value; no binning. Power scales with n — a big D on tiny samples still gives a large p (worked example: D=0.4, p=0.87 at n=5+5).
KS vs PSI: KS for continuous + enough samples + significance; PSI for categorical/bucketed, bounded interpretable magnitude, and large-n where KS flags every trivial shift. Neither sees concept drift alone — that needs labels.
Signal → action: worst feature wins → no-shift / monitor / retrain; retrain trigger re-opens the offline-eval gate (P13.ml-lifecycle.01), it does not auto-ship.
Ops trade-off: thresholds too low → alert fatigue → real drift ignored; too high → drift caught late. Require persistence across windows before paging.

📚 Go Further

Where the pieces of this module lead.

Type	Resource
In-app	P5.ml-methods.07 — Production ML & MLOps: drift is step 1 of the loop this module deepens
In-app	P13.ml-lifecycle.01 — Evaluation infrastructure: the offline half of the loop a retrain trigger re-opens
In-app	P13.operations.01 — the reliability/on-call foundations this monitoring plugs into
Docs	SciPy `stats.ks_2samp`; Evidently / NannyML drift monitors; the PSI credit-scoring literature

✅ Key takeaways

Name the drift before fixing it: data/covariate drift moves P(x) (mapping still right), concept drift moves P(y|x) (model now wrong), label shift moves P(y) (thresholds miscalibrated). Monitor inputs because labels lag or never arrive.
PSI = Σ (a% − e%)·ln(a%/e%) over frozen reference bins gives a bounded, interpretable magnitude (<0.1 stable · 0.1–0.2 moderate · ≥0.2 significant); guard empty bins with an epsilon so ln never blows up.
The two-sample KS test compares empirical CDFs with no binning and returns a statistic plus a p-value — but its power scales with sample size, so a large gap on a tiny window (D=0.4, p=0.87) is no evidence at all. Use KS for continuous features with enough data, PSI for categorical/bucketed features and large n.
A drift score is useless without a decision policy: worst-feature-wins → no-shift / monitor / retrain, with a retrain trigger that re-opens the offline-eval gate (P13.ml-lifecycle.01). Tune thresholds to balance alert fatigue against missed drift, and require persistence before paging.

Try it yourself

A scratch console for this page's ideas — ungraded, nothing you run here is recorded.

Scratch console

A scratch console with the scientific stack (pandas, numpy, scikit-learn). Runs on the server — no network, resource-limited and measured.

Output appears here.

Where next?

Continue

Applied LLMOps at scale →

LLMOps · P13

This module unlocks

Go up a level