ML in Production (method view)

P5.ml-methods.07 · Audience: guest, it-ml, language-pro · Prerequisites: Model Evaluation, Selection & Interpretability

Model Evaluation, Selection & Interpretability Evaluation Metrics for NLP

Production ML one idea at a time: drift monitoring, the MLOps stack and training-serving skew, metrics vs business impact, statistical-rigour failures, and recommenders with cold-start — each with a concrete worked example. Interview depth for the data-scientist track. Back to the foundations: ML6 — Evaluation & Model Selection, and Advanced Probability.

Framing. In production the model is the easy part — the data moves, the offline/online paths diverge, and the proxy metric gets gamed. Each step turns one production failure into numbers.

ⓘ Concept: Method view here — case studies in the P13 pillar

Read this module for the concepts and the interview framing; head to the P13 pillar (Production ML & MLOps) when you want to build the infrastructure — deployment pipelines, feature stores, eval harnesses, and monitoring dashboards — around them.

Why it matters — This module is the method-level view of production concerns: what drift, skew, and Goodhart failures are and how to reason about them in an interview. The Production ML & MLOps pillar (P13) goes deep on the same topics with hands-on case studies — packaging a model, serving it, and wiring up drift monitoring — and its modules reference this one rather than re-teaching it.

Step 1 / 5 — Drift & Feedback Loops (s35)

1 — Monitoring: data drift, concept drift, feedback loops (s35). Data drift = the input $p(x)$ shifts while $p(y \mid x)$ holds; concept drift = the relationship $p(y \mid x)$ itself changes (the dangerous one — only delayed-label performance catches it). Watch inputs with the population stability index (PSI) or KS test; watch the relationship with performance on labels as they arrive.

Worked example — a PSI alert on an input feature. Compare a feature's live distribution against the training reference by bucket:

PSI = sum( (live% - ref%) * ln(live% / ref%) )  over buckets

rule of thumb:  < 0.10      no shift
              0.10-0.25   moderate shift -> investigate
              > 0.25      major shift -> retrain

measured PSI = 0.25  -> major drift, trigger a retrain

A PSI of 0.25 trips the major-shift threshold, so the input distribution has moved far enough to retrain. But PSI only sees inputs — if fraud tactics evolve (concept drift) with $p(x)$ unchanged, PSI stays quiet and only delayed-label AUC reveals it. Beware feedback loops too: a recommender that only shows popular items starves the tail, and future training data confirms the bias.

Note. Monitoring is the online half of a loop whose offline half is the eval harness: drift here is the trigger to refresh the eval set and re-run evaluation before retraining. The evaluation infrastructure for that loop — eval harnesses and experiment tracking — lives in the P13 pillar, Production ML & MLOps.

⚡ Interview Ref — the quick-scan Reference face

Interview-depth production ML for the data-scientist track: drift monitoring and feedback loops, the MLOps stack and training-serving skew, aligning metrics with business impact, statistical-rigour failures, and recommenders — with the assumptions and the 'why' framing. Back to the foundations: ML6 — Evaluation & Model Selection, and Advanced Probability.

1 — Monitoring: data drift, concept drift, feedback loops (s35)

Once a model is live the world moves underneath it. Distinguish the failure modes:

Data drift (covariate shift) — the input distribution shifts, p(x) changes while the learned relationship p(y | x) stays fixed (new user segment, seasonality, a changed upstream sensor). The model is still "correct" but now extrapolates outside its training support.
Concept drift — the relationship itself changes: p(y | x) moves even if p(x) looks stable (fraud tactics evolve, a pricing policy changes buyer behaviour). This is the dangerous one — the mapping the model encodes is now wrong. Watching only inputs misses it, so concept drift demands label-based performance monitoring.
Label / prediction drift — the marginal p(y) or the distribution of the model's own outputs shifts; a cheap early proxy when true labels are delayed.

Detection. For inputs, compare live vs reference distributions with the population stability index (PSI), KS test, or per-feature histograms/embeddings. For the relationship, track performance on delayed labels (AUC, error) as ground truth arrives — the only direct signal for concept drift.

Feedback loops. Beware when the model's own actions change future data: a recommender that only shows popular items starves the tail of impressions, so future training data confirms the bias (a self-fulfilling loop). Left unmonitored these degrade silently and entrench bias.

Respond with alerting thresholds on drift metrics, scheduled or trigger-based retraining, shadow evaluation before promotion, and logging exploration to break feedback loops.

2 — What MLOps covers and training-serving skew (s62)

MLOps is DevOps adapted to the fact that an ML system is code + data + model, all three of which drift. It covers:

Versioning — code, data, and model artefacts all versioned so any prediction is reproducible (DVC, model registries, dataset snapshots).
Reproducible pipelines — orchestrated, parameterised training runs with tracked lineage.
CI/CD for models — automated tests (data validation, model quality gates), then automated deployment.
A feature store — a central, versioned place to define and serve features, giving the same transform to training (offline/batch) and serving (online) code.
Deployment patterns — batch (precompute), online (real-time), shadow (mirror live traffic, don't act), and canary (small % rollout with automatic rollback).
Monitoring — drift, latency, and quality (section 1).
Governance — access control, audit trails, lineage, and model documentation.

Training-serving skew. The classic silent killer: training-serving skew is any difference between how features are computed offline for training vs online for serving — a subtly different transform, a unit mismatch, or stale features (training on fresh data, serving on an hours-old cache). Metrics look fine offline yet production accuracy quietly degrades because the model sees inputs it was never trained on. The structural fix is a shared feature store or a single library of feature transforms invoked identically in both paths, so offline and online computations are provably the same.

3 — Aligning metrics with business impact; Goodhart's law (s36)

A model metric is only ever a proxy for the value the business actually wants. Optimising AUC, log-loss, or click-through rate is a stand-in for revenue, retention, or user well-being — and the two can diverge sharply.

Goodhart's law — "when a measure becomes a target, it ceases to be a good measure." The moment you optimise a proxy hard, the system finds ways to raise the number without delivering the underlying value: a CTR-maximising feed learns clickbait; a "time-on-site" target breeds addictive dark patterns. Goodhart is the reason a metric that correlated with value in observation stops correlating once it is the optimisation target.

Defences:

Pick a single north-star metric tied to real value, plus guardrail metrics (latency, complaint rate, churn, fairness) that must not regress while you optimise the proxy.
Validate causally with online experiments (A/B tests) — offline gains often don't materialise online.
Watch for proxy gaming and guardrail regressions explicitly; treat a proxy win with a guardrail loss as a net loss.

4 — p-hacking, pre-registration & the replication crisis (s52)

Running experiments analytically honestly is harder than it looks.

p-hacking — trying many analyses, subgroups, or stopping rules and reporting only the ones that cross p < 0.05. The garden of forking paths captures the subtler version: even without conscious cheating, data-dependent choices at each fork inflate the false-positive rate far above the nominal α.
HARKing ("Hypothesising After the Results are Known") — presenting a post-hoc pattern as if it were a pre-planned prediction, which converts exploratory noise into a spurious "confirmed" hypothesis.
The replication crisis — the empirical result: large swaths of published findings fail to replicate, because the literature is filtered for significance rather than truth (publication bias + the above).

Defences:

Pre-registration — commit the hypothesis, metric, sample size, and analysis plan before seeing outcomes, separating confirmatory from exploratory claims.
A locked holdout / confirmatory test that is touched only once.
Multiple-comparison correction (Bonferroni, Benjamini–Hochberg FDR) when many hypotheses are tested.
Report effect sizes and confidence intervals, not just p-values — significance is not magnitude.
Reproducible analysis (versioned code + data + seeds) so a result can be re-run exactly.

5 — Recommendation systems and the cold-start problem (s60)

Three families of recommender:

Collaborative filtering — learn from the user–item interaction matrix alone. Matrix factorisation decomposes it into latent user and item vectors, r̂ᵤᵢ = pᵤᵀ qᵢ, so "users who liked what you liked" is captured geometrically. Powerful, but needs interaction history.
Content-based — recommend items whose features (text, genre, embeddings) resemble what a user already engaged with. Works from item attributes, so it survives sparse interactions.
Hybrid — combine both (and context) to get collaborative signal where data is dense and content signal where it is sparse.

The cold-start problem. A pure collaborative system is helpless when there is no interaction history — a brand-new user or a just-added item has no rows/columns to factorise. Mitigations:

Content features — fall back to item/user attributes and embeddings so a new item can be matched before it has any clicks.
Popularity fallbacks — serve trending/top items until personal signal accrues.
Onboarding — explicitly elicit preferences (pick genres, follow topics) to seed the profile.
Contextual bandits — treat recommendation as exploration vs exploitation, deliberately probing uncertain new items (with a Hoeffding/UCB-style bonus) to gather the interaction data the model lacks.

Interview one-liners

Data drift = inputs p(x) shift; concept drift = the p(y | x) mapping changes — the latter needs delayed-label performance monitoring, not just input tests.
Training-serving skew is offline vs online feature computation diverging (or stale features); a shared feature store / identical transforms is the structural fix.
Goodhart's law: optimise a proxy hard and it stops tracking value — use a north-star plus guardrail metrics and validate online.
p-hacking / the garden of forking paths inflate false positives — pre-registration and a locked holdout are the antidotes; the replication crisis is what happens without them.
Cold-start: no history for new users/items — lean on content features, popularity fallbacks, onboarding, and bandit exploration.

📚 Go Further

Practical references for production ML and MLOps.

Type	Resource
Book	Huyen, Designing Machine Learning Systems — drift, feature stores, deploy
Book	Kleppmann, Designing Data-Intensive Applications — data pipelines
Paper	Sculley et al., Hidden Technical Debt in Machine Learning Systems
In-app	ML4 — Causal Inference (P5.ml-methods.04), for the online experiments that validate these metrics
In-app	ML6 — Evaluation & Model Selection (P5.ml-methods.06), for the offline metrics these systems monitor
In-app	P13 — Production ML & MLOps pillar, for the case-study deep dives (packaging, serving, drift monitoring)

✅ Key takeaways

Data drift means the inputs p(x) shift; concept drift means the mapping p(y | x) itself changes — PSI and KS tests watch the inputs, but only performance on delayed labels catches concept drift, and feedback loops silently entrench bias.
MLOps versions code + data + model with reproducible pipelines, CI/CD gates, and shadow/canary deployment; training-serving skew — features computed differently offline vs online — is the classic silent killer, and a shared feature store is the structural fix.
Every model metric is a proxy for business value, and Goodhart's law says a proxy optimised hard stops tracking value — defend with a north-star metric, guardrails that must not regress, and online A/B validation.
p-hacking and the garden of forking paths inflate false positives far above α (20 null tests give a 64% chance of a false 'discovery') — pre-registration, a locked holdout, and multiple-comparison correction are the antidotes.
Collaborative filtering needs interaction history, so new users and items hit cold-start — mitigate with content features, popularity fallbacks, onboarding elicitation, and bandit exploration. This module closes the ml-methods track: from algorithms to what survives contact with production.

Next: P5.evaluation.01 — Evaluation Metrics for NLP

Ask the mentor about this module

Ask a question about this content. The mentor explains and grounds its answer in what you are studying; asking is recorded as a learning signal, not a grade.

Ctrl/Cmd + Enter to send

Try it yourself

A scratch console for this page's ideas — ungraded, nothing you run here is recorded.

Scratch console

A scratch console with the scientific stack (pandas, numpy, scikit-learn). Runs on the server — no network, resource-limited and measured.

Output appears here.

Where next?

Continue

Evaluation Metrics for NLP →

Evaluation · P5

Go up a level