Evaluation infrastructure & experiment tracking
P13.ml-lifecycle.01 · Audience: guest, it-ml, language-pro · Prerequisites: LLM & Generation Evaluation
A quality score computed once, in a notebook, on whatever data was handy, proves nothing a week later — you cannot tell whether the model improved or the evaluation itself drifted. That is why production teams treat evaluation as infrastructure rather than a notebook cell, and this module, which opens the Production ML and MLOps pillar, shows what the shift means in practice. The P5 evaluation track taught which metric to compute; here the metric becomes a system, with eval datasets frozen and versioned, every run tracked in a tool such as Weights and Biases, MLflow, or Braintrust, an automated gate that stops a model or prompt from shipping when its scores regress, and a model-graded harness whose judge is itself calibrated against human raters. The same signals then carry forward into production monitoring, where offline evaluation and online observation turn out to be two views of one thing.
P5.evaluation.01–03 give you the numbers — precision/recall/F1, BLEU/ROUGE, BERTScore, nDCG, LLM-as-judge scores. In production none of that is trustworthy unless it is reproducible and comparable across runs. That is what eval infrastructure buys you.
Three pillars turn a one-off metric into an eval system:
- Versioned eval datasets — the eval set is a dataset artefact with a version, not a notebook cell. Freeze it, hash it, and never let training data leak into it. A model only improves against a fixed target.
- Tracked runs — every evaluation is a logged run: which model/prompt/commit, which eval-set version, the resulting metrics, and enough metadata to reproduce it.
- A regression gate — before a model or prompt ships, an automated eval run must clear a bar (offline-eval gate), exactly like a test suite gates code.
ⓘ Concept: Mental model — eval is CI for models
The Doctolib System-Design interview (Q9) asks for exactly this stack: offline eval + online A/B + LLM-specific evaluation, tracked in W&B / MLflow / Braintrust. The next steps build each piece.
Ask the mentor about this module
Ask a question about this content. The mentor explains and grounds its answer in what you are studying; asking is recorded as a learning signal, not a grade.
Keeping your files on this device
Off by default. The mentor always gets your file; this only decides whether your own copy stays here. Copies live in this browser only - they do not follow you to another device, and clearing site data removes them.
🎓 Practice ladder
4 graded rungs · ~45 minBuild the experiment-tracking loop yourself — a file-based run tracker, a
primary-metric-with-guardrail selector, and an offline-eval regression gate —
then do it with the real tool: a fourth rung runs in the
mlops kernel and logs to a genuine MLflow store.
Local-first — the kernel has no network, so MLflow uses a local
sqlite: store, no server (ADR-011).
Rung 1 — a file-based run tracker
Loading exercise…
Rung 2 — pick the winner on a primary metric under a guardrail
Loading exercise…
Rung 3 — an offline-eval regression gate (senior)
Loading exercise…
Rung 4 — track runs with real MLflow (kernel)
Loading exercise…
⚡ Interview Ref — the quick-scan Reference face
- Eval is CI for models — versioned eval set = fixture · tracked run = CI run · offline-eval gate = required check before promote.
- Three pillars of an eval system: versioned eval datasets (freeze + hash, no leakage) · tracked runs (config + eval-set version + metrics + repro metadata) · regression gate.
- Trackers: MLflow (OSS, self-host, model registry) · W&B (dashboards, sweeps) · Braintrust (LLM-eval-first, prompt/dataset versioning). Pick on workflow (log → compare → gate), not vendor; health data → weigh EU residency / self-host.
- Run comparison = pick winner on the primary metric while watching guardrails (latency p95, groundedness); every run pins its
eval_setversion so a metric jump is unambiguous. - Model-graded harness must: batch-score the frozen set · calibrate vs humans (Cohen's κ) · run a prompt-regression suite · guard against leakage. A judge you never calibrate is a metric you can't defend.
- Offline eval ≡ online monitoring — same signals at two lifecycle points; drift closes the loop back to a fresh eval set. Doctolib Q9 = offline eval + online A/B + LLM-eval, all tracked.
📚 Go Further
Where the pieces of this module lead.
| Type | Resource |
|---|---|
| In-app | P5.evaluation.01–03 — the metrics this infrastructure runs (precision/recall/F1, BERTScore, LLM-as-judge, Cohen's κ) |
| In-app | P13.llmops.01 — Applied LLMOps: cost/latency routing, eval-in-the-loop, incident playbooks |
| In-app | P13.operations.02 — Monitoring & drift (reserved) — the online half of the loop |
| Docs | MLflow tracking & model registry; Weights & Biases experiments; Braintrust eval |
Try it yourself
A scratch console for this page's ideas — ungraded, nothing you run here is recorded.
Scratch console
A scratch console with the scientific stack (pandas, numpy, scikit-learn). Runs on the server — no network, resource-limited and measured.
Output appears here.
My notes on this module
Loading your notes...
Where next?
Later in ML Lifecycle
This module unlocks
Go up a level