Account

Evaluation infrastructure & experiment tracking

P13.ml-lifecycle.01 · Audience: guest, it-ml, language-pro · Prerequisites: LLM & Generation Evaluation

Real LLM grading for this pageLLM grading (this page):

The tooling layer a production ML/LLM team lives in: versioned eval datasets, experiment tracking (Weights & Biases / MLflow / Braintrust), and a model-graded-eval harness — how a team runs, compares, and trusts evaluations, then feeds them into production monitoring. P5.evaluation.01–03 taught which metric to compute; this module treats evaluation as versioned, tracked infrastructure rather than a notebook cell, and opens the Production ML & MLOps pillar.

Step 1 / 4 — From a metric to an eval system

P5.evaluation.01–03 give you the numbers — precision/recall/F1, BLEU/ROUGE, BERTScore, nDCG, LLM-as-judge scores. In production none of that is trustworthy unless it is reproducible and comparable across runs. That is what eval infrastructure buys you.

Three pillars turn a one-off metric into an eval system:

  1. Versioned eval datasets — the eval set is a dataset artefact with a version, not a notebook cell. Freeze it, hash it, and never let training data leak into it. A model only improves against a fixed target.
  2. Tracked runs — every evaluation is a logged run: which model/prompt/commit, which eval-set version, the resulting metrics, and enough metadata to reproduce it.
  3. A regression gate — before a model or prompt ships, an automated eval run must clear a bar (offline-eval gate), exactly like a test suite gates code.
ⓘ Concept: Mental model — eval is CI for models
A versioned eval set is the test fixture, a tracked run is a CI run, and the offline-eval gate is the required check before merge/promote.

The Doctolib System-Design interview (Q9) asks for exactly this stack: offline eval + online A/B + LLM-specific evaluation, tracked in W&B / MLflow / Braintrust. The next steps build each piece.

Ask the mentor about this module

Ask a question about this content. The mentor explains and grounds its answer in what you are studying; asking is recorded as a learning signal, not a grade.

Ctrl/Cmd + Enter to send

🎓 Practice ladder

4 graded rungs · ~45 min

Build the experiment-tracking loop yourself — a file-based run tracker, a primary-metric-with-guardrail selector, and an offline-eval regression gate — then do it with the real tool: a fourth rung runs in the mlops kernel and logs to a genuine MLflow store. Local-first — the kernel has no network, so MLflow uses a local sqlite: store, no server (ADR-011).

Rung 1 — a file-based run tracker

Loading exercise…

Rung 2 — pick the winner on a primary metric under a guardrail

Loading exercise…

Rung 3 — an offline-eval regression gate (senior)

Loading exercise…

Rung 4 — track runs with real MLflow (kernel)

Loading exercise…

⚡ Interview Ref — the quick-scan Reference face
  • Eval is CI for models — versioned eval set = fixture · tracked run = CI run · offline-eval gate = required check before promote.
  • Three pillars of an eval system: versioned eval datasets (freeze + hash, no leakage) · tracked runs (config + eval-set version + metrics + repro metadata) · regression gate.
  • Trackers: MLflow (OSS, self-host, model registry) · W&B (dashboards, sweeps) · Braintrust (LLM-eval-first, prompt/dataset versioning). Pick on workflow (log → compare → gate), not vendor; health data → weigh EU residency / self-host.
  • Run comparison = pick winner on the primary metric while watching guardrails (latency p95, groundedness); every run pins its eval_set version so a metric jump is unambiguous.
  • Model-graded harness must: batch-score the frozen set · calibrate vs humans (Cohen's κ) · run a prompt-regression suite · guard against leakage. A judge you never calibrate is a metric you can't defend.
  • Offline eval ≡ online monitoring — same signals at two lifecycle points; drift closes the loop back to a fresh eval set. Doctolib Q9 = offline eval + online A/B + LLM-eval, all tracked.
📚 Go Further

Where the pieces of this module lead.

TypeResource
In-appP5.evaluation.01–03 — the metrics this infrastructure runs (precision/recall/F1, BERTScore, LLM-as-judge, Cohen's κ)
In-appP13.llmops.01 — Applied LLMOps: cost/latency routing, eval-in-the-loop, incident playbooks
In-appP13.operations.02 — Monitoring & drift (reserved) — the online half of the loop
DocsMLflow tracking & model registry; Weights & Biases experiments; Braintrust eval

Try it yourself

A scratch console for this page's ideas — ungraded, nothing you run here is recorded.

Scratch console

A scratch console with the scientific stack (pandas, numpy, scikit-learn). Runs on the server — no network, resource-limited and measured.

Output appears here.

Where next?

Continue

Packaging & deployment →

ML Lifecycle · P13

Later in ML Lifecycle

This module unlocks

Go up a level