Evaluating for Safety

P15.governance-safety.03 · Audience: guest, it-ml, language-pro · Prerequisites: LLM Safety — Failure Modes & Misuse

LLM Safety — Failure Modes & Misuse When It Goes Wrong — Incident Thinking

"We tested it and it passed" is one of the most reassuring — and most misleading — sentences in the whole field. A safety evaluation is not proof that a system is safe; it is evidence about the specific things you thought to check, on the specific inputs you happened to try. Governance depends on that evidence, but a practitioner who mistakes a passing eval for a guarantee has learned the single most dangerous lesson available. This module treats safety evaluation and red-teaming as a governance practice — how you gather trustworthy evidence, and, just as important, what that evidence stubbornly does not prove. The mechanics of running evals as a system live in the LLM pillar; here they are a tool of accountability.

Step 1 / 5 — An eval is evidence, not proof

The mechanics of LLM evaluation — building test sets, scoring outputs, benchmark design — are a systems topic owned by the LLM systems & frontier track. This module takes the governance seat: an evaluation is how a team produces evidence that a system meets its safety bar, so that the accountable owner can point to something real when they say "this was safe to ship". The value of an eval is exactly the value of the evidence it produces — and evidence is never the same as proof.

ⓘ Concept: A safety eval produces evidence for an accountability claim

good safety eval = tests the harms that matter, on realistic inputs → evidence a reviewer can trust (not merely a green checkmark)

Why it matters — Governance runs on documented evidence, not on a team's confidence. When a high-risk system must show it was built responsibly (module 01), the safety evaluation is a core exhibit: here is what we tested for, here is how it behaved, here is who signed off. That reframes what makes an eval good — not 'did it pass?' but 'does it produce evidence a reviewer can trust, about the harms that actually matter for this deployment?' An eval that tests the wrong harms, or tests them on inputs an attacker would never use, produces confident evidence about nothing.

✅ Key takeaways

A safety eval is evidence, not proof. Its job in governance is to produce a trustworthy exhibit for the accountability claim 'this was safe to ship' — so the right question is not 'did it pass?' but 'does it produce evidence a reviewer can trust about the harms that matter?'
Safety evals deliberately probe the failure classes — hallucination on high-stakes prompts, injection attempts, misuse/refusal — rather than the happy path. What you probe is a governance decision set by the deployment's tier and its specific harms.
Red-teaming is the adversarial search for what a fixed suite misses: people and models trying to break the system before a real attacker does. Findings both fix the system and get added to the standing suite so the same hole cannot silently reopen.
A pass does NOT prove safety in two nameable ways: coverage gaps (the eval saw only the cases you wrote, never the infinite rest) and spec gaming (a system can learn to pass the test without gaining the property). 'Passed the tests we ran' is the only honest claim.
Evaluation is a practice, not an event: the suite grows, the claim stays modest, and the record — harms tested, red-team findings and fixes, honest coverage limits — is the exhibit that makes accountability real. An overclaiming eval is worse than none, because it launders confidence.

Next: P15.governance-safety.04 — When It Goes Wrong: Incident Thinking

The nuts and bolts of building an eval as a running system — datasets, scoring, regression tracking — belong to the LLM systems & frontier track; this module added the safety-and-governance lens on top. The technical track can go one level deeper below.

Going deeper (technical) — designing a safety eval-harness

Interview-grade notes for the technical (it-ml) track. The governance framing above is audience-broad; these are the design choices you face once you actually build the harness that produces the evidence.

A safety eval is a scored dataset plus a grader plus a threshold — and each is a design decision. The dataset is a curated set of adversarial and high-stakes prompts, versioned like code, tagged by harm category (hallucination, injection, misuse-refusal) so you can report per-category pass rates rather than one opaque number. The grader is the hard part: exact-match works only for narrow tasks, so safety graders are usually a rubric applied by a stronger model (LLM-as-judge), human raters on a sampled subset, or a classifier — each with its own failure mode (a judge model shares blind spots with the model under test; humans are slow and inconsistent; classifiers drift). Report inter-rater agreement, not just the score.

Guard against contamination and gaming at the harness level. If eval prompts leak into training data, the pass is memorisation, not capability — hold out a fresh, never-published slice and watch for a gap between it and the public suite (a large gap is a contamination tell). Rotate and expand the suite so a static target cannot be over-fit; a score that climbs while red-team findings hold steady is a spec-gaming warning, not a win.

Separate the standing regression suite from exploratory red-teaming. The regression suite is cheap, deterministic, and run on every change — it stops known holes reopening. Red-teaming is expensive, creative, and run in campaigns — its job is to generate new suite entries. A healthy harness has a pipeline from the second into the first: every confirmed red-team break becomes a permanent regression case with a stable id, so coverage only ratchets up.

Design the report for the reviewer, not the engineer. The eval's governance value is the exhibit it produces, so the output is a document: per-category rates, the exact input distribution tested, known coverage gaps stated explicitly, red-team campaign summary, and a version stamp tying the result to a specific model build. An eval that emits only a single number is engineering-useful and governance-useless — the honest limits are the most important line in the report.

Ask the mentor about this module

Ask a question about this content. The mentor explains and grounds its answer in what you are studying; asking is recorded as a learning signal, not a grade.

Ctrl/Cmd + Enter to send

Try it yourself

A scratch console for this page's ideas — ungraded, nothing you run here is recorded.

Scratch console

A scratch console with the scientific stack (pandas, numpy, scikit-learn). Runs on the server — no network, resource-limited and measured.

Output appears here.

Where next?

Continue

When It Goes Wrong — Incident Thinking →

Governance & Safety · P15

Later in Governance & Safety

When It Goes Wrong — Incident Thinking

Go up a level