LLM Safety — Failure Modes & Misuse

P15.governance-safety.02 · Audience: guest, it-ml, language-pro · Prerequisites: The Governance Landscape

The Governance Landscape Evaluating for Safety

A large language model is a new kind of component: it is fluent, confident, and wrong in ways that older software never was. A database that fails returns an error; an LLM that fails returns a beautifully-worded falsehood, or quietly follows an instruction hidden in the text it was asked to summarise. Governance put every system on a risk map (module 01); this module zooms into one especially slippery kind of system and asks the safety question about it directly: what are the ways an LLM goes wrong, what makes each one dangerous, and who is responsible when it does? The mechanics of how these failures work live in the LLM pillar — here we take the safety seat and frame them as risks to govern.

Step 1 / 5 — Safety is a lens, not a mechanism

There are two different questions you can ask about an LLM failure. How does it happen? — the mechanics of tokens, attention, training and context — is answered by the LLM systems & frontier track. Why is it dangerous, and who must prevent it? — is the safety question, and it is what this module owns. The same failure looks like an engineering curiosity from one seat and a governed risk from the other.

ⓘ Concept: The safety lens: from 'how it breaks' to 'what harm it does and who answers'

mechanics (P10): how the failure happens · safety (here): what harm it causes + who is accountable

Why it matters — Keeping the two seats separate is what stops this module from re-teaching the LLM pillar and what stops the LLM pillar from having to moralise. A hallucination is, mechanically, a model sampling a plausible continuation with no ground truth to anchor it — that is the P10 story. From the safety seat, the same event is 'a confident falsehood presented to a user who may act on it', and the questions become: how bad is the harm, in which tier does this deployment sit, and whose job was it to catch this? This module always takes the second seat.

So throughout, when a mechanism matters, we point to the pillar that owns it and keep our attention on the harm and the responsibility. Three failure classes cover most of what a practitioner must govern: hallucination, prompt injection, and misuse.

✅ Key takeaways

Safety is a lens, not a mechanism: how an LLM failure happens is the LLM pillar's story (P10), while what harm it causes and who must prevent it is the safety question this module owns. Both seats look at the same event.
Hallucination is confident wrongness — the model is exactly as fluent when wrong as when right, so the user has no signal to distrust it. Danger scales with the stakes; the safeguard is grounding, review, or narrower scope where a false answer could harm.
Prompt injection exploits the fact that an LLM cannot separate instructions from data — hidden text in the content it reads becomes a command. The risk grows with the model's ability to act, so untrusted text must be contained and powerful actions gated by a human.
Misuse is the model working as built, aimed at harm: capability is dual-use, scales cheaply, and cannot be removed without making the model useless. Safeguards move outward — refusal training, access controls, rate limits, abuse monitoring — and the misuse categories are a design input.
Responsibility follows the deployment, not the model. Each failure becomes harm through a choice with a named owner — deploying ungrounded, wiring tools to untrusted input, shipping capability without controls. The failure modes are foreseeable, so failing to safeguard them is where accountability lands.

Next: P15.governance-safety.03 — Evaluating for Safety

The mechanics behind these failures — why a model samples a plausible-but-false continuation, why the instruction boundary is unenforceable, how capability emerges — belong to the LLM systems & frontier track. This module took the safety seat on top of them. Next: how a team checks, before shipping, that its safeguards actually hold — evaluation as a governance practice.

Ask the mentor about this module

Ask a question about this content. The mentor explains and grounds its answer in what you are studying; asking is recorded as a learning signal, not a grade.

Ctrl/Cmd + Enter to send

Try it yourself

A scratch console for this page's ideas — ungraded, nothing you run here is recorded.

Scratch console

A scratch console with the scientific stack (pandas, numpy, scikit-learn). Runs on the server — no network, resource-limited and measured.

Output appears here.

Where next?

Continue

Evaluating for Safety →

Governance & Safety · P15

Later in Governance & Safety

This module unlocks

When It Goes Wrong — Incident Thinking

Go up a level