Terminology & judging quality
P10.llm-language-practice.03 · Audience: guest, it-ml, language-pro · Prerequisites: Dialects, register & voice
Take it one idea at a time: how to make the model use your exact terms, how to judge output on more than fluency, and how to decide a model is ‘good enough’ with real evidence. Each step has a worked example. No maths required.
ⓘ Concept: One mental model to keep
Left alone, the model drifts toward variety (synonyms, near-terms) and fluency (reads well). Professional quality is the opposite: consistency and faithfulness. Both must be engineered and verified, never assumed from one instruction.
Why it matters — Left alone the model drifts toward variety and fluency; professional quality is the opposite — consistency and faithfulness — and both must be engineered and verified, never assumed.
You improve terminology adherence by giving the model an explicit glossary in the prompt, instructing it to use those exact terms, and providing examples. For larger jobs you enforce it with a defined term list and a post-processing check that flags deviations. On its own the model tends toward variation and synonyms.
Worked example — one concept, three names in one document
Left to itself, the model refers to the same entity three different ways across a support article — a real problem for legal, medical, and brand content:
| Paragraph | Term the model used | Glossary says | Deviation check |
|---|---|---|---|
| Intro | “the customer” | “account holder” | 🚩 flag |
| Step 2 | “the client” | “account holder” | 🚩 flag |
| FAQ | “the user” | “account holder” | 🚩 flag |
A deviation check — glossary in the prompt plus a post-processing pass that scans for the wrong variants — catches all three. Retrieval of approved references and, at scale, fine-tuning or a terminology-checking step tighten this further. The senior point: consistency must be engineered and verified, not assumed from a single instruction.
⚡ Interview Ref — the quick-scan Reference face
In plain language: how to make the model use your exact terms, how to judge output on more than fluency, and how to decide a model is 'good enough' with real evidence. No maths required.
1 — Consistent terminology & glossary adherence (l19)
Improve adherence by giving the model an explicit glossary in the prompt, instructing it to use those exact terms, and providing examples. For larger jobs, enforce it with a defined term list and a post-processing check that flags deviations — on its own the model tends toward variation and synonyms, a real problem for legal, medical, and brand content. Retrieval of approved references and, at scale, fine-tuning tighten this further. Consistency must be engineered and verified, not assumed from a single instruction.
2 — Evaluating output beyond 'reads well' (l18)
Fluency is only one dimension, and the easiest to be fooled by. Also judge adequacy/faithfulness (fully and only conveys the source), accuracy of facts and numbers, terminology and consistency, register/tone, and cultural appropriateness. A polished, confident text can be subtly wrong on any of these. Good evaluation uses defined criteria, comparison against the source, and ideally multiple reviewers or a rubric — not a gut sense that it "sounds fine".
3 — Deciding a model is 'good enough' — and the evidence (l26)
Define the task and its quality bar, then test on a representative sample of real inputs, measuring the dimensions that matter per language or locale — ideally with human raters and comparison against a trusted reference. Look at worst cases and error types, not just averages, and test the edge cases your content actually contains. Demand this evidence rather than accepting benchmark claims or a vendor demo.
Interview one-liners
- Terminology must be engineered: glossary in the prompt + examples + a deviation check — the model defaults to synonyms.
- Judge output on adequacy, accuracy, terminology, tone, and cultural fit — not just fluency, which is the easiest to fake.
- 'Good enough' = evidence: test on your real samples, per locale, worst-cases included — not a benchmark or a demo.
📚 Go Further
Plain-language explainers on quality and terminology.
| Type | Resource |
|---|---|
| Framework | MQM (Multidimensional Quality Metrics) — adequacy/fluency/terminology error typology |
| In-app | Evaluation modules, for the technical metrics behind quality scoring |
| In-app | P10.llm-language-practice.05 — Strategy, workflow & adoption, for turning these checks into a QA process |
Ask the mentor about this module
Ask a question about this content. The mentor explains and grounds its answer in what you are studying; asking is recorded as a learning signal, not a grade.
🎓 Practice ladder
3 graded rungs · ~28 minMake "quality is evidence" concrete: three rungs that measure terminology, effort, and the ship/revise/reject call. The workspace is three panels — instructions, a code editor, and live output with test results. Run checks the visible tests; Submit grades against hidden edge cases; the reference solution unlocks once you pass.
Explorer rung 1 — Terminology consistency check
Loading exercise…
Explorer rung 2 — Measure the post-edit effort
Loading exercise…
Explorer rung 3 — 'Good enough' is a verdict
Loading exercise…
Where next?
Later in LLM Language Practice
Go up a level