Evaluation Metrics for NLP

P5.evaluation.01 · Audience: guest, it-ml, language-pro

ML in Production (method view)Embedding Evaluation

LLM grading (this page):Real gradingOff — no LLM calls

A model's quality is invisible until you can measure it: output that looks fluent and confident can still fail on exactly the cases that matter, and staring at examples will never tell you how often. What turns "it seems to work" into a number you can defend is a metric, an agreed formula that scores a model's predictions against a known ground truth — and every metric sees some failures while staying blind to others, so you also need to know which one to trust in which situation. In this module you work the standard metrics out by hand, from classification through generated text to a model's own confidence, and finish with the discipline of holding data back and correcting for repeated comparisons that keeps the resulting numbers honest; every computation runs on the fixed demo corpus or a toy small enough to check on paper. The statistical machinery underneath — significance testing in Statistics for NLP Evaluation, and the statistical laws of language behind language-model scoring — lives in the P1 statistics pillar whenever you want the deeper roots.

🗺️ So far you have built: the full pipeline from raw text to sentence embeddings — and the P1 statistics pillar, whose hypothesis testing (S3), resampling (S4), evaluation statistics (S5), and statistical laws of language (S6) all feed directly into this track.

In this module you will: building a language model is only half the job; the other half is knowing whether it works — and for what. This track is the measurement layer that sits on top of the pipeline. In this module you compute the classic metrics by hand on the demo corpus — precision, recall, F1, and ROC/AUC for classification; perplexity, BLEU, and ROUGE for sequences — then check whether a model's confidence can be trusted (calibration, ECE) and how to design an evaluation that doesn't fool you (splits, cross-validation, Bonferroni).

Step 1 / 4 — Classification Metrics — P/R/F1 and ROC/AUC

Task: classify each demo-corpus sentence as “discusses language” (positive) or “does not” (negative). Ground truth: a sentence is positive when it contains the token language — sentences S4 and S6. A toy keyword classifier fires on any sentence containing the token representative or language — so it also fires on S2, where “representative of a population” triggers it.

S#	Sentence	Ground truth	Classifier	Outcome
S0	Lions are the most dangerous of all wild animals.	✗ negative	✗ negative	TN
S1	Humans have always hunted animals as a source of food.	✗ negative	✗ negative	TN
S2	The king is a representative of a population.	✗ negative	✓ positive	FP
S3	The human population of the world has rapidly increased.	✗ negative	✗ negative	TN
S4	A naïve algorithm often fails when the morphology of a language is complex.	✓ positive	✓ positive	TP
S5	The representation of a word captures the semantic patterns in text.	✗ negative	✗ negative	TN
S6	A café is a place where the community uses language as a form of identity.	✓ positive	✓ positive	TP
S7	The tokenization of a word splits it into meaningful subword units.	✗ negative	✗ negative	TN

Every prediction lands in exactly one of four cells — the confusion matrix:

	Actually positive	Actually negative
Predicted positive	TP = 2	FP = 1
Predicted negative	FN = 0	TN = 5

Explore this live — the panel below is the interactive version of exactly this worked example. Move predictions between cells and watch precision, recall, and F1 respond.

Confusion-matrix explorer — the Ev1 demo classifier by default

TPFPFNTN

Computing…

Precision, Recall, F1.

\text{Precision} = \frac{TP}{TP + FP} \qquad \text{Recall} = \frac{TP}{TP + FN} \qquad F_1 = \frac{2 \cdot P \cdot R}{P + R}

Precision — of everything we labelled positive, what fraction was correct?
Recall — of everything actually positive, what fraction did we find?
F1 — harmonic mean of P and R; punishes extreme imbalances between the two.

On the demo corpus: Precision = 2 / (2 + 1) = 0.6667, Recall = 2 / (2 + 0) = 1.0000, F1 = 2 × 0.667 × 1.000 / (0.667 + 1.000) = 0.8000.

Precision

0.6667

Recall

1.0000

0.8000

Precision = 0.667: 2 of the 3 positive predictions were correct. Recall = 1.000: both actually-positive sentences were found. F1 = 0.800: one false positive penalises precision more than recall here.

When to prioritise Precision vs Recall.

High Recall matters when missing a positive is costly (medical diagnosis, fraud detection, content moderation).
High Precision matters when false positives are costly (spam filters that delete legitimate emails, triggering expensive downstream processes).
F1 is the default when neither error dominates — but Fβ (β > 1 for Recall, β < 1 for Precision) lets you tune the trade-off explicitly.

ROC Curve and AUC. P/R/F1 depend on a fixed decision threshold (≥ 0.5 → positive). The ROC curve sweeps the threshold from 0 to 1 and plots TPR (sensitivity/recall) vs FPR (1 − specificity) at every point. AUC (Area Under the Curve) summarises the entire curve in one number: AUC = 1.0 is perfect; AUC = 0.5 is a random classifier.

A toy 5-example ROC: sort the examples by descending confidence and record TPR and FPR after each one (2 positives, 3 negatives):

k (threshold rank)	Confidence	True label	TPR	FPR
1	0.90	+	1/2 = 0.50	0/3 = 0.00
2	0.80	−	1/2 = 0.50	1/3 = 0.33
3	0.70	+	2/2 = 1.00	1/3 = 0.33
4	0.40	−	2/2 = 1.00	2/3 = 0.67
5	0.30	−	2/2 = 1.00	3/3 = 1.00

The trapezoidal rule over these FPR/TPR points gives AUC = 5/6 ≈ 0.833 — the model ranks a randomly chosen positive above a randomly chosen negative 83.3 % of the time. (Computing from the table's two-decimal roundings gives 0.835; the exact fractions 1/3 and 2/3 give 5/6.)

Sweep the threshold yourself in the lab below — it reproduces this table point by point and reports the exact AUC.

ROC / threshold explorer — the Ev1 5-example toy classifier

Decision threshold: 0.5

Computing…

Slide the threshold through the five scores (0.90, 0.80, 0.70, 0.40, 0.30) to reproduce each row of the worked ROC table above.

📚 Go Further

Where these metrics connect to the rest of the curriculum.

Type	Resource
In-app	P5.ml-methods.06 — Evaluation, Selection & Interpretability, for the model-selection view of these metrics (bias–variance, information criteria, interpretability)
In-app	Statistics for NLP Evaluation (P1 statistics pillar), for significance testing of F1 and BLEU differences with the paired bootstrap
In-app	Statistical Laws of Language (P1 statistics pillar), for the n-gram language-model perplexity this module's perplexity metric builds on
In-app	Statistical Inference and Resampling Methods (P1 statistics pillar), the hypothesis-testing and bootstrap machinery behind every comparison in this module

Ask the mentor about this module

Ask a question about this content. The mentor explains and grounds its answer in what you are studying; asking is recorded as a learning signal, not a grade.

Images, PDF or text. Kept on this device only.

Keeping your files on this device

Keep a copy of the files I attach, in this browser

Off by default. The mentor always gets your file; this only decides whether your own copy stays here. Copies live in this browser only - they do not follow you to another device, and clearing site data removes them.

Ctrl/Cmd + Enter to send

🎓 Practice ladder

5 graded rungs · ~47 min

Now grade a classifier yourself. Each rung is a three-panel workspace: instructions on the left, a code editor in the middle, and output + test results on the right. Run checks the visible tests; Submit grades against hidden edge cases; the reference solution unlocks once you pass. The explorer rung is one knob — trade recall for precision on a spam filter; the senior rung picks the metric for a fraud deployment and defends it.

Rung 1 — Precision, recall and F1 from a confusion matrix

Loading exercise…

Rung 2 — ROC points by threshold sweep

Loading exercise…

Rung 3 — Pick the operating point

Loading exercise…

Explorer rung — Raise the bar: trade recall for precision

Loading exercise…

Senior rung — Pick the metric for the deployment and defend it

Loading exercise…

✅ Key takeaways

Precision = TP/(TP+FP), Recall = TP/(TP+FN), and F1 is their harmonic mean — on the demo-corpus keyword classifier: P = 0.6667, R = 1.0000, F1 = 0.8000, with one false positive dragging precision below recall.
The ROC curve sweeps the decision threshold and plots TPR vs FPR; AUC summarises it as the probability of ranking a random positive above a random negative — 5/6 ≈ 0.833 on the 5-example toy.
Perplexity = exp(cross-entropy) reads as the number of equally-likely options the model hesitates between (CE = 1.5350 → PP ≈ 4.64); BLEU is precision-oriented with a brevity penalty (BLEU-1 = 0.513), ROUGE is recall-oriented (ROUGE-1 F1 = 0.750) — they penalise different errors.
A calibrated model's confidence matches its accuracy; ECE is the bin-weighted |acc − conf| gap (worked 2-bin example: ECE = 0.100), and the reliability diagram shows over- vs under-confidence against the diagonal.
Methodology is a metric too: the test set is looked at only once, k-fold CV stabilises estimates when data is scarce, and comparing k systems pairwise needs a Bonferroni-corrected threshold α / C(k,2).

Next: P5.evaluation.02 — Embedding Evaluation: intrinsic benchmarks (word similarity, analogies) and extrinsic downstream evaluation of the embeddings the pipeline produces.

Try it yourself

A scratch console for this page's ideas — ungraded, nothing you run here is recorded.

Scratch console

A scratch console with the scientific stack (pandas, numpy, scikit-learn). Runs on the server — no network, resource-limited and measured.

Output appears here.

My notes on this module

Loading your notes...

Where next?

Continue

Embedding Evaluation →

Evaluation · P5

Later in Evaluation

This module unlocks

Applied Case — The Honest Tabular Model, End to End

Go up a level