Statistics for NLP Evaluation

P1.statistics-core.05 · Audience: guest, it-ml, language-pro · Prerequisites: Resampling & Confidence Intervals

Resampling & Confidence Intervals Statistical Laws of Language

LLM grading (this page):Real gradingOff — no LLM calls

When two NLP systems produce different scores, statistics decides whether the gap is real or just noise. This module shows you how — with the paired bootstrap, F1 and BLEU worked examples, and a practical guide for choosing the right test.

Step 1 / 5 — The Problem: Is the Difference Real?

🗣️ From a linguist's perspective: evaluation as a noisy measurement of a latent system quality

In linguistics, a judgment study measures how native speakers rate the acceptability of sentences. Any single judgment is noisy — it depends on the speaker's mood, fatigue, and the specific test sentences. To trust a result, you need many judgments and a significance test. NLP evaluation is identical: a BLEU score on 500 sentences is a noisy estimate of the model's true translation quality. A difference of 0.5 BLEU between two systems might be noise or might be a real improvement — only a significance test can tell you which.

Two named-entity recognition systems, A and B, are evaluated on the same 10 test examples. Each example gets a per-example F1 score for each system. The core question of this module: is Δ = mean(A) − mean(B) real, or noise?

System A mean F1

0.7850

System B mean F1

0.7640

Δ = A − B

+0.0210

Example	System A F1	System B F1	Δ = A − B
1	0.82	0.79	+0.03
2	0.71	0.68	+0.03
3	0.90	0.85	+0.05
4	0.65	0.63	+0.02
5	0.78	0.80	−0.02
6	0.85	0.81	+0.04
7	0.60	0.58	+0.02
8	0.92	0.89	+0.03
9	0.74	0.77	−0.03
10	0.88	0.84	+0.04

System A wins on 8 of 10 examples, with a mean advantage of +0.0210. But is this significant?

⚠️ A higher mean score does not by itself mean A is better. The spread of the per-example differences determines whether the gap is real — that is what the next step measures.

🎓 Going deeper — the paired bootstrap, derived

Setup — why pair? Let aᵢ and bᵢ be the sentence-level F1 scores for Models A and B on sentence i, for i = 1, …, n, and define the per-sentence difference δᵢ = aᵢ − bᵢ. The observed test statistic is the mean difference:

\Delta_{\text{obs}} = \bar{\delta} = \frac{1}{n}\sum_{i=1}^n \delta_i

Both models are evaluated on the same sentences, so aᵢ and bᵢ are correlated within each sentence. Pairing removes this sentence-level variability. Compare the two variances:

\mathrm{Var}(\bar{a} - \bar{b}) = \frac{\sigma^2_a + \sigma^2_b}{n} \qquad\qquad \mathrm{Var}(\bar{\delta}) = \frac{\sigma^2_a + \sigma^2_b - 2\,\mathrm{Cov}(a,b)}{n}

When the scores are positively correlated (hard sentences are hard for both models), the covariance term makes the paired variance strictly smaller — a tighter estimator and a more powerful test. This construction is also called the bootstrap approximate randomisation test.

Test procedure. The null hypothesis is H₀: 𝔼[δᵢ] = 0 (no difference in expected F1). Bootstrap under H₀ by resampling the centred differences:

Centre: δ̃ᵢ = δᵢ − δ̄, so that 𝔼[δ̃ᵢ] = 0 under H₀.
For b = 1, …, B: draw δ̃*₁, …, δ̃*ₙ with replacement from the centred differences and compute Δ*_b as their mean.
Compute the p-value as the fraction of null samples at least as extreme as the observation:

p = \frac{\#\{\,|\Delta^*_b| \geq |\Delta_{\text{obs}}|\,\}}{B}

Reject H₀ at level α if p ≤ α. Why centre? Without centring, the bootstrap distribution of Δ* is centred on δ̄, not on 0. We need the null distribution (mean 0) to compute the p-value — measuring deviations from the wrong centre gives an incorrect p-value.

Connection to the permutation test. The paired permutation test randomly flips the sign of each δᵢ (swapping A and B for that sentence) and recomputes Δ*. For large n, the paired bootstrap and the paired permutation test give very similar p-values. The bootstrap is more flexible — it also applies when the scores are continuous and when you want a CI, not just a p-value.

Worked example — toy sentence-level F1 scores. Eight test sentences, each with a sentence-level F1 for Model A and Model B:

Sentence	F1 Model A (aᵢ)	F1 Model B (bᵢ)	δᵢ = aᵢ − bᵢ
S1	0.72	0.68	+0.04
S2	0.85	0.80	+0.05
S3	0.61	0.70	−0.09
S4	0.78	0.65	+0.13
S5	0.90	0.82	+0.08
S6	0.55	0.60	−0.05
S7	0.83	0.77	+0.06
S8	0.69	0.71	−0.02

Mean F1: Model A = 0.7413, Model B = 0.7163, so Δ_obs = mean(δᵢ) = +0.0250. Following the procedure: (1) centre the eight differences by subtracting 0.0250; (2) draw B = 5 000 bootstrap resamples of the centred δ̃ and take each mean Δ*; (3) count how many satisfy |Δ*| ≥ 0.0250. The 95th percentile of this null distribution sits at roughly ±0.06, and |Δ_obs| = 0.025 falls well inside it — we fail to reject H₀: the observed advantage is not statistically significant at α = 0.05 with n = 8. With only eight test sentences the test has very low power; in practice, NLP evaluations use hundreds to thousands of test instances.

Self-test — 5 questions

Q1 — Define the paired bootstrap test statistic Δ_obs for comparing two NLP models.

Q2 — Why must we centre the differences before bootstrapping under H₀? What happens if we don't?

Q3 — Why does pairing reduce the variance of the test statistic compared to an unpaired t-test?

Q4 — How is the p-value computed from the bootstrap null distribution?

Q5 — In the worked example, Δ_obs = 0.0250 with n = 8. The 95th percentile of the bootstrap null distribution is approximately ±0.06. Without computing, is the result statistically significant at α = 0.05? Why?

Ask the mentor about this module

Ask a question about this content. The mentor explains and grounds its answer in what you are studying; asking is recorded as a learning signal, not a grade.

Ctrl/Cmd + Enter to send

🎓 Practice ladder

2 graded rungs · ~13 min

Two graded exercises, worked directly in the browser — the code runs locally in Pyodide and is checked against visible and hidden tests. The explorer rung fills the one missing cell of the systems-comparison table; the other grades the deep-dive's numeric self-test: the paired bootstrap on the module's fixed per-sentence F1 scores.

Rung 1 — Is system A really better?

Loading exercise…

Explorer rung — Fill the missing cell of the comparison table

Loading exercise…

📝 Check Your Understanding

Q1. If you test 20 models against a baseline at α = 0.05, how many false positives do you expect by chance?

Q2. Given precision = 0.8 and recall = 0.6, what is the F1 score? (round to 4 decimal places)

Q3. Why is a paired bootstrap test preferred over a simple t-test for comparing F1 scores from the same test set?

Q4. A model improvement is statistically significant (p < 0.05) but the effect size is 0.001 F1. Is this practically meaningful?

Q5. A test has p = 0.001 but Cohen's d = 0.05. What does this suggest about the difference?

Module Review. Name one situation where you would use a permutation test instead of a bootstrap confidence interval.

📚 Go Further

Curated resources to explore evaluation statistics for ML models further.

Type	Resource
Video	StatQuest with Josh Starmer — Confusion Matrix, Sensitivity and Specificity, ROC and AUC (YouTube)
Course	Coursera — Deep Learning Specialization, Course 3: Structuring Machine Learning Projects — evaluation metrics, Andrew Ng (free audit)
Reference	Scikit-learn — Model evaluation: quantifying the quality of predictions (scikit-learn.org/stable/modules/model_evaluation.html)
Exercise	Kaggle — Evaluation Metrics for Classification notebook (free)

✅ Key takeaways

A higher evaluation score does not imply a better system — the difference might be noise from test-set sampling variability.
The paired bootstrap test is the NLP community standard: resample (Aᵢ − Bᵢ) pairs with replacement, compute the mean difference each time, and check how often it is ≤ 0.
F1 = 2PR/(P+R) is the harmonic mean of precision and recall; it is dominated by the smaller of the two. Always report a bootstrap CI alongside the point estimate.
BLEU is computed over n-gram matches; small BLEU differences (below 1 point) are rarely significant on standard test sets. Use paired bootstrap with B ≥ 1 000.
Report both effect size (Δ and its CI) and the p-value. Statistical significance and practical significance are different things.

Next: Statistical Laws of Language covers Zipf's law, Heaps' law, n-gram language-model perplexity, and smoothing — the statistical foundations of language-model evaluation.

Try it yourself

A scratch console for this page's ideas — ungraded, nothing you run here is recorded.

Scratch console

A scratch console with the scientific stack (pandas, numpy, scikit-learn). Runs on the server — no network, resource-limited and measured.

Output appears here.

Where next?

Continue

Statistical Laws of Language →

Statistics Core · P1

Later in Statistics Core

Statistical Laws of Language

Go up a level