Account

Statistical Inference & Hypothesis Testing

P1.statistics-core.03 · Audience: guest, it-ml, language-pro · Prerequisites: Probability Distributions

Real LLM grading for this pageLLM grading (this page):

Inference asks: what can we conclude about the world from a limited sample? Hypothesis testing asks: is what we observed surprising under a baseline assumption?

Step 1 / 5 — Estimators and the Sampling Distribution
🗣️ From a linguist's perspective: estimators as summaries from a sample of the linguistic population
A corpus is a sample of all text that could have been written in a language. The mean sentence length measured in a corpus is an estimator of the true mean sentence length in the language. Different samples of the same language give slightly different estimates — this variability is the sampling distribution of the estimator. The Central Limit Theorem tells us the sample mean's sampling distribution is approximately Normal, allowing us to build confidence intervals and conduct hypothesis tests even without knowing the full language.
ⓘ Concept: Estimators and the sampling distribution
An estimator θ̂ is a function of a sample that estimates a population parameter θ. The sample mean x̄ = Σxᵢ / n estimates the population mean μ; the sample variance s² = Σ(xᵢ − x̄)² / (n−1) estimates the population variance σ². Three quantities describe how good an estimator is: bias = E[θ̂] − θ (zero for unbiased estimators), variance — how much θ̂ fluctuates across different samples — and MSE = bias² + variance, the full error budget. The standard error (SE) of the sample mean is SE(x̄) = σ/√n, approximated by s/√n when σ is unknown.
SE(xˉ)=sn\text{SE}(\bar{x}) = \frac{s}{\sqrt{n}}

Why it matters — When you report model performance as 'F1 = 0.85 ± 0.02 (SE)', the ±0.02 is the standard error of the mean F1 across k evaluation runs. It quantifies how much the estimate would vary if you re-ran with different seeds.

Live demo — the corpus is split into two halves, and each half's sentence lengths (in tokens) are summarised: sample size n, mean x̄, sample standard deviation s, and the standard error SE = s/√n. The 95 % confidence intervals shown here are the subject of the next step.

Computing…

🎓 Going deeper — Statistical Inference Deep Dive

For the quantitatively inclined — the t-statistic derived from the standard error, the Neyman-Pearson likelihood-ratio framework, and Bonferroni / Benjamini-Hochberg correction with a worked 5-hypothesis example.

1 — From Z to t: why the distribution changes. Under H₀: μ = μ₀, if the population standard deviation σ were known, the standardised mean would be exactly standard Normal:

ⓘ Concept: The Z statistic (σ known)

But σ is usually unknown. Substituting s (the sample standard deviation, dividing by n−1) gives t = (X̄ − μ₀)/(s/√n). The denominator s/√n is the standard error of the mean (SE). Because s is random — estimated from the same sample — the ratio t is not standard Normal: it follows a Student's t-distribution with n−1 degrees of freedom, whose heavier tails absorb the extra uncertainty. As n → ∞, s → σ and t₍n−1₎ → N(0,1).

Z=Xˉ−μ0σ/n∼N(0,1)Z = \frac{\bar{X} - \mu_0}{\sigma/\sqrt{n}} \sim N(0,1)

Why it matters — This is the idealised case — in practice σ is almost never known, which is exactly why the t-distribution exists.

Why n−1 degrees of freedom? The deviations Xᵢ − X̄ sum to zero, so only n−1 of them are free. Formally, (n−1)s²/σ² follows a χ²₍n−1₎ distribution, independent of X̄ (Cochran's theorem), and the t-statistic can be written as a Normal divided by the square root of an independent scaled chi-square:

ⓘ Concept: t as a ratio — the definition of Student's t

Numeric example on the demo corpus: test H₀: μ = 10 (average sentence length is 10 words) against H₁: μ ≠ 10. Compute x̄ and s from the 8 sentence lengths, form SE = s/√8, then t = (x̄ − 10)/SE with df = 7. Compare |t| against the two-tailed critical value t*(7, 0.975) ≈ 2.365: if |t| exceeds it (equivalently p < 0.05), reject H₀; otherwise there is no significant evidence that μ ≠ 10.

t=Zχn−12/(n−1),Z∼N(0,1)t = \frac{Z}{\sqrt{\chi^2_{n-1}/(n-1)}}, \quad Z \sim N(0,1)

Why it matters — This ratio is exactly the definition of Student's t with n−1 degrees of freedom — the t-distribution is not an approximation but the exact law of the studentised mean under normality.

Try this ▶ The panel below runs that one-sample t-test live: at μ₀ = 10 it reproduces t = 0.9165 with df = 7. Slide μ₀ away from the sample mean and watch |t| cross the critical value ≈ 2.365.

Computing…

2 — Neyman-Pearson lemma: Type I/II errors and power. In hypothesis testing we choose between H₀ and H₁, and the two error rates are α = P(reject H₀ | H₀ true) — the significance level — and β = P(fail to reject H₀ | H₁ true). Power = 1 − β is the probability of detecting a true effect.

ⓘ Concept: Neyman-Pearson lemma (1933)

For testing a simple H₀: θ = θ₀ against a simple H₁: θ = θ₁, the uniformly most powerful (UMP) test at level α is the likelihood ratio test: reject H₀ when Λ(x) exceeds the threshold k_α, chosen so that P(Λ(X) > k_α | θ₀) = α. Interpretation: compare how much more likely the data are under H₁ vs H₀, and reject when this ratio exceeds a threshold calibrated to the desired α. No test with the same α can have higher power (lower β) against θ₁.

Λ(x)=p(x;θ1)p(x;θ0)>kα\Lambda(x) = \frac{p(x;\theta_1)}{p(x;\theta_0)} > k_\alpha

Why it matters — Likelihood ratios are everywhere in ML: this lemma is why the likelihood-ratio test is the gold standard, and why ROC curves — which sweep the threshold k — measure a test's full discrimination ability.

The trade-off: lowering α raises β. The threshold k_α controls the exchange. Decrease α (a stricter test, 0.01 instead of 0.05) and k_α must increase, the rejection region shrinks, and β increases — power drops. Increase α (more lenient, 0.10) and the opposite happens. The ROC curve traces this trade-off by sweeping the threshold k; the area under it (AUC) measures the test's overall discrimination ability. For a t-test with fixed n, the only way to reduce both α and β simultaneously is to increase the sample size n, which reduces the variance of x̄ and sharpens the test. On the demo corpus (n = 8, α = 0.05), power against H₁: μ₁ = 10.5 is barely above α, but grows steadily as μ₁ drifts further from μ₀ = 10 — power increases with effect size (μ₁ − μ₀)/s.

Try this ▶ The panel below sweeps the alternative μ₁ and shows the resulting power (n = 8, α = 0.05) as both a table and a curve — see how slowly power grows near μ₀ and how it climbs with effect size.

Computing…

3 — Multiple comparison correction: Bonferroni and BH/FDR. When testing m hypotheses simultaneously, the probability of at least one false positive inflates well beyond α.

ⓘ Concept: Family-wise error rate (FWER) and Bonferroni

FWER is the probability of at least one false rejection when all null hypotheses are true; the formula holds for independent tests. The Bonferroni correction runs each test at α/m, which guarantees FWER ≤ α regardless of dependence between the tests. Cost: it is very conservative — power decreases substantially for large m.

FWER=1−(1−α)m≈mα\text{FWER} = 1 - (1-\alpha)^m \approx m\alpha

Why it matters — Run 20 comparisons at α = 0.05 and the chance of at least one spurious 'significant' result is about 64 % — uncorrected multiple testing is the classic way to fool yourself with NLP leaderboards.

ⓘ Concept: False Discovery Rate and Benjamini-Hochberg (1995)

FDR is the expected proportion of false positives V among all rejected hypotheses R. The Benjamini-Hochberg procedure controls FDR ≤ q: (1) order the m p-values p₍₁₎ ≤ p₍₂₎ ≤ … ≤ p₍m₎; (2) find the largest k such that p₍k₎ ≤ (k/m)·q; (3) reject H₍₁₎, …, H₍k₎. Less conservative than Bonferroni — more discoveries while controlling the FDR.

FDR=E ⁣[VR],p(k)≤km q\text{FDR} = \mathbb{E}\!\left[\frac{V}{R}\right], \qquad p_{(k)} \le \frac{k}{m}\, q

Why it matters — BH is the standard in genomics and increasingly in large-scale NLP ablation studies: it buys more discoveries than Bonferroni at the cost of allowing a controlled fraction of them to be false.

Worked example — 5 NLP model comparisons, q = 0.05. Raw p-values: H1 (Model A vs B, precision) 0.001; H2 (A vs B, recall) 0.013; H3 (A vs C, BLEU) 0.042; H4 (B vs C, F1) 0.150; H5 (A vs C, perplexity) 0.420.

HypothesisRaw pBH rank kBH threshold k/m·qBH reject?Bonferroni reject? (p < α/m = 0.01)
H1: A vs B (precision)0.00110.010✓ Yes✓ Yes
H2: A vs B (recall)0.01320.020✓ Yes✗ No
H3: A vs C (BLEU)0.04230.030✗ No✗ No
H4: B vs C (F1)0.15040.040✗ No✗ No
H5: A vs C (perplexity)0.42050.050✗ No✗ No

BH rejects 2 hypotheses vs Bonferroni's 1 — one extra discovery, at the cost of allowing up to 5 % of rejected hypotheses to be false positives (in expectation).

Self-test — 5 questions

Write the formula for the one-sample t-statistic and name the distribution it follows under H₀. Why is it not a standard normal?

State the Neyman-Pearson lemma. What does 'uniformly most powerful' mean?

If you run 20 independent tests each at α = 0.05, what is the approximate probability of at least one false positive?

What does the Bonferroni correction do, and what is its main drawback?

In the Benjamini-Hochberg procedure with m=5 hypotheses and q=0.05, what BH threshold applies to the hypothesis ranked 3rd (k=3)?

Ask the mentor about this module

Ask a question about this content. The mentor explains and grounds its answer in what you are studying; asking is recorded as a learning signal, not a grade.

Ctrl/Cmd + Enter to send

🎓 Practice ladder

5 graded rungs · ~49 min

Five graded exercises, worked directly in the browser — the code runs locally in Pyodide and is checked against visible and hidden tests. The explorer rung asks only for a verdict — which claim does the data support?; the core arc runs the module's t-test with scipy, rebuilds the permutation test from scratch, then grades the deep-dive's multiple-testing numbers; the senior rung defends a test choice for a messy real-world comparison.

Rung 1 — Run the t-test with scipy

Loading exercise…

Rung 2 — Build the permutation test

Loading exercise…

Rung 3 — How many false alarms?

Loading exercise…

Explorer rung — Which claim does the data support?

Loading exercise…

Senior rung — Choose and defend a test for a messy comparison

Loading exercise…

📝 Check Your Understanding

Given sentence lengths [3, 5, 4, 6] for the first half of the corpus, what is the sample mean?

A 95 % CI for mean sentence length is [4.2, 5.8]. Which interpretation is correct? (a) 95 % of sentences have length in [4.2, 5.8]; (b) if we repeated the sampling many times, 95 % of such intervals would contain the true mean.

If p = 0.03 and significance level α = 0.05, do we reject or fail to reject H₀?

What is the null hypothesis in a t-test comparing mean sentence lengths of two corpus subsets?

A Type I error is a false positive. In an NLP significance test, what would a Type I error mean?

Module Review: what is the difference between a one-tailed and a two-tailed hypothesis test?

📚 Go Further

Curated resources to explore statistical inference and hypothesis testing further.

TypeResource
VideoStatQuest with Josh Starmer — P-values, Clearly Explained! and Hypothesis Testing series (YouTube)
CourseCoursera — Inferential Statistics, Duke University / Statistics with R Specialization (free audit)
ReferenceAllen Downey, Think Bayes 2nd ed. (free online at greenteapress.com)
ExerciseKhan Academy — Significance tests (hypothesis testing) practice problems

Try it yourself

A scratch console for this page's ideas — ungraded, nothing you run here is recorded.

Scratch console

A scratch console with the scientific stack (pandas, numpy, scikit-learn). Runs on the server — no network, resource-limited and measured.

Output appears here.

Where next?

Continue

Resampling & Confidence Intervals →

Statistics Core · P1

Later in Statistics Core

Go up a level