Descriptive Statistics

P1.statistics-core.01 · Audience: guest, it-ml, language-pro

LLM grading (this page):Real gradingOff — no LLM calls

Descriptive statistics summarise a distribution in a handful of numbers. This module builds that toolkit on the demo corpus — mean, median, mode, variance, standard deviation, IQR, skewness — and connects each one to LayerNorm and the Zipf-like skew of natural language.

Step 1 / 5 — Measures of Centre: Mean, Median, Mode

🗣️ From a linguist's perspective: measures of centre as summary of a word distribution

Corpus linguists routinely summarise distributions: the mean sentence length, the median word frequency, the mode (most frequent word). These give a quick picture of a corpus without listing every data point. In the Transformer, LayerNorm computes the mean of an activation vector at each token position — the same arithmetic mean you use to summarise a frequency list.

ⓘ Concept: Mean, median, mode

For a dataset x₁, x₂, …, xₙ: the mean μ is the arithmetic average Σxᵢ / n; the median is the middle value when sorted (average of the two middle values if n is even); the mode is the most frequently occurring value (it may not be unique).

The mean is sensitive to outliers; the median is robust. When the mean > median, the distribution is right-skewed (pulled by large values). When mean < median, it is left-skewed.

\mu = \frac{1}{n}\sum_{i=1}^{n} x_i

Why it matters — LayerNorm uses the mean of the activation vector to centre it: x_centred = x − μ. After centring, the mean is 0, so the normalised activations do not drift.

Computing…

Mean > median on the token counts indicates right skew — many rare words, a few very common ones. This is the signature shape of natural language.

🎓 Going deeper — why n−1? The unbiasedness proof

Expectation and the variance identity. The expectation of a random variable is its probability-weighted average, and it is linear:

\mathbb{E}[aX + bY] = a\,\mathbb{E}[X] + b\,\mathbb{E}[Y]

With μ = 𝔼[X], variance is the expected squared deviation from the mean. Expanding the square and applying linearity gives the workhorse identity:

\mathrm{Var}(X) = \mathbb{E}\big[(X - \mu)^2\big] = \mathbb{E}[X^2] - (\mathbb{E}[X])^2

(μ and μ² are constants, so they pull straight out of the expectation; substituting 𝔼[X] = μ collapses the middle term.) Two rules follow: constants shift but do not spread — Var(aX + b) = a²·Var(X) — and for independent X, Y the covariance vanishes, so Var(X + Y) = Var(X) + Var(Y).

The two variance formulas. Use the population formula when the data is the full population; use the sample formula when the data is a sample of size n from a larger population:

\sigma^2 = \frac{1}{N} \sum_{i=1}^{N}(X_i - \mu)^2 \qquad\qquad s^2 = \frac{1}{n-1} \sum_{i=1}^{n}(X_i - \bar{X})^2

The only difference is the denominator: N vs n−1. The n−1 version is called Bessel's correction. On the demo-corpus sentence lengths, s² comes out slightly larger than σ² — dividing by n−1 inflates the estimate to compensate for the downward bias introduced by using x̄ instead of the true μ.

Setup for the proof. Let X₁, …, Xₙ be i.i.d. with 𝔼[Xᵢ] = μ and Var(Xᵢ) = σ², and let X̄ be the sample mean. Four key facts: 𝔼[X̄] = μ; Var(X̄) = σ²/n (i.i.d. terms, variances add); 𝔼[Xᵢ²] = σ² + μ²; and 𝔼[X̄²] = σ²/n + μ² (both from the variance identity above).

Step 1 — expand the sum of squared deviations. Write each deviation as (Xᵢ − μ) − (X̄ − μ) and expand the square. Because Σ(Xᵢ − μ) = n(X̄ − μ), the cross-term collapses:

\sum_{i=1}^n (X_i - \bar{X})^2 = \sum_{i=1}^n (X_i-\mu)^2 - n(\bar{X}-\mu)^2

Step 2 — take the expectation. 𝔼[(Xᵢ − μ)²] = σ² for each i, so the first term contributes nσ². And 𝔼[(X̄ − μ)²] = Var(X̄) = σ²/n, so the second term contributes n · σ²/n = σ². Therefore:

\mathbb{E}\!\left[\sum_{i=1}^n (X_i-\bar{X})^2\right] = n\sigma^2 - \sigma^2 = (n-1)\sigma^2

Step 3 — conclude. Dividing both sides by n−1:

\mathbb{E}[s^2] = \frac{(n-1)\sigma^2}{n-1} = \sigma^2 \qquad \blacksquare

If we had divided by n instead, we would get 𝔼 = (n−1)/n · σ² — strictly less than σ²: a biased estimator that systematically underestimates the true variance. The n−1 denominator corrects exactly for the degree of freedom lost when x̄ is fitted to the very same data.

Self-test — 5 questions

Q1 — Why do we divide by n−1 rather than n when computing sample variance?

Q2 — What is Var(X̄) for an i.i.d. sample of size n with population variance σ²?

Q3 — Use the algebraic identity from the proof to write Σᵢ(Xᵢ−X̄)² in terms of (Xᵢ−μ) and (X̄−μ).

Q4 — A sample of n=5 has values {2, 4, 4, 4, 6}. Compute the population variance σ² and the sample variance s².

Q5 — If you compute the biased estimator σ̂² = Σ(xᵢ−x̄)²/n on a sample of size n=10, by what multiplicative factor must you correct it to get the unbiased s²?

Ask the mentor about this module

Ask a question about this content. The mentor explains and grounds its answer in what you are studying; asking is recorded as a learning signal, not a grade.

Ctrl/Cmd + Enter to send

🎓 Practice ladder

5 graded rungs · ~36 min

Five graded exercises, worked directly in the browser — the code runs locally in Pyodide and is checked against visible and hidden tests. The explorer rung asks for no code at all — predict three numbers straight off the token table; each core rung applies one step of this module to real data; the fourth grades the deep-dive's numeric self-test.

Rung 1 — Describe the token counts

Loading exercise…

Rung 2 — Rebuild the sample variance

Loading exercise…

Rung 3 — Outliers by IQR fences

Loading exercise…

Rung 4 — Bias in action: the n−1 factor

Loading exercise…

Explorer rung — Predict what describe() will report

Loading exercise…

📝 Check Your Understanding

Q1 — Given the dataset [2, 4, 4, 6], what is the median?

Q2 — For [2, 4, 4, 6] (mean = 4), what is the population variance? (round to 1 decimal place)

Q3 — Why does sample variance divide by n−1 rather than n?

Q4 — A distribution of word frequencies across a corpus is right-skewed. Does the mean exceed or fall below the median?

Q5 — LayerNorm subtracts the mean and divides by the standard deviation. Which two descriptive statistics does it use?

Module Review — If 'the' appears 8 times in 40 tokens, what is its relative frequency? (round to 4 decimal places)

📚 Go Further

Curated resources to explore descriptive statistics further.

Type	Resource
Video	StatQuest with Josh Starmer — Statistics Fundamentals playlist: mean, variance, standard deviation (YouTube)
Course	Khan Academy — Statistics and Probability: Summarising quantitative data (free, exercises included)
Reference	Allen Downey, Think Stats 2nd ed. Ch. 1–3 (free online at greenteapress.com)
Exercise	Khan Academy — built-in practice problems on mean, median, and variance

✅ Key takeaways

The mean is the arithmetic average; the median is the middle value; the mode is the most common value. Mean > median signals right skew.
Variance σ² = Σ(xᵢ−μ)²/n measures average squared deviation; std σ = √σ² measures spread in the original units. IQR = Q3−Q1 measures middle-50 % spread.
Use n−1 (Bessel's correction) for sample variance — it removes the downward bias introduced by using the sample mean instead of the unknown population mean.
Natural-language token distributions are right-skewed: a few function words have very high frequency while the vast majority of words are rare.
LayerNorm directly applies mean and std to activation vectors, centring and normalising them before each sub-layer to stabilise deep network training.

Next: Probability Distributions introduces the Bernoulli, Binomial, Normal, and Poisson distributions and their NLP interpretations.

Try it yourself

A scratch console for this page's ideas — ungraded, nothing you run here is recorded.

Scratch console

A scratch console with the scientific stack (pandas, numpy, scikit-learn). Runs on the server — no network, resource-limited and measured.

Output appears here.

Where next?

Continue

Probability Distributions →

Statistics Core · P1

Later in Statistics Core

Go up a level