Account

Statistical Laws of Language

P1.statistics-core.06 · Audience: guest, it-ml, language-pro · Prerequisites: Statistics for NLP Evaluation

Real LLM grading for this pageLLM grading (this page):

Every language obeys a handful of powerful statistical laws. Understanding them reveals why NLP models are designed the way they are — from subword tokenisation to smoothing to perplexity, all computed live on the demo corpus.

Step 1 / 6 — Zipf's Law: Rank × Frequency ≈ Constant
🗣️ From a linguist's perspective: Zipf's law as a universal property of natural language
In 1935, linguist George Zipf observed that in any natural language corpus, the most frequent word appears approximately twice as often as the second most frequent, three times as often as the third most frequent, and so on. This rank-frequency relationship holds across languages, genres, and even non-linguistic phenomena (city populations, earthquake magnitudes). The words at the top of the ranking are the function words — the "a", "of", "the" of a language — while thousands of content words trail off into the long tail. For NLP this means vocabulary is always long-tailed: a small core vocabulary accounts for most tokens, while most words appear rarely — the fundamental motivation for subword tokenisation (BPE, SentencePiece).
ⓘ Concept: Zipf's law

For a word at rank r in a corpus (r = 1 is the most frequent word), the frequency is approximately C / r^α, where C is a constant and α ≈ 1 for most natural languages. Equivalently: rank × frequency ≈ constant.

In log space: log(frequency) = log(C) − α · log(rank) — a straight line with slope −α on a log-log plot. Checking Zipf's law is therefore a matter of plotting and fitting a line.

f(r)≈Crαf(r) \approx \frac{C}{r^\alpha}

Why it matters — Zipf's law explains why vocabulary size grows sub-linearly with corpus size (Heaps' law), why most words are rare (data sparsity, smoothing, subword tokenisation), and why frequency-based representations like TF-IDF need the IDF term to down-weight common words.

The panel below computes the rank-frequency plot live on the demo corpus and fits the log-log slope:

Computing…

Ideal Zipf predicts rank × frequency = constant, i.e. a slope of −1. On this tiny corpus the fitted slope comes out around −0.51 — much shallower, because a few dozen tokens cannot fill out the long tail. On real corpora of millions of tokens the slope approaches −1. Check the top-ranked tokens: multiplying each rank by its frequency should give roughly the same constant.

Check your understanding

In a Zipf distribution, if the most frequent word has frequency 100, what frequency does the 10th most frequent word roughly have?

Heaps' law predicts vocabulary size grows as a power of corpus size. If the Heaps exponent β < 1, does vocabulary grow faster or slower than the corpus?

If bigram P('cat' | 'the') = 2/5 = 0.4, what is the log probability (natural log, to 4 decimal places)?

Why does add-one (Laplace) smoothing add 1 to all bigram counts?

If P(w₁) = 0.2, P(w₂ | w₁) = 0.3, and P(w₃ | w₂) = 0.5, what is the bigram sentence probability?

Module review — a language model has perplexity 50. If a better model halves its perplexity, what is the new perplexity?

🎓 Going deeper — from raw counts to principled probabilities

N-gram MLE. For a bigram model, the probability of word w given context c is estimated by maximum likelihood as the relative frequency. The log-likelihood of a corpus w₁, …, w_T under a bigram model is

ℓ(θ)=∑c,wcount(c,w) log⁡θw∣c\ell(\theta) = \sum_{c,w} \mathrm{count}(c,w)\, \log\theta_{w|c}

subject to the constraint that the θ_w|c sum to 1 for each context c. Using a Lagrange multiplier per constraint, the MLE comes out as the relative frequency:

θ^w∣c=count(c,w)∑w′count(c,w′)=count(c,w)count(c)\hat{\theta}_{w|c} = \frac{\mathrm{count}(c,w)}{\sum_{w'}\mathrm{count}(c,w')} = \frac{\mathrm{count}(c,w)}{\mathrm{count}(c)}

The problem: if (c, w) never appeared in training, P_MLE(w|c) = 0 — the model assigns zero probability to any unseen bigram, making the perplexity of any test sentence with new bigrams infinite.

Add-k smoothing as a correction. Add-k smoothing (Lidstone smoothing; k = 1 is Laplace) adds a pseudocount k above 0 to every possible bigram:

Pk(w∣c)=count(c,w)+kcount(c)+k⋅∣V∣P_k(w \mid c) = \frac{\mathrm{count}(c,w) + k}{\mathrm{count}(c) + k \cdot |V|}

where |V| is the vocabulary size. Effects: all bigrams now have positive probability (no zeros); unseen bigrams get probability k / (count(c) + k|V|); seen bigrams have slightly reduced probability (mass is redistributed to unseen events). The normalisation check confirms it is a proper distribution — summing the numerator over all w gives count(c) + k|V|, which cancels the denominator exactly, so the smoothed probabilities sum to 1.

Kneser-Ney intuition. Add-k treats all unseen words equally for a given context. Kneser-Ney (KN) smoothing does better by using how versatile a word is — how many distinct contexts it appears in. The observation: a word like "Francisco" appears frequently — but almost always after "San". A word like "the" appears in many distinct contexts. When predicting an unseen bigram (c, w), we should ask: how likely is w to appear in any new context? This is the continuation probability, proportional to the number of unique left-contexts in which w appears:

Pcont(w)=∣{c:count(c,w)>0}∣∣{(c′,w′):count(c′,w′)>0}∣P_{\text{cont}}(w) = \frac{\left|\{c : \mathrm{count}(c,w) > 0\}\right|}{\left|\{(c',w') : \mathrm{count}(c',w') > 0\}\right|}

Words with high continuation probability ("the", "of", "a") are good fillers for unseen bigrams; words with low continuation probability ("Francisco") are unlikely in new contexts even if frequent overall. On the demo corpus you can verify this directly: the function words precede-and-follow everything, so they have far more distinct left-contexts than any content word — a better proxy for "fill-in" probability than raw unigram frequency. Interpolated KN combines a discounted bigram estimate with this continuation back-off:

PKN(w∣c)=max⁡(count(c,w)−d,  0)count(c)+λ(c)⋅Pcont(w)P_{\text{KN}}(w \mid c) = \frac{\max(\mathrm{count}(c,w) - d,\; 0)}{\mathrm{count}(c)} + \lambda(c) \cdot P_{\text{cont}}(w)

where d in (0, 1) is the discount (counts reduced to redistribute mass to unseen events; typically d ≈ 0.75) and λ(c) = d · |{w : count(c,w) > 0}| / count(c) is the interpolation weight that ensures normalisation. This is a sketch — the full derivation (Kneser & Ney, 1995) uses absolute discounting and shows that the continuation probability emerges from a principled leave-one-out argument.

The Dirichlet-categorical view: add-k is MAP estimation. The add-k formula has a Bayesian interpretation: it is the MAP estimate under a Dirichlet prior on the categorical distribution. A categorical distribution over |V| words is parametrised by θ = (θ₁, …, θ_|V|) with non-negative entries summing to 1, and P(w = k | θ) = θ_k; the MLE from counts n₁, …, n_|V| is n_k / N. The Dirichlet distribution Dir(α₁, …, α_|V|) is the conjugate prior for this likelihood, with density

p(θ∣α)=1B(α)∏k=1∣V∣θkαk−1p(\theta \mid \alpha) = \frac{1}{B(\alpha)} \prod_{k=1}^{|V|} \theta_k^{\alpha_k - 1}

where B(α) is the multivariate Beta function. Interpret each α_k as a pseudocount — as if we had seen word k occur α_k − 1 times before observing any real data: α_k = 1 for all k is the uniform (Laplace) prior with no preference for any word; α_k above 1 encodes a prior belief that word k is common; α_k below 1 gives a sparse prior (most words are rare). Conjugacy: given counts n₁, …, n_|V|, the posterior is

p(θ∣n1,…,n∣V∣)=Dir(α1+n1,  α2+n2,  …,  α∣V∣+n∣V∣)p(\theta \mid n_1,\dots,n_{|V|}) = \mathrm{Dir}(\alpha_1 + n_1,\; \alpha_2 + n_2,\; \dots,\; \alpha_{|V|} + n_{|V|})

— the same family as the prior, with updated pseudocounts. The MAP estimate (the mode of the Dirichlet posterior) is

θ^kMAP=αk+nk−1∑k′(αk′+nk′−1)=αk+nk−1N+∑k′αk′−∣V∣\hat{\theta}^{\text{MAP}}_k = \frac{\alpha_k + n_k - 1}{\sum_{k'} (\alpha_{k'} + n_{k'} - 1)} = \frac{\alpha_k + n_k - 1}{N + \sum_{k'} \alpha_{k'} - |V|}

Now set α_k = k + 1 for all k (a symmetric Dirichlet with concentration k + 1):

θ^kMAP=k+nkN+k⋅∣V∣\hat{\theta}^{\text{MAP}}_k = \frac{k + n_k}{N + k \cdot |V|}

— exactly the add-k formula. Add-k smoothing corresponds to MAP inference under a symmetric Dirichlet prior. As k → 0 the MAP converges to the MLE; larger k redistributes more mass to rare words, shrinking high-frequency estimates. The Bayesian view also gives a principled way to choose k: it is the strength of a symmetric prior belief, and empirical Bayes methods estimate the optimal k from held-out data.

Self-test — 5 questions

Q1 — Write the MLE estimate for a bigram probability P(w|c). What happens when count(c, w) = 0?

Q2 — Write the add-k smoothed bigram probability formula. Show it sums to 1 over all words w for a fixed context c.

Q3 — What is the Kneser-Ney continuation probability P_cont(w)? Why is it better than the standard unigram for backing off to unseen bigrams?

Q4 — State the conjugate prior for the categorical distribution. What is the posterior family after observing counts n₁, …, n_|V|?

Q5 — Show that the MAP estimate under a symmetric Dirichlet prior with α_k = k+1 (concentration k+1) reduces to the add-k smoothing formula.

Ask the mentor about this module

Ask a question about this content. The mentor explains and grounds its answer in what you are studying; asking is recorded as a learning signal, not a grade.

Ctrl/Cmd + Enter to send

🎓 Practice ladder

4 graded rungs · ~34 min

Four graded exercises, worked directly in the browser — the code runs locally in Pyodide and is checked against visible and hidden tests. The explorer rung predicts the next rank's frequency straight from Zipf's recipe; the core arc builds the rank-frequency table, fits the Zipf slope from scratch, then smooths a bigram model and measures its perplexity.

Rung 1 — Build the rank-frequency table

Loading exercise…

Rung 2 — Fit the Zipf slope from scratch

Loading exercise…

Rung 3 — Smooth and measure surprise

Loading exercise…

Explorer rung — Predict the next rank's frequency

Loading exercise…

📝 Check Your Understanding

Q1 — In a Zipf distribution, if the most frequent word has frequency 100, what frequency does the 10th most frequent word roughly have?

Q2 — Heaps' law predicts vocabulary size grows as a power of corpus size. If the Heaps exponent β < 1, does vocabulary grow faster or slower than the corpus?

Q3 — If bigram P('cat' | 'the') = 2/5 = 0.4, what is the log probability? (round to 4 decimal places, use natural log)

Q4 — Why does add-one (Laplace) smoothing add 1 to all bigram counts?

Q5 — If P(w₁) = 0.2, P(w₂ | w₁) = 0.3, P(w₃ | w₂) = 0.5, what is the bigram sentence probability?

Module Review — A language model has perplexity 50. If a better model halves its perplexity, what is the new perplexity?

📚 Go Further

Curated resources to explore NLP evaluation statistics further.

TypeResource
VideoStatQuest with Josh Starmer — F1 Score, Precision, and Recall, Clearly Explained! (YouTube)
ReferenceJurafsky & Martin, Speech and Language Processing 3rd ed. Ch. 4: precision/recall/F1 section (free PDF draft)
ReferencePapineni et al. (2002), BLEU: a Method for Automatic Evaluation of Machine Translation (aclanthology.org)
CourseHugging Face — NLP Course Chapter 3: Fine-Tuning a Pretrained Model — evaluation with metrics (free)
ReferenceHugging Face — Evaluate library documentation (huggingface.co/docs/evaluate)

Try it yourself

A scratch console for this page's ideas — ungraded, nothing you run here is recorded.

Scratch console

A scratch console with the scientific stack (pandas, numpy, scikit-learn). Runs on the server — no network, resource-limited and measured.

Output appears here.

Where next?

Continue

Advanced Probability →

Statistics Advanced · P1

Go up a level