Probability Distributions

P1.statistics-core.02 · Audience: guest, it-ml, language-pro · Prerequisites: Descriptive Statistics

Descriptive Statistics Statistical Inference & Hypothesis Testing

LLM grading (this page):Real gradingOff — no LLM calls

A distribution describes all possible outcomes and their probabilities. Understanding four key distributions — Bernoulli, Binomial, Normal, Poisson — unlocks most of statistical NLP, and the module ends with a live Central Limit Theorem lab.

Step 1 / 6 — Bernoulli Distribution — Binary Events

🗣️ From a linguist's perspective: Bernoulli as a binary linguistic event

Many linguistic observations are binary: a word is either present in a document or absent; a sentence is either grammatical or ungrammatical; a translation is either accepted or rejected. The Bernoulli distribution models exactly this: one trial, two outcomes, one parameter p (probability of success). Summing many Bernoulli trials gives the Binomial distribution — for example, how many sentences in a corpus contain a passive construction.

ⓘ Concept: Bernoulli distribution Bernoulli(p)

Models a single binary trial with two outcomes: X = 1 with probability p (success), X = 0 with probability 1−p (failure).

Mean: E[X] = p. Variance: Var(X) = p(1−p) — maximised at p = 0.5. At p = 0 only failure is possible; at p = 1 only success; at p = 0.5 both outcomes are equally likely and the uncertainty is greatest.

P(X=x) = p^x (1-p)^{1-x}, \quad x \in \{0,1\}

Why it matters — Each attention weight is in (0, 1) — not a strict Bernoulli, but sparse attention mechanisms that hard-threshold weights use the Bernoulli intuition: attend (1) or ignore (0) each key position.

Try this ▶ Move the p slider to 0 — only failure is possible. Move it to 1 — only success. Set p = 0.5 and the two bars become equal.

p (probability of success): 0.3

Computing…

Corpus example: does the word 'a' appear in a sentence? Count the sentences of the demo corpus that contain it and divide by the number of sentences — that fraction p̂ is the natural estimate of the Bernoulli parameter p (you will see in the deep dive below that it is exactly the maximum-likelihood estimate).

🎓 Going deeper — the CLT via MGFs and maximum likelihood

Data-generating stories. Each distribution answers a specific "how was this data made?" question — naming the story is the fastest way to pick the right model. Bernoulli/Binomial: successes in n independent yes/no trials. Geometric: trials until the first success. Negative Binomial: trials until the r-th success. Poisson: rare independent events in a fixed interval (the Binomial limit for large n, small p). Exponential: waiting time between Poisson events (memoryless). Gamma: waiting time until the k-th Poisson event. Uniform: every outcome in a range equally likely. Normal: sums/averages of many small independent effects (via the CLT). Beta: a distribution over a probability in [0, 1] (conjugate prior for the Bernoulli).

The CLT, precisely. For i.i.d. X₁, …, Xₙ with finite mean μ and finite variance σ², the standardised sample mean converges in distribution to a standard normal:

\sqrt{n}\,\frac{\bar{X}_n - \mu}{\sigma} \xrightarrow{d} \mathcal{N}(0,1)

equivalently X̄ₙ ≈ N(μ, σ²/n) for large n. Assumptions and limitations: the CLT needs a finite variance — the Cauchy distribution has no finite mean or variance, so the CLT fails for it (the sample mean of Cauchy samples is itself Cauchy, never Normal). The convergence rate is O(1/√n) (Berry–Esseen theorem); heavy skew or heavy tails require a larger n before the normal approximation is accurate.

The moment-generating function. For a random variable X, the MGF is

M_X(t) = \mathbb{E}[e^{tX}], \quad t \in \mathbb{R}

Three key properties: the k-th moment is the k-th derivative at 0; the uniqueness theorem — if two distributions share the same MGF in a neighbourhood of 0, they are identical in distribution; and the convolution property — for independent X, Y, the MGF of the sum is the product of the MGFs. The standard normal N(0, 1) has MGF e^(t²/2).

Proof sketch of the CLT. Standardise each observation: Yᵢ = (Xᵢ − μ)/σ has mean 0 and variance 1, and the standardised sample mean is Zₙ = (1/√n)·ΣYᵢ. Step 1 — by the convolution property and scaling:

M_{Z_n}(t) = \left[M_Y\!\left(\frac{t}{\sqrt{n}}\right)\right]^n

Step 2 — Taylor-expand M_Y around 0. Since E[Yᵢ] = 0 and E[Yᵢ²] = 1, the first derivative at 0 vanishes and the second is 1:

M_Y(u) = 1 + \tfrac{u^2}{2} + O(u^3) \quad\Rightarrow\quad M_Y\!\left(\tfrac{t}{\sqrt{n}}\right) = 1 + \frac{t^2}{2n} + O\!\left(n^{-3/2}\right)

Step 3 — take the limit, using lim (1 + a/n)ⁿ = eᵃ:

M_{Z_n}(t) = \left(1 + \frac{t^2}{2n} + O\!\left(n^{-3/2}\right)\right)^n \xrightarrow{n\to\infty} e^{t^2/2}

The limiting MGF e^(t²/2) is exactly the MGF of N(0, 1); by the uniqueness theorem, Zₙ converges in distribution to N(0, 1). ∎

The MLE framework. Maximum Likelihood Estimation finds the parameter θ that makes the observed data most probable. Given i.i.d. observations x₁, …, xₙ from p(x; θ): the likelihood is the product of the densities; the log-likelihood turns that product into a sum,

\ell(\theta) = \sum_{i=1}^n \log p(x_i; \theta)

the score function is its gradient ∂ℓ/∂θ, and the MLE is the θ that maximises ℓ — found by solving score = 0.

Case 1 — Gaussian (e.g. sentence lengths). The log-likelihood is

\ell(\mu,\sigma^2) = -\frac{n}{2}\log(2\pi\sigma^2) - \frac{1}{2\sigma^2}\sum_{i=1}^n(x_i-\mu)^2

Setting the score w.r.t. μ to zero gives μ̂_MLE = x̄; setting the score w.r.t. σ² to zero gives σ̂²_MLE = Σ(xᵢ − x̄)²/n. Note: the MLE for σ² divides by n, not n−1 — it is biased (see the Descriptive Statistics deep dive for the unbiasedness proof).

Case 2 — Bernoulli (e.g. does the sentence contain 'language'?). The log-likelihood is

\ell(p) = \sum_{i=1}^n \bigl[x_i \log p + (1-x_i)\log(1-p)\bigr]

Setting the score Σxᵢ/p − (n − Σxᵢ)/(1−p) to zero gives p̂_MLE = Σxᵢ/n = x̄ — the MLE is simply the sample proportion, the fraction of successes. On the demo corpus, counting the sentences that contain the word 'language' and dividing by the number of sentences confirms the derivation numerically.

Self-test — 5 questions

Q1 — State the Central Limit Theorem. What does it say about Zₙ = (X̄ₙ − μ)/(σ/√n) as n → ∞?

Q2 — In the MGF proof, what Taylor expansion is applied to M_Y(t/√n), and why does the O(u³) term vanish?

Q3 — Write the log-likelihood for a Gaussian sample x₁, …, xₙ and show that the MLE for μ is x̄.

Q4 — The Gaussian MLE for σ² divides by n. Is this biased or unbiased? What correction gives the unbiased estimator?

Q5 — 4 out of 8 corpus sentences are longer than 10 words. What is the MLE estimate p̂ for the probability that a random sentence is longer than 10 words? Write the log-likelihood ℓ(p̂) for this Bernoulli case.

Ask the mentor about this module

Ask a question about this content. The mentor explains and grounds its answer in what you are studying; asking is recorded as a learning signal, not a grade.

Ctrl/Cmd + Enter to send

🎓 Practice ladder

4 graded rungs · ~30 min

Four graded exercises, worked directly in the browser — the code runs locally in Pyodide and is checked against visible and hidden tests. The explorer rung needs no code at all — turn one knob and watch the bell narrow; the next two build the CLT lab's machinery yourself; the last grades the deep-dive's numeric self-test.

Rung 1 — Simulate the sampling distribution

Loading exercise…

Rung 2 — The CLT prediction, by hand

Loading exercise…

Rung 3 — Bernoulli MLE and its log-likelihood

Loading exercise…

Explorer rung — Turn one knob and watch the bell narrow

Loading exercise…

📝 Check Your Understanding

Q1. A token appears in a sentence with probability 0.3. What is the probability it does NOT appear?

Q2. If each of 4 sentences independently contains 'cat' with p = 0.5, what is P(exactly 2 sentences contain 'cat')? (round to 4 decimal places)

Q3. Token frequencies across a large corpus roughly follow which heavy-tailed distribution?

Q4. If 'the' appears on average 2 times per sentence, what distribution models its count per sentence?

Q5. In a softmax language model, the output distribution over the vocabulary is an example of which distribution family?

Module Review. Why can sentence-length distributions often be approximated with a Gaussian for large corpora?

📚 Go Further

Curated resources to explore probability distributions further.

Type	Resource
Video	StatQuest with Josh Starmer — Probability Distributions series: Normal, Binomial, Poisson (YouTube)
Video	3Blue1Brown — Bayes Theorem and probability series (YouTube)
Course	Khan Academy — Statistics and Probability: Random variables and probability distributions (free)
Reference	Allen Downey, Think Stats 2nd ed. Ch. 3–5 (free online at greenteapress.com)

✅ Key takeaways

Bernoulli(p) models one binary trial. Binomial(n, p) counts successes in n trials. Mean = np, Variance = np(1−p).
Normal N(μ, σ²) is symmetric and bell-shaped; 68/95/99.7 % of values fall within 1/2/3 standard deviations. The Central Limit Theorem makes it universal.
Poisson(λ) models rare-event counts with mean = variance = λ. It approximates Binomial when n is large and p is small.
The Multinomial generalises Binomial to k categories. The Transformer's softmax output is a Multinomial distribution over the vocabulary at each step.
Cross-entropy loss −log(pᵧ) is the negative log-probability of the true token under the model's Multinomial output distribution.

Next: Statistical Inference introduces estimators, p-values, t-tests, and chi-square tests applied to NLP corpus data.

Try it yourself

A scratch console for this page's ideas — ungraded, nothing you run here is recorded.

Scratch console

A scratch console with the scientific stack (pandas, numpy, scikit-learn). Runs on the server — no network, resource-limited and measured.

Output appears here.

Where next?

Continue

Statistical Inference & Hypothesis Testing →

Statistics Core · P1

Later in Statistics Core

Go up a level