Probability & Statistics Basics
P2.math-foundations.05 · Audience: guest, it-ml, language-pro · Prerequisites: Matrices & Linear Transformations
Probability is the language a Transformer uses to choose its next word. Every time the model generates text, it builds a probability distribution over the whole vocabulary and picks from it. The same statistical ideas — the average and the spread of a set of numbers — reappear inside the model as the machinery of LayerNorm. This module gives you both, with no maths background assumed, using a small corpus of sentences as a running example. For a deeper, dedicated treatment of these same ideas, this pairs with the Statistics pillar in P1 — come back to it whenever you want more worked examples of frequency, mean, and spread.
The simplest way to think about probability is counting. If you take a corpus of words and count how often each word appears, the relative frequency of a word is its count divided by the total number of words. That fraction is a probability: the chance of landing on that word if you reach into the corpus at random. This is the frequentist view — probability as normalised counting.
🗣️ From a linguist's perspective: Probability as the expectation of a word given its context
ⓘ Concept: Probability — the three rules
Why it matters — Softmax turns the model's raw attention scores into a probability distribution over tokens. Its output obeys all three rules: every value sits between 0 and 1, and the values add up to exactly 1 — a valid probability distribution by construction.
A probability P(A) is a number attached to an event that obeys three rules: it is between 0 and 1; the probabilities of all possible outcomes add up to 1 (something must happen); and if two events cannot both occur, the chance of one or the other is the sum of their chances. The relative frequency of a word in a corpus of N words is count(w) / N.
| Token | Count | P(token) |
|---|---|---|
| the | 8 | 0.2000 |
| a | 5 | 0.1250 |
| animals | 3 | 0.0750 |
| wild | 2 | 0.0500 |
Add up P(token) across the whole vocabulary and you always get exactly 1 — the second rule in action.
Going deeper (technical) — joint, marginal & conditional distributions
The joint distribution P(X, Y) gives the probability of X and Y occurring together — a table over every pair of values, or a surface for continuous variables. Everything else about X and Y can be recovered from it.
The marginal distribution of one variable is obtained by summing (or integrating) the other variable out — an operation called marginalisation: P(X = x) is the sum over all y of P(X = x, Y = y). It collapses the joint back down to a single variable.
ⓘ Concept: Conditional distribution and the chain rule
Why it matters — The chain rule is the formal reason an autoregressive language model can be trained one token at a time: the joint probability of a whole sequence factorises into a product of conditional next-token probabilities, each of which is exactly what the network is asked to predict.
The conditional distribution is the distribution of X once Y is fixed: P(X given Y) = P(X, Y) / P(Y), defined whenever P(Y) is positive. Rearranging this definition factors the joint two ways — P(X, Y) = P(X given Y) P(Y) = P(Y given X) P(X) — which is the chain rule of probability. Applied repeatedly it factors any joint distribution into a product of conditionals, one factor per variable given all the earlier ones. That factorisation is precisely how a Transformer models a sentence: the probability of the whole text is the product of the conditional probability of each word given every word before it.
Two variables are independent when the joint factorises into the product of the marginals — equivalently, when conditioning on one changes nothing about the other: P(X, Y) = P(X) P(Y). They are conditionally independent given a third variable Z when P(X, Y given Z) = P(X given Z) P(Y given Z). Conditional independence is the assumption behind Naive Bayes (features independent given the class) and, more generally, behind graphical models. It neither implies nor is implied by marginal independence: two variables can be conditionally independent given Z yet dependent overall, and vice versa.
Ask the mentor about this module
Ask a question about this content. The mentor explains and grounds its answer in what you are studying; asking is recorded as a learning signal, not a grade.
Try it yourself
A scratch console for this page's ideas — ungraded, nothing you run here is recorded.
Scratch console
A scratch console with the scientific stack (pandas, numpy, scikit-learn). Runs on the server — no network, resource-limited and measured.
Output appears here.
Where next?
Later in Mathematics Foundations