AI/ML Foundations
A track of P2 · Mathematics for ML.
From co-occurrence counts to meaning: PPMI, similarity and the first word vectors — the conceptual runway from counting to embeddings.
Take any shelf of books and count: "coffee" and "cup" turn up in the same sentence again and again; "coffee" and "carburettor" essentially never. Nobody wrote down that coffee goes in cups — the pattern is just sitting there in the counts. So here is the bet this track examines: how much of a word's meaning can you recover from nothing but a table of who-appears-near-whom? The answer turns out to be "startlingly much", and finding out exactly how much is the whole journey.
The track answers it in four sharpening passes. It starts with something you could do with a pencil — counting which words appear near which in a small, fixed set of sentences — and ends with dense vectors whose distances measure similarity and can even complete analogies. Raw co-occurrence counts come first; then PPMI, a reweighting that separates real association from coincidence (frequent words co-occur with everything — that proves nothing); then SVD, which squeezes the wide sparse table into short dense vectors; then cosine similarity, which puts a single number on "these two words are alike".
A final module projects the whole thing onto a 2-D map you can actually look at, using PCA — and there, on the screen, words cluster by meaning even though no human ever labelled one of them. That moment is the payoff of the pencil work: meaning emerging from arithmetic on counts.
The numbers stay small enough to check by hand at every step, which is the point — you should be able to distrust the machine and audit it. And nothing here is a museum piece: every idea in this track returns, grown up, inside the Transformer pillar, where the counting is replaced by learning but the goal — words as vectors, similarity as distance — is exactly the same. This is the runway.