Co-occurrence & the Distributional Hypothesis
P2.aiml-foundations.01 · Audience: guest, it-ml, language-pro
How does a machine discover that lion and animal are related without ever being told what either word means? The answer is one of the oldest ideas in linguistics: you shall know a word by the company it keeps. If we simply count which words appear near which, patterns of meaning fall out of the arithmetic. This module builds that count — the co-occurrence matrix — and then sharpens it into a association score called PPMI, the raw material every word embedding is later distilled from.
The linguist J. R. Firth summarised the idea in 1957: you shall know a word by the company it keeps. Words that appear in similar contexts tend to have similar meanings. Tea and coffee share neighbours (cup, drink, hot), and that shared company is exactly what makes them feel related.
🗣️ From a linguist's perspective: Distributional semantics as corpus evidence for meaning
ⓘ Concept: Meaning can be estimated from context alone
Why it matters — Every modern language model rests on this hypothesis. Its embeddings are trained purely on which words appear near which — no dictionary, no grammar rules, just distributional evidence at enormous scale.
Practice — Ladder C: counting your way to meaning
These three graded rungs build the co-occurrence → PPMI → similarity pipeline by hand, one function at a time.
Ask the mentor about this module
Ask a question about this content. The mentor explains and grounds its answer in what you are studying; asking is recorded as a learning signal, not a grade.
Loading exercise…
Loading exercise…
Loading exercise…
Try it yourself
A scratch console for this page's ideas — ungraded, nothing you run here is recorded.
Scratch console
A scratch console with the scientific stack (pandas, numpy, scikit-learn). Runs on the server — no network, resource-limited and measured.
Output appears here.
Where next?
Later in AI/ML Foundations
Go up a level