Embeddings
P10.text-representation.03 · Audience: guest, it-ml, language-pro · Prerequisites: Tokenization
From raw co-occurrence counts to dense word vectors — the corpus as the source of meaning. Every vector on this page is derived from corpus statistics, not random initialisation.
🗣️ From a linguist's perspective: the meaning dictionary
ⓘ Concept: The table
Why it matters — This table is the model's entire lexical knowledge — GloVe shipped exactly such a matrix for 400k words; every LLM still starts with one.
Reference: Pennington et al. 2014 — GloVe
Computing…
📚 Go further
- Jay Alammar — The Illustrated Word2Vec
- Pennington et al. — GloVe project page
- Levy & Goldberg 2014 — Implicit matrix factorization
Check your understanding
Where do the numbers in the embedding table come from on this page?
What does a larger window size change in the co-occurrence matrix?
Why is the embedding lookup so cheap?
Ask the mentor about this module
Ask a question about this content. The mentor explains and grounds its answer in what you are studying; asking is recorded as a learning signal, not a grade.
Try it yourself
A scratch console for this page's ideas — ungraded, nothing you run here is recorded.
Scratch console
A scratch console with the scientific stack (pandas, numpy, scikit-learn). Runs on the server — no network, resource-limited and measured.
Output appears here.
Where next?
Later in Text Representation
This module unlocks
Go up a level