Embeddings

A token ID is only a shelf mark: the number 17 says nothing about what the word means, and a model can compute nothing useful with it. What we need instead is a representation in which similar words receive similar numbers, so that meaning becomes something we can measure. This module builds exactly that from the corpus itself: it counts which words keep company with which, compresses those counts into dense vectors with truncated SVD, and looks up your sentence's rows in the resulting embedding table. Every number you will see here is derived from corpus statistics, never from random initialisation.

Check your understanding