SVD & Dense Word Vectors
P2.aiml-foundations.02 · Audience: guest, it-ml, language-pro · Prerequisites: Co-occurrence & the Distributional Hypothesis
The PPMI matrix from the last module is wide and sparse — 55 numbers per word, almost all zero. Singular Value Decomposition (SVD) squeezes each of those long, sparse rows into a short, dense vector of just a handful of numbers, keeping the strongest patterns and throwing away the noise. The result is a genuine word embedding: a compact list of decimals whose geometry encodes meaning.
A vocabulary of 55 words gives a 55×55 PPMI matrix — over 90% zeros. Real corpora have hundreds of thousands of words, so the matrix would be astronomically large and almost entirely empty. We need a way to keep the information without keeping all the zeros.
ⓘ Concept: Sparse high-dimensional rows waste space and blur similarity
Why it matters — Two words can be similar without sharing a single non-zero column — they might co-occur with different-but-related words. Compression lets these deeper similarities surface.
Going deeper (technical) — the mathematics of SVD
SVD from the eigendecomposition of AᵀA. For any real matrix A, the matrix AᵀA is symmetric and positive semi-definite, so the spectral theorem gives AᵀA = V Λ Vᵀ with non-negative eigenvalues. The singular values are σᵢ = √λᵢ, the right singular vectors are the columns of V, and the left singular vectors are uᵢ = A vᵢ / σᵢ. Assembling these gives A = U Σ Vᵀ.
Eckart–Young theorem. The best rank-k approximation of A in Frobenius norm is Aₖ = Σᵢ₌₁..ₖ σᵢ uᵢ vᵢᵀ — keep the k largest singular triples. The residual error is ‖A − Aₖ‖²_F = σ²ₖ₊₁ + ... + σ²ₙ, so truncating at k discards exactly the variance carried by the dropped components. This is why keeping the top dimensions is provably optimal, not just a heuristic.
Randomised SVD. Computing the full decomposition costs O(V³). For
a rank-k answer, the Halko–Martinsson–Tropp randomised algorithm projects A onto a
small random subspace, giving a near-optimal rank-k factorisation in O(V² k). The
TruncatedSVD estimator used here is a randomised solver, seeded with
random_state=42 for reproducibility.
Practice — Ladder D: singular values by hand
Three graded rungs that build SVD from its eigen-roots, reconstruct a matrix at low rank, and read off explained variance.
Ask the mentor about this module
Ask a question about this content. The mentor explains and grounds its answer in what you are studying; asking is recorded as a learning signal, not a grade.
Loading exercise…
Loading exercise…
Loading exercise…
Try it yourself
A scratch console for this page's ideas — ungraded, nothing you run here is recorded.
Scratch console
A scratch console with the scientific stack (pandas, numpy, scikit-learn). Runs on the server — no network, resource-limited and measured.
Output appears here.
Where next?
Later in AI/ML Foundations
Go up a level