PCA & Visualising Embeddings
P2.aiml-foundations.04 · Audience: it-ml · Prerequisites: Word Similarity & Analogies
We can measure word similarity, but eight-dimensional vectors cannot be drawn. Principal Component Analysis (PCA) finds the two directions along which the vocabulary spreads out the most and projects every word onto them — turning the embedding cloud into a flat map you can actually look at. PCA turns out to be nothing more than SVD applied to centred data, tying this whole track together.
You cannot draw an 8-dimensional scatter plot. PCA finds the two directions of greatest variation in the cloud of word vectors and projects everything onto them. You lose some information (six dimensions are discarded), but you keep the two contrasts that separate the words the most — the best flat photograph of a high-dimensional sculpture.
🗣️ From a linguist's perspective: PCA as finding the most informative view of a feature matrix
Going deeper (technical) — the mathematics of PCA
Covariance and Bessel's correction. After centring the data X̂ = X − mean(X), the sample covariance is C = X̂ᵀX̂ / (n − 1). The divisor (n − 1) rather than n is Bessel's correction: it removes the downward bias that arises because the mean was estimated from the same data. C is symmetric and positive semi-definite, so it has an orthonormal eigenbasis.
PCA-via-SVD equivalence. Substituting the SVD X̂ = U Σ Vᵀ into the covariance gives C = V (Σ² / (n − 1)) Vᵀ. So the eigenvectors of the covariance matrix are exactly the right singular vectors V — the principal components — and the eigenvalues are λᵢ = σᵢ² / (n − 1). The scores (projected coordinates) are P = X̂ V = U Σ. This is why PCA and SVD are the same computation seen from two angles.
Whitening. Rescaling each principal coordinate by 1/√λᵢ gives Z = X̂ V D⁻¹ with Cov(Z) = I — a decorrelated, unit-variance representation. Whitening is a common preprocessing step before clustering or downstream models, ensuring no single direction dominates purely because of scale.
Ask the mentor about this module
Ask a question about this content. The mentor explains and grounds its answer in what you are studying; asking is recorded as a learning signal, not a grade.
Try it yourself
A scratch console for this page's ideas — ungraded, nothing you run here is recorded.
Scratch console
A scratch console with the scientific stack (pandas, numpy, scikit-learn). Runs on the server — no network, resource-limited and measured.
Output appears here.
Where next?
Go up a level