Account
Guided pathThis is part of Understand Transformers & BERTBack to the path

Mathematical Foundations

P10.text-representation.01 · Audience: guest, it-ml, language-pro

Customise the corpus these labs use

Prerequisite concepts needed before diving into Transformers — four mathematical operations that appear in every layer, each computed live with every intermediate value shown.

Step 1 / 5 — Vectors & Cosine Similarity
What this means in plain words

A word can be described by several properties at once — for example, how formal it is, how concrete (versus abstract) it is, or how emotionally charged it is. If we write each property as a number, we get a list: [0.8, 0.2, 0.5]. That list is a vector. Two words that describe similar things will have similar numbers in their lists; two unrelated words will have very different numbers.

Cosine similarity is a way to measure how alike two such lists are — a score of 1 means the two word-descriptions point in exactly the same direction, 0 means they share nothing in common.

Note: a vector here is a list of numbers — not a directed arrow anchored at a point in physical space. The geometric diagram below is just one way to picture the angle between two lists; the list itself is what matters.

🗣️ From a linguist's perspective: vectors as semantic feature profiles
In lexical semantics, a word's meaning can be described by listing its features: [+animate, +human, −male, +adult] for woman. A vector is the same idea extended to numbers: an ordered list of numeric values, each representing one 'feature dimension'. Words with similar meanings end up with similar numbers in similar positions — which is why two similar vectors point in nearly the same direction. Cosine similarity measures that directional agreement: 1.0 means identical direction (synonyms), 0.0 means unrelated (orthogonal), −1.0 means opposite meaning.
ⓘ Concept: Vectors and Cosine Similarity
A vector is an ordered list of numbers representing a point in n-dimensional space. In NLP, we map words and sentences to vectors so we can compute distances and similarities. The dot product of two vectors a and b is the sum of their element-wise products. Dividing by the product of their magnitudes gives cosine similarity: a value in [−1, 1] where 1 = identical direction, 0 = orthogonal, −1 = opposite.
cos⁡(θ)=a⋅b∥a∥∥b∥\cos(\theta) = \frac{\mathbf{a} \cdot \mathbf{b}}{\|\mathbf{a}\| \|\mathbf{b}\|}

Why it matters — Cosine similarity is used throughout Transformers: in attention to measure query–key alignment, and at inference time to find the nearest sentence embedding.

Reference: 3Blue1Brown — Essence of Linear Algebra

Try it — adjust vectors and see similarity live

Try this ▶ Drag vector b to match vector a exactly and watch the cosine similarity reach 1.0 — then point them in opposite directions to see it drop toward −1.0.

Computing…

📚 Go further

Curated resources to explore text representation foundations further.

Check your understanding

Given a = [1, 0] and b = [0, 1], what is their cosine similarity?

Given E = [[1,0],[0,1]] and W = [[2,3],[4,5]], what is entry [0, 1] of E × W?

Given logits [1.0, 2.0], what is softmax([1, 2])[1]? (round to 4 d.p.)

Given x = [1.0, 3.0, 5.0, 7.0], what is the mean that LayerNorm subtracts?

Which Transformer operation uses dot products of Q and K vectors to decide which tokens to attend to?

Ask the mentor about this module

Ask a question about this content. The mentor explains and grounds its answer in what you are studying; asking is recorded as a learning signal, not a grade.

Ctrl/Cmd + Enter to send

Try it yourself

A scratch console for this page's ideas — ungraded, nothing you run here is recorded.

Scratch console

A scratch console with the scientific stack (pandas, numpy, scikit-learn). Runs on the server — no network, resource-limited and measured.

Output appears here.

Where next?

Continue

Tokenization →

Text Representation · P10

Later in Text Representation

Go up a level