Mathematical Foundations

P10.text-representation.01 · Audience: guest, it-ml, language-pro

Does it understand? Recognition and the question of sentience Tokenization

A Transformer can look impenetrable from the outside, yet everything it does to a sentence is built from just four mathematical operations, repeated layer after layer. Once you can read those four — comparing vectors (lists of numbers) with cosine similarity, transforming them with matrix multiplication, turning raw scores into shares with softmax, and keeping the numbers stable with layer normalisation — no later module will hold anything unfamiliar. This module builds that intuition one operation at a time, computing each of them live so that every intermediate value stays visible.

Step 1 / 5 — Vectors & Cosine Similarity

What this means in plain words

A word can be described by several properties at once — for example, how formal it is, how concrete (versus abstract) it is, or how emotionally charged it is. If we write each property as a number, we get a list: [0.8, 0.2, 0.5]. That list is a vector. Two words that describe similar things will have similar numbers in their lists; two unrelated words will have very different numbers.

Cosine similarity is a way to measure how alike two such lists are — a score of 1 means the two word-descriptions point in exactly the same direction, 0 means they share nothing in common.

Note: a vector here is a list of numbers — not a directed arrow anchored at a point in physical space. The geometric diagram below is just one way to picture the angle between two lists; the list itself is what matters.

🗣️ From a linguist's perspective: vectors as semantic feature profiles

In lexical semantics, a word's meaning can be described by listing its features: [+animate, +human, −male, +adult] for woman. A vector is the same idea extended to numbers: an ordered list of numeric values, each representing one 'feature dimension'. Words with similar meanings end up with similar numbers in similar positions — which is why two similar vectors point in nearly the same direction. Cosine similarity measures that directional agreement: 1.0 means identical direction (synonyms), 0.0 means unrelated (orthogonal), −1.0 means opposite meaning.

ⓘ Concept: Vectors and Cosine Similarity

A vector is an ordered list of numbers representing a point in n-dimensional space. In NLP, we map words and sentences to vectors so we can compute distances and similarities. The dot product of two vectors a and b is the sum of their element-wise products. Dividing by the product of their magnitudes gives cosine similarity: a value in [−1, 1] where 1 = identical direction, 0 = orthogonal, −1 = opposite.

\cos(\theta) = \frac{\mathbf{a} \cdot \mathbf{b}}{\|\mathbf{a}\| \|\mathbf{b}\|}

Why it matters — Cosine similarity is used throughout Transformers: in attention to measure query–key alignment, and at inference time to find the nearest sentence embedding.

Reference: 3Blue1Brown — Essence of Linear Algebra

Try it — adjust vectors and see similarity live

a[0]: 3a[1]: 1b[0]: 1b[1]: 3

Try this ▶ Drag vector b to match vector a exactly and watch the cosine similarity reach 1.0 — then point them in opposite directions to see it drop toward −1.0.

Computing…

Open in Labs: Dot product and cosine

Open in Labs: Dot product against cosine

✅ Key takeaways

A vector is an ordered list of numbers, and direction (angle) matters more than magnitude — which is why cosine similarity is the standard measure of how alike two word representations are.
Matrix multiplication applies a learned transformation to every token vector simultaneously — this is how the Q, K, and V projections work.
Softmax converts raw scores into a probability distribution summing to 1, and layer normalisation rescales each representation to zero mean and unit variance — the pair that turns scores into attention weights and keeps activations stable across layers.
These four operations are the entire toolkit — every later stage just chains them — but none of them can run until text becomes numbers, and that conversion of raw text into token IDs is where the next module begins.

Next: P10.text-representation.02 — Tokenization

📚 Go further

Curated resources to explore text representation foundations further.

Jurafsky & Martin, Speech and Language Processing (3rd ed.) — Ch. 2: Regular Expressions, Text Normalisation, Edit Distance (free PDF draft)
Jay Alammar — The Illustrated Word2Vec — visual introduction to word representations
Hugging Face — NLP Course, Chapter 1: Transformer Models (free)

Check your understanding

Self-check

Given a = [1, 0] and b = [0, 1], what is their cosine similarity?

Self-check

Given E = [[1,0],[0,1]] and W = [[2,3],[4,5]], what is entry [0, 1] of E × W?

Self-check

Given logits [1.0, 2.0], what is softmax([1, 2])[1]? (round to 4 d.p.)

Self-check

Given x = [1.0, 3.0, 5.0, 7.0], what is the mean that LayerNorm subtracts?

Self-check

Which Transformer operation uses dot products of Q and K vectors to decide which tokens to attend to?

Ask the mentor about this module

Ask a question about this content. The mentor explains and grounds its answer in what you are studying; asking is recorded as a learning signal, not a grade.

Images, PDF or text. Kept on this device only.

Keeping your files on this device

Keep a copy of the files I attach, in this browser

Off by default. The mentor always gets your file; this only decides whether your own copy stays here. Copies live in this browser only - they do not follow you to another device, and clearing site data removes them.

Ctrl/Cmd + Enter to send

Try it yourself

A scratch console for this page's ideas — ungraded, nothing you run here is recorded.

Scratch console

A scratch console with the scientific stack (pandas, numpy, scikit-learn). Runs on the server — no network, resource-limited and measured.

Output appears here.

My notes on this module

Loading your notes...

Where next?

Continue

Tokenization →

Text Representation · P10

Later in Text Representation

Go up a level