Labs

The attention matrix

Each query scores every key, the scores become weights, and the weights mix the values.

6 tokens · d_k = d_v = 4

Softmax

Logits become probabilities; temperature, masking and scale decide how peaked they are and whether a gradient survives.

8 logits

Heads split and mixed

One attention split into heads by slicing the projections, each head's map on its own, the contexts concatenated and mixed back by W_O.

6 tokens · d_model = 8 · 2 heads of d_k = 4

Dot product and cosinefrom a module

Two vectors, one number: how much they point the same way, with and without their lengths.

Matrix multiplicationfrom a module

Every cell of the product is one dot product — rows of A against columns of B.

Pooling with and without the mask

One sentence vector from a batch's token rows — [CLS], mean or max — and what the padding rows do to it unless the mask keeps them out.

8 rows × 4 dims · 2 of them padding

Dot product against cosine

The same vectors under both measures, whose neighbour changes, and anisotropy — a shared offset that makes every cosine high until the mean is subtracted.

8 vectors × 4 dims · two clusters

Dropoutfrom a module

Zero a random share of activations while training, and scale the rest so the sum holds.

Layer normalisationfrom a module

Centre and scale one row to zero mean and unit variance, then let γ and β undo what training wants undone.

The sinusoidal tablefrom a module

Position as a fingerprint of sines and cosines at geometrically spaced frequencies.

RoPE and ALiBifrom a module

Position as a rotation of the query–key pair, or as a penalty on distance — the two modern answers.

Positions encoded and compared

The sinusoidal table and why PE·PEᵀ depends only on the offset; the same property as a rotation (RoPE) and as a penalty on distance (ALiBi).

32 positions × d_model = 16

Rows normalised

Each row to mean 0 and variance 1 on its own, then γ and β; where ε matters, and what batch norm would have done to the same matrix instead.

4 rows × 8 features

Tokens expanded and projected

The feed-forward sublayer one position at a time: widen to d_ff, activate, narrow back — and what the activation and the expansion each buy.

4 rows · d_model = 4 → d_ff = 16 → 4

The perceptronfrom a module

A weighted sum, a threshold, and the smallest unit that can learn a line.

Cross-entropy against MSE

The same batch of predictions scored two ways — how hard each loss punishes a confident mistake, what a batch mean hides, and which positions a mask lets count.

5 classes · a batch of 4

Forward and backward

A small MLP run forward, its loss sent back through the chain rule, every gradient checked against the definition of a derivative, and one step taken.

3 → 4 → 4 → 3 · a batch of 4

SGD, momentum and Adam

The same gradient taken three ways across a drawn surface: plain descent, descent with velocity, and a per-coordinate step — and what the learning rate does to each.

a 2-D surface · 60 steps

A gradient down twelve layers

What the backward pass still carries at each depth — with the residual path, without it, and with LayerNorm before or after.

12 layers × width 8 · a batch of 4

Parameters, FLOPs and memory

A model's size from its dials — every parameter counted by the platform's own derivation, the FLOPs a token costs, and what training it holds in memory.

this platform's model, up to a 7B decoder

One distribution, one token

Greedy, temperature, top-k and top-p on one next-token distribution, with a thousand seeded draws to show what each keeps and what each throws away.

12 candidates · 1,000 draws

BPE against WordPiece

Two merge rules on one editable word table — the most frequent pair against the highest-scoring one — and how each then splits a word it has never seen.

10 words · 20 merges