Multi-Head Attention

P10.attention.03 · Audience: guest, it-ml, language-pro · Prerequisites: Scaled Dot-Product Attention

Scaled Dot-Product Attention Encoder Block

From a single attention pass to several running in parallel — head splits, per-head patterns, concatenation, and the output projection W_O.

Step 1 / 5 — Motivation: Why Multiple Heads?

🗣️ From a linguist's perspective: simultaneous levels of analysis

A linguist reads one sentence at several levels at once — phonology, syntax, semantics, pragmatics. Each analysis attends to different words for different reasons. Multi-head attention gives the model that faculty: each head is one analytical reading, run in parallel.

ⓘ Concept: Why multiple attention heads?

A single head must compress every kind of relationship — syntactic, semantic, positional — into one weight pattern. Multiple heads, each in a smaller d_k-dimensional subspace, can each specialise.

\text{MultiHead}(X) = \text{Concat}(\text{head}_1,\dots,\text{head}_H)\,W_O,\qquad \text{head}_h = \text{Attention}(QW_{Q_h}, KW_{K_h}, VW_{V_h})

Why it matters — One head averages all relationship types into one pattern; separate heads can specialise — BERT-base uses 12, GPT-3 uses 96.

Reference: Vaswani et al. 2017, §3.2.2

d_modelnum_heads

Computing…

📚 Go further

Vaswani et al. 2017, §3.2.2 — Attention Is All You Need
Jay Alammar — The Illustrated Transformer
Harvard NLP — The Annotated Transformer
Lilian Weng — Attention? Attention!

Check your understanding

In multi-head attention, is the input matrix X split across heads?

If d_model = 16 and num_heads = 4, what is d_k?

What would be lost without the final W_O projection?

Ask the mentor about this module

Ask a question about this content. The mentor explains and grounds its answer in what you are studying; asking is recorded as a learning signal, not a grade.

Ctrl/Cmd + Enter to send

Try it yourself

A scratch console for this page's ideas — ungraded, nothing you run here is recorded.

Scratch console

A scratch console with the scientific stack (pandas, numpy, scikit-learn). Runs on the server — no network, resource-limited and measured.

Output appears here.

Where next?

Continue

Encoder Block →

Transformer Architecture · P10

Go up a level