Scaled Dot-Product Attention

Causal vs bidirectional masking. Encoders (BERT) attend in both directions; decoders (GPT) apply a causal mask — token i may only attend to positions ≤ i — implemented by setting the upper triangle of the score matrix to −∞ before softmax. The padding mask you toggled above is the same trick applied to a different pattern.

Complexity. Attention is O(n²·d) in sequence length — the reason long-context models need FlashAttention (tiling that never materialises the full n×n matrix in slow memory) and activation checkpointing during training.

• Vaswani et al. 2017 — Attention Is All You Need
• Jay Alammar — The Illustrated Transformer
• Lilian Weng — Attention? Attention!
• Andrej Karpathy — Let's build GPT

What attention scores measure. Every word sends out a question (its Query vector) asking the rest of the sentence for relevant context, and simultaneously publishes an advert (its Key vector) describing what it has to offer. An attention score is the dot product of one word's question with another word's advert — a high score means the two are a good match, so the first word should attend to the second.

Why divide by √dₖ. With many dimensions (large dₖ) each dot product sums many products, so its magnitude grows with dₖ. Very large scores push softmax into saturation — one weight becomes ≈ 1 and the rest ≈ 0 — which slows learning. Dividing by √dₖ scales the scores back to a manageable range and keeps the distribution spread across positions.

How softmax turns scores into weights. Softmax exponentiates every score (making them positive) and divides each by the total, so they sum to exactly 1 — a probability distribution. Each token's context vector is then a weighted average of all Value vectors: a token that assigns 0.7 to the subject takes 70 % of its new representation from the subject's value vector.