Encoder Block
P10.transformer-architecture.01 · Audience: guest, it-ml, language-pro · Prerequisites: Multi-Head Attention, Positional Encoding
MHA → Add & Norm → FFN → Add & Norm — one complete Transformer encoder block traced on the demo corpus, every intermediate tensor visible.
🗣️ From a linguist's perspective: an enrichment cycle
ⓘ Concept: The Transformer encoder block
Why it matters — This exact unit, stacked 12–96 times, IS the body of BERT and GPT — everything else is embellishment.
Reference: Vaswani et al. 2017
Computing…
🎓 Interview deep-dive
Pre-LN vs Post-LN. This page shows Post-LN — LN(x + Sublayer(x)) — the original arrangement, which needs learning-rate warmup for stability. Modern models mostly use Pre-LN — x + Sublayer(LN(x)) — which trains stably without warmup because the residual path stays an identity. Why LayerNorm, not BatchNorm? LayerNorm normalises per token row, so it is independent of batch composition and sequence length; RMSNorm (LLaMA) drops the mean subtraction for speed.
📚 Go further
- Vaswani et al. 2017 — Attention Is All You Need
- Jay Alammar — The Illustrated Transformer
- Harvard NLP — The Annotated Transformer
- 3Blue1Brown — Attention in transformers, visually
- Andrej Karpathy — Let's build GPT
Check your understanding
How many sub-layers does an encoder block contain?
What is the shape of X for 3 tokens with d_model = 8?
Which activation sits between the FFN's two linear layers?
What gradient problem do residual connections solve?
If output − input is near zero everywhere, what did the block do?
Why does Post-LN apply LayerNorm after the residual addition?
Ask the mentor about this module
Ask a question about this content. The mentor explains and grounds its answer in what you are studying; asking is recorded as a learning signal, not a grade.
🎓 Practice ladder
1 graded rung · ~12 minImplement the block's core mechanic yourself. The rung is a three-panel workspace: instructions, a code editor, and output + test results. Run checks the visible tests; Submit grades against hidden edge cases — the residual is added before layer-norm, the output has unit variance, and a constant vector does not blow up (the ε keeps the division stable).
Rung 1 — Trace one token through the block
Loading exercise…
Capstone — Pipeline synthesis
A condensed walkthrough of the full pipeline for one sentence, from raw text to a single sentence embedding. Try to name each stage yourself, then reveal the summary.
🎯 The six stages, end to end
- Raw text — the input sentence, before any processing.
- Tokens — preprocessing (lowercase, strip punctuation) and word tokenisation split the sentence into vocabulary tokens.
- Embeddings + PE — each token ID is looked up in the embedding table, then a sinusoidal positional-encoding vector is added to inject sequence order.
- Attention — multi-head self-attention lets every token attend to every
other, producing a new
(seq_len × d_model)matrix. - Encoder output — the block wraps attention in residual + LayerNorm, applies a position-wise FFN, then a second residual + LayerNorm. The output shape equals the input — same tokens, now fully contextualised.
- Sentence embedding — mean pooling averages the token rows into one
d_model-dimensional sentence vector.
You have traced a sentence from raw text to a fixed-width embedding — the same path every Transformer encoder walks.
Try it yourself
A scratch console for this page's ideas — ungraded, nothing you run here is recorded.
Scratch console
A scratch console with the scientific stack (pandas, numpy, scikit-learn). Runs on the server — no network, resource-limited and measured.
Output appears here.
Where next?
Later in Transformer Architecture
Go up a level