Account
Guided pathThis is part of Understand Transformers & BERTBack to the path

Encoder Block

P10.transformer-architecture.01 · Audience: guest, it-ml, language-pro · Prerequisites: Multi-Head Attention, Positional Encoding

Customise the corpus these labs use
Real LLM grading for this pageLLM grading (this page):

Attention lets every token adjust its meaning by looking at its neighbours, but on its own that mixing is raw and unstable: repeat it a few dozen times and the original signal drifts away while gradients vanish. The Transformer's remedy is a disciplined wrapper — keep the original alongside every change through a residual connection, and re-normalise after every addition. Wrapped this way, attention plus a small feed-forward network become the encoder block, the self-contained unit that stacks into BERT and GPT. In this module you assemble that full block on the active corpus and inspect the tensor each sub-layer produces — the attention output, the residual sums, the norms, the feed-forward — with Q, K, V and the attention weights themselves on the attention modules.

Step 1 / 7 — Encoder Block Overview
🗣️ From a linguist's perspective: an enrichment cycle
Construction grammar sees understanding as cycles of enrichment: each pass adds a layer of analysis without erasing what came before. The encoder block is one such cycle — attend, keep the original, normalise; transform, keep again, normalise.
ⓘ Concept: The Transformer encoder block
Two sub-layers — multi-head attention, then a position-wise feed-forward network — each wrapped in a residual connection and layer normalisation. Input and output shapes are identical, which is what lets blocks stack. BERT stacks this unit as it stands; GPT adds one rule — a causal mask that hides the positions ahead — and that single line, not a different block, is the whole difference between the two families. One step precedes the first block and is easy to miss: the position-aware sum is LayerNorm-normalised row by row before any attention reads it (BERT's embedding LayerNorm). The overview below shows both matrices — the sum, and the normalised input the block consumes.
norm_1=LN(x+MHA(x))output=LN(norm_1+FFN(norm_1))\text{norm\_1} = \text{LN}(x + \text{MHA}(x))\qquad\text{output} = \text{LN}(\text{norm\_1} + \text{FFN}(\text{norm\_1}))

Why it matters — This exact unit, stacked 12–96 times, IS the body of BERT and GPT — everything else is embellishment.

Reference: Vaswani et al. 2017

Computing…

🎓 Interview deep-dive

Pre-LN vs Post-LN. This page shows Post-LN — LN(x + Sublayer(x)) — the original arrangement, which needs learning-rate warmup for stability. Modern models mostly use Pre-LN — x + Sublayer(LN(x)) — which trains stably without warmup because the residual path stays an identity. Why LayerNorm, not BatchNorm? LayerNorm normalises per token row, so it is independent of batch composition and sequence length; RMSNorm (LLaMA) drops the mean subtraction for speed.

📚 Go further

Check your understanding

Self-check

How many sub-layers does an encoder block contain?

Self-check

What is the shape of X for 3 tokens with d_model = 8?

Self-check

Which activation sits between the FFN's two linear layers?

Self-check

What gradient problem do residual connections solve?

Self-check

If output − input is near zero everywhere, what did the block do?

Self-check

Why does Post-LN apply LayerNorm after the residual addition?

Open in Labs: A gradient down twelve layers

Ask the mentor about this module

Ask a question about this content. The mentor explains and grounds its answer in what you are studying; asking is recorded as a learning signal, not a grade.

Images, PDF or text. Kept on this device only.
Keeping your files on this device

Off by default. The mentor always gets your file; this only decides whether your own copy stays here. Copies live in this browser only - they do not follow you to another device, and clearing site data removes them.

Ctrl/Cmd + Enter to send

🎓 Practice ladder

1 graded rung · ~12 min

Implement the block's core mechanic yourself. The rung is a three-panel workspace: instructions, a code editor, and output + test results. Run checks the visible tests; Submit grades against hidden edge cases — the residual is added before layer-norm, the output has unit variance, and a constant vector does not blow up (the ε keeps the division stable).

Rung 1 — Trace one token through the block

Loading exercise…

Capstone — Pipeline synthesis

A condensed walkthrough of the full pipeline for one sentence, from raw text to a single sentence embedding. Try to name each stage yourself, then reveal the summary.

🎯 The six stages, end to end
  1. Raw text — the input sentence, before any processing.
  2. Tokens — preprocessing (lowercase, strip punctuation) and word tokenisation split the sentence into vocabulary tokens.
  3. Embeddings + PE — each token ID is looked up in the embedding table, then a sinusoidal positional-encoding vector is added to inject sequence order.
  4. Attention — multi-head self-attention lets every token attend to every other, producing a new (seq_len × d_model) matrix.
  5. Encoder output — the block wraps attention in residual + LayerNorm, applies a position-wise FFN, then a second residual + LayerNorm. The output shape equals the input — same tokens, now fully contextualised.
  6. Sentence embedding — mean pooling averages the token rows into one d_model-dimensional sentence vector.

You have traced a sentence from raw text to a fixed-width embedding — the same path every Transformer encoder walks.

Try it yourself

A scratch console for this page's ideas — ungraded, nothing you run here is recorded.

Scratch console

A scratch console with the scientific stack (pandas, numpy, scikit-learn). Runs on the server — no network, resource-limited and measured.

Output appears here.

My notes on this module

Loading your notes...

Where next?

Continue

Sentence Embeddings →

Transformer Architecture · P10

Later in Transformer Architecture

Go up a level

Encoder Block — TransformerLab