Transformer Architecture

You arrive here holding the two hard-won pieces: vectors that know what their word means and where it stands, and a mechanism that lets those vectors talk to each other. This track is where the pieces stop being separate demos and click into the unit that everything else is built from — the encoder block in every Transformer diagram, reassembled here part by moving part.

Start with a puzzle. If attention is the breakthrough, why does every block follow it with a second sublayer — a feed-forward network that looks at each token entirely on its own, no neighbours consulted? The answer reframes what attention actually does: it moves information between positions, but it does remarkably little with it. Gathering without digesting. The feed-forward sublayer is where each token processes what attention just delivered — and a block needs both, in that order, or the whole design underperforms. Attention alone, it turns out, is not all you need.

The block's remaining parts look like plumbing and turn out to be load-bearing: residual connections that let information skip past each sublayer untouched, and layer normalisation that keeps the numbers tame. They are the reason these blocks can stack — two of them or ninety-six — without the signal drowning on the way up. The second module then asks the output question: after the stack has run, how do a dozen token vectors become the one vector that stands for the whole sentence — and which of the three ways of pooling them actually works best on your data?

The assembly runs on the corpus below — change a sentence and follow it through every stage of the block.