Sequence Models

P6.dl-architectures.08 · Audience: guest, language-pro, it-ml · Prerequisites: CNNs for Text

CNNs for Text Training Dynamics & Regularisation Clinic

Recurrent Neural Networks read text one word at a time, carrying a hidden state — a running memory — through the sequence. This module traces that hidden-state update numerically, shows why gradients vanish over long sequences, explains LSTM gates in plain language, and arrives at the insight that motivated the Transformer: full pairwise attention.

Step 1 / 5 — Reading One Word at a Time

The core idea: memory across time steps. A CNN sees only a short window. What if the network could remember everything it has read so far?

That is the Recurrent Neural Network (RNN) idea:

At each time step t, the network reads the current word xₜ
It also takes in a hidden state hₜ₋₁ — a summary of everything read so far
It produces a new hidden state hₜ that will be passed to the next step

  h₀  →  [RNN cell]  →  h₁  →  [RNN cell]  →  h₂  →  [RNN cell]  →  h₃
              ↑                       ↑                       ↑
             x₁                     x₂                     x₃
           (The)                   (king)                  (is)

The hidden state is the RNN's running memory. After reading the whole sentence, h_T is a compressed summary of the entire input.

🗣️ From a linguist's perspective: The RNN hidden state as a running summary memo

An interpreter working in consecutive mode takes brief notes while the speaker talks, then delivers a complete translation from those notes. At each point during the speech, the notes summarise everything heard so far — but as the speech grows longer, the notes become more compressed and earlier details get overwritten. The RNN's hidden state works the same way: it is a fixed-size vector (the 'notes') updated at each word. It can capture recent context well but tends to lose information from early in a long sentence, just as a page of notes cannot perfectly preserve an hour-long speech.

ⓘ Concept: Recurrent Neural Network (RNN)

xₜ — current input (word embedding at position t)
hₜ₋₁ — hidden state from the previous step
Wₓ, Wₕ — learnable weight matrices
b — bias vector
tanh — squashes the output into (−1, +1)

The hidden state hₜ is a fixed-size vector. Increasing the sequence length does not increase its size.

h_t = \tanh(W_x x_t + W_h h_{t-1} + b)

Why it matters — RNNs were the dominant architecture for language tasks from the mid-2010s (language models, machine translation with seq2seq) until Transformers took over from 2017. Understanding RNNs is the fastest path to understanding why the Transformer's design choices (no recurrence, full attention) were revolutionary.

Q1 — RNN processing order: in an RNN, input tokens are processed in what order?

✅ Key takeaways

RNNs process text word-by-word, carrying a hidden state that summarises everything seen so far.
The vanishing gradient makes it hard for plain RNNs to learn dependencies more than ~20 steps apart.
LSTMs add a gated cell state that provides a gradient highway through time, extending useful range to ~100–200 steps.
Both RNNs and LSTMs are sequential — steps cannot be parallelised — making training on long texts slow.
The Transformer replaces recurrence with full pairwise attention: every token compares directly to every other in one parallelisable matrix operation.

Next: The track closes with its capstone: the training-dynamics clinic (module .10) — real PyTorch training rungs in the deep-learning kernel (slot .09 remains reserved). And the story continues in P10.attention.01 — What Is Attention? (/learn/p10/attention/01-what-is-attention), where the full pairwise attention mechanism motivated here is built step by step.

🎓 Interview deep-dive — choosing an architecture & why decoder-only won (for it-ml readers)

Interview-grade notes on the architecture families — written for the it-ml track, open to anyone curious.

Encoder-only / decoder-only / encoder-decoder — when to use each:

Family	Attention	Best for	Examples
Encoder-only	bidirectional	understanding: classification, NER, extractive QA, retrieval embeddings	BERT, RoBERTa
Decoder-only	causal	open-ended generation; via in-context learning, almost any task	GPT, LLaMA
Encoder-decoder	bidir. encoder + causal decoder + cross-attention	seq2seq where input and output are distinct sequences: translation, summarization	original Transformer, T5, BART

Rule of thumb: bidirectional understanding of a fixed input → encoder-only; generate text → decoder-only; map one sequence to a different sequence → encoder-decoder.

Why decoder-only came to dominate LLMs (despite strong T5-style encoder-decoders):

Loss density / sample efficiency: next-token prediction supervises every position, so every token is a training signal.
Simplicity & scale: one homogeneous stack (no separate encoder, no cross-attention) is easier to scale, shard, and optimize.
Generality via in-context learning: at scale a decoder-only model absorbs seq2seq tasks by putting the input in the prompt and generating the output — no task-specific encoder needed.
Efficient inference: the KV cache makes autoregressive generation cheap (reuse past keys/values).
Unified pretrain = inference objective, so there is no train/serve mismatch.

Encoder-decoders still win on some fixed seq2seq benchmarks, but decoder-only won on generality + scaling.

📚 Go Further

Curated resources to explore sequence models and the road to Transformers further.

Type	Resource
Blog	Andrej Karpathy — The unreasonable effectiveness of Recurrent Neural Networks (karpathy.github.io, blog + code)
Video	StatQuest with Josh Starmer — Recurrent Neural Networks (RNN), Clearly Explained! and Long Short-Term Memory (LSTM), Clearly Explained! (YouTube)
Course	Coursera — Deep Learning Specialization, Course 5: Sequence Models, Andrew Ng (free audit)
Reference	Christopher Olah — Understanding LSTM Networks (colah.github.io, seminal blog post)

Ask the mentor about this module

Ask a question about this content. The mentor explains and grounds its answer in what you are studying; asking is recorded as a learning signal, not a grade.

Ctrl/Cmd + Enter to send

Try it yourself

A scratch console for this page's ideas — ungraded, nothing you run here is recorded.

Scratch console

A scratch console with the scientific stack (pandas, numpy, scikit-learn). Runs on the server — no network, resource-limited and measured.

Output appears here.

Where next?

Continue

Training Dynamics & Regularisation Clinic →

Deep-Learning Architectures · P6

Later in Deep-Learning Architectures

Training Dynamics & Regularisation Clinic

Go up a level