Regularisation and Normalisation
P6.dl-architectures.05 · Audience: guest, language-pro, it-ml · Prerequisites: Training Loop and Optimisers
A network with enough parameters can memorise its training set perfectly — and still fail on every sentence it has never seen. This module explains why that happens (overfitting and the bias-variance trade-off) and works through the two regularisation techniques built into the Transformer: Dropout, which randomly silences neurons during training, and Layer Normalisation, which keeps activations on a stable scale in deep networks.
Imagine a student who memorises every answer in the practice exam but cannot solve problems they haven't seen before. That student has overfitted to the practice questions.
Neural networks do the same thing. Given enough parameters (weights) and enough training, the model can memorise the training examples perfectly — but fails on new, unseen examples.
- Overfitting = low training error + high validation/test error.
- Underfitting = both training error and test error are high — the model has not learned enough (e.g. too few parameters or too little training).
🗣️ From a linguist's perspective: Overfitting as memorising examples rather than learning the rule
ⓘ Concept: Overfitting and Generalisation
A model generalises when it performs well on data it has never seen. Generalisation fails when the model captures the noise in the training data instead of the underlying pattern.
Common causes of overfitting:
- Too many parameters for the amount of training data
- Training for too many epochs without early stopping
- No regularisation
Common solutions:
- More training data
- Dropout — randomly silence neurons during training
- Weight decay (L2 regularisation)
- Early stopping — stop when validation loss starts rising
- Layer Normalisation — stabilises activations, acts as a mild regulariser
Why it matters — Transformer models use Dropout after every attention and FFN sub-layer, and LayerNorm after every residual connection. These two techniques are directly responsible for the Transformer's ability to train stably for millions of steps on large datasets.
Check your understanding
A model achieves 99 % training accuracy but only 60 % test accuracy. What problem does this describe?
A model that is too simple for the data has high bias or high variance?
During training with keep probability = 0.8 (drop probability = 0.2) and 100 input units, how many units are expected to survive each forward pass?
In the Transformer encoder block, which technique normalises activations between sublayers to reduce internal covariate shift?
Module review — given weights [0.5, −1.0, 0.2] and λ = 0.01, what is the L2 penalty term λ/2 · ‖w‖²? (round to 4 d.p. — recall ‖w‖² = w₁² + w₂² + w₃²)
📚 Go Further
Curated resources to explore overfitting and regularisation further.
| Type | Resource |
|---|---|
| Video | StatQuest with Josh Starmer — Regularization Part 1: Ridge Regression and Dropout (YouTube) |
| Course | Coursera — Deep Learning Specialization, Course 2 Week 1: Regularisation, Andrew Ng (free audit) |
| Reference | Goodfellow, Bengio & Courville, Deep Learning Ch. 7: Regularisation for Deep Learning (free at deeplearningbook.org) |
| Reference | CS231n Stanford — Neural Networks Part 2: Setting up the Data and the Loss (regularisation section) |
Ask the mentor about this module
Ask a question about this content. The mentor explains and grounds its answer in what you are studying; asking is recorded as a learning signal, not a grade.
Try it yourself
A scratch console for this page's ideas — ungraded, nothing you run here is recorded.
Scratch console
A scratch console with the scientific stack (pandas, numpy, scikit-learn). Runs on the server — no network, resource-limited and measured.
Output appears here.
Where next?
Later in Deep-Learning Architectures
This module unlocks