Training Loop and Optimisers

P6.dl-architectures.03 · Audience: guest, language-pro, it-ml · Prerequisites: Backpropagation Intuition

Backpropagation Deep Dive Optimisers Mathematical Deep Dive

A trained network is the product of millions of tiny weight updates. This module walks through the machinery that produces them: epochs and mini-batches organise the data, SGD turns each gradient into a weight update, Adam adapts the step size per parameter, and the loss curve shows whether the whole process is working — including the tell-tale signature of overfitting.

Step 1 / 4 — Epochs and Batches

The training loop in plain language. Training a neural network means repeatedly:

Show the model some examples (a batch of data)
Compute the loss (how wrong is the prediction?)
Backpropagate (compute gradients — who is to blame?)
Update weights (nudge them in the direction that reduces loss)
Repeat until the loss is acceptably small

One complete pass through all the training data is called an epoch. Within each epoch, data is processed in small groups called batches — usually 32, 64, or 256 examples at a time.

🗣️ From a linguist's perspective: Epochs and batches as multiple passes through a training corpus

Corpus linguists build frequency lists by scanning the entire corpus once. Machine learning goes further: the model scans the training data many times (epochs), gradually refining its parameters with each pass. Within each epoch, the data is processed in small groups (batches) rather than all at once — analogous to a language learner who studies one chapter at a time and reviews multiple times rather than trying to memorise the entire textbook in one sitting. The key insight: repeated exposure to the same examples, each time adjusting weights slightly, is how the model learns stable, generalisable patterns.

ⓘ Concept: Epochs and Mini-Batches

Epoch — one full pass through the training dataset.

Mini-batch — a random subset of training examples processed together.

Why mini-batches instead of one example at a time?

Matrix operations on batches are faster on GPUs (parallelism)
Noisy gradient estimates from small batches can help escape local minima
Too large a batch (full dataset) is slow and can converge to sharp minima

Typical schedule: 100–300 epochs × 1,000–50,000 batches/epoch = millions of weight updates.

Why it matters — BERT-base was trained for 1 million steps on batches of 256 sequences. GPT-3 was trained on 300 billion tokens. The training loop ran continuously for weeks on thousands of GPUs.

Concept	Typical size	One iteration covers
Dataset	10,000 – 1B examples	All examples (once per epoch)
Mini-batch	32 – 256 examples	One batch
Step	1 gradient update	One batch forward+backward+update
Epoch	All batches	Full dataset once

How epochs, batches, and steps nest:

Level	Contains	Triggers
Training run	all epochs	starts learning
Epoch	all mini-batches (= dataset size ÷ batch size)	one full data pass
Mini-batch	B examples (e.g. B = 32)	1 forward pass + 1 backward pass + 1 weight update
Example	1 input → label pair	contributes to the batch gradient

Example: 10,000 training sentences, batch size 32 → 313 steps per epoch. At 100 epochs → 31,300 weight updates in total.

✅ Key takeaways

An epoch is one full pass through the training data; a mini-batch is the random subset processed in a single gradient update step.
SGD updates every weight by subtracting a fixed fraction (learning rate) of its gradient — simple but sensitive to the choice of learning rate.
Adam maintains per-parameter running averages of the gradient (direction) and squared gradient (scale) to adapt the effective learning rate for each weight automatically.
The training loss curve tracks model error over epochs; a rising validation loss while training loss falls is the diagnostic signature of overfitting.

Next: P6.dl-architectures.04 — Optimisers Mathematical Deep Dive (IT/ML track): the SGD derivation, momentum as an exponential moving average, the Adam bias-correction proof, and learning-rate schedules. Hands-on PyTorch training rungs live in the track capstone, the training-dynamics clinic (module .10).

Check your understanding

Q1 — If the dataset has 100 samples, batch size = 10, and we train for 3 epochs, how many gradient update steps are there in total?

Q2 — Given w = 1.0, ∂L/∂w = 0.4, η = 0.1, what is w after one SGD step? (round to 2 d.p. — recall w_new = w − η × ∂L/∂w)

Q3 — In Adam, what quantity does the second moment (v) track that vanilla SGD ignores?

Q4 — If the learning rate is too large, what risk does the optimiser face?

Module review — what is the main advantage of Adam over SGD with a fixed learning rate?

📚 Go Further

Curated resources to explore training loops and optimisers further.

Type	Resource
Video	3Blue1Brown — Gradient descent, how neural networks learn (Deep Learning, Chapter 2, YouTube)
Video	StatQuest with Josh Starmer — Stochastic Gradient Descent, Clearly Explained! (YouTube)
Course	Coursera — Deep Learning Specialization, Course 2 Week 2: Optimisation Algorithms, Andrew Ng (free audit)
Reference	Sebastian Ruder — An overview of gradient descent optimisation algorithms (ruder.io/optimizing-gradient-descent)
Course	fast.ai — Practical Deep Learning for Coders Lesson 4–5: training loop and optimisers (free)

Ask the mentor about this module

Ask a question about this content. The mentor explains and grounds its answer in what you are studying; asking is recorded as a learning signal, not a grade.

Ctrl/Cmd + Enter to send

Try it yourself

A scratch console for this page's ideas — ungraded, nothing you run here is recorded.

Scratch console

A scratch console with the scientific stack (pandas, numpy, scikit-learn). Runs on the server — no network, resource-limited and measured.

Output appears here.

Where next?

Continue

Optimisers Mathematical Deep Dive →

Deep-Learning Architectures · P6

Later in Deep-Learning Architectures

This module unlocks

Regularisation and Normalisation

Go up a level