Backpropagation Deep Dive
P6.dl-architectures.02 · Audience: it-ml · Prerequisites: Backpropagation Intuition
Jacobian matrices for multi-layer networks, a minimal tape-based autograd implementation, and gradient checking — mathematical derivations for IT/ML practitioners. Work through the four steps with a pencil at hand; the numeric exercises in the last step carry their worked solutions behind reveals.
Prerequisite: Backpropagation Intuition (the base module). This page assumes familiarity with matrix multiplication, the sigmoid function, and the scalar chain rule.
The base module showed the chain rule for a single scalar expression. Real networks have vector-valued layers, so we need Jacobian matrices — the generalisation of the derivative to vector-to-vector functions.
Jacobian — definition. For a function f : ℝⁿ → ℝᵐ, the Jacobian matrix J ∈ ℝᵐˣⁿ is defined as:
Each row i gives the gradient of the i-th output with respect to all inputs. Each column j gives how all outputs change when input j increases by 1. For a scalar loss L, the chain rule through a sequence of layers becomes a product of Jacobians:
where J₂ = ∂a₂/∂a₁ and J₁ = ∂a₁/∂x.
Concrete 3-layer example. Consider a 2-input, 2-hidden, 1-output network with no bias for clarity:
- Layer 1: a₁ = W₁x (linear, W₁ ∈ ℝ²ˣ²)
- Layer 2: a₂ = σ(a₁) element-wise sigmoid
- Output: L = ½‖a₂ − y‖²
with x = [1.0, 2.0], W₁ = [[0.3, −0.1], [0.2, 0.5]], and target y = [0.5, 0.5].
Forward pass:
| Variable | Value |
|---|---|
| x | [1.0, 2.0] |
| a₁ = W₁x | [0.1, 1.2] |
| a₂ = σ(a₁) | [0.525, 0.7685] |
Jacobians (note σ'(a₁) = a₂ ⊙ (1 − a₂) = [0.2494, 0.1779]):
| Jacobian | Shape | Note |
|---|---|---|
| J₁ = ∂a₁/∂x = W₁ | 2×2 | Same as weight matrix |
| J₂ = ∂a₂/∂a₁ = diag(σ'(a₁)) | 2×2 (diagonal) | Non-zero only on diagonal |
Gradient of L with respect to x:
| Step | Value |
|---|---|
| ∂L/∂a₂ | [0.025, 0.2685] |
| ∂L/∂a₁ = J₂ᵀ · ∂L/∂a₂ | [0.0062, 0.0478] |
| ∂L/∂x = J₁ᵀ · ∂L/∂a₁ | [0.0114, 0.0233] |
The chain rule is a right-to-left Jacobian product: start at the loss, transpose each Jacobian, and multiply backward through the graph.
Why transpose?
The forward pass applies J (shape m×n) to map an n-vector to an m-vector. The backward pass maps an m-vector (upstream gradient) to an n-vector (gradient w.r.t. input), so we need the transpose Jᵀ (shape n×m). This is why loss.backward() in PyTorch traverses the graph in reverse and multiplies transposed Jacobians at every node.
Where the graded practice lives: this module fulfils the "backprop from
scratch" slot of the taxonomy with Ladder B — its hands-on exercise, the
tape-based autograd you studied in Step 2, is graded as rung
P6.dl-architectures.01.ex03 ("Finish the autograd tape") on the previous
module's ladder, alongside ex01 (chain rule, checked numerically) and ex02
(backprop through f = (a + b) × c). No new rungs are embedded here: the maths
and NumPy exercises above carry their worked solutions behind reveals instead.
Cross-references: Backpropagation Intuition (base module) · Optimisers Deep Dive — SGD derivation, momentum, and the full Adam proof — arrives with this track.
Ask the mentor about this module
Ask a question about this content. The mentor explains and grounds its answer in what you are studying; asking is recorded as a learning signal, not a grade.
Try it yourself
A scratch console for this page's ideas — ungraded, nothing you run here is recorded.
Scratch console
A scratch console with the scientific stack (pandas, numpy, scikit-learn). Runs on the server — no network, resource-limited and measured.
Output appears here.
Where next?
Later in Deep-Learning Architectures