Optimisers Mathematical Deep Dive

The SGD derivation from the stochastic objective, momentum as an exponential moving average with its heavy-ball origin, Adam with the full bias-correction proof, and learning-rate schedule formulas — mathematical details for IT/ML practitioners, with pencil-and-paper exercises, NumPy programming tasks, and a 5-question self-test.

Prerequisite: Training Loop and Optimisers. This page assumes familiarity with expectations, the chain rule, and geometric series. All derivations are self-contained.

Exercises

Work these on paper or in the scratch console at the bottom of the page. Each exercise carries its worked solution behind a reveal, so check yourself as you go. (Hands-on PyTorch training rungs arrive with the VN-056 training clinic — coming soon.)

Maths exercises (pencil and paper)

M1 — One Adam step. Given $g = 0.1$, $m_0 = 0$, $v_0 = 0$, $\beta_1 = 0.9$, $\beta_2 = 0.999$, $\varepsilon = 10^{-8}$, $\alpha = 0.001$, $t = 1$: compute the Adam weight update at $t = 1$ for a single parameter $\theta = 0.5$.

Recall: $m_1 = \beta_1 \cdot 0 + (1-\beta_1) \cdot g$, $v_1 = \beta_2 \cdot 0 + (1-\beta_2) \cdot g^2$, then bias-correct and update.

M2 — Step-decay LR at t = 100. Compute $\alpha(100) = \alpha_0 \cdot \gamma^{\lfloor t/s \rfloor}$ for $\alpha_0 = 0.1$, $s = 30$, $\gamma = 0.5$.

M3 — Bias-correction proof: show that $\hat{m}_1 = g_1$ when $\beta_1 = 0.9$. Starting from $m_0 = 0$, after one step: $m_1 = \beta_1 \cdot m_0 + (1-\beta_1) \cdot g_1$. Show that $\hat{m}_1 = m_1/(1-\beta_1^1) = g_1$.

Programming exercises (NumPy)

Try each exercise in your own Python environment (or the scratch console), then reveal the reference solution.

P1 — Implement one Adam step in NumPy.

import numpy as np

def adam_step(theta, g, m, v, t, alpha=0.001, beta1=0.9, beta2=0.999, eps=1e-8):
    """Perform one Adam update.

    Parameters
    ----------
    theta : np.ndarray  Current parameters.
    g     : np.ndarray  Gradient at current step.
    m     : np.ndarray  First moment (EMA of gradients).
    v     : np.ndarray  Second moment (EMA of squared gradients).
    t     : int         Current step (1-based).

    Returns
    -------
    tuple[np.ndarray, np.ndarray, np.ndarray]
        (theta_new, m_new, v_new)
    """
    # TODO: implement Adam
    pass

theta = np.array([0.5, -0.3])
g     = np.array([0.1, -0.2])
m     = np.zeros(2)
v     = np.zeros(2)
print(adam_step(theta, g, m, v, t=1))

def adam_step(theta, g, m, v, t, alpha=0.001, beta1=0.9, beta2=0.999, eps=1e-8):
    m_new = beta1 * m + (1 - beta1) * g
    v_new = beta2 * v + (1 - beta2) * g**2
    m_hat = m_new / (1 - beta1**t)
    v_hat = v_new / (1 - beta2**t)
    theta_new = theta - alpha / (np.sqrt(v_hat) + eps) * m_hat
    return theta_new, m_new, v_new

theta_new, m_new, v_new = adam_step(theta, g, m, v, t=1)
print(theta_new)   # [0.499   -0.299  ] — very small step at t=1

P2 — Implement a step-decay learning rate schedule.

import numpy as np

def step_decay_schedule(t_max, alpha0=0.1, step_size=30, gamma=0.5):
    """Return LR values for steps 0..t_max.

    Parameters
    ----------
    t_max     : int    Total number of steps.
    alpha0    : float  Initial learning rate.
    step_size : int    Number of steps between decays.
    gamma     : float  Decay factor per drop.

    Returns
    -------
    np.ndarray  LR at each step.
    """
    # TODO: compute alpha(t) = alpha0 * gamma^floor(t/step_size) for t in range(t_max+1)
    pass

lrs = step_decay_schedule(150)
print(lrs[:5], "...", lrs[30], "...", lrs[90])

def step_decay_schedule(t_max, alpha0=0.1, step_size=30, gamma=0.5):
    t = np.arange(t_max + 1)
    return alpha0 * gamma ** np.floor(t / step_size)

lrs = step_decay_schedule(150)
print(lrs[:5])         # [0.1 0.1 0.1 0.1 0.1]
print(lrs[30])         # 0.05  (first drop at t=30)
print(lrs[90])         # 0.0125  (three drops: 0.1 × 0.5³)

P3 — Implement a cosine annealing schedule.

import numpy as np

def cosine_annealing(t_max, alpha_max=0.1, alpha_min=0.0):
    """Return cosine-annealed LR values for steps 0..t_max.

    Parameters
    ----------
    t_max     : int    Total number of steps (one cycle).
    alpha_max : float  Peak learning rate.
    alpha_min : float  Minimum learning rate.

    Returns
    -------
    np.ndarray  LR at each step.
    """
    # TODO: alpha(t) = alpha_min + 0.5*(alpha_max - alpha_min)*(1 + cos(pi*t/t_max))
    pass

lrs = cosine_annealing(150)
print(lrs[0], lrs[75], lrs[150])   # should be ~alpha_max, ~midpoint, ~alpha_min

def cosine_annealing(t_max, alpha_max=0.1, alpha_min=0.0):
    t = np.arange(t_max + 1)
    return alpha_min + 0.5 * (alpha_max - alpha_min) * (1 + np.cos(np.pi * t / t_max))

lrs = cosine_annealing(150)
print(lrs[0])     # 0.1  (= alpha_max)
print(lrs[75])    # 0.05 (= midpoint)
print(lrs[150])   # 0.0  (= alpha_min)

Self-Test — 5 Questions

Write your answer before revealing.

Cross-references: Training Loop and Optimisers (base module) · Regularisation and Normalisation — dropout and LayerNorm, the two regularisation techniques built into every Transformer encoder layer. Hands-on PyTorch training rungs arrive with the VN-056 training clinic (coming soon).