Regularisation Mathematical Deep Dive

Mathematical derivation of the bias-variance decomposition, the ridge regression closed-form solution, LASSO sparsity, dropout as approximate Bayesian inference, and LayerNorm. Step through the five derivations one at a time (keep a pencil at hand), then work the pencil-and-paper exercises, NumPy programming tasks, and 5-question self-test below.

Prerequisite: Regularisation and Normalisation. This page assumes familiarity with basic probability, linear algebra, and calculus (partial derivatives, matrix inverse). All derivations are self-contained.

Part 2 — Exercises

Work these on paper or in the scratch console at the bottom of the page. Each exercise carries its worked solution behind a reveal, so check yourself as you go.

Maths exercises (pencil and paper)

M1 — Bias and variance of a constant predictor. Consider a predictor that always predicts the training mean: $\hat{f}(x) = \bar{y} = \frac{1}{n} \sum_i y_i$, independent of $x$. Assume the true function is $f(x) = 3x$ and noise $\varepsilon \sim \mathcal{N}(0, \sigma^2)$.

1. What is $\mathbb{E}[\hat{f}(x)]$ in terms of $f(x)$ values seen during training?
2. What is the Bias² of this predictor at $x = 1$ if $f(1) = 3$ and the training data has $\mathbb{E}[\bar{y}] = 2$?
3. Is the variance of this predictor zero? Explain.
4. What can you say about the bias-variance trade-off for this predictor?

M2 — Ridge regression limit as λ → ∞. Show algebraically that as $\lambda \to \infty$, the ridge regression solution $w^* \to 0$.

1. Start from $w^* = (X^\top X + \lambda I)^{-1} X^\top y$.
2. Factor out $\lambda$: $w^* = \frac{1}{\lambda}(X^\top X / \lambda + I)^{-1} X^\top y$.
3. Apply the matrix limit as $\lambda \to \infty$.
4. Interpret: what does a model with $w^* \to 0$ predict for every input?

M3 — Soft-thresholding: LASSO solution in 1D. In the 1D LASSO problem, the objective (after orthonormal $X$) is:

$\min_w \; (w - c)^2 + \lambda |w|$

where $c$ is the OLS estimate.

1. For $c = 3.0$, $\lambda = 2.0$: compute the LASSO solution $w^*$. (Soft-thresholding: $w^* = \mathrm{sign}(c) \cdot \max(|c| - \lambda/2, 0)$.)
2. For $c = 0.8$, $\lambda = 2.0$: compute $w^*$. Is it zero?
3. What is the threshold on $|c|$ below which $w^* = 0$?

Programming exercises (NumPy)

P1 — Ridge regression closed-form vs OLS. Fill in the stub to implement ridge regression and compare with OLS (λ = 0):

import numpy as np

np.random.seed(42)
n, d = 20, 5
X = np.random.randn(n, d)
w_true = np.array([3.0, -1.5, 0.0, 0.5, 2.0])
y = X @ w_true + 0.5 * np.random.randn(n)

lam = 1.0

# STUB: compute the ridge regression solution
w_ridge = ___

# OLS (lambda = 0): may be ill-conditioned if n < d
w_ols = np.linalg.lstsq(X, y, rcond=None)[0]

print('True w:  ', w_true.round(3))
print('OLS w:   ', w_ols.round(3))
print('Ridge w: ', w_ridge.round(3))
print('OLS  MSE:', ((X @ w_ols  - y)**2).mean().round(4))
print('Ridge MSE:', ((X @ w_ridge - y)**2).mean().round(4))

w_ridge = np.linalg.solve(X.T @ X + lam * np.eye(d), X.T @ y)

P2 — Compare L1 and L2 regularisation: sparsity. Fill in the stub to apply soft-thresholding (LASSO) and ridge shrinkage to a 1D array of OLS coefficients and compare the results:

import numpy as np

# Simulate OLS estimates (orthonormal X, so w_ols = XᵀY directly)
np.random.seed(7)
c = np.random.uniform(-3, 3, 10)  # 10 OLS estimates
lam = 2.0

# Ridge shrinkage: w_ridge_j = c_j / (1 + lambda)  [when XᵀX = I]
w_ridge = c / (1 + lam)

# STUB: soft-thresholding (LASSO solution when XᵀX = I)
# w_lasso_j = sign(c_j) * max(|c_j| - lam/2, 0)
w_lasso = ___

print('OLS:   ', c.round(3))
print('Ridge: ', w_ridge.round(3), f'  ({(w_ridge == 0).sum()} zeros)')
print('LASSO: ', w_lasso.round(3), f'  ({(w_lasso == 0).sum()} zeros)')

w_lasso = np.sign(c) * np.maximum(np.abs(c) - lam / 2, 0)

P3 — LayerNorm: implement and verify. Fill in the stub to implement LayerNorm and verify mean = 0, variance = 1:

import numpy as np

def layer_norm(x: np.ndarray, eps: float = 1e-5) -> np.ndarray:
    """Normalise x across feature dimension (γ=1, β=0)."""
    # STUB: compute mean and variance, return (x - mean) / sqrt(var + eps)
    return ___

np.random.seed(0)
x = np.random.randn(8) * 5 + 10  # large scale and offset
x_hat = layer_norm(x)

print('Input mean:  ', x.mean().round(4))
print('Input std:   ', x.std().round(4))
print('LN mean:     ', x_hat.mean().round(8))  # should be ~0
print('LN var:      ', x_hat.var().round(8))   # should be ~1
print('LN output:   ', x_hat.round(4))

mu = x.mean()
var = x.var()  # population variance: (1/d) Σ (xᵢ - μ)²
return (x - mu) / np.sqrt(var + eps)

Part 3 — Self-Test

Write your answer before revealing.

Cross-references: Regularisation and Normalisation (base module) · Training Loop and Optimisers — module 03 of this track (AdamW = Adam + L2 weight decay) · The Encoder Block (dropout and LayerNorm in the full architecture).