Loss Functions Mathematical Deep Dive

Mathematical derivation of cross-entropy from MLE, MSE from the Gaussian noise assumption, the softmax+CE combined gradient, and label smoothing. Step through the four derivations one at a time (keep a pencil at hand), then work the pencil-and-paper exercises, NumPy programming tasks, and 5-question self-test below.

Prerequisite: Loss Functions and the Learning Objective. This page assumes familiarity with logarithms, probability distributions, and basic calculus (chain rule). All derivations are self-contained.

Part 2 — Exercises

Work these on paper or in the scratch console at the bottom of the page. Graded numeric practice on these formulas lives on the module 03 practice rung — here each exercise carries its worked solution behind a reveal, so check yourself as you go.

Maths exercises (pencil and paper)

M1 — Binary cross-entropy from MLE. Consider binary classification ($K = 2$, $y \in \{0, 1\}$). The model outputs a scalar $\hat{y} = p(y = 1 \mid x)$.

1. Write the likelihood of observing label $y$ given $\hat{y}$ as a single expression using $\hat{y}$ and $(1 - \hat{y})$ without an if-else.
2. Take the negative log-likelihood for one example to obtain $\mathrm{BCE}(\hat{y}, y)$.
3. Verify: for $y = 1$ and $\hat{y} \to 1$, BCE $\to 0$; for $y = 1$ and $\hat{y} \to 0$, BCE $\to \infty$.
4. Show that BCE is a special case of CE with $K = 2$ classes.

M2 — Softmax + CE gradient for K = 3. Let $z = [2.0, 1.0, 0.0]$ and the correct class be $k^* = 0$.

1. Compute $\mathrm{softmax}(z) = (p_0, p_1, p_2)$. (Hint: subtract max for numerical stability, then normalise.)
2. Compute the CE loss $L = -\log p_0$.
3. Use the formula $\partial L / \partial z = p - y$ to write down the gradient vector.
4. Verify: the gradient at position 0 is $p_0 - 1$, and at positions 1, 2 it is $p_j$.

M3 — Label smoothing: compute soft targets. Given $K = 4$ classes, correct class $k^* = 2$, smoothing factor $\varepsilon = 0.1$:

1. Write down the hard one-hot target $y$.
2. Compute the label-smoothed target $y^{\mathrm{LS}}$ using $y^{\mathrm{LS}}_j = (1-\varepsilon)\cdot y_j + \varepsilon/K$.
3. Verify that $y^{\mathrm{LS}}$ sums to 1.
4. What is the optimal logit gap $\log\big((K-1)(1-\varepsilon)/\varepsilon\big)$ for these values? (Give a numerical answer.)

Programming exercises (NumPy)

P1 — Implement cross-entropy from logits. Fill in the stub to compute CE loss directly from logits (numerically stable):

import numpy as np

def cross_entropy_from_logits(z: np.ndarray, k_star: int) -> float:
    """Cross-entropy loss for one example."""
    # STUB: compute log-softmax of z (numerically stable), then return −log_p[k_star]
    log_p = ___
    return float(-log_p[k_star])

z = np.array([2.0, 1.0, 0.0])
print(cross_entropy_from_logits(z, 0))  # should be ≈ 0.4076

log_p = z - np.log(np.sum(np.exp(z - z.max()))) - z.max()
# or more compactly:
log_p = z - (z.max() + np.log(np.sum(np.exp(z - z.max()))))

P2 — Verify the gradient formula ∂L/∂z = ŷ − y numerically. Fill in the stub to verify the analytical gradient against a numerical approximation:

import numpy as np

def ce_loss(z: np.ndarray, k_star: int) -> float:
    z_s = z - z.max()
    log_p = z_s - np.log(np.sum(np.exp(z_s)))
    return float(-log_p[k_star])

z = np.array([2.0, 1.0, 0.0])
k_star = 0
h = 1e-5

# Numerical gradient (finite differences)
num_grad = np.array([
    (ce_loss(z + h * np.eye(len(z))[i], k_star) -
     ce_loss(z - h * np.eye(len(z))[i], k_star)) / (2 * h)
    for i in range(len(z))
])

# STUB: compute the analytical gradient ∂L/∂z = ŷ − y
analytic_grad = ___

print('Numerical:', num_grad.round(6))
print('Analytic: ', analytic_grad.round(6))
print('Match:', np.allclose(num_grad, analytic_grad, atol=1e-5))

z_s = z - z.max()
y_hat = np.exp(z_s) / np.sum(np.exp(z_s))  # softmax probabilities
y = np.zeros(len(z))
y[k_star] = 1.0
analytic_grad = y_hat - y

P3 — Label smoothing: compare losses and gradients. Fill in the stub to implement label smoothing and compare its CE loss with the hard-target CE:

import numpy as np

def smooth_labels(y: np.ndarray, eps: float) -> np.ndarray:
    """Apply label smoothing to a one-hot vector."""
    K = len(y)
    # STUB: return (1-eps)*y + eps/K
    return ___

K, k_star, eps = 4, 0, 0.1
y_hard = np.zeros(K); y_hard[k_star] = 1.0
y_soft = smooth_labels(y_hard, eps)

# Suppose the model is very confident: logits favour class k_star
z = np.array([5.0, 1.0, 1.0, 1.0])
z_s = z - z.max()
log_p = z_s - np.log(np.sum(np.exp(z_s)))
p = np.exp(log_p)

loss_hard = float(-log_p[k_star])
loss_soft = float(-y_soft @ log_p)  # − Σⱼ yⱼ^LS · log pⱼ

print(f'Hard CE: {loss_hard:.4f}')
print(f'Smoothed CE: {loss_soft:.4f}')
print(f'Softmax probs: {p.round(4)}')

return (1 - eps) * y + eps / K

Part 3 — Self-Test

Write your answer before revealing.

Cross-references: Loss Functions and the Learning Objective (base module) · Backpropagation Intuition — where the gradient flows backward from the CE loss — arrives with the dl-architectures track (coming soon).