Loss Functions and the Learning Objective

Entropy $H(p) = -\sum_x p(x)\log p(x)$ is the average uncertainty of a distribution $p$ — equivalently, the minimum expected number of bits (with $\log_2$) needed to encode samples drawn from $p$. It is maximised by the uniform distribution (every outcome equally likely = maximal surprise) and is $0$ when $p$ puts all mass on one outcome.

Cross-entropy $H(p, q) = -\sum_x p(x)\log q(x)$ is the expected number of bits to encode data drawn from $p$ using a code optimised for $q$. This is the standard classification loss: $p$ is the one-hot label vector and $q$ is the model's predicted probability distribution, so $H(p, q) = -\log q(k^*)$ collapses to the $-\log p_{k^*}$ seen above.

KL divergence measures the extra bits incurred by coding with $q$ instead of $p$:

$D_{\mathrm{KL}}(p\,\|\,q) = \sum_x p(x)\log\frac{p(x)}{q(x)} = H(p, q) - H(p) \ge 0.$

It is $\ge 0$ (Gibbs' inequality), zero iff $p = q$, and asymmetric — $D_{\mathrm{KL}}(p\|q) \neq D_{\mathrm{KL}}(q\|p)$ in general — so it is a divergence, not a true distance.

Key ML link. Writing $\hat{p}$ for the empirical data distribution and $p_\theta$ for the model, minimising cross-entropy loss equals minimising $D_{\mathrm{KL}}(\hat{p}\,\|\,p_\theta)$, because $H(\hat{p}, p_\theta) = H(\hat{p}) + D_{\mathrm{KL}}(\hat{p}\,\|\,p_\theta)$ and the entropy $H(\hat{p})$ is constant in $\theta$. Both are therefore identical to maximum-likelihood estimation of the model parameters.

Mutual information quantifies how much knowing $Y$ reduces uncertainty about $X$:

$I(X; Y) = D_{\mathrm{KL}}\big(p(x, y)\,\|\,p(x)\,p(y)\big) = H(X) - H(X\mid Y).$

It is the KL divergence between the joint distribution and the product of the marginals, is symmetric, and is zero iff $X$ and $Y$ are independent (the joint factorises).

Curated resources to explore loss functions and the learning objective further.