Regression & GLMs

P5.ml-methods.03 · Audience: guest, it-ml, language-pro · Prerequisites: Ensembles & Unsupervised Learning

Ensembles & Unsupervised Learning Causal Inference & Experiments

Regression one idea at a time: OLS and the Gauss-Markov assumptions, generalized linear models and the link function, and logistic regression — each with a small worked calculation. Interview depth for the ml-engineer track. Back to the foundations: M5 — Probability and Statistics Basics, and Advanced Probability.

Framing. Linear regression is the identity-link Gaussian GLM; swap the link and the response distribution and you get logistic, Poisson, and the rest. Each step turns the algebra into a number.

Step 1 / 3 — OLS & Gauss-Markov (s19)

1 — OLS and the Gauss-Markov assumptions (s19). OLS minimises squared residuals; setting the gradient to zero gives the closed-form normal equations:

\hat\beta = (X^\top X)^{-1} X^\top y

Under the five Gauss-Markov assumptions — linearity, exogeneity, homoskedasticity, no autocorrelation, no perfect multicollinearity — OLS is BLUE (the best linear unbiased estimator). Normality is only needed for exact inference, not unbiasedness.

Worked example — the closed form in code. Fit a slope+intercept to three points with the normal equations:

import numpy as np
X = np.array([[1,0],[1,1],[1,2]])   # intercept col + feature
y = np.array([1.0, 2.0, 3.0])
beta = np.linalg.inv(X.T @ X) @ X.T @ y
# beta = [1., 1.]  ->  y_hat = 1 + 1*x  (a perfect fit)

The X.T @ X inverse is the whole solution. If two columns were collinear it would be singular (no inverse) — that's the no-perfect-multicollinearity assumption; ridge's (XᵀX + λI)⁻¹ restores a well-conditioned inverse. Heteroskedasticity wouldn't bias β̂ but would break the standard errors (use robust SEs / WLS).

⚡ Interview Ref — the quick-scan Reference face

Interview-depth regression for the ml-engineer track: OLS and the Gauss-Markov theorem, generalized linear models and the link function, and logistic regression — log-odds, odds ratios, and why we train with cross-entropy rather than MSE.

1 — OLS and the Gauss-Markov assumptions (s19)

Ordinary least squares (OLS) fits a linear model y = Xβ + ε by minimising the sum of squared residuals ‖y − Xβ‖². Setting the gradient to zero gives the normal equations and the closed-form solution:

\hat\beta = (X^\top X)^{-1} X^\top y

The Gauss-Markov theorem says that under five assumptions OLS is BLUE — the best linear unbiased estimator (lowest variance among all linear unbiased estimators):

Linearity — the model is linear in the parameters.
Exogeneity — errors have zero conditional mean, E[ε ∣ X] = 0.
Homoskedasticity — constant error variance, Var(εᵢ) = σ².
No autocorrelation — errors are uncorrelated across observations.
No perfect multicollinearity — XᵀX is invertible (columns are linearly independent).

(Normality of errors is not needed for BLUE; it is only needed for exact t/F inference.)

Violations and fixes:

Heteroskedasticity (non-constant variance) → robust / White standard errors or weighted least squares (WLS); coefficients stay unbiased but their SEs are wrong.
Autocorrelation (correlated errors, common in time series) → Newey-West standard errors or generalized least squares (GLS).
Endogeneity (E[ε ∣ X] ≠ 0) → instrumental variables (IV); OLS is biased and inconsistent here.
Multicollinearity → drop redundant features or regularise (ridge shrinks (XᵀX + λI)⁻¹, restoring a well-conditioned inverse).

2 — Generalized linear models and the link function (s48)

A generalized linear model (GLM) extends linear regression to response distributions in the exponential family via a link function g that relates the mean of the response to the linear predictor:

g\big(\mathbb{E}[y]\big) = X\beta

Every GLM has three components:

Random component — the distribution of y (Normal, Bernoulli, Poisson, …).
Systematic component — the linear predictor η = Xβ.
Link function — the invertible g connecting the mean to η; its inverse g⁻¹ (the mean function) maps the linear predictor back to the response scale.

Common choices:

Distribution	Link	Model
Normal	identity, g(μ) = μ	ordinary linear regression (OLS)
Bernoulli	logit, g(μ) = log(μ / (1 − μ))	logistic regression
Poisson	log, g(μ) = log μ	Poisson regression (counts)

The identity link with a Normal response recovers OLS exactly, so linear regression is the simplest GLM. GLMs are fit by maximum likelihood, solved numerically with iteratively reweighted least squares (IRLS) rather than a closed form.

3 — Logistic regression: log-odds and coefficients (s25)

Logistic regression is the GLM with a Bernoulli response and a logit link. It models the log-odds (the logit of the success probability p) as a linear function:

\log\frac{p}{1-p} = X\beta, \qquad p = \sigma(X\beta) = \frac{1}{1 + e^{-X\beta}}

The sigmoid σ is the inverse link — it squashes the unbounded linear predictor into a probability in (0, 1).

Interpreting coefficients:

A coefficient βⱼ is the change in log-odds per unit increase in xⱼ (holding the others fixed).
Exponentiating gives the odds ratio: e^βⱼ is the multiplicative factor by which the odds change per unit of xⱼ. e^βⱼ > 1 raises the odds, < 1 lowers them, = 1 means no effect.

Why cross-entropy, not MSE: logistic regression is trained by minimising cross-entropy (log-loss), which is the negative Bernoulli log-likelihood:

\mathcal{L} = -\sum_i \big[\,y_i \log p_i + (1 - y_i)\log(1 - p_i)\,\big]

This loss is convex in β, so gradient descent reaches the global optimum. MSE on the sigmoid is non-convex and produces vanishing gradients in the saturated regions (p → 0 or 1), where σ′(z) = σ(z)(1 − σ(z)) → 0 stalls learning — which is why we use log-loss instead.

Interview one-liners

OLS closed form β̂ = (XᵀX)⁻¹Xᵀy is BLUE under Gauss-Markov; normality is only needed for exact inference, not unbiasedness.
Heteroskedasticity breaks the SEs, not the coefficients — fix with robust SEs or WLS.
A GLM = distribution + linear predictor + link function; identity+Normal is OLS, logit+Bernoulli is logistic, log+Poisson is count regression.
Logistic regression is linear in the log-odds; e^βⱼ is the odds ratio per unit of xⱼ.
Train logistic with cross-entropy, not MSE — log-loss is convex; MSE on the sigmoid is non-convex with vanishing gradients.

📚 Go Further

Rigorous references for regression and GLMs.

Type	Resource
Book	Hastie, Tibshirani & Friedman, Elements of Statistical Learning — GLMs
Book	Wooldridge, Introductory Econometrics — Gauss-Markov, robust SEs, IV
Book	McCullagh & Nelder, Generalized Linear Models — the definitive GLM text
In-app	P5.ml-methods.06 — Evaluation, Selection & Interpretability, for calibration of these probability outputs
In-app	P1.statistics-advanced.01 — Advanced Probability (S7-adv), for the likelihood foundations

✅ Key takeaways

OLS minimises squared residuals; the normal equations give the closed form β̂ = (XᵀX)⁻¹Xᵀy, and under the five Gauss-Markov assumptions OLS is BLUE — normality is only needed for exact inference.
Heteroskedasticity and autocorrelation break the standard errors, not the coefficients — fix with robust / Newey-West SEs, WLS or GLS; endogeneity biases OLS itself and needs instrumental variables.
A GLM is a response distribution + a linear predictor + a link function: identity+Normal is OLS, logit+Bernoulli is logistic, log+Poisson is count regression — fit by maximum likelihood (IRLS), not a closed form.
The link is chosen to match the response's support — the log link keeps a Poisson mean positive — and the inverse link maps the linear predictor back to the response scale.
Logistic regression is linear in the log-odds: e^βⱼ is the odds ratio per unit of xⱼ, and training uses cross-entropy (convex) rather than MSE on the sigmoid (non-convex, with vanishing gradients).

Next: P5.ml-methods.04 — Causal Inference & Experiments

Ask the mentor about this module

Ask a question about this content. The mentor explains and grounds its answer in what you are studying; asking is recorded as a learning signal, not a grade.

Ctrl/Cmd + Enter to send

Try it yourself

A scratch console for this page's ideas — ungraded, nothing you run here is recorded.

Scratch console

A scratch console with the scientific stack (pandas, numpy, scikit-learn). Runs on the server — no network, resource-limited and measured.

Output appears here.

Where next?

Continue

Causal Inference & Experiments →

ML Methods · P5

Later in ML Methods

Go up a level