Ensembles & Unsupervised Learning

Interview-depth machine learning for the ml-engineer track: ensemble methods (bagging, boosting, random forests, gradient boosting), clustering (k-means, hierarchical, DBSCAN), and the curse of dimensionality with PCA — assumptions, trade-offs, and the 'why it matters' framing.

1 — Bagging vs boosting; random forests vs gradient boosting (s26)

Bagging (bootstrap aggregating) trains many high-variance learners in parallel, each on a different bootstrap sample (sample-with-replacement) of the data, then averages their predictions (or majority-votes for classification):

Because the base learners are (near) unbiased but noisy, averaging reduces variance without inflating bias. Random forests are bagging on decision trees plus per-split feature subsampling — at each split only a random subset of features is considered — which decorrelates the trees so their averaged variance drops further.

Boosting instead fits weak learners sequentially, each one correcting the previous ensemble's errors:

• AdaBoost reweights the training points so misclassified examples get more attention in the next round.
• Gradient boosting frames the ensemble as stagewise additive modelling and fits each new tree to the negative gradient of the loss (the residuals for squared error) — literally gradient descent in function space.

Boosting reduces bias and can drive training error very low, so it can overfit if left unregularised. Random forests are parallel, robust, and low-maintenance; gradient boosting machines (XGBoost / LightGBM) are usually more accurate on tabular data but need careful tuning of the learning rate, tree depth, and early stopping.

Why it matters. This is the classic bias–variance lever: reach for bagging / random forests to kill variance, reach for boosting to kill bias — and know that GBMs win most tabular Kaggle competitions precisely because staged bias reduction plus shrinkage is so effective.

2 — Clustering: k-means, hierarchical, DBSCAN; choosing k (s56)

• k-means minimises the within-cluster sum of squares via Lloyd's algorithm (alternate: assign each point to its nearest centroid, then recompute centroids):

  $$\min_{\{C_k\}} \sum_{k}\sum_{x \in C_k} \lVert x - \mu_k \rVert^2.$$

  You must specify k up front; it assumes roughly spherical, equal-size clusters, is sensitive to initialisation (use k-means++ seeding), and is scale-dependent so features must be standardised.

• Hierarchical (agglomerative) clustering builds a dendrogram by repeatedly merging the closest clusters under a linkage rule (single, complete, average, Ward). No k is needed upfront — you cut the tree at the height that gives the clusters you want.

• DBSCAN is density-based: with parameters eps (neighbourhood radius) and minPts, it grows clusters from dense cores, finds arbitrarily shaped clusters, needs no k, and explicitly labels low-density points as outliers/noise. Its weakness is varying density — a single global eps struggles when clusters have very different densities.

Choosing k: the elbow method (plot inertia / within-cluster SS vs k and look for the bend), the silhouette score (how well each point sits in its cluster vs the next-nearest, in [−1, 1] — higher is better), and the gap statistic (compare inertia against a null reference distribution).

Why it matters. Picking the wrong algorithm silently distorts results: k-means will happily split one banana-shaped cluster in two, while DBSCAN handles it but demands density tuning — and without a principled k (silhouette / elbow) the whole unsupervised story is unfalsifiable.

3 — The curse of dimensionality and what PCA does (s30)

In high dimensions data becomes sparse, distances concentrate (the nearest and farthest neighbours become almost equidistant, so 'nearest' loses meaning), and volume grows exponentially with the number of dimensions — so the sample size needed to cover the space blows up. This curse of dimensionality degrades neighbourhood- and distance-based methods (kNN, k-means, kernel density) and inflates overfitting risk because there are far more ways to fit noise.

PCA (principal component analysis) fights back by projecting the data onto its top principal components — the orthogonal directions of maximum variance, which are the leading eigenvectors of the covariance matrix (equivalently the top singular vectors of the centred data; the P2 Mathematics pillar develops the eigen-decomposition and SVD behind this):

Keeping the first d components retains most of the variance (the explained-variance ratio is the kept eigenvalues' share of the total, Σᵢ≤d λᵢ / Σᵢ λᵢ) while collapsing the rest, which reduces dimensionality, denoises, and speeds up downstream models. The costs: components are linear combinations of features so they lose interpretability, PCA assumes linear structure and that variance = signal, and it requires standardised inputs.

Why it matters. Almost every high-dimensional pipeline (embeddings, images, genomics) leans on dimensionality reduction to stay tractable and generalisable — PCA is the linear baseline that non-linear methods (t-SNE, UMAP, autoencoders) are measured against.

Interview one-liners

• Bagging reduces variance, boosting reduces bias: bagging averages parallel high-variance learners; boosting fits weak learners sequentially on the previous errors.
• Random forest = bagged trees + feature subsampling (decorrelation); gradient boosting fits each tree to the negative gradient / residuals and can overfit without shrinkage.
• k-means needs k and assumes spherical clusters; DBSCAN needs none and finds outliers but struggles with varying density; hierarchical gives a dendrogram you cut.
• Choose k with the elbow, silhouette, or gap statistic — never eyeball it.
• The curse of dimensionality makes distances meaningless; PCA projects onto the top-variance eigenvectors of the covariance matrix to reduce dimensions while keeping most variance.

Rigorous references for ensembles and unsupervised learning.