Unsupervised Learning

a single gaussian is a committed statement: one bump, symmetric, thin tails. real data is usually several stories overlaid — different regimes, different subpopulations — and a gaussian mixture says so explicitly: each point was generated by one of \(k\) gaussians, we just don’t get told which. 𐃏 fitting one is the canonical latent-variable problem, and the algorithm that fits it — expectation-maximisation — is one of the great workhorses of statistics.

k-means is unsupervised learning’s hello world: pick \(k\) prototype points, assign every datum to its nearest prototype, move each prototype to the centre of its flock, repeat. 𐃏 it is fast, it always terminates, and it is wrong in ways that are so instructive that every clustering course starts here anyway.

pca is the linear algebra exam question that escaped into industry. given a cloud of points in \(\mathbb{R}^d\), it finds the orthogonal directions along which the cloud spreads the most, and lets you throw away the rest. 𐃏 two apparently different questions — “which directions carry the most variance?” and “which subspace loses the least when i project onto it?” — turn out to have the same answer, and that answer is an eigendecomposition.