Clustering

a single gaussian is a committed statement: one bump, symmetric, thin tails. real data is usually several stories overlaid — different regimes, different subpopulations — and a gaussian mixture says so explicitly: each point was generated by one of \(k\) gaussians, we just don’t get told which. 𐃏 fitting one is the canonical latent-variable problem, and the algorithm that fits it — expectation-maximisation — is one of the great workhorses of statistics.

Network intrusion detection at competition scale: nearly five million TCP connection records, each labelled normal or with one of dozens of attack names. Historically the most-used intrusion-detection benchmark ever — and, by broad consensus, one that should now be used only with its flaws stated up front.

Provenance#

The dataset was built for the Third International Knowledge Discovery and Data Mining Tools Competition (1999) (held with KDD-99, the fifth KDD conference) by processing the tcpdump portions of the 1998 DARPA Intrusion Detection System Evaluation, run by MIT Lincoln Laboratory. Traffic was generated on a closed simulated air-force network with hand-injected attacks over seven weeks (training) plus two weeks (test), then summarised into per-connection feature vectors.

k-means is unsupervised learning’s hello world: pick \(k\) prototype points, assign every datum to its nearest prototype, move each prototype to the centre of its flock, repeat. 𐃏 it is fast, it always terminates, and it is wrong in ways that are so instructive that every clustering course starts here anyway.