Kernels

kernel methods are the great arbitrage of classical machine learning: keep the algorithm linear — with all its convexity and closed forms — but run it in a feature space so large it can bend around anything, and never pay for that space explicitly. 𐃏 one identity powers everything: if your algorithm touches the data only through inner products, you may replace every \(\langle x, x’\rangle\) with a kernel \(k(x, x’)\) and thereby work in the implicit feature space of \(k\) — possibly infinite-dimensional — at the cost of an \(n \times n\) matrix.

a linearly separable dataset admits infinitely many separating hyperplanes, and the perceptron will happily hand you whichever one it trips over first. 𐃏 the support vector machine asks a better question: of all the hyperplanes that separate the data, which one is farthest from everybody? the answer — the maximum-margin hyperplane — is determined by a handful of boundary points (the support vectors), drops out of a beautiful convex dual, and generalises via the kernel trick from lines to nearly anything.