one model is an opinion; a committee is an estimator. 𐃏 ensemble methods build many imperfect predictors and combine them, and the two great families attack opposite ends of the bias-variance decomposition: bagging averages low-bias, high-variance models to cancel their wobble; boosting stacks up high-bias, low-variance weak learners to build accuracy that none of them has alone.
Supervised Learning
a linearly separable dataset admits infinitely many separating hyperplanes, and the perceptron will happily hand you whichever one it trips over first. 𐃏 the support vector machine asks a better question: of all the hyperplanes that separate the data, which one is farthest from everybody? the answer — the maximum-margin hyperplane — is determined by a handful of boundary points (the support vectors), drops out of a beautiful convex dual, and generalises via the kernel trick from lines to nearly anything.
naive bayes is the classifier you get by taking bayes’ rule seriously and probability theory not seriously at all. 𐃏 it assumes every feature is independent of every other feature given the class — an assumption that is false for essentially all real data — and yet it filters spam, routes support tickets and triages documents well enough that it has survived five decades of fancier competition. this page derives it, counts why the “naive” part is the whole point, builds a spam filter from scratch, and is honest about where it breaks (its probabilities, not its decisions).
Entropy and Information Gain
Definition
(Entropy)
The entropy of a dataset \(S\) with classes \(C\) is:
\[H(S) = -\sum_{c \in C} p_c \log_2(p_c)\]
where \(p_c\) is the proportion of examples belonging to class \(c\). Entropy is maximised when classes are equally distributed and zero when all examples belong to a single class.
Definition
(Information Gain)
The information gain of splitting dataset \(S\) on attribute \(A\) is:
\[\text{IG}(S, A) = H(S) - \sum_{v \in \text{Values}(A)} \frac{|S_v|}{|S|} H(S_v)\]
Origins
The perceptron learning algorithm is the most simple algorithm we have for Binary Classification.
It was introduced by Frank Rosenblatt in his seminal paper: “The Perceptron: A Probabilistic Model for Information Storage and Organization in the Brain” in 1958. The history however dates back further to the theoretical foundations of Warren McCulloch and Walter Pitts in 1943 and their paper “A Logical Calculus of the Ideas Immanent in Nervous Activity”. The interested reader may visit these links for annotations and the original pdfs.
Backlinks (3)
1. MNIST /tags/mnist/
An Embedded Notebook
History
Abstract
The MNIST dataset (Modified National Institute of Standards and Technology) has been very influential in machine learning and computer vision. It is an easy and popular dataset that has been used since it’s inception in 1998 as a benchmark for Machine Learning Models. Historically it has enhanced the evolution of OCR (Optical Character Recognition) and assisted in the emergence of neural networks.
2. Wiki /wiki/
Knowledge is a paradox. The more one understand, the more one realises the vastness of his ignorance.
3. Machine Learning /wiki/ml/
Type 1 error