an autoencoder is a network trained to do the one thing that sounds useless: output its own input. the trick is the obstacle course in the middle — a bottleneck, a corruption, a penalty — that makes verbatim copying impossible, so the network is forced to learn what about the input is worth keeping. 𐃏 the family tree below runs from the linear special case (which is pca wearing a trenchcoat) to the variational autoencoder, which turns the whole construction into a generative model (Goodfellow, Ian, 2016).
Deep Learning
the perceptron is the hydrogen atom of neural networks: one neuron, one weight vector, one threshold — and yet it already exhibits the two behaviours that define the whole field. 𐃏 it learns from mistakes with a provable convergence guarantee, and it fails on problems its geometry cannot express. this page covers both, then swaps the hard threshold for a sigmoid so that gradient descent can take over — a companion to the sign-loss perceptron page, which treats the classical algorithm on its own.
a gan trains a generator by making it play a game against a learned critic: the generator \(G\) maps noise to samples, the discriminator \(D\) tries to tell those samples from real data, and each improves by exploiting the other’s current weakness — density estimation recast as a two-player minimax game (goodfellow et al. 2014, generative adversarial networks). 𐃏 the framework is treated in ch. 20 of (Goodfellow, Ian, 2016).
before a transformer sees a single number, text must be cut into pieces and each piece mapped to an integer. the tokeniser is that cut — the least glamorous and most consequential preprocessing step in the whole pipeline, since it fixes the vocabulary, the sequence length, and what the model can represent at all. 𐃏
the vision transformer asks a blunt question: if attention replaced recurrence for text, can it replace convolution for images? the answer (dosovitskiy et al. 2020, an image is worth 16x16 words) is yes — chop the image into patches, treat each patch as a token, and feed the sequence to a standard transformer encoder with almost no vision-specific machinery. 𐃏
the vanilla rnn cannot learn long-range dependencies: its gradient signal is a product of jacobians that shrinks or blows up geometrically with distance. the lstm’s answer is architectural, not numerical — give the network a second state, updated additively rather than by repeated matrix multiplication, and let learned gates decide what enters, what stays, and what leaves. 𐃏 the design dates to hochreiter and schmidhuber’s 1997 paper (long short-term memory, neural computation 9(8)), and for two decades it was simply what “rnn” meant in practice (Goodfellow, Ian, 2016).
feedforward networks eat fixed-size vectors. sequences — text, audio, sensor streams — have no fixed size, and worse, their order carries the meaning. the recurrent neural network solves both problems with one idea: maintain a hidden state that is updated by the same function at every time step. 𐃏 parameter count stops depending on sequence length, and the state becomes a lossy summary of everything seen so far (Goodfellow, Ian, 2016).
We have seen what can be learned by the perceptron algorithm — namely, linear decision boundaries for binary classification problems.
It may also be of interest to know that the perceptron algorithm can also be used for regression with the simple modification of not applying an activation function (i.e. the sigmoid). I refer the interested reader to open another tab.
We begin with the punchline:
XOR
Not linearly separable in \(\mathbb{R}^2\)
Backlinks (3)
1. Literate Programming and Donald Knuth /blog/literate-programming/
I was first introduced to this concept by Distrotube (Derek Taylor’s) “literate config” files. At the time I was not using emacs and thus all the code I was writing was sparingly commented.
Since then, I have entered a world of Machine Learning and Deep Learning, where suddenly in 4 lines, I can sit atop my high-horse and perform sentiment analysis with tensorflow and keras!
from transformers import pipeline
classifier = pipeline('sentiment-analysis')
prediction = classifier("Donald Knuth was the greatest computer scientist.")[0]
print(prediction)
In such an age of abstraction complexity, it becomes paramount to distill what is happening at the last few \((n-k)\) layers.
2. Wiki /wiki/
Knowledge is a paradox. The more one understand, the more one realises the vastness of his ignorance.
3. Machine Learning /wiki/ml/
Type 1 error