the transformer (Vaswani, Ashish and Shazeer, Noam and Parmar, Niki and Uszkoreit, Jakob and Jones, Llion and Gomez, Aidan N. and Kaiser, Lukasz and Polosukhin, Illia, 2017) deleted recurrence from sequence modelling and replaced it with a single primitive — attention — applied in parallel over the whole sequence.𐃏every token gets to look at every other token in one matrix multiply, the maximum path length between any two positions drops from \(O(n)\) to \(O(1)\), and training parallelises across the sequence dimension. this page derives the machinery; a from-scratch decoder-only build lives at nanogpt.
a python dict maps a query to the value whose key matches exactly. attention softens this: every token emits a query vector \(q\), a key vector \(k\), and a value vector \(v\); the output for a query is a weighted average of all values, weighted by how well each key matches:
stack the \(n\) queries, keys, values into matrices \(Q, K \in \mathbb{R}^{n \times d_k}\), \(V \in \mathbb{R}^{n \times d_v}\) and the whole thing is one expression — scaled dot-product attention:
suppose the components of \(q\) and \(k\) are independent with mean \(0\) and variance \(1\). then the raw dot product \(q \cdot k = \sum_{i=1}^{d_k} q_i k_i\) has
since each product term has variance \(\mathbb{E}[q_i^2]\,\mathbb{E}[k_i^2] = 1\). so raw logits have standard deviation \(\sqrt{d_k}\) — at \(d_k = 64\) that is \(8\), and softmax over logits of magnitude \(\pm 8\) is essentially an argmax: one weight near \(1\), the rest near \(0\), and gradients through the softmax vanish. dividing by \(\sqrt{d_k}\) restores unit variance regardless of head width (Vaswani, Ashish and Shazeer, Noam and Parmar, Niki and Uszkoreit, Jakob and Jones, Llion and Gomez, Aidan N. and Kaiser, Lukasz and Polosukhin, Illia, 2017). empirically:
one attention pattern per layer is a bottleneck: a token may simultaneously want its syntactic head, the previous mention of its referent, and the adjacent word. the fix is to run \(h\) attention functions in parallel on learned projections into lower-dimensional subspaces:
with \(W_i^Q, W_i^K \in \mathbb{R}^{d_{\text{model}} \times d_k}\), \(W_i^V \in \mathbb{R}^{d_{\text{model}} \times d_v}\), \(W^O \in \mathbb{R}^{h d_v \times d_{\text{model}}}\). the original paper uses \(h = 8\) heads with \(d_k = d_v = d_{\text{model}}/h = 64\), so the total compute is close to single-head attention at full width (Vaswani, Ashish and Shazeer, Noam and Parmar, Niki and Uszkoreit, Jakob and Jones, Llion and Gomez, Aidan N. and Kaiser, Lukasz and Polosukhin, Illia, 2017).𐃏each head learns its own notion of relevance; concatenation plus \(W^O\) lets the block mix what the heads found.
attention is permutation-equivariant: shuffle the input rows and the output rows shuffle identically. word order must therefore be injected explicitly. the original transformer adds a fixed sinusoidal code to each embedding:
i.e. each embedding dimension pair \((2i, 2i+1)\) traces a sinusoid whose wavelength grows geometrically from \(2\pi\) to \(10000 \cdot 2\pi\) as \(i\) increases. two reasons for this choice:
relative positions are linear. for any fixed offset \(k\), \(PE_{pos+k}\) is a linear function of \(PE_{pos}\) — each frequency pair transforms by a plain 2d rotation1 — so a learned weight matrix can express “attend 3 tokens back” without knowing the absolute position.
extrapolation. sinusoids are defined for every \(pos\), so the code exists (if not necessarily well-behaved) beyond the training length; a learned position table simply runs out of rows.
learned absolute embeddings perform about the same at training lengths (Vaswani, Ashish and Shazeer, Noam and Parmar, Niki and Uszkoreit, Jakob and Jones, Llion and Gomez, Aidan N. and Kaiser, Lukasz and Polosukhin, Illia, 2017); modern decoder stacks mostly use rotary embeddings (rope), which apply the rotation to \(q\) and \(k\) directly instead of adding a code to the input.
a transformer layer is two sublayers, each wrapped in a residual connection and a layer normalisation; the encoder stacks \(N = 6\) of these, the decoder adds a third sublayer for cross-attention.
encoder layer: self-attention (every position attends to every position), then a position-wise feed-forward network
applied identically to each token, with inner width \(d_{ff} = 4\,d_{\text{model}}\) in the original (2048 vs 512).
decoder layer: masked self-attention over the output-so-far, then cross-attention where \(Q\) comes from the decoder and \(K, V\) from the encoder output, then the ffn.
the original is post-ln: \(x \leftarrow \operatorname{LN}(x + \operatorname{Sublayer}(x))\) — normalisation sits on the residual path itself. modern stacks (gpt-2 onwards) are pre-ln: \(x \leftarrow x + \operatorname{Sublayer}(\operatorname{LN}(x))\).𐃏the difference matters at depth: in pre-ln the identity path from input to output is unbroken, so gradients reach early layers unattenuated and deep stacks train without the delicate learning-rate warmup that post-ln requires. post-ln, when it does converge, tends to give slightly stronger final models at shallow depth — which is why the 6-layer original got away with it. see xiong et al. 2020, on layer normalization in the transformer architecture, for the gradient analysis.
a language model must not see the future: position \(t\) may attend only to positions \(\le t\). implementation is one line — before the softmax, set
\begin{equation}
S_{ij} \leftarrow \begin{cases} S_{ij} & j \le i \\ -\infty & j > i \end{cases}
\end{equation}
so the softmax assigns zero weight above the diagonal. the same trick handles padding tokens in batched training. note the mask acts on scores, not weights: masking after the softmax would break the rows-sum-to-1 property.
the full encoder-decoder transformer. left: encoder stack. right: decoder stack with masked self-attention and cross-attention reading the encoder output. add-and-norm wraps every sublayer (post-ln shown, as in the original).
three architectural families fall out of this diagram: encoder-only (bert: bidirectional attention, good for classification/retrieval), decoder-only (gpt: causal mask everywhere, see llms), and the full encoder-decoder (translation, whisper, t5).
per layer, for sequence length \(n\) and model width \(d\):
component
time
memory
self-attention
\(O(n^2 d)\)
\(O(n^2)\) scores
feed-forward
\(O(n d^2)\)
\(O(n d)\)
rnn layer (contrast)
\(O(n d^2)\)
\(O(n d)\), sequential
attention dominates once \(n > d\) — the quadratic term is the price of every-token-sees-every-token, and the reason long-context research obsesses over sparse, linear, and flash attention variants.𐃏against recurrence (Goodfellow, Ian, 2016) the trade is: transformers pay a factor of \(n\) in compute to gain \(O(1)\) path length between any two tokens and full parallelism over the sequence during training.
inference is different. generating token \(n+1\) autoregressively only needs the new token’s query against all previous keys and values. recomputing \(K, V\) for the whole prefix at every step would cost \(O(n^2 d)\) per token; instead the kv-cache stores each layer’s keys and values as they are produced, so each new token costs \(O(nd)\) attention per layer. the price is memory: \(2 \times N_{\text{layers}} \times n \times d\) activations per sequence, which is exactly the resource that limits context length on real hardware (and why serving stacks quantise the cache and invent tricks like grouped-query attention to shrink it).
scaled dot-product attention and a single head, in numpy. note the row sums, and how the causal mask zeroes everything above the diagonal — row \(t\) of the causal matrix is exactly the unmasked row \(t\) renormalised over positions \(\le t\).
importnumpyasnpdefsoftmax(z,axis=-1):z=z-z.max(axis=axis,keepdims=True)# numerical stabilitye=np.exp(z)returne/e.sum(axis=axis,keepdims=True)defscaled_dot_product_attention(Q,K,V,mask=None):d_k=Q.shape[-1]scores=Q@K.T/np.sqrt(d_k)# (n, n)ifmaskisnotNone:scores=np.where(mask,scores,-1e9)# forbid masked positionsA=softmax(scores,axis=-1)# rows sum to 1returnA@V,Arng=np.random.default_rng(42)n,d_model,d_k=4,8,4# 4 tokens, model dim 8, head dim 4X=rng.normal(size=(n,d_model))# toy token embeddingsW_q=rng.normal(size=(d_model,d_k))/np.sqrt(d_model)W_k=rng.normal(size=(d_model,d_k))/np.sqrt(d_model)W_v=rng.normal(size=(d_model,d_k))/np.sqrt(d_model)Q,K,V=X@W_q,X@W_k,X@W_v# one head's projectionsout,A=scaled_dot_product_attention(Q,K,V)print("attention weights (rows = queries):")print(np.round(A,3))print("row sums:",np.round(A.sum(axis=1),6))causal=np.tril(np.ones((n,n),dtype=bool))# causal maskout_c,A_c=scaled_dot_product_attention(Q,K,V,mask=causal)print("\ncausal attention weights:")print(np.round(A_c,3))print("head output shape:",out_c.shape)
the near-uniform weights are what untrained attention looks like — random projections give logits close to zero, and softmax of near-zero logits is near-uniform. training sharpens the rows into the syntax- and coreference-shaped patterns that make attention maps fun to stare at.
with \(\omega_i = 10000^{-2i/d_{\text{model}}}\), the pair \((\sin \omega_i (pos+k),\ \cos \omega_i (pos+k))\) equals the rotation matrix \(\begin{pmatrix} \cos \omega_i k & \sin \omega_i k \\ -\sin \omega_i k & \cos \omega_i k \end{pmatrix}\) applied to \((\sin \omega_i\, pos,\ \cos \omega_i\, pos)\) — the standard angle-addition identities.
Vaswani, Ashish and Shazeer, Noam and Parmar, Niki and Uszkoreit, Jakob and Jones, Llion and Gomez, Aidan N. and Kaiser, Lukasz and Polosukhin, Illia (2017). Attention is All You Need. ↩︎
stoi={ch:ifori,chinenumerate(chars)}itos={i:chfori,chinenumerate(chars)}encode=lambdas:[stoi[c]forcins]# defines function taking in string, outputs list of intsdecode=lambdal:''.join([itos[i]foriinl])# input: list of integers, outputs stringprint(encode("hello world"))print(decode(encode("hello world")))
the vanilla rnn cannot learn long-range dependencies: its gradient signal is a product of jacobians that shrinks or blows up geometrically with distance. the lstm’s answer is architectural, not numerical — give the network a second state, updated additively rather than by repeated matrix multiplication, and let learned gates decide what enters, what stays, and what leaves.𐃏the design dates to hochreiter and schmidhuber’s 1997 paper (long short-term memory, neural computation 9(8)), and for two decades it was simply what “rnn” meant in practice (Goodfellow, Ian, 2016).
feedforward networks eat fixed-size vectors. sequences — text, audio, sensor streams — have no fixed size, and worse, their order carries the meaning. the recurrent neural network solves both problems with one idea: maintain a hidden state that is updated by the same function at every time step.𐃏parameter count stops depending on sequence length, and the state becomes a lossy summary of everything seen so far (Goodfellow, Ian, 2016).
before a transformer sees a single number, text must be cut into pieces and each piece mapped to an integer. the tokeniser is that cut — the least glamorous and most consequential preprocessing step in the whole pipeline, since it fixes the vocabulary, the sequence length, and what the model can represent at all.𐃏
the vision transformer asks a blunt question: if attention replaced recurrence for text, can it replace convolution for images? the answer (dosovitskiy et al. 2020, an image is worth 16x16 words) is yes — chop the image into patches, treat each patch as a token, and feed the sequence to a standard transformer encoder with almost no vision-specific machinery.𐃏
a large language model is a decoder-only transformer trained on one absurdly simple objective — predict the next token — scaled until the emergent behaviour stops looking simple.𐃏this page is the map from that objective to a deployed assistant: the loss, the scaling laws that size the model, the pretrain-align pipeline, and the inference tricks. the hands-on build is nanogpt.
diffusion models generate by learning to undo noise: destroy an image with a fixed gaussian corruption process, train a network to reverse one small step of the destruction, then chain the reversals from pure noise back to data (ho et al. 2020, denoising diffusion probabilistic models).𐃏stable diffusion (rombach et al. 2022, high-resolution image synthesis with latent diffusion models) runs this machinery not on pixels but in the latent space of an autoencoder, with a text-conditioned u-net doing the denoising. first the maths, then the architecture.
a convolutional neural network is a feedforward network with its linear layers put on a diet: instead of every unit seeing every input, each unit sees a small local window, and every window is processed by the same small set of weights.𐃏this page builds the operation from its definition, gets the geometry formulas straight, walks the canonical architectures, and ends with a convolution written in loops and checked against scipy (Goodfellow, Ian, 2016).
retrieval-augmented generation bolts a search engine onto a language model: fetch relevant documents at query time and paste them into the prompt, so the model answers from evidence rather than from its frozen weights (lewis et al. 2020, retrieval-augmented generation for knowledge-intensive nlp).𐃏it is the pragmatic alternative to baking every fact into an llm’s weights.