Transformers

Thu, 09 Jul 2026 21:02:56 +1000

the transformer (Vaswani, Ashish and Shazeer, Noam and Parmar, Niki and Uszkoreit, Jakob and Jones, Llion and Gomez, Aidan N. and Kaiser, Lukasz and Polosukhin, Illia, 2017) deleted recurrence from sequence modelling and replaced it with a single primitive — attention — applied in parallel over the whole sequence. 𐃏 every token gets to look at every other token in one matrix multiply, the maximum path length between any two positions drops from \(O(n)\) to \(O(1)\), and training parallelises across the sequence dimension. this page derives the machinery; a from-scratch decoder-only build lives at nanogpt.

Self-Attention on Aayush Bajaj's Augmenting Infrastructure

Transformers