Tokenisers

Fri, 10 Jul 2026 01:42:07 +1000

before a transformer sees a single number, text must be cut into pieces and each piece mapped to an integer. the tokeniser is that cut — the least glamorous and most consequential preprocessing step in the whole pipeline, since it fixes the vocabulary, the sequence length, and what the model can represent at all. 𐃏

Subword on Aayush Bajaj's Augmenting Infrastructure

Tokenisers