Inductive-Bias

averaged over all possible problems, every learning algorithm is exactly as good as random guessing — and every optimiser is exactly as good as blind enumeration. 𐃏 this sounds like nihilism but is actually the sharpest possible argument for inductive bias: an algorithm can only beat chance on some problems by losing to chance on others, so the whole game of machine learning is choosing whose lunch to eat.

the vision transformer asks a blunt question: if attention replaced recurrence for text, can it replace convolution for images? the answer (dosovitskiy et al. 2020, an image is worth 16x16 words) is yes — chop the image into patches, treat each patch as a token, and feed the sequence to a standard transformer encoder with almost no vision-specific machinery. 𐃏