Fine Tuning LLM

a large language model is a decoder-only transformer trained on one absurdly simple objective — predict the next token — scaled until the emergent behaviour stops looking simple. 𐃏 this page is the map from that objective to a deployed assistant: the loss, the scaling laws that size the model, the pretrain-align pipeline, and the inference tricks. the hands-on build is nanogpt.

retrieval-augmented generation bolts a search engine onto a language model: fetch relevant documents at query time and paste them into the prompt, so the model answers from evidence rather than from its frozen weights (lewis et al. 2020, retrieval-augmented generation for knowledge-intensive nlp). 𐃏 it is the pragmatic alternative to baking every fact into an llm’s weights.

why retrieve at all#

three problems that no amount of scaling fixes cleanly, and retrieval fixes cheaply: