LLM from scratch

a large language model is a decoder-only transformer trained on one absurdly simple objective — predict the next token — scaled until the emergent behaviour stops looking simple. 𐃏 this page is the map from that objective to a deployed assistant: the loss, the scaling laws that size the model, the pretrain-align pipeline, and the inference tricks. the hands-on build is nanogpt.

the objective#

given a token sequence \(x_1, \dots, x_T\), the model factorises the joint probability autoregressively and is trained to maximise its log-likelihood — equivalently, to minimise the average next-token cross-entropy:

\begin{equation} \mathcal{L}(\theta) = -\frac{1}{T} \sum_{t=1}^{T} \log p_\theta\!\left(x_t \mid x_{<t}\right). \end{equation}

that is the entire pretraining loss. three things worth stating plainly:

• the label for position \(t\) is just \(x_{t+1}\) — the data is its own supervision, which is why the internet’s worth of raw text suffices (self-supervised, no annotation).
• the causal mask makes all \(T\) predictions computable in one forward pass, so a length-\(T\) sequence yields \(T\) training signals at once.
• \(\exp(\mathcal{L})\) is perplexity, the effective branching factor — “how many equally-likely tokens is the model choosing among”. a perplexity of \(10\) means the model is as uncertain as a fair 10-sided die at each step.

scaling laws#

the empirical discovery that made the field industrial: test loss falls as a smooth power law in model parameters \(N\), dataset tokens \(D\), and compute \(C\), over many orders of magnitude (kaplan et al. 2020, scaling laws for neural language models):

\begin{equation} L(N) \approx \left(\frac{N_c}{N}\right)^{\alpha_N}, \qquad L(D) \approx \left(\frac{D_c}{D}\right)^{\alpha_D}, \end{equation}

with small exponents (\(\alpha_N \approx 0.076\) in the original fit). the practical question is how to spend a fixed compute budget \(C \approx 6ND\) ¹ between a bigger model and more data.

kaplan et al. concluded compute should go mostly to model size. the chinchilla correction (hoffmann et al. 2022, training compute-optimal large language models) re-ran the sweep more carefully and found \(N\) and \(D\) should scale in equal proportion: roughly 20 training tokens per parameter at the compute-optimal point. 𐃏 honest caveats: the 20:1 ratio is compute-optimal for training, not for deployment — if you will serve a model billions of times, it is rational to “over-train” a smaller model far past 20:1 to save inference cost forever, which is exactly what the llama models do. and the constants are dataset- and architecture-dependent; 20:1 is a landmark, not a law of nature.

the training pipeline#

a shipped assistant is three stages stacked on the pretrained base.

• pretraining. the cross-entropy objective above, on trillions of tokens. produces a base model — a superb next-token predictor that will happily continue a prompt but has no notion of “being helpful”.
• supervised fine-tuning (sft). continue the same next-token training on a curated set of (instruction, ideal-response) pairs. cheap, and it teaches the format of following instructions. detail lives at fine tuning.
• preference optimisation. sft imitates demonstrations; the last stage optimises against comparisons, which are easier for humans to give (“A is better than B”) than perfect answers.
  • rlhf (ouyang et al. 2022, training language models to follow instructions): train a reward model on human preference pairs, then optimise the policy against it with ppo, penalised by a kl term to stay near the sft model.
  • dpo (rafailov et al. 2023, direct preference optimization): skip the reward model and rl entirely. dpo shows the rlhf objective has a closed-form optimum, giving a simple classification loss directly on preference pairs \((y^+ \text{ preferred over } y^-)\):

\begin{equation} \mathcal{L}_{\text{DPO}} = -\,\mathbb{E}_{(x, y^+, y^-)} \left[ \log \sigma\!\left( \beta \log \frac{\pi_\theta(y^+ \mid x)}{\pi_{\text{ref}}(y^+ \mid x)} - \beta \log \frac{\pi_\theta(y^- \mid x)}{\pi_{\text{ref}}(y^- \mid x)} \right) \right], \end{equation}

where \(\pi_{\text{ref}}\) is the frozen sft model and \(\beta\) controls how far the policy may drift. same goal as rlhf, no reward model, no rollout instability — which is why it took over.

inference#

sampling#

the base model outputs a distribution over the vocabulary; how you draw from it is a decoding choice, not a training one.

• temperature \(T\): divide logits by \(T\) before softmax. \(T<1\) sharpens toward greedy (safe, repetitive), \(T>1\) flattens (diverse, riskier), \(T\to 0\) is argmax.
• top-k: keep only the \(k\) most probable tokens, renormalise, sample.
• top-p (nucleus): keep the smallest set of tokens whose cumulative probability exceeds \(p\), renormalise, sample — an adaptive cutoff that widens on flat distributions and narrows on peaked ones.

import numpy as np

vocab = ["the", "a", "cat", "dog", "ran", "sat", "quickly", "blue", "?", "!"]
logits = np.array([3.2, 2.9, 1.5, 1.4, 0.8, 0.7, -0.3, -1.0, -2.0, -3.5])

def softmax(z):
    z = z - z.max();  e = np.exp(z);  return e / e.sum()

def temperature(logits, T):
    return softmax(logits / T)

def top_k(logits, k):
    idx = np.argsort(logits)[::-1][:k]          # keep k largest
    masked = np.full_like(logits, -np.inf); masked[idx] = logits[idx]
    return softmax(masked)

def top_p(logits, p):
    probs = softmax(logits)
    order = np.argsort(probs)[::-1]
    cum = np.cumsum(probs[order])
    keep = order[:np.searchsorted(cum, p) + 1]  # smallest set with mass >= p
    masked = np.full_like(logits, -np.inf); masked[keep] = logits[keep]
    return softmax(masked)

def show(name, probs):
    top = np.argsort(probs)[::-1][:5]
    print(f"{name:<16} " + "  ".join(f"{vocab[i]}={probs[i]:.3f}" for i in top if probs[i] > 0))

show("T=1.0", temperature(logits, 1.0))
show("T=0.5 (sharp)", temperature(logits, 0.5))
show("T=2.0 (flat)",  temperature(logits, 2.0))
show("top-k=3",       top_k(logits, 3))
show("top-p=0.9",     top_p(logits, 0.9))

T=1.0            the=0.432  a=0.320  cat=0.079  dog=0.071  ran=0.039
T=0.5 (sharp)    the=0.615  a=0.338  cat=0.021  dog=0.017  ran=0.005
T=2.0 (flat)     the=0.271  a=0.233  cat=0.116  dog=0.110  ran=0.082
top-k=3          the=0.520  a=0.385  cat=0.095
top-p=0.9        the=0.479  a=0.355  cat=0.087  dog=0.079

lowering \(T\) concentrates mass on the and a; raising it spreads probability toward the tail. top-k=3 hard-truncates to three tokens; top-p=0.9 keeps four here because that is the smallest nucleus reaching 90% mass.

making it fast#

• kv-cache. the dominant inference optimisation: cache each layer’s keys and values so generating token \(n+1\) costs \(O(nd)\) attention instead of recomputing the prefix at \(O(n^2 d)\). its memory footprint is the main limit on context length — see transformers.
• speculative decoding. run a small cheap “draft” model to propose several tokens, then verify them all in one forward pass of the big model, accepting each draft token with probability \(\min(1, p_{\text{big}}/p_{\text{draft}})\) and resampling from the corrected residual distribution on the first rejection — which provably preserves the big model’s output distribution (greedy decoding degenerates to “longest agreeing prefix”). because verification is parallel while generation is sequential, this gives a 2–3x wall-clock speedup with no change to the output distribution — the big model’s samples are provably unchanged.

emergent abilities, honestly#

some capabilities (multi-step arithmetic, instruction following, in-context learning) appear to switch on abruptly past a scale threshold rather than improving smoothly (wei et al. 2022, emergent abilities of large language models). the honest caveat: a well-cited rebuttal (schaeffer et al. 2023, are emergent abilities a mirage?) argues much of the apparent discontinuity is an artefact of discontinuous metrics — exact-match accuracy jumps, but the underlying per-token log-likelihood improves smoothly. the phenomenon is real and useful to plan around; the “phase transition” framing is contested.

prompting and evaluation#

• few-shot / in-context learning. put \(k\) worked examples in the prompt and the model generalises the pattern with no weight update — a behaviour that itself emerged with scale (brown et al. 2020, language models are few-shot learners).
• chain-of-thought. prompting the model to “think step by step” before answering elicits intermediate reasoning and sharply improves multi-step tasks — the model conditions its final answer on its own generated scratchpad (wei et al. 2022, chain-of-thought prompting).
• evaluation. benchmarks (mmlu for knowledge, gsm8k for maths, humaneval for code) give comparable numbers but leak into training sets over time; pairwise human or llm-judge comparisons (arena-style elo) capture preference but are noisy and gameable. no single number suffices — every eval measures a proxy.

(Vaswani, Ashish and Shazeer, Noam and Parmar, Niki and Uszkoreit, Jakob and Jones, Llion and Gomez, Aidan N. and Kaiser, Lukasz and Polosukhin, Illia, 2017) is the architecture underneath all of it; the tokeniser fixes what counts as a token in the loss above.

see also#

• nanogpt — build the decoder that this page scales up
• running llms locally — inference on consumer hardware
• fine tuning llms — the sft and adapter stage in detail
• retrieval-augmented generation — grounding the model in external documents
• tokenisers — how the input sequence is built
• transformers — the block that is stacked and scaled