Model Selection

every fitted model so far has been judged on the data it was fitted to, and that judge is bribed: least squares chose \(\hat\beta\) to make the training error small, so the training error flatters it. this page is about the two honest questions that follow — which of several candidate models to use, and how wrong the chosen one will be on new data — and the three families of answer: resample the data (cross-validation, the bootstrap), correct the training error analytically (\(C_p\), AIC, BIC), or search over subsets of predictors with one of those scores as the objective. 𐃏

the bias-variance decomposition explains why test error is U-shaped in model flexibility and the cross-validation notebook tunes \(\lambda\) empirically; neither is repeated here. what this page adds is the regression statistician’s bookkeeping: precise definitions of the errors being estimated, the optimism identity that makes \(C_p\) and AIC inevitable, the leave-one-out shortcut that makes LOOCV free for linear smoothers, and one running data set — thirty cheddar cheeses — on which every method is made to give a number.

selection versus assessment#

the training error cannot supply that estimate. in the course’s polynomial example, twenty points are drawn from \(Y = X + \varepsilon\) with \(\varepsilon \sim N(0, 0.25^2)\) and polynomials of degree 5, 10 and 15 are fitted. every degree \(k \ge 1\) contains the truth, and the training MSE can only fall as \(k\) grows, since the larger model nests the smaller. yet the degree-15 fit (16 coefficients, 4 residual degrees of freedom) threads every point and leaves the plotting window three times; its test MSE is far above the noise floor \(\sigma^2 = 1/16 = 0.0625\) that no model can beat. small training error with large test error is overfitting, and the rest of the page is about measuring the gap instead of eyeballing it.

statistical decision theory#

the marginal \(p(x)\) is a modelling decision in disguise: it says which regions of input space the predictor must serve well (uniform on \([-1, 1]\) if all inputs matter equally, \(N(0,1)\) if those near zero matter more).

this is what regression is: the linear model and the GLM are parametric guesses at \(\mathbb{E}(Y \mid X = x)\), and logistic regression and discriminant analysis are parametric guesses at \(\Pr(Y = j \mid X = x)\).

three errors, and the optimism#

fix a training set \(\mathcal{T} = \{(x_i, y_i)\}_{i=1}^N\) drawn from \(p(x, y)\) and a fit \(\hat f\) built from it.

\(\mathrm{Err}(\mathcal{T})\) is the quantity one wants — the quality of this fit — but resampling methods end up estimating its average \(\mathrm{Err}\), and \(\overline{\mathrm{err}}\) estimates neither: the fit was chosen to minimise loss at the training responses, so redrawing those responses can only hurt it on average. the optimism has an exact form.

the optimism is the covariance of a fitted value with its own response: how hard the fit chases the noise. it grows linearly in the number of coefficients, and for a shrinkage smoother such as ridge \(\operatorname{tr}\mathbf H\) is the effective degrees of freedom. every information criterion below is this identity with \(\sigma^2\) estimated.

proofs#

cross-validation#

the validation-set approach splits the data once at random, fits on one part and scores on the other. it is simple and has two flaws: the estimate depends heavily on the split, and the model is trained on far fewer points than the final model, so it overestimates that model’s error. cross-validation repairs both by letting every observation take a turn as test data.

one fit replaces \(N\). the leave-one-out residual is the ordinary residual inflated by \(1/(1 - h_i)\): barely for typical points (leverages average \(d/N\)), enormously for high-leverage ones, which are precisely the points whose removal moves the fit — see regression diagnostics, where the same quantity is the PRESS residual. the proof uses only that \(\hat{\mathbf y} = \mathbf H \mathbf y\) is linear in \(\mathbf y\) and that refitting to a response with \(y_i\) replaced by its own leave-one-out prediction reproduces \(\hat\beta_{(-i)}\), so the identity holds for any linear smoother with that self-consistency property (ridge, smoothing splines, kernel smoothers). in R: mean(((y - predict(fit)) / (1 - hatvalues(fit)))^2).

proofs#

the bootstrap#

for taste ~ H2S + Lactic with \(B = 500\), the bootstrap standard errors are 0.73 (H2S) and 6.52 (Lactic) against the model-based 1.14 and 7.96 — the bootstrap assumes neither normality nor constant variance, and here judges the fit a little more stable than normal theory does. as a prediction-error estimator the bootstrap needs more care.

the naive estimate is too optimistic because training and test sets overlap; the leave-one-out bootstrap predicts each point only from fits that never saw it, but is pessimistic for a reason the next result makes precise.

the convergence is fast: \(1 - (1 - 1/N)^N\) is 0.750 at \(N = 2\), 0.651 at \(N = 10\), 0.638 at \(N = 30\) (19.2 distinct cheeses per sample on average).

proofs#

information criteria#

instead of resampling, estimate the optimism and add it back. by the optimism identity, every criterion has the shape training error + penalty growing with model size.

proofs#

variable selection#

two facts organise the search. at a fixed size \(R^2_{\mathrm{adj}}\), AIC, AICc and BIC all agree — the penalty is the same for every subset of that size, so each is a monotone function of RSS — and within one stepwise step the AIC ranking is the ranking by single-term \(F\)-test \(p\)-value. only across sizes do the penalties disagree. and the shortcut has a price:

> drop1(lm(taste ~ ., data = cheddar))
       Df Sum of Sq    RSS    AIC
<none>              2668.4 142.64
Acetic  1      0.55 2669.0 140.65
H2S     1   1007.66 3676.1 150.25
Lactic  1    533.32 3201.7 146.11

proofs#

beyond selection#

subset selection is discrete: a coefficient is fitted freely or forced to zero, so small changes in the data can flip the chosen model, and the search is exponential in \(p\). two continuous alternatives use all \(p\) predictors while controlling the effective model size. shrinkage — ridge, lasso, elastic net — penalises coefficients (regularised regression), and its effective degrees of freedom \(\operatorname{tr}\mathbf H_\lambda\) slot straight into \(C_p\) via the optimism identity. dimension reduction — principal components regression — regresses on \(M < p\) linear combinations of the predictors. in both, the tuning parameter (\(\lambda\) or \(M\)) is chosen by the cross-validation of this page.

see also#

• the bias-variance decomposition — why test error is U-shaped; the derivation is there, not here
• cross-validation — a worked notebook tuning ridge and lasso by LOOCV
• regularised regression — shrinkage as the continuous alternative to subset selection
• regression diagnostics — leverage and the deleted residuals the LOOCV shortcut is built from
• linear regression — the hat matrix and RSS every criterion here consumes
• generalised linear models — deviance loss, and AIC/BIC for non-gaussian likelihoods
• analysis of variance — the \(F\)-tests that stepwise \(p\)-value rules reduce to
• performance metrics — error rates, sensitivity and ROC for classifiers