Locally Weighted Regression

Thu, 09 Jul 2026 21:02:56 +1000

a straight line is too rigid for a wiggly world, and a global degree-9 polynomial is a hostage negotiation. 𐃏 locally weighted regression (LWR — and its robust cousin LOWESS) takes the diplomatic route: fit the simplest possible model, but fit it freshly at every query point, paying attention only to the training points nearby.

motivation

linear regression commits to one \(\theta\) for the whole input space. if the true \(f\) bends, the residue of that commitment is bias everywhere.
the fix need not be a fancier global family. any smooth function is locally linear — taylor says so — so a linear fit weighted toward a neighbourhood of \(x_0\) can track an arbitrary smooth \(f\).
the price: there is no longer a “trained model”. LWR is memory-based and non-parametric — like k-nearest neighbours, it keeps the entire training set and does all of its work at prediction time. training is \(O(1)\); every query costs a fresh weighted least-squares solve. 𐃏

the estimator

weighted least squares at a query point

fix a query \(x_0\). assign each training point a weight \(w_i(x_0) \ge 0\) that decays with distance from \(x_0\), then solve the weighted least-squares problem

Bandwidth on Aayush Bajaj's Augmenting Infrastructure

Locally Weighted Regression

motivation#

the estimator#

weighted least squares at a query point#

motivation

the estimator

weighted least squares at a query point