this page collects the closed-form solutions to regularised regression (where they exist) and the iterative approximations we fall back on (where they don’t). 𐃏 along the way we will see that regularisation is not an ad-hoc hack but a perfectly sensible artefact of estimation: it drops straight out of MAP (maximum a posteriori) inference once you put a prior on the coefficients.
Bayesian
2026-07-19
The content here-in has been influenced by Mung Chiang’s Networked Life and Introduction to Algorithms by CLRS.
This page pairs well with Statistics.
Elements of Probability Theory
Definition
(Random Experiment, Sample Space, Events)
A random experiment has uncertain outcomes. The sample space \(S\) is the set of all possible outcomes. An event \(E\) is a subset of \(S\). The certain event is \(S\); the impossible event is \(\varnothing\).
Definition
(Probability Measure (Kolmogorov Axioms))
A probability space \((S,\mathcal{F},P)\) consists of a sample space \(S\), a \(\sigma\)-algebra \(\mathcal{F}\subseteq 2^S\), and a function \(P:\mathcal{F}\to[0,1]\) such that:
This blog post has been created to convince you that real-world probability, is in fact Bayesian probability.
Anyone who believes that a frequentist approach is superior may be correct (for that particular example), but it must be said that the Bayesian framework is a superset of this naive and trivial card-playing model of probability.
We are no longer trying to determine the probability of landing a
double-six dice roll, and rather we are trying to figure out what
the probability is that Mia (our cat) will be waiting for us on the
porch when we get home.
Backlinks (2)
1. Wiki /wiki/
Knowledge is a paradox. The more one understand, the more one realises the vastness of his ignorance.