approximate inference in graphical models — forward and rejection sampling, likelihood weighting, gibbs sampling and mcmc, and loopy belief propagation, with the trade-offs that decide which to reach for.
Explainer
2026-08-31
bayesian networks from first principles — the chain rule, conditional probability tables, d-separation and its valve intuition, markov blankets, i-maps, naive bayes classifiers and gaussian networks.
how regression models are estimated and judged, from MATH5806: score and fisher information, newton-raphson and the method of scoring, the wald/score/likelihood-ratio trio, and exact t and F inference in the linear gaussian model.
exact inference in graphical models — factors, variable elimination, elimination orders and treewidth, the jointree (junction tree) algorithm with message passing, and mpe/map queries.
exponential-family GLM theory from MATH5806: the canonical form, link functions, fisher scoring as iteratively reweighted least squares, deviance, poisson and log-linear regression, and quasi-likelihood.
learning bayesian networks from data — maximum-likelihood estimation of cpts, additive smoothing, handling missing data with expectation-maximisation, and structure learning from chow-liu trees to score-based search.
sequence models as graphical models — markov chains and stationary distributions, hidden markov models, the forward and viterbi algorithms on the state trellis, particle filtering, and dynamic bayesian networks.
undirected graphical models — gibbs distributions and the partition function, hammersley-clifford, factor graphs, energy minimisation and map-mrf, separation semantics, moralisation, and conditional random fields.
In 1931 a twenty-five-year-old in Vienna published a paper proving that mathematics cannot fully formalise itself: any consistent formal system rich enough to do arithmetic contains true statements it cannot prove, and — worse — cannot even prove its own consistency (Gödel, Kurt, 1931). 𐃏 This page walks the actual argument: what the theorem says precisely, the two technical inventions that power it (Gödel numbering and diagonalisation), and — just as important — the long list of things it does not say.
Entropy and Information Gain
Definition
(Entropy)
The entropy of a dataset \(S\) with classes \(C\) is:
\[H(S) = -\sum_{c \in C} p_c \log_2(p_c)\]
where \(p_c\) is the proportion of examples belonging to class \(c\). Entropy is maximised when classes are equally distributed and zero when all examples belong to a single class.
Definition
(Information Gain)
The information gain of splitting dataset \(S\) on attribute \(A\) is:
\[\text{IG}(S, A) = H(S) - \sum_{v \in \text{Values}(A)} \frac{|S_v|}{|S|} H(S_v)\]