The Bayesian Cat

This blog post has been created to convince you that real-world probability, is in fact Bayesian probability.

Anyone who believes that a frequentist approach is superior may be correct (for that particular example), but it must be said that the Bayesian framework is a superset of this naive and trivial card-playing model of probability.

We are no longer trying to determine the probability of landing a double-six dice roll, and rather we are trying to figure out what the probability is that Mia (our cat) will be waiting for us on the porch when we get home.

History (optional)#

It starts with Cardano, a gambler first and a mathematician second, who counted dice outcomes in the 1560s and prudently didn’t publish. It leads to Pascal and Fermat, whose 1654 letters on the problem of points founded the subject, and to Leibniz, who dreamed of a calculus for weighing evidence in courtrooms — a distinctly Bayesian ambition, two centuries early.

Then the Reverend Thomas Bayes dies (1761) leaving a theorem in a drawer, Richard Price mails it to the Royal Society, and Laplace — who actually did the work — rediscovers it, generalises it, and uses it to weigh Saturn. For Laplace, probability was simply “good sense reduced to calculus”: a degree of belief, updated by evidence. Everyone did probability this way for a century.

Then came the great freeze. Boole muttered about the priors, Venn and von Mises rebuilt probability as long-run frequency, Fisher and Neyman industrialised it, and the Russian dude — Kolmogorov, 1933 — gave the whole thing its measure-theoretic constitution: a probability is a measure of total mass one on a σ-algebra, and nothing more is said about what it means. 𐃏

The thaw came from a physicist. R.T. Cox proved in 1946 that any system for reasoning about plausibility that satisfies a few common-sense desiderata — degrees of belief are real numbers, reasoning is consistent, equivalent states of knowledge get equal plausibilities — must be isomorphic to probability theory. And E.T. Jaynes spent forty years driving that theorem to its conclusion in Probability Theory: The Logic of Science: probability is not about randomness in the world, it is extended logic — the unique consistent generalisation of Boolean deduction to incomplete information. Jaynes is the patron saint of this blog post. Mia is merely its instrument.

The Crux of the Matter#

Prior. That is the crux. What is the probability Mia will be waiting on the chair on the porch? Well that depends on a few things: is it raining? did we feed her before we left? is it really hot? did she get into another cat fight?

Each of these questions, either answered or unanswered influence the probability.

Formally we have

\[ P(\text{Mia on Red Chair} \mid I) \;=\; \frac{P(I \mid \text{Mia})\, P(\text{Mia})}{P(I)} \]

where \(I\) is everything we know as we walk up the street: the weather, the hour, the state of her bowl, the neighbourhood cat politics. There is no ensemble here. There is no long run. We will arrive home exactly once tonight, and yet the number is perfectly meaningful: it is the odds at which we would bet on the silhouette being on the chair.

Watch it update. Base rate over the last year: she’s there maybe 6 evenings in 10, so \(P(\text{Mia}) = 0.6\). It’s raining — she hates rain — and the posterior slides toward 0.2. But we forgot to feed her this morning, and hunger has historically overpowered meteorology: back up to 0.7. Every scrap of information moves the number, and Bayes’ theorem is nothing but the bookkeeping that keeps the moves consistent. A frequentist can say none of this — there is no “population of tonights” to sample from. He must remain silent precisely where all the interesting questions live. 𐃏

Comparisons#

Physical Examples#

On the frequentist’s home turf the two frameworks agree, as they must. Dice, coins, shuffled decks, radioactive counts: symmetric physical setups repeated indefinitely, where the frequency is the rational degree of belief. 𐃏 The double-six roll is \(1/36\) under both flags; the Bayesian just derives it from indifference over the 36 states of knowledge rather than from an imagined infinity of throws.

The classical machinery — Casella & Berger’s estimators, van der Vaart’s asymptotics — is not wrong. It is a special case: what Bayesian updating collapses to when the prior is flat, the data are plentiful, and the question is repeatable. Where those conditions hold, use it gladly; the Bernstein–von Mises theorem even guarantees the two answers merge in the limit.

Meta-Physical Examples#

The nature of the matter is that the frequentist requires a black-box that can be sampled from infinitely, and reality rarely supplies one. What is the probability this bridge design fails? That this email is spam? That it rains tomorrow — this tomorrow, not an ensemble of tomorrows? That Mia is on the chair? Each event happens once. Frequentism must either stay silent or invent a fictitious population; the Bayesian simply conditions on what is known and reports a degree of belief. Thus superset.

And this is not merely philosophy — it is the working machinery of modern inference. Gelman’s Bayesian Data Analysis runs entire scientific workflows on posterior distributions; Gammerman’s MCMC and its descendants made the integrals tractable; O’Hagan & Forster wrote it into Kendall’s Advanced Theory. Machine learning quietly surrendered decades ago: Bishop, Murphy, and Barber are Bayesian textbooks wearing ML titles, and the graphical models of Koller & Friedman and Darwiche are Bayes’ theorem drawn as a network — the field even named them Bayesian networks without blushing. 𐃏

Coda#

So: when we round the corner tonight and the porch light catches an orange shape on the red chair — or doesn’t — no long-run frequency will be confirmed or refuted. A belief will be updated. That is all probability has ever been: Laplace’s good sense, Cox’s consistency, Jaynes’ logic of science, reduced to calculus and applied to a cat.

Further Reading#

The books that make the case, in escalating order of commitment: 𐃏

• E.T. Jaynes, Probability Theory: The Logic of Science — the book this post is downstream of. Read the first three chapters and you will not recover.
• L. Wasserman, All of Statistics — both frameworks, honestly labelled.
• G. Casella & R. Berger, Statistical Inference — the classical canon, worth knowing precisely so you know what the special case is.
• A. Gelman, J. Carlin, H. Stern & D. Rubin, Bayesian Data Analysis — the practice.
• D. Gammerman, Markov Chain Monte Carlo — how the integrals actually get done.
• P. Billingsley, Probability and Measure & J.S. Rosenthal, A First Look at Rigorous Probability Theory — Kolmogorov’s foundations, for when you want the measure theory under the philosophy.
• D. Barber, Bayesian Reasoning and Machine Learning; K. Murphy, Machine Learning: A Probabilistic Perspective; C. Bishop, Pattern Recognition and Machine Learning — the surrender documents of machine learning.
• A. Darwiche, Modelling and Reasoning with Bayesian Networks; D. Koller & N. Friedman, Probabilistic Graphical Models — Bayes’ theorem, drawn as a graph.