Measure Theory
This page sits between Real Analysis (metric spaces, completeness, uniform convergence) and Probability (which it puts on a rigorous footing). It condenses my notes for UNSW MATH5825 Measure, Integration and Probability (Term 3, 2026), which follow T. Tao’s An Introduction to Measure Theory for the analysis and then turn to probability.
The spine, in one paragraph: the Riemann integral cannot handle limits, so we rebuild “length” from countable covers (Lebesgue outer measure), keep only the sets on which it is additive (Carathéodory), and integrate by approximating from below with simple functions. Three convergence theorems (monotone, Fatou, dominated) are the payoff. Everything then abstracts to an arbitrary \(\sigma\)-algebra with a measure; products of measure spaces give Fubini and Tonelli; the integrable functions form complete normed spaces \(L^p\). A probability space is a measure space of total mass one, a random variable is a measurable function, and expectation is the integral — so the laws of large numbers, the central limit theorem, conditional expectation and martingales are all theorems about integrals.
Each section opens with a short overview that stays visible; the definitions, theorems, proofs and worked examples are inside the folded subsections.
why measure theory
The Riemann integral is defined by upper and lower sums over finite partitions. That is enough for continuous functions on compact intervals and fails almost everywhere else: it cannot integrate the indicator of the rationals, it ignores unbounded functions, and — the decisive defect — a pointwise limit of Riemann-integrable functions need not be Riemann integrable, so the space of Riemann-integrable functions is not complete under the \(L^1\) distance. Measure theory repairs this by changing what we sum over: not intervals in a partition, but arbitrary measurable sets weighted by their size.
the Riemann integral and its failures
For bounded \(f:[a,b]\to\mathbb{R}\) and a partition \(P: a=x_0<x_1<\dots<x_n=b\), the upper and lower sums are \[ U(f,P)=\sum_{k=1}^n M_k (x_k-x_{k-1}),\qquad L(f,P)=\sum_{k=1}^n m_k(x_k-x_{k-1}), \] with \(M_k=\sup_{[x_{k-1},x_k]}f\) and \(m_k=\inf_{[x_{k-1},x_k]}f\). \(f\) is Riemann integrable when \(\inf_P U(f,P)=\sup_P L(f,P)\), the common value being \(\int_a^b f\). Equivalently (the Riemann criterion), for every \(\varepsilon>0\) some partition has \(U(f,P)-L(f,P)<\varepsilon\).
\(f=\mathbf{1}_{\mathbb{Q}\cap[0,1]}\). Every subinterval of every partition contains rationals and irrationals, so \(U(f,P)=1\) and \(L(f,P)=0\) for all \(P\): \(f\) is not Riemann integrable. Yet \(f\) is zero except on a countable set, and any sensible notion of area should give \(\int f=0\). Note that \(f\) is the pointwise limit of \(f_n=\mathbf{1}_{\{q_1,\dots,q_n\}}\), each of which is Riemann integrable with integral \(0\).
Start from \([0,1]\) and at stage \(n\) remove an open middle interval of length \(4^{-n}\) from each of the \(2^{n-1}\) remaining intervals. The removed length is \(\sum_n 2^{n-1}4^{-n}=\tfrac12\), so what is left, \(K\), is a closed nowhere-dense set that “should” have length \(\tfrac12\). The indicator \(\mathbf{1}_K\) is discontinuous at every point of \(K\) and is not Riemann integrable (its upper sums stay at least \(\tfrac12\), its lower sums are \(0\)). Approximating \(K\) from outside by the finite stages gives a decreasing sequence of step functions, Cauchy in the \(L^1\) seminorm, with no Riemann-integrable limit.
Let \(R[a,b]\) be the Riemann-integrable functions with seminorm \(\|f\|_1=\int_a^b|f|\), and \(N[a,b]\) those with \(\|f\|_1=0\). The normed space \(R[a,b]/N[a,b]\) is not complete.
The fat Cantor example witnesses the theorem. This is the structural reason for measure theory: \(L^1\) is, among other things, the completion of \(R[a,b]/N[a,b]\), in the same way that \(\mathbb{R}\) is the completion of \(\mathbb{Q}\) (see Real Analysis for completions of metric spaces).
One might hope for a “volume” defined on all subsets of \(\mathbb{R}^3\) that is countably additive and invariant under rigid motions. The Banach–Tarski paradox (using the axiom of choice) splits a ball into finitely many pieces and reassembles them by rotations and translations into two balls of the same size, so no such volume exists. We must give up measuring some sets; the work is to keep enough of them. A one-dimensional version (Vitali’s set) is proved below.
Ten flips of a fair coin live on a finite sample space of \(2^{10}\) outcomes and need no measure theory. Infinitely many flips do: the sample space \(\{H,T\}^{\mathbb{N}}\) is uncountable, every single outcome has probability \(0\), and questions such as “does the proportion of heads tend to \(\tfrac12\)?” concern events that are not finite combinations of finitely-described ones. Writing \(\omega\in[0,1)\) in binary turns the infinite coin into Lebesgue measure on \([0,1)\); this example recurs throughout.
Lebesgue measure
Jordan measure, the measure implicit in the Riemann integral, approximates a set from inside and outside by finite unions of boxes. Lebesgue’s step is to allow countable covers from outside. That gives an outer measure on every subset of \(\mathbb{R}^d\) which is countably subadditive but not additive on all sets; restricting to the sets that split every other set additively (Carathéodory’s criterion) gives a \(\sigma\)-algebra on which it is countably additive. That restriction is Lebesgue measure.
Jordan measure
A box in \(\mathbb{R}^d\) is a product \(B=I_1\times\dots\times I_d\) of bounded intervals, with volume \(|B|=|I_1|\cdots|I_d|\). An elementary set is a finite union of boxes. Every elementary set is a finite disjoint union of boxes, and its elementary measure \(\operatorname{em}(E)=\sum|B_k|\) does not depend on the decomposition chosen.
For bounded \(E\subseteq\mathbb{R}^d\), \[ m_{*,J}(E)=\sup_{A\subseteq E\text{ elementary}}\operatorname{em}(A),\qquad m^{*,J}(E)=\inf_{B\supseteq E\text{ elementary}}\operatorname{em}(B). \] \(E\) is Jordan measurable when the two agree; the common value is its Jordan measure. Equivalently, for every \(\varepsilon>0\) there are elementary \(A\subseteq E\subseteq B\) with \(\operatorname{em}(B\setminus A)<\varepsilon\).
Jordan measure is finitely additive, monotone and translation invariant, and triangles, discs and other regions bounded by finitely many smooth curves are Jordan measurable.
A bounded \(f:[a,b]\to[0,\infty)\) is Riemann integrable iff the region \(\{(x,t):x\in[a,b],\ 0\le t\le f(x)\}\subseteq\mathbb{R}^2\) is Jordan measurable, and then \(\int_a^bf\) is its Jordan measure.
\(E=\mathbb{Q}\cap[0,1]\) is not Jordan measurable. An elementary set inside \(E\) can contain no interval of positive length, so \(m_{*,J}(E)=0\). An elementary set \(B\supseteq E\) is a finite union of intervals whose closure contains the closure of \(E\), namely \([0,1]\); removing finitely many endpoints does not change measure, so \(\operatorname{em}(B)\ge1\) and \(m^{*,J}(E)=1\).
Lebesgue outer measure
For any \(E\subseteq\mathbb{R}^d\), \[ m^*(E)=\inf\Bigl\{\sum_{n=1}^\infty|B_n|:\ B_1,B_2,\dots\text{ boxes},\ E\subseteq\bigcup_{n=1}^\infty B_n\Bigr\}\in[0,\infty]. \]
- \(m^*(\emptyset)=0\);
- (monotonicity) \(E\subseteq F\Rightarrow m^*(E)\le m^*(F)\);
- (countable subadditivity) \(m^*\bigl(\bigcup_n E_n\bigr)\le\sum_n m^*(E_n)\);
- (translation invariance) \(m^*(E+x)=m^*(E)\);
- for every box, and more generally every elementary set, \(m^*(E)=\operatorname{em}(E)\).
(1), (2), (4) are immediate from the definition. For (3), fix \(\varepsilon>0\) and cover each \(E_n\) by boxes \(B_{n,k}\) with \(\sum_k|B_{n,k}|\le m^*(E_n)+\varepsilon 2^{-n}\). The doubly indexed family covers the union, so \(m^*(\bigcup E_n)\le\sum_n m^*(E_n)+\varepsilon\). For (5), \(\le\) is clear (cover by the box itself). For \(\ge\), enlarge each covering box \(B_n\) to an open box of volume at most \(|B_n|+\varepsilon2^{-n}\) and shrink the target to a closed box of volume at least \(|B|-\varepsilon\); compactness (Heine–Borel) extracts a finite subcover, and finite additivity of \(\operatorname{em}\) then gives \(\sum|B_n|\ge|B|-2\varepsilon\).
Countable sets are null: \(m^*(\{q_1,q_2,\dots\})\le\sum_n\varepsilon2^{-n}=\varepsilon\). So \(m^*(\mathbb{Q}\cap[0,1])=0\) while its outer Jordan measure is \(1\) — the countable cover is exactly what Jordan measure lacks. In general \(m_{*,J}(E)\le m^*(E)\le m^{*,J}(E)\).
\(m^*\) is defined on every set, but it is not additive: there are disjoint \(A,B\subseteq\mathbb{R}\) with \(m^*(A\cup B)<m^*(A)+m^*(B)\) (take a Vitali set and a translate). So we restrict it.
measurable sets and Carathéodory’s criterion
\(E\subseteq\mathbb{R}^d\) is Lebesgue measurable if it splits every set additively: \[ m^*(A)=m^*(A\cap E)+m^*(A\cap E^c)\qquad\text{for every }A\subseteq\mathbb{R}^d. \] By subadditivity only \(\ge\) needs checking, and only for \(A\) with \(m^*(A)<\infty\).
A collection \(\mathcal{A}\subseteq\mathcal{P}(X)\) is a \(\sigma\)-algebra if \(\emptyset\in\mathcal{A}\), it is closed under complements, and it is closed under countable unions. (Hence also under countable intersections, differences and \(\limsup/\liminf\) of sequences of sets.)
The Lebesgue measurable sets \(\mathcal{L}\) form a \(\sigma\)-algebra, and \(m^*\) restricted to \(\mathcal{L}\) is countably additive: if \(E_1,E_2,\dots\in\mathcal{L}\) are disjoint then \(m^*(\bigcup E_n)=\sum m^*(E_n)\).
Complements are immediate from the symmetry of the criterion. For a finite union of two measurable sets \(E,F\), split \(A\) first by \(E\) and then split \(A\cap E^c\) by \(F\): \[ m^*(A)=m^*(A\cap E)+m^*(A\cap E^c\cap F)+m^*(A\cap E^c\cap F^c)\ge m^*(A\cap(E\cup F))+m^*(A\cap(E\cup F)^c), \] using subadditivity on the first two terms. So \(\mathcal{L}\) is an algebra. Now let \(E_n\) be disjoint and measurable, \(S_N=\bigcup_{n\le N}E_n\), \(S=\bigcup_n E_n\). Splitting \(A\cap S_N\) by \(E_N\) repeatedly gives \(m^*(A\cap S_N)=\sum_{n\le N}m^*(A\cap E_n)\). Hence \[ m^*(A)=m^*(A\cap S_N)+m^*(A\cap S_N^c)\ge\sum_{n\le N}m^*(A\cap E_n)+m^*(A\cap S^c). \] Let \(N\to\infty\) and use subadditivity once more: \(m^*(A)\ge\sum_n m^*(A\cap E_n)+m^*(A\cap S^c)\ge m^*(A\cap S)+m^*(A\cap S^c)\). So \(S\in\mathcal{L}\), and with \(A=S\) all the inequalities are equalities, which is countable additivity. A general countable union is a disjoint union of differences \(E_n\setminus(E_1\cup\dots\cup E_{n-1})\).
The proof uses nothing about boxes, which is why it reappears verbatim for abstract outer measures below.
- Every null set (\(m^*(E)=0\)) is measurable, as is every subset of a null set.
- Half-spaces \(\{x:x_i>a\}\) are measurable; hence (as countable unions of boxes) open sets are measurable, and so are closed sets and all Borel sets.
- Every Jordan measurable set is Lebesgue measurable, with the same measure.
For every \(E\subseteq\mathbb{R}^d\), \(m^*(E)=\inf\{m^*(U):U\supseteq E\text{ open}\}\). If \(E\) is measurable and \(\varepsilon>0\), there is an open \(U\supseteq E\) with \(m^*(U\setminus E)<\varepsilon\) and a closed \(F\subseteq E\) with \(m^*(E\setminus F)<\varepsilon\).
Tao takes this approximation property as the definition of measurability and derives the Carathéodory criterion; the course goes the other way. They define the same \(\sigma\)-algebra.
Lebesgue measure and its examples
\(m=m^*|_{\mathcal{L}}:\mathcal{L}\to[0,\infty]\) is Lebesgue measure on \(\mathbb{R}^d\).
- (from below) If \(E_1\subseteq E_2\subseteq\cdots\) are measurable, then \(m(\bigcup E_n)=\lim m(E_n)\).
- (from above) If \(E_1\supseteq E_2\supseteq\cdots\) are measurable and \(m(E_1)<\infty\), then \(m(\bigcap E_n)=\lim m(E_n)\).
(1) Write \(\bigcup E_n\) as the disjoint union of \(E_1\) and the rings \(E_{n+1}\setminus E_n\) and apply countable additivity; the partial sums are \(m(E_n)\). (2) Apply (1) to \(E_1\setminus E_n\), and subtract from the finite number \(m(E_1)\).
Finiteness in (2) matters: \(E_n=[n,\infty)\) has \(m(E_n)=\infty\) for all \(n\) and empty intersection.
The middle-thirds Cantor set \(C\) is what remains after removing \(2^{n-1}\) open intervals of length \(3^{-n}\) at stage \(n\). The removed length is \(\sum_n 2^{n-1}3^{-n}=1\), so \(m( C)=0\) by continuity from above (stage \(n\) has measure \((2/3)^n\)). Yet \(C\) is uncountable: it is the set of numbers with a ternary expansion using only the digits \(0\) and \(2\), which maps onto \([0,1]\) by halving the digits and reading in binary. The fat Cantor set \(K\) above is closed, nowhere dense, and has \(m(K)=\tfrac12\). Size in measure, in cardinality, and in topology are three different things.
The infinite strip \(\mathbb{R}\times[0,1]\) is not Jordan measurable (it is unbounded) but is Lebesgue measurable with measure \(\infty\); the region under \(x\mapsto x^{-1/2}\) on \((0,1]\) has Lebesgue measure \(2\) though it is unbounded, and its boundary graph may even be discontinuous without affecting measurability.
a non-measurable set
There is a subset of \([0,1]\) that is not Lebesgue measurable.
Call \(x\sim y\) when \(x-y\in\mathbb{Q}\). By the axiom of choice pick one representative in \([0,1]\) from each equivalence class; call the set \(V\). Enumerate \(\mathbb{Q}\cap[-1,1]=\{q_1,q_2,\dots\}\). The translates \(V+q_n\) are pairwise disjoint (two points of \(V\) differing by a rational would be in the same class) and \[ [0,1]\subseteq\bigcup_n(V+q_n)\subseteq[-1,2]. \] If \(V\) were measurable, so would each \(V+q_n\) be, all with the same measure \(m(V)\), and countable additivity would give \(1\le\sum_n m(V)\le 3\). No value of \(m(V)\) satisfies this: \(0\) makes the sum \(0\) and anything positive makes it \(\infty\).
The argument adapts to show that every set of positive measure contains a non-measurable subset. The axiom of choice is essential: there are models of set theory without full choice in which every subset of \(\mathbb{R}\) is Lebesgue measurable.
the Lebesgue integral
Integrate an unsigned function by approximating it from below by simple functions — finite combinations of indicators of measurable sets — and taking the supremum of their obvious integrals. The functions for which this behaves well are the measurable ones, those whose superlevel sets \(\{f>a\}\) are measurable. Where Riemann partitions the domain, Lebesgue partitions the range. The pay-off is three theorems about interchanging limits and integrals.
simple and measurable functions
Unsigned functions take values in \([0,\infty]\), with \(x+\infty=\infty\), \(x\cdot\infty=\infty\) for \(x\ne0\), and the convention \(0\cdot\infty=0\) (so that the integral of \(\infty\) over a null set is \(0\)). In \([0,\infty]\) every set has a supremum and every monotone sequence converges.
A simple function is \(g=\sum_{i=1}^n c_i\mathbf{1}_{E_i}\) with \(E_i\) measurable. For unsigned \(g\) define \[ \operatorname{Simp}\int g=\sum_{i=1}^n c_i\, m(E_i). \] This is independent of the representation (refine to a common disjoint partition), monotone and linear over non-negative scalars.
\(f:\mathbb{R}^d\to[0,\infty]\) is measurable if \(\{x:f(x)>a\}\) is measurable for every \(a\in\mathbb{R}\). Equivalently \(f^{-1}(I)\) is measurable for every interval \(I\), or for every Borel set.
Continuous functions and indicators of measurable sets are measurable. Sums, products, maxima and minima of measurable functions are measurable, and if each \(f_n\) is measurable then so are \(\sup_n f_n\), \(\inf_n f_n\), \(\limsup f_n\), \(\liminf f_n\) and the pointwise limit when it exists.
\(\{\sup_n f_n>a\}=\bigcup_n\{f_n>a\}\) and \(\{\inf_n f_n\ge a\}=\bigcap_n\{f_n\ge a\}\); then \(\limsup f_n=\inf_n\sup_{k\ge n}f_k\). Closure under countable operations is exactly what a \(\sigma\)-algebra provides, which is why the Riemann-integrable functions lack it.
An unsigned \(f\) is measurable iff it is the pointwise limit of unsigned simple functions, and the approximants can be chosen increasing: \(0\le g_1\le g_2\le\cdots\uparrow f\).
Take \[ g_n=\min\Bigl(n,\ 2^{-n}\lfloor 2^n f\rfloor\Bigr)\cdot\mathbf{1}_{[-n,n]^d}, \] i.e. round \(f\) down to the grid \(2^{-n}\mathbb{Z}\), cap at height \(n\) and restrict to a bounded box. Each level set of \(g_n\) is a preimage of an interval under \(f\), the grids refine so \(g_n\le g_{n+1}\), and \(g_n(x)\to f(x)\) (to within \(2^{-n}\) once \(n>f(x)\), and \(g_n(x)=n\to\infty\) when \(f(x)=\infty\)). The converse is the closure property for limits.
the unsigned integral
For unsigned measurable \(f\), \[ \int_{\mathbb{R}^d}f=\sup\Bigl\{\operatorname{Simp}\int g:\ g\text{ unsigned simple},\ g\le f\Bigr\}\in[0,\infty]. \]
For unsigned measurable \(f,g\):
- (compatibility) for simple \(f\) this agrees with \(\operatorname{Simp}\int f\); in particular \(\int\mathbf{1}_E=m(E)\);
- (monotonicity) \(f\le g\Rightarrow\int f\le\int g\);
- (linearity) \(\int(f+cg)=\int f+c\int g\) for \(c\ge0\);
- (Markov) \(m(\{f\ge t\})\le\frac1t\int f\) for \(t>0\);
- \(\int f=0\) iff \(f=0\) almost everywhere, and \(\int f<\infty\) implies \(f<\infty\) a.e.
(1) and (2) are from the definition. (4): \(t\mathbf{1}_{\{f\ge t\}}\le f\). (5): if \(\int f=0\) then \(m(\{f\ge1/n\})=0\) for every \(n\) by (4), and \(\{f>0\}\) is their countable union. Additivity in (3) is the one non-trivial claim; superadditivity is easy from the definition, and full additivity follows most cleanly from the monotone convergence theorem below applied to increasing simple approximants of \(f\) and \(g\).
A property holds almost everywhere (a.e.) if the set where it fails is null. Integrals do not see null sets: if \(f=g\) a.e. then \(\int f=\int g\).
\(\mathbf{1}_{\mathbb{Q}}\) is simple (an indicator of a measurable null set), so \(\int\mathbf{1}_{\mathbb{Q}}=m(\mathbb{Q})=0\). Likewise \(\int\mathbf{1}_K=\tfrac12\) for the fat Cantor set. Both functions defeated Riemann.
monotone convergence and Fatou
Limits and integrals do not commute in general. Three ways mass can escape, all on \(\mathbb{R}\) with every \(f_n\to0\) pointwise but \(\int f_n=1\):
- escape to horizontal infinity: \(f_n=\mathbf{1}_{[n,n+1]}\);
- escape to width infinity: \(f_n=\tfrac1n\mathbf{1}_{[0,n]}\);
- escape to vertical infinity: \(f_n=n\mathbf{1}_{(0,1/n)}\).
The theorems below say that for unsigned functions, mass can only be lost in the limit, never created, and that it cannot be lost at all if the sequence increases or is dominated.
If \(0\le f_1\le f_2\le\cdots\) are measurable and \(f=\sup_n f_n\), then \(f\) is measurable and \[ \lim_{n\to\infty}\int f_n=\int f. \]
Monotonicity gives \(\int f_n\uparrow L\le\int f\). For the reverse, take any simple \(g=\sum c_i\mathbf{1}_{E_i}\le f\) and any \(t\in(0,1)\). The sets \(A_n=\{f_n\ge tg\}\) are measurable and increase to all of \(\mathbb{R}^d\) (where \(g(x)>0\), eventually \(f_n(x)>tg(x)\) because \(f(x)\ge g(x)>tg(x)\)). Then \[ \int f_n\ge\int tg\,\mathbf{1}_{A_n}=t\sum_i c_i\,m(E_i\cap A_n)\longrightarrow t\sum_ic_i\,m(E_i)=t\operatorname{Simp}\int g \] by continuity from below. So \(L\ge t\operatorname{Simp}\int g\); let \(t\uparrow1\) and take the supremum over \(g\).
For unsigned measurable \(f_n\), \(\int\sum_n f_n=\sum_n\int f_n\) (Tonelli for sums). The integral can therefore be computed as \(\int f=\lim\int g_n\) for any increasing simple \(g_n\uparrow f\).
For unsigned measurable \(f_n\), \[ \int\liminf_{n\to\infty}f_n\le\liminf_{n\to\infty}\int f_n. \]
Put \(g_n=\inf_{k\ge n}f_k\). Then \(g_n\uparrow\liminf f_n\) and \(g_n\le f_k\) for all \(k\ge n\), so \(\int g_n\le\inf_{k\ge n}\int f_k\). Apply monotone convergence to \(g_n\) and take limits.
All three escaping examples show the inequality can be strict. Monotone convergence also needs the increase: \(f_n=\mathbf{1}_{[n,\infty)}\) decreases to \(0\) with \(\int f_n=\infty\). For decreasing sequences one needs \(\int f_1<\infty\) (continuity from above, in integral form).
integrable functions and dominated convergence
A real measurable \(f\) is (absolutely) integrable if \(\int|f|<\infty\); then with \(f_+=\max(f,0)\), \(f_-=\max(-f,0)\) define \(\int f=\int f_+-\int f_-\). A complex \(f=u+iv\) is integrable if \(u,v\) are, with \(\int f=\int u+i\int v\). The integral is linear on integrable functions and \(|\int f|\le\int|f|\).
There is no conditionally convergent Lebesgue integral: \(\sin x/x\) is not Lebesgue integrable on \((0,\infty)\), although its improper Riemann integral exists. Riemann-integrable functions on \([a,b]\) are Lebesgue integrable with the same integral, so nothing is lost on compact intervals.
Let \(f_n\) be measurable with \(f_n\to f\) pointwise (or a.e.), and suppose \(|f_n|\le g\) for all \(n\) with \(g\) integrable. Then \(f\) is integrable and \[ \int|f_n-f|\to0,\qquad\text{in particular}\qquad\int f_n\to\int f. \]
\(|f|\le g\), so \(f\) is integrable. The functions \(2g-|f_n-f|\) are unsigned and converge to \(2g\). By Fatou, \[ \int2g\le\liminf\int(2g-|f_n-f|)=\int2g-\limsup\int|f_n-f|. \] As \(\int2g<\infty\) it can be cancelled, leaving \(\limsup\int|f_n-f|\le0\).
In each escaping example the smallest possible dominating function is \(\sup_nf_n\), which is not integrable (\(\sum_n\mathbf{1}_{[n,n+1]}\), \(\sup_n\frac1n\mathbf{1}_{[0,n]}\approx\frac1x\), \(\sup_n n\mathbf{1}_{(0,1/n)}\approx\frac1x\)). Domination is the condition that stops mass from escaping.
\(\displaystyle\lim_{n\to\infty}\int_0^\infty\frac{n\sin(x/n)}{x(1+x^2)}\,dx\). The integrand converges to \(\frac1{1+x^2}\) pointwise, and \(|n\sin(x/n)|\le x\) gives the domination \(\frac{1}{1+x^2}\), which is integrable. So the limit is \(\int_0^\infty\frac{dx}{1+x^2}=\frac\pi2\).
Littlewood’s three principles
Littlewood’s heuristic: every measurable set is nearly a finite union of intervals, every measurable function is nearly continuous, every convergent sequence of measurable functions is nearly uniformly convergent. The first is outer regularity; the other two are made precise below.
For integrable \(f\) on \(\mathbb{R}^d\) and \(\varepsilon>0\) there is a continuous compactly supported \(\varphi\) (or a step function) with \(\int|f-\varphi|<\varepsilon\).
If \(f:\mathbb{R}^d\to\mathbb{R}\) is integrable and \(\varepsilon>0\), there is a measurable \(E\) with \(m(E)<\varepsilon\) such that \(f\) restricted to \(\mathbb{R}^d\setminus E\) is continuous.
Choose continuous compactly supported \(\varphi_n\) with \(\int|f-\varphi_n|\le\varepsilon4^{-n}\). By Markov the set \(A_n=\{|f-\varphi_n|>2^{-n}\}\) has measure at most \(\varepsilon2^{-n}\). Off \(A=\bigcup_nA_n\) (measure at most \(\varepsilon\)), \(\varphi_n\to f\) uniformly, so \(f|_{A^c}\) is continuous. Enlarging \(A\) to an open set of measure \(<2\varepsilon\) keeps the complement closed.
The restriction is continuous; \(f\) itself need not be continuous at any point (the Dirichlet function: remove \(\mathbb{Q}\)).
If measurable \(f_n\to f\) pointwise on \(\mathbb{R}^d\) and \(\varepsilon>0\), there is a measurable \(E\) with \(m(E)<\varepsilon\) such that \(f_n\to f\) uniformly on every bounded subset of \(\mathbb{R}^d\setminus E\).
Work on a bounded set \(B\) first. For fixed \(k\) the sets \(E_{N,k}=\{x\in B:|f_n(x)-f(x)|\ge1/k\text{ for some }n\ge N\}\) decrease to \(\emptyset\) as \(N\to\infty\), and \(m(B)<\infty\), so continuity from above gives \(N_k\) with \(m(E_{N_k,k})<\varepsilon2^{-k}\). Off \(E=\bigcup_kE_{N_k,k}\), \(|f_n-f|<1/k\) for all \(n\ge N_k\): uniform convergence. Exhaust \(\mathbb{R}^d\) by countably many bounded pieces with budgets \(\varepsilon2^{-j}\).
“Bounded” cannot be dropped: \(\mathbf{1}_{[n,n+1]}\to0\) pointwise but not uniformly off any set of finite measure.
differentiation and integration
The fundamental theorem of calculus has two halves — differentiate an integral, integrate a derivative — and both break for Riemann integration. Lebesgue’s versions: an indefinite integral of an integrable function is differentiable almost everywhere with the right derivative, and the functions that are indefinite integrals are exactly the absolutely continuous ones. The Cantor staircase shows that continuity and monotonicity are not enough.
what goes wrong with the classical FTC
- If \(f\) is continuous on \([a,b]\) then \(g(x)=\int_a^xf\) is differentiable with \(g’=f\).
- If \(f\) is Riemann integrable and \(g’=f\) everywhere on \([a,b]\), then \(\int_a^bf=g(b)-g(a)\).
\(g(x)=x^2\sin(1/x)\) (with \(g(0)=0\)) is differentiable everywhere with \(g’(x)=2x\sin(1/x)-\cos(1/x)\) for \(x\ne0\) and \(g’(0)=0\). The derivative is bounded with one discontinuity, so it is Riemann integrable and part 2 applies.
\(g(x)=x^2\sin(1/x^2)\) is also differentiable everywhere, but \(g’(x)=2x\sin(1/x^2)-\frac2x\cos(1/x^2)\) is unbounded near \(0\); it is not Riemann integrable, not even Lebesgue integrable (\(\int|g’|=\infty\) near \(0\)). A function can be differentiable everywhere without being the integral of its derivative in any sense we have.
\(f=\mathbf{1}_{[0,1]}\) is Riemann integrable but has no antiderivative (derivatives have the intermediate value property, Darboux). Its integral \(g(x)=\int_0^xf\) is differentiable with \(g’=f\) everywhere except at the two jump points — a preview of “almost everywhere”.
the Lebesgue differentiation theorem
If \(f:\mathbb{R}\to\mathbb{R}\) is integrable, then \(g(x)=\int_a^xf\) is continuous, differentiable almost everywhere, and \(g’=f\) almost everywhere. Equivalently, for a.e. \(x\), \[ \lim_{h\to0}\frac1h\int_x^{x+h}f(t)\,dt=f(x). \]
(Outline.) Continuity of \(g\) follows from dominated convergence. For continuous compactly supported \(f\) the statement is the classical FTC. In general, approximate \(f\) in \(L^1\) by such a \(\varphi\) with \(\int|f-\varphi|<\varepsilon\); then for every \(x\) \[ \limsup_{h\to0}\Bigl|\frac1h\int_x^{x+h}f-f(x)\Bigr|\le M|h|(x)+|h(x)|,\qquad h=f-\varphi, \] where \(M\) is the Hardy–Littlewood maximal function. The one-sided maximal inequality \(m(\{M|h|>\lambda\})\le\frac{C}{\lambda}\int|h|\), proved with Riesz’s rising sun lemma, and Markov’s inequality for \(|f-\varphi|\), show that the set where the left side exceeds \(\lambda\) has measure \(O(\varepsilon/\lambda)\). Let \(\varepsilon\to0\), then \(\lambda\to0\) through a sequence.
Let \(F:[a,b]\to\mathbb{R}\) be continuous and \(U=\{x\in(a,b):F(y)>F(x)\text{ for some }y>x\}\) (the points in shadow when the sun rises on the right). Then \(U\) is open, hence a countable disjoint union of intervals \((a_n,b_n)\), and \(F(a_n)\le F(b_n)\) on each, with equality unless \(a_n=a\).
\(g(x)=\int_0^x\mathbf{1}_K\) is continuous and increasing with \(g(1)=\tfrac12\). By the theorem \(g’=\mathbf{1}_K\) a.e.: the slope is \(1\) at almost every point of a set with empty interior.
bounded variation and absolute continuity
\(F:[a,b]\to\mathbb{R}\) has bounded variation if \(\sup_P\sum_k|F(x_k)-F(x_{k-1})|<\infty\) over partitions. \(F\) is absolutely continuous if for every \(\varepsilon>0\) there is \(\delta>0\) such that \(\sum_k|F(b_k)-F(a_k)|<\varepsilon\) for every finite family of disjoint intervals \((a_k,b_k)\) with \(\sum_k(b_k-a_k)<\delta\).
On \([a,b]\): continuously differentiable \(\Rightarrow\) Lipschitz \(\Rightarrow\) absolutely continuous \(\Rightarrow\) (uniformly continuous and of bounded variation). Every function of bounded variation is the difference of two monotone functions, and every monotone function is differentiable almost everywhere. Each implication is strict.
\(g\) on \([a,b]\) has the form \(g(x)=\int_a^xf+C\) with \(f\) integrable iff \(g\) is absolutely continuous; then \(f=g’\) a.e. Consequently, if \(g\) is absolutely continuous, \(\int_a^bg’=g(b)-g(a)\).
(Forward direction.) Absolute continuity of the integral: given \(\varepsilon\), dominated convergence applied to \(|f|\mathbf{1}_{\{|f|>N\}}\) finds \(N\) with \(\int_{\{|f|>N\}}|f|<\varepsilon/2\); then any union \(E\) of intervals with \(m(E)<\varepsilon/(2N)\) has \(\int_E|f|<\varepsilon\). (Converse, sketched.) An absolutely continuous \(g\) is of bounded variation, so \(g’\) exists a.e. and is integrable; \(h=g-\int_a^xg’\) is absolutely continuous with \(h’=0\) a.e., and the Vitali covering lemma shows such a function is constant.
Define \(\varphi\) on \([0,1]\) by \(\varphi=\tfrac12\) on the first removed middle third, \(\tfrac14,\tfrac34\) on the next two, and so on, extended by continuity. Then \(\varphi\) is continuous, increasing, \(\varphi(0)=0\), \(\varphi(1)=1\), and \(\varphi’=0\) on the complement of the Cantor set, i.e. almost everywhere. So \[ \int_0^1\varphi’=0\ne1=\varphi(1)-\varphi(0). \] \(\varphi\) is not absolutely continuous: the stage-\(n\) cover of \(C\) has total length \((2/3)^n\to0\) yet \(\varphi\) rises by \(1\) across it. All of its increase happens on a null set.
when is a function Riemann integrable?
A bounded \(f:[a,b]\to\mathbb{R}\) is Riemann integrable iff its set of discontinuities has Lebesgue measure zero.
(Idea.) Let \(\omega_f(x)\) be the oscillation of \(f\) at \(x\), so \(D=\{\omega_f>0\}=\bigcup_k\{\omega_f\ge1/k\}\), each piece closed. \(U(f,P)-L(f,P)\) is at least \(\frac1k\) times the length of the partition intervals meeting \(\{\omega_f\ge 1/k\}\) in their interior, and at most that length times \(2\sup|f|\) plus \(\frac1k(b-a)\). If \(m(D)=0\) each compact piece is covered by finitely many intervals of small total length; if \(m(D)>0\) some piece has positive measure and the gap \(U-L\) cannot be made small.
\(\mathbf{1}_C\) is discontinuous exactly on \(C\), a null set, so it is Riemann integrable (with integral \(0\)). \(\mathbf{1}_K\) for the fat Cantor set is discontinuous on \(K\), of measure \(\tfrac12\), so it is not.
abstract measure spaces
Nothing in the construction of the integral used the geometry of \(\mathbb{R}^d\) once Lebesgue measure was in hand; only countable additivity on a \(\sigma\)-algebra. So we abstract: a measurable space \((X,\mathcal{B})\) and a measure \(\mu\) on it. Counting measure turns integrals into sums, a probability measure turns them into expectations, and the convergence theorems hold verbatim. Two new questions arise: how measures compare (absolute continuity, Radon–Nikodym), and how to build them (Carathéodory again, and the Hahn–Kolmogorov extension).
sigma-algebras and Borel sets
A measurable space is a pair \((X,\mathcal{B})\) with \(\mathcal{B}\) a \(\sigma\)-algebra on \(X\). The trivial \(\{\emptyset,X\}\) and discrete \(\mathcal{P}(X)\) \(\sigma\)-algebras are the extremes; the Lebesgue \(\sigma\)-algebra \(\mathcal{L}\) is in between.
Any intersection of \(\sigma\)-algebras on \(X\) is a \(\sigma\)-algebra. So for any \(\mathcal{S}\subseteq\mathcal{P}(X)\) there is a smallest \(\sigma\)-algebra \(\sigma(\mathcal{S})\) containing \(\mathcal{S}\): the intersection of all of them.
For a topological (or metric) space \(X\), \(\mathcal{B}(X)=\sigma(\text{open sets})\). On \(\mathbb{R}\) it is also generated by the intervals \((a,b)\), by the rays \((-\infty,a]\), or by the rays with \(a\) rational.
If \(f:X\to Y\) is continuous then \(f^{-1}(B)\in\mathcal{B}(X)\) for every \(B\in\mathcal{B}(Y)\).
The “good sets” \(\{B\subseteq Y:f^{-1}(B)\in\mathcal{B}(X)\}\) form a \(\sigma\)-algebra (preimages commute with complements and unions) containing the open sets, hence containing \(\mathcal{B}(Y)\).
This good-sets argument is the standard way to prove anything about all sets in a generated \(\sigma\)-algebra: show the property holds for generators and is preserved by the \(\sigma\)-algebra operations.
On \(\Omega=[0,1)\) with binary digits \(d_1(\omega),d_2(\omega),\dots\), let \(\mathcal{F}_n\) be generated by the dyadic intervals \([k2^{-n},(k+1)2^{-n})\): the events decided by the first \(n\) flips. Then \(\mathcal{F}_1\subseteq\mathcal{F}_2\subseteq\cdots\) and \(\sigma(\bigcup_n\mathcal{F}_n)=\mathcal{B}([0,1))\). The increasing family is the prototype of a filtration.
measures
A measure on \((X,\mathcal{B})\) is \(\mu:\mathcal{B}\to[0,\infty]\) with \(\mu(\emptyset)=0\) and countable additivity over disjoint sequences. \((X,\mathcal{B},\mu)\) is a measure space; a probability space if \(\mu(X)=1\).
Every measure is monotone, countably subadditive, continuous from below, and continuous from above on sets of finite measure (same proofs as for Lebesgue measure).
- Counting measure \(\#\) on \((X,\mathcal{P}(X))\): \(\#(E)\) is the number of elements.
- Dirac measure \(\delta_x(E)=\mathbf{1}_E(x)\).
- Borel measure \(m|_{\mathcal{B}(\mathbb{R}^d)}\): Lebesgue measure on the smaller \(\sigma\)-algebra.
- Poisson measure on \(\mathbb{N}_0\): \(\mu(E)=\sum_{k\in E}e^{-\lambda}\lambda^k/k!\), a weighted counting measure.
- Standard Gaussian measure on \(\mathbb{R}\): \(\gamma(E)=\int_E\frac{1}{\sqrt{2\pi}}e^{-x^2/2}\,dx\).
integration on a measure space
\(f:X\to[0,\infty]\) or \(\mathbb{R}\) is measurable if \(f^{-1}(B)\in\mathcal{B}\) for every Borel \(B\) (equivalently, for every ray \((a,\infty)\)). The integral \(\int_Xf\,d\mu\) is defined exactly as for Lebesgue measure: simple functions, supremum from below, positive and negative parts.
Fatou’s lemma, the monotone convergence theorem and the dominated convergence theorem hold on every measure space, with the same statements and proofs.
On \((\mathbb{N},\mathcal{P}(\mathbb{N}),\#)\) a function is a sequence \(a_n\) and \(\int a\,d\#=\sum_na_n\). Integrable means absolutely summable. Monotone convergence becomes “increasing limits of non-negative series commute with summation”, and dominated convergence becomes Tannery’s theorem.
For measurable \(f:X\to\mathbb{R}\), the pushforward \(f_*\mu(B)=\mu(f^{-1}(B))\) is a measure on \((\mathbb{R},\mathcal{B}(\mathbb{R}))\).
For Borel \(\varphi:\mathbb{R}\to[0,\infty]\), \(\displaystyle\int_{\mathbb{R}}\varphi\,d(f_*\mu)=\int_X\varphi\circ f\,d\mu\). For real or complex \(\varphi\), one side is integrable iff the other is, and then they are equal.
For \(\varphi=\mathbf{1}_B\) it is the definition of \(f_*\mu\). Linearity gives simple \(\varphi\), monotone convergence gives unsigned \(\varphi\), and splitting into positive and negative parts gives the rest. (This four-step bootstrap — indicators, simple, unsigned, integrable — is how most identities between integrals are proved.)
In probability this is the law of the unconscious statistician: \(E(\varphi(X))=\int\varphi\,dP_X\).
For \(E\in\mathcal{B}\), \(\int_Ef\,d\mu=\int_Xf\mathbf{1}_E\,d\mu\), which equals the integral of \(f|_E\) against the restricted measure on \((E,\mathcal{B}|_E)\).
comparing measures: Radon–Nikodym
If \(f\ge0\) is measurable on \((X,\mathcal{B},\mu)\), then \(\nu(E)=\int_Ef\,d\mu\) is a measure, and \(\int g\,d\nu=\int gf\,d\mu\) for unsigned measurable \(g\).
- \(\mu\) is \(\sigma\)/-finite/ if \(X\) is a countable union of sets of finite measure (Lebesgue measure: yes; counting measure on \(\mathbb{R}\): no).
- \(\nu\ll\mu\) (\(\nu\) is absolutely continuous with respect to \(\mu\)) if \(\mu(E)=0\Rightarrow\nu(E)=0\).
- \(\nu\perp\mu\) (mutually singular) if \(X=A\sqcup B\) with \(\mu(A)=0\) and \(\nu(B)=0\).
Lebesgue measure and counting measure on \(\mathbb{N}\subseteq\mathbb{R}\) are mutually singular (split \(\mathbb{R}=\mathbb{N}\sqcup\mathbb{N}^c\)); so are Lebesgue measure and the Cantor measure \(d\varphi\) carried by the Cantor set. Any measure with a density, \(\nu(E)=\int_Ef\,dm\), is absolutely continuous with respect to \(m\).
If \(\mu_1,\mu_2\) are \(\sigma\)-finite measures on \((X,\mathcal{B})\), there are unique measures with \(\mu_2=\mu_{ac}+\mu_s\), \(\mu_{ac}\ll\mu_1\), \(\mu_s\perp\mu_1\).
If \(\mu_1,\mu_2\) are \(\sigma\)-finite and \(\mu_2\ll\mu_1\), there is an unsigned measurable \(f\), unique \(\mu_1\)-a.e., with \[ \mu_2(E)=\int_Ef\,d\mu_1\qquad\text{for all }E\in\mathcal{B}. \] We write \(f=\frac{d\mu_2}{d\mu_1}\), the Radon–Nikodym derivative.
(Outline, finite measures.) Consider the densities \(\mathcal{F}=\{f\ge0:\int_Ef\,d\mu_1\le\mu_2(E)\ \forall E\}\). It is closed under maxima, so a maximising sequence for \(\int f\,d\mu_1\) can be taken increasing; monotone convergence gives a maximiser \(f\). If \(\nu=\mu_2-\int f\,d\mu_1\) were not zero, a positive-set lemma (a Hahn-decomposition argument) finds \(\varepsilon>0\) and \(A\) with \(\mu_1(A)>0\) on which \(\nu\ge\varepsilon\mu_1\), using \(\mu_2\ll\mu_1\); then \(f+\varepsilon\mathbf{1}_A\in\mathcal{F}\) contradicts maximality. Uniqueness: if two densities differ, integrate over the set where one exceeds the other. \(\sigma\)-finiteness reduces the general case to countably many finite pieces.
\(\sigma\)-finiteness is needed: counting measure \(\#\) on \([0,1]\) satisfies \(m\ll\#\) (only \(\emptyset\) is \(\#\)-null), but no \(f\) has \(\int_Ef\,d\#=m(E)\), since singletons would force \(f=0\). In probability, the Radon–Nikodym derivative is the density of a distribution, and (below) it is what makes conditional expectation exist.
constructing measures
An outer measure on \(X\) is \(\mu^*:\mathcal{P}(X)\to[0,\infty]\) with \(\mu^*(\emptyset)=0\), monotone, and countably subadditive.
For any outer measure \(\mu^*\), the sets \(E\) with \(\mu^*(A)=\mu^*(A\cap E)+\mu^*(A\cap E^c)\) for all \(A\) form a \(\sigma\)-algebra on which \(\mu^*\) is a measure, and that measure is complete.
An algebra \(\mathcal{A}\subseteq\mathcal{P}(X)\) contains \(\emptyset\) and is closed under complements and finite unions. A pre-measure \(\mu_0:\mathcal{A}\to[0,\infty]\) has \(\mu_0(\emptyset)=0\) and is countably additive whenever a countable disjoint union of sets of \(\mathcal{A}\) happens to lie in \(\mathcal{A}\).
A pre-measure \(\mu_0\) on an algebra \(\mathcal{A}\) extends to a measure on \(\sigma(\mathcal{A})\). The extension is unique if \(\mu_0\) is \(\sigma\)-finite.
Define \(\mu^*(E)=\inf\{\sum_n\mu_0(A_n):A_n\in\mathcal{A},\ E\subseteq\bigcup A_n\}\), the same recipe as Lebesgue outer measure with \(\mathcal{A}\) playing the role of boxes. It is an outer measure; countable additivity of \(\mu_0\) shows \(\mu^*=\mu_0\) on \(\mathcal{A}\), and finite additivity shows each \(A\in\mathcal{A}\) satisfies Carathéodory’s criterion. Carathéodory’s lemma then gives a measure on a \(\sigma\)-algebra containing \(\mathcal{A}\), hence containing \(\sigma(\mathcal{A})\). Uniqueness is a monotone-class (or \(\pi\)–\(\lambda\)) argument on sets of finite measure.
Finite disjoint unions of half-open intervals \((a,b]\) form an algebra on \(\mathbb{R}\). For a non-decreasing right-continuous \(F\), setting \(\mu_0((a,b])=F(b)-F(a)\) is a pre-measure (right-continuity is exactly what countable additivity needs), so Hahn–Kolmogorov produces a Borel measure \(\mu_F\). \(F(x)=x\) gives Lebesgue measure on Borel sets; \(F\) the Cantor staircase gives the singular Cantor measure; \(F\) a distribution function gives every Borel probability measure on \(\mathbb{R}\).
A measure space is complete if every subset of a null set is measurable. Given \((X,\mathcal{B},\mu)\), the sets \(B\cup N\) with \(B\in\mathcal{B}\) and \(N\) a subset of a \(\mu\)-null set form a \(\sigma\)-algebra \(\overline{\mathcal{B}}\), and \(\overline\mu(B\cup N)=\mu(B)\) is a well-defined complete measure extending \(\mu\).
\((\mathbb{R},\mathcal{L},m)\) is complete; \((\mathbb{R},\mathcal{B}(\mathbb{R}),m)\) is not, and its completion is \((\mathbb{R},\mathcal{L},m)\). A Lebesgue-null set that is not Borel: let \(c\) be the Cantor staircase and \(h(x)=x+c(x)\), a homeomorphism \([0,1]\to[0,2]\) that maps the Cantor set \(C\) onto a set of measure \(1\) (it maps the removed intervals, total length \(1\), onto intervals of total length \(1\)). Choose a non-measurable \(V\subseteq h( C)\) and put \(B=h^{-1}(V)\subseteq C\). Then \(m(B)=0\), so \(B\in\mathcal{L}\); but if \(B\) were Borel, \(V=(h^{-1})^{-1}(B)\) would be Borel (continuous preimage), hence measurable.
product measures
On \(X\times Y\), the natural sets are measurable rectangles \(A\times B\) and the natural measure gives them \(\mu(A)\nu(B)\). The product \(\sigma\)-algebra is generated by the rectangles; Hahn–Kolmogorov extends the rectangle measure to it; and the pay-off is the pair of theorems that let us compute a double integral as an iterated one. Tonelli needs only non-negativity; Fubini needs absolute integrability. Both need \(\sigma\)-finiteness, and each hypothesis has a counterexample showing it is not decoration.
the product sigma-algebra and cross-sections
For measurable spaces \((X,\Sigma_X)\) and \((Y,\Sigma_Y)\), \(\Sigma_X\times\Sigma_Y=\sigma(\{A\times B:A\in\Sigma_X,\ B\in\Sigma_Y\})\).
\(\mathcal{B}(\mathbb{R})\times\mathcal{B}(\mathbb{R})=\mathcal{B}(\mathbb{R}^2)\), since every open set in the plane is a countable union of open rectangles with rational corners. But \(\mathcal{L}(\mathbb{R})\times\mathcal{L}(\mathbb{R})\subsetneq\mathcal{L}(\mathbb{R}^2)\): \(V\times\{0\}\) with \(V\) a Vitali set is a subset of a null line, hence in \(\mathcal{L}(\mathbb{R}^2)\), but its cross-section at height \(0\) is not Lebesgue measurable, which (by the next lemma) rules it out of the product.
If \(E\in\Sigma_X\times\Sigma_Y\) then every cross-section \(E_x=\{y:(x,y)\in E\}\) is in \(\Sigma_Y\) and every \(E^y\) is in \(\Sigma_X\). If \(f\) is \(\Sigma_X\times\Sigma_Y\)-measurable, every \(y\mapsto f(x,y)\) is \(\Sigma_Y\)-measurable.
Good sets: \(\{E:E_x\in\Sigma_Y\ \forall x\}\) contains rectangles (\((A\times B)_x\) is \(B\) or \(\emptyset\)) and is a \(\sigma\)-algebra because sections commute with complements and unions.
product measure
If \((X,\Sigma_X,\mu_X)\) and \((Y,\Sigma_Y,\mu_Y)\) are \(\sigma\)-finite, there is a unique measure \(\mu_X\times\mu_Y\) on \(\Sigma_X\times\Sigma_Y\) with \((\mu_X\times\mu_Y)(A\times B)=\mu_X(A)\mu_Y(B)\).
(Outline.) Finite disjoint unions of rectangles form an algebra. Define \(\mu_0\) on it by summing \(\mu_X(A_i)\mu_Y(B_i)\); to check countable additivity, if \(A\times B=\bigsqcup_n A_n\times B_n\) then \(\mathbf{1}_A(x)\mathbf{1}_B(y)=\sum_n\mathbf{1}_{A_n}(x)\mathbf{1}_{B_n}(y)\); integrate in \(y\) and then in \(x\), each time by monotone convergence. Hahn–Kolmogorov extends \(\mu_0\), uniquely by \(\sigma\)-finiteness.
Lebesgue measure on \(\mathbb{R}^{d+d’}\) is the completion of the product of Lebesgue measures on \(\mathbb{R}^d\) and \(\mathbb{R}^{d’}\). Products of complete spaces need not be complete (the Vitali line above), which is why Tonelli and Fubini come in two versions.
\(X=Y=\{H,T\}\) with \(\mu(\{H\})=\mu(\{T\})=\tfrac12\). The product gives each of the four outcomes probability \(\tfrac14\): product measure is how independence is built.
Tonelli and Fubini
Let \(\mu_X,\mu_Y\) be \(\sigma\)-finite and \(f:X\times Y\to[0,\infty]\) be \(\Sigma_X\times\Sigma_Y\)-measurable. Then \(x\mapsto\int_Yf(x,y)\,d\mu_Y\) is \(\Sigma_X\)-measurable, symmetrically in \(y\), and \[ \int_{X\times Y}f\,d(\mu_X\times\mu_Y)=\int_X\Bigl(\int_Yf(x,y)\,d\mu_Y\Bigr)d\mu_X=\int_Y\Bigl(\int_Xf(x,y)\,d\mu_X\Bigr)d\mu_Y. \]
For \(f=\mathbf{1}_E\) the statement says that \(x\mapsto\mu_Y(E_x)\) is measurable and integrates to \((\mu_X\times\mu_Y)(E)\). This holds for rectangles, and the collection of \(E\) for which it holds is closed under increasing unions (monotone convergence) and, on pieces of finite measure, decreasing intersections; the monotone class lemma says such a collection containing an algebra contains the \(\sigma\)-algebra it generates. Then bootstrap: simple functions by linearity, unsigned functions by monotone convergence.
Let \(\mu_X,\mu_Y\) be complete and \(\sigma\)-finite, and \(f\) be real or complex, measurable for the completed product, with \(\int_{X\times Y}|f|<\infty\). Then for a.e. \(x\) the section \(y\mapsto f(x,y)\) is integrable, \(x\mapsto\int_Yf(x,y)\,d\mu_Y\) is integrable, and the double integral equals both iterated integrals.
Apply Tonelli (in its completed-product form, which holds for almost every section) to \(f_\pm\) or to the real and imaginary parts. Absolute integrability makes every iterated integral finite a.e., so the subtractions are legitimate.
In practice: use Tonelli on \(|f|\) to check \(\int|f|<\infty\), then use Fubini on \(f\).
Let \(I=\int_{\mathbb{R}}e^{-x^2}\,dx\). By Tonelli on the non-negative \(e^{-(x^2+y^2)}\) and polar coordinates, \[ I^2=\int_{\mathbb{R}^2}e^{-(x^2+y^2)}\,d(x,y)=\int_0^{2\pi}\!\!\int_0^\infty e^{-r^2}r\,dr\,d\theta=\pi, \] so \(I=\sqrt\pi\).
\(X=[0,1]\) with Lebesgue measure, \(Y=[0,1]\) with counting measure (not \(\sigma\)-finite), and \(f=\mathbf{1}_D\) for the diagonal \(D=\{(x,x)\}\), which is product-measurable. Then \[ \int_X\Bigl(\int_Y\mathbf{1}_D\,d\#\Bigr)dm=\int_X1\,dm=1,\qquad\int_Y\Bigl(\int_X\mathbf{1}_D\,dm\Bigr)d\#=\int_Y0\,d\#=0. \]
On \(\mathbb{N}\times\mathbb{N}\) with counting measure let \(a_{m,n}=1\) if \(n=m\), \(-1\) if \(n=m+1\), \(0\) otherwise. Each row sums to \(0\), so summing rows first gives \(0\); column \(1\) sums to \(1\) and every other column to \(0\), so summing columns first gives \(1\). Here \(\sum|a_{m,n}|=\infty\). A continuous analogue: \(f(x,y)=\frac{x^2-y^2}{(x^2+y^2)^2}\) on \((0,1)^2\) has iterated integrals \(\frac\pi4\) and \(-\frac\pi4\).
\(\displaystyle\int_0^1\!\!\int_y^1\frac{\sin x}{x}\,dx\,dy\) has no elementary inner integral. The integrand is bounded on the triangle \(0<y<x<1\), so Fubini applies; integrating in \(y\) first gives \(\displaystyle\int_0^1\frac{\sin x}{x}\cdot x\,dx=1-\cos1\).
Lp spaces
Identify functions that agree almost everywhere and the integrable functions become a normed space \(L^1\), which — unlike its Riemann counterpart — is complete. More generally \(L^p\) uses \(\|f\|_p=(\int|f|^p)^{1/p}\). Young’s inequality gives Hölder’s, Hölder gives Minkowski’s (the triangle inequality), and a fast-Cauchy-subsequence argument gives completeness. \(L^2\) is a Hilbert space, and for \(1<p<\infty\) the dual of \(L^p\) is \(L^q\) with \(\frac1p+\frac1q=1\). This is where measure theory hands over to Functional Analysis.
normed spaces and L1
A norm on a real or complex vector space \(V\) satisfies \(\|v\|=0\iff v=0\), \(\|\lambda v\|=|\lambda|\|v\|\), and \(\|u+v\|\le\|u\|+\|v\|\). If only the first condition fails it is a seminorm. A Banach space is a normed space that is complete in the metric \(\|u-v\|\).
\(\|f\|_\infty=\max|f|\) makes \(C[0,1]\) a Banach space (uniform limits of continuous functions are continuous). \(\|f\|_1=\int_0^1|f|\) is a norm on \(C[0,1]\) but not a complete one: continuous ramps converging to \(\mathbf{1}_{[1/2,1]}\) form a Cauchy sequence with no continuous limit.
On a measure space, \(\int|f|\) is a seminorm on the integrable functions, vanishing exactly when \(f=0\) a.e. \(L^1(X,\mu)\) is the quotient by the a.e.-zero functions: its elements are equivalence classes \([f]\), with \(\|[f]\|_1=\int|f|\,d\mu\).
\(L^1(X,\mu)\) is complete.
It is enough that every absolutely convergent series converges. Let \(\sum_n\|f_n\|_1=M<\infty\) and \(G=\sum_n|f_n|\). By monotone convergence \(\int G=M\), so \(G<\infty\) a.e. and \(F=\sum_nf_n\) converges absolutely a.e. The partial sums are dominated by \(G\), so dominated convergence gives \(\|F-\sum_{n\le N}f_n\|_1\to0\). (A Cauchy sequence has a subsequence with \(\|f_{n_{k+1}}-f_{n_k}\|_1<2^{-k}\); telescoping turns it into an absolutely convergent series, and a Cauchy sequence with a convergent subsequence converges.)
The proof shows more: an \(L^1\)-convergent sequence has a subsequence converging almost everywhere. The full sequence need not (the “typewriter” sequence in the probability section).
Lp spaces and the inequalities of Hölder and Minkowski
For \(0<p<\infty\), \(L^p\) is the set of (classes of) measurable \(f\) with \(\|f\|_p=(\int|f|^p\,d\mu)^{1/p}<\infty\). \(L^\infty\) uses the essential supremum \(\|f\|_\infty=\inf\{M:|f|\le M\text{ a.e.}\}\), which is attained. Exponents \(p,q\in[1,\infty]\) are conjugate if \(\frac1p+\frac1q=1\).
For \(x,y\ge0\) and conjugate \(1<p,q<\infty\): \(\displaystyle xy\le\frac{x^p}p+\frac{y^q}q\).
For \(x,y>0\), concavity of \(\log\) gives \(\log\bigl(\tfrac1px^p+\tfrac1qy^q\bigr)\ge\tfrac1p\log x^p+\tfrac1q\log y^q=\log(xy)\).
For conjugate \(p,q\in[1,\infty]\), \(f\in L^p\) and \(g\in L^q\): \(fg\in L^1\) and \(\|fg\|_1\le\|f\|_p\|g\|_q\).
The case \(p=1,q=\infty\) is \(|fg|\le|f|\,\|g\|_\infty\) a.e. Otherwise, if either norm is \(0\) the product is \(0\) a.e.; if not, scale so \(\|f\|_p=\|g\|_q=1\). Young pointwise gives \(|fg|\le\frac{|f|^p}p+\frac{|g|^q}q\); integrate to get \(\|fg\|_1\le\frac1p+\frac1q=1\).
For \(1\le p\le\infty\), \(\|f+g\|_p\le\|f\|_p+\|g\|_p\). Hence \(\|\cdot\|_p\) is a norm on \(L^p\).
For \(1<p<\infty\): \(|f+g|^p\le2^{p-1}(|f|^p+|g|^p)\) by convexity, so \(f+g\in L^p\). Then \[ \int|f+g|^p\le\int|f|\,|f+g|^{p-1}+\int|g|\,|f+g|^{p-1}\le(\|f\|_p+\|g\|_p)\,\bigl\||f+g|^{p-1}\bigr\|_q \] by Hölder, and \(\||f+g|^{p-1}\|_q=\|f+g\|_p^{p-1}\) since \((p-1)q=p\). Divide (when \(\|f+g\|_p\ne0\)).
- If \(\mu(X)<\infty\) and \(p<q\), then \(L^q\subseteq L^p\), with \(\|f\|_p\le\mu(X)^{\frac1p-\frac1q}\|f\|_q\).
- For sequences (counting measure on \(\mathbb{N}\)) the inclusion reverses: \(\ell^p\subseteq\ell^q\) for \(p<q\).
- On \(\mathbb{R}\) with Lebesgue measure neither inclusion holds: \(x^{-1/2}\mathbf{1}_{(0,1)}\in L^1\setminus L^2\) and \(x^{-1}\mathbf{1}_{(1,\infty)}\in L^2\setminus L^1\).
(1) Hölder with exponents \(\frac qp\) and its conjugate, applied to \(|f|^p\cdot1\). (2) If \(\sum|a_n|^p<\infty\) then \(|a_n|\le1\) eventually and \(|a_n|^q\le|a_n|^p\) there. On a probability space, (1) says higher moments control lower ones.
completeness, density, duality
\(L^p(X,\mu)\) is complete for every \(1\le p\le\infty\).
For Lebesgue measure on \(\mathbb{R}^d\) and \(1\le p<\infty\), each of the following is dense in \(L^p\): simple functions supported on sets of finite measure; step functions; continuous compactly supported functions.
Density fails for \(p=\infty\): a continuous function within \(\tfrac13\) of \(\mathbf{1}_{[0,\infty)}\) a.e. would have to jump at \(0\). In particular \(L^1([a,b])\) is the completion of \(C[a,b]\) under \(\|\cdot\|_1\), the promise made in the first section.
\(\langle f,g\rangle=\int f\overline g\,d\mu\) is an inner product on \(L^2\) with \(\langle f,f\rangle=\|f\|_2^2\), and \(L^2\) is complete, so it is a Hilbert space. Cauchy–Schwarz is Hölder with \(p=q=2\).
For conjugate \(1<p,q<\infty\), every bounded linear functional on \(L^q\) has the form \(g\mapsto\int fg\,d\mu\) for a unique \(f\in L^p\), with operator norm \(\|f\|_p\): \((L^q)^*\cong L^p\). If \(\mu\) is \(\sigma\)-finite then also \((L^1)^*\cong L^\infty\); but \((L^\infty)^*\) is strictly larger than \(L^1\) in general.
For sequences: \((\ell^q)^*=\ell^p\) and \((\ell^1)^*=\ell^\infty\), but a Banach limit (a functional extending \(\lim\) from convergent sequences, by Hahn–Banach) is a bounded functional on \(\ell^\infty\) that no \(\ell^1\) sequence represents. The proof of duality for \(1<p<\infty\) on a finite measure space defines \(\nu(E)=\Lambda(\mathbf{1}_E)\), checks \(\nu\ll\mu\), and takes \(f=d\nu/d\mu\) — Radon–Nikodym again.
probability on a measure space
A probability space is a measure space \((\Omega,\mathcal{B},P)\) with \(P(\Omega)=1\). Events are measurable sets, random variables are measurable functions, the distribution of \(X\) is the pushforward \(P_X=X_*P\), and \(E(X)=\int_\Omega X\,dP\). Every Borel probability measure on \(\mathbb{R}\) is determined by its distribution function, independence of random variables means their joint distribution is the product measure, and for any prescribed sequence of distributions there is a probability space carrying independent random variables with those distributions. The more elementary treatment is on the Probability page.
probability spaces, independence, random variables
A measure space \((\Omega,\mathcal{B},P)\) with \(P(\Omega)=1\). Sets in \(\mathcal{B}\) are events; a property holding a.e. holds almost surely (a.s.).
- Fair coin flips: \(([0,1),\mathcal{B},m)\) with the binary digits \(d_n(\omega)\) as the flips.
- Poisson: \((\mathbb{N}_0,\mathcal{P}(\mathbb{N}_0),\mu)\) with \(\mu(\{k\})=e^{-\lambda}\lambda^k/k!\).
- Any \(f\ge0\) with \(\int_{\mathbb{R}}f=1\) gives \((\mathbb{R},\mathcal{B},f\,dm)\).
Events \(A_1,A_2,\dots\) are independent if \(P(A_{i_1}\cap\dots\cap A_{i_k})=P(A_{i_1})\cdots P(A_{i_k})\) for every finite set of distinct indices. Pairwise independence is strictly weaker.
A random variable is a measurable \(X:\Omega\to\mathbb{R}\). Its distribution (law) is the Borel probability measure \(P_X(B)=P(X\in B)\), and its distribution function is \(F_X(x)=P(X\le x)\). Its expected value is \(E(X)=\int_\Omega X\,dP\) when \(X\) is integrable, and by change of variables \(E(\varphi(X))=\int_{\mathbb{R}}\varphi\,dP_X\).
On \(([0,1],\mathcal{B},m)\): a constant \(X=c\) has law \(\delta_c\) and \(F_X=\mathbf{1}_{[c,\infty)}\); the identity \(X(\omega)=\omega\) is uniform with \(F_X(x)=x\) on \([0,1]\); the square \(X(\omega)=\omega^2\) has \(F_X(x)=P(\omega\le\sqrt x)=\sqrt x\), density \(\frac1{2\sqrt x}\) on \((0,1)\), and \(E(X)=\int_0^1\omega^2\,d\omega=\frac13=\int_0^1x\cdot\frac{1}{2\sqrt x}\,dx\). The two calculations of \(E(X)\) are the two sides of change of variables.
distribution functions, densities, quantiles
- For a Borel probability measure \(P\) on \(\mathbb{R}\), \(F_P(x)=P((-\infty,x])\) is non-decreasing, right-continuous, with \(F_P(-\infty)=0\) and \(F_P(\infty)=1\).
- \(P\mapsto F_P\) is a bijection from Borel probability measures onto functions with these three properties.
(1) Monotonicity of \(P\); right-continuity and the limits are continuity from above and below along \((-\infty,x+\frac1n]\), \((-\infty,-n]\) and \((-\infty,n]\). (2) Injective: the rays \((-\infty,x]\) are closed under intersection and generate \(\mathcal{B}\), so two probability measures agreeing on them agree everywhere (a \(\pi\)–\(\lambda\) argument). Surjective: given \(F\), Lebesgue–Stieltjes via Hahn–Kolmogorov, or more concretely the quantile construction below.
For a distribution function \(F\), let \(G(\omega)=\inf\{x:F(x)\ge\omega\}\) for \(\omega\in(0,1)\). Then \(G(\omega)\le x\iff\omega\le F(x)\), so on \(((0,1),\mathcal{B},m)\) the random variable \(G\) has distribution function \(F\).
This is the theory behind inverse-transform sampling: \(G(U)\) with \(U\) uniform has law \(F\). For the exponential, \(F(x)=1-e^{-x}\) and \(G(u)=-\log(1-u)\).
\(X\) has a density \(f\) if \(P_X(B)=\int_Bf\,dm\) for all Borel \(B\), i.e. \(P_X\ll m\) and \(f=dP_X/dm\) (Radon–Nikodym). Then \(F_X(x)=\int_{-\infty}^xf\) and \(F_X’=f\) a.e. (Lebesgue differentiation). \(X\) is discrete with mass function \(p\) if \(P_X=\sum_kp(x_k)\delta_{x_k}\). By Lebesgue decomposition, a general law is a discrete part plus an absolutely continuous part plus a singular continuous part such as the Cantor distribution.
joint distributions and independent random variables
\(X_1,\dots,X_n\) are independent if \(P(X_1\in B_1,\dots,X_n\in B_n)=\prod_iP(X_i\in B_i)\) for all Borel \(B_i\); equivalently the joint law \(P_{(X_1,\dots,X_n)}\) on \(\mathbb{R}^n\) is the product \(P_{X_1}\times\dots\times P_{X_n}\). An infinite family is independent if every finite subfamily is.
If \(X_1,X_2,\dots\) are independent, then functions of disjoint groups of them, such as \(g(X_1,X_2)\) and \(h(X_3,X_4,X_5)\), are independent. The binary digits \(d_n\) of a uniform \(\omega\in[0,1)\) are independent fair coin flips.
If \(X,Y\) are independent and integrable then \(XY\) is integrable and \(E(XY)=E(X)E(Y)\).
By change of variables and independence, \(E(|XY|)=\int_{\mathbb{R}^2}|xy|\,d(P_X\times P_Y)\). Tonelli computes this as \(\int|x|\,dP_X\int|y|\,dP_Y<\infty\); then Fubini, without the absolute values, gives \(E(XY)=E(X)E(Y)\).
The converse fails: \(X\) uniform on \(\{-1,0,1\}\) and \(Y=X^2\) have \(E(XY)=E(X^3)=0=E(X)E(Y)\) but are dependent. Linearity \(E(X+Y)=E(X)+E(Y)\) needs no independence, and for independent \(X,Y\) with finite variance, \(\operatorname{Var}(X+Y)=\operatorname{Var}X+\operatorname{Var}Y\).
For any Borel probability measures \(P_1,P_2,\dots\) on \(\mathbb{R}\) there is a probability space carrying independent random variables \(X_1,X_2,\dots\) with \(X_n\sim P_n\).
On \([0,1)\) with Lebesgue measure, split the binary digits of \(\omega\) into infinitely many disjoint infinite subsequences (for example by the \(2\)-adic valuation of the index). Each subsequence, read as a binary expansion, is an independent uniform variable \(U_n\); then \(X_n=G_n(U_n)\) with \(G_n\) the quantile function of \(P_n\).
This theorem is what makes “let \(X_1,X_2,\dots\) be i.i.d.” a meaningful sentence.
sums of independent random variables
The classical limit theorems are statements about \(S_n=X_1+\dots+X_n\) for i.i.d. \(X_k\). Their meaning depends on the mode of convergence, so we first separate almost sure, in probability, in \(L^p\) and in distribution convergence. The weak law is a one-line consequence of Chebyshev’s inequality; the strong law needs a.s. convergence and (in the proof here) a fourth-moment estimate. The central limit theorem describes the fluctuations of size \(\sqrt n\) around \(n\mu\), and its proof runs through characteristic functions — Fourier transforms of distributions.
convolution
If \(X,Y\) are independent, \(P_{X+Y}\) is the pushforward of \(P_X\times P_Y\) under \((x,y)\mapsto x+y\), and \[ F_{X+Y}(z)=\int_{\mathbb{R}}F_X(z-y)\,dP_Y(y). \] If \(X,Y\) have densities \(f,g\), then \(X+Y\) has density \(f*g(z)=\int f(z-y)g(y)\,dy\).
For independent \(U,V\) uniform on \([0,1]\), \(\mathbf{1}_{[0,1]}*\mathbf{1}_{[0,1]}(z)\) is the length of \([0,1]\cap[z-1,z]\): the triangle density \(z\) on \([0,1]\) and \(2-z\) on \([1,2]\). Iterating convolutions smooths the shape towards a bell curve, which the CLT makes precise.
modes of convergence
For random variables \(X_n,X\):
- almost surely: \(P(\{\omega:X_n(\omega)\to X(\omega)\})=1\);
- in probability: \(P(|X_n-X|>\varepsilon)\to0\) for every \(\varepsilon>0\);
- in \(L^p\) (in mean when \(p=1\)): \(E(|X_n-X|^p)\to0\);
- in distribution: \(F_{X_n}(x)\to F_X(x)\) at every continuity point \(x\) of \(F_X\).
Almost sure convergence and \(L^p\) convergence each imply convergence in probability, which implies convergence in distribution. Convergence in probability implies a.s. convergence along a subsequence. No other implications hold in general.
\(L^p\Rightarrow\) probability: Markov, \(P(|X_n-X|>\varepsilon)\le\varepsilon^{-p}E|X_n-X|^p\). A.s. \(\Rightarrow\) probability: dominated convergence on \(\mathbf{1}_{\{|X_n-X|>\varepsilon\}}\le1\), which tends to \(0\) a.s. Subsequence: choose \(n_k\) with \(P(|X_{n_k}-X|>2^{-k})<2^{-k}\); the sum of these probabilities is finite, so a.s. only finitely many of the events occur (Borel–Cantelli), giving a.s. convergence.
On \(([0,1],\mathcal{B},m)\):
- \(X_n=n\mathbf{1}_{(0,1/n)}\to0\) a.s. and in probability, but \(E(X_n)=1\): not in mean.
- The typewriter: sweep indicators of \([0,1],[0,\tfrac12],[\tfrac12,1],[0,\tfrac13],\dots\) across the interval. \(E(X_n)\to0\), so \(X_n\to0\) in \(L^1\) and in probability, but every \(\omega\) is covered infinitely often, so \(X_n(\omega)\) converges for no \(\omega\).
- \(X\) a fair \(\pm1\) coin and \(X_n=-X\) for all \(n\): \(X_n\to X\) in distribution (same law) but not in probability.
the laws of large numbers
If \(X\ge0\) and \(a>0\), then \(P(X\ge a)\le E(X)/a\). If \(X\) has mean \(\mu\) and variance \(\sigma^2>0\), then \(P(|X-\mu|\ge a\sigma)\le1/a^2\).
\(a\mathbf{1}_{\{X\ge a\}}\le X\); take expectations. Chebyshev is Markov for \((X-\mu)^2\) at level \(a^2\sigma^2\).
Both are sharp: \(X\) equal to \(a\) with probability \(p\) and \(0\) otherwise attains Markov’s bound, and Markov fails for variables that can be negative.
If \(X_n\) are i.i.d. with finite variance \(\sigma^2\) and mean \(\mu\), then \(\bar X_n=\frac1n(X_1+\dots+X_n)\to\mu\) in probability.
\(E(\bar X_n)=\mu\) and, by independence, \(\operatorname{Var}(\bar X_n)=\sigma^2/n\). Chebyshev: \(P(|\bar X_n-\mu|>\varepsilon)\le\frac{\sigma^2}{n\varepsilon^2}\to0\).
To be 95% sure that the proportion of heads in \(n\) fair flips is within \(0.01\) of \(\tfrac12\), Chebyshev asks for \(\frac{1/4}{n(0.01)^2}\le0.05\), i.e. \(n\ge50\,000\). The normal approximation (CLT) says about \(9\,604\) suffice: Chebyshev is universal but crude.
If \(X_n\) are i.i.d. and integrable with mean \(\mu\), then \(\bar X_n\to\mu\) almost surely.
(Under the extra assumption \(E(X_1^4)<\infty\); the general case needs truncation.) Centre: \(Y_k=X_k-\mu\), \(S_n=Y_1+\dots+Y_n\), \(K=E(Y_1^4)\), \(\sigma^2=E(Y_1^2)\). Expanding \(E(S_n^4)=\sum_{i,j,k,l}E(Y_iY_jY_kY_l)\), any term in which some index appears exactly once vanishes by independence and \(E(Y)=0\). What survives is \(n\) terms \(E(Y_i^4)\) and \(3n(n-1)\) terms \(E(Y_i^2)E(Y_j^2)\), so \[ E(S_n^4)=nK+3n(n-1)\sigma^4\le Cn^2,\qquad E\Bigl(\sum_n\Bigl(\frac{S_n}n\Bigr)^4\Bigr)=\sum_n\frac{E(S_n^4)}{n^4}\le\sum_n\frac{C}{n^2}<\infty, \] using monotone convergence to swap sum and expectation. A random variable with finite expectation is finite a.s., so \(\sum_n(S_n/n)^4<\infty\) a.s., and the terms of a convergent series tend to \(0\): \(S_n/n\to0\) a.s.
Apply the strong law to the binary digits \(d_n(\omega)\) of a uniform \(\omega\in[0,1)\): for almost every \(\omega\), the proportion of \(1\)s among the first \(n\) digits tends to \(\tfrac12\). “Almost” is necessary — \(\omega=0\) and \(\omega=\tfrac17=0.001001\ldots_2\) are exceptions — but the exceptions form a null set.
the central limit theorem
\(N(\mu,\sigma^2)\) has density \(\frac{1}{\sigma\sqrt{2\pi}}e^{-(x-\mu)^2/2\sigma^2}\); \(\Phi\) is the distribution function of \(N(0,1)\). The standardisation of a random variable \(S\) with mean \(m\) and variance \(v>0\) is \((S-m)/\sqrt v\). If \(Z\sim N(0,1)\) then \(\mu+\sigma Z\sim N(\mu,\sigma^2)\).
If \(X_n\sim B(n,p)\) with \(0<p<1\), then \(P\Bigl(\frac{X_n-np}{\sqrt{np(1-p)}}\le x\Bigr)\to\Phi(x)\) for every \(x\).
Let \(X_n\) be i.i.d. with mean \(\mu\) and finite variance \(\sigma^2>0\). Then the standardisations \[ Z_n=\frac{X_1+\dots+X_n-n\mu}{\sigma\sqrt n} \] converge in distribution to \(N(0,1)\): \(P(Z_n\le x)\to\Phi(x)\) for every \(x\in\mathbb{R}\).
A sum of \(n\) independent \(\mathrm{Poisson}(1)\) variables is \(\mathrm{Poisson}(n)\), so the CLT says the standardised \(\mathrm{Poisson}(n)\) law tends to \(N(0,1)\). At \(x=0\): \(e^{-n}\sum_{k\le n}\frac{n^k}{k!}\to\tfrac12\), a fact that is awkward to prove directly.
characteristic functions and the proof of the CLT
\(\varphi_X(u)=E(e^{iuX})=\int_{\mathbb{R}}e^{iux}\,dP_X(x)\), defined for every real \(u\) because \(|e^{iux}|=1\).
- \(\varphi_X(0)=1\), \(|\varphi_X|\le1\), and \(\varphi_X\) is uniformly continuous (dominated convergence).
- \(\varphi_{aX+b}(u)=e^{iub}\varphi_X(au)\).
- If \(X,Y\) are independent, \(\varphi_{X+Y}=\varphi_X\varphi_Y\) (expectation of a product).
- If \(E(X^2)<\infty\), then \(\varphi_X(u)=1+iuE(X)-\tfrac12u^2E(X^2)+o(u^2)\) as \(u\to0\).
- \(\varphi_{N(0,1)}(u)=e^{-u^2/2}\); \(\varphi_{\mathrm{Poisson}(\lambda)}(u)=e^{\lambda(e^{iu}-1)}\); a fair \(\pm1\) coin has \(\cos u\).
If \(\varphi_{X_n}(u)\to\varphi_X(u)\) for every \(u\), then \(X_n\to X\) in distribution. In particular the characteristic function determines the distribution.
(Idea.) Smooth by an independent small Gaussian: \(X_n+\varepsilon Z\) has a density that is an explicit Fourier integral of \(\varphi_{X_n}(u)e^{-\varepsilon^2u^2/2}\), and dominated convergence passes the pointwise limit of \(\varphi_{X_n}\) inside. So \(X_n+\varepsilon Z\to X+\varepsilon Z\) in distribution for each \(\varepsilon\); let \(\varepsilon\to0\), which costs nothing at continuity points of \(F_X\).
(Of the CLT.) Let \(Y_k=(X_k-\mu)/\sigma\), so \(E(Y_k)=0\), \(E(Y_k^2)=1\), and \(Z_n=n^{-1/2}\sum_kY_k\). By facts 2–4, \[ \varphi_{Z_n}(u)=\varphi_Y\Bigl(\frac u{\sqrt n}\Bigr)^n=\Bigl(1-\frac{u^2}{2n}+o\Bigl(\frac1n\Bigr)\Bigr)^n\longrightarrow e^{-u^2/2}=\varphi_{N(0,1)}(u). \] (The limit of a power with a complex \(o(1/n)\) term is justified by \(|a^n-b^n|\le n|a-b|\) for \(|a|,|b|\le1\).) The continuity theorem finishes the proof. Since \(\Phi\) is continuous everywhere, convergence holds at every \(x\).
Continuity points cannot be dropped from the definition of convergence in distribution: \(X_n=\frac1n\) converges in distribution to \(0\), yet \(F_{X_n}(0)=0\not\to1=F_0(0)\).
conditional expectation and martingales
Conditioning on an event of positive probability is elementary; conditioning on “everything known at time \(n\)” is not, because that information is a \(\sigma\)-algebra and may consist of events of probability zero. The fix is to define \(E(X\mid\mathcal{A})\) by what it does — an \(\mathcal{A}\)-measurable variable with the same integrals as \(X\) over every set in \(\mathcal{A}\) — and prove existence with Radon–Nikodym. A martingale is a sequence whose conditional expectation of tomorrow, given today, is today: a fair game. Doob’s theorems say that you cannot beat a fair game by choosing your stakes or your stopping time (under conditions), and that a fair game bounded in \(L^1\) must settle down.
conditional expectation
For an integrable \(X\) and an event \(A\) with \(P(A)>0\): \(E(X\mid A)=\frac{E(X\mathbf{1}_A)}{P(A)}\), the average of \(X\) over \(A\). If \(\Omega=A_1\sqcup\dots\sqcup A_n\) with each \(P(A_k)>0\), the conditional expectation given the partition is the random variable equal to \(E(X\mid A_k)\) on \(A_k\).
Let \(X\) be integrable and \(\mathcal{A}\subseteq\mathcal{B}\) a sub-\(\sigma\)-algebra. A random variable \(Y\) is a conditional expectation of \(X\) given \(\mathcal{A}\), written \(E(X\mid\mathcal{A})\), if
- \(Y\) is \(\mathcal{A}\)-measurable, and
- \(\int_AY\,dP=\int_AX\,dP\) for every \(A\in\mathcal{A}\).
For a random variable \(Z\), \(E(X\mid Z)=E(X\mid\sigma(Z))\) with \(\sigma(Z)=\{Z^{-1}(B):B\text{ Borel}\}\).
For a finite partition the elementary definition satisfies (1) and (2), so this generalises it. Condition (1) says \(Y\) uses only the information in \(\mathcal{A}\); condition (2) says it is right on average over every piece of that information. When \(X\in L^2\), \(E(X\mid\mathcal{A})\) is the orthogonal projection of \(X\) onto the closed subspace \(L^2(\Omega,\mathcal{A},P)\): the best mean-square prediction of \(X\) from \(\mathcal{A}\).
\(E(X\mid\mathcal{A})\) exists and is unique up to almost sure equality.
Existence: for \(X\ge0\), \(\nu(A)=\int_AX\,dP\) is a finite measure on \((\Omega,\mathcal{A})\), absolutely continuous with respect to \(P|_{\mathcal{A}}\). Radon–Nikodym on \((\Omega,\mathcal{A})\) gives an \(\mathcal{A}\)-measurable \(Y=d\nu/dP|_{\mathcal{A}}\), which is (1) and (2). For general \(X\) apply this to \(X_+\) and \(X_-\). Uniqueness: if \(Y,Y’\) both qualify, \(A=\{Y>Y’\}\in\mathcal{A}\) and \(\int_A(Y-Y’)\,dP=0\) with a positive integrand on \(A\), so \(P(A)=0\); symmetrically \(P(Y<Y’)=0\).
On \(([0,1],\mathcal{B},m)\) let \(X(t)=t\mathbf{1}_{[0,1/2]}(t)\) and \(Y(t)=t\). \(X\) reveals \(t\) exactly when \(t\le\tfrac12\) and only “\(t>\tfrac12\)” otherwise, so \(\sigma(X)\) consists of the Borel subsets of \([0,\tfrac12]\) together with their unions with \((\tfrac12,1]\). Hence \[ E(Y\mid\sigma(X))(t)=\begin{cases}t,&t\le\tfrac12,\\ \tfrac34,&t>\tfrac12,\end{cases} \] the second value being the average of \(t\) over \((\tfrac12,1]\).
For integrable \(X,Y\), almost surely:
- linearity, and monotonicity: \(X\le Y\Rightarrow E(X\mid\mathcal{A})\le E(Y\mid\mathcal{A})\);
- \(E(E(X\mid\mathcal{A}))=E(X)\);
- if \(X\) is \(\mathcal{A}\)-measurable, \(E(X\mid\mathcal{A})=X\); more generally if \(Z\) is \(\mathcal{A}\)-measurable and \(ZX\) integrable, \(E(ZX\mid\mathcal{A})=ZE(X\mid\mathcal{A})\) (“taking out what is known”);
- (tower) if \(\mathcal{A}_1\subseteq\mathcal{A}_2\), \(E(E(X\mid\mathcal{A}_2)\mid\mathcal{A}_1)=E(X\mid\mathcal{A}_1)\);
- if \(\sigma(X)\) is independent of \(\mathcal{A}\), \(E(X\mid\mathcal{A})=E(X)\);
- \(|E(X\mid\mathcal{A})|\le E(|X|\mid\mathcal{A})\).
Each is checked against the definition: exhibit the candidate, observe it is measurable for the smaller \(\sigma\)-algebra, and verify the integral identity over sets of that \(\sigma\)-algebra; uniqueness does the rest. For (4), \(A\in\mathcal{A}_1\subseteq\mathcal{A}_2\) gives \(\int_AE(X\mid\mathcal{A}_2)=\int_AX\). For (5), \(\int_AX=E(X\mathbf{1}_A)=E(X)P(A)=\int_AE(X)\) by independence of \(X\) and \(\mathbf{1}_A\). For (3), check indicators \(Z=\mathbf{1}_B\) and bootstrap.
filtrations and martingales
A filtration is an increasing sequence of sub-\(\sigma\)-algebras \(\mathcal{F}_0\subseteq\mathcal{F}_1\subseteq\cdots\subseteq\mathcal{B}\) (the information available at each time). A sequence \(X_n\) is adapted if each \(X_n\) is \(\mathcal{F}_n\)-measurable. The natural filtration of \(X_n\) is \(\mathcal{F}_n=\sigma(X_0,\dots,X_n)\).
An adapted sequence of integrable \(X_n\) is a martingale if \(E(X_{n+1}\mid\mathcal{F}_n)=X_n\), a supermartingale if \(E(X_{n+1}\mid\mathcal{F}_n)\le X_n\) (unfavourable to the gambler) and a submartingale if \(\ge\). By the tower property, \(E(X_m\mid\mathcal{F}_n)=X_n\) for all \(m\ge n\) for a martingale, and \(E(X_n)\) is constant (non-increasing for a supermartingale).
- Sums: for independent integrable \(Y_k\) with mean \(0\), \(S_n=Y_1+\dots+Y_n\) is a martingale in its natural filtration (take out \(S_n\), and \(Y_{n+1}\) is independent of \(\mathcal{F}_n\)). The simple random walk is the case \(Y_k=\pm1\) with probability \(\tfrac12\).
- Products: for independent \(Y_k\ge0\) with mean \(1\), \(M_n=Y_1\cdots Y_n\) is a martingale.
- Doob martingale: for any integrable \(X\), \(X_n=E(X\mid\mathcal{F}_n)\) is a martingale (tower property): successive predictions of one quantity as information accumulates.
If \(S_n\) is the simple random walk, \(S_n^2\) is a submartingale and \(S_n^2-n\) is a martingale: \(E(S_{n+1}^2\mid\mathcal{F}_n)=S_n^2+2S_nE(Y_{n+1})+E(Y_{n+1}^2)=S_n^2+1\). In general a convex function of a martingale is a submartingale (conditional Jensen), given integrability.
betting systems and stopping times
\(H_n\) (\(n\ge1\)) is previsible if \(H_n\) is \(\mathcal{F}_{n-1}\)-measurable: the stake on round \(n\) is decided before round \(n\) is played. The martingale transform is the gambler’s winnings \[ (H\cdot X)_n=\sum_{k=1}^nH_k(X_k-X_{k-1}). \]
If \(X\) is a martingale and \(H\) is bounded and previsible, then \(H\cdot X\) is a martingale with \((H\cdot X)_0=0\), so \(E((H\cdot X)_n)=0\) for every \(n\). If \(X\) is a supermartingale and \(0\le H\le K\), then \(H\cdot X\) is a supermartingale.
\(E(H_{n+1}(X_{n+1}-X_n)\mid\mathcal{F}_n)=H_{n+1}E(X_{n+1}-X_n\mid\mathcal{F}_n)\), taking out what is known, and the last factor is \(0\) (or \(\le0\), and \(H_{n+1}\ge0\) preserves the sign). Boundedness of \(H\) supplies integrability.
\(T:\Omega\to\{0,1,2,\dots,\infty\}\) is a stopping time if \(\{T=n\}\in\mathcal{F}_n\) for every \(n\) (equivalently \(\{T\le n\}\in\mathcal{F}_n\)): whether to stop now depends only on what has happened so far. The stopped process is \(X_{T\wedge n}\).
If \(X\) is a (super)martingale and \(T\) a stopping time, then \(X_{T\wedge n}\) is a (super)martingale. In particular \(E(X_{T\wedge n})=E(X_0)\) for a martingale.
\(X_{T\wedge n}-X_0=(H\cdot X)_n\) with \(H_k=\mathbf{1}_{\{T\ge k\}}\), which is previsible since \(\{T\ge k\}=\Omega\setminus\{T\le k-1\}\in\mathcal{F}_{k-1}\), and takes values in \(\{0,1\}\).
Bet \(1\) on a fair coin; after each loss double the stake; stop at the first win. If the first win is on round \(T\), the net gain is \(2^{T-1}-(1+2+\dots+2^{T-2})=1\), and \(T<\infty\) a.s. So \(X_T=1\) a.s. while \(E(X_0)=0\): a sure profit from a fair game. The catch is unbounded credit — the loss just before the win is \(2^{T-1}-1\), with infinite expectation — and the optional stopping theorem says exactly which hypothesis has been violated.
martingale convergence
For \(a<b\), \(U_n[a,b](\omega)\) is the number of completed upcrossings of \([a,b]\) by \(X_0(\omega),\dots,X_n(\omega)\): the number of times the path climbs from below \(a\) to above \(b\).
For a supermartingale \(X\) and \(a<b\): \((b-a)\,E(U_n[a,b])\le E((X_n-a)^-)\).
Use the strategy “buy one unit when \(X\) drops below \(a\), hold until it rises above \(b\), then sell and wait”. Its stakes \(H_k\in\{0,1\}\) are previsible, so \(W=H\cdot X\) is a supermartingale with \(E(W_n)\le0\). Each completed upcrossing earns at least \(b-a\), and the possibly unfinished last one loses at most \((X_n-a)^-\). Hence \(0\ge E(W_n)\ge(b-a)E(U_n[a,b])-E((X_n-a)^-)\).
If \(X_n\) is a supermartingale bounded in \(L^1\), i.e. \(\sup_nE(|X_n|)<\infty\), then \(X_n\) converges almost surely to a finite limit \(X_\infty\).
By the upcrossing lemma and monotone convergence, \(E(U_\infty[a,b])\le\frac{\sup_nE|X_n|+|a|}{b-a}<\infty\), so a.s. the path makes only finitely many upcrossings of \([a,b]\). If \(X_n(\omega)\) does not converge in \([-\infty,\infty]\) then \(\liminf X_n<a<b<\limsup X_n\) for some rationals \(a<b\), forcing infinitely many upcrossings; the union of these events over countably many rational pairs is null. So \(X_n\to X_\infty\) a.s., and Fatou gives \(E|X_\infty|\le\liminf E|X_n|<\infty\), so the limit is finite a.s.
A non-negative supermartingale converges almost surely (since \(E|X_n|=E(X_n)\le E(X_0)\)).
- The simple random walk \(S_n\) does not converge (it visits every integer infinitely often a.s.); it is not bounded in \(L^1\), since \(E|S_n|\approx\sqrt{2n/\pi}\).
- The product martingale \(M_n=\prod_{k\le n}Y_k\) with \(Y_k\in\{\tfrac12,\tfrac32\}\) equally likely is non-negative, so it converges a.s.; by the strong law applied to \(\log Y_k\), whose mean \(\tfrac12\log\tfrac34<0\), the limit is \(0\), even though \(E(M_n)=1\) for every \(n\). A.s. convergence does not give \(L^1\) convergence; that needs uniform integrability.
- A Doob martingale \(E(X\mid\mathcal{F}_n)\) is bounded in \(L^1\) by \(E|X|\), so it converges a.s.
optional stopping
Let \(X\) be a supermartingale and \(T\) an a.s. finite stopping time. If any one of
- \(X\) is bounded: \(|X_n|\le K\) for all \(n\);
- \(T\) is bounded: \(T\le N\);
- \(E(T)<\infty\) and the increments are bounded: \(|X_{n+1}-X_n|\le K\);
holds, then \(X_T\) is integrable and \(E(X_T)\le E(X_0)\). For a martingale, \(E(X_T)=E(X_0)\).
The stopped process is a supermartingale, so \(E(X_{T\wedge n})\le E(X_0)\) for every \(n\), and \(X_{T\wedge n}\to X_T\) a.s. because \(T<\infty\) a.s. It remains to pass the limit through \(E\). Under (2), take \(n=N\). Under (1), dominated convergence with bound \(K\). Under (3), \(|X_{T\wedge n}-X_0|\le\sum_{k<T}|X_{k+1}-X_k|\le KT\), integrable, so dominated convergence again. For a martingale apply the result to \(X\) and \(-X\).
For the simple random walk from \(S_0=0\), let \(T=\min\{n:S_n=1\}\). The walk is recurrent, so \(T<\infty\) a.s. and \(S_T=1\), hence \(E(S_T)=1\ne0=E(S_0)\). None of the conditions holds: the walk is unbounded below, \(T\) is unbounded, and although the increments are bounded, \(E(T)=\infty\). (So the theorem also proves \(E(T)=\infty\): otherwise (3) would force \(E(S_T)=0\).)
Start the simple random walk at \(0\) and stop at \(T\), the first time it hits \(-a\) or \(b\) (\(a,b>0\) integers). The stopped walk is bounded by \(\max(a,b)\), so condition (1) applies: \(0=E(S_T)=-aP(S_T=-a)+bP(S_T=b)\), giving \[ P(S_T=b)=\frac{a}{a+b}. \] For the duration, \(S_n^2-n\) is a martingale, so \(E(S_{T\wedge n}^2)=E(T\wedge n)\); letting \(n\to\infty\) (bounded convergence on the left, monotone convergence on the right) gives \(E(T)=E(S_T^2)=ab\).
where it leads
- Functional analysis. \(L^p\) spaces, duality and Hilbert space structure are the standard examples on the Functional Analysis page; Riesz representation identifies the dual of \(C(K)\) with a space of measures.
- Stochastic processes. Martingales, stopping times and filtrations run continuous-time stochastic calculus (Brownian motion, Itô integrals) and the pricing of derivatives, where a change of measure is a Radon–Nikodym derivative (Girsanov).
- Statistics. Likelihoods are Radon–Nikodym derivatives with respect to a dominating measure, which is why discrete and continuous models fit one theory; consistency of estimators is a law of large numbers and asymptotic normality is a central limit theorem. See Statistics.
- Harmonic analysis and PDE. The maximal function and Lebesgue differentiation start Calderón–Zygmund theory; Sobolev spaces are \(L^p\) spaces of functions with weak derivatives.
- Ergodic theory. The strong law is the i.i.d. case of Birkhoff’s ergodic theorem: time averages equal space averages for measure-preserving systems.
Backlinks (5)
1. Bayesian Inference /wiki/mathematics/statistics/bayesian/
2. λambda /roam/lambda/
λambda — Learner-Adaptive, Marks-Bound Drilling Agent — is a study tool that keeps a working image of your mind: what you hold, where you stall, and which exam marks each gap is costing you. It probes before it teaches, teaches only what you missed, and drills until the fix survives a variant. The protocol is open source at lambda-agent; the hosted study lounge is being built at lambda.fatfort.com.
■ your mind, as λ understands it ■ what the Language Model knows that is relevant to you ■ everything else the Language Model knows
3. Probability /wiki/mathematics/probability/
This page pairs well with Statistics.
Elements of Probability Theory
Rigorously, \((\Omega,\mathcal{F},P)\) is a measure space of total mass one, a random variable is a measurable function and \(\mathbb{E}[X]=\int_\Omega X\,dP\). The measure-theoretic foundations — distribution functions, independence as product measure, modes of convergence, the laws of large numbers, the CLT via characteristic functions, conditional expectation and martingales — are developed on the Measure Theory page.
A random experiment has uncertain outcomes. The sample space \(S\) is the set of all possible outcomes. An event \(E\) is a subset of \(S\). The certain event is \(S\); the impossible event is \(\varnothing\).
4. Real Analysis /wiki/mathematics/analysis/real/
I am finding Real Analysis to be more difficult than any other mathematics that I have studied before. I can seem to verify the truth of statements because they seem right; but I am having a difficult time producing rigorous and correct proofs.
It seems that High-School children (on the internet) are able to self-study Fomin with success. Bitterly, we remind ourselves:
“Comparison is the thief of Joy”—Theodore Roosevelt (probably)
Comments