Autoencoders

an autoencoder is a network trained to do the one thing that sounds useless: output its own input. the trick is the obstacle course in the middle — a bottleneck, a corruption, a penalty — that makes verbatim copying impossible, so the network is forced to learn what about the input is worth keeping. 𐃏 the family tree below runs from the linear special case (which is pca wearing a trenchcoat) to the variational autoencoder, which turns the whole construction into a generative model (Goodfellow, Ian, 2016).

the undercomplete autoencoder#

two maps and a loss. an encoder \(f\) compresses, a decoder \(g\) reconstructs, and training minimises the reconstruction error

\begin{equation} \mathcal{L}(\theta) = \frac{1}{n}\sum_{i=1}^{n} \bigl\lVert x_i - g\bigl(f(x_i)\bigr) \bigr\rVert^2 , \qquad f: \mathbb{R}^d \to \mathbb{R}^m, \quad g: \mathbb{R}^m \to \mathbb{R}^d , \end{equation}

with \(m < d\): the code \(h = f(x)\) is undercomplete, so the identity function is unrepresentable and the network must ration its \(m\) numbers on the directions of greatest structure. with deep nonlinear \(f\) and \(g\), the model learns a curved \(m\)-dimensional manifold that hugs the data — nonlinear pca in the honest sense of the phrase.

the linear case is pca — stated carefully#

strip the nonlinearities: \(f(x) = W_e x\), \(g(h) = W_d h\), squared error, centred data. then the optimal reconstruction subspace is exactly the pca subspace, but the equivalence has fine print worth getting right:

• at any global optimum, the columns of \(W_d\) span the subspace of the top \(m\) principal components, and the achieved loss equals the sum of the discarded eigenvalues \(\sum_{j > m} \lambda_j\) — the same optimum pca attains.
• the individual weight vectors need not be the eigenvectors: any invertible \(m \times m\) mixing \(A\) leaves the model invariant (\(W_d A^{-1}\), \(A W_e\)), so the network identifies the principal subspace, not the principal axes, and codes are not decorrelated or variance-ordered as pca’s are.
• the loss surface has no spurious local minima — every additional critical point is a saddle corresponding to picking a non-top subset of eigenvectors (baldi and hornik, 1989).¹
• gradient descent therefore reliably finds the pca solution up to that mixing — which is precisely what the experiment at the bottom of this page verifies numerically.

the corollary: depth and nonlinearity are not decoration. a linear autoencoder of any depth collapses to this same solution; everything genuinely new about autoencoders lives in the nonlinear case.

regularised autoencoders#

capacity control by bottleneck is blunt. the alternative family keeps the code wide — even overcomplete, \(m \ge d\) — and prevents copying with a penalty or a corruption instead (Goodfellow, Ian, 2016).

denoising autoencoders#

corrupt the input, demand the clean version back:

\begin{equation} \mathcal{L} = \mathbb{E}_{x,\, \tilde{x} \sim C(\tilde{x} \mid x)} \bigl\lVert x - g\bigl(f(\tilde{x})\bigr) \bigr\rVert^2 , \end{equation}

with \(C\) a corruption process — gaussian noise, or masking a random subset of coordinates to zero. 𐃏 the identity is no longer a solution even with infinite capacity, because \(\tilde{x} \ne x\). what the network learns instead is a map from off-manifold points back onto the data manifold; for small gaussian corruption the optimal reconstruction points along the gradient of the log-density, so the dae implicitly estimates the shape of the data distribution.

sparse autoencoders#

keep the code big but mostly asleep. with sigmoid hidden units, let \(\hat{\rho}_j\) be unit \(j\)’s average activation over the training set, pick a small target \(\rho\) (say 0.05), and add a kl-divergence penalty between two bernoulli distributions:

\begin{equation} \mathcal{L}_{\text{sparse}} = \mathcal{L}_{\text{recon}} + \beta \sum_{j=1}^{m} \left[ \rho \log\frac{\rho}{\hat{\rho}_j} + (1 - \rho) \log\frac{1 - \rho}{1 - \hat{\rho}_j} \right]. \end{equation}

each unit is pushed to fire rarely, so any single input touches only a few of them — a distributed dictionary where codes are sparse combinations of learned features. (the cheaper cousin: an \(\ell_1\) penalty \(\beta \sum_j |h_j|\) directly on the code.)

contractive autoencoders#

penalise the encoder’s sensitivity itself, via the frobenius norm of its jacobian:

\begin{equation} \mathcal{L}_{\text{cae}} = \mathcal{L}_{\text{recon}} + \lambda \left\lVert \frac{\partial f}{\partial x}(x) \right\rVert_F^2 = \mathcal{L}_{\text{recon}} + \lambda \sum_{j,k} \left( \frac{\partial h_j}{\partial x_k} \right)^{2}. \end{equation}

the penalty wants \(f\) locally constant; reconstruction wants \(f\) to preserve information. the compromise: the encoder becomes flat in directions orthogonal to the data manifold (noise is contracted away) while staying sensitive along it (structure survives). denoising and contracting are two prices for the same product — local robustness — one paid stochastically, one analytically.

the variational autoencoder#

everything above gives representations; none of it gives a principled way to generate. the vae (kingma and welling, 2013, auto-encoding variational bayes) fixes that by making the story probabilistic from the start: a prior \(p(z) = \mathcal{N}(0, I)\) over latents, a decoder network defining \(p_\theta(x \mid z)\), and the ambition to maximise the marginal likelihood

\begin{equation} \log p_\theta(x) = \log \int p_\theta(x \mid z)\, p(z)\, dz , \end{equation}

which is intractable — as is the posterior \(p_\theta(z \mid x)\) needed by em. the move: introduce an encoder network \(q_\phi(z \mid x) = \mathcal{N}\bigl(\mu_\phi(x), \operatorname{diag}(\sigma_\phi^2(x))\bigr)\) as a learned approximate posterior.

the elbo, by jensen#

multiply and divide inside the marginal by \(q_\phi\), then push the (concave) logarithm through the expectation with jensen’s inequality:

\begin{align*} \log p_\theta(x) &= \log \mathbb{E}_{z \sim q_\phi(z \mid x)} \left[ \frac{p_\theta(x \mid z)\, p(z)}{q_\phi(z \mid x)} \right] \\ &\ge \mathbb{E}_{z \sim q_\phi(z \mid x)} \left[ \log \frac{p_\theta(x \mid z)\, p(z)}{q_\phi(z \mid x)} \right] \\ &= \underbrace{\mathbb{E}_{q_\phi}\bigl[\log p_\theta(x \mid z)\bigr]}_{\text{reconstruction}} \;-\; \underbrace{D_{\mathrm{KL}}\bigl(q_\phi(z \mid x) \,\Vert\, p(z)\bigr)}_{\text{regulariser}} \;=\; \mathcal{L}_{\text{ELBO}}(x; \theta, \phi). \end{align*}

the slack in jensen is exactly \(D_{\mathrm{KL}}\bigl(q_\phi(z \mid x) \,\Vert\, p_\theta(z \mid x)\bigr) \ge 0\): the bound is tight precisely when the encoder matches the true posterior. maximising the elbo therefore does two jobs at once — pushes the likelihood up, and squeezes the approximate posterior toward the real one. and squint at the two terms: reconstruction error plus a penalty keeping codes near a fixed prior — the vae is an autoencoder whose regulariser fell out of a probability argument rather than being bolted on.

the reparameterisation trick#

the elbo contains an expectation over \(z \sim q_\phi\), and \(\phi\) lives inside the sampling distribution — you cannot backpropagate through a random draw. rewrite the sample as a deterministic function of \(\phi\) and parameter-free noise:

\begin{equation} z = \mu_\phi(x) + \sigma_\phi(x) \odot \varepsilon, \qquad \varepsilon \sim \mathcal{N}(0, I), \end{equation}

which has exactly the law \(q_\phi(z \mid x)\), but now gradients flow through \(\mu_\phi\) and \(\sigma_\phi\) while the randomness sits harmlessly in \(\varepsilon\). a single monte carlo sample per datum gives a low-variance unbiased gradient estimate, and the whole model trains end-to-end by sgd.

the gaussian kl in closed form#

for \(q = \mathcal{N}(\mu, \operatorname{diag}(\sigma^2))\) against \(p = \mathcal{N}(0, I)\), the kl term needs no sampling at all:

\begin{equation} D_{\mathrm{KL}}\bigl(q \,\Vert\, p\bigr) = \frac{1}{2} \sum_{j=1}^{m} \left( \mu_j^2 + \sigma_j^2 - \log \sigma_j^2 - 1 \right), \end{equation}

each term zero iff \(\mu_j = 0, \sigma_j = 1\): unused latent coordinates are shrunk exactly onto the prior. 𐃏 generation is then two cheap steps: draw \(z \sim \mathcal{N}(0, I)\), decode.

beta-vae#

scale the kl term by \(\beta > 1\),

\begin{equation} \mathcal{L}_{\beta} = \mathbb{E}_{q_\phi}\bigl[\log p_\theta(x \mid z)\bigr] - \beta\, D_{\mathrm{KL}}\bigl(q_\phi(z \mid x) \,\Vert\, p(z)\bigr), \end{equation}

and the model trades reconstruction fidelity for a code squeezed harder against the isotropic prior — empirically encouraging disentangled latents, one factor of variation per coordinate (higgins et al., 2017). \(\beta\) is a dial between the two things an autoencoder can be: at low \(\beta\) a compressor, at high \(\beta\) an explainer.

code: a linear autoencoder rediscovers pca#

2-d data with one dominant direction, squeezed through a 1-d code. gradient descent on reconstruction error, then compare the learned subspace against the eigendecomposition:

import numpy as np

rng = np.random.default_rng(0)

# --- synthetic 2d data with one dominant direction ---
n = 500
true_dir = np.array([2.0, 1.0]) / np.sqrt(5.0)          # unit principal direction
z = rng.normal(0, 2.0, n)                                # large variance along it
noise = rng.normal(0, 0.3, (n, 2))                       # small isotropic residue
X = np.outer(z, true_dir) + noise
X -= X.mean(axis=0)                                      # centre the data

# --- pca answer, for reference ---
C = X.T @ X / n
eigvals, eigvecs = np.linalg.eigh(C)
v_pca = eigvecs[:, -1]                                   # top principal direction

# --- linear autoencoder: 2 -> 1 -> 2, no biases, untied weights ---
W_enc = rng.normal(0, 0.1, (1, 2))                       # encoder
W_dec = rng.normal(0, 0.1, (2, 1))                       # decoder
lr = 0.05
for step in range(2001):
    h = X @ W_enc.T                                      # codes      (n, 1)
    Xr = h @ W_dec.T                                     # reconstructions
    err = Xr - X
    loss = (err ** 2).sum(axis=1).mean()
    if step % 400 == 0:
        print(f"step {step:>4}  reconstruction mse {loss:.4f}")
    dW_dec = 2 * err.T @ h / n                           # (2, 1)
    dW_enc = 2 * (err @ W_dec).T @ X / n                 # (1, 2)
    W_dec -= lr * dW_dec
    W_enc -= lr * dW_enc

# the learned decoder column spans the model's 1-d reconstruction subspace
v_ae = W_dec[:, 0] / np.linalg.norm(W_dec[:, 0])
align_pca = abs(v_ae @ v_pca)
align_true = abs(v_ae @ true_dir)
print(f"\npca direction:      {np.round(v_pca, 4)}")
print(f"ae direction:       {np.round(v_ae, 4)}")
print(f"|cos| ae vs pca:    {align_pca:.6f}")
print(f"|cos| ae vs truth:  {align_true:.6f}")
print(f"residual variance (pca optimum): {eigvals[0]:.4f}")

step    0  reconstruction mse 4.3774
step  400  reconstruction mse 0.0877
step  800  reconstruction mse 0.0877
step 1200  reconstruction mse 0.0877
step 1600  reconstruction mse 0.0877
step 2000  reconstruction mse 0.0877

pca direction:      [-0.8919 -0.4522]
ae direction:       [-0.8919 -0.4522]
|cos| ae vs pca:    1.000000
|cos| ae vs truth:  0.999984
residual variance (pca optimum): 0.0877

the theory section, verified line by line:

• the learned decoder direction matches the top principal component to six decimal places (cosine similarity 1.000000), and both sit within sampling noise of the true generating direction.
• the converged reconstruction mse, 0.0877, equals the discarded eigenvalue \(\lambda_2\) — the pca optimum — to four decimals. gradient descent found the global minimum of the linear autoencoder, as baldi–hornik promise.
• absolute values are used in the comparison because the subspace, not the sign, is identified: negate encoder and decoder together and nothing changes.

see also#

• pca — the linear case, done with eigenvectors instead of gradients
• generative adversarial networks — generation by game instead of by likelihood bound
• stable diffusion — a vae compresses the pixels; diffusion happens in its latent space
• feedforward deep neural networks — the machinery inside \(f\) and \(g\)

References#

Goodfellow, Ian (2016). Deep Learning, MIT Press.