GAN: Generative Adversarial Networks

a gan trains a generator by making it play a game against a learned critic: the generator \(G\) maps noise to samples, the discriminator \(D\) tries to tell those samples from real data, and each improves by exploiting the other’s current weakness — density estimation recast as a two-player minimax game (goodfellow et al. 2014, generative adversarial networks). 𐃏 the framework is treated in ch. 20 of (Goodfellow, Ian, 2016).

the minimax game#

let \(p_{\text{data}}\) be the data distribution, \(p_z\) a fixed noise prior (gaussian, say), and \(p_g\) the distribution of \(G(z)\), \(z \sim p_z\). the value function is

\begin{equation} \min_G \max_D \; V(D, G) = \mathbb{E}_{x \sim p_{\text{data}}}\!\left[\log D(x)\right] + \mathbb{E}_{z \sim p_z}\!\left[\log\!\big(1 - D(G(z))\big)\right]. \end{equation}

\(D\) is a binary classifier maximising the log-likelihood of “real vs fake”; \(G\) minimises the same quantity — it wants its fakes classified real.

why the optimum is \(p_g = p_{\text{data}}\)#

step 1 — the optimal discriminator. fix \(G\) and write the value as an integral:

\begin{equation} V(D, G) = \int_x \Big[ p_{\text{data}}(x) \log D(x) + p_g(x) \log\big(1 - D(x)\big) \Big] \, dx. \end{equation}

the integrand can be maximised pointwise: for constants \(a, b \ge 0\) (not both zero), \(y \mapsto a \log y + b \log(1-y)\) peaks at \(y^\ast = \frac{a}{a+b}\).¹ hence

\begin{equation} D^\ast(x) = \frac{p_{\text{data}}(x)}{p_{\text{data}}(x) + p_g(x)}. \end{equation}

the perfect discriminator is a density ratio estimator — this observation powers half of modern self-supervised learning.

step 2 — the jensen-shannon divergence emerges. substitute \(D^\ast\) back:

\begin{align*} C(G) &= \mathbb{E}_{x \sim p_{\text{data}}}\!\left[\log \frac{p_{\text{data}}(x)}{p_{\text{data}}(x) + p_g(x)}\right] + \mathbb{E}_{x \sim p_g}\!\left[\log \frac{p_g(x)}{p_{\text{data}}(x) + p_g(x)}\right] \\ &= -\log 4 + \mathrm{KL}\!\left(p_{\text{data}} \,\Big\|\, \tfrac{p_{\text{data}} + p_g}{2}\right) + \mathrm{KL}\!\left(p_g \,\Big\|\, \tfrac{p_{\text{data}} + p_g}{2}\right) \\ &= -\log 4 + 2\, \mathrm{JSD}\!\left(p_{\text{data}} \,\|\, p_g\right), \end{align*}

where the middle line follows by multiplying and dividing each ratio by \(2\). since \(\mathrm{JSD} \ge 0\) with equality iff the arguments coincide, the global minimum of \(C(G)\) is \(-\log 4\), attained exactly when \(p_g = p_{\text{data}}\) — at which point \(D^\ast \equiv \tfrac12\): the discriminator is reduced to coin-flipping. the caveats are real, though: the proof is in the space of distributions, assumes \(D\) is optimal at every step, and says nothing about whether alternating sgd on network parameters converges there.

the non-saturating trick#

early in training \(G\) is terrible, \(D\) confidently outputs \(D(G(z)) \approx 0\), and the generator’s minimax loss \(\log(1 - D(G(z)))\) is flat there — its gradient vanishes exactly when the generator most needs signal. the standard fix flips the objective: instead of minimising \(\log(1-D(G(z)))\), maximise \(\log D(G(z))\). 𐃏 near \(D(G(z)) = 0\) the new objective has enormous gradient rather than none.

training pathologies#

• mode collapse. the generator maps many \(z\) to few outputs — it parks all its mass on whatever currently fools \(D\), then hops when \(D\) catches up. symptom: low sample diversity; in the 1d experiment below it appears as generated std far below target.
• vanishing gradients. if \(D\) gets too good and \(p_g, p_{\text{data}}\) have (near-)disjoint supports — generic for low-dimensional manifolds in high-dimensional pixel space — the js divergence saturates at \(\log 2\) and gradients to \(G\) die. hence the folklore of carefully rationing discriminator updates.
• non-convergence. simultaneous gradient descent-ascent can orbit the equilibrium instead of approaching it (the classic example: \(\min_x \max_y xy\) spirals). gans oscillate; losses are nearly useless as progress meters, which is why people monitor samples and fid instead.

wasserstein gans, honestly#

wgan (arjovsky et al. 2017, wasserstein gan) replaces the js divergence with the earth-mover (wasserstein-1) distance

\begin{equation} W(p, q) = \inf_{\gamma \in \Pi(p,q)} \mathbb{E}_{(x,y) \sim \gamma} \left[\, \lVert x - y \rVert \,\right], \end{equation}

the minimum cost of transporting the mass of \(p\) onto \(q\). the point: when supports are disjoint, js is a constant but \(W\) still measures how far apart they are — it gives gradients everywhere. by kantorovich-rubinstein duality,

\begin{equation} W(p_{\text{data}}, p_g) = \sup_{\lVert f \rVert_{L} \le 1} \; \mathbb{E}_{x \sim p_{\text{data}}}[f(x)] - \mathbb{E}_{x \sim p_g}[f(x)], \end{equation}

so the discriminator becomes a critic \(f\) (no sigmoid, no log) constrained to be 1-lipschitz. enforcing that constraint is the whole engineering problem:

• weight clipping (original wgan): clamp every weight to \([-c, c]\). crude — it enforces a lipschitz bound but biases the critic toward simple functions, and \(c\) is fiddly. the authors themselves call it “clearly terrible”.
• gradient penalty (wgan-gp, gulrajani et al. 2017, improved training of wasserstein gans): add \(\lambda\, \mathbb{E}_{\hat{x}}\big[(\lVert \nabla_{\hat{x}} f(\hat{x}) \rVert - 1)^2\big]\) with \(\hat{x}\) sampled on lines between real and fake points, exploiting the fact that an optimal critic has unit gradient norm almost everywhere on those lines. much more stable; the standard choice.

honest summary: wgan’s clean theory (a loss that correlates with sample quality, meaningful even for disjoint supports) is bought with an approximation — the critic is trained finitely many steps (typically 5 per generator step), so you never actually have the supremum, and the duality argument holds only at optimality. it trains more robustly in practice; it is not magic.

architectures, two lines#

• dcgan (radford et al. 2015, unsupervised representation learning with dcgans): the recipe that made image gans train at all — all-convolutional \(G\) and \(D\), batchnorm, relu/leakyrelu, no pooling. see cnns for the building blocks.
• stylegan (karras et al. 2019, a style-based generator architecture): maps \(z\) through an mlp to a style vector that modulates each convolution layer separately, giving coarse-to-fine control and the era’s best faces.

evaluation: fid#

likelihood is unavailable, so evaluation compares feature statistics. fréchet inception distance fits gaussians to inception-v3 features of real and generated samples and computes

\begin{equation} \mathrm{FID} = \lVert \mu_r - \mu_g \rVert^2 + \operatorname{Tr}\!\left(\Sigma_r + \Sigma_g - 2\big(\Sigma_r \Sigma_g\big)^{1/2}\right), \end{equation}

the squared wasserstein-2 (fréchet) distance between the two gaussian fits (heusel et al. 2017, gans trained by a two time-scale update rule). lower is better; it punishes both poor fidelity and mode collapse (a collapsed \(p_g\) has tiny \(\Sigma_g\)). caveats: biased at small sample counts and inherits inception-v3’s imagenet-centric view of what matters in an image.

experiment: a 1d gan learns a gaussian#

everything above, in ~70 lines of numpy: target \(\mathcal{N}(3, 0.5)\), a linear generator \(G(z) = Wz + b\) (so \(p_g\) is exactly gaussian: learning the target means learning \(W \to \pm 0.5\), \(b \to 3\)), a 16-unit tanh mlp discriminator, adam, non-saturating generator loss — backprop written by hand.

import numpy as np

rng = np.random.default_rng(0)
MU, SIGMA = 3.0, 0.5      # target: N(3, 0.5)
G_HIDDEN = 0              # 0 = linear generator; try 16 and watch the std collapse

def mlp_init(n_in, n_h, n_out):
    if n_h == 0:          # linear model: G(z) = zW + b, so p_g is exactly gaussian
        return {"W1": np.ones((n_in, n_out)), "b1": np.zeros(n_out)}
    return {"W1": rng.normal(0, 0.5, (n_in, n_h)), "b1": np.zeros(n_h),
            "W2": rng.normal(0, 0.5, (n_h, n_out)), "b2": np.zeros(n_out)}

def forward(p, x):
    if "W2" not in p:
        return None, x @ p["W1"] + p["b1"]
    h = np.tanh(x @ p["W1"] + p["b1"])
    return h, h @ p["W2"] + p["b2"]

def backward(p, x, h, dout):
    """grads of scalar loss w.r.t. params and input, given dL/d(output)."""
    if "W2" not in p:
        return {"W1": x.T @ dout, "b1": dout.sum(0)}, dout @ p["W1"].T
    g = {"W2": h.T @ dout, "b2": dout.sum(0)}
    dh = (dout @ p["W2"].T) * (1 - h**2)          # tanh'
    g["W1"], g["b1"] = x.T @ dh, dh.sum(0)
    return g, dh @ p["W1"].T

def adam(p, g, m, v, t, lr=1e-3):
    for k in p:
        m[k] = 0.9*m[k] + 0.1*g[k]
        v[k] = 0.999*v[k] + 0.001*g[k]**2
        p[k] -= lr * (m[k]/(1-0.9**t)) / (np.sqrt(v[k]/(1-0.999**t)) + 1e-8)

sigmoid = lambda z: 1/(1+np.exp(-z))

G, D = mlp_init(1, G_HIDDEN, 1), mlp_init(1, 16, 1)
mG = {k: 0.0 for k in G}; vG = {k: 0.0 for k in G}
mD = {k: 0.0 for k in D}; vD = {k: 0.0 for k in D}
B = 128

print(f"target: mean={MU}, std={SIGMA}")
print(f"{'step':>6} {'gen mean':>9} {'gen std':>8} {'D(real)':>8} {'D(fake)':>8}")
for t in range(1, 10001):
    # --- discriminator step: ascend log D(x) + log(1 - D(G(z)))
    x_real = rng.normal(MU, SIGMA, (B, 1))
    z = rng.normal(0, 1, (B, 1))
    hG, x_fake = forward(G, z)
    hr, sr = forward(D, x_real); pr = sigmoid(sr)
    hf, sf = forward(D, x_fake); pf = sigmoid(sf)
    gr, _ = backward(D, x_real, hr, (pr - 1)/B)   # descend the negative
    gf, _ = backward(D, x_fake, hf, pf/B)
    adam(D, {k: gr[k] + gf[k] for k in D}, mD, vD, t)

    # --- generator step (non-saturating): ascend log D(G(z))
    z = rng.normal(0, 1, (B, 1))
    hG, x_fake = forward(G, z)
    hf, sf = forward(D, x_fake); pf = sigmoid(sf)
    _, dxf = backward(D, x_fake, hf, (pf - 1)/B)  # d(-log D)/dx_fake
    gG, _ = backward(G, z, hG, dxf)
    adam(G, gG, mG, vG, t)

    if t in (1, 1000, 2000, 4000, 7000, 10000):
        z = rng.normal(0, 1, (4000, 1))
        _, xs = forward(G, z)
        _, sr4 = forward(D, rng.normal(MU, SIGMA, (4000, 1)))
        _, sf4 = forward(D, xs)
        print(f"{t:>6} {xs.mean():>9.3f} {xs.std():>8.3f} "
              f"{sigmoid(sr4).mean():>8.3f} {sigmoid(sf4).mean():>8.3f}")

target: mean=3.0, std=0.5
  step  gen mean  gen std  D(real)  D(fake)
     1    -0.004    0.997    0.739    0.498
  1000     1.406    0.522    0.885    0.129
  2000     2.752    0.000    0.607    0.395
  4000     3.018    0.001    0.505    0.492
  7000     2.977    0.405    0.505    0.497
 10000     3.000    0.508    0.501    0.501

read the last row against the theory: generated mean \(3.000\) and std \(0.508\) against targets \(3.0\) and \(0.5\), with \(D(\text{real}) = D(\text{fake}) = 0.501 \approx \tfrac12\) — the coin-flipping equilibrium \(D^\ast = \tfrac12\) predicted by the optimality proof. the trajectory is also honest about the dynamics: around steps 2000–4000 the std collapses to nearly zero (the generator camps at the target mean) before the discriminator forces it back out — a transient mode collapse and recovery, live.

set G_HIDDEN = 16 (a tanh-mlp generator) and the same run gets the mean right but the std never recovers:

 step  gen mean  gen std  D(real)  D(fake)
    1    -0.011    0.462    0.421    0.500
 1000     3.180    0.120    0.494    0.495
 2000     2.710    0.087    0.503    0.507
 4000     3.823    0.074    0.469    0.495
 7000     2.714    0.033    0.514    0.512
10000     2.783    0.053    0.516    0.512

that is textbook mode collapse — more generator capacity, worse distribution — and a taste of why gan training earned its reputation.

vs. stable diffusion#

the modern comparison the stub asked for: diffusion models replaced gans as the default image generator by trading the adversarial game for a stable regression objective (predict the noise) — no equilibrium to chase, no mode collapse, a usable training loss, at the cost of many network evaluations per sample instead of one. gans survive where single-forward-pass sampling matters (super-resolution, real-time synthesis) and as perceptual losses inside other pipelines — including the adversarial term used to train stable diffusion’s own vae decoder.

see also#

• stable diffusion — the successor regime for image generation
• autoencoders — the other classical route to a latent-variable generator
• cnn from scratch — the convolutional machinery inside dcgan
• feedforward networks — backprop, the thing written by hand above