Denoising

diffusion models generate by learning to undo noise: destroy an image with a fixed gaussian corruption process, train a network to reverse one small step of the destruction, then chain the reversals from pure noise back to data (ho et al. 2020, denoising diffusion probabilistic models). 𐃏 stable diffusion (rombach et al. 2022, high-resolution image synthesis with latent diffusion models) runs this machinery not on pixels but in the latent space of an autoencoder, with a text-conditioned u-net doing the denoising. first the maths, then the architecture.

an autoencoder is a network trained to do the one thing that sounds useless: output its own input. the trick is the obstacle course in the middle — a bottleneck, a corruption, a penalty — that makes verbatim copying impossible, so the network is forced to learn what about the input is worth keeping. 𐃏 the family tree below runs from the linear special case (which is pca wearing a trenchcoat) to the variational autoencoder, which turns the whole construction into a generative model (Goodfellow, Ian, 2016).