Research Scientist, Google Research
- Date & time
- Monday, April 10, 2023 · 1:00 PM
- Location
- Donald Bren Hall 4011
Abstract
Some believe that maximum likelihood is incompatible with high-quality image generation. We provide counter-evidence: diffusion models with SOTA FIDs are actually optimized with the ELBO, with very simple data augmentation (additive noise). We show that diffusion models in the literature are optimized with various objectives that are special cases of a weighted loss, where the weighting function specifies the weight per noise level. Uniform weighting corresponds to maximizing the ELBO, a principled approximation of maximum likelihood. We expose a direct relationship between the weighted loss (with any weighting) and the ELBO objective: the weighted loss can be written as a weighted integral of ELBOs, with one ELBO per noise level. If the weighting function is monotonic, then the weighted loss is a likelihood-based objective. Our main contribution is a deeper theoretical understanding of the diffusion objective.
About the speaker
Durk (Diederik) Kingma does research on principled and scalable methods for machine learning, with a focus on generative models. His contributions include the Variational Autoencoder (VAE), the Adam optimizer, Glow, and Variational Diffusion Models. He obtained a PhD (cum laude) from University of Amsterdam in 2017, and was part of the founding team of OpenAI in 2015.