FFJORD: Free-Form Continuous Dynamics for Scalable Reversible Generative Models
Authors: Will Grathwohl, Ricky T. Q. Chen, Jesse Bettencourt, Ilya Sutskever, David Duvenaud
Keywords: generative models, density estimation, approximate inference, ordinary differential equations
TL;DR: We use continuous time dynamics to define a generative model with exact likelihoods and efficient sampling that is parameterized by unrestricted neural networks.
Abstract: A promising class of generative models maps points from a simple distribution to a complex distribution through an invertible neural network.   Likelihood-based training  of  these  models  requires  restricting  their  architectures  to  allow  cheap computation of Jacobian determinants.  Alternatively, the Jacobian trace can be used if the transformation is specified by an ordinary differential equation. In this paper, we use Hutchinson’s trace estimator to give a scalable unbiased estimate of the log-density.  The result is a continuous-time invertible generative model with unbiased density estimation and one-pass sampling, while allowing unrestricted neural network architectures. We demonstrate our approach on high-dimensional density  estimation,  image  generation,  and  variational  inference,  achieving  the state-of-the-art among exact likelihood methods with efficient sampling.