Diffusion models are latent variable models of the form p theta of x_0 is defined as the integral of p theta of x_0 through x_T with respect to x_1 through x_T, where x_1 through x_T are latents of the same dimensionality as the data x_0 sampled from q of x_0. The joint distribution p theta of x_0 through x_T is called the reverse process, and it is defined as a Markov chain with learned Gaussian transitions starting at p of x_T, which is a standard normal distribution. What distinguishes diffusion models from other types of latent variable models is that the approximate posterior q of x_1 through x_T given x_0, called the forward process or diffusion process, is fixed to a Markov chain that gradually adds Gaussian noise to the data according to a variance schedule beta_1 through beta_T. Training is performed by optimizing the usual variational bound on negative log likelihood. The forward process variances beta_t can be learned by reparameterization or held constant as hyperparameters, and expressiveness of the reverse process is ensured in part by the choice of Gaussian conditionals in p theta of x_t-1 given x_t, because both processes have the same functional form when beta_t are small. A notable property of the forward process is that it admits sampling x_t at an arbitrary timestep t in closed form: using the notation alpha_t is defined as 1 minus beta_t and alpha_bar_t is defined as the product of alpha_s from s equals 1 to t. Efficient training is therefore possible by optimizing random terms of L with stochastic gradient descent. Further improvements come from variance reduction by rewriting L. This rewritten form uses KL divergence to directly compare p theta of x_t-1 given x_t against forward process posteriors, which are tractable when conditioned on x_0. Consequently, all KL divergences are comparisons between Gaussians, so they can be calculated in a Rao-Blackwellized fashion with closed form expressions instead of high variance Monte Carlo estimates. Section 3: Diffusion models and denoising autoencoders  Diffusion models might appear to be a restricted class of latent variable models, but they allow a large number of degrees of freedom in implementation. One must choose the variances beta_t of the forward process and the model architecture and Gaussian distribution parameterization of the reverse process. To guide our choices, we establish a new explicit connection between diffusion models and denoising score matching that leads to a simplified, weighted variational bound objective for diffusion models. Ultimately, our model design is justified by simplicity and empirical results. Our discussion is categorized by the terms of the variational bound. Subsection 3.1: Forward process and L_T  We ignore the fact that the forward process variances beta_t are learnable by reparameterization and instead fix them to constants. Thus, in our implementation, the approximate posterior q has no learnable parameters, so L_T is a constant during training and can be ignored. Subsection 3.2: Reverse process and L_1 through L_T-1  Now we discuss our choices in the reverse process for t between 1 and T. First, we set the covariance matrix to untrained time dependent constants. Experimentally, both sigma_t squared equals beta_t and sigma_t squared equals a scaled version of beta_t had similar results. The first choice is optimal for x_0 sampled from a standard normal distribution, and the second is optimal for x_0 deterministically set to one point. These are the two extreme choices corresponding to upper and lower bounds on reverse process entropy for data with coordinatewise unit variance. Second, to represent the mean, we propose a specific parameterization motivated by the following analysis of L_t. We see that the most straightforward parameterization of the mean is a model that predicts the forward process posterior mean. However, we can expand this term further. This reveals that the mean must predict a specific function given x_t. Since x_t is available as input to the model, we may choose a parameterization where a function approximator is intended to predict epsilon from x_t. To sample x_t-1 is to compute a value based on x_t, the function approximator, and some added noise. The complete sampling procedure resembles Langevin dynamics with the function approximator as a learned gradient of the data density. Furthermore, with this parameterization, the variational bound term simplifies to a form which resembles denoising score matching over multiple noise scales indexed by t. As this term is equal to one term of the variational bound for the Langevin-like reverse process, we see that optimizing an objective resembling denoising score matching is equivalent to using variational inference to fit the finite-time marginal of a sampling chain resembling Langevin dynamics. To summarize, we can train the reverse process mean function approximator to predict the forward process posterior mean, or by modifying its parameterization, we can train it to predict epsilon. (There is also the possibility of predicting x_0, but we found this to lead to worse sample quality early in our experiments.) We have shown that the epsilon-prediction parameterization both resembles Langevin dynamics and simplifies the diffusion model's variational bound to an objective that resembles denoising score matching. Nonetheless, it is just another parameterization of the reverse process, so we verify its effectiveness in our experiments in an ablation where we compare predicting epsilon against predicting the forward process posterior mean. Subsection 3.3: Data scaling, reverse process decoder, and L_0  We assume that image data consists of integers from 0 to 255 scaled linearly to the range of -1 to 1. This ensures that the neural network reverse process operates on consistently scaled inputs starting from the standard normal prior. To obtain discrete log likelihoods, we set the last term of the reverse process to an independent discrete decoder derived from a Gaussian distribution. (It would be straightforward to instead incorporate a more powerful decoder like a conditional autoregressive model, but we leave that to future work.) Similar to the discretized continuous distributions used in VAE decoders and autoregressive models, our choice here ensures that the variational bound is a lossless codelength of discrete data, without need of adding noise to the data or incorporating the Jacobian of the scaling operation into the log likelihood. At the end of sampling, we display the mean of the distribution at time 1 noiselessly. Subsection 3.4: Simplified training objective  With the reverse process and decoder defined above, the variational bound is clearly differentiable with respect to theta and is ready to be employed for training. However, we found it beneficial to sample quality (and simpler to implement) to train on a variant of the variational bound. This variant is a simplified objective where t is uniform between 1 and T. The t equals 1 case corresponds to L_0. The t greater than 1 cases correspond to an unweighted version of the variational bound term, analogous to the loss weighting used by the NCSN denoising score matching model. (L_T does not appear because the forward process variances beta_t are fixed.) The complete training procedure uses this simplified objective. Since our simplified objective discards the weighting in the original term, it is a weighted variational bound that emphasizes different aspects of reconstruction compared to the standard variational bound. In particular, our diffusion process setup causes the simplified objective to down-weight loss terms corresponding to small t. These terms train the network to denoise data with very small amounts of noise, so it is beneficial to down-weight them so that the network can focus on more difficult denoising tasks at larger t terms. We will see in our experiments that this reweighting leads to better sample quality. Section 4: Experiments  We set T equals 1000 for all experiments so that the number of neural network evaluations needed during sampling matches previous work. We set the forward process variances to constants increasing linearly from beta_1 equals 10 to the power of -4 to beta_T equals 0.02. These constants were chosen to be small relative to data scaled to the range of -1 to 1, ensuring that reverse and forward processes have approximately the same functional form while keeping the signal-to-noise ratio at x_T as small as possible. To represent the reverse process, we use a U-Net backbone similar to an unmasked PixelCNN++ with group normalization throughout. Parameters are shared across time, which is specified to the network using the Transformer sinusoidal position embedding. We use self-attention at the 16 by 16 feature map resolution. Subsection 4.1: Sample quality  On CIFAR10, with our FID score of 3.17, our unconditional model achieves better sample quality than most models in the literature, including class conditional models. Our FID score is computed with respect to the training set, as is standard practice; when we compute it with respect to the test set, the score is 5.24, which is still better than many of the training set FID scores in the literature. We find that training our models on the true variational bound yields better codelengths than training on the simplified objective, as expected, but the latter yields the best sample quality. We also generated samples for CIFAR10, CelebA-HQ 256 by 256, and LSUN 256 by 256. Subsection 4.2: Reverse process parameterization and training objective ablation  We show the sample quality effects of reverse process parameterizations and training objectives. We find that the baseline option of predicting the forward process posterior mean works well only when trained on the true variational bound instead of unweighted mean squared error, a simplified objective. We also see that learning reverse process variances leads to unstable training and poorer sample quality compared to fixed variances. Predicting epsilon, as we proposed, performs approximately as well as predicting the posterior mean when trained on the variational bound with fixed variances, but much better when trained with our simplified objective. Subsection 4.3: Progressive coding  The codelengths of our CIFAR10 models show a gap between train and test of at most 0.03 bits per dimension, which is comparable to the gaps reported with other likelihood-based models and indicates that our diffusion model is not overfitting. Still, while our lossless codelengths are better than the large estimates reported for energy based models and score matching using annealed importance sampling, they are not competitive with other types of likelihood-based generative models. Since our samples are nonetheless of high quality, we conclude that diffusion models have an inductive bias that makes them excellent lossy compressors. Treating the variational bound terms L_1 through L_T as rate and L_0 as distortion, our CIFAR10 model with the highest quality samples has a rate of 1.78 bits per dimension and a distortion of 1.97 bits per dimension, which amounts to a root mean squared error of 0.95 on a scale from 0 to 255. More than half of the lossless codelength describes imperceptible distortions. Paragraph: Progressive lossy compression. We can probe further into the rate-distortion behavior of our model by introducing a progressive lossy code. This code assumes access to a procedure that can transmit a sample using approximately a certain number of bits on average for any distributions p and q, for which only p is available to the receiver beforehand. When applied to a data sample, this procedure transmits x_T through x_0 in sequence using a total expected codelength equal to the variational bound. The receiver, at any time t, has the partial information x_t fully available and can progressively estimate x_0. A rate-distortion plot on the CIFAR10 test set shows the resulting behavior. At each time t, the distortion is calculated as the root mean squared error, and the rate is calculated as the cumulative number of bits received so far. The distortion decreases steeply in the low-rate region of the rate-distortion plot, indicating that the majority of the bits are indeed allocated to imperceptible distortions. Paragraph: Progressive generation. We also run a progressive unconditional generation process given by progressive decompression from random bits. In other words, we predict the result of the reverse process, a predicted x_0, while sampling from the reverse process. The resulting sample quality of the predicted x_0 over the course of the reverse process shows that large scale image features appear first and details appear last. Stochastic predictions of x_0 given x_t for various t show that when t is small, all but fine details are preserved, and when t is large, only large scale features are preserved. Perhaps these are hints of conceptual compression. Paragraph: Connection to autoregressive decoding. Note that the variational bound can be rewritten in an alternate form. Now consider setting the diffusion process length T to the dimensionality of the data, defining the forward process so that it masks out coordinates one by one, setting the prior to place all mass on a blank image, and taking the reverse process to be a fully expressive conditional distribution. With these choices, minimizing the KL divergence trains the model to predict the masked coordinate given the others. Thus, training with this particular diffusion is training an autoregressive model. We can therefore interpret the Gaussian diffusion model as a kind of autoregressive model with a generalized bit ordering that cannot be expressed by reordering data coordinates.