Original: L EARNING E NERGY -B ASED M ODELS BY D IFFUSION R ECOVERY L IKELIHOOD
Revised: L EARNING E NERGY -B ASED M ODELS BY D IFFUSION R ECOVERY L IKELIHOOD

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: While energy-based models (EBMs) exhibit a number of desirable properties, training and sampling on high-dimensional datasets remains challenging.
Revised: While energy-based models (EBMs) exhibit a number of desirable properties, training and sampling on high-dimensional datasets remains challenging.

Original: Inspired by recent progress on diffusion probabilistic models, we present a diffusion recovery likelihood method to tractably learn and sample from a sequence of EBMs trained on increasingly noisy versions of a dataset.
Revised: Inspired by recent progress on diffusion probabilistic models, we present a diffusion recovery likelihood method to tractably learn and sample from a sequence of EBMs trained on increasingly noisy versions of a dataset.

Original: Each EBM is trained with recovery likelihood, which maximizes the conditional probability of the data at a certain noise level given their noisy versions at a higher noise level.
Revised: Each EBM is trained with recovery likelihood, which maximizes the conditional probability of the data at a certain noise level given their noisy versions at a higher noise level.

Original: Optimizing recovery likelihood is more tractable than marginal likelihood, as sampling from the conditional distributions is much easier than sampling from the marginal distributions.
Revised: Optimizing recovery likelihood is more tractable than marginal likelihood, as sampling from the conditional distributions is much easier than sampling from the marginal distributions.

Original: After training, synthesized images can be generated by the sampling process that initializes from Gaussian white noise distribution and progressively samples the conditional distributions at decreasingly lower noise levels.
Revised: After training, synthesized images can be generated by the sampling process that initializes from Gaussian white noise distribution and progressively samples the conditional distributions at decreasingly lower noise levels.

Original: Our method generates high ﬁdelity samples on various image datasets.
Revised: Our method generates high ﬁdelity samples on various image datasets.

Original: On unconditional CIFARour method achieves FID 9.60 and inception score 8.58, superior to the majority of GANs.
Revised: On unconditional CIFARour method achieves FID 9.58 and inception score 8.30, superior to the majority of GANs.

Original: Moreover, we demonstrate that unlike previous work on EBMs, our long-run MCMC samples from the conditional distributions do not diverge and still represent realistic images, allowing us to accurately estimate the normalized density of data even for high-dimensional datasets.
Revised: Moreover, we demonstrate that unlike previous work on EBMs, our long-run MCMC samples from the conditional distributions do not diverge and still represent realistic images, allowing us to accurately estimate the normalized density of data even for high-dimensional datasets.

Revised: Our implementation is available at https://github.com/ruiqigao/recovery_likelihood .

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: EBMs (LeCun et al., 2006; Ngiam et al., 2011; Kim & Bengio, 2016; Zhao et al., 2016; Goyal et al., 2017; Xie et al., 2016b; Finn et al., 2016; Gao et al., 2018; Kumar et al., 2019; Nijkamp et al., 2019b; Du & Mordatch, 2019; Grathwohl et al., 2019; Desjardins et al., 2011; Gao et al., 2020; Che et al., 2020; Grathwohl et al., 2020; Qiu et al., 2019; Rhodes et al., 2020) are an appealing class of probabilistic models, which can be viewed as generative versions of discriminators (Jin et al., 2017;
Revised: EBMs (LeCun et al., 2006; Ngiam et al., 2011; Kim & Bengio, 2016; Zhao et al., 2016; Goyal et al., 2017; Xie et al., 2016b; Finn et al., 2016; Gao et al., 2018; Kumar et al., 2019; Nijkamp et al., 2019b; Du & Mordatch, 2019; Grathwohl et al., 2019; Desjardins et al., 2011; Gao et al., 2020; Che et al., 2020; Grathwohl et al., 2020; Qiu et al., 2019; Rhodes et al., 2020) are an appealing class of probabilistic models, which can be viewed as generative versions of discriminators (Jin et al., 2017;

Original: Lazarow et al., 2017; Lee et al., 2018; Grathwohl et al., 2020), yet can be learned from unlabeled data.
Revised: Lazarow et al., 2017; Lee et al., 2018; Grathwohl et al., 2020), yet can be learned from unlabeled data.

Original: Despite a number of desirable properties, two challenges remain for training EBMs on highdimensional datasets.
Revised: Despite a number of desirable properties, two challenges remain for training EBMs on highdimensional datasets.

Original: First, learning EBMs by maximum likelihood requires Markov Chain Monte
Revised: First, learning EBMs by maximum likelihood requires Markov Chain Monte

Original: Carlo (MCMC) to generate samples from the model, which can be extremely expensive.
Revised: Carlo (MCMC) to generate samples from the model, which can be extremely expensive.

Original: Second, as pointed out in Nijkamp et al.
Revised: Second, as pointed out in Nijkamp et al.

Original: (2019a), the energy potentials learned with non-convergent MCMC do not have a valid steady-state, in the sense that samples from long-run Markov chains can differ greatly from observed samples, making it difﬁcult to evaluate the learned energy potentials.
Revised: (2019a), the energy potentials learned with non-convergent MCMC do not have a valid steady-state, in the sense that samples from long-run Markov chains can differ greatly from observed samples, making it difﬁcult to evaluate the learned energy potentials.

Original: Another line of work, originating from Sohl-Dickstein et al. (2015), is to learn from a diffused version of the data, which are obtained from the original data via a diffusion process that sequentially adds Gaussian white noise.
Revised: Another line of work, originating from Sohl-Dickstein et al. (2015), is to learn from a diffused version of the data, which are obtained from the original data via a diffusion process that sequentially adds Gaussian white noise.

Original: From such diffusion data, one can learn the conditional model of the data at a certain noise level given their noisy versions at the higher noise level.
Revised: From such diffusion data, one can learn the conditional model of the data at a certain noise level given their noisy versions at the higher noise level of the diffusion process.

Original: After learning the sequence of conditional models that invert the diffusion process, one can  generate synthesized images from Gaussian white noise images by ancestral sampling.
Revised: After learning the sequence of conditional models that invert the diffusion process, one can then generate synthesized images from Gaussian white noise images by ancestral sampling.

Original: Building on Sohl-Dickstein et al.
Revised: Building on Sohl-Dickstein et al.

Original: (2015), Ho et al.
Revised: (2015), Ho et al.

Original: (2020) further developed the method, obtaining strong image synthesis results.
Revised: (2020) further developed the method, obtaining strong image [Figure] results

Original: Inspired by Sohl-Dickstein et al.
Revised: Inspired by Sohl-Dickstein et al.

Original: (2015) and Ho et al.
Revised: (2015) and Ho et al.

Original: (2020), we propose a diffusion recovery likelihood method to tackle the challenge of training EBMs directly on a dataset by instead learning a sequence of EBMs for the marginal distributions of the diffusion process.
Revised: (2020), we propose a diffusion recovery likelihood method to tackle the challenge of training EBMs directly on a dataset by instead learning a sequence of EBMs for the marginal distributions of the diffusion process.

Original: The sequence of marginal
Revised: The sequence of marginal

Original: EBMs are learned with recovery likelihoods that are deﬁned as the conditional distributions that invert the diffusion process.
Revised: EBMs are learned with recovery likelihoods that are deﬁned as the conditional distributions that invert the diffusion process.

Original: Compared to standard maximum likelihood estimation (MLE) of EBMs, learning marginal EBMs by diffusion recovery likelihood only requires sampling from the conditional distributions, which is much easier than sampling from the marginal distributions.
Revised: Compared to standard maximum likelihood estimation (MLE) of EBMs, learning marginal EBMs by diffusion recovery likelihood only requires sampling from the conditional distributions, which is much easier than sampling from the marginal distributions.

Original: After learning the marginal EBMs, we can generate synthesized images by a sequence of conditional samples initialized from the Gaussian white noise distribution.
Revised: After learning the marginal EBMs, we can generate synthesized images by a sequence of conditional samples initialized from the Gaussian white noise distribution.

Original: Unlike Ho et al.
Revised: Unlike Ho et al.

Original: (2020) that approximates the reverse process by normal distributions, in our case the conditional distributions are derived from the marginal EBMs, which are more ﬂexible.
Revised: (2020) that approximates the reverse process by normal distributions, in our case the conditional distributions are derived from the marginal EBMs, which are more ﬂexible.

Original: The framework of recovery likelihood was originally proposed in Bengio et al.
Revised: The framework of recovery likelihood was originally proposed in Bengio et al.

Original: In our work, we adapt it to learning the sequence of marginal
Revised: In our work, we adapt it to learning the sequence of marginal

Original: EBMs from the diffusion data.
Revised: EBMs from the diffusion data.

Original: Rhodes et al.

Original: (2020) proposes a related framework where they learn the sequence of EBMs based on noise contrastive estimation.

Original: Our work is also related to the denoising score matching method of Vincent (2011), which was further developed by Song & Ermon (2019; 2020) for learning from diffusion data.
Revised: Our work is also related to the denoising score matching method of Vincent (2011), which was further developed by Song & Ermon (2019; 2020) for learning from diffusion data.

Original: The training objective used for diffusion probabilisitic models is a weighted version of the denoising score matching objective, as revealed by Ho et al.
Revised: The training objective used for diffusion probabilisitic models is a weighted version of the denoising score matching objective, as revealed by Ho et al.

Original: These methods learn the score functions (the gradients of the energy functions) directly, instead of using the gradients of learned energy functions as in
Revised: These methods learn the score functions (the gradients of the energy functions) directly, instead of using the gradients of learned energy functions as in

Original: EBMs.
Revised: EBMs.

Original: On the other hand, Saremi et al.
Revised: On the other hand, Saremi et al.

Original: (2018) parametrizes the score function as the gradient of a
Revised: (2018) parametrizes the score function as the gradient of a

Original: MLP energy function, and Saremi & Hyvarinen (2019) further uniﬁes denoising score matching and neural empirical Bayes.
Revised: MLP energy function, and Saremi & Hyvarinen (2019) further uniﬁes denoising score matching and neural empirical Bayes.

Original: We demonstrate the efﬁcacy of diffusion recovery likelihood on CIFAR-10, CelebA and LSUN datasets.
Revised: We demonstrate the efﬁcacy of diffusion recovery likelihood on CIFAR-10, CelebA and LSUN datasets.

Original: The generated samples are of high ﬁdelity and comparable to GAN-based methods.
Revised: The generated samples are of high ﬁdelity and comparable to GAN-based methods.

Original: On
Revised: On

Original: CIFAR-10, we achieve FID 9.60 and inception score 8.58, exceeding existing methods of learning explicit EBMs to a large extent.
Revised: CIFAR-10, we achieve FID 9.58 and inception score 8.30, exceeding existing methods of learning explicit EBMs to a large extent.

Original: We also demonstrate that diffusion recovery likelihood outperforms denoising score matching from diffusion data if we naively take the gradients of explicit energy functions as the score functions.
Revised: We also demonstrate that diffusion recovery likelihood outperforms denoising score matching from diffusion data if we naively take the gradients of explicit energy functions as the score functions.

Original: More interestingly, by using a thousand diffusion time steps, we demonstrate that even very long MCMC chains from the sequence of conditional distributions produce samples that represent realistic images.
Revised: More interestingly, by using a thousand diffusion time steps, we demonstrate that even very long MCMC chains from the sequence of conditional distributions produce samples that represent realistic images.

Original: With the faithful long-run MCMC samples from the conditional distributions, we can accurately estimate the marginal partition function at zero noise level by importance sampling, and thus evaluate the normalized density of data under the EBM.
Revised: With the faithful long-run MCMC samples from the conditional distributions, we can accurately estimate the marginal partition function at zero noise level by importance sampling, and thus evaluate the normalized density of data under the EBM.

Original:  B ACKGROUND
Revised:  B ACKGROUND

Original: Let x ∼ p data ( x ) denote a training example, and p θ ( x ) denote a model’s probability density function that aims to approximates p data ( x ) .
Revised: Let x ∼ p data ( x ) denote a training example, and p θ ( x ) denote a model’s probability density function that aims to approximates p data ( x ) .

Original: An energy-based model (EBM) is deﬁned as:
Revised: An energy-based model (EBM) is deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: where Z θ = (cid:82) exp( f θ ( x ))
Revised: where Z θ = (cid:82) exp( f θ ( x ))

Original: d x is the partition function, which is analytically intractable for highdimensional x .
Revised: d x is the partition function, which is analytically intractable for highdimensional x .

Original: For images, we parameterize f θ ( x ) with a convolutional neural network with a scalar output.
Revised: For images, we parameterize f θ ( x ) with a convolutional neural network with a scalar output.

Original: The energy-based model in equation 1 can, in principle, be learned through MLE.
Revised: The energy-based model in equation 1 can, in principle, be learned through MLE.

Original: Speciﬁcally, suppose we observe samples x i ∼ p data ( x ) for i = 1 , 2 , ..., n .
Revised: Speciﬁcally, suppose we observe samples x i ∼ p data ( x ) for i = 1 , 2 , ..., n .

Original: The log-likelihood function is
Revised: The log-likelihood function is

Original: [Equation]
Revised: [Equation]

Original: In MLE, we seek to maximize the log-likelihood function, where the gradient approximately follows (Xie et al., 2016b)
Revised: In MLE, we seek to maximize the log-likelihood function, where the gradient approximately follows (Xie et al., 2016b)

Original: [Equation]
Revised: [Equation]

Original: The expectations can be approximated by averaging over the observed samples and the synthesized samples drawn from the model distribution p θ ( x ) respectively.
Revised: The expectations can be approximated by averaging over the observed samples and the synthesized samples drawn from the model distribution p θ ( x ) respectively.

Original: Generating synthesized samples from p θ ( x ) can be done with Markov Chain Monte Carlo (MCMC) such as Langevin dynamics (or
Revised: Generating synthesized samples from p θ ( x ) can be done with Markov Chain Monte Carlo (MCMC) such as Langevin dynamics (or

Original: Hamiltonian Monte Carlo (Girolami & Calderhead, 2011)), which iterates
Revised: Hamiltonian Monte Carlo (Girolami & Calderhead, 2011)), which iterates

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where τ indexes the time, δ is the step size, and (cid:15)
Revised: where τ indexes the time, δ is the step size, and (cid:15)

Original: τ ∼
Revised: τ ∼

Original: N (0 , I ) .
Revised: N (0 , I ) .

Original: The difﬁculty lies in the fact that for highdimensional and multi-modal distributions, MCMC sampling can take a long time to converge, and the sampling chains may have difﬁculty traversing modes.
Revised: The difﬁculty lies in the fact that for highdimensional and multi-modal distributions, MCMC sampling can take a long time to converge, and the sampling chains may have difﬁculty traversing modes.

Original: As demonstrated in Figure 2, training EBMs with synthesized samples from non-convergent MCMC results in malformed energy landscapes (Nijkamp et al., 2019b), even if the samples from the model look reasonable.Qiu et al. (2019) proposes coupled MCMC to tackle this problem.
Revised: As demonstrated in Figure 2, training EBMs with synthesized samples from non-convergent MCMC results in malformed energy landscapes (Nijkamp et al., 2019b), even if the samples from the model look reasonable.

Original:  R ECOVERY L IKELIHOOD 3. F ROM M ARGINAL TO C ONDITIONAL
Revised:  R ECOVERY L IKELIHOOD 3. F ROM M ARGINAL TO C ONDITIONAL

Original: Given the difﬁculty of sampling from the marginal density p θ ( x ) , following Bengio et al.
Revised: Given the difﬁculty of sampling from the marginal density p θ ( x ) , following Bengio et al.

Original: (2013), we use the recovery likelihood deﬁned by the density of the observed sample conditional on a noisy sample perturbed by isotropic Gaussian noise.
Revised: (2013), we use the recovery likelihood deﬁned by the density of the observed sample conditional on a noisy sample perturbed by isotropic Gaussian noise.

Original: Speciﬁcally, let ˜ x
Revised: Speciﬁcally, let ˜ x

Original: = x + σ (cid:15) be the noisy observation of x , where (cid:15) ∼ N (0 , I ) .
Revised: = x + σ (cid:15) be the noisy observation of x , where (cid:15) ∼ N (0 , I ) .

Original: Suppose p θ ( x ) is deﬁned by the EBM in equation 1, then the conditional EBM can be derived as
Revised: Suppose p θ ( x ) is deﬁned by the EBM in equation 1, then the conditional EBM can be derived as

Original: [Equation]
Revised: [Equation]

Original: where ˜ Z θ (˜ x ) = (cid:82) exp (cid:0) f
Revised: where ˜ Z [Equation] = (cid:82) [Equation]

Revised: (cid:0)

Revised: [Equation]

Revised: ( x )

Original: θ ( x ) − 12 σ 2 (cid:107) ˜ x − x (cid:107) 2 (cid:1) d x is the partition function of this conditional EBM.
Revised: [Equation] [Equation] 2 (cid:107) ˜ x − x (cid:107) 2 (cid:1) d x is the partition function of this conditional EBM.

Original: See Appendix A.1 for the derivation.
Revised: See Appendix A.1 for the derivation.

Original: Compared to p θ ( x ) (equation 1), the extra quadratic term 12 σ 2 (cid:107) ˜ x − x (cid:107) 2 in p θ ( x | ˜ x ) constrains the energy landscape to be localized around ˜ x , making the latter less multi-modal and easier to sample from.
Revised: Compared to p θ ( x ) (equation 1), the extra quadratic term 12 σ 2 (cid:107) ˜ x − x (cid:107) 2 in p θ ( x | ˜ x ) constrains the energy landscape to be localized around ˜ x , making the latter less multi-modal and easier to sample from.

Original: As we will show later, when σ is small, p θ ( x | ˜ x ) is approximately a single mode Gaussian distribution, which greatly reduces the burden of MCMC.
Revised: As we will show later, when σ is small, p θ ( x | ˜ x ) is approximately a single mode Gaussian distribution, which greatly reduces the burden of MCMC.

Original: A more general formulation is ˜ x
Revised: A more general formulation is ˜ x

Original: = a x + σ (cid:15) , where a is a positive constant.
Revised: = a x + σ (cid:15) , where a is a positive constant.

Original: In that case, we can let y =
Revised: In that case, we can let y =

Original: a x , and treat y as the observed sample.
Revised: a x , and treat y as the observed sample.

Original: Assume p θ ( y )
Revised: Assume p θ ( y )

Original: = 1 Z θ exp( f θ ( y )) , then by change of variable , the density function of x can be derived as g θ ( x ) =
Revised: = 1 Z θ exp( f θ ( y )) , then by change of variable , the density function of x can be derived as g θ ( x ) =

Original: ap θ ( a x ) .
Revised: ap θ ( a x ) .

Original: 3. M AXIMIZING RECOVERY LIKELIHOOD
Revised: 3. M AXIMIZING RECOVERY LIKELIHOOD

Original: With the conditional EBM, assume we have observed samples x i ∼ p data ( x ) and the corresponding perturbed samples ˜ x
Revised: With the conditional EBM, assume we have observed samples x i ∼ p data ( x ) and the corresponding perturbed samples ˜ x

Original: i = x
Revised: i = x

Original: i + σ (cid:15) i for i
Revised: i + σ (cid:15) i for i

Original: = 1 , ..., n .
Revised: = 1 , ..., n .

Original: We deﬁne the recovery log-likelihood function as
Revised: We deﬁne the recovery log-likelihood function as

Original: [Equation]
Revised: [Equation]

Original: The term recovery indicates that we attempt to recover the clean sample
Revised: The term recovery indicates that we attempt to recover the clean sample

Original: x i from the noisy sample ˜ x i .
Revised: x i from the noisy sample ˜ x i .

Original: Thus, instead of maximizing L ( θ ) in equation 2, we can maximize J ( θ ) , whose distributions are easier to sample from.
Revised: Thus, instead of maximizing L ( θ ) in equation 2, we can maximize J ( θ ) , whose distributions are easier to sample from.

Original: Speciﬁcally, we generate synthesized samples by K steps of Langevin dynamics that iterates
Revised: Speciﬁcally, we generate synthesized samples by K steps of Langevin dynamics that iterates

Original: [Equation]
Revised: [Equation]

Original: The model is then updated following the same learning gradients as MLE (equation 3), because the quadratic term − 12 σ 2 (cid:107) ˜ x − x (cid:107) 2 is not related to θ .
Revised: The model is then updated following the same learning gradients as MLE (equation 3), because the quadratic term − 12 σ 2 (cid:107) ˜ x − x (cid:107) 2 is not related to θ .

Original: Following the classical analysis of MLE, we can show that the point estimate given by maximizing recovery likelihood is an unbiased estimator of the true parameters, which means that given enough data, a rich enough model and exact synthesis, maximizing the recovery likelihood learns θ such that p data ( x ) =
Revised: Following the classical analysis of MLE, we can show that the point estimate given by maximizing recovery likelihood is an unbiased estimator of the true parameters, which means that given enough data, a rich enough model and exact synthesis, maximizing the recovery likelihood learns θ such that p data ( x ) =

Original: p θ ( x ) .
Revised: p θ ( x ) .

Original: See Appendix A.2 for a theoretical explanation.
Revised: See Appendix A.2 for a theoretical explanation.

Original: 3. N ORMAL A PPROXIMATION TO C ONDITIONAL
Revised: 3. N ORMAL A PPROXIMATION TO C ONDITIONAL

Original: When the variance of perturbed noise σ 2 is small, p θ ( x | ˜ x ) can be approximated by a normal distribution via a ﬁrst order Taylor expansion at ˜ x .
Revised: When the variance of perturbed noise σ 2 is small, p θ ( x | ˜ x ) can be approximated by a normal distribution via a ﬁrst order Taylor expansion at ˜ x .

Original: Speciﬁcally, the negative conditional energy is
Revised: Speciﬁcally, the negative conditional energy is

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: [Equation]

Original: where c include terms irrelevant of x (see Appendix A.3 for a detailed derivation).
Revised: where c include terms irrelevant of x (see Appendix A.3 for a detailed derivation).

Original: In the above approximation, we do not perform second order Taylor expansion because σ 2 is small, and (cid:107) ˜ x − x (cid:107) 2 / 2 σ 2 will dominate all the second order terms from Taylor expansion.
Revised: In the above approximation, we do not perform second order Taylor expansion because σ 2 is small, and (cid:107) ˜ x − x (cid:107) 2 / 2 σ 2 will dominate all the second order terms from Taylor expansion.

Original: Thus we can approximate p θ ( x | ˜ x ) by a Gaussian approximation (cid:101) p θ ( x | ˜ x ) :
Revised: Thus we can approximate p θ ( x | ˜ x ) by a Gaussian approximation (cid:101) p θ ( x | ˜ x ) :

Original: [Equation]
Revised: [Equation]

Original: We can sample from this distribution using:
Revised: We can sample from this distribution using:

Original: [Equation]
Revised: [Equation]

Original: where (cid:15) ∼ N (0 , I ) .
Revised: where (cid:15) ∼ N (0 , I ) .

Original: This resembles a single step of Langevin dynamics, except that σ (cid:15) is replaced by √ 2 σ (cid:15) in Langevin dynamics.
Revised: This resembles a single step of Langevin dynamics, except that σ (cid:15) is replaced by √ 2 σ (cid:15) in Langevin dynamics.

Original: This normal approximation has two traits: (1) it veriﬁes the fact that the conditional density p θ ( x | ˜ x ) can be generally easier to sample from when σ is small; (2) it provides hints of choosing the step size of Langevin dynamics, as discussed in section 3.5.
Revised: This normal approximation has two traits: (1) it veriﬁes the fact that the conditional density p θ ( x | ˜ x ) can be generally easier to sample from when σ is small; (2) it provides hints of choosing the step size of Langevin dynamics, as discussed in section 3.5.

Original: 3. C ONNECTION TO VARIATIONAL INFERENCE AND SCORE MATCHING
Revised: 3. C ONNECTION TO VARIATIONAL INFERENCE AND SCORE MATCHING

Original: The normal approximation to the conditional distribution leads to a natural connection to diffusion probabilistic models (Sohl-Dickstein et al., 2015; Ho et al., 2020) and denoising score matching (Vincent, 2011; Song & Ermon, 2019; 2020; Saremi et al., 2018; Saremi & Hyvarinen, 2019).
Revised: The normal approximation to the conditional distribution leads to a natural connection to diffusion probabilistic models (Sohl-Dickstein et al., 2015; Ho et al., 2020) and denoising score matching (Vincent, 2011; Song & Ermon, 2019; 2020; Saremi et al., 2018; Saremi & Hyvarinen, 2019).

Original: Speciﬁcally, in diffusion probabilistic models, instead of modeling p θ ( x ) as an energy-based model, it recruits variational inference and directly models the conditional density as 
Revised: Speciﬁcally, in diffusion probabilistic models, instead of modeling p θ ( x ) as an energy-based model, it recruits variational inference and directly models the conditional density as [Equation]

Original: [Equation]
Revised: [Equation]

Original: which is in agreement with the normal approximation (equation 11), with s θ ( x ) =
Revised: which is in agreement with the normal approximation (equation 11), with s θ ( x ) =

Original: ∇ x f θ ( x ) .
Revised: ∇ x f θ ( x ) .

Original: On the other hand, the training objective of denoising score matching is to minimize
Revised: On the other hand, the training objective of denoising score matching is to minimize

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original:  where s θ ( x ) is the score of the density of ˜ x .
Revised: 14) where s θ ( x ) is the score of the density of ˜ x .

Original: This objective is in agreement with the objective of maximizing log-likelihood of the normal approximation (equation 10), except that for normal approximation, ∇ x f θ ( · ) is the score of density of x , instead of ˜ x .
Revised: This objective is in agreement with the objective of maximizing log-likelihood of the normal approximation (equation 10), except that for normal approximation, ∇ x f θ ( · ) is the score of density of x , instead of ˜ x .

Original: However, the difference between the scores of density of x and ˜ x is of O ( σ 2 ) , which is negligible when σ is sufﬁciently small (see Appendix A.4 for details).
Revised: However, the difference between the scores of density of x and ˜ x is of O ( σ 2 ) , which is negligible when σ is sufﬁciently small (see Appendix A.4 for details).

Original: We can further show that the learning gradient of maximizing log-likelihood of the normal approximation is approximately the same as the learning gradient of maximizing the original recovery log-likelihood with one step of Langevin dynamics (see Appendix A.5).
Revised: We can further show that the learning gradient of maximizing log-likelihood of the normal approximation is approximately the same as the learning gradient of maximizing the original recovery log-likelihood with one step of Langevin dynamics (see Appendix A.5).

Original: It indicates that the training process of maximizing recovery likelihood agrees with the one of diffusion probabilistic models and denoising score matching when σ is small.
Revised: It indicates that the training process of maximizing recovery likelihood agrees with the one of diffusion probabilistic models and denoising score matching when σ is small.

Original: As the normal approximation is accurate only when σ is small, it requires many time steps in the diffusion process for this approximation to work well, which is also reported in Ho et al.
Revised: As the normal approximation is accurate only when σ is small, it requires many time steps in the diffusion process for this approximation to work well, which is also reported in Ho et al.

Original: and Song & Ermon (2020).
Revised: and Song & Ermon (2020).

Original: In contrast, the diffusion recovery likelihood framework can be more ﬂexible in choosing the number of time steps and the magnitude of σ .
Revised: In contrast, the diffusion recovery likelihood framework can be more ﬂexible in choosing the number of time steps and the magnitude of σ .

Original: 3. D IFFUSION RECOVERY LIKELIHOOD
Revised: 3. D IFFUSION RECOVERY LIKELIHOOD

Original: As we discuss, sampling from p θ ( x | ˜ x ) becomes simple only when σ is small.
Revised: As we discuss, sampling from p θ ( x | ˜ x ) becomes simple only when σ is small.

Original: In the extreme case when σ → ∞ , p θ ( x | ˜ x ) converges to the marginal distribution p θ ( x ) , which is again highly multimodal and difﬁcult to sample from.
Revised: In the extreme case when σ → ∞ , p θ ( x | ˜ x ) converges to the marginal distribution p θ ( x ) , which is again highly multimodal and difﬁcult to sample from.

Original: To keep σ small and meanwhile equip the model with the ability to generate new samples initialized from white noise, inspired by Sohl-Dickstein et al.
Revised: To keep σ small and meanwhile equip the model with the ability to generate new samples initialized from white noise, inspired by Sohl-Dickstein et al.

Original: (2015) and Ho et al.
Revised: (2015) and Ho et al.

Original: (2020), we propose to learn a sequence of recovery likelihoods, on gradually perturbed observed data based on a diffusion process.
Revised: (2020), we propose to learn a sequence of recovery likelihoods, on gradually perturbed observed data based on a diffusion process.

Original: Speciﬁcally, assume a sequence of perturbed observations x 0 , x 1 , ..., x T such that
Revised: Speciﬁcally, assume a sequence of perturbed observations x 0 , x 1 , ..., x T such that

Original: [Equation]
Revised: [Equation]

Original: The scaling factor (cid:113) 1 − σ 2 t +1 ensures that the sequence is a spherical interpolation between the observed sample and Gaussian white noise.
Revised: The scaling factor (cid:113) 1 − σ 2 t +1 ensures that the sequence is a spherical interpolation between the observed sample and Gaussian white noise.

Original: Let y t = (cid:113) 1 − σ 2 t +1 x t , and we assume a sequence of conditional EBMs
Revised: Let y t = (cid:113) 1 − σ 2 t +1 x t , and we assume a sequence of conditional EBMs

Original: [Equation]
Revised: [Equation]

Original: where f θ ( y t , t ) is deﬁned by a neural network conditioned on t .
Revised: where f θ ( y t , t ) is deﬁned by a neural network conditioned on t .

Original: We follow the learning algorithm in section 3.2.
Revised: We follow the learning algorithm in section 3.2.

Original: A question is how to determine the step size schedule δ t of Langevin dynamics.
Revised: A question is how to determine the step size schedule δ t of Langevin dynamics.

Original: Inspired by the sampling procedure of the normal approximation (equation 12), we set the step size δ t =
Revised: Inspired by the sampling procedure of the normal approximation (equation 12), we set the step size δ t =

Original: bσ t , where b < 1 is a tuned hyperparameter.
Revised: bσ t , where b < 1 is a tuned hyperparameter.

Original: This schedule turns out to work well in practice.
Revised: This schedule turns out to work well in practice.

Original: Thus the K steps of Langevin dynamics iterates
Revised: Thus the K steps of Langevin dynamics iterates

Original: [Equation]
Revised: [Equation]

Original: Algorithm 1 summarizes the training procedure.
Revised: Algorithm 1 summarizes the training procedure.

Original: After training, we initialize the MCMC sampling from Gaussian white noise, and the synthesized sample at each time step serves to initialize the
Revised: After training, we initialize the MCMC sampling from Gaussian white noise, and the synthesized sample at each time step serves to initialize the

Original: MCMC that samples from the model of the previous time step.
Revised: MCMC that samples from the model of the previous time step.

Original: See algorithm 2.
Revised: See algorithm 2.

Original: To show the efﬁcacy of our method, Figures 3 and 2 display several 2D toy examples learned by diffusion recovery likelihood.
Revised: To show the efﬁcacy of our method, Figures 3 and 2 display several 2D toy examples learned by diffusion recovery likelihood.

Original:  repeat
Revised: Algorithm repeat

Original: Sample t ∼ Unif( { 0 , ..., T − 1 } ) .
Revised: Sample t ∼ Unif( { 0 , ..., T − 1 } ) .

Original: Sample pairs ( y t , x t +1 ) .
Revised: Sample pairs ( y t , x t +1 ) .

Original: Set synthesized sample y − t
Revised: Set synthesized sample y − t

Original: = x t +1 .
Revised: = x t +1 .

Original: for τ ← 1 to K do
Revised: for τ ← 1 to K do

Original: Update y − t according to equation 17.
Revised: Update y − t according to equation 17.

Original: end for
Revised: end for

Original: Update θ following the gradients ∂∂θ f θ ( y t , t ) − ∂∂θ f θ ( y − t , t ) .
Revised: Update θ following the gradients ∂∂θ f θ ( y t , t ) − ∂∂θ f θ ( y − t , t ) .

Original: until converged.
Revised: until converged.

Revised: Algorithm

Original: Sample x T ∼ N (0 , I ) .
Revised: Sample x T ∼ N (0 , I ) .

Original: for t ← T − 1 to 0 do y t
Revised: for t ← T − 1 to 0 do y t

Original: = x t +1 .
Revised: = x t +1 .

Original: for τ ← 1 to K do
Revised: for τ ← 1 to K do

Original: Update y t according to equation 17.
Revised: Update y t according to equation 17.

Original: end for x t =
Revised: end for x t =

Original: y t / (cid:113) 1 − σ 2 t
Revised: y t / (cid:113) 1 − σ 2 t

Original: +1 .
Revised: +1 .

Original: end for return x 0 .
Revised: end for return x 0 .

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: To show that diffusion recovery likelihood is ﬂexible for diffusion process of various magnitudes of noise, we test the method under two settings: (1) T = 6 , with K = 30 steps of Langevin dynamic per time step, b = 0 . 0002 ; (2) T = 1000 , with sampling from normal approximation.
Revised: To show that diffusion recovery likelihood is ﬂexible for diffusion process of various magnitudes of noise, we test the method under two settings: (1) T = 6 , with K = 30 steps of Langevin dynamic per time step; (2) T = 1000 , with sampling from normal approximation.

Original: (2) resembles the noise schedule of Ho et al.
Revised: (2) resembles the noise schedule of Ho et al.

Original: and the magnitude of noise added at each time step is much smaller compared to (1).
Revised: and the magnitude of noise added at each time step is much smaller compared to (1).

Original: For both settings, we set σ 2 t to increase linearly.
Revised: For both settings, we set σ 2 t to increase linearly.

Original: The network structure of f θ ( x, t ) is based on Wide ResNet (Zagoruyko & Komodakis, 2016) and we remove weight normalization.
Revised: The network structure of f θ ( x, t ) is based on Wide ResNet (Zagoruyko & Komodakis, 2016) and we remove weight normalization.

Original: t is encoded by Transformer sinusoidal position embedding as in (Ho et al., 2020).
Revised: t is encoded by Transformer sinusoidal position embedding as in (Ho et al., 2020).

Revised: For (1), we ﬁnd that adding another scaling factor c t to the step size δ t helps.

Original: Architecture and training details are in Appendix B. Henceforth we simply refer the two settings as T6 and T1k .
Revised: Architecture and training details are in Appendix B. Henceforth we simply refer the two settings as T6 and T1k .

Original: 4. I MAGE GENERATION
Revised: 4. I MAGE GENERATION

Original: Figures 1 and 4 display uncurated samples generated from learned models on CIFAR-10, CelebA 64 × 64 , LSUN 64 × 64 and 128 × 128 datasets under T6 setting.
Revised: Figures 1 and 4 display uncurated samples generated from learned models on CIFAR-10, CelebA 64 × 64 , LSUN 64 × 64 and 128 × 128 datasets under T6 setting.

Original: The samples are of high ﬁdelity and comparable to GAN-based methods.
Revised: The samples are of high ﬁdelity and comparable to GAN-based methods.

Original: Appendix C.4 provides more generated samples.
Revised: Appendix C.5 provides more generated samples.

Original: Tables 1 and 3 summarize the quantitative evaluations on CIFAR-10 and CelebA datasets, in terms of
Revised: Tables 1 and 3 summarize the quantitative evaluations on CIFAR-10 and CelebA datasets, in terms of

Original: Frechet Inception Distance (FID) (Heusel et al., 2017) and inception scores (Salimans et al., 2016).
Revised: Frechet Inception Distance (FID) (Heusel et al., 2017) and inception scores (Salimans et al., 2016).

Original: On CIFAR-10, our model achieves FID 9.60 and inception score 8.58, which outperforms existing methods of learning explicit energy-based models to a large extent, and is superior to a majority of
Revised: On CIFAR-10, our model achieves FID 9.58 and inception score 8.30, which outperforms existing methods of learning explicit energy-based models to a large extent, and is superior to a majority of

Original: GAN-based methods.
Revised: GAN-based methods.

Original: On CelebA, our model obtains results comparable with the state-of-the-art
Revised: On CelebA, our model obtains results comparable with the state-of-the-art

Original: GAN-based methods, and outperforms score-based methods (Song & Ermon, 2019; 2020).
Revised: GAN-based methods, and outperforms score-based methods (Song & Ermon, 2019; 2020).

Original: Note that the score-based methods (Song & Ermon, 2019; 2020) and diffusion probabilistic models (Ho et al., 2020) directly parametrize and learn the score of data distribution, whereas our goal is to learn explicit energy-based models.
Revised: Note that the score-based methods (Song & Ermon, 2019; 2020) and diffusion probabilistic models (Ho et al., 2020) directly parametrize and learn the score of data distribution, whereas our goal is to learn explicit energy-based models.

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: and grain of salt
Revised: that we estimate the function instead of analytically computing it See section 4.2.

Original: [Table]
Revised: [Table]

Original: Interpolation.
Revised: Interpolation.

Original: As shown in Figure 5, our model is capable of smooth interpolation between two generated samples.
Revised: As shown in Figure 5, our model is capable of smooth interpolation between two generated samples.

Original: Speciﬁcally, for two samples x (0) 0 and x (1) 0 , we do a sphere interpolation between the initial white noise images x (0) T and x (1) T and the noise terms of Langevin dynamics (cid:15) (0) t,τ and (cid:15) (1) t,τ for every sampling step at every time step.
Revised: Speciﬁcally, for two samples x (0) 0 and x (1) 0 , we do a sphere interpolation between the initial white noise images x (0) T and x (1) T and the noise terms of Langevin dynamics (cid:15) (0) t,τ and (cid:15) (1) t,τ for every sampling step at every time step.

Original: More interpolation results can be found in Appendix C.2.
Revised: More interpolation results can be found in Appendix C.3.

Original: Image inpainting.
Revised: Image inpainting.

Original: A promising application of energy-based models is to use the learned model as a prior model for image processing, such as image inpainting, denoising and super-resolution.
Revised: A promising application of energy-based models is to use the learned model as a prior model for image processing, such as image inpainting, denoising and super-resolution

Original: Such applications have been explored in Gao et al. ; Du & Mordatch (2019); Song & Ermon (2019).
Revised: (Gao et al., 2018; Du & Mordatch, 2019; Song & Ermon, 2019).

Original: In Figure 6, we demonstrate that the learned models by maximizing recovery likelihoods are capable of realistic and semantically meaningful image inpainting.
Revised: In Figure 6, we demonstrate that the learned models by maximizing recovery likelihoods are capable of realistic and semantically meaningful image inpainting.

Original: Speciﬁcally, given a masked image and the corresponding mask, we ﬁrst obtain a sequence of perturbed masked images at different noise levels.
Revised: Speciﬁcally, given a masked image and the corresponding mask, we ﬁrst obtain a sequence of perturbed masked images at different noise levels.

Original: The inpainting can be easily achieved by running Langevin dynamics progressively on the masked pixels while keeping the observed pixels ﬁxed at decreasingly lower noise levels.
Revised: The inpainting can be easily achieved by running Langevin dynamics progressively on the masked pixels while keeping the observed pixels ﬁxed at decreasingly lower noise levels.

Original: Additional image inpainting results can be found in Appendix C.3.
Revised: Additional image inpainting results can be found in Appendix C.4.

Original: 4. A BLATION STUDY
Revised: Ablation study.

Original: Table 2 summarizes the results of ablation study on CIFAR-10.
Revised: Table 2 summarizes the results of ablation study on CIFAR-10.

Original: We investigate the effect of choosing different number of time steps T andnumber of sampling steps K .
Revised: We investigate the effect of changing the numbers of time steps T and sampling steps K .

Original: First, to show that it is beneﬁcial to learn by diffusion recovery likelihood, we compare against a baseline approach ( T = 1 , K = 180 ) where we use only one time step, so that the recovery likelihood becomes marginal likelihood.
Revised: First, to show that it is beneﬁcial to learn by diffusion recovery likelihood, we compare against a baseline approach ( T = 1 , K = 180 ) where we use only one time step, so that the recovery likelihood becomes marginal likelihood.

Original: The approach is adopted by Nijkamp et al.
Revised: The approach is adopted by Nijkamp et al.

Original: (2019b)
Revised: (2019b)

Original: and Du & Mordatch (2019).
Revised: and Du & Mordatch (2019).

Original: For fair comparison, we equip the baseline method the same budget of MCMC sampling as our T6 setting (i.e., 180 sampling steps).
Revised: For fair comparison, we equip the baseline method the same budget of MCMC sampling as our T setting (i.e., 180 sampling steps).

Original: Our method outperforms this baseline method by a large margin.
Revised: Our method outperforms this baseline method by a large margin.

Original: Also the models can be trained more efﬁciently as the number of sampling steps per iteration is reduced and amortized by time steps.
Revised: Also the models are trained more efﬁciently as the number of sampling steps per iteration is reduced and amortized by time steps.

Original: Next, we report the sample quality of setting T1k .
Revised: Next, we report the sample quality of setting T1k .

Original: We test two training objectives for this setting: (1) maximizing recovery likelihoods (T = 1000, K = 0) and (2) maximizing the approximated normal distributions (T=1000, K=0 (DSM)).
Revised: We test two training objectives for this setting: (1) maximizing recovery likelihoods (T = 1000, K = 0) and (2) maximizing the approximated normal distributions (T=1000, K=0 (DSM)).

Original: As mentioned in section 3.4, (2) is equivalent to the training objectives of denoising score matching (Song & Ermon, 2019; 2020) and diffusion probabilistic model (Ho et al., 2020), except that the score functions are taken as the gradients of explicit energy functions.
Revised: As mentioned in section 3.4, (2) is equivalent to the training objectives of denoising score matching (Song & Ermon, 2019; 2020) and diffusion probabilistic model (Ho et al., 2020), except that the score functions are taken as the gradients of explicit energy functions.

Original: In practice, for a direct comparison, (2) follows the same implementation as in Ho et al.
Revised: In practice, for a direct comparison, (2) follows the same implementation as in Ho et al.

Original: (2020), except that the score function is parametrized as the gradients of the explicit energy function used in our method.
Revised: (2020), except that the score function is parametrized as the gradients of the explicit energy function used in our method.

Original: (1) and (2) achieve similar sample quality in terms of quantitative metrics, where (2) results in a slightly better FID score yet a slightly worse inception score.
Revised: (1) and (2) achieve similar sample quality in terms of quantitative metrics, where (2) results in a slightly better FID score yet a slightly worse inception score.

Original: This veriﬁes the fact that the training objectives of (1) and (2) are consistent.
Revised: This veriﬁes the fact that the training objectives of (1) and (2) are consistent.

Original: Both (1) and (2) performs worse than setting T6.
Revised: Both (1) and (2) performs worse than setting T6.

Original: A possible explanation is that the sampling error may accumulate with many time steps, so that a more ﬂexible schedule of time steps accompanied with certain amount of sampling steps is preferred.
Revised: A possible explanation is that the sampling error may accumulate with many time steps, so that a more ﬂexible schedule of time steps accompanied with certain amount of sampling steps is preferred.

Original: Last, we examine the inﬂuence of varying the number of sampling steps while ﬁxing the number of time steps.
Revised: Last, we examine the inﬂuence of varying the number of sampling steps while ﬁxing the number of time steps.

Original: Figure 7 demonstrates FID scores computed on 2,500 samples every 15,000 iterations.

Original: The training becomes unstable when the number of sampling steps are not enough ( T = 6 , K = 10 ), and more sampling steps lead to better sample quality.
Revised: The training becomes unstable when the number of sampling steps are not enough ( T = 6 , K = 10 ), and more sampling steps lead to better sample quality.

Original: However, since K = 50 does not gain signiﬁcant improvement versus K = 30 , yet of much higher computational cost, we keep K =for image generation on all datasets.
Revised: However, since K =does not gain signiﬁcant improvement versus K = 30 , yet of much higher computational cost, we keep K = 30 for image generation on all datasets.

Revised: See Appendix C.1 for a plot of FID scores over iterations.

Original: 4. L ONG - RUN CHAIN ANALYSIS
Revised: 4. L ONG - RUN CHAIN ANALYSIS

Original: Besides achieving high quality generation, a perhaps equally important aspect of learning EBMs is to obtain a faithful energy potential.
Revised: Besides achieving high quality generation, a perhaps equally important aspect of learning EBMs is to obtain a faithful energy potential.

Original: A principle way to check the validity of the learned potential is to perform long-run sampling chains and see if the samples still remain realistic.
Revised: A principle way to check the validity of the learned potential is to perform long-run sampling chains and see if the samples still remain realistic.

Original: However, as pointed out in Nijkamp et al.
Revised: However, as pointed out in Nijkamp et al.

Original: (2019a), almost all existing methods of learning EBMs fail in getting realistic long-run chain samples.
Revised: (2019a), almost all existing methods of learning EBMs fail in getting realistic long-run chain samples.

Original: In this subsection, we demonstrate that by composing a thousand diffusion time steps ( T1k setting), we canuse MCMC to form steady long-run  chains for the conditional distributions.
Revised: In this subsection, we demonstrate that by composing a thousand diffusion time steps ( T1k setting), we can form steady long-run MCMC chains for the conditional distributions.

Original: First we prepare a faithful sampler for conducting long-run sampling.
Revised: First we prepare a faithful sampler for conducting long-run sampling.

Original: Speciﬁcally, after training the model under T 1 k setting by maximizing diffusion recovery likelihood, for each time step, we ﬁrst sample from the normal approximation and count it as one sampling step, and then use Hamiltonian
Revised: Speciﬁcally, after training the model under T 1 k setting by maximizing diffusion recovery likelihood, for each time step, we ﬁrst sample from the normal approximation and count it as one sampling step, and then use Hamiltonian

Original: Monte Carlo (HMC) (Neal et al., 2011) with 2 leapfrog steps to perform the consecutive sampling steps.
Revised: Monte Carlo (HMC) (Neal et al., 2011) with 2 leapfrog steps to perform the consecutive sampling steps.

Original: To obtain a reasonable schedule of sampling step size, for each time step we adaptively adjust the step size of HMC to make the average acceptance rate range in [0 . 6 , 0 . 9] , which is computed over 1000 chains for 100 steps.
Revised: To obtain a reasonable schedule of sampling step size, for each time step we adaptively adjust the step size of HMC to make the average acceptance rate range in [0 . 6 , 0 . 9] , which is computed over 1000 chains for 100 steps.

Original: Figure 8 displays the adjusted step size( left ) and acceptance rate ( center ) over time step.
Revised: Figure 7 displays the adjusted step size left ) and acceptance rate ( center ) over time step.

Original: The adjusted step size increases logarithmically.
Revised: The adjusted step size increases logarithmically.

Original: With this step size schedule, we generate long-run chains from the learned sequence of conditional distributions.
Revised: With this step size schedule, we generate long-run chains from the learned sequence of conditional distributions.

Original: As shown in
Revised: As shown in

Original: Figure 9, images remain realistic for even 100 k sampling steps in total (i.e., 100 sampling steps per time step), resulting in FID 24.89.
Revised: Figure 8, images remain realistic for even 100 k sampling steps in total (i.e., 100 sampling steps per time step), resulting in FID 24.89.

Original: This score is close to the one computed on samples generated by 1 k steps (i.e., sampled from normal approximation), which is 25.12.
Revised: This score is close to the one computed on samples generated by 1 k steps (i.e., sampled from normal approximation), which is 25.12.

Original: As a further check, we recruit a No-U-Turn Sampler (Hoffman & Gelman, 2014) with the same step size schedule as HMC to perform long-run sampling, where the samples also remain realistic.
Revised: As a further check, we recruit a No-U-Turn Sampler (Hoffman & Gelman, 2014) with the same step size schedule as HMC to perform long-run sampling, where the samples also remain realistic.

Original: See Appendix C.1 for details.
Revised: See Appendix C.2 for details.

Original: More interestingly, given the faithful long-run MCMC samples from the conditional distributions, we can estimate the log ratio of the partition functions of the marginal distributions, and further estimate the partition function of p θ ( y 0 ) .
Revised: More interestingly, given the faithful long-run MCMC samples from the conditional distributions, we can estimate the log ratio of the partition functions of the marginal distributions, and further estimate the partition function of p θ ( y 0 ) .

Original: The strategy is based on annealed importance sampling (Neal, 2001).
Revised: The strategy is based on annealed importance sampling (Neal, 2001).

Original: See Appendix A.6 for the implementation details.
Revised: See Appendix A.6 for the implementation details.

Original: The right subﬁgure of Figure 8 depicts the estimated log partition function of p θ ( y 0 ) over the number of
Revised: The right subﬁgure of Figure 7 depicts the estimated log partition function of p θ ( y 0 ) over the number of

Original: MCMC samples used.
Revised: MCMC samples used.

Original: To verify the estimation strategy and again check the long-run chain samples, we conduct multiple runs using samples generated with different numbers of HMC steps and display the estimation curves.
Revised: To verify the estimation strategy and again check the long-run chain samples, we conduct multiple runs using samples generated with different numbers of HMC steps and display the estimation curves.

Original: All the curves saturate to values close to each other at the end, indicating the stability of long-run chain samples and the effectiveness of the estimation strategy.
Revised: All the curves saturate to values close to each other at the end, indicating the stability of long-run chain samples and the effectiveness of the estimation strategy.

Original: With the estimated partition function, by change of variable , we can estimate the normalized density of data as g θ ( x 0 ) =
Revised: With the estimated partition function, by change of variable , we can estimate the normalized density of data as g θ ( x 0 ) =

Original: (cid:112) 1 − σ 21 p θ ( (cid:112) 1 − σ 21 x 0 ) .
Revised: (cid:112) 1 − σ 21 p θ ( (cid:112) 1 − σ 21 x 0 ) .

Original: We report test bits per dimension on CIFAR-10 in Table 4.
Revised: We report test bits per dimension on CIFAR-10 in Table 4.

Original: Note that the result should be taken with a grain of salt, because the partition function is estimated by samples and as shown in Appendix
Revised: Note that the result should be taken with a grain of salt, because the partition function is estimated by samples and as shown in Appendix

Original: A.6, it is a stochastic lower bound of the true value, that will converge to the true value when the number of samples grows large.
Revised: A.6, it is a stochastic lower bound of the true value, that will converge to the true value when the number of samples grows large.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We propose to learn EBMs by diffusion recovery likelihood, a variant of MLE applied to diffusion processes.
Revised: We propose to learn EBMs by diffusion recovery likelihood, a variant of MLE applied to diffusion processes.

Original: We achieve high quality image synthesis, and with a thousand noise levels, we obtain faithful long-run MCMC samples that indicate the validity of the learned energy potentials.
Revised: We achieve high quality image synthesis, and with a thousand noise levels, we obtain faithful long-run MCMC samples that indicate the validity of the learned energy potentials.

Original: In future work, we look forward to investigating settings to combine the high quality synthesis of model- T and faithful long-run MCMC samples of model- T1k , to achieve the best of two worlds.

Original: Since this method can learn EBMs efﬁciently with small budget of MCMC, we are also interested in scaling it up to higher resolution images and investigating this method in other data modalities.
Revised: Since this method can learn EBMs efﬁciently with small budget of MCMC, we are also interested in scaling it up to higher resolution images and investigating this method in other data modalities in the future. A CKNOWLEDGEMENT

Revised: The work was done while Ruiqi Gao and Yang Song were interns at Google Brain during the summer of 2020.

Revised: The work of Ying Nian Wu is supported by NSF DMS-2015577.

Revised: We thank Alexander A.

Revised: Alemi, Jonathan Ho, Tim Salimans and Kevin Murphy for their insightful discussions during the course of this project.

Original: A E XTENDED DERIVATIONS A. D ERIVATION OF EQUATION
Revised: A E XTENDED DERIVATIONS A. D ERIVATION OF EQUATION

Original: Let ˜ x
Revised: Let ˜ x

Original: = x + σ (cid:15) , where (cid:15) ∼ N (0 , I ) .
Revised: = x + σ (cid:15) , where (cid:15) ∼ N (0 , I ) .

Original: Given the marginal distribution of
Revised: Given the marginal distribution of

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: We can derive the conditional distribution of x given ˜ x as
Revised: We can derive the conditional distribution of x given ˜ x as

Original: [Equation]
Revised: [Equation]

Original: where we absorb all the terms that are irrelevant of x as ˜
Revised: where we absorb all the terms that are irrelevant of x as ˜

Original: Z θ (˜ x ) .
Revised: Z θ (˜ x ) .

Original: A. T HEORETICAL UNDERSTANDING
Revised: A. T HEORETICAL UNDERSTANDING

Original: In this subsection, we analyze the asymptotic behavior of maximizing the recovery log-likelihood.
Revised: In this subsection, we analyze the asymptotic behavior of maximizing the recovery log-likelihood.

Original: For model class { p θ ( x ) , ∀ θ } , suppose there exists θ ∗ such that p data = p θ ∗ .
Revised: For model class { p θ ( x ) , ∀ θ } , suppose there exists θ ∗ such that p data = p θ ∗ .

Original: According to the classical theory of MLE, let ˆ θ 

Revised: According to the classical theory of MLE, let ˆ θ

Original: be the point estimate by MLE.
Revised: be the point estimate by MLE.

Original: Then we have ˆ θ is an unbiased estimator of θ ∗ with asymptotic normality: √
Revised: Then we have ˆ θ is an unbiased estimator of θ ∗ with asymptotic normality: √

Original: [Equation]
Revised: [Equation]

Original: (22) where I 0 ( θ ) =
Revised: (22) where I 0 ( θ ) =

Original: E x ∼ p θ [ −∇ 2 θ log p θ ( x )] is the Fisher information, and n is the number of observed samples.
Revised: E x ∼ p θ [ −∇ 2 θ log p θ ( x )] is the Fisher information, and n is the number of observed samples.

Original: Let ˆ θ be the point estimate given by maximizing recovery log-likelihood, we can derive a result in parallel to that of MLE:
Revised: Let ˆ θ be the point estimate given by maximizing recovery log-likelihood, we can derive a result in parallel to that of MLE:

Original: [Equation]
Revised: [Equation]

Original: ) where I ( θ ) =
Revised: where I ( θ ) =

Original: E p θ ( x , ˜ x ) [ −∇ 2 θ log p θ ( x | ˜ x )] .
Revised: E p θ ( x , ˜ x ) [ −∇ 2 θ log p θ ( x | ˜ x )] .

Original: The relationship between I 0 ( θ ) and I ( θ ) is that
Revised: The relationship between I 0 ( θ ) and I ( θ ) is that

Original: [Equation]
Revised: [Equation]

Original: Thus there is loss of information
Revised: Thus there is loss of information

Original: [Equation]
Revised: [Equation]

Original: but ˆ θ is still an unbiased estimator of θ ∗ with asymptotic normality
Revised: but ˆ θ is still an unbiased estimator of θ ∗ with asymptotic normality

Original: [Equation]
Revised: [Equation]

Original: A. D ETAILED DERIVATION OF NORMAL APPROXIMATION
Revised: A. D ETAILED DERIVATION OF NORMAL APPROXIMATION

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: A. 
Revised: A. D IFFERENCE

Original: p ( x ) AND p (˜ x )
Revised: p ( x ) AND p (˜ x )

Original: For notation clarity, with ˜ x
Revised: For notation clarity, with ˜ x

Original: = x + (cid:15) , we let (cid:101) p be the distribution of ˜ x , and p be the distribution of x .
Revised: = x + (cid:15) , we let (cid:101) p be the distribution of ˜ x , and p be the distribution of x .

Original: Then for a smooth testing function with vanishing tails,
Revised: Then for a smooth testing function with vanishing tails,

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Integral by part,
Revised: Integral by parts,

Original: [Equation]
Revised: [Equation]

Original: Thus we have the heat equation
Revised: Thus we have the heat equation

Original: [Equation]
Revised: [Equation]

Original: The score
Revised: The score

Original: [Equation]
Revised: [Equation]

Original: Thus the difference between the score of p and (cid:101) p is of the order σ 2 , which is negligible when σ 2 is small.
Revised: Thus the difference between the score of p and (cid:101) p is of the order σ 2 , which is negligible when σ 2 is small.

Original: A. L EARNING GRADIENTS OF NORMAL APPROXIMATION AND ORIGINAL RECOVERY LIKELIHOOD
Revised: A. L EARNING GRADIENTS OF NORMAL APPROXIMATION AND ORIGINAL RECOVERY LIKELIHOOD

Original: In this subsection we demonstrate that the learning gradient of maximizing likelihood of the normal approximation is approximately the same as the gradient of maximizing the original recovery likelihood with one step of Langevin sampling.
Revised: In this subsection we demonstrate that the learning gradient of maximizing likelihood of the normal approximation is approximately the same as the gradient of maximizing the original recovery likelihood with one step of Langevin sampling.

Original: Speciﬁcally, the gradient of the normal approximation of recovery log-likelihood for an observed x obs is
Revised: Speciﬁcally, the gradient of the normal approximation of recovery log-likelihood for an observed x obs is

Original: [Equation]
Revised: [Equation]

Original: On the other hand
Revised: On the other hand

Original: [Equation]
Revised: [Equation]

Original: to maximize the original recovery likelihood, suppose we sample x syn ∼ p θ ( x | ˜ x ) , then the gradient ascent of the original recovery log-likelihood is
Revised: to maximize the original recovery likelihood, suppose we sample x syn ∼ p θ ( x | ˜ x ) , then the gradient ascent of the original recovery log-likelihood is

Original: [Equation]
Revised: [Equation]

Original: where h θ ( x ) = ∇ θ f θ ( x ) .
Revised: where h θ ( x ) = ∇ θ f θ ( x ) .

Original: Approximately, if we perform one step of Langevin dynamics from ˜ x to obtain x syn , i
Revised: Approximately, if we perform one step of Langevin dynamics from ˜ x to obtain x syn , i

Original: [Equation]
Revised: [Equation]

Original: e., x syn =
Revised: e., x syn =

Original: ˜ x + σ 2 f (cid:48)
Revised: ˜ x + σ 2 f (cid:48)

Original: θ (˜ x ) + √ 2 σe , and assume f θ ( x ) is locally linear in x , then
Revised: θ (˜ x ) + √ 2 σe , and assume f θ ( x ) is locally linear in x , then

Original: [Equation]
Revised: [Equation]

Original: Comparing equations 37 and 43, we see that the two gradients agree with each other [Equation]
Revised: Comparing equations 37 and 43, we see that the two gradients agree with each other.

Original: A. E STIMATING THE PARTITION FUNCTION
Revised: A. E STIMATING THE PARTITION FUNCTION

Original: We can utilize the sequence of learned distributions of y t ( = (cid:113) 1 − σ 2 t +1 x t ) to estimate the partition function.
Revised: We can utilize the sequence of learned distributions of y t ( = (cid:113) 1 − σ 2 t +1 x t ) to estimate the partition function.

Original: Speciﬁcally, the marginal distribution of y t is
Revised: Speciﬁcally, the marginal distribution of y t is

Original: [Equation]
Revised: [Equation]

Original: We can estimate the ratio of the partition functions at two consecutive time steps using importance sampling
Revised: We can estimate the ratio of the partition functions at two consecutive time steps using importance sampling

Original: [Equation]
Revised: [Equation]

Original: where y t +1 ,i are samples generated by progressive sampling.
Revised: where y t +1 ,i are samples generated by progressive sampling.

Original: Starting from t =
Revised: Starting from t =

Original: T , where p T ( x ) follows Gaussian distribution, we can compute log Z t along the reverse path of the diffusion process, until we reach t = 0 :
Revised: T , where p T ( x ) follows Gaussian distribution, we can compute log Z θ,t along the reverse path of the diffusion process, until we reach t = 0 :

Original: [Equation]
Revised: [Equation]

Original: In practice, since the ratio given by MCMC samples can vary across many orders of magnitude, it is more meaningful to estimate
Revised: In practice, since the ratio given by MCMC samples can vary across many orders of magnitude, it is more meaningful to estimate

Original: [Equation]
Revised: [Equation]

Original: Unfortunately, although equation 46 is an unbiased estimator of Z θ,t /Z
Revised: Unfortunately, although equation 46 is an unbiased estimator of Z θ,t /Z

Original: θ,t +1 , the logarithm of this estimator is generally a stochastic lower bound of log( Z θ,t /Z
Revised: θ,t +1 , the logarithm of this estimator is generally a stochastic lower bound of log( Z θ,t /Z

Original: θ,t +1 )
Revised: θ,t +1 )

Original: (Grosse et al., 2016).
Revised: (Grosse et al., 2016).

Original: However, as we show below, this bound will gradually converge to an unbiased estimator of log( Z θ,t /Z
Revised: However, as we show below, this bound will gradually converge to an unbiased estimator of log( Z θ,t /Z

Original: θ,t +1 ) , as the number of samples becomes large.
Revised: θ,t +1 ) , as the number of samples becomes large.

Original: Speciﬁcally, let A be the estimator in equation 46, µ be the true value of Z θ,t /Z
Revised: Speciﬁcally, let A be the estimator in equation 46, µ be the true value of Z θ,t /Z

Original: θ,t +1 .
Revised: θ,t +1 .

Original: We have E [ A ] = µ , then by second order Taylor expansion,[Equation]
Revised: We have E [ A ] = µ , then by second order Taylor expansion,

Original: [Equation]
Revised: [Equation]

Original: By law of large number , Var( A ) → 0 as M → ∞ , and thus E [log A ] → log µ .
Revised: By law of large number , Var( A ) → 0 as M → ∞ , and thus E [log A ] → log µ .

Original: This is also consistent with the estimation curves in the right subﬁgure of Figure 8: since Var( A ) ≥ 0 , the estimation curve increases from below as the number of samples becomes larger.
Revised: This is also consistent with the estimation curves in the right subﬁgure of Figure 7: since Var( A ) ≥ 0 , the estimation curve increases from below as the number of samples becomes larger.

Original: When the curve becomes stable, it indicates the convergence.
Revised: When the curve becomes stable, it indicates the convergence.

Original: B E XPERIMENTAL DETAILS
Revised: B E XPERIMENTAL DETAILS

Original: Model architecture.
Revised: Model architecture.

Original: Our network structure is based on Wide ResNet (Zagoruyko & Komodakis, 2016).
Revised: Our network structure is based on Wide ResNet (Zagoruyko & Komodakis, 2016).

Original: Table 5 lists the detailed network structures of various resolutions.
Revised: Table 5 lists the detailed network structures of various resolutions.

Original: The number of ResBlocks at every level N is a hyperparameter that we sweep over.
Revised: The number of ResBlocks at every level N is a hyperparameter that we sweep over.

Original: The values of N for various datasets are listed in Table 6.
Revised: The values of N for various datasets are listed in Table 6.

Original: Each ResBlock consists of two Conv2D layers.
Revised: Each ResBlock consists of two Conv2D layers.

Original: For the second Conv2D layer, we use zero initialization for the weights, and add a trainable channel-wise scaling parameter to the output.
Revised: For the second Conv2D layer, we use zero initialization for the weights, and add a trainable channel-wise scaling parameter to the output.

Original: We remove the weight normalization, and use leaky ReLU (slope = 0 . 2 ) as the activation function in ResBlocks.
Revised: We remove the weight normalization, and use leaky ReLU (slope = 0 . 2 ) as the activation function in ResBlocks.

Original: Spectral normalization (Miyato et al., 2018) is used to regularize parameters in Conv2D layer, ResBlocks and Dense layer.
Revised: Spectral normalization (Miyato et al., 2018) is used to regularize parameters in Conv2D layer, ResBlocks and Dense layer.

Original: For encoding time step t , we follow the scheme in (Ho et al., 2020).
Revised: For encoding time step t , we follow the scheme in (Ho et al., 2020).

Original: Speciﬁcally, the time step t is ﬁrst transformed into sinusoidal embedding, and then two Dense layers is added.
Revised: Speciﬁcally, the time step t is ﬁrst transformed into sinusoidal embedding, and then two Dense layers is added.

Original: The time embedding is added after the ﬁrst Conv2D layer of each
Revised: The time embedding is added after the ﬁrst Conv2D layer of each

Original: ResBlock.
Revised: ResBlock.

Original: Training.
Revised: Training.

Original: We use Adam (Kingma & Ba, 2014) optimizer for all the experiments.
Revised: We use Adam (Kingma & Ba, 2014) optimizer for all the experiments.

Original: We ﬁnd that for high resolution images, using a smaller β 1 in Adam help stabilize training.
Revised: We ﬁnd that for high resolution images, using a smaller β 1 in Adam help stabilize training.

Original: We use learning rate 0 .
Revised: We use learning rate 0 .

Original: for all the experiments.
Revised: for all the experiments.

Original: For the values of β 1 , batch sizes and the number of training iterations for various datasets, see Table 6.
Revised: For the values of β 1 , batch sizes and the number of training iterations for various datasets, see Table 6.

Original: Datasets.
Revised: Datasets.

Original: We use the following datasets in our experiments: CIFAR-10 (Krizhevsky et al., 2009),
Revised: We use the following datasets in our experiments: CIFAR-10 (Krizhevsky et al., 2009),

Original: CelebA (Liu et al., 2018) and LSUN (Yu et al., 2015).
Revised: CelebA (Liu et al., 2018) and LSUN (Yu et al., 2015).

Original: CIFAR-10 is of resolution 32 × 32 , and contains 50 , 000 training images and 10 , 000 test images.
Revised: CIFAR-10 is of resolution 32 × 32 , and contains 50 , 000 training images and 10 , 000 test images.

Original: CelebA contains 202,599 face images, of which 162,770 are training images and 19,962 are test images.
Revised: CelebA contains 202,599 face images, of which 162,770 are training images and 19,962 are test images.

Original: For processing, we ﬁrst clip each image to 178 × 178 and then resize it to 64 × 64 .
Revised: For processing, we ﬁrst clip each image to 178 × 178 and then resize it to 64 × 64 .

Original: For LSUN, we use church outdoor and bedroom categories, which contains 126,227 and 3,033,042 training images respectively.
Revised: For LSUN, we use church outdoor and bedroom categories, which contains 126,227 and 3,033,042 training images respectively.

Original: Both categories contain 300 test images.
Revised: Both categories contain 300 test images.

Original: For processing, we ﬁrst crop each image to a square image of the smaller size among the height and weight, and then we resize it to 64 × 64 or 128 × 128 .
Revised: For processing, we ﬁrst crop each image to a square image of the smaller size among the height and weight, and then we resize it to 64 × 64 or 128 × 128 .

Original: For resizing, we set antialias to True.
Revised: For resizing, we set antialias to True.

Original: We apply horizontal random ﬂip as data augmentation for all datasets during training.
Revised: We apply horizontal random ﬂip as data augmentation for all datasets during training.

Original: Evaluation metrics.
Revised: Evaluation metrics.

Original: We use FID and inception scores as quantitative evaluation metrics of sample quality.
Revised: We use FID and inception scores as quantitative evaluation metrics of sample quality.

Original: On all the datasets, we calculate FID and inception scores on 50,000 samples using the original code from Salimans et al.
Revised: On all the datasets, we calculate FID and inception scores on 50,000 samples using the original code from Salimans et al.

Original: (2016) and Heusel et al.
Revised: (2016) and Heusel et al.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: C A DDITIONAL EXPERIMENTAL RESULTS C. L ONG - RUN C HAIN SAMPLING WITH NUTS
Revised: C A DDITIONAL EXPERIMENTAL RESULTS C. FID SCORES OVER ITERATIONS

Revised: Figure 9 demonstrates FID scores computed on 2,500 samples every 15,000 iterations

Revised: . C. L ONG - RUN C HAIN SAMPLING WITH NUTS

Original: As a further check, we use a No-U-Turn Sampler (Hoffman & Gelman, 2014) to perform the longrun chain sampling, with the same step size schedule obtained for HMC sampler.
Revised: As a further check, we use a No-U-Turn Sampler (Hoffman & Gelman, 2014) to perform the longrun chain sampling, with the same step size schedule obtained for HMC sampler.

Original: Figure 10 displays samples with different number of sampling steps.
Revised: Figure 10 displays samples with different number of sampling steps.

Original: The samples remain realistic after 100 k sampling steps in total and the FID score remains stable.
Revised: The samples remain realistic after 100 k sampling steps in total and the FID score remains stable.

Revised: [Figure]

Original: C. A DDITIONAL INTERPOLATION RESULTS
Revised: C. A DDITIONAL INTERPOLATION RESULTS

Original: Figures 11, 12 and 13 display more examples of interpolation between two generated samples on
Revised: Figures 11, 12 and 13 display more examples of interpolation between two generated samples on

Original: CelebA 64 2 , LSUN church outdoor 128 2 and LSUN bedroom 128 2 .
Revised: CelebA 64 2 , LSUN church outdoor 128 2 and LSUN bedroom 128 2 .

Original: [Figure]

Original: C. A DDITIONAL IMAGE INPAINTING RESULTS
Revised: C. A DDITIONAL IMAGE INPAINTING RESULTS

Original: Figures 14 and 15 show additional examples of image inpainting on CelebA 64 2 and LSUN church outdoor [-
-]
Revised: Figures 14 and 15 show additional examples of image inpainting on CelebA 64 2 and LSUN church outdoor [-
-]128 2 .

Original: . C. A DDITIONAL UNCURATED SAMPLES
Revised: C. A DDITIONAL UNCURATED SAMPLES

Original: Figures 16, 17, 18, 19, 20 and 21 show uncurated samples from the learned models under T setting on CIFAR-10, CelebA 64 2 , LSUN church outdoor 128 2 , LSUN bedroom 128 2 , LSUN church outdoor 64 2 and LSUN bedroom 64 2 datasets.
Revised: Figures 16, 17, 18, 19, 20 and 21 show uncurated samples from the learned models under T setting on CIFAR-10, CelebA 64 2 , LSUN church outdoor 128 2 , LSUN bedroom 128 2 , LSUN church outdoor 64 2 and LSUN bedroom 64 2 datasets.