Original: I MPROVED C ONTRASTIVE D IVERGENCE T RAINING OF E NERGY B ASED M ODELS
Revised: I MPROVED C ONTRASTIVE D IVERGENCE T RAINING OF E NERGY B ASED M ODELS

Original: A BSTRACT
Revised: A BSTRACT

Original: Contrastive divergence is a popular method of training energy-based models, but is known to have difﬁculties with training stability.
Revised: Contrastive divergence is a popular method of training energy-based models, but is known to have difﬁculties with training stability.

Original: We propose an adaptation to improve contrastive divergence training by scrutinizing a gradient term that is difﬁcult to calculate and is often left out for convenience.
Revised: We propose an adaptation to improve contrastive divergence training by scrutinizing a gradient term that is difﬁcult to calculate and is often left out for convenience.

Original: We show that this gradient term is numerically signiﬁcant and in practice is important to avoid training instabilities, while being tractable to estimate.
Revised: We show that this gradient term is numerically signiﬁcant and in practice is important to avoid training instabilities, while being tractable to estimate.

Original: We further highlight how data augmentation and multi-scale processing can be used to improve model robustness and generation quality.
Revised: We further highlight how data augmentation and multi-scale processing can be used to improve model robustness and generation quality.

Original: Finally, we empirically evaluate stability of model architectures and show improved performance on a host of benchmarks and use cases,such as image generation, OOD detection, and compositional generation
Revised: Finally, we empirically evaluate stability of model architectures and show improved performance on a host of benchmarks and use cases,such as image generation, OOD detection, and compositional generation

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Energy-Based models (EBMs) have received an inﬂux of interest recently and have been applied to realistic image generation (Han et al., 2019; Du & Mordatch, 2019),
Revised: Energy-Based models (EBMs) have received an inﬂux of interest recently and have been applied to realistic image generation (Han et al., 2019; Du & Mordatch, 2019),

Original: 3D shapes synthesis (Xie et al., 2018b) , out of distribution and adversarial robustness (Lee et al., 2018; Du & Mordatch, 2019; Grathwohl et al., 2019), compositional generation (Hinton, 1999; Du et al., 2020a), memory modeling (Bartunov et al., 2019), text generation (Deng et al., 2020), video generation (Xie et al., 2017), reinforcement learning (Haarnoja et al., 2017; Du et al., 2019), protein design and folding (Ingraham et al.;
Revised: 3D shapes synthesis (Xie et al., 2018b) , out of distribution and adversarial robustness (Lee et al., 2018; Du & Mordatch, 2019; Grathwohl et al., 2019), compositional generation (Hinton, 1999; Du et al., 2020a), memory modeling (Bartunov et al., 2019), text generation (Deng et al., 2020), video generation (Xie et al., 2017), reinforcement learning (Haarnoja et al., 2017; Du et al., 2019), protein design and folding (Ingraham et al.;

Original: Du et al., 2020b), continual learning (Li et al., 2020) and biologically-plausible training (Scellier & Bengio, 2017).
Revised: Du et al., 2020b), continual learning (Li et al., 2020) and biologically-plausible training (Scellier & Bengio, 2017).

Original: Contrastive divergence is a popular and elegant procedure for training EBMs proposed by (Hinton, 2002) which lowers the energy of the training data and raises the energy of the sampled confabulations generated by the model.
Revised: Contrastive divergence is a popular and elegant procedure for training EBMs proposed by (Hinton, 2002) which lowers the energy of the training data and raises the energy of the sampled confabulations generated by the model.

Original: The model confabulations are generated via an MCMC process (commonly Gibbs sampling or Langevin dynamics), leveraging the extensive body of research on sampling and stochastic optimization.
Revised: The model confabulations are generated via an MCMC process (commonly Gibbs sampling or Langevin dynamics), leveraging the extensive body of research on sampling and stochastic optimization.

Original: The appeal of contrastive divergence is its simplicity and extensibility.
Revised: The appeal of contrastive divergence is its simplicity and extensibility.

Original: It does not require training additional auxiliary networks (Kim & Bengio, 2016;
Revised: It does not require training additional auxiliary networks (Kim & Bengio, 2016;

Original: Dai et al., 2019) (which introduce additional tuning and balancing demands), and can be used to compose models zero-shot.
Revised: Dai et al., 2019) (which introduce additional tuning and balancing demands), and can be used to compose models zero-shot.

Original: Despite these advantages, training EBMs with contrastive divergence has been challenging due to training instabilities.
Revised: Despite these advantages, training EBMs with contrastive divergence has been challenging due to training instabilities.

Original: Ensuring training stability required either combinations of spectral normalization and Langevin dynamics gradient clipping (Du & Mordatch, 2019), parameter tuning (Grathwohl et al., 2019), early stopping of MCMC chains (Nijkamp et al., 2019b), or avoiding the use of modern deep learning components, such as self-attention or layer normalization (Du & Mordatch, 2019).
Revised: Ensuring training stability required either combinations of spectral normalization and Langevin dynamics gradient clipping (Du & Mordatch, 2019), parameter tuning (Grathwohl et al., 2019), early stopping of MCMC chains (Nijkamp et al., 2019b), or avoiding the use of modern deep learning components, such as self-attention or layer normalization (Du & Mordatch, 2019).

Original: These requirements limit modeling power, prevent the compatibility with modern deep learning architectures, and prevent long-running training procedures required for scaling to larger datasets.
Revised: These requirements limit modeling power, prevent the compatibility with modern deep learning architectures, and prevent long-running training procedures required for scaling to larger datasets.

Original: With this work, we aim to maintain the simplicity and advantages of contrastive divergence training, while resolving stability issues and incorporating complementary deep learning advances.
Revised: With this work, we aim to maintain the simplicity and advantages of contrastive divergence training, while resolving stability issues and incorporating complementary deep learning advances.

Original: An often overlooked detail of contrastive divergence formulation is that changes to the energy function change the MCMC samples, which introduces an additional gradient term in the objective function (see Section 2.1 for details).
Revised: An often overlooked detail of contrastive divergence formulation is that changes to the energy function change the MCMC samples, which introduces an additional gradient term in the objective function (see Section 2.1 for details).

Original: This term was claimed to be empirically negligible in the original formulation and is typically ignored (Hinton, 2002; Liu & Wang, 2017) or estimated via highvariance likelihood ratio approaches (Ruiz & Titsias, 2019).
Revised: This term was claimed to be empirically negligible in the original formulation and is typically ignored (Hinton, 2002; Liu & Wang, 2017) or estimated via highvariance likelihood ratio approaches (Ruiz & Titsias, 2019).

Original: We show that this term can be efﬁciently estimated for continuous data via a combination of auto-differentiation and nearest-neighbor entropy estimators.
Revised: We show that this term can be efﬁciently estimated for continuous data via a combination of auto-differentiation and nearest-neighbor entropy estimators.

Original: We also empirically show that this term contributes signiﬁcantly to the overall training gradient and has the effect of stabilizing training.
Revised: We also empirically show that this term contributes signiﬁcantly to the overall training gradient and has the effect of stabilizing training.

Original: It enables inclusion of self-attention blocks into network architectures, removes the need for capacity-limiting spectral normalization, and allows us to train the networks for longer periods.
Revised: It enables inclusion of self-attention blocks into network architectures, removes the need for capacity-limiting spectral normalization, and allows us to train the networks for longer periods.

Original: We do not introduce any new objectives or complexity - our procedure is simply a more complete form of the original formulation.
Revised: We do not introduce any new objectives or complexity - our procedure is simply a more complete form of the original formulation.

Original: We further present techniques to improve mixing and mode exploration of MCMC transitions in contrastive divergence.
Revised: We further present techniques to improve mixing and mode exploration of MCMC transitions in contrastive divergence.

Original: We propose data augmentation as a useful tool to encourage mixing in
Revised: We propose data augmentation as a useful tool to encourage mixing in

Original: MCMC by directly perturbing input images to related images.
Revised: MCMC by directly perturbing input images to related images.

Original: By incorporating data augmentation as semantically meaningful perturbations, we are able to greatly improve mixing and diversity of
Revised: By incorporating data augmentation as semantically meaningful perturbations, we are able to greatly improve mixing and diversity of

Original: MCMC chains.
Revised: MCMC chains.

Original: We also leverage compositionality of EBMs to evaluate an image sample at multiple image resolutions when computing energies.
Revised: We also leverage compositionality of EBMs to evaluate an image sample at multiple image resolutions when computing energies.

Original: Such evaluation and coarse and ﬁne scales leads to samples with greater spatial coherence, but leaves MCMC generation process unchanged.
Revised: Such evaluation and coarse and ﬁne scales leads to samples with greater spatial coherence, but leaves MCMC generation process unchanged.

Original: Our contributions are as follows: ﬁrstly, we show that a gradient term neglected in the popular contrastive divergence formulation is both tractable to estimate and is important in avoiding training instabilities that previously limited applicability and scalability of energy-based models.
Revised: Our contributions are as follows: ﬁrstly, we show that a gradient term neglected in the popular contrastive divergence formulation is both tractable to estimate and is important in avoiding training instabilities that previously limited applicability and scalability of energy-based models.

Original: Secondly, we highlight how data augmentation and multi-scale processing can be used to improve model robustness and generation quality.
Revised: Secondly, we highlight how data augmentation and multi-scale processing can be used to improve model robustness and generation quality.

Original: Thirdly, we empirically evaluate stability of model architectures and show improved performance on a host of benchmarks and use cases, such as image generation, OOD detection, and compositional generation.
Revised: Thirdly, we empirically evaluate stability of model architectures and show improved performance on a host of benchmarks and use cases, such as image generation, OOD detection, and compositional generation.

Original:  A N I MPROVED C ONTRASTIVE D IVERGENCE F RAMEWORK FOR E NERGY -B ASED M ODELS
Revised:  A N I MPROVED C ONTRASTIVE D IVERGENCE F RAMEWORK FOR E NERGY -B ASED M ODELS

Original: Energy-Based Models (EBMs) represent the likelihood of a probability distribution for x ∈ R D as [Equation]
Revised: Energy-Based Models (EBMs) represent the likelihood of a probability distribution for x ∈ R D as p θ ( x ) = exp( − E θ ( x ))

Original:  where the function E θ ( x ) : R D → R , is known as the energy function , and
Revised: Z ( θ ) where the function E θ ( x ) : R D → R , is known as the energy function , and

Original: Z ( θ ) =
Revised: Z ( θ ) =

Original: (cid:82) x exp − E θ ( x ) is known as the partition function.
Revised: (cid:82) x exp − E θ ( x ) is known as the partition function.

Original: Thus, an EBM can be represented by an neural network that takes x as input and outputs a scalar.
Revised: Thus, an EBM can be represented by an neural network that takes x as input and outputs a scalar.

Original: Training an EBM through maximum likelihood (ML) is not straightforward, as Z ( θ ) cannot be reliably computed, since this involves integration over the entire input domain of x .
Revised: Training an EBM through maximum likelihood (ML) is not straightforward, as Z ( θ ) cannot be reliably computed, since this involves integration over the entire input domain of x .

Original: However, the gradient of log-likelihood with respect to a data sample x can be represented as
Revised: However, the gradient of log-likelihood with respect to a data sample x can be represented as

Original: [Equation]
Revised: [Equation]

Original: Note that Equation 1 is still not tractable, as it requires using Markov Chain Monte Carlo (MCMC) to draw samples from the model distribution p θ ( x ) , which often takes exponentially long to mix.
Revised: Note that Equation 1 is still not tractable, as it requires using Markov Chain Monte Carlo (MCMC) to draw samples from the model distribution p θ ( x ) , which often takes exponentially long to mix.

Original: As a practical approximation to the above objective, (Hinton, 2002) proposes the contrastive divergence objective
Revised: As a practical approximation to the above objective, (Hinton, 2002) proposes the contrastive divergence objective

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where Π θ represents a MCMC transition kernel for p θ , and Π tθ ( p ( x )) represents t sequential MCMC transitions starting from p ( x ) .
Revised: where Π θ represents a MCMC transition kernel for p θ , and Π tθ ( p ( x )) represents t sequential MCMC transitions starting from p ( x ) .

Original: The above objective can be seen as an improvement operator, where
Revised: The above objective can be seen as an improvement operator, where

Original: KL [Equation] p
Revised: KL ( p ( x ) || p

Original: [Equation]

Original: x ) || p θ ( x )) ≥ KL (Π tθ ( p ( x ))
Revised: θ ( x )) ≥ KL (Π tθ ( p ( x ))

Original: ||
Revised: ||

Original: p θ ( x )) , because Π θ is converging to equilibrium distribution p θ ( x ) (Lyu, 2011).
Revised: p θ ( x )) , because Π θ is converging to equilibrium distribution p θ ( x ) (Lyu, 2011).

Original: Furthermore, the above objective is only zero (at its ﬁxed point), when Π θ does not change the distribution of p ( x ) , which corresponds to p θ ( x )
Revised: Furthermore, the above objective is only zero (at its ﬁxed point), when Π θ does not change the distribution of p ( x ) , which corresponds to p θ ( x )

Original: = p ( x ) .
Revised: = p ( x ) .

Original: 2. A M ISSING T ERM IN C ONTRASTIVE D IVERGENCE
Revised: 2. A M ISSING T ERM IN C ONTRASTIVE D IVERGENCE

Original: When taking the negative gradient of the contrastive divergence objective (Equation 2)
Revised: When taking the negative gradient of the contrastive divergence objective (Equation 2)

Original: [Equation]
Revised: [Equation]

Original: we obtain the expression
Revised: we obtain the expression

Original: [Equation]
Revised: [Equation]

Original: where for brevity
Revised: where for brevity

Original: [Equation]
Revised: [Equation]

Original: we summarize Π tθ ( p ( x ))
Revised: we summarize Π tθ ( p ( x ))

Original: =
Revised: =

Original: q θ ( x ) .
Revised: q θ ( x ) .

Original: The ﬁrst two terms are identical to those of
Revised: The ﬁrst two terms are identical to those of

Original: Equation 1 and the third gradient term (which we refer to as the KL divergence term) corresponds to minimizing the divergence between q θ ( x ) and p θ ( x ) .
Revised: Equation 1 and the third gradient term (which we refer to as the KL divergence term) corresponds to minimizing the divergence between q θ ( x ) and p θ ( x ) .

Original: In practice, past contrastive divergence approaches have ignored the third gradient term, which was difﬁcult to estimate and claimed to be empirically negligible (Hinton, 1999) (which in Figure 13 we show to be non-negligible).
Revised: In practice, past contrastive divergence approaches have ignored the third gradient term, which was difﬁcult to estimate and claimed to be empirically negligible (Hinton, 1999) (which in Figure 13 we show to be non-negligible).

Original: These gradients correspond to a joint loss expression L Full , consisting of traditional contrastive loss L CD and a new loss expression L KL .
Revised: These gradients correspond to a joint loss expression L Full , consisting of traditional contrastive loss L CD and a new loss expression L KL .

Original: Speciﬁcally, we have L Full =
Revised: Speciﬁcally, we have L Full =

Original: L CD + L KL where L CD is
Revised: L CD + L KL where L CD is

Original: [Equation]
Revised: [Equation]

Original: and the ignored KL divergence term corresponding to the following KL loss:
Revised: and the ignored KL divergence term corresponding to the following KL loss:

Original: [Equation]
Revised: [Equation]

Original: Despite being difﬁcult to estimate, we show that L KL is a useful tool for both speeding up and stabilizing training of EBMs.
Revised: Despite being difﬁcult to estimate, we show that L KL is a useful tool for both speeding up and stabilizing training of EBMs.

Original: Figure 2 illustrates the overall effects of both losses.
Revised: Figure 2 illustrates the overall effects of both losses.

Original: Equationencourage the energy function to assign low energy to real samples and high energy for generated samples.
Revised: Equationencourage the energy function to assign low energy to real samples and high energy for generated samples.

Original: However, only optimizing Equation 4 often leads to an adversarial mode where the energy function learns to simply generate an energy landscape that makes sampling difﬁcult.
Revised: However, only optimizing Equation 4 often leads to an adversarial mode where the energy function learns to simply generate an energy landscape that makes sampling difﬁcult.

Original: The
Revised: The

Original: KL divergence term counteracts this effect and encourages sampling to closely approximate the underlying distribution p θ ( x ) , by encouraging samples to be both low energy under the energy function as well as diverse.
Revised: KL divergence term counteracts this effect and encourages sampling to closely approximate the underlying distribution p θ ( x ) , by encouraging samples to be both low energy under the energy function as well as diverse.

Original: Empirically, we ﬁnd that including for KL term signiﬁcantly improves both the stability, generation quality, and robustness to different model architectures (Figure 8).
Revised: Empirically, we ﬁnd that including for KL term signiﬁcantly improves both the stability, generation quality, and robustness to different model architectures (Figure 8).

Original: Next, we will discuss our approach towards estimating this KL divergence.
Revised: Next, we will discuss our approach towards estimating this KL divergence.

Original: 2. E STIMATING THE MISSING GRADIENT TERM
Revised: 2. E STIMATING THE MISSING GRADIENT TERM

Original: Estimating L KL can further be decomposed into two separate objectives, minimizing the energy of samples from q θ ( x ) , which we refer to as L opt (Equation 6) and maximizing the entropy of samples from q θ ( x ) which we refer to as L ent (Equation 7).
Revised: Estimating L KL can further be decomposed into two separate objectives, minimizing the energy of samples from q θ ( x ) , which we refer to as L opt (Equation 6) and maximizing the entropy of samples from q θ ( x ) which we refer to as L ent (Equation 7).

Original: Minimizing Sampler Energy.
Revised: Minimizing Sampler Energy.

Original: To minimize the energy of samples from q θ ( x ) we can directly differentiate through both the energy function and MCMC sampling.
Revised: To minimize the energy of samples from q θ ( x ) we can directly differentiate through both the energy function and MCMC sampling.

Original: We follow recent work in EBMs and utilize Langevin dynamics (Du & Mordatch, 2019; Nijkamp et al., 2019b; Grathwohl et al., 2019)
Revised: We follow recent work in EBMs and utilize Langevin dynamics (Du & Mordatch, 2019; Nijkamp et al., 2019b; Grathwohl et al., 2019)

Original: [Figure]
Revised: [Figure]

Original: for our MCMC transition kernel, and note that each step of Langevin sampling is fully differentiable with respect to underlying energy function parameters.
Revised: for our MCMC transition kernel, and note that each step of Langevin sampling is fully differentiable with respect to underlying energy function parameters.

Original: Precisely, gradient of L opt , ∂ L opt ∂θ becomes
Revised: Precisely, gradient of L opt , ∂ L opt ∂θ becomes

Original: [Figure]
Revised: [Figure]

Original: 0 ,..., x (cid:48) t ) (cid:34) ∂E stop grad ( θ ) ( x (cid:48)
Revised: 0 ,..., x (cid:48) t ) (cid:34) ∂E stop grad ( θ ) ( x (cid:48)

Original: t − 1 − ∇ x (cid:48)
Revised: t − 1 − ∇ x (cid:48)

Original: t − 1 E θ ( x (cid:48)
Revised: t − 1 E θ ( x (cid:48)

Original: t − 1 ) + ω ) ∂θ (cid:35) , (6) where ω ∼ N (0 , λ ) and x (cid:48)
Revised: t − 1 ) + ω ) ∂θ (cid:35) , (6) where ω ∼ N (0 , λ ) and x (cid:48)

Original: i represents the i th step of Langevin sampling.
Revised: i represents the i th step of Langevin sampling.

Original: To reduce the memory overhead of this differentiation procedure, we only differentiate through the last step of Langevin sampling as also done in (Vahdat et al., 2020).
Revised: To reduce the memory overhead of this differentiation procedure, we only differentiate through the last step of Langevin sampling as also done in (Vahdat et al., 2020).

Original: In the appendix we show that this leads to the same effect as differentiation through Langevin sampling.
Revised: In the appendix we show that this leads to the same effect as differentiation through Langevin sampling.

Original: Entropy Estimation.
Revised: Entropy Estimation.

Original: To maximize the entropy of samples from q θ ( x ) , we use a non-parametric nearest neighbor entropy estimator (Beirlant et al., 1997), which is shown to be mean square consistent (Kozachenko & Leonenko, 1987) with root-n convergence rate (Tsybakov & Van der Meulen, 1996).
Revised: To maximize the entropy of samples from q θ ( x ) , we use a non-parametric nearest neighbor entropy estimator (Beirlant et al., 1997), which is shown to be mean square consistent (Kozachenko & Leonenko, 1987) with root-n convergence rate (Tsybakov & Van der Meulen, 1996).

Original: The entropy H of a distribution p ( x ) can be estimated through a set X = x 1 , x 2 , . . .
Revised: The entropy H of a distribution p ( x ) can be estimated through a set X = x 1 , x 2 , . . .

Original: , x n of n different points sampled from p ( x ) as H ( p θ ( x )) = 1 n (cid:80) ni
Revised: , x n of n different points sampled from p ( x ) as H ( p θ ( x )) = 1 n (cid:80) ni

Original: =1 ln( n · NN( x i , X ))
Revised: =1 ln( n · NN( x i , X ))

Original: + O (1) where the function NN( x i , X ) denotes the nearest neighbor distance of x i to any other data point in X .
Revised: + O (1) where the function NN( x i , X ) denotes the nearest neighbor distance of x i to any other data point in X .

Original: Based off the above entropy estimator, we write L ent as the entropy loss:
Revised: Based off the above entropy estimator, we write L ent as the entropy loss:

Original: [Equation]
Revised: [Equation]

Original: Energy !

Original: +

Original: [Figure]

Original: [Equation]

Original: [Figure]

Original: [Equation]

Original: [Figure]
Revised: [Figure]

Original: where we measure the nearest neighbor with respect to a set B of 100 past samples from
Revised: where we measure the nearest neighbor with respect to a set B of 100 past samples from

Original: MCMC chains.
Revised: MCMC chains.

Original: We utilize L2 distance as the metric for computing nearest neighbors.
Revised: We utilize L2 distance as the metric for computing nearest neighbors.

Original: 2. D ATA A UGMENTATION T RANSITIONS
Revised: 2. D ATA A UGMENTATION T RANSITIONS

Original: Langevin sampling, our MCMC transition kernel, is prone to falling into local probability modes (Neal, 2011).
Revised: Langevin sampling, our MCMC transition kernel, is prone to falling into local probability modes (Neal, 2011).

Original: In the image domain, this manifests with sampling chains always converging to a ﬁxed image (Du & Mordatch, 2019).
Revised: In the image domain, this manifests with sampling chains always converging to a ﬁxed image (Du & Mordatch, 2019).

Original: To encourage greater exploration between similar inputs in our model, we propose to augment chains of MCMC sampling with periodic data 
Revised: To encourage greater exploration between similar inputs in our model, we propose to augment chains of MCMC sampling with periodic data augmentation transitions that encourages movement between “similar” inputs. In particular, we

Original: [Figure]
Revised: [Figure]

Original: augmentation transitions that encourages movement between “similar” inputs.

Original: In particular, we utilize a combination of color, horizontal ﬂip, rescaling, and Gaussian blur augmentations.
Revised: utilize a combination of color, horizontal ﬂip, rescaling, and Gaussian blur augmentations.

Original: Such combinations of augmentation has recently seen success applied in unsupervised learning (Chen et al., 2020).
Revised: Such combinations of augmentation has recently seen success applied in unsupervised learning (Chen et al., 2020).

Original: During training time, we initialize MCMC sampling from a data augmentation applied to an input sampled from the buffer of past samples.
Revised: During training time, we initialize MCMC sampling from a data augmentation applied to an input sampled from the buffer of past samples.

Original: At test time, during the generation, we apply a random augmentation to the input after every 20 steps of Langevin sampling.
Revised: At test time, during the generation, we apply a random augmentation to the input after every 20 steps of Langevin sampling.

Original: We illustrate this process in the bottom of Figure 2.
Revised: We illustrate this process in the bottom of Figure 2.

Original: Data augmentation transitions are always taken.
Revised: Data augmentation transitions are always taken.

Original: 2. C OMPOSITIONAL M ULTI - SCALE G ENERATION
Revised: 2. C OMPOSITIONAL M ULTI - SCALE G ENERATION

Original: To encourage energy functions to focus on features in both low and high resolutions, we deﬁne our energy function as the composition (sum) of a set of energy functions operating on different scales of an image, illustrated in Figure 5.
Revised: To encourage energy functions to focus on features in both low and high resolutions, we deﬁne our energy function as the composition (sum) of a set of energy functions operating on different scales of an image, illustrated in Figure 5.

Original: Since the downsampling operation is fully differentiable, Langevin based sampling can be directly applied to the energy function.
Revised: Since the downsampling operation is fully differentiable, Langevin based sampling can be directly applied to the energy function.

Original: In our experiments, we utilize full, half, and quarter resolution image as input and show that this improves the generation performance.
Revised: In our experiments, we utilize full, half, and quarter resolution image as input and show that this improves the generation performance.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We perform empirical experiments to validate the following set of questions: (1) What are the effects of each proposed component towards training EBMs?
Revised: We perform empirical experiments to validate the following set of questions: (1) What are the effects of each proposed component towards training EBMs?

Original: (2) Are our trained EBMs able to perform well on downstream applications of EBMs, such as image generation, out-of-distribution detection, and concept compositionality?
Revised: (2) Are our trained EBMs able to perform well on downstream applications of EBMs, such as image generation, out-of-distribution detection, and concept compositionality?

Original: 3. I MAGE G ENERATION
Revised: 3. I MAGE G ENERATION

Original: We evaluate our approach on CIFAR-10, Imagenet 32x32 (Deng et al., 2009), and CelebA-HQ (Karras et al., 2017) datasets.
Revised: We evaluate our approach on CIFAR-10, Imagenet 32x32 (Deng et al., 2009), and CelebA-HQ (Karras et al., 2017) datasets.

Original: Additional quantitative comparisons, results, and ablations can be found in the appendix of the paper.
Revised: Additional quantitative comparisons, results, and ablations can be found in the appendix of the paper.

Original: Image Quality.
Revised: Image Quality.

Original: We evaluate our approach on unconditional generation in Table 3.
Revised: We evaluate our approach on unconditional generation in Table 3.

Original: We utilize
Revised: We utilize

Original: Inception (Salimans et al., 2016) and FID (Heusel et al., 2017) implementations from (Du & Mordatch, 2019) to evaluate samples.
Revised: Inception (Salimans et al., 2016) and FID (Heusel et al., 2017) implementations from (Du & Mordatch, 2019) to evaluate samples.

Original: On CIFAR-10, we ﬁnd that our approach outperforms many past EBM approaches in both FID and Inception scores using a similar number of parameters.
Revised: On CIFAR-10, we ﬁnd that our approach outperforms many past EBM approaches in both FID and Inception scores using a similar number of parameters.

Original: We ﬁnd that our performance is slightly worse than that of SNGAN and SSGAN.
Revised: We ﬁnd that our performance is slightly worse than that of SNGAN and SSGAN.

Original: Effect of Data Augmentation.
Revised: Effect of Data Augmentation.

Original: We evaluate the effect of data augmentation on sampling in EBMs.
Revised: We evaluate the effect of data augmentation on sampling in EBMs.

Original: In Figure 6 we show that by combining Langevin sampling with data augmentation transitions, we are able to enable chains to mix across different images, whereas prior works have shown Langevin converging to ﬁxed images.
Revised: In Figure 6 we show that by combining Langevin sampling with data augmentation transitions, we are able to enable chains to mix across different images, whereas prior works have shown Langevin converging to ﬁxed images.

Original: In Figure 12 we show that given a ﬁxed random noise initialization, data augmentation transitions enable to reach a diverse number of different samples, while sampling without data augmentation transitions leads all chains to converge to the same face.
Revised: In Figure 12 we show that given a ﬁxed random noise initialization, data augmentation transitions enable to reach a diverse number of different samples, while sampling without data augmentation transitions leads all chains to converge to the same face.

Original: Mode Convergence.
Revised: Mode Convergence.

Original: We further investigate high likelihood modes of our model.
Revised: We further investigate high likelihood modes of our model.

Original: In Figure 7, we compare very low energy samples (obtained after running gradient descent 1000 steps on an energy function) for both our model with data augmentation and KL loss and the IGEBM model.
Revised: In Figure 7, we compare very low energy samples (obtained after running gradient descent 1000 steps on an energy function) for both our model with data augmentation and KL loss and the IGEBM model.

Original: Due to improved mode exploration, we ﬁnd that low temperature samples under our model with data augmentation/KL loss reﬂect typical high likelihood ”modes” in the training dataset, while our baseline models converges to odd shapes, also noted in (Nijkamp et al., 2019a).
Revised: Due to improved mode exploration, we ﬁnd that low temperature samples under our model with data augmentation/KL loss reﬂect typical high likelihood ”modes” in the training dataset, while our baseline models converges to odd shapes, also noted in (Nijkamp et al., 2019a).

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure] [Table] [Figure] [Table]
Revised: No

Original: [Figure]
Revised: [Figure]

Revised: KL Loss

Original: I n [Figure] p ti [Figure]
Revised: I n ce p ti on

Original: S c
Revised: S c

Original: o r e
Revised: o r e

Original: E n e r gy
Revised: E n e r gy

Original: D i ff e r e n ce
Revised: D i ff e r e n ce

Original: Stability/KL Loss.
Revised: Stability/KL Loss.

Original: EBMs are difﬁcult to train and are sensitive to both the exact architecture and to various hyper-parameters.
Revised: EBMs are difﬁcult to train and are sensitive to both the exact architecture and to various hyper-parameters.

Original: We found that the addition of a KL term into our training objective signiﬁcantly improved the stability of training, by encouraging the sampling distribution to match the model distribution.
Revised: We found that the addition of a KL term into our training objective signiﬁcantly improved the stability of training, by encouraging the sampling distribution to match the model distribution.

Original: In Figure 8, we measure training stability by measuring the energy differences between real and generated images.
Revised: In Figure 8, we measure training stability by measuring the energy differences between real and generated images.

Original: Stable training occurs when the energy difference is close to zero.
Revised: Stable training occurs when the energy difference is close to zero.

Original: Without L KL , we found that training our model with or without self-attention were both unstable, with differences spiking.
Revised: Without L KL , we found that training our model with or without self-attention were both unstable, with differences spiking.

Original: Adding spectral normalization stabilizes training, but the addition of selfattention once again destabilizes training.
Revised: Adding spectral normalization stabilizes training, but the addition of selfattention once again destabilizes training.

Original: In contrast with L KL , the addition of self-attention is also stable.
Revised: In contrast with L KL , the addition of self-attention is also stable.

Original: We further compare Inception scores in Figure 8 over training and ﬁnd that while spectral normalization stabilizes training, it does at the expense of decreased improvement of Inception score.
Revised: We further compare Inception scores in Figure 8 over training and ﬁnd that while spectral normalization stabilizes training, it does at the expense of decreased improvement of Inception score.

Original: 3. O UT OF D ISTRIBUTION R OBUSTNESS
Revised: 3. O UT OF D ISTRIBUTION R OBUSTNESS

Original: Energy-Based Models (EBMs) have also been shown to exhibit robustness to both out-ofdistribution and adversarial samples (Du & Mordatch, 2019; Grathwohl et al., 2019; 2020a).
Revised: Energy-Based Models (EBMs) have also been shown to exhibit robustness to both out-ofdistribution and adversarial samples (Du & Mordatch, 2019; Grathwohl et al., 2019; 2020a).

Original: We evaluate out-of-distribution detection of our trained energy function through log-likelihood using the
Revised: We evaluate out-of-distribution detection of our trained energy function through log-likelihood using the

Original: AUROC evaluation metrics proposed in Hendrycks & Gimpel (2016).
Revised: AUROC evaluation metrics proposed in Hendrycks & Gimpel (2016).

Original: Results.
Revised: Results.

Original: We present out-of-distribution results in Table 1, comparing with both likelihood models and EBMs using log-likelihood to detect outliers.
Revised: We present out-of-distribution results in Table 1, comparing with both likelihood models and EBMs using log-likelihood to detect outliers.

Original: We ﬁnd that our approach signiﬁcantly outperforms other baselines, with the exception of CIFAR-10 interpolations.
Revised: We ﬁnd that our approach signiﬁcantly outperforms other baselines, with the exception of CIFAR-10 interpolations.

Original: We note the JEM (Grathwohl et al., 2019) further requires supervised labels to train the energy function, which has to shown to improve out-of-distribution performance.
Revised: We note the JEM (Grathwohl et al., 2019) further requires supervised labels to train the energy function, which has to shown to improve out-of-distribution performance.

Original: We posit that by more efﬁciently exploring modes of the energy distribution at training time, we are able to reduce the spurious modes of the energy function and thus improve out-of-distribution performance.
Revised: We posit that by more efﬁciently exploring modes of the energy distribution at training time, we are able to reduce the spurious modes of the energy function and thus improve out-of-distribution performance.

Original: 3. C OMPOSITIONALITY
Revised: 3. C OMPOSITIONALITY

Original: Energy-Based Models (EBMs) have the ability to compose with other models at generation time (Hinton, 1999; Haarnoja et al., 2017; Du et al., 2020a).
Revised: Energy-Based Models (EBMs) have the ability to compose with other models at generation time (Hinton, 1999; Haarnoja et al., 2017; Du et al., 2020a).

Original: We investigate to what extent EBMs trained under our new framework can also exhibit compositionality.
Revised: We investigate to what extent EBMs trained under our new framework can also exhibit compositionality.

Original: See (Du et al., 2020a) for a discussion of various compositional operators and applications in EBMs.
Revised: See (Du et al., 2020a) for a discussion of various compositional operators and applications in EBMs.

Original: In particular, we train independent
Revised: In particular, we train independent

Original: EBMs E ( x | c 1 ) , E ( x | c 2 ) , E ( x | c 3 ) , that learn conditional generative distribution over concept factors c such as facial expression.
Revised: EBMs E ( x | c 1 ) , E ( x | c 2 ) , E ( x | c 3 ) , that learn conditional generative distribution over concept factors c such as facial expression.

Original: We test whether we can compose independent energy functions together to generate images with each concept factor simultaneously.
Revised: We test whether we can compose independent energy functions together to generate images with each concept factor simultaneously.

Original: We test compositions on the CelebA-HQ dataset, where we train separate EBMs on face attributes of age, gender, smiling, and wavy hair and on a rendered Blender dataset where we train separate EBMs on object attributes of size, position, rotation, and identity.
Revised: We test compositions on the CelebA-HQ dataset, where we train separate EBMs on face attributes of age, gender, smiling, and wavy hair and on a rendered Blender dataset where we train separate EBMs on object attributes of size, position, rotation, and identity.

Original: [Figure]
Revised: [Figure]

Original: Qualitative Results.
Revised: Qualitative Results.

Original: We present qualitative results on compositions of energy functions on
Revised: We present qualitative results on compositions of energy functions on

Original: CelebA-HQ in Figure 9.
Revised: CelebA-HQ in Figure 9.

Original: We consider compositional generation on the factors of young, female, smiling, and wavy hair.
Revised: We consider compositional generation on the factors of young, female, smiling, and wavy hair.

Original: Compared to baselines, our approach is able to successfully generate images with each of conditioned factor, with faces being signiﬁcantly higher resolution than baselines.
Revised: Compared to baselines, our approach is able to successfully generate images with each of conditioned factor, with faces being signiﬁcantly higher resolution than baselines.

Original: In Figure 11, we further consider compositions of energy functions over object attributes.
Revised: In Figure 11, we further consider compositions of energy functions over object attributes.

Original: We consider compositional generation on factors of size, type, position, and rotation.
Revised: We consider compositional generation on factors of size, type, position, and rotation.

Original: Again, we ﬁnd that each associated image generation exhibits the corresponding conditioned attribute.
Revised: Again, we ﬁnd that each associated image generation exhibits the corresponding conditioned attribute.

Original: Images are further visually consistent in terms of lighting, shadows and reﬂections.
Revised: Images are further visually consistent in terms of lighting, shadows and reﬂections.

Original: We note that different from most past work, generations of these combinations of different factors are only speciﬁed at generation time, with models being trained independently .
Revised: We note that different from most past work, generations of these combinations of different factors are only speciﬁed at generation time, with models being trained independently .

Original: Our results indicate that our framework for training EBMs is a promising direction for high resolution compositional visual generation.
Revised: Our results indicate that our framework for training EBMs is a promising direction for high resolution compositional visual generation.

Original: Quantitative Results.
Revised: Quantitative Results.

Original: We quantitatively compare compositional generations of our model with
Revised: We quantitatively compare compositional generations of our model with

Original: IGEBM and JVAE (Vedantam et al., 2018) models on the CelebA-HQ dataset in Section B.3.
Revised: IGEBM and JVAE (Vedantam et al., 2018) models on the CelebA-HQ dataset in Section B.3 utilizing a pretrained ResNet 18 classiﬁer.

Revised: We ﬁnd better performance at recovering the underlying attributes.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We propose a simple and general framework for improving generation and ease of training of EBMs.
Revised: We propose a simple and general framework for improving generation and ease of training of EBMs.

Original: We show that the framework enables high resolution compositional image generation and out-ofdistribution robustness.
Revised: We show that the framework enables high resolution compositional image generation and out-ofdistribution robustness.

Original: In the future, we are interested in further computational scaling of our framework and its applications to additional domains such as text, video, and reasoning.
Revised: In the future, we are interested in further computational scaling of our framework and its applications to additional domains such as text, video, and reasoning.

Original: A R ELATED W ORK
Revised: A R ELATED W ORK

Original: Our work is related to a large, growing body of work on different approaches for training EBMs.
Revised: Our work is related to a large, growing body of work on different approaches for training EBMs.

Original: Our approach is based on contrastive divergence (Hinton, 2002), where an energy function is trained to contrast negative samples drawn from a model distribution and from real data.
Revised: Our approach is based on contrastive divergence (Hinton, 2002), where an energy function is trained to contrast negative samples drawn from a model distribution and from real data.

Original: In recent years, such approaches have been applied to the image domain (Salakhutdinov & Hinton, 2009; Xie et al., 2016;
Revised: In recent years, such approaches have been applied to the image domain (Salakhutdinov & Hinton, 2009; Xie et al., 2016;

Original: Gao et al., 2018; Du & Mordatch, 2019).
Revised: Gao et al., 2018; Du & Mordatch, 2019).

Original: (Gao et al., 2018) also proposes a multiscale approach towards generating images from EBMs, but different from our work, uses each sub-scale EBM to initialize the generation of the next EBM, while we jointly sample across each resolution.
Revised: (Gao et al., 2018) also proposes a multiscale approach towards generating images from EBMs, but different from our work, uses each sub-scale EBM to initialize the generation of the next EBM, while we jointly sample across each resolution.

Original: Our work builds on existing works, and aims to provide improvements in generation and stability.
Revised: Our work builds on existing works, and aims to provide improvements in generation and stability.

Original: A difﬁculty with contrastive divergence training is negative sample generation.
Revised: A difﬁculty with contrastive divergence training is negative sample generation.

Original: To sidestep this issue, a separate line of work utilizes an auxiliary network to amortize the negative portions of the sampling procedure (Kim & Bengio, 2016;
Revised: To sidestep this issue, a separate line of work utilizes an auxiliary network to amortize the negative portions of the sampling procedure (Kim & Bengio, 2016;

Original: Kumar et al., 2019; Han et al., 2019; Xie et al., 2018a; Song & Ou, 2018; Dai et al., 2019; Che et al., 2020; Grathwohl et al., 2020a; Arbel et al., 2020).
Revised: Kumar et al., 2019; Han et al., 2019; Xie et al., 2018a; Song & Ou, 2018; Dai et al., 2019; Che et al., 2020; Grathwohl et al., 2020a; Arbel et al., 2020).

Original: One line of work (Kim & Bengio, 2016; Kumar et al., 2019; Song & Ou, 2018), utilizes a separate generator network for negative image sample generations.
Revised: One line of work (Kim & Bengio, 2016; Kumar et al., 2019; Song & Ou, 2018), utilizes a separate generator network for negative image sample generations.

Original: In contrast, (Xie et al., 2018a), utilizes a generator to warm start generations for negative samples and (Han et al., 2019) minimizes a divergence triangle between three models.
Revised: In contrast, (Xie et al., 2018a), utilizes a generator to warm start generations for negative samples and (Han et al., 2019) minimizes a divergence triangle between three models.

Original: While such approaches enable better qualitative generation, they also lose some of the ﬂexibility of the EBM formulation.
Revised: While such approaches enable better qualitative generation, they also lose some of the ﬂexibility of the EBM formulation.

Original: For example, separate energy models can no longer be composed together for generation.
Revised: For example, separate energy models can no longer be composed together for generation.

Original: In addition, other approaches towards training EBMs seek instead to investigate separate objectives to train the EBM.
Revised: In addition, other approaches towards training EBMs seek instead to investigate separate objectives to train the EBM.

Original: One such approach is score matching, where the gradients of an energy function are trained to match the gradients of real data (Hyv ¨ arinen, 2005; Song & Ermon, 2019), or with a related denoising (Sohl-Dickstein et al., 2015;
Revised: One such approach is score matching, where the gradients of an energy function are trained to match the gradients of real data (Hyv ¨ arinen, 2005; Song & Ermon, 2019), or with a related denoising (Sohl-Dickstein et al., 2015;

Original: Saremi et al., 2018; Ho et al., 2020) approach.
Revised: Saremi et al., 2018; Ho et al., 2020) approach.

Original: Additional objectives include noise contrastive estimation (Gao et al., 2020), learned Steins discrepancy (Grathwohl et al., 2020b), and learned F divergences (Yu et al., 2020).
Revised: Additional objectives include noise contrastive estimation (Gao et al., 2020), learned Steins discrepancy (Grathwohl et al., 2020b), and learned F divergences (Yu et al., 2020).

Original: Most prior work in contrastive divergence has ignored the KL term (Hinton, 1999; Salakhutdinov &
Revised: Most prior work in contrastive divergence has ignored the KL term (Hinton, 1999; Salakhutdinov &

Original: Hinton, 2009).
Revised: Hinton, 2009).

Original: A notable exception is (Ruiz & Titsias, 2019), which obtains a similar KL divergence term to ours.
Revised: A notable exception is (Ruiz & Titsias, 2019), which obtains a similar KL divergence term to ours.

Original: Ruiz & Titsias (2019) use a high variance REINFORCE estimator to estimate the gradient of the KL term, while our approach relies on auto-differentiation and nearest neighbor entropy estimators.
Revised: Ruiz & Titsias (2019) use a high variance REINFORCE estimator to estimate the gradient of the KL term, while our approach relies on auto-differentiation and nearest neighbor entropy estimators.

Original: Differentiation through model generation procedures has previously been explored in other models (Finn & Levine, 2017;
Revised: Differentiation through model generation procedures has previously been explored in other models (Finn & Levine, 2017;

Original: Metz et al., 2016).
Revised: Metz et al., 2016).

Original: Other related entropy estimators include those based on Stein’s identity (Liu et al., 2017) and MINE (Belghazi et al., 2018).
Revised: Other related entropy estimators include those based on Stein’s identity (Liu et al., 2017) and MINE (Belghazi et al., 2018).

Original: In contrast to these approaches, our entropy estimator relies only on nearest neighbor calculation, and does not require the training of an independent neural network.
Revised: In contrast to these approaches, our entropy estimator relies only on nearest neighbor calculation, and does not require the training of an independent neural network.

Original: B M ORE I MAGE R ESULTS B. N EAREST N EIGHBOR G ENERATIONS
Revised: B M ORE I MAGE R ESULTS B. N EAREST N EIGHBOR G ENERATIONS

Original: We present L2 nearest neighbors in CelebA-HQ training dataset of unconditional image samples from our trained EBM in Figure 14.
Revised: We present L2 nearest neighbors in CelebA-HQ training dataset of unconditional image samples from our trained EBM in Figure 14.

Original: We ﬁnd that our approach generates images distinct from the training set.
Revised: We ﬁnd that our approach generates images distinct from the training set.

Original: B. LSUN Q UANTITATIVE R ESULTS
Revised: B. LSUN Q UANTITATIVE R ESULTS

Original: We further quantitatively compare our generations with those of SNGAN on LSUN 128x128 bedroom scenes.
Revised: We further quantitatively compare our generations with those of SNGAN on LSUN 128x128 bedroom scenes.

Original: We ﬁnd that an SNGAN model trained on LSUN 128x128 bedroom scenes obtains an FID of 64.05 compared to our approach, which obtains an FID of 33.46.
Revised: We ﬁnd that an SNGAN model trained on LSUN 128x128 bedroom scenes obtains an FID of 64.05 compared to our approach, which obtains an FID of 33.46.

Original: To report SNGAN scores, we re-implemented the SNGAN model using the default hyper parameters to train models on ImageNet
Revised: To report SNGAN scores, we re-implemented the SNGAN model using the default hyper parameters to train models on ImageNet

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: B. C OMPOSITIONAL Q UANTITATIVE R ESULTS
Revised: B. C OMPOSITIONAL Q UANTITATIVE R ESULTS

Original: We quantitatively compare compositional generations of our model with IGEBM and JVAE (Vedantam et al., 2018) models on the CelebA-HQ dataset.
Revised: We quantitatively compare compositional generations of our model with IGEBM and JVAE (Vedantam et al., 2018) models on the CelebA-HQ dataset.

Original: We assess the compositional generation accuracy of different models by measuring the accuracy with which a pretrained ResNet18 classiﬁer can recover the underlying conditioned attributes.
Revised: We assess the compositional generation accuracy of different models by measuring the accuracy with which a pretrained ResNet18 classiﬁer can recover the underlying conditioned attributes.

Original: In Table 3, we ﬁnd that our model has signiﬁcantly higher attribute recovery compared to baselines across all compositions of attributes.
Revised: In Table 3, we ﬁnd that our model has signiﬁcantly higher attribute recovery compared to baselines across all compositions of attributes.

Original: B. A BLATIONS
Revised: B. A BLATIONS

Original: We ablate each portion of our proposed approach in Table 2.
Revised: We ablate each portion of our proposed approach in Table 2.

Original: We ﬁnd that each our proposed components have signiﬁcant gains in generation performance.
Revised: We ﬁnd that each our proposed components have signiﬁcant gains in generation performance.

Original: In particular, we ﬁnd a large gain in overall generative performance when adding the KL loss.
Revised: In particular, we ﬁnd a large gain in overall generative performance when adding the KL loss.

Original: This is in part due to a large boost in training stability (Figure 8), enabling signiﬁcantly longer training times with both multiscale sampling and data augmentation.
Revised: This is in part due to a large boost in training stability (Figure 8), enabling signiﬁcantly longer training times with both multiscale sampling and data augmentation.

Original: B. Q UALITATIVE C OMPOSITIONAL S AMPLES
Revised: B. Q UALITATIVE C OMPOSITIONAL S AMPLES

Original: In Figure 11, we further consider compositions of energy functions over object attributes.
Revised: In Figure 11, we further consider compositions of energy functions over object attributes.

Original: We consider compositional generation on factors of size, type, position, and rotation.
Revised: We consider compositional generation on factors of size, type, position, and rotation.

Original: We ﬁnd that each associated image generation exhibits the corresponding conditioned attribute.
Revised: We ﬁnd that each associated image generation exhibits the corresponding conditioned attribute.

Original: Images are further visually consistent in terms of lighting, shadows and reﬂections.
Revised: Images are further visually consistent in terms of lighting, shadows and reﬂections.

Original: We note that different from most past work, generations of these combinations of different factors are only speciﬁed at generation time, with models being trained independently .
Revised: We note that different from most past work, generations of these combinations of different factors are only speciﬁed at generation time, with models being trained independently .

Original: Our results indicate that our framework for training EBMs is a promising direction for high resolution compositional visual generation.
Revised: Our results indicate that our framework for training EBMs is a promising direction for high resolution compositional visual generation.

Original: B. U NCONDITIONAL I MAGE S AMPLES
Revised: B. U NCONDITIONAL I MAGE S AMPLES

Original: We present qualitative visualizations of unconditional samples generated from an EBM.
Revised: We present qualitative visualizations of unconditional samples generated from an EBM.

Original: Figureshows unconditional image generations from LSUN bedroom scenes.
Revised: Figureshows unconditional image generations from LSUN bedroom scenes.

Original: Figure 16 shows unconditional image generations on the CIFAR-10 dataset.
Revised: Figure 16 shows unconditional image generations on the CIFAR-10 dataset.

Original: Finally, Figure 17 shows unconditional image generations on the ImageNet 32x32 dataset.
Revised: Finally, Figure 17 shows unconditional image generations on the ImageNet 32x32 dataset.

Original: In all three different settings, we ﬁnd that our generated unconditional images are relatively globally coherent.
Revised: In all three different settings, we ﬁnd that our generated unconditional images are relatively globally coherent.

Original: [Figure]
Revised: [Figure]

Original: In this part, we provide the model architectures used in our experiments.
Revised: In this part, we provide the model architectures used in our experiments.

Original: When training multiscale energy functions, our ﬁnal output energy function is the sum of energy functions applied to the full resolution image, the half resolution image, and the quarter resolution image.
Revised: When training multiscale energy functions, our ﬁnal output energy function is the sum of energy functions applied to the full resolution image, the half resolution image, and the quarter resolution image.

Original: We use the architecture reported in Table 4 for the full resolution image on CIFAR-10 and ImageNet 32x32 (used in the main paper Section 3.2 and 3.3).
Revised: We use the architecture reported in Table 4 for the full resolution image on CIFAR-10 and ImageNet 32x32 (used in the main paper Section 3.2 and 3.3).

Original: The model architecture used on the CelebA-HQ and LSUN datasets are reported in Table 5 (used in the main paper Section 3. and 3.4).
Revised: The model architecture used on the CelebA-HQ and LSUN datasets are reported in Table 5 (used in the main paper Section 3. and 3.4).

Original: The half-resolution models share the architecture listed in Table 4, but with the ﬁrst down-sampled residual block removed.
Revised: The half-resolution models share the architecture listed in Table 4, but with the ﬁrst down-sampled residual block removed.

Original: Similarly, the quarter resolution models share the architectures listed, but with the ﬁrst two downsampled residual blocks removed.
Revised: Similarly, the quarter resolution models share the architectures listed, but with the ﬁrst two downsampled residual blocks removed.

Original: We utilize group normalization (Wu & He, 2018) inside each residual block and utilize the Swish nonlinearity (Ramachandran et al., 2018).
Revised: We utilize group normalization (Wu & He, 2018) inside each residual block and utilize the Swish nonlinearity (Ramachandran et al., 2018).

Original: C. O VERALL E XPERIMENTAL S ETUP
Revised: C. O VERALL E XPERIMENTAL S ETUP

Original: Models are trained using the Adam Optimizer (Kingma & Ba, 2015), on a single 32GB Volta GPU for CIFAR-10 for 1 day, and for 3 days on 8 32GB Volta GPUs for CelebaHQ, LSUN and ImageNet 32x32 datasets.
Revised: Models are trained using the Adam Optimizer (Kingma & Ba, 2015), on a single 32GB Volta GPU for CIFAR-10 for 1 day, and for 3 days on 8 32GB Volta GPUs for CelebaHQ, LSUN and ImageNet 32x32 datasets.

Original: We provide detailed training conﬁguration details in the appendix.
Revised: We provide detailed training conﬁguration details in the appendix.

Original: Our proposed improvements are built on top of the EBMs training framework proposed in (Du &
Revised: Our proposed improvements are built on top of the EBMs training framework proposed in (Du &

Original: Mordatch, 2019).
Revised: Mordatch, 2019).

Original: We use a buffer size of 10000, with a resampling rate of 0.1% with L2 regularization applied on output energies.
Revised: We use a buffer size of 10000, with a resampling rate of 0.1% with L2 regularization applied on output energies.

Original: Our approach is signiﬁcantly more stable than IGEBM, allowing us to remove aspects of regularization in (Du & Mordatch, 2019).
Revised: Our approach is signiﬁcantly more stable than IGEBM, allowing us to remove aspects of regularization in (Du & Mordatch, 2019).

Original: We remove the clipping of gradients in
Revised: We remove the clipping of gradients in

Original: Langevin sampling as well as spectral normalization on the weights of the network.
Revised: Langevin sampling as well as spectral normalization on the weights of the network.

Original: In addition, we add self-attention blocks and layer normalization blocks in residual networks of our trained models.
Revised: In addition, we add self-attention blocks and layer normalization blocks in residual networks of our trained models.

Original: In multi-scale architectures, we utilize 3 different resolutions of an image, the original image resolution, half the image resolution and a quarter the image resolution.
Revised: In multi-scale architectures, we utilize 3 different resolutions of an image, the original image resolution, half the image resolution and a quarter the image resolution.

Original: We report detailed architectures in the appendix.
Revised: We report detailed architectures in the appendix.

Original: When evaluating models, we utilize the EMA model with EMA weight of 0.9999.
Revised: When evaluating models, we utilize the EMA model with EMA weight of 0.9999.

Original: C. E XPERIMENT C ONFIGURATIONS F OR D IFFERENT D ATASETS
Revised: C. E XPERIMENT C ONFIGURATIONS F OR D IFFERENT D ATASETS

Original: CIFAR-10/ImageNet 32x
Revised: CIFAR-10/ImageNet 32x

Original: For CIFAR-10 and ImageNet 32x32, we use 40 steps of Langevin sampling to generate a negative sample.
Revised: For CIFAR-10 and ImageNet 32x32, we use 40 steps of Langevin sampling to generate a negative sample.

Original: The Langevin sampling step size is set to be 500, with
Revised: The Langevin sampling step size is set to be 500, with

Original: Gaussian noise of magnitude 0.001 at each iteration.
Revised: Gaussian noise of magnitude 0.001 at each iteration.

Original: The data augmentation transform consists of color augmentation of strength 1.0 from (Chen et al., 2020), a random horizontal ﬂip, and a image resize between 0.02 and 1.0.
Revised: The data augmentation transform consists of color augmentation of strength 1.0 from (Chen et al., 2020), a random horizontal ﬂip, and a image resize between 0.02 and 1.0.

Original: This is used in the main paper Section 3.2 and 3.3.
Revised: This is used in the main paper Section 3.2 and 3.3.

Original: CelebA/LSUN Bedroom.
Revised: CelebA/LSUN Bedroom.

Original: For the CelebA-HQ and LSUN bed datasets, we use 40 steps of
Revised: For the CelebA-HQ and LSUN bed datasets, we use 40 steps of

Original: Langevin sampling to generate negative samples.
Revised: Langevin sampling to generate negative samples.

Original: The Langevin sampling step size is set to be 1000, with Gaussian noise of magnitude 0.001 applied at each iteration.
Revised: The Langevin sampling step size is set to be 1000, with Gaussian noise of magnitude 0.001 applied at each iteration.

Original: The data augmentation transform consists of color augmentation of strength 1.0 from (Chen et al., 2020), a random horizontal ﬂip, and a image resize between 0.08 and 1.0.
Revised: The data augmentation transform consists of color augmentation of strength 1.0 from (Chen et al., 2020), a random horizontal ﬂip, and a image resize between 0.08 and 1.0.

Original: This is used in the main paper Section 3.2 and 3.4.
Revised: This is used in the main paper Section 3.2 and 3.4.

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: D A DDITIONAL A NALYSIS D. A NALYSIS OF T RUNCATED L ANGEVIN B ACKPROPAGATION
Revised: D A DDITIONAL A NALYSIS D. A NALYSIS OF T RUNCATED L ANGEVIN B ACKPROPAGATION

Original: To better understand the training effect of L KL , we analyze the effect of truncating backpropogation through Langevin sampling.
Revised: To better understand the training effect of L KL , we analyze the effect of truncating backpropogation through Langevin sampling.

Original: We train two separate models on MNIST, one with backpropogation through all Langevin steps, and one with backpropogation through only the last Langevin step.
Revised: We train two separate models on MNIST, one with backpropogation through all Langevin steps, and one with backpropogation through only the last Langevin step.

Original: We obtain an FID of 90.54 with backpropogation through only 1 step of Langevin sampling and an FID of 94.85 with backpropogation through all steps of Langevin sampling.
Revised: We obtain an FID of 90.54 with backpropogation through only 1 step of Langevin sampling and an FID of 94.85 with backpropogation through all steps of Langevin sampling.

Original: We present illustrations of samples generated with one step in Figure 18 and with all steps in Figure 19.
Revised: We present illustrations of samples generated with one step in Figure 18 and with all steps in Figure 19.

Original: Overall, we ﬁnd little degradation in performance with the truncation of backpropogation, but note that backpropogation through all steps of sampling is over 3 times slower to train.
Revised: Overall, we ﬁnd little degradation in performance with the truncation of backpropogation, but note that backpropogation through all steps of sampling is over 3 times slower to train.

Original: D. A NALYSIS OF E FFECT OF KL L OSS ON M ODE S AMPLING
Revised: D. A NALYSIS OF E FFECT OF KL L OSS ON M ODE S AMPLING

Original: We illustrate the effect of L KL as a regularizer to prevent EBM sampling collapse.
Revised: We illustrate the effect of L KL as a regularizer to prevent EBM sampling collapse.

Original: When training an
Revised: When training an

Original: EBM, L KL serves as a repelling term encouraging MCMC samples from an EBM to both have low energy and exhibit diversity.
Revised: EBM, L KL serves as a repelling term encouraging MCMC samples from an EBM to both have low energy and exhibit diversity.

Original: In the absence of L KL , we ﬁnd that EBM sampling always collapses and eventually always generates samples illustrated in Figure 20.
Revised: In the absence of L KL , we ﬁnd that EBM sampling always collapses and eventually always generates samples illustrated in Figure 20.

Original: These samples are signiﬁcantly less diverse than those generated when training with L KL ( Figure 4), which never suffers from sampling collapse.
Revised: These samples are signiﬁcantly less diverse than those generated when training with L KL ( Figure 4), which never suffers from sampling collapse.

Original: D. E FFECT OF D ATA A UGMENTATION
Revised: D. E FFECT OF D ATA A UGMENTATION

Original: We evaluate the effect of data augmentation on sampling in EBMs.
Revised: We evaluate the effect of data augmentation on sampling in EBMs.

Original: In Figure 6 we show that by combining Langevin sampling with data augmentation transitions, we are able to enable chains to mix across different images, whereas prior works have shown Langevin converging to ﬁxed images.
Revised: In Figure 6 we show that by combining Langevin sampling with data augmentation transitions, we are able to enable chains to mix across different images, whereas prior works have shown Langevin converging to ﬁxed images.

Original: In Figure 12 we show that given a ﬁxed random noise initialization, data augmentation transitions enable to reach a diverse number of different samples, while sampling without data augmentation transitions leads all chains to converge to the same face.
Revised: In Figure 12 we show that given a ﬁxed random noise initialization, data augmentation transitions enable to reach a diverse number of different samples, while sampling without data augmentation transitions leads all chains to converge to the same face.

Original: D. KL G RADIENT
Revised: D. KL G RADIENT

Original: We plot the overall gradient magnitudes of L CD and L KL when training an EBM on CIFAR-10 in
Revised: We plot the overall gradient magnitudes of L CD and L KL when training an EBM on CIFAR-10 in

Original: Figure 13.
Revised: Figure 13.

Original: We ﬁnd that relative magnitude of gradients of both training objectives remains constant across training, and that the gradient of the KL objective is non-negligible.
Revised: We ﬁnd that relative magnitude of gradients of both training objectives remains constant across training, and that the gradient of the KL objective is non-negligible.

Original: [Table] [Table]
Revised: [Table] [Table]