L EARNING E NERGY -B ASED M ODELS IN H IGHD IMENSIONAL S PACES WITH M ULTI - SCALE DENOISING SCORE MATCHING
Anonymous authors
Paper under double-blind review
A BSTRACT
Energy-Based Models (EBMs) output unnormalized log-probability values given data samples.
Such an estimation is essential in a variety of applications such as sample generation, denoising, sample restoration, outlier detection, Bayesian reasoning, and many more.
However, standard maximum likelihood training is computationally expensive due to the requirement of sampling the model distribution.
Score matching potentially alleviates this problem, and denoising score matching (Vincent, 2011) is a particularly convenient version.
However, previous works do not produce models capable of high quality sample synthesis in high dimensional datasets from random initialization.
We believe that is because the score is only matched over a single noise scale, which corresponds to a small set in high-dimensional space.
To overcome this limitation, here we instead learn an energy function using denoising score matching over all noise scales.
When sampled from random initialization using Annealed Langevin Dynamics and single step denoising jump, our model produced high-quality samples comparable to state-of-the-art techniques such as GANs.
The learned model also provide density information and set a new sample quality baseline in energy-based models.
We further demonstrate that the proposed method generalizes well with an image inpainting task.
I NTRODUCTION AND M OTIVATION
Treating data as stochastic samples from a probability distribution and developing models that can learn such distributions is at the core for solving a large variety of application problems, such as error correction/denoising (Vincent et al., 2010), outlier/novelty detection (Zhai et al., 2016; Choi and Jang, 2018), invariant pattern recognition, Bayesian reasoning (Welling and Teh, 2011) which relies on good data priors, and many others.
Energy-Based Models (EBMs) (LeCun et al., 2006; Ngiam et al., 2011) assign an energy E ( xxx ) to each data point xxx which implicitly deﬁnes a probability by the Boltzmann distribution p m ( xxx ) =
e − E
( xxx ) /Z .
Sampling from this distribution can be used as a generative process that yield plausible samples of xxx .
Compared to other generative models, like GANs (Goodfellow et al., 2014), ﬂowbased models (Dinh et al., 2015; Kingma and Dhariwal, 2018), or auto-regressive models (van den Oord et al., 2016; Ostrovski et al., 2018), energy-based models have signiﬁcant advantages.
First, they provide explicit (unnormalized) density information, compositionality (Hinton, 1999; Haarnoja et al., 2017), better mode coverage (Kumar et al., 2019) and ﬂexibility (Du and Mordatch, 2019).
Secondly, they do not require special model architecture, unlike auto-regressive and ﬂow-based models.
Recently, Energy-based models has been successfully trained with maximum likelihood (Nijkamp et al., 2019; Du and Mordatch, 2019), but training can be very computationally demanding due to the need of sampling model distribution.
Variants with a truncated sampling procedure have been proposed, such as contrastive divergence (Hinton, 2002).
Such models learn much faster with the draw back of not exploring the state space thoroughly (Tieleman, 2008).
1. S CORE M ATCHING , D ENOISING S CORE M ATCHING AND D EEP E NERGY E STIMATOR
Score matching (SM) (Hyv¨arinen, 2005) circumvents the requirement of sampling the model distribution.
In score matching, the score function is deﬁned to be the gradient of log-density or the negative energy function.
The expected L 2 norm of difference between the model score function and the data score function are minimized.
One convenient way of using score matching is learning the energy function corresponding to a Gaussian kernel Parzen density estimator (Parzen, 1962) of the data: p σ 0 (˜ xxx )
= (cid:82) q σ
(˜ xxx | xxx ) p ( xxx ) dx .
Though hard to evaluate, the data score is well deﬁned: s d (˜ xxx )
= ∇ ˜
xxx log( p σ
(˜ xxx )) .
And the corresponding objective is:
[Equation]
Vincent (2011) studied the connection between denoising auto-encoder and score matching, and proved a remarkable result that the following objective, named Denoising Score Matching (DSM), is equivalent to the objective above:
[Equation]
Note that the Parzen density score is replaced by the derivative of log density of the single noise kernel ∇ ˜ xxx log( q σ 0 (˜ xxx | xxx )) , which is much easier to evaluate.
In the particular case of Gaussian noise log( q σ 0 (˜ xxx | xxx )) = − ( ˜ x ˜ x ˜ x − xxx i ) 2 2 σ+ C and:
[Equation]
This objective (3) forces the energy gradient to align with the vector pointing from the noisy sample to the clean data sample, therefore it is named Denoising score matching .
To optimize such an objective involving derivative of a function deﬁned by a neural network, double backpropagation (Drucker and Le Cun, 1991) is needed, e.g. in (Kingma and LeCun, 2010). Deep energy estimator networks (Saremi et al., 2018)
ﬁrst applied the same technique to learn an energy function deﬁned by a deep neural network.
While learning the Parzen kernel density estimator under small noise was only proposed to analyze denoising autoencoder, Saremi et al. (2018) learned an energy-based model to match a Parzen estimator with a particular noise magnitude. Later, Song and Ermon (2019) learned a series of score estimators, each matching the score of a Parzen estimator with a different noise magnitude. In this work, we combine some of the key ideas from Saremi et al.
; Saremi and Hyvarinen (2019); Song and Ermon (2019) and aim to learn an energy-based model using denoising score matching with multiple noise magnitudes. In Section 2, we ﬁrst provide a geometric view to illustrate the training process of denoising score matching in high dimensional space, analyze its potential issues, and to motivate our own approach.
Then we formulate Multiscale Denoising Score Matching (MDSM), which is followed by a series of empirical results.
A G EOMETRIC VIEW
In Saremi and Hyvarinen (2019), author arrived at Equation 2 using a different theory based on Empirical Bayes estimator in the case of Gaussian noise. In the same paper the behavior of noise corrupted distribution was analyzed in high-dimensional case using measure concentration results. Here we develop upon this picture to further understand denoise score matching learning process.
We adopt the common assumption that the data distribution to be learned is high-dimensional, but only has support around a relatively low-dimensional manifold (Tenenbaum et al., 2000; Roweis and Saul, 2000; Lawrence, 2005).
This causes a problem that density, or the gradient of the density for data distribution is not deﬁned outside the manifold, making it difﬁcult to train a valid density model for the data distribution deﬁned on the entire space. In Saremi and Hyvarinen (2019) and Song and Ermon (2019), authors analyzed this problem and motivated smoothing the data distribution with Gaussian kernel to alleviate this issue.
To understand the learning process in relation to the data manifold X , we recall two elementary properties of random Gaussian vectors in high-dimensional spaces: First, the length distribution of A. Training B. Sampling [Figure] random vectors becomes concentrated at √ dσ (Vershynin, 2018), where σ 2 is the variance of a single dimension.
Second, a random vector is always close to orthogonal to a ﬁxed vector (Tao, 2012).
With these premises one can visualize the conﬁguration of noisy and noiseless data points used in the learning of denoising score matching: A data point xxx sampled from X and its noisy version ˜ xxx always lie on a line which is almost perpendicular to the tangent space T xxx X and intersects X at xxx .
Further, the distance vectors between ( xxx, ˜ xxx ) pairs all have similar length √ dσ .
As a consequence, the set of noisy data points concentrate on a set ˜ X √ dσ,(cid:15) that has a distance with ( √ dσ − (cid:15), √ dσ + (cid:15) ) from the data manifold X , where (cid:15) (cid:28) √ dσ .
All told, performing denoising score matching learning with ( xxx, ˜ xxx ) pairs generated with a ﬁxed noise level σ , which is the approach taken previously except in Song and Ermon (2019), will match the score on the set ˜ X √ dσ,(cid:15) and enable denoising of noisy points in ˜ X √ dσ,(cid:15) .
However, the learning provides little information about the density outside this set, farther or closer to the data manifold, as noisy samples outside ˜ X √ dσ,(cid:15) rarely appear in the training process. An illustration is presented in
Figure 1A. Denoting the complement of the set ˜ X √ dσ,(cid:15) as ˜ X C √ dσ,(cid:15) , though p σ 0 (˜ xxx ∈ ˜ X C √ dσ,(cid:15) ) is very small in high-dimensional space, the score in ˜ X C √ dσ,(cid:15) still plays a critical role in sampling from random noise.
This may explain the difﬁculty previous models based on denoising score matching with single noise scale encountered when sampled with noise initialization.
We demonstrate this experiment in Appendix B with multiple single noise magnitudes. This phenomenon in high-dimensional space is not sufﬁciently appreciated by the classical score matching objective, where the expectation almost conﬁnes score estimation to ˜ X √ dσ,(cid:15) . To remedy this problem, we propose modifying the expectation in score matching objective to a distribution consist of mixture of Gaussian noise scales which will sample the ambient space more evenly, avoiding issue caused by measure concentration, see Section 3.1 and Appendix A for more detailed discussions. Song and Ermon (2019) also used multiple noise scales in denoise score-matching, but they did not motivate this choice in the context of high-dimensional geometry, and a low dimensional picture would suggest that a single noise scale large enough to cover entire space would be sufﬁcient.
Another property of denoising score matching was observed in the denoising auto-encoder literature (Vincent et al., 2010; Karklin and Simoncelli, 2011) that with higher noise level the learned representation tend to have larger spatial scale.
An plausible explanation is that a lot of natural signals have multi-scale structure with 1 /f power characteristic, meaning that features at low spatial frequencies have larger signal power than features at high spacial frequencies (Field, 1987).
Thus, a higher noise levels will wash away features above certain spacial frequencies, but insufﬁciently corrupting features below certain spacial frequencies to drive learning.
Thus, a model trained with only one noise level will only learn structures roughly of a single scale.
To illustrate this point, in Figure 1B we show some examples of intermediate samples during annealed sample carrying different levels of noise. One can see that as the sample point approach the data manifold, more and more details are present in the denoised sample.
We provide more result from this and similar experiment in Figure 3 and Figure E.1.
The analysis above suggest that it is not sufﬁcient to learn energy-based model with denoising score matching using a single noise scale, at least if high-quality sample synthesis is part of the goal for the learning.
To achieve this while maintaining the speed advantage of score matching, we propose multiscale denoising score matching (MDSM), a denoising score matching procedure using multiple noise levels.
We expect this sampling-free learning method to shape an energy function that can drive samples anywhere in the state space towards high probability regions of the data. For sampling from the learned model, we propose annealed Langevin dynamics and additionally use single step denoise jump. We show that our model generates high-quality samples similar in quality to the-state-of-the-art image generation techniques, such as GANs (Goodfellow et al., 2014), as well as providing density information.
L EARNING E NERGY - BASED MODEL WITH M ULTISCALE D ENOISING SCORE MATCHING 3. M ULTISCALE D ENOISING S CORE MATCHING
Motivated by the geometric view, we strive to learn an energy function corresponding to the Parzen density estimator of the data p σ 0 (˜ xxx )
= (cid:82) q σ {+
+}
(˜ xxx | xxx ) p ( xxx ) dx by minimizing the difference between derivative of the energy function with the score of p σ 0 under the expectation E p σ (cid:48)
(˜ xxx ) {+rather than E p σ 
+}
(˜ xxx ) , where p σ (cid:48)
(˜ xxx )
= (cid:82) q σ (cid:48)
(˜ xxx | xxx ) p ( xxx ) dx covers the signal space better than p σ 0 (˜ xxx ) . This new expectation avoids the measure concentration issue and leads to the following Multiscale Score
Matching (MSM) objective:
[Equation]
Note that the only thing has changed in Equation 4 is the expectation compared to Equation 1. In the asymptotic case, they are consistent if p σ (cid:48) (˜ xxx ) and p σ 0 (˜ xxx ) have the same support.
We show this formally in Proposition 1 of Appendix A.
In Proposition 2, we prove that Equation 4 is equivalent to the following denoising score matching objective: [Equation]
The above results apply to any noise kernel q σ (cid:48)
(˜ xxx | xxx ) , but Equation 5 has a reversed expectation form that is not easy to work with.
To proceed, we consider the case q σ 0 (˜ xxx | xxx ) is Gaussian in which we can choose the distribution q σ (cid:48) (˜ xxx | xxx ) to be a Gaussian scale mixture: q σ (cid:48) (˜ xxx | xxx ) = (cid:82) q σ (˜ xxx | xxx ) p ( σ ) dσ and q σ (˜ xxx | xxx )
=
N ( xxx, σ 2 I d ) . After some manipulation and approximation (See analysis following Proposition 2 in Appendix A), we can change Equation 5 into a more convenient form, which we call Multiscale Denoising Score Matching (MDSM):
[Equation]
The the square loss term evaluated at noisy points ˜ xxx at larger distances from true data points xxx will have larger magnitude. Therefore, in practice we add an extra monotonic decreasing term l ( σ ) to balance different noise scales, e.g. l ( σ ) = 1 σ 2 .
Ideally we want our model to learn the correct gradient everywhere so we need to add noise of all levels, but applying denoising score matching to very large or very small noise is rather pointless.
For very large noise the original sample’s information is completely lost.
Conversely, in the limit of small noise, the noisy sample is virtually indistinguishable from real data.
In neither case can one expect to learn an informative gradient.
Thus, the noise range needs only be chosen broad enough to encourage learning of data features over all scales.
Particularly, we do not sample σ but instead choose a series of ﬁxed σ values in a range.
We also substitute log( q [Equation] + C into Equation 6 and arrive the the ﬁnal objective: [Equation]
On the surface, this objective looks similar to the one in Song and Ermon (2019), but the important difference is that we match the score to a single distribution, while Song and Ermon (2019) learns the score of multiple distributions.
Our model also does not require noise magnitude as input since there’s only one target distribution p σ 0 (˜ xxx ) to be learned.
3. S AMPLING BY A NNEALED L ANGEVIN D YNAMICS AND J UMP
In Saremi and Hyvarinen (2019), the energy function corresponding to Parzen density estimator of a single noise scale is sampled by Langevin dynamics, and clean sample can be obtained using single step denoise jump.
We extend this approach to sample from our learned energy function with annealed Langevin dynamics, as our energy function is trained over all noise levels.
Additionally we use the same single step denoise jump.
Simulated annealing (Kirkpatrick et al., 1983; Neal, 2001) has been successfully applied to challenging computational problems, such as combinatorial optimization.
In essence, the mode exploration in an objective function can be improved by sampling ﬁrst at high temperature and then cooling down gradually.
Langevin dynamics has been applied to sampling from neural network energy functions (Du and Mordatch, 2019; Nijkamp et al., 2019).
However, they reported difﬁculty with mode exploration or the need for very large number of sampling steps.
To improve mode exploration even with small numbers of sampling steps, we propose Annealed Langevin dynamics.
In this sampling process the temperature parameter in the Langevin dynamics is adjusted from high to low according to an annealing schedule.
In the model of a physical particle undergoing Brownian motion, temperature in its Langevin equation enters as a √ T factor in front of the noise term, some literature use β − 1 where β = 1 /T
(Jordan et al., 1998).
Adopting the √ T convention yields the following sampling process:
[Equation]
where T t follows some annealing schedule, and (cid:15) denotes step length, which is ﬁxed.
In the NCSN model (Song and Ermon, 2019), author also proposed annealed Langevin dynamics, which is best interpreted as sequentially sampling a series of different distributions.
Our formulation differs as our algorithm samples the same energy function at different temperatures, and correspond exactly to the form of damped Langevin dynamics in physics and statistics literature.
During sampling, samples behave very much like physical particles under Brownian motion in a potential ﬁeld.
Because the particles have average energies close to the their current thermic energy, they explore the state space at different distances from data manifold depending on temperature.
Eventually, they settle somewhere on the data manifold.
The behavior of the particle’s energy value during a typical annealing process is depicted in Appendix Figure F.1B.
If the obtained sample is still noisy, we can make a single step gradient denoise jump to improve sample quality:
[Equation]
[Equation] This denoising procedure can be applied to noisy sample with any level of Gaussian noise because in our model the gradient automatically has the right magnitude to denoise the sample.
This process is also justiﬁed by the Empirical Bayes interpretation of this denoising process, as studied in Saremi and Hyvarinen (2019).
I MAGE MODELING R ESULTS
Training and Sampling Details.
The proposed energy-based model is trained on standard image datasets, namely MNIST, Fashion MNIST, CelebA
(Liu et al., 2015) and CIFAR-10 (Krizhevsky et al., 2009).
During training we set σ 0 = 0 .
1 and train over a noise range of σ ∈ [0 . 05 , 1 . 2] , with the different noise uniformly spaced on the batch dimension, for MNIST and Fashion MNIST we used geometrically distributed noise in the range [0 . 1 , 3] .
l ( σ ) is set to 1 / ( σ 2 ) at all times to make the square term roughly independent of σ .
We ﬁx batch size at 128 and use the Adam optimizer with a learning rate of 5 × 10 − 5 .
For MNIST and Fashion MNIST, we use a 12-Layer ResNet withﬁlters, for the CelebA and CIFAT-10 data sets we used a 18-Layer ResNet with 128 ﬁlters (He et al., 2016; Zagoruyko and Komodakis, 2016).
No normalization layer was used in any of the networks.
We designed the output layer of all networks to take a generalized quadratic form (Fan et al., 2018), because the energy function is anticipated to be approximately quadratic with respect to the noise level, we found this modiﬁcation to boost the performance signiﬁcantly.
For more detail on training and model architecture, see Appendix D. One notable result is that since our training method does not involve sampling, we achieved a speed up of roughly an order of magnitude compared to the common maximum-likelihood training 1 .
Our method thus enables the training of energy-based models even when limited computational resources prohibit maximum likelihood methods.
We found that the choice of the maximum noise level has little effect on learning as long as it is large enough to encourage learning of the longest range features in the data.
However, as expected, learning with too small or too large noise levels is not beneﬁcial and can even destabilize the training process.
Further, our method appeared to be relatively insensitive to how the noise levels are distributed over a chosen range.
Geometrically spaced noise as in (Song and Ermon, 2019) and linearly spaced noise both work, although in our case learning with linearly spaced noise was somewhat more robust.
For sampling the learned energy function we used annealed Langevin dynamics with an empirically optimized annealing schedule,see Figure F.1 B for the particular shape of annealing schedule we used.
In contrast, annealing schedules with theoretical guaranteed convergence property takes extremely long (Geman and Geman, 1984).
Range of temperatures to use in the sampling process depends on ones choice of σ 0 , as the equilibrium distribution is roughly images with Gaussian noise of magnitude √ Tσ 0 added on top.
To ease traveling between modes far apart and ensure even sampling, the initial temperature need to be high enough to inject noise of sufﬁcient magnitude.
In our case T = 100 , which correspond to added noise of magnitude √ 100 ∗ 0 .
1 = 1 seems to be sufﬁcient starting point.
For step length (cid:15) we generally used 0 .
02 , although any value within the range [0 . 015 , 0 . 05] seemed to work ﬁne.
After annealing process we perform a single step denoising jump to further enhance sample quality.
Unconditional Image Generation.
We demonstrate the generative ability of our model by displaying samples obtained by annealed Langevin sampling and single step denoise jump.
We evaluated 50k sampled images after training on CIFAR-10 with two performance scores, Inception (Salimans et al., 2016) and FID (Heusel et al., 2017).
We achieved Inception Score of 8.31 and FID of 31.7, comparable to modern GAN approaches.
Scores for CelebA dataset are not reported here as they are not commonly reported and may depend on the speciﬁc pre-processing used.
More samples and training images are provided in Appendix for visual inspection.
We believe that visual assessment is still essential because of issues with the Inception score (Barratt and Sharma, 2018).
Indeed, we also found that the visually impressive samples was not the one achieving the highest Inception Score. [Table] [Table]
[Table] Although overﬁtting is not a common concern for generative models [Table] we still tested our model for overﬁtting.
We found no indication for overﬁtting by comparing model samples with their nearest neighbors in the data set, see Figure C.1 in Appendix.
Mode Coverage.
We repeated with our model the 3 channel MNIST mode coverage experiment similar to the one in Kumar et al.
An energy-based model was trained on 3-channel data where each channel is a random MNIST digit.
Then 8000 samples were taken from the model and each channel is classiﬁed using a small MNIST classiﬁer network.
We obtained results of the 966 modes, comparable to GAN approaches.
Training was successful and our model assigned low energy to all the learned modes, but some modes were not accessed during sampling, likely due to the Langevin Dynamics failing to explore these modes.
A better sampling technique such as HMC Neal et al.
(2011) or a Maximum Entropy Generator (Kumar et al., 2019) could improve this result.
Image Inpainting.
Image impainting can be achieved with our model by clamping part of the image to ground truth and performing the same annealed Langevin and Jump sampling procedure on the missing part of the image.
Noise appropriate to the sampling temperature need to be added to the clamped inputs.
The quality of inpainting results of our model trained on CelebA and CIFARcan be assessed in Figure 3.
For CIFAR inpainting we used test set,
Log likelihood estimation.
For energy-based models the log density can be obtained after estimating the partition function with Annealed Importance Sampling (AIS)
(Salakhutdinov and Murray, 2008) or Reverse AIS (Burda et al., 2015).
In our experiment on CIFAR-10 model, similar to report in Du and Mordatch (2019), there’s still a substantial gap between AIS and Reverse AIS estimation after very substantial computational effort.
In Table 1, We report result from Reverse AIS as it tends to over-estimate the partition function thus underestimate the density.
Note that although density values and likelihood values are not directly comparable, we list them together due to the sheer lack of density model on CIFAR-10. We also report density of 1.21 bits/dim on MNIST dataset, and we refer readers to Du and Mordatch (2019) for comparison to other models on this dataset.
More details on this experiment is provided in the Appendix.
Outlier Detection.
Choi and Jang (2018) and Nalisnick et al.
(2019) have reported intriguing behavior of high dimensional density models on out of distribution samples.
Speciﬁcally, they showed that a lot of models assign higher likelihood to out of distribution samples than real data samples.
We investigated whether our model behaves similarly.
Our energy function is only trained outside the data manifold where samples are noisy, so the energy value at clean data points may not always be well behaved.
Therefore, we added noise with magnitude σ 0 before measuring the energy value.
We report that our network behaves similarly to previous likelihood models that it assigns lower energy, thus higher density, to some OOD samples.
We show one example of this phenomenon in Appendix Figure F.1A.
We also attempted to use the denoise performance, or the objective function to perform outlier detection. Intriguingly, it also behaves similarly as using energy value, as denoise performance seems to correlate more with the variance of original image rather than the content of the image.  D ISCUSSION Our approach is conceptually similar to the idea of combining denoising autoencoder and annealing (Geras and Sutton, 2015; Chandra and Sharma, 2014; Zhang and Zhang, 2018) though they were proposed in the context of pre-training neural networks for classiﬁcation applications.
Previous efforts of learning energy-based models with score matching (Kingma and LeCun, 2010; Song et al., 2019) were either computationally intensive or unable to produce high-quality samples comparable to those obtained by other generative models such as GANs.
Saremi et al.
and Saremi and Hyvarinen (2019) trained energy-based model with denoising score matching objective but the resulting models cannot perform sample synthesis from random noise initializations. We provide analysis and empirical results both suggesting that the limitation of previous denoising score matching models is due to the objective doesn’t sufﬁciently appreciate low-density regions.
To remedy this issue we proposed a learning procedure using a wide range of different noise scales, which we named Multiscale Denoising Score Matching .
Models learned with MDSM is capable of denoising, producing high-quality samples from random noise, and performing image inpainting. They achieve those while providing density information, being an order of magnitude faster over maximum likelihood method during training, and admitting a more efﬁcient annealed Langevin sampling procedure.
Our approach is perhaps most closely related to the NCSN model (Song and Ermon, 2019), a network that is also trained by denoising score matching with Gaussian noise corruption of different scales.
This model essentially learns p σ (˜ xxx ) for a series of σ values.
Our method instead learn the energy function correspond to {+p σ 
+}
(˜ xxx ) for one speciﬁc σ 0 value.
Our model provide density information and is able to perform single step denoising because it doesn’t require noise magnitude input like the NCSN model. In conclusion, our model provides an unique combination of density information with high-quality sample generation.
A MDSM O BJECTIVE
In this section, we provide a formal discussion of the MDSM objective and suggest it as an improved score matching formulation in high-dimensional space.
Vincent (2011) illustrated the connection between the model score −∇
˜ xxx E (˜ xxx ; θ ) with the {+score of Parzen window density estimator ∇ ˜ xxx log( p σ 
+}
(˜ xxx )) . Speciﬁcally, the objective is Equation 1 which we restate here:
[Equation]
Our key observation is: in high-dimensional space, due the concentration of measure, the expectation w.r.t.
p σ 0 (˜ xxx ) over weighs a thin shell at roughly distance √ dσ to the empirical distribution p ( x ) .
Though in theory this is not a problem, in practice this leads to results that the score are only well matched on this shell.
Based on this observation, we suggest to replace the expectation w.r.t.
{+p σ 0 (˜ xxx ) with a distribution p σ (cid:48) (˜ xxx ) that has the same support as p σ 
+}
(˜ xxx ) but can avoid the measure concentration problem.
We call this multiscale score matching and the objective is the following:
[Equation]
Proposition 1.
{+L MSM ( θ ) = 
+}
{+⇐⇒ L SM ( θ ) = 
+}
⇐⇒ θ
= θ ∗ .
Given that p σ (cid:48) (˜ xxx ) and p σ 0 (˜ xxx ) has the same support, it’s clear that L MSM = 0 would be equivalent to
L SM = 0 .
Due to the proof of the Theorem 2 in Hyv ¨ arinen (2005), we have L SM ( θ ) ⇐⇒ θ
= θ ∗ .
{+Thus, L MSM ( θ ) = 
+}
⇐⇒ θ
= θ ∗ .
Proposition 2.
L
[Equation]
We follow the same procedure as in Vincent (2011) to prove this result
[Equation]
[Table]
Thus we have:
[Equation]
So [Equation] The above analysis applies to any noise distribution, not limited to Gaussian. but L MDSM ∗ has a reversed expectation form that is not easy to work with.
To proceed further we study the case where q σ 0 (˜ xxx | xxx ) is Gaussian and choose q σ (cid:48) (˜ xxx | xxx ) as a Gaussian scale mixture (Wainwright and Simoncelli, 2000) and p σ (cid:48) (˜ xxx ) = (cid:82) q σ (cid:48) (˜ xxx | xxx ) p ( xxx ) dx .
By Proposition 1 and Proposition 2, we have the following form to optimize: [Equation] To minimize Equation (*)
[Equation] we can use the following importance sampling procedure (Russell and Norvig, 2016): we can sample from the empirical distribution p ( xxx ) , then sample the Gaussian scale mixture q σ (cid:48) (˜ xxx | xxx ) and ﬁnally weight the sample by q σ 0 ( xxx | ˜ xxx )
q σ (cid:48) ( xxx | ˜ xxx ) . We expect the ratio to be close to 1 for the following reasons: Using Bayes rule, q σ 0 ( xxx | [Equation] we can see that q σ 0 ( xxx | ˜ xxx ) only has support on discret data points xxx , same thing holds for q σ (cid:48)
( xxx | ˜ xxx ) .
because in ˜ xxx is generated by adding
Gaussian noise to real data sample, both estimators should give results highly concentrated on the original sample point xxx . Therefore, in practice, we ignore the weighting factor and use Equation ?? . Improving upon this approximation is left for future work. B P ROBLEM WITH SINGLE NOISE DENOISING SCORE MATCHING
To compare with previous method, we trained energy-based model with denoising score matching using one noise level on MNIST, initialized the sampling with Gaussian noise of the same level, and sampled with Langevin dynamics at T = 1 for 1000 steps and perform one denoise jump to recover the model’s best estimate of the clean sample, see Figure B.1.
We used the same 12-layer ResNet as other MNIST experiments.
Models were trained for 100000 steps before sampling.
C O VERFITTING TEST
We demonstrate that the model does not simply memorize training examples by comparing model samples with their nearest neighbors in the training set.
We use Fashion MNIST for this demonstration because overﬁtting can occur there easier than on more complicated datasets, see Figure C.1.
D D ETAILS ON T RAINING AND S AMPLING
We used a custom designed ResNet architecture for all experiments.
For MNIST and FashionMNIST we used a 12-layer ResNet with 64 ﬁlters on ﬁrst layer, while for CelebA and CIFAR dataset we used a 18-layer ResNet with 128 ﬁlters on the ﬁrst layer.
All network used the ELU activation function.
We did not use any normalization in the ResBlocks and the ﬁler number is doubled at each downsampling block.
Details about the structure of our networks used can be found in our code release Here.
All mentioned models can be trained on 2 GPUs within 2 days.
Since the gradient of our energy model scales linearly with the noise, we expected our energy function to scale quadratically with noise magnitude.
Therefore, we modiﬁed the standard energy-based network output layer to take a ﬂexible quadratic form (Fan et al., 2018):
[Equation]
where a i , c i , d i and b 1 , b 2 , b 3 are learnable parameters, and h i is the (ﬂattened) output of last residual block.
We found this modiﬁcation to signiﬁcantly improve performance compared to using a simple linear last layer.
For CIFAR and CelebA results we trained for 300k weight updates, saving a checkpoint everyupdates.
We then took 1000 samples from each saved networks and used the network with the lowest FID score.
For MNIST and fashion MNIST we simply trained for 100k updates and used the last checkpoint.
During training we pad MNIST and Fashion MNIST to 32*32 for convenience and randomly ﬂipped CelebA images.
No other modiﬁcation was performed.
We only constrained the gradient of the energy function, the energy value itself could in principle be unbounded.
However, we observed that they naturally stabilize so we did not explicitly regularize them.
The annealing sampling schedule is optimized to improve sample quality for CIFAR-10 dataset, and consist of a total of 2700 steps.
For other datasets the shape has less effect on sample quality, see Figure F.1 B for the shape of annealing schedule used.
For the Log likelihood estimation we initialized reverse chain on test images, then sampleintermediate distribution using 10 steps HMC updates each. Temperature schedule is roughly exponential shaped and the reference distribution is an isotropic Gaussian.
The variance of estimation was generally less than 10% on the log scale.
Due to the high variance of results, and to avoid getting dominated by a single outlier, we report average of the log density instead of log of average density.
E E XTENDED S AMPLES AND I NPAINTING RESULTS
We provide more inpainting examples and further demonstrate the mixing during sampling process in Figure E.1.
We also provide more samples for readers to visually judge the quality of our sample generation in Figure E.2, E.3 and E.4.
All samples are randomly selected.
F S AMPLING PROCESS AND E NERGY VALUE COMPARISONS
Here we show how the average energy of samples behaves vs the sampling temperature.
We also show an example of our model making out of distribution error that is common in most other likelihood based models (Nalisnick et al., 2019)
Figure F.
[Figure]