L EARNING E NERGY -B ASED M ODELS IN H IGHD IMENSIONAL S PACES WITH M ULTI - SCALE DENOISING SCORE MATCHING
Anonymous authors
Paper under double-blind review
A BSTRACT
Energy-Based Models (EBMs) assign unnormalized log-probability to data samples.
This functionality has a variety of applications, such as sample synthesis, data denoising, sample restoration, outlier detection, Bayesian reasoning, and many more.
But training of EBMs using standard maximum likelihood is extremely slow because it requires sampling from the model distribution.
Score matching potentially alleviates this problem. In particular, denoising score matching (Vincent, 2011) has been successfully used to train EBMs.
Using noisy data samples with one ﬁxed noise level, these models learn fast and yield good results in data denoising (Saremi and Hyvarinen, 2019).
However, demonstrations of such models in high quality sample synthesis of high dimensional data were lacking.
Recently, Song and Ermon (2019) have shown that a generative model trained by denoising score matching accomplishes excellent sample synthesis, when trained with data samples corrupted with multiple levels of noise. Here we provide analysis and empirical evidence showing that training with multiple noise levels is necessary when the data dimension is high.
Leveraging this insight, we propose a novel EBM trained with multi-scale denoising score matching. Our model exhibits data generation performance comparable to state-of-the-art techniques such as (Song and Ermon, 2019) and GANs, and sets a new baseline for EBMs.
The proposed model also provides density information and performs well in an image inpainting task.
I NTRODUCTION AND M OTIVATION
Treating data as stochastic samples from a probability distribution and developing models that can learn such distributions is at the core for solving a large variety of application problems, such as error correction/denoising (Vincent et al., 2010), outlier/novelty detection (Zhai et al., 2016; Choi and Jang, 2018), sample generation (Nijkamp et al., 2019; Du and Mordatch, 2019), invariant pattern recognition, Bayesian reasoning (Welling and Teh, 2011) which relies on good data priors, and many others.
Energy-Based Models (EBMs) (LeCun et al., 2006; Ngiam et al., 2011) assign an energy E ( xxx ) to each data point xxx which implicitly deﬁnes a probability by the Boltzmann distribution p m ( xxx ) =
e − E
( xxx ) /Z .
Sampling from this distribution can be used as a generative process that yield plausible samples of xxx .
Compared to other generative models, like GANs (Goodfellow et al., 2014), ﬂowbased models (Dinh et al., 2015; Kingma and Dhariwal, 2018), or auto-regressive models (van den
Oord et al., 2016; Ostrovski et al., 2018), energy-based models have signiﬁcant advantages.
First, they provide explicit (unnormalized) density information, compositionality (Hinton, 1999; Haarnoja et al., 2017), better mode coverage (Kumar et al., 2019) and ﬂexibility (Du and Mordatch, 2019).
Further, they do not require special model architecture, unlike auto-regressive and ﬂow-based models.
Recently, Energy-based models has been successfully trained with maximum likelihood (Nijkamp et al., 2019; Du and Mordatch, 2019), but training can be very computationally demanding due to the need of sampling model distribution.
Variants with a truncated sampling procedure have been proposed, such as contrastive divergence (Hinton, 2002).
Such models learn much faster with the draw back of not exploring the state space thoroughly (Tieleman, 2008).
1. S CORE M ATCHING , D ENOISING S CORE M ATCHING AND D EEP E NERGY E STIMATORS
Score matching (SM) (Hyv¨arinen, 2005) circumvents the requirement of sampling the model distribution.
In score matching, the score function is deﬁned to be the gradient of log-density or the negative energy function.
The expected L 2 norm of difference between the model score function and the data score function are minimized.
One convenient way of using score matching is learning the energy function corresponding to a Gaussian kernel Parzen density estimator (Parzen, 1962) of the data: p σ 0 (˜ xxx )
= (cid:82) q σ
(˜ xxx | xxx ) p ( xxx ) dxxx .
Though hard to evaluate, the data score is well deﬁned: s d (˜ xxx )
= ∇ ˜
xxx log( p σ
(˜ xxx )) , and the corresponding objective is:
[Equation]
Vincent (2011) studied the connection between denoising auto-encoder and score matching, and proved the remarkable result that the following objective, named Denoising Score Matching (DSM), is equivalent to the objective above:
[Equation]
Note that in (2) the Parzen density score is replaced by the derivative of log density of the single noise kernel ∇ ˜ xxx log( q σ 0 (˜ xxx | xxx )) , which is much easier to evaluate.
In the particular case of Gaussian noise, log( q σ 0 (˜ xxx | xxx ))
= − (
˜ x ˜ x ˜ x − xxx ) 2 2 σ+ C , and therefore:
[Equation]
The interpretation of objective (3) is simple, it forces the energy gradient to align with the vector pointing from the noisy sample to the clean data sample.
To optimize an objective involving the derivative of a function deﬁned by a neural network, Kingma and LeCun (2010) proposed the use of double backpropagation (Drucker and Le Cun, 1991).
Deep energy estimator networks (Saremi et al., 2018)
ﬁrst applied this technique to learn an energy function deﬁned by a deep neural network.
In this work and similarly in Saremi and Hyvarinen (2019), an energy-based model was trained to match a Parzen density estimator of data with a certain noise magnitude. The previous models were able to perform denoising task, but they were unable to generate high-quality data samples from a random input initialization.
Recently, Song and Ermon (2019) trained an excellent generative model by ﬁtting a series of score estimators coupled together in a single neural network, each matching the score of a Parzen estimator with a different noise magnitude.
The questions we address here is why learning energy-based models with single noise level does not permit high-quality sample generation and what can be done to improve energy based models.
Our work builds on key ideas from Saremi et al.
; Saremi and Hyvarinen (2019); Song and Ermon (2019). Section 2, provides a geometric view of the learning problem in denoising score matching and provides a theoretical explanation why training with one noise level is insufﬁcient if the data dimension is high. Section 3 presents a novel method for training energy based model, Multiscale
Denoising Score Matching (MDSM). Section 4 describes empirical results of the MDSM model and comparisons with other models.
A G EOMETRIC VIEW OF D ENOISING S CORE M ATCHING
Song and Ermon (2019) used denoising score matching with a range of noise levels, achieving great empirical results.
The authors explained that large noise perturbation are required to enable the learning of the score in low-data density regions.
But it is still unclear why a series of different noise levels are necessary, rather than one single large noise level. Following Saremi and Hyvarinen (2019), we analyze the learning process in denoising score matching based on measure concentration properties of high-dimensional random vectors.
We adopt the common assumption that the data distribution to be learned is high-dimensional, but only has support around a relatively low-dimensional manifold (Tenenbaum et al., 2000; Roweis and Saul, 2000; Lawrence, 2005).
If the assumption holds, it causes a problem for score matching:
The density, or the gradient of the density is then undeﬁned outside the manifold, making it difﬁcult A. Training B. Sampling [Figure] to train a valid density model for the data distribution deﬁned on the entire space.
Saremi and Hyvarinen (2019) and Song and Ermon (2019) discussed this problem and proposed to smooth the data distribution with a Gaussian kernel to alleviate the issue.
To further understand the learning in denoising score matching when the data lie on a manifold X and the data dimension is high, two elementary properties of random Gaussian vectors in highdimensional spaces are helpful:
First, the length distribution of random vectors becomes concentrated at √ dσ (Vershynin, 2018), where σ 2 is the variance of a single dimension.
Second, a random vector is always close to orthogonal to a ﬁxed vector (Tao, 2012).
With these premises one can visualize the conﬁguration of noisy and noiseless data points that enter the learning process: A data point xxx sampled from X and its noisy version ˜ xxx always lie on a line which is almost perpendicular to the tangent space T xxx X and intersects X at xxx .
Further, the distance vectors between ( xxx, ˜ xxx ) pairs all have similar length √ dσ .
As a consequence, the set of noisy data points concentrate on a set ˜ X √ dσ,(cid:15) that has a distance with ( √ dσ − (cid:15), √ dσ + (cid:15) ) from the data manifold X , where (cid:15) (cid:28) √ dσ .
Therefore, performing denoising score matching learning with ( xxx, ˜ xxx ) pairs generated with a ﬁxed noise level σ , which is the approach taken previously except in Song and Ermon (2019), will match the score in the set ˜ X √ dσ,(cid:15) and enable denoising of noisy points in the same set.
However, the learning provides little information about the density outside this set, farther or closer to the data manifold, as noisy samples outside ˜ X √ dσ,(cid:15) rarely appear in the training process.
An illustration is presented in Figure 1A.
Let ˜ X C √ dσ,(cid:15) denote the complement of the set ˜ X √ dσ,(cid:15) . Even if p σ 0 (˜ xxx ∈ ˜ X C √ dσ,(cid:15) ) is very small in high-dimensional space, the score in ˜ X C √ dσ,(cid:15) still plays a critical role in sampling from random initialization.
This analysis may explain why models based on denoising score matching, trained with a single noise level encounter difﬁculties in generating data samples when initialized at random.
For an empirical support of this explanation, see our experiments with models trained with single noise magnitudes (Appendix B).
To remedy this problem, one has to apply a learning procedure of the sort proposed in Song and Ermon (2019), in which samples with different noise levels are used. Depending on the dimension of the data, the different noise levels have to be spaced narrowly enough to avoid empty regions in the data space. In the following, we will use Gaussian noise and employ a Gaussian scale mixture to produce the noisy data samples for the training (for details, See
Section 3.1 and Appendix A). Another interesting property of denoising score matching was suggested in the denoising autoencoder literature (Vincent et al., 2010; Karklin and Simoncelli, 2011).
With increasing noise level, the learned features tend to have larger spatial scale.
In our experiment we observe similar phenomenon when training model with denoising score matching with single noise scale.
If one compare samples in Figure B.1, Appendix B, it is evident that noise level of 0.3 produced a model that learned short range correlation that spans only a few pixels, noise level of 0.6 learns longer stroke structure without coherent overall structure, and noise level of 1 learns more coherent long range structure without details such as stroke width variations.
This suggests that training with single noise level in denoising score matching is not sufﬁcient for learning a model capable of high-quality sample synthesis, as such a model have to capture data structure of all scales.
L EARNING E NERGY - BASED MODEL WITH M ULTISCALE D ENOISING SCORE MATCHING 3. M ULTISCALE D ENOISING S CORE MATCHING
Motivated by the analysis in section 2, we strive to develop an EBM based on denoising score matching that can be trained with noisy samples in which the noise level is not ﬁxed but drawn from a distribution.
The model should approximate the Parzen density estimator of the data p σ 0 (˜ xxx )
= (cid:82) q σ
(˜ xxx | xxx ) p ( xxx ) dx . Speciﬁcally, the learning should minimize the difference between the derivative of the energy and the score of p σ 0 under the expectation E p M (˜ xxx ) rather than E p σ [-
-]0 (˜ xxx ) , the expectation taken in standard denoising score matching.
Here p M (˜ xxx ) =
(cid:82) q M (˜ xxx
| xxx ) p ( xxx ) dx is chosen to cover the signal space more evenly to avoid the measure concentration issue described above. The resulting
Multiscale Score Matching (MSM) objective is:
[Equation]
Compared to the objective of denoising score matching (1), the only change in the new objective (4) is the expectation.
Both objectives are consistent, if p M (˜ xxx ) and p σ 0 (˜ xxx ) have the same support, as shown formally in Proposition 1 of Appendix A.
In Proposition 2, we prove that Equation 4 is equivalent to the following denoising score matching objective:
[Equation]
The above results hold for any noise kernel q σ 0 (˜ xxx | xxx ) , but Equation 5 contains the reversed expectation, which is difﬁcult to evaluate in general.
To proceed, we choose q σ 0 (˜ xxx | xxx ) to be Gaussian, and also choose q M (˜ xxx | xxx ) to be a Gaussian scale mixture: q M (˜ xxx | xxx )
= (cid:82) q σ (˜ xxx | xxx ) p ( σ ) dσ and q σ (˜ xxx | xxx )
=
N ( xxx, σ 2 I d ) .
After algebraic manipulation and one approximation (see the derivation following Proposition 2 in Appendix A), we can transform Equation 5 into a more convenient form, which we call Multiscale Denoising Score Matching (MDSM): [Equation]
[Equation]
The square loss term evaluated at noisy points ˜ xxx at larger distances from the true data points xxx will have larger magnitude.
Therefore, in practice it is convenient to add a monotonically decreasing term l ( σ ) for balancing the different noise scales, e.g. l ( σ ) = 1 σ 2 .
Ideally, we want our model to learn the correct gradient everywhere, so we would need to add noise of all levels. However, learning denoising score matching at very large or very small noise levels is useless.
At very large noise levels the information of the original sample is completely lost.
Conversely, in the limit of small noise, the noisy sample is virtually indistinguishable from real data.
In neither case one can learn a gradient which is informative about the data structure.
Thus, the noise range needs only to be broad enough to encourage learning of data features over all scales.
Particularly, we do not sample σ but instead choose a series of ﬁxed σ values σ 1 · · · σ K .
Further, substituting log( q σ 0 (˜ xxx | xxx ))
= − (
˜ x ˜ x ˜ x − xxx ) 2 2 σ+ C into Equation 4, we arrive at the ﬁnal objective: [Equation]
[Equation]
It may seem that σ 0 is an important hyperparameter to our model, but after our approximation σbecome just a scaling factor in front of the energy function, and can be simply set to one as long as the temperature range during sampling is scaled accordingly (See Section 3.2).
Therefore the only hyper-parameter is the rang of noise levels used during training. On the surface, objective (7) looks similar to the one in Song and Ermon (2019).
The important difference is that Equation 7 approximates a single distribution, namely p σ {+
 (˜ xxx ) , the data smoothed with one ﬁxed kernel q σ+} 0 (˜ xxx | xxx ) .
In contrast, Song and Ermon (2019) approximate the score of multiple distributions, the family of distributions { p σ i (˜ xxx ) : i = 1 , ..., n } , resulting from the data smoothed by kernels of different widths σ i .
Because our model learns only a single target distribution, it does not require noise magnitude as input.
3. S AMPLING BY A NNEALED L ANGEVIN D YNAMICS
Langevin dynamics has been used to sample from neural network energy functions (Du and Mordatch, 2019; Nijkamp et al., 2019).
However, the studies described difﬁculties with mode exploration unless very large number of sampling steps is used.
To improve mode exploration, we propose incorporating simulated annealing in the Langevin dynamics.
Simulated annealing (Kirkpatrick et al., 1983; Neal, 2001) improves mode exploration by sampling ﬁrst at high temperature and then cooling down gradually. This has been successfully applied to challenging computational problems, such as combinatorial optimization. To apply simulated annealing to Langevin dynamics.
Note that in a model of Brownian motion of a physical particle, the temperature in the Langevin equation enters as a factor √ T in front of the noise term, some literature uses (cid:112) β − 1 where β
= 1 /T
(Jordan et al., 1998).
Adopting the √ T convention, the Langevin sampling process (Bellec et al., 2017) is given by:
[Equation]
(8) where T t follows some annealing schedule, and (cid:15) denotes step length, which is ﬁxed.
During sampling, samples behave very much like physical particles under Brownian motion in a potential ﬁeld.
Because the particles have average energies close to the their current thermic energy, they explore the state space at different distances from data manifold depending on temperature.
Eventually, they settle somewhere on the data manifold.
The behavior of the particle’s energy value during a typical annealing process is depicted in Appendix Figure F.1B.
If the obtained sample is still slightly noisy, we can apply a single step gradient denoising jump (Saremi and Hyvarinen, 2019) to improve sample quality:
[Equation]
This denoising procedure can be applied to noisy sample with any level of Gaussian noise because in our model the gradient automatically has the right magnitude to denoise the sample.
This process is justiﬁed by the Empirical Bayes interpretation of this denoising process, as studied in Saremi and Hyvarinen (2019).
Song and Ermon (2019) also call their sample generation process annealed Langevin dynamics.
It should be noted that their sampling process does not coincide with Equation 8.
Their sampling procedure is best understood as sequentially sampling a series of distributions corresponding to data distribution corrupted by different levels of noise.
I MAGE M ODELING R ESULTS
Training and Sampling Details.
The proposed energy-based model is trained on standard image datasets, speciﬁcally MNIST, Fashion MNIST, CelebA
(Liu et al., 2015) and CIFAR-10 (Krizhevsky et al., 2009).
During training we set σ 0 = 0 .
1 and train over a noise range of σ ∈ [0 . 05 , 1 . 2] , with the different noise uniformly spaced on the batch dimension. For MNIST and Fashion MNIST we used geometrically distributed noise in the range [0 . 1 , 3] .
The weighting factor l ( σ ) is always set to 1 /σ 2 to make the square term roughly independent of σ .
We ﬁx the batch size at 128 and use the Adam optimizer with a learning rate of 5 × 10 − 5 .
For MNIST and Fashion MNIST, we use a 12-Layer ResNet with 64 ﬁlters, for the CelebA and CIFAT-10 data sets we used a 18-Layer ResNet with 128 ﬁlters
(He et al., 2016a;b).
No normalization layer was used in any of the networks.
We designed the output layer of all networks to take a generalized quadratic form (Fan et al., 2018). Because the energy function is anticipated to be approximately quadratic with respect to the noise level, this modiﬁcation was able to boost the performance signiﬁcantly.
For more detail on training and model architecture, see Appendix D. One notable result is that since our training method does not involve sampling, we achieved a speed up of roughly an order of magnitude compared to the maximum-likelihood training using Langevin dynamics 1 .
Our method thus enables the training of energy-based models even when limited computational resources prohibit maximum likelihood methods.
We found that the choice of the maximum noise level has little effect on learning as long as it is large enough to encourage learning of the longest range features in the data.
However, as expected, learning with too small or too large noise levels is not beneﬁcial and can even destabilize the training process.
Further, our method appeared to be relatively insensitive to how the noise levels are distributed over a chosen range.
Geometrically spaced noise as in (Song and Ermon, 2019) and linearly spaced noise both work, although in our case learning with linearly spaced noise was somewhat more robust.
For sampling the learned energy function we used annealed Langevin dynamics with an empirically optimized annealing schedule,see Figure F.1B for the particular shape of annealing schedule we used.
In contrast, annealing schedules with theoretical guaranteed convergence property takes extremely long (Geman and Geman, 1984).
The range of temperatures to use in the sampling process depends on the choice of σ 0 , as the equilibrium distribution is roughly images with Gaussian noise of magnitude √ Tσ 0 added on top.
To ease traveling between modes far apart and ensure even sampling, the initial temperature needs to be high enough to inject noise of sufﬁcient magnitude.
A choice of T = 100 , which corresponds to added noise of magnitude √ 100 ∗ 0 .
1 = 1 , seems to be sufﬁcient starting point.
For step length (cid:15) we generally used 0 .
02 , although any value within the range [0 . 015 , 0 . 05] seemed to work ﬁne.
After the annealing process we performed a single step denoising to further enhance sample quality. [Table]
Unconditional Image Generation.
We demonstrate the generative ability of our model by displaying samples obtained by annealed Langevin sampling and single step denoising jump.
We evaluated 50k sampled images after training on CIFAR-10 with two performance scores, Inception (Salimans et al., 2016) and FID (Heusel et al., 2017).
We achieved Inception Score of 8.31 and FID of 31.7, comparable to modern GAN approaches.
Scores for CelebA dataset are not reported here as they are not commonly reported and may depend on the speciﬁc pre-processing used.
More samples and training images are provided in Appendix for visual inspection.
We believe that visual assessment is still essential because of the possible issues with the Inception score (Barratt and Sharma, 2018).
Indeed, we also found that the visually impressive samples were not necessarily the one achieving the highest Inception Score.
Although overﬁtting is not a common concern for generative models, we still tested our model for overﬁtting.
We found no indication for overﬁtting by comparing model samples with their nearest neighbors in the data set, see Figure C.1 in Appendix.
Mode Coverage.
We repeated with our model the 3 channel MNIST mode coverage experiment similar to the one in Kumar et al.
An energy-based model was trained on 3-channel data where each channel is a random MNIST digit.
Then 8000 samples were taken from the model and each channel was classiﬁed using a small MNIST classiﬁer network.
We obtained results of the 966 modes, comparable to GAN approaches.
Training was successful and our model assigned low energy to all the learned modes, but some modes were not accessed during sampling, likely due to the Langevin Dynamics failing to explore these modes.
A better sampling technique such as HMC
Neal et al.
(2011) or a Maximum Entropy Generator (Kumar et al., 2019) could improve this result.
Image Inpainting.
Image impainting can be achieved with our model by clamping a part of the image to ground truth and performing the same annealed Langevin and Jump sampling procedure on the missing part of the image.
Noise appropriate to the sampling temperature need to be added to the clamped inputs.
The quality of inpainting results of our model trained on CelebA and CIFARcan be assessed in Figure 3.
For CIFAR-10 inpainting results we used the test set.
Log likelihood estimation.
For energy-based models, the log density can be obtained after estimating the partition function with Annealed Importance Sampling (AIS)
(Salakhutdinov and Murray, 2008) or Reverse AIS (Burda et al., 2015).
In our experiment on CIFAR-10 model, similar to reports in Du and Mordatch (2019), there is still a substantial gap between AIS and Reverse AIS estimation, even after very substantial computational effort.
In Table 1, we report result from Reverse AIS, as it tends to over-estimate the partition function thus underestimate the density.
Note that although density values and likelihood values are not directly comparable, we list them together due to the sheer lack of a density model for CIFAR-10.
We also report a density of 1.21 bits/dim on MNIST dataset, and we refer readers to Du and Mordatch (2019) for comparison to other models on this dataset.
More details on this experiment is provided in the Appendix.
Outlier Detection.
Choi and Jang (2018) and Nalisnick et al.
(2019) have reported intriguing behavior of high dimensional density models on out of distribution samples.
Speciﬁcally, they showed that a lot of models assign higher likelihood to out of distribution samples than real data samples.
We investigated whether our model behaves similarly.
Our energy function is only trained outside the data manifold where samples are noisy, so the energy value at clean data points may not always be well behaved.
Therefore, we added noise with magnitude σ 0 before measuring the energy value.
We ﬁnd that our network behaves similarly to previous likelihood models, it assigns lower energy, thus higher density, to some OOD samples.
We show one example of this phenomenon in Appendix Figure F.1A.
We also attempted to use the denoising performance, or the objective function to perform outlier detection.
Intriguingly, the results are similar as using the energy value. Denoising performance seems to correlate more with the variance of the original image than the content of the image.
D ISCUSSION
In this work we provided analyses and empirical results for understanding the limitations of learning the structure of high-dimensional data with denoising score matching.
We found that the objective function conﬁnes learning to a small set due to the measure concentration phenomenon in random vectors.
Therefore, sampling the learned distribution outside the set where the gradient is learned does not produce good result.
One remedy to learn meaningful gradients in the entire space is to use samples during learning that are corrupted by different amounts of noise. Indeed, Song and Ermon (2019) applied this strategy very successfully. The central contribution of our paper is to investigate how to use a similar learning strategy in EBMs. Speciﬁcally, we proposed a novel EBM model, the Multiscale Denoising Score Matching (MDSM) model.
The new model is capable of denoising, producing high-quality samples from random noise, and performing image inpainting.
While also providing density information, our model learns an order of magnitude faster than models based on maximum likelihood.
Our approach is conceptually similar to the idea of combining denoising autoencoder and annealing (Geras and Sutton, 2015; Chandra and Sharma, 2014; Zhang and Zhang, 2018) though this idea was proposed in the context of pre-training neural networks for classiﬁcation applications.
Previous efforts of learning energy-based models with score matching (Kingma and LeCun, 2010; Song et al., 2019) were either computationally intensive or unable to produce high-quality samples comparable to those obtained by other generative models such as GANs.
Saremi et al.
and Saremi and Hyvarinen (2019) trained energy-based model with the denoising score matching objective but the resulting models cannot perform sample synthesis from random noise initialization.
Recently, Song and Ermon (2019) proposed the NCSN model, capable of high-quality sample synthesis. This model approximates the score of a family of distributions obtained by smoothing the data by kernels of different widths.
The sampling in the NCSN model starts with sampling the distribution obtained with the coarsest kernel and successively switches to distributions obtained with ﬁner kernels.
Unlike NCSN, our method learns an energy-based model corresponding to p σ
(˜ xxx ) for a ﬁxed σ 0 .
{+This method improves score matching in high-dimensional space by matching the gradient of an energy function to the score of p σ 
+}
(˜ xxx ) in a set that avoids measure concentration issue. All told, we offer a novel EBM model that achieves high-quality sample synthesis, which among other EBM approaches provides a new state-of-the art. Compared to the NCSN model, our model is more parsimonious than NCSN and can support single step denoising without prior knowledge of the noise magnitude. But our model performs sightly worse than the NCSN model, which could have several reasons.
First, the derivation of Equation 6 requires an approximation to keep the training procedure tractable, which could reduce the performance. Second, the NCSNs output is a vector that, at least during optimization, does not always have to be the derivative of a scalar function. In contrast, in our model the network output is a scalar function. Thus it is possible that the NCSN model performs better because it explores a larger set of functions during optimization.
A MDSM O BJECTIVE
In this section, we provide a formal discussion of the MDSM objective and suggest it as an improved score matching formulation in high-dimensional space.
Vincent (2011) illustrated the connection between the model score −∇
˜ xxx E (˜ xxx ; θ ) with the score of
Parzen window density estimator ∇ ˜ xxx log( p σ
(˜ xxx )) .
Speciﬁcally, the objective is Equation 1 which we restate here:
[Equation]
Our key observation is: in high-dimensional space, due the concentration of measure, the expectation w.r.t.
p σ 0 (˜ xxx ) over weighs a thin shell at roughly distance √ dσ to the empirical distribution p ( x ) .
Though in theory this is not a problem, in practice this leads to results that the score are only well matched on this shell.
Based on this observation, we suggest to replace the expectation w.r.t.
p σ 0 (˜ xxx ) with a distribution p σ (cid:48) (˜ xxx ) that has the same support as p σ
(˜ xxx ) but can avoid the measure concentration problem.
We call this multiscale score matching and the objective is the following:
[Equation]
Proposition 1.
L MSM ( θ ) =
⇐⇒ L SM ( θ ) =
⇐⇒ θ
= θ ∗ .
Given that p M (˜ xxx ) and p σ 0 (˜ xxx ) has the same support, it’s clear that L MSM = 0 would be equivalent to
L SM = 0 .
Due to the proof of the Theorem 2 in Hyv ¨ arinen (2005), we have L SM ( θ ) ⇐⇒ θ
= θ ∗ .
Thus, L MSM ( θ ) =
⇐⇒ θ
= θ ∗ .
Proposition 2.
L
[Equation]
We follow the same procedure as in Vincent (2011) to prove this result
[Equation]
Thus we have:
[Equation]
So L MSM ( θ ) (cid:94) L MDSM ∗ .
The above analysis applies to any noise distribution, not limited to Gaussian.
but L MDSM ∗ has a reversed expectation form that is not easy to work with.
To proceed further we study the case where q σ 0 (˜ xxx | xxx ) is Gaussian and choose q M (˜ xxx | xxx ) as a Gaussian scale mixture (Wainwright and Simoncelli, 2000) and p M (˜ xxx ) =
(cid:82) q M (˜ xxx
| xxx ) p ( xxx ) dx .
By Proposition 1 and Proposition 2, we have the following form to optimize:
[Equation]
To minimize Equation (*), we can use the following importance sampling procedure (Russell and
Norvig, 2016): we can sample from the empirical distribution p ( xxx ) , then sample the Gaussian scale mixture q M (˜ xxx | xxx ) and ﬁnally weight the sample by q σ 0 ( xxx | ˜ xxx ) q M
( xxx | ˜ xxx ) .
We expect the ratio to be close to 1 for the following reasons: Using Bayes rule, q σ 0 ( xxx | ˜ xxx )
= p ( xxx ) q σ 0 (˜ xxx | xxx ) p σ 0 (˜ xxx )
we can see that q σ 0 ( xxx | ˜ xxx ) only has support on discret data points xxx , same thing holds for q M ( xxx | ˜ xxx ) .
because in ˜ xxx is generated by adding
Gaussian noise to real data sample, both estimators should give results highly concentrated on the original sample point xxx .
Therefore, in practice, we ignore the weighting factor and use Equation 6.
Improving upon this approximation is left for future work.
B P ROBLEM WITH SINGLE NOISE DENOISING SCORE MATCHING
To compare with previous method, we trained energy-based model with denoising score matching using one noise level on MNIST, initialized the sampling with Gaussian noise of the same level, and sampled with Langevin dynamics at T = 1 for 1000 steps and perform one denoise jump to recover the model’s best estimate of the clean sample, see Figure B.1.
We used the same 12-layer ResNet as other MNIST experiments.
Models were trained for 100000 steps before sampling.
C O VERFITTING TEST
We demonstrate that the model does not simply memorize training examples by comparing model samples with their nearest neighbors in the training set.
We use Fashion MNIST for this demonstration because overﬁtting can occur there easier than on more complicated datasets, see Figure C.1.
D D ETAILS ON T RAINING AND S AMPLING
We used a custom designed ResNet architecture for all experiments.
For MNIST and FashionMNIST we used a 12-layer ResNet with 64 ﬁlters on ﬁrst layer, while for CelebA and CIFAR dataset we used a 18-layer ResNet with 128 ﬁlters on the ﬁrst layer.
All network used the ELU activation function.
We did not use any normalization in the ResBlocks and the ﬁler number is doubled at each downsampling block.
Details about the structure of our networks used can be found in our code release.
All mentioned models can be trained on 2 GPUs within 2 days.
Since the gradient of our energy model scales linearly with the noise, we expected our energy function to scale quadratically with noise magnitude.
Therefore, we modiﬁed the standard energy-based network output layer to take a ﬂexible quadratic form (Fan et al., 2018):
[Equation]
where a i , c i , d i and b 1 , b 2 , b 3 are learnable parameters, and h i is the (ﬂattened) output of last residual block.
We found this modiﬁcation to signiﬁcantly improve performance compared to using a simple linear last layer.
For CIFAR and CelebA results we trained for 300k weight updates, saving a checkpoint everyupdates.
We then took 1000 samples from each saved networks and used the network with the lowest
FID score.
For MNIST and fashion MNIST we simply trained for 100k updates and used the last checkpoint.
During training we pad MNIST and Fashion MNIST to 32*32 for convenience and randomly ﬂipped CelebA images.
No other modiﬁcation was performed.
We only constrained the gradient of the energy function, the energy value itself could in principle be unbounded.
However, we observed that they naturally stabilize so we did not explicitly regularize them.
The annealing sampling schedule is optimized to improve sample quality for CIFAR-10 dataset, and consist of a total of 2700 steps.
For other datasets the shape has less effect on sample quality, see Figure F.1 B for the shape of annealing schedule used.
For the Log likelihood estimation we initialized reverse chain on test images, then sampleintermediate distribution using 10 steps HMC updates each.
Temperature schedule is roughly exponential shaped and the reference distribution is an isotropic Gaussian.
The variance of estimation was generally less than 10% on the log scale.
Due to the high variance of results, and to avoid getting dominated by a single outlier, we report average of the log density instead of log of average density.
E E XTENDED S AMPLES AND I NPAINTING RESULTS
We provide more inpainting examples and further demonstrate the mixing during sampling process in Figure E.1.
We also provide more samples for readers to visually judge the quality of our sample generation in Figure E.2, E.3 and E.4.
All samples are randomly selected.
F S AMPLING PROCESS AND E NERGY VALUE COMPARISONS
Here we show how the average energy of samples behaves vs the sampling temperature.
We also show an example of our model making out of distribution error that is common in most other likelihood based models (Nalisnick et al., 2019)
Figure F.
[Figure]