I MPROVED A UTOREGRESSIVE M ODELING WITH D ISTRIBUTION S MOOTHING
A BSTRACT
While autoregressive models excel at image compression, their sample quality is often lacking.
Although not realistic, generated images often have high likelihood according to the model, resembling the case of adversarial examples.
Inspired by a successful adversarial defense method, we incorporate randomized smoothing into autoregressive generative modeling.
We ﬁrst model a smoothed version of the data distribution, and then reverse the smoothing process to recover the original data distribution.
This procedure drastically improves the sample quality of existing autoregressive models on several synthetic and real-world image datasets while obtaining competitive likelihoods on synthetic datasets.
I NTRODUCTION
Autoregressive models have exhibited promising results in a variety of downstream tasks.
For instance, they have shown success in compressing images (Minnen et al., 2018), synthesizing speech (Oord et al., 2016a) and modeling complex decision rules in games (Vinyals et al., 2019).
However, the sample quality of autoregressive models on real-world image datasets is still lacking.
Poor sample quality might be explained by the manifold hypothesis: many real world data distributions ( e.g . natural images) lie in the vicinity of a low-dimensional manifold (Belkin & Niyogi, 2003), leading to complicated densities with sharp transitions ( i.e . high Lipschitz constants), which are known to be difﬁcult to model for density models such as normalizing ﬂows (Cornish et al., 2019). Since each conditional of an autoregressive model is a 1-dimensional normalizing ﬂow (given a ﬁxed context of previous pixels), a high Lipschitz constant will likely hinder learning of autoregressive models.
Another reason for poor sample quality is the “compounding error” issue in autoregressive modeling.
To see this, we note that an autoregressive model relies on the previously generated context to make a prediction; once a mistake is made, the model is likely to make another mistake which compounds (K ¨ a ¨ ari ¨ ainen, 2006), eventually resulting in questionable and unrealistic samples. Intuitively, one would expect the model to assign low-likelihoods to such unrealistic images, however, this is not always the case.
In fact, the generated samples, although appearing unrealistic, often are assigned high-likelihoods by the autoregressive model, resembling an “adversarial example” (Szegedy et al., 2013; Biggio et al., 2013), an input that causes the model to output an incorrect answer with high conﬁdence.
Inspired by the recent success of randomized smoothing techniques in adversarial defense (Cohen et al., 2019), we propose to apply randomized smoothing to autoregressive generative modeling.
More speciﬁcally, we propose to address a density estimation problem via a two-stage process.
Unlike Cohen et al.
(2019) which applies smoothing to the model to make it more robust
, we apply smoothing to the data distribution. Speciﬁcally, we convolve a symmetric and stationary noise distribution with the data distribution to obtain a new “smoother” distribution.
In the ﬁrst stage, we model the smoothed version of the data distribution using an autoregressive model.
In the second stage, we reverse the smoothing process—a procedure which can also be understood as “denoising”—by either applying a gradient-based denoising approach (Alain & Bengio, 2014) or introducing another conditional autoregressive model to recover the original data distribution from the smoothed one.
By choosing an appropriate smoothing distribution, we aim to make each step easier than the original learning problem: smoothing facilitates learning in the ﬁrst stage by making the input distribution fully supported without sharp transitions in the density function; generating a sample given a noisy one is easier than generating a sample from scratch.
We show with extensive experimental results that our approach is able to drastically improve the sample quality of current autoregressive models on several synthetic datasets and real-world image datasets, while obtaining competitive likelihoods on synthetic datasets.
We empirically demonstrate that our method can also be applied to density estimation, image inpainting, and image denoising.
B ACKGROUND
We consider a density estimation problem. Given D -dimensional i.i.d samples { x 1 , x 2 , ..., x N } from a continuous data distribution p data ( x ) , the goal is to approximate p data ( x ) with a model p θ ( x ) parameterized by θ .
A commonly used approach for density estimation is maximum likelihood estimation (MLE), where the objective is to maximize L ( θ ) (cid:44) 1 N (cid:80)
Ni =1 log p θ ( x i ) .
2. A UTOREGRESSIVE MODELS
An autoregressive model (Larochelle & Murray, 2011; Salimans et al., 2017) decomposes a joint distribution p θ ( x ) into the product of univariate conditionals:
[Equation]
(1) where x i stands for the i -th component of x , and x <i refers to the components with indices smaller than i .
In general, an autoregressive model parameterizes each conditional p θ ( x i | x <i ) using a prespeciﬁed density function ( e.g . mixture of logistics).
This bounds the capacity of the model by limiting the number of modes for each conditional.
Although autoregressive models have achieved top likelihoods amongst all types of density based models, their sample quality is still lacking compared to energy-based models (Du & Mordatch, 2019) and score-based models (Song & Ermon, 2019).
We believe this can be caused by the following two reasons.
2. M ANIFOLD H YPOTHESIS
Several existing methods (Roweis & Saul, 2000; Tenenbaum et al., 2000) rely on the manifold hypothesis, i.e . that real-world high-dimensional data tends to lie on a low-dimensional manifold (Narayanan & Mitter, 2010).
If the manifold hypothesis is true, then the density of the data distribution is not well deﬁned in the ambient space; if the manifold hypothesis holds only approximately and the data lies in the vicinity of a manifold, then only points that are very close to the manifold would have high density, while all other points would have close to zero density.
Thus we may expect the data density around the manifold to have large ﬁrst-order derivatives, i.e. the density function has a high Lipschitz constant (if not inﬁnity).
To see this, let us consider a 2-d example where the data distribution is a thin ring distribution (almost a unit circle) formed by rotating the 1-d Gaussian distribution N (1 , 0 .
01 2 ) around the origin.
The density function of the ring has a high Lipschitz constant near the “boundary”.
Let us focus on a data point travelling along the diagonal as shown in the leftmost panel in ﬁgure 2.
We plot the ﬁrst-order directional derivatives of the density for the point as it approaches the boundary from the inside, then lands on the ring, and ﬁnally moves outside the ring (see ﬁgure 2).
As we can see, when the point is far from the boundary, the derivative has a small magnitude.
When the point moves closer to the boundary, the magnitude increases and changes signiﬁcantly near the boundary even with small displacements in the trajectory.
However, once the point has landed on the ring, the magnitude starts to decrease.
As it gradually moves off the ring, the magnitude ﬁrst increases and then decreases just like when the point approached the boundary from the inside.
It has been observed that certain likelihood models, such as normalizing ﬂows, exhibit pathological behaviors on data distributions whose densities have high Lipschitz constants (Cornish et al., 2019).
Since each conditional of an autoregressive model is a 1-d normalizing ﬂow given a ﬁxed context, a high Lipschitz constant on data density could also hinder learning of autoregressive models.
2. C OMPOUNDING ERRORS IN AUTOREGRESSIVE MODELING
Autoregressive models can also be susceptible to compounding errors from the conditional distributions (Lamb et al., 2016) during sampling time.
We notice that an autoregressive model p θ ( x ) learns the joint density p data ( x ) by matching each of the conditional p θ ( x i | x <i ) with p data ( x i | x <i ) .
In practice, we typically have access to a limited amount of training data, which makes it hard for an autoregressive model to capture all the conditional distributions correctly due to the curse of dimensionality.
During sampling, since a prediction is made based on the previously generated context, once a mistake is made at a previous step, the model is likely to make more mistakes in the later steps, eventually generating a sample ˆ x that is far from being an actual image, but is mistakenly assigned a high-likelihood by the model.
The generated image ˆ x , being unrealistic but assigned a high-likelihood, resembles an adversarial example, i.e., an input that causes the model to make mistakes.
Recent works (Cohen et al., 2019) in adversarial defense have shown that random noise can be used to improve the model’s robustness to adversarial perturbations — a process during which adversarial examples that are close to actual data are generated to fool the model.
We hypothesize that such approach can also be applied to improve an autoregressive modeling process by making the model less vulnerable to compounding errors occurred during density estimation.
Inspired by the success of randomized smoothing in adversarial defense (Cohen et al., 2019), we propose to apply smoothing to autoregressive modeling to address the problems mentioned above.
G ENERATIVE MODELS WITH DISTRIBUTION SMOOTHING
In the following, we propose to decompose a density estimation task into a smoothed data modeling problem followed by an inverse smoothing problem where we recover the true data density from the smoothed one.
3. R ANDOMIZED SMOOTHING PROCESS
Unlike Cohen et al.
(2019) where randomized smoothing is applied to a model, we apply smoothing directly to the data distribution p data ( x ) .
To do this, we introduce a smoothing distribution q (˜ x | x ) — a distribution that is symmetric and stationary ( e.g . a Gaussian or Laplacian kernel) — and convolve it with p data ( x ) to obtain a new distribution q (˜ x ) (cid:44) (cid:82) q
(˜ x | x ) p data ( x ) d x .
When q (˜ x | x ) is a normal distribution, this convolution process is equivalent to perturbing the data distribution with Gaussian [Figure] noise, which, intuitively, will make the data distribution smoother.
In the following, we formally prove that convolving a 1-d distribution p data ( x ) with a suitable noise can indeed “smooth” p data ( x ) .
Theorem 1.
Given a continuous and bounded 1-d distribution p data ( x ) that is supported on R , for any 1-d distribution q (˜ x | x ) that is symmetric ( i.e . q (˜ x | x ) =
q ( x | ˜ x ) ), stationary ( i.e . translation invariant) and satisﬁes
lim x →∞ p data ( x ) q ( x | ˜ x ) = [-
-]0 for any given ˜ x , we have Lip ( q (˜ x )) ≤
Lip ( p data ( x )) , where q (˜ x ) (cid:44) (cid:82) q
(˜ x | x ) p data ( x ) dx and Lip ( · ) denotes the Lipschitz constant of the given 1-d function.
Theorem 1 shows that convolving a 1-d data distribution p data ( x ) with a suitable noise distribution q (˜ x | x ) ( e.g . N (˜ x | x, σ 2 ) ) can reduce the Lipschitzness ( i.e . increase the smoothness) of p data ( x ) .
We provide the proof of Theorem 1 in Appendix A.
Given p data ( x ) with a high Lipschitz constant, we empirically verify that density estimation becomes an easier task on the smoothed distribution q (˜ x ) than directly on p data ( x ) .
To see this, we visualize a 1-d example in ﬁgure 3a, where we want to model a ten-mode data distribution with a mixture of logistics model.
If our model has three logistic components, there is almost no way for the model, which only has three modes, to perfectly ﬁt this data distribution, which has ten separate modes with sharp transitions.
The model, after training (see ﬁgure 3a), mistakenly assigns a much higher density to the low density regions between nearby modes.
If we convolve the data distribution with q (˜ x | x ) =
N (˜ x | x, 0 .
5 2 ) , the new distribution becomes smoother (see ﬁgure 3b) and can be captured reasonably well by the same mixture of logistics model with only three modes (see ﬁgure 3b).
Comparing the same model’s performance on the two density estimation tasks, we can see that the model is doing a better job at modeling the smoothed version of the data distribution than the original data distribution, which has a high Lipschitz constant.
This smoothing process can also be understood as a regularization term for the original maximum likelihood objective (on the un-smoothed data distribution), encouraging the learned model to be smooth, as formalized by the following statement:
Proposition 1 (Informal) .
Assume that the symmetric and stationary smoothing distribution q (˜ x | x ) has small variance and negligible higher order moments, then
[Equation]
for some constant η .
Proposition 1 shows that our smoothing process provides a regularization effect on the original objective E p data ( x ) [log p θ ( x )] when no noise is added, where the regularization aims to maximize
[Equation]
Since samples from p data should be close to a local maximum of the model, this encourages the second order gradients computed at a data point x to become closer to zero (if it were positive then x will not be a local maximum), creating a smoothing effect.
This extra term is also the trace of the score function (up to a multiplicative constant) that can be found in the score matching objective (Hyv¨arinen, 2005), which is closely related to many denoising methods (Vincent, 2011;
Hyv¨arinen, 2008).
This regularization effect can, intuitively, increase the generalization capability of the model.
In fact, it has been demonstrated empirically that training with noise can lead to improvements in network generalization (Sietsma & Dow, 1991; Bishop, 1995).
Our argument is also similar to that used in (Bishop, 1995) except that we consider a more general generative modeling case as opposed to supervised learning with squared error.
We provide the formal statement and proof of Proposition 1 in Appendix A.
3. A UTOREGRESSIVE DISTRIBUTION SMOOTHING MODELS
Motivated by the previous 1-d example, instead of directly modeling p data ( x ) , which can have a high
Lipschitz constant, we propose to ﬁrst train an autoregressive model on the smoothed version of the data distribution q (˜ x ) .
Although the smoothing process makes the distribution easier to learn, it also introduces bias.
Thus, we need an extra step to debias the learned distribution by reverting the smoothing process.
If our goal is to generate approximate samples for p data ( x ) , when q (˜ x | x ) =
N (˜ x | x , σ 2 I ) and σ is small, we can use the gradient of p θ (˜ x ) for denoising (Alain & Bengio, 2014).
More speciﬁcally, given smoothed samples ˜ x from p θ (˜ x ) , we can “denoise” samples via:
[Equation]
which only requires the knowledge of p θ (˜ x ) and the ability to sample from it.
However, this approach does not provide a likelihood estimate and Eq.
(2) only works when q (˜ x | x ) is Gaussian (though alternative denoising updates for other smoothing processes could be derived under the Empirical Bayes framework (Raphan & Simoncelli, 2011)).
Although Eq.
(2) could provide reasonable denoising results when the smoothing distribution has a small variance, ¯ x obtained in this way is only a point estimation of ¯ x = E [ x | ˜ x ] and does not capture the uncertainty of p ( x | ˜ x ) .
To invert more general smoothing distributions (beyond Gaussians) and to obtain likelihood estimations, we introduce a second autoregressive model p θ ( x | ˜ x ) .
The parameterized joint density p θ ( x , ˜ x ) can then be computed as p θ ( x , ˜ x )
= p θ ( x | ˜ x ) p θ (˜ x ) .
To obtain our approximation of p data ( x ) , we need to integrate over ˜ x on the joint distribution p θ ( x , ˜ x ) to obtain p θ ( x ) = (cid:82) p θ ( x , ˜ x )
d ˜ x , which is in general intractable.
However, we can easily obtain an evidence lower bound (ELBO):
[Equation]
Note that when q (˜ x | x ) is ﬁxed, the entropy term E q (˜ x | x ) [log q (˜ x | x )] is a constant with respect to the optimization parameters.
Maximizing ELBO on p data ( x ) is then equivalent to maximizing:
[Equation]
From equation 4, we can see that optimizing the two models p θ (˜ x ) and p θ ( x | ˜ x ) separately via maximum likelihood estimation is equivalent to optimizing J ( θ ) .
3. T RADEOFF IN MODELING
In general, there is a trade-off between the difﬁculty of modeling p θ (˜ x ) and p θ ( x | ˜ x ) .
To see this, let us consider two extreme cases for the variance of q (˜ x | x ) — when q (˜ x | x ) has a zero variance and an inﬁnite variance.
When q (˜ x | x ) has a zero variance, q (˜ x | x ) is a distribution with all its probability mass at x , meaning that no noise is added to the data distribution.
In this case, modeling the smoothed distribution would be equivalent to modeling p data ( x ) , which can be hard as discussed above.
The reverse smoothing process, however, would be easy since p θ ( x | ˜ x ) can simply be an identity map to perfectly invert the smoothing process.
In the second case when q (˜ x | x ) has an inﬁnite variance, modeling p (˜ x ) would be easy because all the information about the original data is lost, and p (˜ x ) would be close to the smoothing distribution.
Modeling p ( x | ˜ x ) , on the other hand, is equivalent to directly modeling p data ( x ) , which can be challenging.
Thus, the key here is to appropriately choose a smoothing level so that both q (˜ x ) and p ( x | ˜ x ) can be approximated relatively well by existing autoregressive models.
In general, the optimal variance might be hard to ﬁnd.
Although one can train q (˜ x | x ) by jointly optimizing ELBO, in practice, we ﬁnd this approach often assigns a very large variance to q (˜ x | x ) , which can trade-off sample quality for better likelihoods on high dimensional image datasets.
We ﬁnd empirically that a pre-speciﬁed q (˜ x | x ) chosen by heuristics (Saremi & Hyvarinen, 2019; Garreau et al., 2017) is able to generate much better samples than training q (˜ x | x ) via ELBO.
In this paper, we will focus on the sample quality and leave the training of q ( ˜ x | x ) for future work. [Table]
E XPERIMENTS
In this section, we demonstrate empirically that by appropriately choosing the smoothness level of randomized smoothing, our approach is able to drastically improve the sample quality of existing autoregressive models on several synthetic and real-world datasets while retaining competitive likelihoods on synthetic datasets.
We also present results on image inpainting in Appendix C.2.
4. C HOOSING THE SMOOTHING DISTRIBUTION
To help us build insights into the selection of the smoothing distribution q (˜ x | x ) , we ﬁrst focus on a 1-d multi-modal distribution (see ﬁgure 4 leftmost panel).
We use model-based methods to invert the smoothed distribution and provide analysis on “single-step denoising” in Appendix B.1.
We start with the exploration of three different types of smoothing distributions – Gaussian distribution,
Laplace distribution, and uniform distribution.
For each type of distribution, we perform a grid search to ﬁnd the optimal variance.
Since our approach requires the modeling of both p θ (˜ x ) and p θ ( x | ˜ x ) , we stack ˜ x and x together, and use a MADE model (Germain et al., 2015) with a mixture of two logistic components to parameterize p θ (˜ x ) and p θ ( x | ˜ x ) at the same time.
For the baseline model, we train a mixture of logistics model directly on p data ( x ) .
We compare the results in the middle two panels in ﬁgure 4.
We ﬁnd that although the baseline with eight logistic components has the capacity to perfectly model the multi-modal data distribution, which has six modes, the baseline model still fails to do so.
We believe this can be caused by optimization or initialization issues for modeling a distribution with a high Lipschitz constant.
Our method, on the other hand, demonstrates more robustness by successfully modeling the different modes in the data distribution even when using only two mixture components for both p θ (˜ x ) and p θ ( x | ˜ x ) .
For all the three types of smoothing distributions, we observe a reverse U-shape correlation between the variance of q (˜ x | x ) and ELBO values — with ELBO ﬁrst increasing as the variance increases and then decreasing as the variance grows beyond a certain point.
The results match our discussion on the trade-off between modeling p θ (˜ x ) and p θ ( x | ˜ x ) in Section 3.3.
We notice from the empirical results that Gaussian smoothing is able to obtain better ELBO than the other two distributions.
Thus, we will use q (˜ x | x ) =
N (˜ x | x , σ 2 I ) for the later experiments.
4. 2-D SYNTHETIC DATASETS
In this section, we consider two challenging 2-d multi-modal synthetic datasets (see ﬁgure 5).
We focus on model-based denoising methods and present discussion on “single-step denoising” in Appendix B.2.
We use a MADE model with comparable number of total parameters for both the baseline and our approach.
For the baseline, we train the MADE model directly on the data.
For [Figure] [Figure] our randomized smoothing model, we choose q (˜ x | x ) =
N (˜ x | x , 0 .
3 2 I ) to be the smoothing distribution.
We observe that with this randomized smoothing approach, our model is able to generate better samples than the baseline (according to a human observer) even when using less logistic components (see ﬁgure 5).
We provide more analysis on the model’s performance in Appendix B.2.
We also provide the negative log-likelihoods in Tab. 1.
4. I MAGE EXPERIMENTS
In this section, we focus on three common image datasets, namely MNIST, CIFAR-10 (Krizhevsky et al., 2009) and CelebA (Liu et al., 2015).
We select q (˜ x | x ) =
N (˜ x | x , σ 2 I ) to be the smoothing distribution.
We use PixelCNN++ (Salimans et al., 2017) as the model architecture for both p θ (˜ x ) and p θ ( x | ˜ x ) .
We provide more details about settings in Appendix C.
Image generation.
For image datasets, we select the σ of q (˜ x | x ) =
N (˜ x | x , σ 2 I ) according to analysis in (Saremi & Hyvarinen, 2019)
(see Appendix C for more details).
Since q (˜ x | x ) is a
Gaussian distribution, we can apply “single-step denoising” to reverse the smoothing process for samples drawn from p θ (˜ x ) .
In this case, the model p θ ( x | ˜ x ) is not required for sampling since the gradient of p θ (˜ x ) can be used to denoise samples (also from p θ (˜ x ) )
(see equation 2).
We present smoothed samples from p θ (˜ x ) , reversed smoothing samples processed by “single-step denoising” and processed by p θ ( x | ˜ x ) in ﬁgure 6.
For comparison, we also present samples from a PixelCNN++ with parameters comparable to the sum of total parameters of p θ (˜ x ) and p θ ( x | ˜ x ) .
We ﬁnd that by using this randomized smoothing approach, we are able to drastically improve the sample quality of PixelCNN++ (see the rightmost panel in ﬁgure 6).
We note that with only p θ (˜ x ) , a PixelCNN++ optimized on the smoothed data, we already obtain more realistic samples compared to the original PixelCNN++ method.
However, p θ ( x | ˜ x ) is needed to compute the likelihood lower bounds.
We report the sample quality evaluated by Fenchel Inception Distance (FID (Heusel et al., 2017)), Kernel Inception Distance (KID (Bi´nkowski et al., 2018)), and Inception scores (Salimans et al., 2016) in Tab.
Although our method obtains better samples compared to the original PixelCNN++, our model has worse likelihoods as evaluated in BPDs.
We believe this is because likelihood and sample quality are not always directly correlated as discussed in Theis et al.
We also tried training the variance for q ( ˜ x | x ) by jointly optimizing ELBO.
Although training the variance can produce better likelihoods, it does not generate samples with comparable quality as our method ( i [Table] e . choosing variance by heuristics).
Thus, it is hard to conclusively determine what is the best way of choosing q (˜ x | x ) .
We provide more image samples in Appendix C.4 and nearest neighbors analysis in Appendix C.
[Table]
A DDITIONAL EXPERIMENTS ON NORMALIZING FLOWS
In this section [Table] we demonstrate empirically on 2-d synthetic datasets that randomized smoothing techniques can also be applied to improve the sample quality of normalizing ﬂow models (Rezende & Mohamed, 2015).
We focus on RealNVP (Dinh et al., 2016).
We compare the RealNVP model trained with randomized smoothing, where we use p θ ( x | ˜ x ) (also a RealNVP) to revert the smoothing process, with a RealNVP trained with the original method but with comparable number of parameters.
We observe that smoothing is able to improve sample quality on the datasets we consider (see ﬁgure 7) while also obtaining competitive likelihoods.
On the checkerboard dataset, our method has negative log-likelihoods 3.64 while the original RealNVP has 3.72; on the Olympics dataset, our method has negative log-likelihoods 1.32 while the original RealNVP has 1.80.
This example demonstrates that randomized smoothing techniques can also be applied to normalizing ﬂow models.
R ELATED W ORK
Our approach shares some similarities with denoising autoencoders (DAE, Vincent et al.
(2008)) which recovers a clean observation from a corrupted one.
However, unlike DAE which has a train- [Figure] able encoder and a ﬁxed prior distribution, our approach ﬁxes the encoder and models the prior using an autoregressive model.
Generative stochastic networks (GSN, Bengio et al.
(2014)) use DAEs to train a Markov chain whose equilibrium distribution matches the data distribution.
However, GSN needs to start the chain from a sample that is very close to the training distribution.
Denoising diffusion model (Sohl-Dickstein et al., 2015; Ho et al., 2020) and NCSN (Song & Ermon (2019; 2020)) address the issue of GSNs by considering a sequence of distributions corresponding to data corrupted with various noise levels.
By setting multiple noise levels that are close to each other, the sample from the previous level can serve as a proper initialization for the sampling process at the next level.
This way, the model can start from a distribution that is easy to model and gradually move to the desired distribution.
However, due to the large number of noise levels, such approaches require many steps for the chain to converge to the right data distribution.
In this paper, we instead propose to use only one level of smoothing by modeling each step with a powerful autoregressive model instead of deterministic autoencoders.
Motivated by the success of “randomized smoothing” techniques in adversarial defense (Cohen et al., 2019), we perform randomized smoothing directly on the data distribution.
Unlike denoising score matching (Vincent, 2011), a technique closely related to denoising diffusion models and NCSN, which requires the perturbed noise to be a Gaussian distribution, we are able to work with different noise distributions.
Our smoothing method is also relevant to “dequantization” approaches that are common in normalizing ﬂow models, where the discrete data distribution is converted to a continuous one by adding continuous noise (Uria et al., 2013; Ho et al., 2019).
However the added noise for “dequantization” in ﬂows is often indistinguishable to human eyes, and the reverse “dequantization” process is often ignored.
In contrast, we consider noise scales that are signiﬁcantly larger and thus a denoising process is required.
Our method is also related to “quantization” approaches which reduce the number of “signiﬁcant” bits that are modeled by a generative model (Kingma & Dhariwal, 2018; Menick & Kalchbrenner, 2018).
For instance, Glow (Kingma & Dhariwal, 2018) only models the 5 most signiﬁcant bits of an image, which improves the visual quality of samples but decreases color ﬁdelity.
SPN (Menick & Kalchbrenner, 2018) introduces another network to predict the remaining bits conditioned on the 3 most signiﬁcant bits already modeled.
Modeling the most signiﬁcant bits can be understood as capturing a data distribution perturbed by bit-wise correlated noise, similar to modeling smoothed data in our method.
Modeling the remaining bits conditioned on the most signiﬁcant ones in SPN is then similar to denoising.
However, unlike these quantization approaches which process an image at the “signiﬁcant” bits level, we apply continuous data independent Gaussian noise to the entire image with a different motivation to smooth the data density function.
D ISCUSSION
In this paper, we propose to incorporate randomized smoothing techniques into autoregressive modeling.
By choosing the smoothness level appropriately, this seemingly simple approach is able to drastically improve the sample quality of existing autoregressive models on several synthetic and real-world datasets while retaining reasonable likelihoods.
Our work provides insights into how recent adversarial defense techniques can be leveraged to building more robust generative models.
Since we apply randomized smoothing technique directly to the target data distribution other than the model, we believe our approach is also applicable to other generative models such as variational autoencoders (VAEs) and generative adversarial networks (GANs). A CKNOWLEDGEMENTS
The authors would like to thank Kristy Choi for reviewing the draft of the paper.
This research was supported by NSF (#1651565, #1522054, #1733686), ONR (N00014-19-1-2145), AFOSR (FA9550-19-1-0024), ARO, and Amazon AWS.
A P ROOFS
Theorem 1.
Given a continuous and bounded 1-d distribution p data ( x ) that is supported on R , for any 1-d distribution q (˜ x | x ) that is symmetric ( i.e . q (˜ x | x ) =
q ( x | ˜ x ) ), stationary ( i.e . translation invariant) and satisﬁes
lim x →∞ p data ( x ) q ( x | ˜ x ) = [-
-]0 for any given ˜ x , we have Lip ( q (˜ x )) ≤
Lip ( p data ( x )) , where q (˜ x ) (cid:44) (cid:82) q
(˜ x | x ) p data ( x ) dx and Lip ( · ) denotes the Lipschitz constant of the given 1-d function.
Proof.
First, we have that:
[Equation]
and if we assume symmetry, i.e. q ( x | ˜ x )
=
q (˜ x | x ) then by integration by parts we have:
[Equation]
Therefore,
[Equation]
which proves the result.
Proposition 1 (Formal) .
Given a D -dimensional data distribution p data ( x ) and model distribution p θ ( x ) , assume that the smoothing distribution q (˜ x | x ) satisﬁes: • log p θ is inﬁnitely differentiable on the support of p θ ( x ) • q (˜ x | x ) is symmetric ( i.e . q (˜ x | x ) = q ( x | ˜ x ) ) • q (˜ x | x ) is stationary (
i e . translation invariant) • q (˜ x | x ) is bounded and fully supported on R D • q (˜ x | x ) is element-wise independent • E q (˜ x | x ) [(˜ x − x ) 2 ] is bounded, and E q (˜ x | x ) [(˜ x i − x i ) 2 ] =
η at each dimension i
Denote (cid:15) = ˜ x − x , then [Equation]
where o ( (cid:15) 2 ) : R D → R is a function of (cid:15) such that lim (cid:15) → 0 o ( (cid:15) 2 ) (cid:15)= 0 . Thus when (cid:82) (cid:82) o ( (cid:15) 2 ) p data ( x ) p ( (cid:15) ) d x d (cid:15) → 0 , we have
[Equation]
Proof. To see this, we ﬁrst note that the new training objective for the smoothed data distribution is E
[Equation] p θ (˜ x )] .
Let (cid:15) = ˜ x − x , because of the assumptions we have, the PDF function q (˜ x | x ) can be reparameterized as p ( (cid:15) ) which satisﬁes: p is bounded and fully supported on R D ; p is element-wise independent and E p ( (cid:15) )
[ (cid:15) 2 i ] =
E p ( (cid:15) i ) [ (cid:15) 2 i ] = η at each dimension i ( i = 1 , ..., D ).
Then we have
[Equation]
Using Taylor expansion, we have:
[Equation]
Since (cid:15) is independent of x and
[Equation]
where δ i,j is the Kronecker delta function, the right hand side of Equation 9 becomes [Equation] When [Equation]
we have
[Equation]
where the second term on the right hand side serves as a regularization for the original objective E p data ( x ) [log p θ ( x )] .
B D ENOISING EXPERIMENTS B. A NALYSIS ON 1- D DENOISING
To provide more insights into denoising, we ﬁrst study “single-step denoising” (see equation 2) on a 1-d dataset.
We choose the data distribution to be a two mixture of Gaussian distribution 0 .
5 N ( − 0 .
1 2 ) + 0 .
5 N (0 . 3 , 0 .
1 2 ) and the smoothing distribution to be q (˜ x | x ) =
N (˜ x | x, 0 .
3 2 ) (see ﬁgure 8a).
Since the convolution of two Gaussian distributions is also a Gaussian distribution, the smoothed data is a mixture of Gaussian distribution given by 0 .
5 N ( − 0 .
1 2 + 0 .
3 2 ) + 0 .
5 N (0 . 3 , 0 .
1 2 + 0 .
The ground truth of ∇ ˜ x log p (˜ x ) can then be calculated in closed form.
Thus, given the smoothed data ˜ x , we can calculate the ground truth ∇ ˜ x log p (˜ x ) in equation 2 and obtain ¯ x using “single-step denoising”.
We visualize the denoising results in ﬁgure 8b.
We ﬁnd that the low density region between the two modes in p data ( x ) are not modeled properly in ﬁgure 8b.
However, this is very expected since “single-step denosing” uses ¯ x = E [ x | ˜ x ] as the substitute for the denoised result.
When the smoothing distribution has a large variance (like in ﬁgure 8a where the smoothed data has merged into a one mode distribution), datapoints like ˜ x 0 in the middle low density region of p data ( x ) can have high density in the smoothed distribution.
Since ˜ x 0 , as well as other points in the middle low density region of p data ( x ) , can come from both modes of p data ( x ) with high probability before the smoothing process (see ﬁgure 11a), the denoised ¯ x = E [ x | ˜ x = ˜ x 0 ] can still be located in the middle low density region (see ﬁgure 8b).
Since a large proportion of the smoothed data is located in the middle low density region of p data ( x ) , we would expect certain proportion of the density to remain in the low density region after “single-step denoising” just as shown in ﬁgure 8b.
However, when the smoothing distribution has a smaller variance, “single-step denoising” can achieve much better denoising results (see ﬁgure 9, where we use q (˜ x | x ) =
N (˜ x | x, 0 .
Although denoising can be easier when the smoothing distribution has a smaller variance, modeling the smoothed distribution could be harder as we discussed before.
In general, the right denoising results should be samples coming from p ( x | ˜ x ) , which is the reason why samples from p θ ( x | ˜ x ) ( i.e . introducing the model p θ ( x | ˜ x ) ) is more ideal than using E θ
[ x | ˜ x ] as a denoising substitute ( i.e . “single-step denoising”).
In general, the capacity of the denoising model p θ ( x | ˜ x ) also matters in terms of denoising results.
Let us again consider the datapoint ˜ x
0 shown in ﬁgure 8a.
If the invert smoothing model p θ ( x | ˜ x ) is a one mode logistic distribution, due to the mode covering property of maximum likelihood estimation, given the smoothed observation ˜ x 0 , the best the model can do is to center its only mode at ˜ x 0 for approximating p ( x | ˜ x
= ˜ x 0 ) (see ﬁgure 10c).
Thus, like ˜ x 0 , the smoothed datapoints at the low density region between the two modes of p data ( x ) are still likely to remain between the two modes after denoising (see ﬁgure 10b).
To solve this issue, we can increase the capacity of p θ ( x | ˜ x ) by making it a two mixture of logistics.
In this case, the distribution p θ ( x | ˜ x
= ˜ x 0 ) can be captured in a better way (see ﬁgure 11c and ﬁgure 11a).
After the invert smoothing process, like ˜ x 0 , most smoothed datapoints in the low density can be mapped to one of the two high density modes (see ﬁgure 11b), resulting in much better denoising effects.
[Figure]
[Figure] [Figure] [Figure]
[Figure]
B. A NALYSIS ON 2- D DENOISING
On the 2-d Olympics dataset in section 4.2, we ﬁnd that the intersections between rings can be poorly modeled with the proposed smoothing approach when only two mixture of logistics are used (see ﬁgure 12e).
We believe this can be caused if the denoising model is not ﬂexible enough to capture the distribution p ( x | ˜ x ) .
More speciﬁcally, we note that the ground truth distribution for p ( x | ˜ x ) at the intersections of the rings is a highly complicated distribution and can be hard to capture using our model which only has two mixtures of logistics for each dimension.
If we increase the ﬂexibility of p θ ( x | ˜ x ) by using three or four mixtures of logistics components (note that we still use fewer mixture components than the MADE baseline and we use comparable number of parameters), the intersection of the rings can be modeled in an improved way (see ﬁgure 12).
We also provide “single-step denoising” results for the experiments in Section 4.2 (see ﬁgure 13), where we use the same smoothing distribution, and the MADE model with three mixture components as used in section 4.2.
We note that “single-step denoising” results are not very good, which is also expected.
As discussed in section B.1, when the smoothing distribution has a relatively large variance, E θ [ x | ˜ x ] is not a good approximation for the denoised result, and we want the denoised sample to come from the distribution p θ ( x | ˜ x ) , in which case introducing a denoising model p θ ( x | ˜ x ) could be a better option.
Although we could select q (˜ x | x ) to have a smaller variance so that “single-step denoing” could work reasonably well, but modeling p (˜ x ) in this case could be more challenging. [Figure]
C I MAGE EXPERIMENTS C. S ETTINGS
For the image experiments, we ﬁrst rescale images to [ − 1 , 1] and then perturb the images with q (˜ x | x ) =
N (˜ x | x , σ 2 I ) .
We use σ = 0 .
5 for MNIST and σ = 0 .
3 for both CIFAR-10 and CelebA.
The selection of σ is mainly based on analysis in (Saremi & Hyvarinen, 2019).
More speciﬁcally, given an image, we consider the median value of the Euclidean distance between two data points in a dataset, and then divide it by 2 √ D , where D is the dimension of the data.
This provides us with a way of selecting the variance of q ( ˜ x | x ) ,
when q ( ˜ x | x ) is a Gaussian distribution.
We ﬁnd this selection of variance able to generate reasonably well samples in practice.
We train all the models with Adam optimizer with learning rate 0 . 0002 .
To model p θ ( x | ˜ x ) , we stack ˜ x and x together at the second dimension to obtain ˆ x =
[˜ x , x ] , which ensures that ˜ x comes before x in the pixel ordering.
For instance, this stacking would provide an image ˆ x with size 1 × (2 × 28) ×on a MNIST image, and an image with size 3 × (2 × 32) × 32 on a CIFAR-10 image.
Since
PixelCNN++ consists of convolutional layers, we can directly feed ˆ x into the default architecture without modifying the model architecture.
As the latter pixels of the input only depend on the previous pixels in an autoregressive model and ˜ x comes before x , we can parameterize p θ ( x | ˜ x ) by computing the likelihoods only on x using the outputs from the autoregressive model.
C. I MAGE I NPAINTING
Since both p θ (˜ x ) and p θ ( x | ˜ x ) are parameterized by an autoregressive model, we can also perform image inpainting using our method.
We present the inpainting results on CIFAR-10 in ﬁgure 14a and CelebA in ﬁgure 14b, where the bottom half of the input image is being inpainted.
[Figure]
C. I MAGE D ENOISING
We notice that the reverse smoothing process can also be understood as a denoising process.
Besides the “single-step denoising” approach shown above, we can also apply p θ ( x | ˜ x ) to denoise images.
To visualize the denoising performance, we sample x test from the test set and perturb x test with q (˜ x | x ) to obtain a noisy sample ˜ x test .
We feed ˜ x test into p θ ( x | ˜ x = ˜ x test ) and draw samples from the model.
We visualize the results in ﬁgure 15.
As we can see, the model exhibits reasonable denoising results, which shows that the autoregressive model is capable of learning the data distribution when conditioned on the smoothed data.
[Figure] [Figure]
C. N EAREST NEIGHBORS
C. A BLATION STUDIES
In this section, we show that gradient-based “single-step denoising” will not improve sample qualities without performing randomized smoothing.
To see this, we draw samples from a PixelCNN++ p θ ( x ) trained directly on p data ( x ) ( i.e . without smoothing).
We perform “single-step denoising” update deﬁned as
[Equation]
We explore various values for σ , and report the results in ﬁgure 26.
This shows that “single-step denoising” alone (without randomized smoothing) will not improve sample quality of PixelCNN++.
[Figure]