I MPROVED A UTOREGRESSIVE M ODELING WITH D ISTRIBUTION S MOOTHING
Anonymous authors
Paper under double-blind review
A BSTRACT
While autoregressive models excel at image compression, their sample quality is often lacking.
Inspired by randomized smoothing for adversarial defense, we incorporate randomized smoothing techniques into autoregressive generative modeling.
We ﬁrst model a smoothed version of the data distribution and then recover the data distribution by learning to reverse the smoothing process.
We demonstrate empirically on a 1-d dataset that by appropriately choosing the smoothing level, we can keep the proposed process relatively easier to model than directly learning a data distribution with a high Lipschitz constant.
Since autoregressive generative modeling consists of a sequence of 1-d density estimation problems, we believe the same arguments can be generalized to an autoregressive model.
This seemingly simple procedure drastically improves the sample quality of existing autoregressive models on several synthetic and real-world datasets while obtaining competitive likelihoods on synthetic datasets.
I NTRODUCTION
Autoregressive models have exhibited promising results in a variety of downstream tasks.
For instance, they have shown success in compressing real-world images (Minnen et al., 2018), synthesizing realistic speech (Oord et al., 2016a) and providing competitive game strategies (Vinyals et al., 2019).
However, the sample quality of autoregressive models on real-world image datasets is still lacking, which, we believe, can be caused by the following two reasons.
The ﬁrst one is the manifold hypothesis, which states that many real world data distributions ( e.g . natural images) may lie in the vicinity of a low-dimensional manifold (Belkin & Niyogi, 2003) and can often have complicated densities with sharp transitions ( i.e . high Lipschitz constants) that are difﬁcult to model.
While addressing the manifold hypothesis can already be a challenging problem even with inﬁnitely many data, in real life, one might only have access to a limited amount of training samples, making learning even harder.
Besides, an insufﬁcient amount of training data can also give rise to “compounding error” issues.
To see this, we note that an autoregressive model relies on the previous context to make prediction for the next-step; once a mistake is made, the model is likely to make another mistake which compounds (K¨a¨ari¨ainen, 2006), eventually resulting in samples that are questionable or unrealistic compared to the data distribution but have high-likelihoods when evaluated by the model.
The unrealistic sample, in some sense, resembles an “adversarial example” (Goodfellow et al., 2014).
Inspired by the recent success of randomized smoothing techniques in adversarial defense (Cohen et al., 2019), we propose to apply randomized smoothing to autoregressive generative modeling.
More speciﬁcally, we propose to address a density estimation problem via a two-stage process.
Unlike Cohen et al.
(2019) which applies smoothing to the model to make it more robust
, we apply smoothing to the data distribution, where a symmetric and stationary distribution is convolved with the data distribution to make the new distribution “smooth”.
In the ﬁrst stage, we model the smoothed version of the data distribution using an autoregressive model.
In the second stage, we reverse the smoothing process — a procedure which can also be understood as “denoising”, by either applying a gradient-based denoising approach (Saremi et al., 2018) or introducing another autoregressive model to recover the original data distribution.
By choosing an appropriate smoothing, we aim to make each step easier than the original learning problem: smoothing facilitates learning in the ﬁrst stage, and generating a sample given a noisy one is easier than generating one from scratch.
We show with extensive experimental results that our approach is able to drastically improve the sample quality of current autoregressive models on several synthetic datasets and real-world image datasets, while obtaining competitive likelihoods on synthetic datasets.
We empirically demonstrate that our method can also be applied to density estimation, image inpainting and image denoising.
B ACKGROUND
Given D -dimensional i.i.d samples { x 1 , x 2 , ..., x N } from a continuous data distribution p data ( x ) , we want to perform density estimation on p data ( x ) with a density function p θ ( x ) often parameterized by a deep network, where θ are parameters of the model.
A commonly used approach for density estimation is maximum likelihood estimation (MLE), i.e., by maximizing L ( θ ) (cid:44) 1 N (cid:80)
Ni =1 log p θ ( x i ) .
2. A UTOREGRESSIVE MODELS
An autoregressive model (Salimans et al., 2017; Germain et al., 2015) decomposes a joint distribution into univariate conditionals:
[Equation]
where x i refers to the i -th index of x , and x <i refers to the components with indices smaller than i in the given ordering.
In general, an autoregressive model parameterizes each conditional p ( x i | x <i ) using a pre-speciﬁed density function ( e.g . mixture of logistics).
However, this approach bounds the capacity of the model by limiting the number of modes for each conditional.
Although autoregressive models have achieved top likelihoods amongst all types of density based models, they are still unable to generate high quality image samples (Du & Mordatch, 2019).
We believe this can be caused by the following two reasons.
2. M ANIFOLD H YPOTHESIS
Several existing methods (Roweis & Saul, 2000; Tenenbaum et al., 2000) rely on the manifold hypothesis, i.e . that real-world high-dimensional data tends to lie on a low-dimensional manifold (Narayanan & Mitter, 2010).
If the manifold hypothesis is true, then the density of the data distribution is not well deﬁned in the ambient space; if the manifold hypothesis holds only approximately and the data lies in the vicinity of a manifold, then only points that are very close to the manifold would have high density, while all other points would have close to zero density.
Thus we may expect the data density around the manifold to have large ﬁrst-order derivatives, i.e. the density function has a high Lipschitz constant (if not inﬁnity).
To see this, let us consider a 2-d example where the data distribution is a very thin ring distribution (almost a unit circle) formed by rotating the 1-d Gaussian distribution N (1 , 0 .
01 2 ) around the origin.
The density function of the ring has a high Lipschitz constant near the “boundary”.
Let us focus on a data point travelling along the diagonal as shown in the leftmost panel in ﬁgure 2.
We plot the the ﬁrst-order derivatives of the density for the data as it approaches the boundary from the inside, then lands on the ring, and ﬁnally moves outside the ring (see ﬁgure 2).
As we can see, when the point is far from the boundary, the ﬁrst-order derivative has a small magnitude.
When the data moves closer to the boundary, the magnitude increases and changes signiﬁcantly near the boundary even with small displacements in the trajectory.
However, once the data has landed on the ring, the magnitude start to decrease.
As the data gradually moves off the ring, the magnitude ﬁrst increases and then decreases just like when the point approached the boundary from the inside.
For certain likelihood models, such as normalizing ﬂows (Cornish et al., 2019), it has been observed that having high
Lipschitz constants from such datasets is detrimental to training; this could also negatively impact learning of autoregressive models.
2. C OMPOUNDING ERRORS IN AUTOREGRESSIVE MODELING
Autoregressive models can also be susceptible to compounding errors from the conditional distributions (Lamb et al., 2016).
We notice that an autoregressive model p θ ( x ) learns the joint density p data ( x ) by matching each of the conditional p θ ( x i | x <i ) with p data ( x i | x <i ) .
In reality, we often have access to very limited amount of training data, which makes it hard for an autoregeressive model to capture all the conditional distributions correctly due to the curse of dimensionality.
During sampling time, since a prediction of the current state is based on the values of the previous states, once a mistake is made at a previous step, the model is likely to make more mistakes in the later steps, eventually generating a sample ˆ x that is far from being an actual image, but is mistakenly assigned a high density by the model.
The unrealistic sample ˆ x , in some sense, resembles an adversarial example, i.e., an input that causes the model to make mistakes.
Recent works (Cohen et al., 2019) in adversarial defense have shown that random noise can be used to improve the model’s robustness to adversarial perturbations — a process during which adversarial examples that are close to actual data are generated to fool the model.
We hypothesize that such approach can also be applied to improve an autoregressive modeling process by making the model less vulnerable to compounding errors occurred during density estimation.
Inspired by the success of randomized smoothing in adversarial defense (Cohen et al., 2019), we propose to apply smoothing to autoregressive modeling to address the problems mentioned above.
G ENERATIVE MODELS WITH DISTRIBUTION SMOOTHING
[Figure]
In the following, we propose to decompose a density estimation task into a smoothed data modeling problem and a reverse smoothing problem.
3. R ANDOMIZED SMOOTHING PROCESS
Inspired by the success of “randomized smoothing” in adversarial defense (Cohen et al., 2019), we propose to apply smoothing to autoregressive generative modeling.
Unlike Cohen et al.
(2019) which uses smoothing to smooth a given model, we apply smoothing directly to p data ( x ) to make the data distribution smoother.
To do this, we introduce a smoothing distribution q (˜ x | x ) — a distribution that is symmetric and stationary ( e.g . a Gaussian or Laplacian kernel), and convolve it with p data ( x ) to obtain a new distribution q (˜ x ) (cid:44) (cid:82) q
(˜ x | x ) p data ( x ) d x .
When q (˜ x | x ) is a normal distribution, this convolution process is equivalent to perturbing the data distribution with Gaussian noise, which, intuitively, will make the data distribution smoother.
Theorem 1.
Given a continuous 1-d distribution p ( x ) that is supported on R , for any 1-d distribution q (˜ x | x ) that is symmetric ( i.e . q (˜ x | x ) =
q ( x | ˜ x ) ), stationary ( i.e . translation invariant) and satisﬁes lim x →∞ p ( x ) q ( x | ˜ x )
=
for any given ˜ x , we have Lip ( q (˜ x ))
≤ Lip ( p data ( x )) , where q (˜ x ) (cid:44) (cid:82) q
(˜ x | x ) p ( x ) dx and Lip ( · ) denotes the Lipschitz constant of the given 1-d function.
Theorem 1 shows that convolving a 1-d data distribution p data ( x ) with certain choices of q (˜ x | x ) ( e.g .
N (˜ x | x, σ 2 ) ) can reduce the Lipschitzness ( i.e . increase the smoothness) of p data ( x ) .
We provide the proof of Theorem 1 in Appendix A.
To provide some intuition, we visualize a 1-d example in ﬁgure 3a, where we want to model a tenmode data distribution with a mixture of logistics model.
If our model has three logistic components, there is almost no way for the model, which only has three modes, to perfectly ﬁt this data distribution, which has ten separate modes with sharp transitions.
The model, after training (see ﬁgure 3a), mistakenly assigns a much higher density to the low density regions between nearby modes.
If we convolve the data distribution with q (˜ x | x ) =
N (˜ x | x, 0 .
5 2 ) , the new distribution becomes smoother than the original data distribution (see ﬁgure 3b) and can be captured reasonably well even by the same mixture of logistics model with only three modes (see ﬁgure 3b).
Comparing the same model’s performance on the two density estimation tasks, we can see that the model is doing a better job at modeling the smoothed version of the data distribution than the original data distribution, which has a high Lipschitz constant.
This smoothing process can also be understood as a regularization term for the original maximum likelihood objective (on the un-smoothed data distribution), encouraging the learned model to be smooth, as formalized by the following statement:
Proposition 1 (Informal) .
Assume that the symmetric and stationary smoothing distribution q (˜ x | x ) has small variance and negligible higher order moments, then
[Equation]
for some constant η .
Proposition 1 shows that our smoothing process provides a regularization effect on the original objective E p data ( x ) [log p θ ( x )] when no noise is added, where the regularization aims to maximize
[Equation]
Since the samples from p data should be close to a local maximum of the model, this encourages the second order gradients computed at a data point x to become closer to zero (if it were positive then x will not be a local maximum), creating a smoothing effect.
This extra term is also the trace of the score function (up to a multiplicative constant) that can be found in the score matching objective (Hyv¨arinen, 2005), which is closely related to many denoising methods (Vincent, 2011;
Hyv¨arinen, 2008).
This regularization effect can, intuitively, increase the generalization capability of the model.
In fact, it has been demonstrated empirically that training with noise can lead to improvements in network generalization (Sietsma & Dow, 1991; Bishop, 1995).
Our argument is also similar to that used in (Bishop, 1995) except that we consider a more general generative modeling case as opposed to supervised learning with squared error.
We provide the proof of Proposition 1 in
Appendix A.
3. A UTOREGRESSIVE DISTRIBUTION SMOOTHING MODELS
Motivated by the previous 1-d example, instead of directly modeling p data ( x ) , which can have a high
Lipschitz constant, we propose to ﬁrst train an autoregressive model on the smoothed version of the data distribution q (˜ x ) .
Although the smoothing process makes the distribution easier to learn, it also introduces bias.
Thus, we need an extra step to debias the learned distribution by reverting the smoothing process.
If our goal is to generate approximated samples for p data ( x ) , when q (˜ x | x ) =
N (˜ x | x , σ 2 I ) and σ is small we can directly apply “single-step denoising” as discussed in (Saremi et al., 2018).
More speciﬁcally, given smoothed samples ˜ x from p θ (˜ x ) , we can “denoise” samples via:
[Equation]
which only requires the knowledge of p θ (˜ x ) and the ability to sample from it.
However, this approach does not provide a likelihood estimation of p data ( x ) and Eq.
(2) only works when q (˜ x | x ) is
Gaussian (though alternative denoising updates for other smoothing processes could be derived under the Empirical Bayes framework (Raphan & Simoncelli, 2011)).
Although Eq.
(2) could provide reasonable denoising results when the smoothing distribution has a small variance, the obtained ¯ x is only an approximation of the denoised result given by ¯ x = E [ x | ˜ x ] and does not capture uncertainty in p ( x | ˜ x ) .
When the smoothing distribution has a large variance, such approximation might not perform well for a complicated data distribution.
To invert more general smoothing distributions (beyond Gaussians) as well as obtaining the likelihood estimation of p data ( x ) , we introduce a second autoregressive model p θ ( x | ˜ x ) .
The parameterized joint density p θ ( x , ˜ x ) can then be computed as p θ ( x , ˜ x )
= p θ ( x | ˜ x ) p θ (˜ x ) .
To obtain our approximation of p data ( x ) , we need to integrate over ˜ x on the joint distribution p θ ( x , ˜ x ) to obtain p θ ( x ) = (cid:82) p θ ( x , ˜ x )
d ˜ x , which is in general intractable.
However, we can easily obtain an evidence lower bound (ELBO):
[Equation]
Note that when q (˜ x | x ) is ﬁxed, the entropy term E p data ( x ) E q (˜ x | x ) [log q (˜ x | x )] is a constant with respect to the optimization parameters.
Maximizing the ELBO is then equivalent to maximizing:
[Equation]
From equation 4, we can see that optimizing J ( θ ) is equivalent to optimizing the two models p θ (˜ x ) and p θ ( x | ˜ x ) separately via maximum likelihood estimation.
3. T RADEOFF IN MODELING
In general, there is a trade-off between the difﬁculty of modeling p θ (˜ x ) and p θ ( x | ˜ x ) .
To see this, let us consider two extreme cases for the variance of q (˜ x | x ) — when q (˜ x | x ) has a zero variance and an inﬁnite variance.
In the ﬁrst case, when q (˜ x | x ) has a zero variance, q (˜ x | x ) is a distribution with all its mass at x , meaning that no noise is added to the data distribution.
In this case, modeling the new distribution would be equivalent to modeling p data ( x ) , which can be hard as discussed above.
The reverse smoothing process, however, would be easy since p θ ( x | ˜ x ) can simply be an identity map to perfectly invert the smoothing process.
In the second case when q (˜ x | x ) has an inﬁnite variance, modeling p (˜ x ) would be easy because all the information about the original data is lost, and p (˜ x ) would be close to the smoothing distribution.
Modeling p ( x | ˜ x ) , on the other hand, is equivalent to directly modeling p data ( x ) , which can be challenging.
In comparison only 2 mixture of components
Thus, the key here is to appropriately choose a smoothing level so that both q (˜ x ) and p ( x | ˜ x ) can be approximated relatively well by existing autoregressive models.
In general, the optimal variance might be hard to ﬁnd.
Although one can train q (˜ x | x ) by jointly optimizing ELBO, in practice, we ﬁnd this approach often assigns a very large variance to q (˜ x | x ) , which can trade-off sample quality for better likelihoods on high dimensional image datasets.
We ﬁnd empirically that a pre-speciﬁed q (˜ x | x ) chosen by heuristics (Saremi & Hyvarinen, 2019; Garreau et al., 2017) is able to generate much better samples than training q (˜ x | x ) via ELBO.
In this paper, we will focus on the sample quality and leave the training of q (˜ x | x ) for future work.
E XPERIMENTS
In this section, we demonstrate empirically that by appropriately choosing the smoothness level of randomized smoothing, our approach is able to drastically improve the sample quality of existing autoregressive models on several synthetic and real-world datasets while retaining competitive likelihoods on synthetic datasets.
4. C HOOSING THE SMOOTHING DISTRIBUTION
To help us build insights into the selection of the smoothing distribution q (˜ x | x ) , we ﬁrst focus on a 1-d multi-modal distribution (see ﬁgure 4 leftmost panel).
We use model-based methods to invert the smoothed distribution and provide analysis on “single-step denoising” in Appendix B.1.
We start with the exploration of three different types of smoothing distribution – Gaussian distribution,
Laplace distribution, and uniform distribution.
For each type of distribution, we perform a grid search to ﬁnd the optimal variance.
Since our approach requires the modeling of both p θ (˜ x ) and p θ ( x | ˜ x ) , we stack ˜ x and x together, and use a MADE model (Germain et al., 2015) with a mixture of two logistic components to parameterize p θ (˜ x ) and p θ ( x | ˜ x ) at the same time.
For the baseline model, we train a mixture of logistics model directly on p data ( x ) .
We compare the results in the middle two panels in ﬁgure 4.
We ﬁnd that although the baseline with eight logistic components has the capacity to perfectly model the multi-modal data distribution, which has six modes, the baseline model still fails to do so.
We believe this can be caused by optimization or initialization issues for modeling a distribution with a high Lipschitz constant.
Our method, on the other hand, demonstrates more robustness by successfully modeling the different modes in the data distribution even when using only two mixture components for both p θ (˜ x ) and p θ ( x | ˜ x ) .
For all the three types of smoothing distributions, we observe a reverse U-shape correlation between the variance of q (˜ x | x ) and ELBO — with ELBO ﬁrst increasing as the variance increases and then decreasing as the variance reaches beyond a certain point.
The results match our discussion on the trade-off between modeling p θ (˜ x ) and p θ ( x | ˜ x ) in Section 3.3.
We notice from the empirical results that Gaussian smoothing is able to obtain better ELBO than the other two distributions.
Thus, we will use q (˜ x | x ) =
N (˜ x | x , σ 2 I ) for the later experiments.
[Table]
4. 2-D SYNTHETIC DATASETS
In this section, we consider two challenging 2-d multi-modal synthetic datasets (see ﬁgure 5).
We focus on model-based denoising methods and present discussion on “single-step denoising” in Appendix B.2.
We use a MADE model with comparable number of total parameters for both the baseline and our approach.
For the baseline, we train the MADE model directly on the data.
For our randomized smoothing model, we choose q (˜ x | x ) =
N (˜ x | x , 0 .
3 2 I ) to be the smoothing distribution.
We observe that with this randomized smoothing approach, our model is able to generate
[Figure] [Figure]
modeled conditioned on labels better samples than the baseline (according to a human observer) even when using less logistic components (see ﬁgure 5).
We provide more analysis on the model’s performance in Appendix B.2.
We also provide the negative log-likelihoods in Tab. 1.
4. I MAGE EXPERIMENTS
In this section, we focus on three common image datasets, namely MNIST, CIFAR-10 (Krizhevsky et al., 2009) and CelebA (Liu et al., 2015).
We select q (˜ x | x ) =
N (˜ x | x , σ 2 I ) to be the smoothing distribution.
We use PixelCNN++ (Salimans et al., 2017) as the model architecture for both p θ (˜ x ) and p θ ( x | ˜ x ) .
We provide more details about settings in Appendix C.
Image generation.
For image datasets, we select the σ of q (˜ x | x ) =
N (˜ x | x , σ 2 I ) according to analysis in (Saremi & Hyvarinen, 2019)
(see Appendix C for more details).
Since q ( ˜ x | x ) is a
[Table]
Gaussian distribution, we can apply “single-step denoising” to reverse the smoothing process for samples drawn from p θ ( ˜ x ) .
In this case, the model p θ ( x | ˜ x ) is not required for sampling since the gradient of p θ (˜ x ) can be used to reverse the smoothed samples drawn from itself ( i
[Table]
e .
p θ (˜ x ) )
(see equation 2).
We present smoothed samples from p θ (˜ x ) , reversed smoothing samples processed by “single-step denoising” and processed by p θ ( x | ˜ x ) in ﬁgure 6.
For comparison, we also present samples from a PixelCNN++ with parameters comparable to the sum of total parameters of p θ (˜ x ) and p θ ( x | ˜ x ) .
We ﬁnd that by using this randomized smoothing approach, we are able to drastically improve the sample quality of PixelCNN++ (see the rightmost panel in ﬁgure 6).
We note that with only p θ (˜ x ) , a PixelCNN++ optimized on the smoothed data, we are already able to obtain more image-like samples compared to the original PixelCNN++ method.
However, p θ (˜ x ) alone does not provide likelihood estimation of p data ( x ) and p θ ( x | ˜ x ) is required for the computation of likelihoods.
We report the sample quality evaluated by FID and Inception scores in Tab. 2.
We provide more image samples in Appendix C.4 and nearest neighbors analysis in Appendix C.5.
Although our method is able to obtain better samples compared to the original PixelCNN++, our model has worse likelihoods.
We believe this is because likelihood and sample quality are not always directly correlated as discussed in Theis et al.
We also tried training the variance for q (˜ x | x ) by jointly optimizing ELBO.
Although training the variance can produce better likelihoods, it does not generate samples with comparable quality as our method ( i.e . choosing variance by heuristics).
Thus, it is hard to conclusively determine what is the best way of choosing q (˜ x | x ) .
A DDITIONAL EXPERIMENTS ON NORMALIZING FLOWS
In this section, we demonstrate empirically on 2-d synthetic datasets that randomized smoothing techniques can also be applied to improve the sample quality of normalizing ﬂow models (Rezende & Mohamed, 2015).
We focus on RealNVP (Dinh et al., 2016).
We compare the RealNVP model trained with randomize smoothing, where we use p θ ( x | ˜ x ) (also a RealNVP) to revert the smoothing process, with a RealNVP trained with the original method but with comparable number of parameters.
We observe that smoothing is able to improve sample quality on the datasets we considered (see ﬁgure 7) while also obtaining competitive likelihoods.
On the checkerboard dataset, our method has negative log-likelihoods 3.64 while the original RealNVP has 3.72; on the Olympics dataset, our method has negative log-likelihoods 1.32 while the original RealNVP has 1.80.
This example demonstrates that randomized smoothing techniques can also be applied to normalizing ﬂow models.
R ELATED W ORK
Our approach shares some similarities with denoising autoencoders (DAE, Vincent et al.
(2008)) which recovers a clean observation from a corrupted one.
However, unlike DAE which has a trainable encoder and a ﬁxed prior distribution, our approach ﬁxes the encoder and models the prior using an autoregressive model.
Generative stochastic networks (GSN, Bengio et al.
(2014)) use DAEs to train a Markov chain whose equilibrium distribution matches the data distribution.
However, GSN
[Figure]
needs to start the chain from a sample that is very close to the training distribution.
Denoising diffusion model (Sohl-Dickstein et al., 2015; Ho et al., 2020) and NCSN (Song & Ermon (2019; 2020)) address the issue of GSNs by considering a sequence of distributions corresponding to data corrupted with various noise levels.
By setting multiple noise levels that are close to each other, the sample from the previous level can serve as a proper initialization for the sampling process at the next level.
This way, the model can start from a distribution that is easy to model and gradually move to the desired distribution.
However, due to the large number of noise levels, such approaches require many steps for the chain to converge to the right data distribution.
In this paper, we instead propose to use only one level of smoothing by modeling each step with a powerful autoregressive model instead of deterministic autoencoders.
Motivated by the success of “randomized smoothing” techniques in adversarial defense (Cohen et al., 2019), we perform randomized smoothing directly on the data distribution.
Unlike denoising score matching (Vincent, 2011), a technique closely related to denoising diffusion models and NCSN, which requires the perturbed noise to be a Gaussian distribution, we are able to work with different noise distributions.
Our smoothing method is also relevant to “dequantization” approaches that are common in normalizing ﬂow models, where the discrete data distribution is converted to a continuous one by adding continuous noise (Uria et al., 2013; Ho et al., 2019).
However the added noise for “dequantization” in ﬂows is often indistinguishable to human eyes, and the reverse “dequantization” process is often ignored.
In contrast, we consider noise scales that are signiﬁcantly larger and thus a denoising process is required.
Our method is also related to “quantization” approaches which reduce the number of “signiﬁcant” bits that are modeled by a generative model (Kingma & Dhariwal, 2018; Menick & Kalchbrenner, 2018).
For instance, Glow (Kingma & Dhariwal, 2018) only models the 5 most signiﬁcant bits of an image, which improves the visual quality of samples but decreases color ﬁdelity.
SPN (Menick & Kalchbrenner, 2018) introduces another network to predict the remaining bits conditioned on the 3 most signiﬁcant bits already modeled.
Modeling the most signiﬁcant bits can be understood as capturing a data distribution perturbed by bit-wise correlated noise, similar to modeling smoothed data in our method.
Modeling the remaining bits conditioned on the most signiﬁcant ones in SPN is then similar to denoising.
However, unlike these quantization approaches which process an image at the “signiﬁcant” bits level, we apply continuous data independent Gaussian noise to the entire image with a different motivation to smooth the data density function.
D ISCUSSION
In this paper, we propose to incorporate randomized smoothing techniques into autoregressive modeling.
By choosing the smoothness level appropriately, this seemingly simple approach is able to drastically improve the sample quality of existing autoregressive models on several synthetic and real-world datasets while retaining reasonable likelihoods.
Our work provides insights into how recent adversarial defense techniques can be leveraged to building more robust generative models.
Since we apply randomized smoothing technique directly to the target data distribution other than the model, we believe our approach is also applicable to other generative models such as normalizing ﬂows, variational autoencoders and GANs.
A P ROOFS
Theorem 1.
Given a continuous 1-d distribution p ( x ) that is supported on R , for any 1-d distribution q (˜ x | x ) that is symmetric ( i.e . q (˜ x | x ) =
q ( x | ˜ x ) ), stationary ( i.e . translation invariant) and satisﬁes lim x →∞ p ( x ) q ( x | ˜ x )
=
for any given ˜ x , we have Lip ( q (˜ x ))
≤ Lip ( p data ( x )) , where q (˜ x ) (cid:44) (cid:82) q
(˜ x | x ) p ( x ) dx and Lip ( · ) denotes the Lipschitz constant of the given 1-d function.
Proof.
First, we have that:
[Equation]
and if we assume symmetry, i.e. q ( x | ˜ x )
=
q (˜ x | x ) then by integration by parts we have:
[Equation]
Therefore,
[Equation]
which proves the result.
Proposition 1 (Informal) .
Assume that the symmetric and stationary smoothing distribution q (˜ x | x ) has small variance and negligible higher order moments, then
[Equation]
for some constant η .
Proof.
To see this
[Equation]
we ﬁrst note that the new training objective for the smoothed data distribution is E p data E q
(˜ x | x ) [log p θ (˜ x )] .
Since q (˜ x | x ) is Gaussian
[Equation]
we can write q (˜ x | x ) = p ( (cid:15) ) for some distribution p , where (cid:15) =
˜ x − x .
Then we have
[Equation]
Using Taylor expansion
[Equation]
we have:
[Equation]
Since that (cid:15) is independent of x and
[Equation]
for some constant η , where δ i,j is the Kronecker delta function, Equation 9 becomes
[Equation]
where the second term serves as a regularization for the original objective E p data [log p θ ( x )] .
B D ENOISING EXPERIMENTS B. A NALYSIS ON 1- D DENOISING
To provide more insights into denoising, we ﬁrst study “single-step denoising” (see equation 2) on a 1-d dataset.
We choose the data distribution to be a two mixture of Gaussian distribution 0 .
5 N ( − 0 .
1 2 ) + 0 .
5 N (0 . 3 , 0 .
1 2 ) and the smoothing distribution to be q (˜ x | x ) =
N (˜ x | x, 0 .
3 2 ) (see ﬁgure 8a).
Since the convolution of two Gaussian distributions is also a Gaussian distribution, the smoothed data is a mixture of Gaussian distribution given by 0 .
5 N ( − 0 .
1 2 + 0 .
3 2 ) + 0 .
5 N (0 . 3 , 0 .
1 2 + 0 .
The ground truth of ∇ ˜ x log p (˜ x ) can then be calculated in closed form.
Thus, given the smoothed data ˜ x , we can calculate the ground truth ∇ ˜ x log p (˜ x ) in equation 2 and obtain ¯ x using “single-step denoising”.
We visualize the denoising results in ﬁgure 8b.
We ﬁnd that the low density region between the two modes in p data ( x ) are not modeled properly in ﬁgure 8b.
However, this is very expected since “single-step denosing” uses ¯ x = E [ x | ˜ x ] as the substitute for the denoised result.
When the smoothing distribution has a large variance (like in ﬁgure 8a where the smoothed data has merged into a one mode distribution), datapoints like ˜ x 0 in the middle low density region of p data ( x ) can have high density in the smoothed distribution.
Since ˜ x 0 , as well as other points in the middle low density region of p data ( x ) , can come from both modes of p data ( x ) with high probability before the smoothing process (see ﬁgure 11a), the denoised ¯ x = E [ x | ˜ x = ˜ x 0 ] can still be located in the middle low density region (see ﬁgure 8b).
Since a large proportion of the smoothed data is located in the middle low density region of p data ( x ) , we would expect certain proportion of the density to remain in the low density region after “single-step denoising” just as shown in ﬁgure 8b.
However, when the smoothing distribution has a smaller variance, “single-step denoising” can achieve much better denoising results (see ﬁgure 9, where we use q (˜ x | x ) =
N (˜ x | x, 0 .
Although denoising can be easier when the smoothing distribution has a smaller variance, modeling the smoothed distribution could be harder as we discussed before.
In general, the right denoising results should be samples coming from p ( x | ˜ x ) , which is the reason why samples from p θ ( x | ˜ x ) ( i.e . introducing the model p θ ( x | ˜ x ) ) is more ideal than using E θ
[ x | ˜ x ] as a denoising substitute ( i.e . “single-step denoising”).
In general, the capacity of the denoising model p θ ( x | ˜ x ) also matters in terms of denoising results.
Let us again consider the datapoint ˜ x
0 shown in ﬁgure 8a.
If the invert smoothing model p θ ( x | ˜ x ) is a one mode logistic distribution, due to the mode covering property of maximum likelihood estimation, given the smoothed observation ˜ x 0 , the best the model can do is to center its only mode at ˜ x 0 for approximating p ( x | ˜ x
= ˜ x 0 ) (see ﬁgure 10c).
Thus, like ˜ x 0 , the smoothed datapoints at the low density region between the two modes of p data ( x ) are still likely to remain between the two modes after denoising (see ﬁgure 10b).
To solve this issue, we can increase the capacity of p θ ( x | ˜ x ) by making it a two mixture of logistics.
In this case, the distribution p θ ( x | ˜ x
= ˜ x 0 ) can be captured in a better way (see ﬁgure 11c and ﬁgure 11a).
After the invert smoothing process, like ˜ x 0 , most smoothed datapoints in the low density can be mapped to one of the two high density modes (see ﬁgure 11b), resulting in much better denoising effects.
[Figure] [Figure]
[Figure] [Figure]
B. A NALYSIS ON 2- D DENOISING
On the 2-d Olympics dataset in section 4. we ﬁnd that the intersections between rings can be poorly modeled with the proposed smoothing approach when only two mixture of logistics are used (see ﬁgure 12e).
We believe this can be caused if the denoising model is not ﬂexible enough to capture the distribution p ( x | ˜ x ) .
More speciﬁcally, we note that the ground truth distribution for p ( x | ˜ x ) at the intersections of the rings is a highly complicated distribution and can be hard to capture using our model which only has two mixtures of logistics for each dimension.
If we increase the ﬂexibility of p θ ( x | ˜ x ) by using three or four mixtures of logistics components (note that we still use fewer mixture components than the MADE baseline and we use comparable number of parameters), the intersection of the rings can be modeled in an improved way (see ﬁgure 12).
We also provide “single-step denoising” results for the experiments in Section 4.2 (see ﬁgure 13), where we use the same smoothing distribution, and the MADE model with three mixture components as used in section 4.2.
We note that “single-step denoising” results are not very good, which is also expected.
As discussed in section B.1, when the smoothing distribution has a relatively
[Figure]
[Figure]
large variance, E θ [ x | ˜ x ] is not a good approximation for the denoised result, and we want the denoised sample to come from the distribution p θ ( x | ˜ x ) , in which case introducing a denoising model p θ ( x | ˜ x ) could be a better option.
Although we could select q (˜ x | x ) to have a smaller variance so that “single-step denoing” could work reasonably well, but modeling p (˜ x ) in this case could be more challenging.
C I MAGE EXPERIMENTS C. S ETTINGS
For the image experiments, we ﬁrst rescale images to [ − 1 , 1] and then perturb the images with q (˜ x | x ) =
N (˜ x | x , σ 2 I ) .
We use σ = 0 .
5 for MNIST and σ = 0 .
3 for both CIFAR-10 and CelebA.
The selection of σ is mainly based on analysis in (Saremi & Hyvarinen, 2019).
More speciﬁcally, given an image, we consider the median value of the Euclidean distance between two data points in a dataset, and then divide it by 2 √ D , where D is the dimension of the data.
This provides us with a way of selecting the variance of q (˜ x | x ) ,
when q (˜ x | x ) is a Gaussian distribution.
We ﬁnd this selection of variance able to generate reasonably well samples in practice.
We train all the models with Adam optimizer with learning rate 0 . 0002 .
To model p θ ( x | ˜ x ) , we stack ˜ x and x together at the second dimension to obtain ˆ x =
[˜ x , x ] , which ensures that ˜ x comes before x in the pixel ordering.
For instance, this stacking would provide an image ˆ x with size 1 × (2 × 28) ×on a MNIST image, and an image with size 3 × (2 × 32) × 32 on a CIFAR-10 image.
Since
PixelCNN++ consists of convolutional layers, we can directly feed ˆ x into the default architecture without modifying the model architecture.
As the latter pixels of the input only depend on the previous pixels in an autoregressive model and ˜ x comes before x , we can parameterize p θ ( x | ˜ x ) by computing the likelihoods only on x using the outputs from the autoregressive model.
C. I MAGE I NPAINTING
Since both p θ (˜ x ) and p θ ( x | ˜ x ) are parameterized by an autoregressive model, we can also perform image inpainting using our method.
We present the inpainting results on CIFAR-10 in ﬁgure 14a and CelebA in ﬁgure 14b, where the bottom half of the input image is being inpainted.
[Figure]
C. I MAGE D ENOISING
We notice that the reverse smoothing process can also be understood as a denoising process.
Besides the “single-step denoising” approach shown above, we can also apply p θ ( x | ˜ x ) to denoise images.
To visualize the denoising performance, we sample x test from the test set and perturb x test with q (˜ x | x ) to obtain a noisy sample ˜ x test .
We feed ˜ x test into p θ ( x | ˜ x = ˜ x test ) and draw samples from the model.
We visualize the results in ﬁgure 15.
As we can see, the model exhibits reasonable denoising results, which shows that the autoregressive model is capable of learning the data distribution when conditioned on the smoothed data.
[Figure] [Figure]
C. N EAREST NEIGHBORS
C. A BLATION STUDIES
In this section, we show that gradient-based “single-step denoising” will not improve sample qualities without performing randomized smoothing.
To see this, we draw samples from a PixelCNN++ p θ ( x ) trained directly on p data ( x ) ( i.e . without smoothing).
We perform “single-step denoising” update deﬁned as
[Equation]
We explore various values for σ , and report the results in ﬁgure 26.
This shows that “single-step denoising” alone (without randomized smoothing) will not improve sample quality of PixelCNN++.
[Figure]