E XPLORING THE C ORRELATION BETWEEN L IKELIHOOD OF F LOW - BASED G ENERATIVE M ODELS AND I MAGE S EMANTICS
Anonymous authors
Paper under double-blind review
A BSTRACT
Among deep generative models, ﬂow-based models, simply referred as ﬂow s in this paper, differ from other models in that they provide tractable likelihood.
Besides being an evaluation metric of synthesized data, ﬂows are supposed to be robust against out-of-distribution (OoD) inputs since they do not discard any information of the inputs.
However, it has been observed that ﬂows trained on FashionMNIST assign higher likelihoods to OoD samples from MNIST.
This counterintuitive observation raises the concern about the robustness of ﬂows’ likelihood.
In this paper, we explore the correlation between ﬂows’ likelihood and image semantics.
We choose two typical ﬂows as the target models: Glow, based on coupling transformations, and pixelCNN, based on autoregressive transformations.
Our experiments reveal surprisingly weak correlation between ﬂows’ likelihoods and image semantics: the predictive likelihoods of ﬂows can be heavily affected by trivial transformations that keep the image semantics unchanged, which we call semantic-invariant transformations (SITs).
We explore three SITs (all small pixel-level modiﬁcations): image pixel translation, random noise perturbation, latent factors zeroing (limited to ﬂows using multi-scale architecture, e.g. Glow).
These ﬁndings, though counter-intuitive, resonate with the fact that the predictive likelihood of a ﬂow is the joint probability of all the image pixels.
So ﬂows’ likelihoods, modeling on pixel-level intensities, is not able to indicate the existence likelihood of the high-level image semantics.
We call for attention that it may be abuse if we use the predictive likelihoods of ﬂows for OoD samples detection.
I NTRODUCTION
Deep generative models have been very successful in image generation (Brock et al., 2018; Kingma & Dhariwal, 2018;
Miyato et al., 2018), natural language generation (Bowman et al., 2015; Yu et al., 2017), audio synthesis(Van Den Oord et al., 2016) and so on.
Among them, generative adversarial networks (GANs) are implicit generative models(Goodfellow et al., 2014) that explicit likelihood function is not required, and are trained by playing a minimax game between the discriminator and the generator; Variational auto-encoders (VAEs,Kingma & Welling (2013); Rezende et al.
(2014)) are latent variable generative models optimized by maximizing a lower bound, called evidence lower bound, of the data log-likelihood.
Flow-based models (Dinh et al., 2016; 2014; van den Oord et al., 2016) differ from them in that they provide exact log-likelihood evaluation with change of variables theorem (Rezende & Mohamed, 2015).
A ﬂow usually starts with a simple base probability distribution, e.g. diagonal Gaussian, then follows a chain of transformations in order to approximate complex distributions.
Each transformation is parameterized by specially designed neural networks so that the log-determinant of its Jacobian can be efﬁciently computed.
Most of the previous works focus on how to design more ﬂexible transformations to achieve tighter log-likelihoods, and generate more realistic samples.
It is also believed that ﬂows can be used to detect out-of-distribution(OoD) samples by assigning low likelihoods on them.
However, it has been observed that ﬂows fail to do so.
For example, ﬂows trained on FashionMNIST surprisingly assign higher likelihoods on MNIST samples (Nalisnick et al., 2018; Choi & Jang, 2018).
Though analyses on pixel-level statistics are performed on this phenomenon (Nalisnick et al., 2018), and density evaluation combined with uncertainty estimation is used to detect OoD samples (Choi &
Jang, 2018), the reasons behind ﬂows’ counter-intuitive behaviours are still not clear.
Humans easily discriminate MNIST images from FashionMNIST images, since their high-level image semantics are perceptually different.
Accordingly, it takes some metrics that can reﬂect the high-level image semantics for OoD detection.
In this paper, we empirically explore the correlation between ﬂows’ likelihoods and image semantics, and question the rationality and applicability of using predictive likelihoods of ﬂows for OoD detection.
We ﬁrst introduce a concept of semanticinvariant transformation (SIT).
An SIT transforms an input without changing its high-level semantics, e.g. a dog image through an SIT is still supposed to be recognized as a dog.
We choose two typical ﬂow-based models as target models: Glow (Kingma & Dhariwal, 2018), based on coupling transformations, and pixelCNN (van den Oord et al., 2016), based on autoregressive transformations.
We evaluate on image datasets MNIST and FashionMNIST under three trivial SITs: image translation, random noise perturbation, and latent factors zeroing (speciﬁc to invertible ﬂows using multi-scale architectures, e.g. Glow).
We demonstrate that the predictive likelihoods of the target models show weak correlation to the image semantics in the following ways:
• Small pixel translations of test images could result in obvious likelihood decreases of Glow.
• Perturbing small random noises, unnoticeable to humans, to test images could lead to catastrophic likelihood decreases of target models.
This also applies even if we keep the semantic object of a test image intact, and only add noises to the background.
• For an invertible ﬂow using multi-scale architecture, e.g. Glow, the inferred latent variables of an image is a list of gaussianized and standardized factors.
We ﬁnd that the contributions of a ﬂow’s blocks to the log-likelihood are constant and independent of inputs.
Thus, simply zeroing the
preceding latent factors of a sample image, and feed them to ﬂow’s reverse function.
We could obtain new samples with surprisingly higher likelihoods, yet with perceptually unnoticeable changes from the original image.
We emphasize that all these SITs are small pixel-level modiﬁcations on test images, and undoubtedly have no inﬂuences on humans’ recognition of the semantic objects in the images.
However, they lead to obvious inconsistency of ﬂows’ likelihoods on test samples.
Considering that the predictive likelihood of a ﬂow is the joint probability of all the image pixels, it may not convincingly indicate the existence of a semantic object in an image.
Thus it could be problematic to use ﬂows for downstream tasks which require metrics that can reﬂect image semantics, e.g. OoD detection.
B ACKGROUND 2. C HANGE OF V ARIABLES T HEOREM
Given a random variable z with probability density function p ( z ) , after applying an invertible function f : R D → R D on z , we get a new random variable z (cid:48)
=
f ( z ) .
Then probability density function of the changed variable z (cid:48) is given by:
[Equation]
We can construct arbitrarily complex probability distributions by transforming a simple base distribution p ( z 0 ) with a chain of mappings f k of length K .
Then we have :
[Equation]
2. F LOW - BASED M ODELS
Flow-based models are generative models designed by applying the above theorem, thus exact loglikelihood evaluation of data is feasible.
Then the practical problem of building ﬂow-based models applied on high-dimensional data, like images, becomes how to design invertible transformations whose Jacobian determinant can be efﬁciently computed.
Research on ﬂows is very active and rapidly evolving.
In this paper, we particularly focus on ﬂowbased generative models on images and the behaviours of their likelihoods.
They can roughly be divided into two categories according to the granularity of the transformation layers:
Coupling Flow A coupling ﬂow contains a sequence of coupling layers which model the transformation in a coarse way.
Coupling layers split a D -dimensional intermediate random variable z into two parts: z 1: d , z d : D .
A general form of coupling layer is afﬁne(Eq. 3).
The ﬁrst part z 1: d is kept still, while the second part x d : D is scaled and shifted with transformations s, t on the ﬁrst part x 1: d .
[Equation]
The afﬁne coupling is proposed in Real NVP (Dinh et al., 2016), whose Jacobian is a lower triangular matrix that can be efﬁciently computed.
An earlier and simpler version is additive coupling proposed in NICE (Dinh et al., 2014), which can be obtained by simply removing the scale item exp s ( x 1: d ) in afﬁne coupling.
Additive coupling layer is volume-preserving and the log-determinant of its
Jacobian is always 0.
Glow improves Real NVP by replacing the ﬁxed shufﬂing permutation with 1 × 1 invertible convolution.
Since forward and inverse operation of a coupling layer have the same computational efﬁciency, both likelihood evaluation and sampling(or generation) for coupling ﬂows are equally efﬁcient.
Autoregressive Flow
As the building blocks of autoregressive ﬂow, autoregressive transformations model the joint probability p ( x ) as the product of one-dimensional conditionals:
[Equation]
where the probability of observation x i conditions only on its previous observations x <i .
The autoregressive property of an autoregressive layer is enforced by specially designed mechanism, e.g. masking.
In PixelCNN (van den Oord et al., 2016), this is implemented as masked convolutional layers, which are inherently easier to be parallelized than its counterpart PixelRNN (Oord et al., 2016).
The likelihood evaluation of PixelCNN takes only one-forward pass, but its inference, i.e. generation, takes O ( D ) , since we have to sample pixel-by-pixel.
PixelCNN can be further parallelized (Reed et al., 2017b) to accelerate its inference speed.
O THER R ELATED W ORKS
Flows for generation Serving as powerful decoders, PixelCNN can also be combined with other generative models, e.g. PixelGAN (Makhzani & Frey, 2017) and PixelVAE (Gulrajani et al., 2016).
Variants of PixelCNN are also used to model audio (Van Den Oord et al., 2016), video (Kalchbrenner et al., 2017), and text (Kalchbrenner et al., 2016).
PixelCNN, combining with attention, is also applied to few-shot autoregressive density estimation Reed et al.
(2017a).
Ho et al.
(2019) proposes several improvements to coupling ﬂow, reducing its gap to autoregressive ﬂow in terms of density estimation.
Autoregressive models for density estimation Autoregressive models can be specially designed for general-purpose density estimation.
Masked Autoencoder for Distribution Estimation (MADE,
Germain et al. (2015)) is a pioneering work that use masked neural networks to model the autoregressive density.
MADE constitutes the building block of two popular normalizing ﬂows : Inverse
Autoregressive Flow (IAF, Kingma et al.
and Masked Autoregressive Flow (MAF, Papamakarios et al.
IAF and MAF are similar but with different computational trade-offs.
IAF, providing efﬁcient sampling, is designed to improve the expressiveness of the approximate posterior of VAE.
MAF is a more powerful density estimator which stacks multiple MADEs.
M ETHOD
Let g be a ﬂow-based generative model trained on dataset X
= { x i } Ni =1 sampled from some unknown p ( x ) , and p g ( x ) be the g ’s predictive probability density function of sample x .
Semantic-Invariant Transformation (SIT)
Roughly speaking, SIT can be any transformation that do not change humans’ recognition of image semantics.
For example, suppose x is a dog image.
After applying SIT T to x , T ( x ) is supposed to be high recognized as a dog image.
As a proof of concept evaluation, we limit our evaluations to three trivial SITs: image translation, random noise perturbation, and latent factors zeroing (speciﬁc to Glow).
Semantic Correlation We probe the correlation between the predictive likelihood of a ﬂow g and image semantics by examining the inﬂuences of SITs on test samples’ likelihoods.
Speciﬁcally, a reasonable observation we should expect is that for SIT T : (cid:12)(cid:12) p g ( T ( x ))
− p g
( x ) (cid:12)(cid:12) <
δ holds for a small positive scalar δ .
We report bits-per-dim (BPD), which is given by:
[Equation]
where NLL is the negative log-likelihood of the test sample, h, w, c are height, width, number of channels.
Lower BPD implies higher likelihood.
Throughout this paper, we use BPD and likelihood interchangeably.
We refer to Supp.
A for setup and training details of the target models.
4. I MAGE T RANSLATION
Translation invariance is a fundamental property in learning image representations that are robust for downstream tasks.
In this section, we evaluate the inﬂuences of image translations on ﬂows’ likelihoods.
The results in Fig.
1 and examples in Fig.
2 show that even 1 or 2-pixel left translation could lead to obvious increase of Glow’s predictive BPDs, while the PixelCNN’s predictive BPDs are robust to pixel translation.
This surprising difference can be attributed to the difference of their architectures.
Glow, like other ﬂows based on coupling transformation layers (Dinh et al., 2016), models the joint probability of the pixels in a coarse-grained way.
They rely on multi-scale architecture modeling different level of abstractions in order to achieve competitive BPDs.
At higher scale levels, the intermediate tensors have smaller spatial sizes and bigger channel sizes.
This is performed at the starting point of each scale level with squeeze operation, which trades spatial sizes for channel sizes by transforming a h × w × c tensor into a h/ 2 × w/ 2 × 4 c tensor.
Note that squeeze operation actually destroys the spatial positions of adjacent pixels, and 1-pixel translation could lead to quite different spatial partitions.
While for PixelCNN, the intermediate tensors are not reshaped, and the spatial positions of pixels are kept still; Furthermore, the prediction of each pixel conditions only on a neighborhood of (previous) pixels in masked convolution, so translation invariance is preserved.
original 1p-left 2p-left original 1p-left 2p-left
Afﬁne 0.
Additive 0.
PixelCNN 0.
Afﬁne 1.
Additive 1.
PixelCNN 1.
Below are predictive BPDs of Glows (Afﬁne and Additive) and PixelCNN.
Problematic Likelihood Comparisons
The foundation of using likelihood-based models for OoD detection is that they are supposed to assign much lower likelihoods for OoD samples x out than indistribution samples x in , i.e. p g ( x in ) (cid:29) p g ( x out ) .
However, it has been observed that ﬂows assign higher likelihoods on OoD samples than even training samples (Nalisnick et al., 2018; Choi & Jang, 2018).
Analyses of pixel-level datasets statistics in (Nalisnick et al., 2018) show that this may be due to OoD datasets just “sit inside of” in-distribution datasets with roughly the same mean, smaller variance.
Surprisingly, similar counter-intuitive likelihood assignment also occurs in in-distribution samples.
For example, in Fig. 1, images with class label 1 are consistently have signiﬁcantly lower
BPDs, i.e. higher likelihoods than samples of other classes.
In OoD detection, we assume that a sample with a higher likelihood indicates that it is more likely to be an in-distribution sample.
Following the same logic, this tells that all images of class label 1 are more likely to be in-distribution samples than samples from other classes, which contradicts the fact that they are all in-distribution samples for sure.
We may reasonably suspect that ﬂows’ counter-intuitive likelihood assignment is dominated by the inherent differences of pixel-level statistics associated to the image semantics, e.g. different numbers.
This kind of counter-intuitive likelihood comparisons exist not only between in-distribution and OoD samples, but also within in-distribution samples from different classes.
Similarly, Theis et al.
ﬁnd that 1-pixel shift of the images could lead to quite different nearest neighbours from the training set, measured in Euclidean distance, which also demonstrate the gap between pixel-level metrics and humans’ perception.
4. R ANDOM N OISE P ERTURBATION
Image pixels are discrete integers; and in practice, right amount of real-valued uniform noise is added to dequantize the pixels.
For images x ∈ [0 , . . .
255 / 256] D scaled to [0 , 1] , we usually do x = x + u , u ∈ [0 , 1 / 256] D .
In our experiments, we ﬁnd that adding small random perturbations out of the coverage of the added noise, i.e. u ∈ [0 , 1 / 256] D here, to test images cause ﬂows to give catastrophically higher BPDs. perturbed/- noise original masked noise perturbed/+ Glow 2. PixelCNN
[Table] [Table]
Glow
[Table]
[Table]
[Table]
PixelCNN
[Table] [Table] [Table] Humans can robustly recognize semantic objects in images regardless of the backgrounds.
So we also evaluate inﬂuences of adding random perturbations to only the backgrounds of test images.
This can be simply implemented using proper mask:
[Equation]
6) where m is the mask, and (cid:15) is a small scaling factor ensuring the noise is small enough.
In our evaluations, we use unit Gaussian noises, and set the scaling factor (cid:15)
= 0 .
Examples in
Fig.
3 manifest that adding small noises catastrophically lower samples’ likelihoods.
Compared to
Glow, PixelCNN is more sensitive to the noises, because its pixel-wise modeling quickly augment and propagate the inﬂuences of the added noise.
We get similar results even if we keep the semantic objects of test images intact, and add noises only to the backgrounds.
Note that the Gaussian noises (cid:15) · N (0 , 1) we added are out of the coverage (with < 0 elements) of the uniform noises [0 , 1 / 256] added during training, so theoretically this is expected since models are not optimized in that areas.
However, it does reveal that ﬂows are not aware of the image semantics, and treat the pixels of objects and the pixels of backgrounds with no discrimination.
Other tested noises include (cid:15) · (1 / 256+
N (0 , 1)) and (cid:15) · [ − 1 / 256 , 0] , and similar results were obtained.
4. F REE L IKELIHOOD L OOPHOLE OF G LOW
Algorithm 1 Generate x ∗ by zeroing the latent factors
1: Input : image x , label y ( optional ), Glow model g , k < L . 2: z ← g.forward ( x , y ) (cid:46) Infer the latent factors.
3: z ∗ ← zero ( z , k ) (cid:46)
Zero the preceding k factors.
4: x ∗ ← g.reverse ( z ∗ , y ) (cid:46) Reverse the zeroed latent factors.
: return x ∗
Let us ﬁrst decompose the Glow architecture into blocks and review their contributions to the ﬁnal log-likelihood.
A Glow consists of a sequence of modules at different scale levels.
At each scale level, it starts with a squeeze operation, which reshapes the intermediate tensor without contribution to the log-likelihood.
Following each squeeze operation is a stack of step ﬂow blocks.
A step ﬂow block consists of three layers: actnorm, invertible 1 × 1 convolutional layer, and coupling layer.
The log-determinants of actnorm and 1 × 1 convolutional layer are input-independent, and depend only on their inner weights (see Table 1 in (Kingma & Dhariwal, 2018)).
Additive coupling layer is volumepreserving whose log-determinant is 0, thus is also input-independent.
For afﬁne coupling layer, its log-determinant depends on the afﬁned half, but is quantitatively small.
Compared to Glow with additive coupling layers, afﬁne layers bring only a small improvement of < 0 .
05 BPD (Kingma &
Dhariwal, 2018).
Then, the intermediary tensor is split into two halves along the channel dimension.
One halve is gaussianized with a convolutional block, then the other halve is factored out after being standardized.
This procedure signiﬁcantly reduces the amount of computation and memory.
We refer to Dinh et al.
(2016) for more details due to limited space.
So for a Glow using additive couplings, the cumulative log-determinant of the ﬂow blocks within a particular scale-level is constant regardless of different samples.
Only the log-determinants of gaussianized factorings (between the transitions of different scale-levels) depend on the individual inputs.
Denote z = { z 1 , . . .
, z L } as the latent variables of input image x .
Each z i is the standardized vector of i -th scale level.
This simply means that a sample x whose latent variables z are close to center 0 will have a higher log-likelihood.
Empirically, it also applies to Glow using afﬁne coupling layers, since the inﬂuence of varying the latent variables on the log-determinants of afﬁne coupling layers is quantitatively small compared to its gain.
We could make use of this property to generate samples with higher likelihoods via the invertibility of Glow for free (see Algorithm 1, label y is optional).
We ﬁnd that the semantic object of a test image depends heavily on the last factored latent z L , rather than the preceding factors.
Examples in
Fig.
5 show that zeroing the preceding 1 and 2 latent factors could give us samples with obviously lower BPDs but without obvious changes (slightly faded pixel intensities) of the semantic objects.
Results in Fig.
4 show that zeroing the ﬁrst latent factor give the maximum increment of likelihoods.
4. I MPLICATIONS ON D ISCRIMINATIVE C LASSIFIERS
We also evaluate the inﬂuences of these SITs on the performance of discriminative classiﬁers.
In contrast to the obvious change of ﬂows’ likelihoods, these small perturbations could decrease the testing accuracies of classiﬁers to some insigniﬁcant, or negligible (on MNIST) extent (see Tab 1).
The discriminative classiﬁer used here is a shallow residual network of 8 layers on 32 × 32 images in the structure speciﬁed in (He et al., 2016).
[Table]
[Table]
Difference to Adversarial Examples Both the SITs we use in this paper and adversarial perturbations are small perturbations, but they are inherently different.
Adversarial perturbations are intentionally crafted to cause misbehaviors, which are usually speciﬁc to individual images, and easily fool a classiﬁer with almost 0% accuracy.
While three SITs above are universal transformations over all images that take no additional computations and basically come for free.
D ISCUSSIONS AND C ONCLUSIONS
What is the problem of likelihood-based generative models?
Discriminative classiﬁers, trained to extract class-relevant features, are known to be vulnerable to adversarial examples, and give over-conﬁdent predictions even for OoD samples.
Generative models are supposed to be more robust since they model every pixel information of an image.
However, likelihood modeling in high-dimensional space can be hard and lead to counter-intuitive observations.
It was observed that likelihood-based generative models can assign even higher likelihoods on OoD samples(Nalisnick et al., 2018; Choi & Jang, 2018). Nalisnick et al. (2018) observe this phenomenon on both ﬂows and VAEs.
They decompose the change-of-variable theorem and investigate the inﬂuences of different transformation layers, ﬁnd that the phenomenon still exists regardless of whether the transformation is volume-preserving or not.
Their second-order analysis on pixel statistics suggests that OoD datasets, e.g. MNIST, just sit inside of in-distribution datasets, e.g. FashinMNIST, with roughly the same mean, smaller variance.
They suspect that ﬂows may simply ﬁt the pixel intensities without really capture the high-level semantics.
Ren (2019) ﬁnd that the likelihood of an image is mostly dominated by the irrelevant background pixels, and propose a remedy to correct the original likelihood with a likelihood ratio.
Though signiﬁcantly improves the accuracy of OoD detection, but still fail to answer the question: whether the likelihood ratio shows high correlation to high-level semantics.
This paper differs from previous works and step further to explore the correlations between the likelihood of ﬂow-based generative models and image semantics.
Theoretical analyses in (Theis et al., 2015; van den Oord & Dambre, 2015) point out an important argument that generative models’ ability to produce plausible samples is neither sufﬁcient nor necessary for high likelihood.
Results in this paper provide more experimental evidences for this simple argument that even for powerful exact likelihood-based generative models-ﬂows, the likelihoods of samples can be largely weakly correlated to the high-level image semantics.
Thus, special attention should be paid to this argument before we apply likelihood-based generative models to downstream tasks.
For example, considering the weak correlation between ﬂows’ likelihoods and image semantics, it may be inappropriate to use them for OoD samples detection.
On the other hand, these counter-intuitive behaviours of ﬂows raise our awareness of the gap between the predictive likelihoods of ﬂows and the expectation that these likelihoods can closely relate to the semantics for OoD detection.
What is exactly the likelihood of a image?
We should keep in mind that the predictive likelihood of a ﬂow is the joint probability of all the image pixels.
There is no doubt that ﬂows, trained by maximizing its likelihood, could generate impressive synthesized data.
There seem to be no problem that in terms of image generation, we expect that every single generated pixel in a image is the most likely one (hinging on its contextual pixels).
However, the likelihood is explicitly modeled on pixels, so can be easily inﬂuenced by pixel-level modiﬁcations.
Images’ likelihoods signiﬁcantly decrease even small noises are added to the pixels of backgrounds.
For downstream tasks that need some “likelihood” to indicate the object in an image is a cat, rather than a car, the pixels of backgrounds are almost irrelevant.
This drive us to think that we may need to model likelihood in some kind of semantic space or with some “perceptual” metrics, rather than on raw pixels.
One promising direction is to deﬁne likelihood of an images on its high-level representation, and successful examples are (Lee, 2018; Nilesh A. Ahuja, 2019).
A S ETUP OF E XPERIMENTS AND M ODELS
The target models Glow and PixelCNN are implemented in Pytorch 1.0, and the implementation of
PixelCNN is based on https://github.com/pclucas14/pixel-cnn-pp .
Though several modiﬁcations proposed in PixenCNN++ are used in our implementation, we still use PixelCNN to annotate our model.
We implement both unconditional and conditional versions of Glow.
We report the results of conditional Glow in our experiments.
For Glow, using multi-scale architecture with more levels is critical to achieve lower BPDs.
We resize the FashionMNIST, MNIST images from 28 × 28 to 32 × 32 and set the number of levels to 5.
So the last factored the latent variables will be 1 × 1 in the spatial size.
We provide pretrained models in code package as supplementary material for reproducing the results (We only provide models on
MNIST due to the upload size limit).
For PixelCNN, both 28 × 28 and 32 versions are provided.
In image translation experiments, we report the BPDs of PixelCNN on MNIST images of size 28 ×in the paper.
Unlike Glow, PixelCNN treat the pixels as discretized intensity levels, and resizing the image size to 32 × 32 could lead to dangling intensity levels.
We also ﬁnd that if performing image translation experiment on PixelCNN( 32 × 32 ), 1-pixel or 2-pixel translation will also lead to considerably BPD decreases.
[Table]
[Table]
B A DDITIONAL R ESULTS OF I MAGE T RANSLATION B. A DDITIONAL RESULTS
B. R ESULTS OF ADDITIVE G LOW
C A DDITIONAL R ESULTS OF Z EROING LATENT FACTORS OF G LOW
C. E XAMPLES OF A DDITIVE G LOW
original zero z ≤zero z ≤original zero z ≤zero z ≤
Glow 0.
Glow 1.
Glow 0.
D A DDITIONAL E XAMPLES OF R ANDOM N OISE P ERTURBATION
perturbed/- noise original masked noise perturbed/+
Glow 2.
PixelCNN 3.
Glow 2.
PixelCNN 2.
perturbed/- noise original masked noise perturbed/+
Glow 2.
PixelCNN 3.
Glow 2.
PixelCNN 3.