A UXILIARY G UIDED A UTOREGRESSIVE V ARIATIONAL A UTOENCODERS
Anonymous authors
Paper under double-blind review
A BSTRACT
Generative modeling of high-dimensional data is a key problem in machine learning.
Successful approaches include latent variable models and autoregressive models.
The complementary strengths of these approaches, to model global and local image statistics respectively, suggest hybrid models combining the strengths of both models.
Our contribution is to train such hybrid models using an auxiliary loss function that controls which information is captured by the latent variables and what is left to the autoregressive decoder.
In contrast, prior work on such hybrid models needed to limit the capacity of the autoregressive decoder to prevent degenerate models that ignore the latent variables and only rely on autoregressive modeling.
Our approach results in models with meaningful latent variable representations, and which rely on powerful autoregressive decoders to model image details.
Our model generates qualitatively convincing samples, and yields stateof-the-art quantitative results.
I NTRODUCTION
Unsupervised modeling of complex distributions with unknown structure is a landmark challenge in machine learning.
The problem is often studied in the context of learning generative models of the complex high-dimensional distributions of natural image collections.
Latent variable approaches can learn disentangled and concise representations of the data (Bengio et al., 2013), which are useful for compression (Gregor et al., 2016) and semi-supervised learning (Kingma et al., 2014; Rasmus et al., 2015).
When conditioned on prior information, generative models can be used for a variety of tasks, such as attribute or class-conditional image generation, text and pose-based image generation, image colorization, etc .
(Yan et al., 2016; van den Oord et al., 2016;
Reed et al., 2017; Deshpande et al., 2017).
Recently signiﬁcant advances in generative (image) modeling have been made along several lines, including adversarial networks (Goodfellow et al., 2014; Arjovsky et al., 2017), variational autoencoders (Kingma & Welling, 2014; Rezende et al., 2014), autoregressive models (Oord et al., 2016;
Reed et al., 2017), and non-volume preserving variable transformations (Dinh et al., 2017).
In our work we seek to combine the merits of two of these lines of work.
Variational autoencoders (VAEs) (Kingma & Welling, 2014; Rezende et al., 2014) can learn latent variable representations that abstract away from low-level details, but model pixels as conditionally independent given the latent variables.
This renders the generative model computationally efﬁcient, but the lack of lowlevel structure modeling leads to overly smooth and blurry samples.
Autoregressive models, such as pixelCNNs (Oord et al., 2016), on the other hand, estimate complex translation invariant conditional distributions among pixels.
They are effective to model low-level image statistics, and yield stateof-the-art likelihoods on test data (Salimans et al., 2017).
This is in line with the observations of Kolesnikov & Lampert (2017) that low-level image details account for a large part of the likelihood.
These autoregressive models, however, do not learn a latent variable representations to support, e.g ., semi-supervised learning.
See Figure 1 for representative samples of VAE and pixelCNN models.
The complementary strengths of VAEs and pixelCNNs, modeling global and local image statistics respectively, suggest hybrid approaches combining the strengths of both.
Prior work on such hybrid models needed to limit the capacity of the autoregressive decoder to prevent degenerate models that completely ignore the latent variables and rely on autoregressive modeling only (Gulrajani et al., 2017; Chen et al., 2017). In this paper we describe Auxiliary Guided Autoregressive Variational autoEncoders (AGAVE), an approach to train such hybrid models using an auxiliary loss function
[Figure]
that controls which information is captured by the latent variables and what is left to the AR decoder, rather than limiting the capacity of the latter.
Using high-capacity VAE and autoregressive components allows our models to obtain quantitative results on held-out data that are on par with the state of the art, and to generate samples with both global coherence and low-level details, see Figure 1.
R ELATED WORK
Generative image modeling has recently taken signiﬁcant strides forward, leveraging deep neural networks to learn complex density models using a variety of approaches.
These include the variational autoencoders and autoregressive models that form the basis of our work, but also generative adversarial networks (GANs) (Goodfellow et al., 2014; Arjovsky et al., 2017) and variable transformation with invertible functions (Dinh et al., 2017).
While GANs produce visually appealing samples, they suffer from mode dropping and their likelihood-free nature prevents measuring how well they model held-out test data.
In particular, GANs can only generate samples on a non-linear manifold in the data space with dimension equal to the number of latent variables.
In contrast, probabilistic models such as VAEs and autoregressive models generalize to the entire data space, and likelihoods of held-out data can be used for compression, and to quantitatively compare different models.
The non-volume preserving (NVP) transformation approach of Dinh et al.
(2017) chains together invertible transformations to map a basic ( e.g . unit Gaussian) prior on the latent space to a complex distribution on the data space.
This method offers tractable likelihood evaluation and exact inference, but obtains likelihoods on held-out data below the values reported using state-of-the-art VAE and autoregressive models.
Moreover, it is restricted to use latent representations with the same dimensionality as the input data, and is thus difﬁcult to scale to model high-resolution images.
Autoregressive density estimation models, such as pixelCNNs (Oord et al., 2016), admit tractable likelihood evaluation, while for variational autoencoders (Kingma & Welling, 2014; Rezende et al., 2014) accurate approximations can be obtained using importance sampling (Burda et al., 2016).
Naively combining powerful pixelCNN decoders in a VAE framework results in a degenerate model which ignores the VAE latent variable structure, as explained through the lens of bits-back coding by Chen et al.
To address this issue, the capacity of the the autoregressive component can be restricted.
This can, for example, be achieved by reducing its depth and/or ﬁeld of view, or by giving the pixelCNN only access to grayscale values, i.e .
modeling p ( x i | x <i , z ) =
p ( x i | gray ( x <i ) , z ) (Chen et al., 2017; Gulrajani et al., 2017).
This forces the model to leverage the latent variables z to model part of the dependencies among the pixels.
This approach, however, has two drawbacks.
(i)
Curbing the capacity of the model is undesirable in unsupervised settings where training data is abundant and overﬁtting unlikely.
(ii) Balancing what is modeled by the VAE and the pixelCNN by means of architectural design choices requires careful hand-design and tuning of the architectures.
To overcome these drawbacks, we propose to instead control what is modeled by the VAE and pixelCNN with an auxiliary loss on the VAE decoder output before it is used to condition the autoregressive decoder.
This allows us to “plug in” powerful high-capacity VAE and pixelCNN architectures, and balance what is modeled by each component by means of the auxiliary loss.
In a similar vein, Kolesnikov & Lampert (2017) force pixelCNN models to capture more high-level image aspects using an auxiliary representation y of the original image x , e.g . a low-resolution version of the original.
They learn a pixelCNN for y , and a conditional pixelCNN to predict x from y , possibly using several intermediate representations.
This approach forces modeling of more high-level aspects in the intermediate representations, and yields visually more compelling samples.
Reed et al.
(2017) similarly learn a series of conditional autoregressive models to upsample coarser intermediate latent images.
By introducing partial conditional independencies in the model they scale the model to efﬁciently sample high-resolution images of up to 512 × 512 pixels.
Gregor et al.
(2016) use a recurrent VAE model to produces a sequence of RGB images with increasing detail derived from latent variables associated with each iteration.
Like our work, all these models work with intermediate representations in RGB space to learn accurate generative image models.
A UXILIARY GUIDED AUTOREGRESSIVE VARIATIONAL AUTOENCODERS
We give a brief overview of variational autoencoders and their limitations in Section 3.1, before we present our approach to learn variational autoencoders with autoregressive decoders in Section 3.2.
3. V ARIATIONAL AUTOENCODERS
Variational autoencoders (Kingma & Welling, 2014; Rezende et al., 2014) learn deep generative latent variable models using two neural networks.
The “decoder” network implements a conditional distribution p θ ( x | z ) over observations x given a latent variable z , with parameters θ .
Together with a basic prior on the latent variable z , e.g . a unit Gaussian, the generative model on x is obtained by marginalizing out the latent variable:
[Equation]
The marginal likelihood can, however, not be optimized directly since the non-linear dependencies in p θ ( x | z ) render the integral intractable.
To overcome this problem, an “encoder” network is used to compute an approximate posterior distribution q φ ( z | x ) , with parameters φ .
The approximate posterior is used to deﬁne a variational bound on the data log-likelihood, by subtracting the
Kullback-Leibler divergence between the true and approximate posterior:
[Equation]
This is the same variational lower-bound that underlies the EM-algorithm (Neal & Hinton, 1998), and inequality (2) follows directly from the non-negativity of the KL divergence.
The decomposition in (3) interprets the bound as the sum of a reconstruction term and a regularization term.
The ﬁrst aims to maximize the expected data log-likelihood p θ ( x | z ) given the posterior estimate q φ ( z | x ) .
The second term regularizes q φ ( z | x ) to prevent it from collapsing to a single point.
Variational autoencoders typically model the dimensions of x as conditionally independent,
[Equation]
for instance using a factored Gaussian or Bernoulli model, see e.g .
Kingma & Welling (2014);
Kingma et al.
; Yan et al. (2016).
The conditional independence assumption makes sampling from the VAE efﬁcient: since the decoder network is evaluated only once for a sample z ∼ p ( z ) to compute all the conditional distributions p θ ( x i | z ) , the x i can then be sampled in parallel.
A result of relying on the latent variables to account for all pixel dependencies, however, is that all low-level variability must also be modeled by the latent variables.
Consider, for instance, a picture of a dog, and variants of that image shifted by one or a few pixels, or in a slightly different pose, with a slightly lighter background, or with less saturated colors, etc .
If these factors of variability are modeled using latent variables, then these low-level aspects are confounded with latent variables relating to the high-level image content.
If the corresponding image variability is not modeled using latent variables, it will be modeled as independent pixel noise.
In the latter case, using the mean of p θ ( x | z ) as the synthetic image for a given z results in blurry samples, since the mean is averaged over the low-level variants of the image.
Sampling from p θ ( x | z ) to obtain synthetic images, on the other hand, results in images with unrealistic independent pixel noise.
3. A UTOREGRESSIVE DECODERS IN VARIATIONAL AUTOENCODERS
Autoregressive density models, see e.g .
(Larochelle & Murray, 2011; Germain et al., 2015), rely on the basic factorization of multi-variate distributions,
[Equation]
with x <i = x 1 , . . .
, x i − 1 , and model the conditional distributions using a (deep) neural network.
For image data, pixelCNNs (Oord et al., 2016; van den Oord et al., 2016) use a scanline pixel ordering, and model the conditional distributions using a convolution neural network.
The convolutional ﬁlters are masked so as to ensure that the receptive ﬁelds only extend to pixels x <i when computing the conditional distribution of x i .
PixelCNNs can be used as a decoder in a VAE by conditioning on the latent variable z in addition to the preceding pixels, leading to a variational bound with a modiﬁed reconstruction term:
[Equation]
The regularization term can be interpreted as a “cost” of using the latent variables.
To effectively use the latent variables, the approximate posterior q φ ( x | z ) must differ from the prior p ( z ) , which increases the KL divergence.
Provided that it has enough capacity, it is thus more cost-effective to model image structure using the pixelCNN rather than the latent variables, see (Chen et al., 2017).
To ensure meaningful latent representation learning without sacriﬁcing the capacity of the pixelCNN decoder (Chen et al., 2017; Gulrajani et al., 2017), we use two decoders in parallel.
The ﬁrst one reconstructs an auxiliary image y from an intermediate representation f θ ( z ) in a non-autoregressive manner.
The auxiliary image can be a compressed version of the original image x , e.g . with lower resolution or with a coarser color quantization, or we can simply set y = x .
The second decoder is a conditional autoregressive model that predicts x conditioned on f θ ( z ) .
Modeling y in a nonautoregressive manner forces the latent variables to learn a meaningful representation of y , which is then “freely” available to the autoregressive decoder.
To train the model we combine both decoders in a single objective function with a shared encoder network:
[Equation]
Treating x and y as two variables that are conditionally independent given a shared underlying latent vairable z leads to λ = 1 .
Summing the lower bounds in Eq.
(3) and Eq.
(6) of the marginal loglikelihoods of y and x , and sharing the encoder network, leads to λ = 2 .
Larger values of λ result in valid but less tight lower bounds of the log-likelihoods.
Encouraging the variational posterior to be closer to the prior, this leads to less informative latent variable representations.
Sharing the encoder across the two decoders is the key of our approach.
The VAE decoder can only model pixel dependencies by means of the latent variables, which ensures that a meaningful representation is learned.
Now, given that the VAE decoder output is informative on the image [Figure] content, there is no incentive for the autoregressive decoder to ignore the intermediate representation f
( z ) on which it is conditioned.
The choice of the regularization parameter λ and auxiliary image y provide two levers to control how much and what type of information should be encoded in the latent variables.
See Figure 2 for a schematic illustration of our approach.
E XPERIMENTAL EVALUATION
In this section we describe our experimental setup, and present results on CIFAR10.
4. D ATASET AND IMPLEMENTATION
The CIFAR10 dataset (Krizhevsky, 2009) contains 6,000 images of 32 × 32 pixels for each of theobject categories airplane, automobile, bird, cat, deer, dog, frog, horse, ship, truck .
The images are split into 50,000 training images and 10,000 test images.
We train all our models in a completely unsupervised manner, ignoring the class information.
We implemented our model based on existing architectures.
In particular we use the VAE architecture of Kingma et al.
and use logistic distributions over the RGB color values.
We let the intermediate representation f ( z ) output by the VAE decoder be the per-pixel and per-channel mean values of the logistics, and learn per-channel scale parameters that are used across all pixels.
The cumulative density function (CDF), given by the sigmoid function, is used to compute probabilities across the 256 discrete color levels, or fewer if a lower quantization level is chosen in y .
Using RGB values y i ∈ [0 , 255] , we let b denote the number of discrete color levels and deﬁne c = 256 /b .
The probabilities over the b discrete color levels are computed from the logistic mean and variance µ i and s i as
[Equation]
For the pixelCNN we use the architecture of Salimans et al.
(2017), and
modify it to be conditioned on the VAE decoder output f ( z ) , or possibly an upsampled version if y has a lower resolution than x .
In particular, we apply standard non-masked convolutional layers to the VAE output, as many as there are pixelCNN layers.
We allow each layer of the pixel-CNN to take additional input using non-masked convolutions from the feature stream based on the VAE output.
This ensures that the conditional pixelCNN remains autoregressive.
To speed up training, we independently pretrain the VAE and pixelCNN in parallel, and then continue training the full model with both decoders.
We use the Adamax optimizer (Kingma & Ba, 2015) with a learning rate of 0 .
002 without learning rate decay.
We will release our TensorFlowbased code to replicate our experiments upon publication.
4. E XPERIMENTAL RESULTS
Quantitative performance evaluation.
Following previous work, we evaluate models on the test images using the bits-per-dimension (BPD) metric: the negative log-likelihood divided by the num-
[Table] [Table]
VR
PS
[Equation]
VR
PS
[Equation]
VR
PS
[Equation]
λ =
VS
PS λ =
VS
PS λ =
VS
PS (b) ber of pixels values (3 × 32 × 32).
It can be interpreted as the average number of bits per RGB value in a lossless compression scheme derived from the model.
The comparison in Table 1 shows that our model performs on par with the state-of-the-art results of the pixelCNN++ model (Salimans et al., 2017).
Here we used the importance sampling-based bound of Burda et al.
with 150 samples to compute the BPD metric for our model.
1 We refer to Figure 1 for qualitative comparison of samples from our model and pixelCNN++, the latter generated using the publicly available code.
Effect of KL regularization strength.
In Figure 3 we show reconstructions of test images and samples generated by the VAE decoder, together with their corresponding conditional pixelCNN samples for different values of λ .
As expected, the VAE reconstructions become less accurate for larger values of λ , mainly by lacking details while preserving the global shape of the input.
At the same time, the samples become more appealing for larger λ , suppressing the unrealistic highfrequency detail in the VAE samples obtained at lower values of λ .
Note that the VAE samples and reconstructions become more similar as λ increases, which makes the input to the pixelCNN during training and sampling more consistent.
[Figure]
[Figure]
For both reconstructions and samples, the pixelCNN clearly takes into account the output of the
VAE decoder, demonstrating the effectiveness of our auxiliary loss to condition high-capacity pixelCNN decoders on latent variable representations.
Samples from the pixelCNN faithfully reproduce the global structure of the VAE output, leading to more realistic samples, in particular for higher values of λ .
For λ = 2 the VAE reconstructions are near perfect during training, and the pixelCNN decoder does not signiﬁcantly modify the appearance of the VAE output.
For larger values of λ , the pixelCNN clearly adds signiﬁcant detail to the VAE outputs.
Figure 4 traces the BPD metrics of both the VAE and pixelCNN decoder as a function of λ .
We also show the decomposition in regularization and reconstruction terms.
By increasing λ , the KL divergence can be pushed closer to zero.
As the KL divergence term drops, the reconstruction term for the VAE rapidly increases and the VAE model obtains worse BPD values, stemming from the inability of the VAE to model pixel dependencies other than via the latent variables.
The reconstruction term of the pixelCNN decoder also increases with λ , as the amount of information it receives drops.
However, in terms of BPD which sums KL divergence and pixelCNN reconstruction, a substantial gain of 0.2 is observed increasing λ from 1 to 2, after which smaller but consistent gains are observed.
Effect of different auxiliary images.
We assess the effect of using coarser RGB quantizations and lower spatial resolutions in the auxiliary image.
Both make the VAE reconstruction task easier, and transfer modeling of color nuances and/or spatial detail to the pixelCNN.
[Figure] The VAE reconstructions in Figure 5 (a) obtained using coarser color quantization carry less detail than reconstructions based on the original images using 256 color values, as expected.
To understand the relatively small impact of the quantization level on the reconstruction, recall that the VAE decoder outputs the continuous means of the logistic distributions regardless of the quantization level.
Only the reconstruction loss is impacted by the quantization level via the computation of the probabilities over the discrete color levels in Eq.
In Figure 5 (b) we observe small but consistent gains in the BPD metric as the number of color bins is reduced, showing that it is more effective to model color nuances using the pixelCNN, rather than the latent variables.
We trained models with auxiliary images downsampled to 16 × 16 and 8 × 8 pixels, which yield 2.94 and 2.93 BPD, respectively.
Which is comparable to the 2.92 BPD obtained using our best model at scale 32 × 32.
In
Figure 6 (a) and (b) we show samples obtained using models trained with 256 and 32 color levels in the auxiliary image, and in Figure 6 (c) and (d) with auxiliary images of size 16 × 16 and 8 × 8.
The samples are qualitatively comparable, showing that in all cases the pixelCNN is able to compensate the less detailed outputs of the VAE decoder.
C ONCLUSION
We presented a new approach to training generative image models that combine a latent variable structure with an autoregressive model component.
Unlike prior approaches, it does not require careful architecture design to trade-off how much is modeled by latent variables and the autoregressive decoder.
Instead, this trade-off can be controlled using a regularization parameter, and the choice of auxiliary target images.
We obtain quantitative performance on par with the state of the art on CIFAR10, and samples from our model exhibit globally coherent structure as well as ﬁne details.