MAE: M UTUAL P OSTERIOR -D IVERGENCE R EGULARIZATION FOR V ARIATIONAL A UTO E NCODERS
A BSTRACT
Variational Autoencoder (VAE), a simple and effective deep generative model, has led to a number of impressive empirical successes and spawned many advanced variants and theoretical investigations.
However, recent studies demonstrate that, when equipped with expressive generative distributions (aka.
decoders), VAE suffers from learning uninformative latent representations with the observation called KL Varnishing , in which case VAE collapses into an unconditional generative model.
In this work, we introduce mutual posterior-divergence regularization , a novel regularization that is able to control the geometry of the latent space to accomplish meaningful representation learning, while achieving comparable or superior capability of density estimation.
Experiments on three image benchmark datasets demonstrate that, when equipped with powerful decoders, our model performs well both on density estimation and representation learning
I NTRODUCTION
Representation learning, besides data distributions estimation, is a principle component in generative models.
The goal is to identify and disentangle the underlying causal factors, to tease apart the underlying dependencies of the data, so that it becomes easier to understand, to classify, or to perform other tasks (Bengio et al., 2013).
Among these generative models, VAE (Kingma & Welling, 2014; Rezende et al., 2014) gains popularity for its capability of estimating densities of complex distributions, while automatically learning meaningful (low-dimensional) representations from raw data.
VAE, as a member of latent variable models (LVMs), deﬁnes the joint distribution between the observed data (visible variables) and a set of latent variables by factorizing it as the product of a prior over the latent variables and a conditional distribution of the visible variables given the latent ones (detailed in §2).
VAEs are usually estimated by maximizing the likelihood of the observed data by marginalizing over the latent variables, typically via optimizing the evidence lower bound (ELBO).
By learning a VAE from the data with the appropriate hierarchical structure of latent variables, the hope is to uncover and untangle the causal sources of variations that we are interested in.
A notorious problem of VAEs, however, is that the marginal likelihood may not guide the model to learn the intended latent variables.
It may instead focus on explaining irrelevant but common correlations in the data (Ganchev et al., 2010).
Extensive previous studies (Bowman et al., 2015;
Chen et al., 2017a; Yang et al., 2017) showed that optimizing the ELBO objective is often completely disconnected from the goal of learning good representations.
An extreme case called KL varnishing , happens when using sufﬁciently expressive decoding distributions such as auto-regressive ones; the latent variables are often completely ignored and the VAE regresses to a standard auto-regressive model (Larochelle & Murray, 2011; Oord et al., 2016).
This problem has spawned signiﬁcant interests in analyzing and solving it from both theoretical and practical perspectives.
We can only name a few here due to space limits.
Some previous work (Bowman et al., 2015; Sønderby et al., 2016b; Serban et al., 2017) attributed the KL varnishing phenomenon to “optimization challenges” of VAEs, and proposed training methods including annealing the relative weight of the KL term in ELBO (Bowman et al., 2015;
Sønderby et al., 2016b) or adding free bits (Kingma et al., 2016; Chen et al., 2017a).
However, Chen et al.
(2017a) pointed out that this phenomenon arises not just due to the optimization challenges, and even if we ﬁnd the exact solution for the optimization problems, the latent code will still be ignored at optimum.
They proposed a solution by limiting the capacity of the decoder and applied PixelCNN (Oord et al., 2016) with small local receptive ﬁelds as the decoder of VAEs to model 2D images, achieving both impressive performance for density estimation and informative latent representations.
Yang et al.
(2017) embraced a similar idea and applied VAE to text modeling by using dilated CNN as the decoder.
Unfortunately, these approaches require manual and problem-speciﬁc design of the decoder’s architecture to learn meaningful representations.
Other studies attempted to explore alternatives of ELBO.
Makhzani et al.
(2015) proposed Adversarial Autoencoders (AAEs) by replacing the KL-divergence between the posterior and prior distributions with Jensen-Shannon divergence on the aggregated posterior distribution.
InfoVAEs (Zhao et al., 2017) generalized the Jensen-Shannon divergence in
AAEs to a divergence family and linked its objective to the Mutual Information between the data and the latent variables.
However, directly optimizing these objectives is intractable, requiring advanced approximate learning methods such as adversarial learning or Maximum-Mean Discrepancy (Gretton et al., 2007;
Dziugaite et al., 2015; Li et al., 2015).
Moreover, these models’ performance on density estimation signiﬁcantly falls behind state-of-the-art models (Salimans et al., 2017; Chen et al., 2017a).
In this paper, we propose to tackle the aforementioned representation learning challenges of VAEs by adding a data-dependent regularization to the ELBO objective.
Our contributions are three-fold: (1) Algorithmically, we introduce the mutual posterior-divergence regularization for VAEs, named
MAEs (§3.2), to control the geometry of the latent space during learning by encouraging the learned variational posteriors to be diverse (i.e. they are favored to be mutually “different” from each other), to achieve low-redundant, interpretable representation learning.
(2) Theoretically, we establish a close relation between MAE and InfoVAE, by showing that the mutual posterior-devergence regularization maximizes a symmetric version of the KL divergence involved in InfoVAE’s mutual information term (§3.3).
(3) Experimentally, on three benchmark datasets for images, we demonstrate the effectiveness of MAE as a density estimator by state-of-the-art log-likelihood results on MNIST and OMNIGLOT, and comparable result on CIFAR-10.
Moreover, by performing image reconstruction, unsupervised and semi-supervised classiﬁcation, we show that MAE is also capable of learning meaningful latent representations, even combined with a sufﬁciently powerful decoder (§4).
V ARIATIONAL A UTOENCODERS 2. N OTATIONS
Throughout we use uppercase letters for random variables, and lowercase letters for realizations of the corresponding random variables.
Let X ∈ X be the randoms variables of the observed data, e.g.,
X is an image or a sentence for image and text generation, respectively.
Let P denote the true distribution of the data, i.e., X ∼ P , and D = { x 1 , . . .
, x N } be our training sample, where x i , i = 1 , . . .
, N, are usually i.i.d.
samples of X .
Let P
=
{ P θ : θ ∈ Θ } denote a parametric statistical model indexed by parameter θ ∈ Θ , where Θ is the parameter space.
p is used to denote the density of corresponding distribution P .
In the literature of deep generative models, deep neural networks are the most widely used parametric models.
The goal of generative models is to learn the parameter θ such that P θ can best approximate the true distribution P .
2. VAE S
In the framework of VAEs, or general LVMs, a set of latent variables Z ∈ Z are introduced to characterize the hidden patterns of X , and the model distribution P θ ( X ) is deﬁned as the marginal of the joint distribution between X and Z :
[Equation]
where the joint distribution p θ ( x, z ) is factorized as the product of a prior p θ ( z ) over the latent Z , and the “generative” distribution p θ ( x | z ) .
µ ( z ) is the base measure on the latent space Z .
Typically, prior p θ ( z ) is modeled with a simple distribution like multivariate Gaussian, or transforming simple priors to complex ones by normalizing ﬂows and variants (Rezende & Mohamed, 2015; Kingma et al., 2016;
Sønderby et al., 2016a).
To learn parameters θ , we wish to minimizes the negative log-likelihood of the parameters:
[Equation]
where ˜ P ( X ) is the empirical distribution derived from training data D .
In general, this marginal likelihood is intractable to compute or differentiate directly for high-dimensional latent space Z .
Variational Inference (Wainwright et al., 2008) provides a solution to optimize the evidence lower bound (ELBO) an alternative objective by introducing a parametric inference model q φ ( z | x ) :
[Equation]
where L elbo could be seen as an autoencoding loss with q φ ( z | x ) being the encoder and p θ ( x | z ) being the decoder, with the ﬁrst term in the RHS in (3) as the reconstruction error.
2. A UTOENCODING P ROBLEM IN VAE S
As discussed in Chen et al. (2017a), without further assumptions, the ELBO objective L elbo in (3) may not guide the model towards the intended role for the latent variables Z , or even learn uninformative Z with the observation that the KL term KL( q φ ( Z | X )
|| p θ ( Z )) varnishes to zero.
For example, suppose we use an auto-regressive decoder, p θ ( x | z ) =
(cid:81) i p θ ( x i | x <i , z ) , which is sufﬁciently expressive that it can model the data distribution P ( X ) without the assistance of Z , i.e, p θ ( x i | x <i , z ) =
p θ ( x i | x <i ) .
In this case, the optimal Z w.r.t.
L elbo is the one independent with X , with the inference model reducing to the prior, i.e., q φ ( Z | X ) =
p θ ( Z ) , ∀ X ∈ X .
The essential reason of this problem is that, under absolutely unsupervised setting, the marginal likelihood based objective L elbo incorporates no (direct) supervision on the latent space to characterize the latent variable Z with preferred properties w.r.t.
representation learning.
The main goal of this work is to explicitly control the geometry of the latent space, in the hope that preferred latent representations would be characterized and selected.
M UTUAL P OSTERIOR -D IVERGENCE R EGULARIZATION 3. G EOMETRIC P ROPERTIES OF M EANINGFUL L ATENT S PACE
Motivated by the Diversity-Inducing Mutual Angular Regularization (Xie et al., 2015) which is widely used in LVMs, we propose to regularize the posteriors q φ ( z | x ) of different data x ∈ X , to encourage them to diversely, smoothly, and evenly spread out in the data space X .
The intuition is: (1) To make posteriors mutually diverse from each other, the patterns captured by different posteriors are likely to have less redundancy and hence characterizing and interpreting different data x .
To make posteriors smoothly and evenly distributed in the whole space of X , the shared patterns of similar data points are likely to be captured by their posteriors, to avoid isolating each data point from others.
By balancing the diversity and smoothness of the distribution of posteriors, learned representations are encouraged to maintain global structured information, discarding detailed texture of local dependencies in the data.
3. MAE S
Measure of Diversity.
We propose to use expectation of the mutual KL-divergence between a pair of data to measure the diversity of posteriors.
Speciﬁcally, the mutual posterior diversity is deﬁned as:
[Equation]
There are two main reasons we use KL-divergence instead of others as the measure of diversity: (1)
KL-divergence is transformation invariant, i.e., for an invertible smooth function f ,
[Equation]
It makes the computation efﬁcient for complex posteriors that are transformed from simple ones, such as applying normalizing ﬂows and variants (Rezende & Mohamed, 2015; Kingma et al., 2016;
[Figure]
Sønderby et al., 2016a).
(2) KL-divergence has a close relation with mutual information, an important information-theoretic measure of the mutual dependence between two variables, which provides us the theoretical justiﬁcation of the proposed regularizer (detailed in §3.3).
In MAEs, we propose to maximize the mutual posterior diversity (MPD) in (4).
A straightforward way is to add negative MPD to the objective L elbo in (3) that VAEs attempt to minimize.
There are, however, two practical issues: (1) The scale of MPD, particularly for continuous Z , is much larger than that of L elbo .
We need to choose a hyperparameter carefully to control the scale of
MPD, making optimization much more challenging and unstable.
(2) For multivariate Z , e.g. a Kdimensional Z =
( Z 1 , Z 2 , . .
, Z K ) , due to the property of KL-divergence, MPD may be dominated by a small group of dimensions, leaving others close to zero.
In this case, most dimensions of Z are uninformative, which is not a desired representation.
To solve the two problems, in practice, we propose to minimize a MPD-based loss instead of directly maximizing MPD itself:
[Equation]
(cid:35) (5)
L diverse has two important good properties: (1) L diverse ≥ 0 .
[Equation]
L diverse
[Equation]
iff
KL( q φ ( Z k | X 1 )
||
q φ ( Z k | X 2 ))
→ ∞ , ∀ k .
The ﬁrst property sets a lower bound of L diverse , making optimization much more stable.
The second one guarantees that all the dimensions of the latent S need to be mutually diverse w.r.t minimizing L diverse .
Measure of Smoothness.
The smoothness of the distribution of posteriors is measured by utilizing the standard deviation of the mutual KL-divergence:
[Equation]
where STD stands for standard deviation of random variables.
L smooth encourages the posteriors to smoothly and evenly spread out to different directions.
Encouraging the standard deviation to be small can prevent the phenomenon that the posteriors fall into several small groups that are isolated from each other.
It is crucially important for unsupervised clustering tasks, in which we want to cluster similar data into a big group instead of splitting them into multiple separated small groups (see §4.1.2 for detailed experimental results).
Figure 1 shows two sets of distributions of data points, where the mean of the pairwise distances of the ﬁrst set (Figure 1 (a)) is roughly the same as the second set (Figure 1 (b)).
But the standard deviation of the ﬁrst set is larger.
In the framework of MAEs, the ﬁnal objective to minimize is:
[Equation] [Equation]
where η > 0 , γ > 0 are regularization constants to balance the three losses in L MAE .
Even though
MAE introduces two extra hyperparameters η and γ , we ﬁnd them easy to tune and MAE shows robust performance with different values of η and γ .
To solve (7), we can approximate L diverse and L smooth using Monte carlo in each mini-batch:
[Equation] [Equation]
where M is the number of valid pairs of data in each mini-batch.
L smooth is approximately computed similarly.
3. T HEORETICAL J USTIFICATION
So far our discussion has been concentrated on the motivation and mathematical formulation of the proposed regularization method for VAE.
In this section, we provide theoretical justiﬁcation by connecting the mutual posterior diversity (MPD) in (4) with the mutual information term deﬁned in
InfoVAE (Zhao et al., 2017).
With the end goal of theoretically justifying the proposed regularizer in mind, we ﬁrst review the background of the mutual information (MI) term involved in the InfoVAE objective, which is central for linking MAE and InfoVAE.
Mutual Information Maximization.
InfoVAE proposed the mutual information by ﬁrst deﬁning the joint “inference distribution”:
[Equation]
where p ( x ) is the density of the true data distribution P ( X ) .
Then they added a mutual information maximization term that prefers high mutual information between X and Z under q φ ( x, z ) to the standard L elbo :
[Equation]
and further proved that
[Equation]
where q φ ( z ) = (cid:82)
X q φ ( x, z ) dµ ( x ) is the marginal of q φ ( x, z ) .
Mutual information inspired objectives have been explored in GANs (Goodfellow et al., 2014; Chen et al., 2016), clustering (Hinton et al., 1995; Krause et al., 2010) and representation learning (Esmaeili et al., 2018; Hjelm et al., 2018).
Relation between MPD and MI.
The following theorem states our major result that reveals the relation between MPD and MI (proof in Appendix A):
Theorem 1.
The mutual posterior diversity (MPD) in (4) is a symmetric version of the KL-divergence of MI in (9) :
[Equation]
Roughly, Theorem 1 states that maximizing MPD and MI achieve the same goal: maximizing the divergence between the posterior distribution q φ ( z | x ) and the marginal q φ ( z ) .
Note that the (approximate) computation of MPD, as described in (8), is much easier than MI, which is generally intractable and requires adversarial learning or Maximum-Mean Discrepancy.
E XPERIMENTS
In this paper, we choose Variational Lossy Autoencoder (VLAE)
(Chen et al., 2017a), VAE with auto-regressive ﬂow (AF) prior, and auto-regressive decoder, as the basic architecture of our MAE models.
More detailed descriptions, results, and analysis of the conducted experiments are provided in Appendix B.
4. B INARY I MAGES
We evaluate MAE on two binary images that are commonly used for evaluating deep generative models: MNIST (LeCun et al., 1998) and OMNIGLOT (Lake et al., 2013; Burda et al., 2015), both with dynamically binarized version (Burda et al., 2015).
VLAE networks used in binary image datasets are similar of that described in Chen et al.
(2017a):
ResNet (He et al., 2016) encoder same as in ResNet VAE (Kingma et al., 2016), PixelCNN (Oord et al., 2016) decoder with 6 layers of masked convolution, and 32-dimensional latent code with AF prior implemented with MADE (Germain et al., 2015).
The only difference is that the PixelCNN decoder has varying ﬁlter sizes: two 7x layers, followed by two 5x5 layers, and ﬁnally two 3x3 layers, instead of a ﬁxed ﬁlter size of 3x used in Chen et al. (2017a).
Hence the decoder we use has larger receptive ﬁeld, to ensure that the decoder is sufﬁciently expressive.
The same architecture is applied to all the experiments on both the two datasets.
For pair comparison, we re-implemented VLAE using the same architecture in our
MAE model.
“Free bits” (Kingma et al., 2016) is used to improve optimization stability of VLAE (not for MAE).
For hyperparameters η and γ , we explored a few conﬁgurations: η is selected from [0 . 5 , 1 . 0 , 2 . 0] , and γ from [0 . 1 , 0 . 5 , 1 . 0] .
[Table]
-supervised classiﬁcation.
For each experiment, we report the average over 5 runs.
[Table]
η = 1 .
[Table]
γ = 0 .
[Table]
: η = 0 .
5 , γ = 0 .
[Table]
: η = 1 . 0 , γ = 0 .
[Table]
: η = 2 .
[Table]
γ
[Table] [Table]
=
[Table]
D ENSITY E STIMATION
We ﬁrst evaluate MAE on density estimation performance.
Table 1 provides the results of MAE with different settings of hyperparameters on MNIST, together with previous top systems for comparison.
Reported marginal negative log-likelihood (NLL) is evaluated with 4096 importance samples (Burda et al., 2015).
Our MAE achieves state-of-the-art performance on both the two datasets, exceeding all previous models and the re-implemented VLAE.
Note that our re-implementation of VLAE obtains better performance than the original one in Chen et al. (2017a), demonstrating the effectiveness of increasing decoder expressiveness by enlarging its receptive ﬁeld.
4.1. R EPRESENTATION L EARNING
In order to evaluate the quality of the learned latent representations, we conduct three sets of experiments — image reconstruction and generation, unsupervised clustering, and semi-supervised classiﬁcation.
Image Reconstruction and Generation.
The visualization of the of image reconstruction and generation on MNIST and OMNIGLOT is shown in Figure 2 and Figure 3.
For comparison, we also show the reconstructed images from VLAE.
MAE achieves better reconstruction ability than VLAE, proving that the latent code from MAE encodes more information from data.
Unsupervised Clustering.
As discussed above, good latent representations need to capture global structured information and disentangle the underlying causal factors, rather than just memorizing the data.
From this perspective, good image reconstruction results cannot guarantee good representations.
To further evaluate the quality of the learned representation from MAE, we conduct the experiments of unsupervised clustering on MNIST.
We perform K-Means clustering algorithm (Hartigan & Wong, 1979) on the learned representations.
The class label of each cluster is assigned by ﬁnding the closest
[Figure] [Figure]
sample in the training data with the cluster head.
Evaluation of clustering accuracy is based on the assigned cluster labels.
We run three experiments with K ∈ [10 , 20 , 30] .
Table 2 (left section) illustrates the clustering performance.
To make a thorough comparison, we also re-implemented a VAE model with a factorized decoder p θ ( x | z ) =
(cid:81) i p θ ( x i | z ) and AF prior, which has been proven to obtain remarkable reconstruction performance.
The VAE model uses ResNet (He et al., 2016) as its encoder and decoder similar to Chen et al.
(2017a).
From Table 2 we see that
MAE signiﬁcantly outperforms ResNet VAE and VLAE, especially when the number of clusters
K is small.
Interestingly, when K keeps increasing, clustering accuracy of ResNet VAE increases rapidly, showing that in its latent space the data are split into small groups.
In addition, towards the affects of η and γ on learned representations, MAEs with larger η obtain worse performance on K = 10 .
The reason might be that large η encourages the posteriors to diverse from each other, by splitting the data into small groups.
Meanwhile, increasing γ is effective to
[Table]
prevent the phenomenon, showing that in practice considerations on the trade-off between space diversity and smoothness are needed.
Semi-supervised Classiﬁcation.
For semi-supervised classiﬁcation, we re-implemented the M model as described in Kingma et al.
To test quality of information encoded in the latent representations, we choose two simple classiﬁers with limited capacity — K-nearest neighbor ( K = 10 ) and linear logistic regression.
For each classiﬁer, we use different numbers of labeled data — 100, 1000 and all the training data from MNIST.
From the results listed in Table 2 (right section)
[Table]
MAE obtains the best classiﬁcation accuracy on all the settings.
Moreover, the improvements of MAE over ResNet VAE and VLAE are more signiﬁcant when the number of labeled training data is small, further proving the meaningful representation learned from MAE.
4. N ATURAL I MAGES
In addition to binary image datasets, we also applied MAE to CIFAR-10 dataset (Krizhevsky &
Hinton, 2009) of natural images.
The VLAE with DenseNet (Huang et al., 2017) encoder and
PixelCNN++ (Salimans et al., 2017) decoder described in Chen et al.
(2017a) is used as the neural architecture of MAE.
To ensure that the decoder is sufﬁciently expressive, the decoder PixelCNN has 5 blocks of 96 feature maps and 7x4 receptive ﬁeld.
Hence the PixelCNN decoder we used is both deeper and wider than that used in Chen et al.
(2017a).
4. D ENSITY E STIMATION
Density estimation performance on CIFAR-10 of MAEs with different hyperparameters is provided in Table 3, compared with the top-performing likelihood-based unconditional generative models (ﬁrst section) and variationally trained latent-variable models (second section).
MAE models obtain improvement over the VLAE re-implemented by us, and slightly fall behind the original one in Chen et al.
(2017a).
Compared with PixelSNAIL (Chen et al., 2017b), the state-of-the-art auto-regressive generative model, the performance of MAE models is around 0.11 bits/dim worse.
Further improving the density estimation performance of MAEs on natural images has been left to future work.
4. I MAGE R ECONSTRUCTION AND G ENERATION
We also investigate learning informative representations on CIFAR-10 dataset.
The visualization of image reconstruction and generation is shown in Figure 4, together with VLAE for comparison.
It is interesting to note that MAE tends to preserve rather detailed shape information than VLAE, whereas the color information, particularly the color for background, is partially omitted.
One reasonable explanation, as discussed in Chen et al. (2017a), is that color is predictable locally.
This serves as one
[Figure]
example showing that MAEs can capture global structured information from data, omitting common correlations.
Image samples from MAE are shown in Figure 4b.
C ONCLUSION
In this paper, we proposed a mutual posterior-divergence regularization for VAEs, which controls the geometry of the latent space during training.
By connecting the mutual posterior diversity with the mutual information, we have formally studied the theoretical properties of the proposed MAEs.
Experiments on three benchmark datasets of images show the capability of MAEs on both density estimation and representation learning, with state-of-the-art or comparable likelihood, and superior performance on image reconstruction, unsupervised clustering and semi-supervised classiﬁcation against previous top-performing models.
One potential direction for future work is to extend MAE to other forms of data, in particular text on which VAEs suffer a more serious KL-varnishing problem.
Another exciting direction is to formally study the properties of the standard deviation of the mutual posterior KL-divergence used to measure smoothness, hence providing further justiﬁcation of the proposed regularizer, or even introducing alternatives to further improve performances. A CKNOWLEDGEMENTS
The authors thank Junxian He, Di Wang and Zhengzhong Liu for their helpful discussions.
This research was supported in part by DARPA grant FA8750-18-2-0018 funded under the AIDA program.
Any opinions, ﬁndings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reﬂect the views of DARPA.
A PPENDIX : MAE: M UTUAL P OSTERIOR -D IVERGENCE R EGULARIZATION FOR V ARIATIONAL A UTO E NCODERS A P ROOF OF T HEOREM
Proof.
[Equation]
where H( · ) denotes the entropy.
Then,
[Equation]
and
[Equation]
So we have,
[Equation]
( Z | X ))]
= E P ( X )
[H( q φ ( Z ) , q φ ( Z | X )) − H( q φ ( Z ))
+ H( q φ ( Z ))
− H( q φ ( Z | X ))]
= E P ( X )
[KL( q φ ( z | x ) ||
q φ ( z ))
+ KL( q φ ( z ) ||
q φ ( z | x ))]
B D ETAILED D ESCRIPTION OF E XPERIMENTS B. E XPERIMENTS FOR B INARY I MAGES B.1. N EURAL N ETWORK A RCHITECTURES AND T RAINING
The neural network architectures, including most of the hyperparameters, are the same as those in
Chen et al.
(2017a).
The only difference in network architecture is the ﬁlter size of the PixelCNN decoder, which has been described in §4.
For ResNet VAE with AF, we use the same ResNet encoder but a symmetric ResNet architecture for decoder.
For encoder, we only use one stochastic layer with 32 dimensions.
In term of training, we use Adam optimizer (JLB, 2015) with learning rate 0.001, instead of Adamax used in Chen et al. (2017a).
0.01 nats/data-dim free bits was used in all the experiments.
In order to get a relatively accurate approximation of L diverse and L smooth , we used a much larger batch size 100 in our experiments.
Polyak averaging (Polyak & Juditsky, 1992) was used to compute the ﬁnal parameters, with α = 0 .
B.1. D ETAILED R ESULTS ON D ENSITY E STIMATION
Table 4 shows the detailed results of density estimation on MNIST.
We see that increasing η always achieving more informative latent Z , but the NLL not always becomes better.
It illustrates the hypothesis that good representations should encode global structured information in the data, rather than local dependencies.
It is interesting to see that, the effect of γ on the latent Z is inconsistent — increasing γ from 0.1 to 0.5 leads to more informative Z (larger KL) and better γ
[Table] [Table]
5 ± 0.
(1.0) prevent the latent Z to learn more information from data (smaller KL), resulting worse NLL.
Hence, in practice considerations on the trade-off between diversity and smoothness of the latent space are needed.
B.1. D ETAILED R ESULTS ON S EMI - SUPERVISED C LASSIFICATION WITH SVM S
Table 5 provides the performance of semi-supervised classiﬁcation using SVMs with two different kernels — linear and RBF.
MAE achieves the best classiﬁcation accuracy on all the settings.
It shoulld be noted that the accuracies of SVMs with non-linear kernels ﬂuctuate more rapidly than linear ones, particularly when the number of labeled training data is small.
B.1. L ATENT S PACE V ISUALIZATION
Figure 5 visualize the latent spaces of VAEs and MAEs with different settings on MNIST, by t-Distributed Stochastic Neighbor Embedding (t-SNE)
(Maaten & Hinton, 2008).
The ﬁrst row displays the visualizations of ResNet VAE, VLAE without free-bits training and VLAE with free-bits training.
The following three rows display visualizations of MAEs with γ ∈ [0 . 1 , 0 . 5 , 1 . 0] and η ∈ [0 . 5 , 1 . 0 , 2 . 0] .
We see that large η encourages the posteriors to diverse from each other, by splitting the data into small groups.
Meanwhile, increasing γ is effective to prevent the phenomenon.
B. E XPERIMENTS FOR CIFAR-
Following Kingma et al.
and
Chen et al.
(2017a)
, latent codes are represented by 16 feature maps of 8x8.
Prior distribution is factorized Gaussian transformed by 8 auto-regressive ﬂows, each of which is implemented by 3-layer masked CNNs (Oord et al., 2016) with 128 feature maps.
Between every other auto-regressive ﬂow, the ordering of stochastic units is reversed.
PixelCNN++ (Salimans et al., 2017) with 7x3 receptive ﬁeld is used as the decoder.
Due to the limitation of computational
[Figure]
VLAE VLAE
[Equation]
resources, we used batch size 64 in our experiments.
Same as experiments on binary images, Polyak averaging (Polyak & Juditsky, 1992) was used to compute the ﬁnal parameters, with α = 0 .
C G ENERATED S AMPLES FROM VLAE AND MAE
Figure 6 provides generated images on MNIST, OMNIGLOT and CIFAR-10 from VLAE and MAE.
[Figure]