Paper:
EPITOMIC VARIATIONAL AUTOENCODER
In this paper, we propose epitomic variational autoencoder (eVAE), a probabilistic generative model of high dimensional data. eVAE is composed of a number of sparse variational autoencoders called ‘epitome’ such that each epitome partially shares its encoder-decoder architecture with other epitomes in the composition. We show that the proposed model greatly overcomes the common problem in variational autoencoders (VAE) of model over-pruning. We substantiate that eVAE is efficient in using its model capacity and generalizes better than VAE, by presenting qualitative and quantitative results on MNIST and TFD datasets.
1 INTRODUCTION
Unsupervised learning holds the promise of learning the inherent structure in data so as to enable many future tasks including generation, prediction and visualization. Generative modeling is an approach to unsupervised learning wherein an explicit stochastic generative model of data is defined, such that independent draws from this model are likely to produce the original data distribution, while the learned latent structure itself is useful in prediction, classification and visualization tasks.
The recently proposed variational autoencoder (VAE) (Kingma & Welling, 2014) is an example of one such generative model. VAE pairs a top down generative model with a bottom up recognition network for amortized probabilistic inference. Both networks are jointly trained to maximize a variational lower bound on the data likelihood. A number of recent works use VAE as a modeling framework, including iterative conditional generation of images (Gregor et al., 2015) and conditional future frame prediction (Xue et al., 2016).
A commonly known problem with the VAE lower bound is that it is known to self-prune or under utilize the model’s capacity (Mackay, 2001). This can lead to poor generalization. A common approach to alleviate this problem is to resort to optimization schedules and regularization techniques (Bowman et al., 2015; Kaae Sonderby et al., 2016) that trade-off two competing terms, latent cost and data reconstruction, in the bound. Fig. 1 provides a quick insight into this problem of over-pruning and how commonly used regularization techniques may not be sufficient. Detailed discussion is provided in § 2.1. In this paper, we take a model-based approach to directly address this problem. We present an extension of variational autoencoders called epitomic variational autoencoder (Epitomic VAE, or eVAE, for short) that automatically learns to utilize its model capacity more effectively, leading to better generalization. Consider the task of learning a D-dimensional representation for the examples in a given dataset. The motivation for our model stems from the hypothesis that a single example in the dataset can be sufficiently embedded in a smaller K-dimensional (K D) subspace of D. However, different data points may need different subspaces, hence the need for D. Sparse coding methods also exploit a similar hypothesis. Epitomic VAE exploits sparsity using an additional categorical latent variable in the encoder-decoder architecture of the VAE. Each value of the variable activates only a contiguous subset of latent stochastic variables to generate an observation. This ∗Work done during an internship at Facebook AI Research.
enables learning multiple shared subspaces such that each subspace specializes, and also increases the use of model capacity (Fig. 4), enabling better representation. The choice of the name Epitomic VAE comes from the fact that multiple miniature models with shared parameters are trained simultaneously.
The rest of the paper is organized as follows. We first describe variational autoencoders and mathematically show the model pruning effect in § 2. We then present our epitomic VAE model in § 3 that overcomes these shortcomings. Experiments showing qualitative and quantitative results are presented in § 4. We finally provide more general context of our work in the related work in § 5, and conclude with discussions.
2 VARIATIONAL AUTOENCODERS
The generative model (decoder) of a VAE consists of first generating a D-dimensional stochastic variable z drawn from a standard multivariate Gaussian
p(z) = N (z; 0; I) (1) and then generating the N-dimensional observation x from a parametric family of distributions such as a Gaussian pθ(x|z) = N (x; f1(z); exp(f2(z))) (2) where f1 and f2 define non-linear deterministic transformations of z modeled using a neural network. The parameters θ of the model are the weights and biases of the neural network that encodes the functions f1 and f2.
Given a dataset X of T i.i.d samples, the model is learned such that it maximizes the likelihood of the parameters to have generated the data, p(X|θ). This maximization requires marginalizing the unobserved z. However, computing p(z|x) is intractable due to dependencies induced between the zi when conditioned on x.
Variational autoencoders, as the name suggests, use variational inference to approximate the exact posterior with a surrogate parameterized distribution. However, instead of having separate parameters for the posterior distribution of each observation, VAE amortizes the cost by learning a neural network with parameters φ that outputs the posterior distribution of the form qφ(z|x) = ∏ d q(zi|x). This results in the lower bound given by
log pθ(X) = T∑ t=1 log ∫ z pθ(x (t), z) (3)
≥ T∑ t=1 Eqφ(z|x(t))[log p(x (t)|z)]−KL ( qφ(z|x(t)) ‖ p(z) ) (4)
VAE is trained with standard backpropagation using minibatch gradient descent to minimize the negative of the lowerbound
Cvae = − T∑ t=1 Eqφ(z|x(t))[log p(x (t)|z)] + T∑ t=1 D∑ i=1 KL ( qφ(zi|x(t)) ‖ p(zi) ) (5)
2.1 AUTOMATIC MODEL OVER-PRUNING IN VAE
Cvae introduces a trade-off between data reconstruction (first term) and satisfying the independence assumption of p(z) (second term, KL).
Of particular interest is the KL term. Since the KL term is the sum of independent contributions from each dimension d of D, it provides unduly freedom for the model in how it minimizes this term. In particular, the model needs to only ensure that the overall KL term is minimized, on average, and not per component wise. The easiest way for the model to do this is to have a large number of components that satisfies the KL term effectively, by turning off the units so that the posterior for those units becomes the same as the prior1. This effect is quite pronounced in the early iterations of
1Since log variance is modeled using the neural network, turning it off will lead to a variance of 1.
training: the model for log p(x|z) is quite impoverished and hence the easiest way to improve the bound is by turning off the KL terms. However, once the units have become inactive, it is almost impossible for them to resurrect, and hence the full capacity of the model is not utilized.
A quantity that is useful in understanding this effect, is the activity level of a unit. Following Burda et al. (2015), we define a unit to be used, or “active”, if Au = Covx(Eu∼q(u|x)[u]) > 0.02.
A commonly used approach to overcome this problem is to use a trade-off between the two terms using parameter λ so that the cost is
C = −Eqφ(z|x)[log p(x|z)] + λ D∑ i=1 KL ( qφ(zi|x) ‖ p(zi) ) (6)
Fig. 1 shows the effect of λ on unit activity and generation, with λ = 1 being the correct objective to optimize. While tuning down λ increases the number of active units, samples generated from the model are still poor. Fig. 2 shows generation using all units, active units only, and dead units only, for λ = 1. The model spends its capacity in ensuring that reconstruction of the training set is optimized (reconstruction visualizations are shown in § 8.1), at the cost of generalization. This has led to more sophisticated schemes such as using an annealed optimization schedule for λ (Bowman et al., 2015; Kaae Sonderby et al., 2016) or enforcing minimum KL contribution from subsets of the latent units (Kingma et al., 2016).
In this paper, we present a model based approach called “epitomic variational autoencoder” to address the problem of over pruning.
3 MODEL
We propose epitomic variational autoencoder (eVAE) to overcome the shortcomings of VAE by enabling more efficient use of model capacity to gain better generalization. We base this on the observation that while we may need a D-dimensional representation to accurately represent every example in a dataset, each individual example can be represented with a smaller K-dimensional subspace. As an example, consider MNIST with its variability in terms of digits, strokes and thick-
ness of ink, to name a few. While the overall D is large, it is likely that only a few K dimensions of D are needed to capture the variability in strokes of some digits (see Fig. 3).
Epitomic VAE can be viewed as a variational autoencoder with latent stochastic dimension D that is composed of a number of sparse variational autoencoders called epitomes, such that each epitome partially shares its encoder-decoder architecture with other epitomes in the composition. In this paper, we assume simple structured sparsity for each epitome: in particular, only K contiguous dimensions of D are active2.
The generative process can be described as follows: A D-dimensional stochastic variable z is drawn from a standard multivariate Gaussian p(z) = N (z; 0; I). In tandem, an epitome is implicitly chosen through an epitome selector variable y, which has a uniform prior over possible epitomes. The N -dimensional observation x is then drawn from a Gaussian distribution:
pθ(x|y, z) = N (x; f1(my z), exp(f2(my z))) (7) my enforces the epitome constraint: it is also aD-dimensional vector that is zero everywhere except in the active dimensions of the epitome. is element-wise multiplication between the two operands. Thus, my masks the dimensions of z other than those dictated by the choice of y. Fig. 3 illustrates this for an 8-d z with epitome size K = 2, so that there are four possible epitomes (the model also allows for overlapping epitomes, but this is not shown for illustration purposes). Epitome structure is defined using size K and stride s, where s = 1 corresponds to full overlap in D dimensions3. Our model generalizes the VAE and collapses to a VAE when D = K = s.
f1( ) and f2( ) define non-linear deterministic transformations of modeled using neural networks. Note that the model does not snip off the K dimensions corresponding to an epitome, but instead deactivates the D-K dimensions that are not part of the chosen epitome. While the same deterministic functions f1 and f2 are used for any choice of epitome, the functions can still specialize due to the
2The model also allows for incorporating other forms of structured sparsity. 3The strided epitome structure allows for learning O(D) specialized subspaces, that when sampled during generation can each produce good samples. In contrast, if only a simple sparsity prior is introduced over arbitrary subsets (e.g. with Bernoulli latent units to specify if a unit is active for a particular example), it can lead to poor generation results, which we confirmed empirically but do not report. The reason for this is as follows: due to an exponential number of potential combinations of latent units, sampling a subset from the prior during generation cannot be straightforwardly guaranteed to be a good configuration for a subconcept in the data, and often leads to uninterpretable samples.
sparsity of their inputs. Neighboring epitomes will have more overlap than non-overlapping ones, which manifests itself in the representation space; an intrinsic ordering in the variability is learned.
3.1 OVERCOMING OVER-PRUNING
Following Kingma & Welling (2014), we use a recognition network q(z, y|x) for approximate posterior inference, with the functional form
q(z, y|x) = q(y|x)q(z|y,x) (8) = q(y|x)N (z;my µ, exp (my φ)) (9)
where µ = h1(x) and φ = h2(x) are neural networks that map x to D dimensional space.
We use a similar masking operation to deactivate units, as decided by the epitome y. Unlike the generative model (eq. 7), the masking operation defined by y operates directly on outputs of the recognition network that characterizes the parameters of q(z|y,x). As in VAE, we can derive the lower bound on the log probability of a dataset, and hence the cost function (negative bound) is
Cevae = − T∑ t=1 Eq(z,y|x(t))[log p(x (t)|y, z)]
− T∑ t=1 KL [ qφ(y|x(t)) ‖ pθ(y) ] − T∑ t=1 ∑ y qφ(y|x(t))KL [ qφ(z|y,x(t)) ‖ pθ(z) ] (10)
The epitomic VAE departs from the VAE in how the contribution from the KL term is constrained. Let us consider the third term in eq. 10, and substituting in eq. 9:
T∑ t=1 ∑ y qφ(y|x(t))KL [ qφ(z|y,x(t)) ‖ pθ(z) ] (11)
= T∑ t=1 ∑ y qφ(y|x(t))KL [ N (z;my µ(t), exp (my φ(t))) ‖ N (z;0, I) ] (12)
= T∑ t=1 ∑ y qφ(y|x(t)) D∑ d=1 1[md,y = 1]KL [ N (zd;µ(t)d , exp(φ (t) d )) ‖ N (0, 1) ] (13)
where 1[?] is an indicator variable that evaluates to 1 if only if its operand ? is true.
For a training example x(t) and for a fixed y (and hence the corresponding epitome), the number of KL terms that will contribute to the bound is exactly K. The dimensions of z that are not part of the corresponding epitome will have zero KL because their posterior parameters are masked to have unit Gaussian, the same as the prior. By design, this ensures that only the K dimensions that explain x(t) contribute to Cevae.
This is quite in contrast to how VAE optimizes Cvae (§. 2.1). For Cvae to have a small contribution from the KL term of a particular zd, it has to infer that unit to have zero mean and unit variance for many examples in the training set. In practice, this results in VAE completely deactivating units, and leading to many dead units. EpitomicVAE chooses the epitome based on x(t) and ensures that the dimensions that are not useful in explaining x(t) are ignored in Cevae. This means that the unit is still active, but by design, only a fraction of examples in the training set contributes a possible non-zero value to zd’s KL term in Cevae. This added flexibility gives the model the freedom to use more total units without deactivating them, while optimizing the bound. With these characteristics, during training, the data points will naturally group themselves to different epitomes, leading to a more balanced use of z.
In Fig. 4 we compare the activity levels of VAE, dropout VAE and our model. We see that compared with VAE, our model is able to better use the model capacity. In the same figure, we also compare with adding dropout to the latent variable z of the VAE (Dropout VAE). While this increases the number of active units, it generalizes poorly as it uses the dropout layers to merely replicate representation, in contrast to eVAE. See Fig. 5 along with the explanation in § 4.1 where we compare generation results for all three models.
3.2 TRAINING
The generative model and the recognition network are trained simultaneously, by minimizing Cevae in eq. 10.
For the stochastic continuous variable z, we use the reparameterization trick as in VAE. The trick involves reparametrizing the recognition distribution in terms of auxiliary variables with fixed distributions. This allows efficient sampling from the posterior distribution as they are deterministic functions of the inputs and auxiliary variables.
For the discrete variable y, we cannot use the reparameterization trick. We therefore approximate q(y|x) by a point estimate y∗ so that q(y|x) = δ(y = y∗), where δ evaluates to 1 only if y = y∗ and the best y∗ = argmin Cevae. We also explored modeling q(y|x) = Cat(h(x)) as a discrete distribution with h being a neural network. In this case, the backward pass requires either using REINFORCE or passing through gradients for the categorical sampler. In our experiments, we found that these approaches did not work well, especially when the number of possible values of y becomes large. We leave this as future work to explore.
The recognition network first computes µ and φ. It is then combined with the optimal y∗ for each example, to arrive at the final posterior. The model is trained using a simple algorithm outlined in Algo. 1. Backpropagation with minibatch updates is used, with each minibatch constructed to be balanced with respect to epitome assignment.
Algorithm 1 Learning Epitomic VAE 1: θ, φ←Initialize parameters 2: for until convergence of parameters (θ, φ) do 3: Assign each x to its best y∗ = argmin Cevae 4: Randomize and then partition data into minibatches with each minibatch having proportion-
ate number of examples ∀ y 5: for k ∈ numbatches do 6: Update model parameters using kth minibatch consisting of x, y pairs 7: end for 8: end for
4 EXPERIMENTS
We present experimental results on two datasets, MNIST (LeCun et al., 1998) and Toronto Faces Database (TFD) (Susskind et al., 2010). We show generation results that illustrate eVAE’s ability to better utilize model capacity for modeling data variability, and then evaluate the effect of epitome choice and model complexity. Finally we present quantitative comparison with other models and qualitative samples from eVAE. We emphasize that in all experiments, we keep the weight of the KL term λ = 1 to evaluate performance under optimizing the true derived lower bound, without introducing an additional hyperparameter to tune.
We use standard splits for both MNIST and TFD. In our experiments, the encoder and decoder are fullyconnected networks, and we show results for different depths and number of units of per layer. ReLU nonlinearities are used, and models are trained using the Adam update rule (Kingma & Ba, 2014) for 200 epochs (MNIST) and 250 epochs (TFD), with base learning rate 0.001.
4.1 OVERCOMING OVER-PRUNING.
We first qualitatively illustrate the ability of eVAE to overcome over-pruning and utilize latent capacity to model greater variability in data. Fig. 5 compares generation results for VAE, Dropout VAE, and eVAE for different dimensionsD of latent variable z. WithD = 2, VAE generates realistic digits but suffers from lack of diversity. When D is increased to 5, the generation exhibits some greater variability but also begins to degrade in quality. As D is further increased to 10 and 20, the degradation continues. As explained in Sec. 2.1, this is due to VAE’s propensity to use only a portion of its latent units for modeling the training data and the rest to minimize the KL term. The under-utilization of model capacity means that VAE learns to model well only regions of the posterior manifold near training samples, instead of generalizing to model the space of possible generations. The effect of this is good reconstruction (examples are shown in Fig. 9) but poor generation samples.
Adding dropout to the latent variable z of the VAE (row 2 of Fig. 5) encourages increased usage of model capacity, as shown in Fig. 4 and the discussion in Sec. 2. However, due to the stochastic nature of dropout, the model is forced to use the additional capacity to encode redundancy in the representation. It therefore does not achieve the desired effect of encoding additional data variability, and furthermore leads to blurred samples due to the redundant encoding. Epitomic VAE addresses the crux of the problem by learning multiple specialized subspaces. Since the effective dimension of any example is still small, eVAE is able to model each subspace well, while encoding variability through multiple possibly shared subspaces. This enables the model to overcome over-pruning from which VAE suffered. Fig. 5 shows that as the dimension D of z is increased
while maintaining epitomes of size K = 2, eVAE is able to model greater variability in the data. Highlighted digits in the 20-d eVAE show multiple styles such as crossed versus un-crossed 7, and pointed, round, thick, and thin 4s. Additional visualization of the variability in the learned 2-d manifolds are shown in Fig. 3. In contrast, the 2-d VAE generates similar-looking digits, and is unable to increase variability and maintain sample quality as the latent dimension is increased.
4.2 CHOICE OF EPITOME SIZE
We next investigate how the choice of epitome size, K, affects generation performance. We evaluate the generative models quantitatively through their samples by measuring the log-density with a Parzen window estimator Rifai et al. (2012). Fig. 6 shows the Parzen log-density for different choices of epitome size on MNIST, with encoder and decoder consisting of a single deterministic layer of 500 units. Epitomes are nonoverlapping, and the results are grouped by total dimension D of the latent variable z. For comparison, we also show the log-density for VAE models with the same dimension D, and for mixture VAE (mVAE), an ablative version of eVAE where parameters are not shared. mVAE can also be seen as a mixture of independent VAEs trained in the same manner as eVAE. The number of deterministic units in each mVAE component is computed so that the total number of parameters is comparable to eVAE.
As we increase D, the performance of VAE drops significantly, due to over-pruning. In fact, the number of active units for VAE are 8, 22 and 24 respectively, forD values of 8, 24 and 48. In contrast, eVAE performance increases as we increase D, with an epitome size K that is significantly smaller than D. Table 1 provides more comparisons. This confirms the advantage of using eVAE to avoid overpruning and effectively capture data distribution.
eVAE also performs comparably or better than mVAE at all epitome sizes. Intuitively, the advantage of parameter sharing in eVAE is that each epitome can also benefit from general features learned across the training set.
4.3 INCREASING COMPLEXITY OF ENCODER AND DECODER
Here, we would like to understand the role of encoder and decoder architectures on over pruning, and the generative performance. We control model complexity through number of layers L of deterministic hidden units, and number of hidden units H in each deterministic layer.
Table 1 shows the Parzen log-densities of VAE, mVAE and eVAE models trained on MNIST and TFD with different latent dimension D. For mVAE and eVAE models on MNIST, the maximum over epitomes of size K = 3 and K = 4 is used, and on TFD epitomes of size K = 5 are used. All epitomes are non-overlapping.
We observe that for VAE, increasing the number of hidden units H (e.g. from 500 to 1000) for a fixed network depth L has a negligible effect on the number of active units and performance. On the other hand, as the depth of the encoder and decoder L is increased, the number of active units in VAE decreases though performance is still able to improve. This illustrates that increase in the complexity of the interactions through use of multiple
layers counteract the perils of the over-pruning. However, this comes with the cost of substantial increase in the number of model parameters to be learned.
In contrast, for any given model configuration, eVAE is able to avoid the over-pruning effect in the number of active units and outperform VAE. While both VAE and eVAE approach what appears to be a ceiling in generative performance with large models for MNIST, the difference between VAE and eVAE is significant for all TFD models.
Table 1 also shows results for mVAE, the ablative version of eVAE where parameters are not shared. The number of deterministic units per layer in each mVAE component is computed so that the total number of parameters is comparable to eVAE. While mVAE and eVAE perform comparably on MNIST especially with larger models (reaching a limit in performance that VAE also nears), eVAE demonstrates an advantage on smaller models and when the data is more complex (TFD). These settings are in line with the intuition that parameter sharing is helpful in more challenging settings when each epitome can also benefit from general features learned across the training set.
H = 500 H = 1000 L = 1 L = 2 L = 3 L = 1 L = 2 L = 3
MNIST
D = 8 VAE 283(8) 292(8) 325(8) 283(8) 290(8) 322(6) mVAE 300(8) 328(8) 337(8) 309(8) 333(8) 335(8) eVAE 300(8) 330(8) 337(8) 312(8) 331(8) 334(8)
D = 24 VAE 213(22) 273(11) 305(8) 219(24) 270(12) 311(7) mVAE 309(24) 330(24) 336(24) 313(24) 333(24) 338(24) eVAE 311(24) 331(24) 336(24) 317(24) 332(24) 336(24)
D = 48 VAE 213(24) 267(13) 308(8) 224(24) 273(12) 309(8) mVAE 314(48) 334(48) 336(48) 315(48) 333(48) 337(48) eVAE 319(48) 334(48) 337(48) 321(48) 334(48) 332(48)
TFD
4.4 COMPARISON WITH OTHER MODELS
In Table 2 we compare the generative performance of eVAE with other models, using Parzen log-density. VAE−, mVAE−, and eVAE− refer to models trained using the same architecture as Adversarial Autoencoders, for comparison. Encoders and decoders have L = 2 layers of H = 1000 deterministic units. D = 8 for MNIST, and D = 15 for TFD. VAE, mVAE, and eVAE refer to the best performing models over all architectures from Table 1. For MNIST, the VAE model is (L,H,D) = (3, 500, 8), mVAE is (3, 1000, 24), and eVAE is (3, 500, 48). For TFD, the VAE model is (3, 500, 15), mVAE is (3, 1000, 50), and eVAE is (3, 500, 25).
We observe that eVAE significantly improves over VAE and is competitive with several state-of-the-art models, notably Adversarial Autoencoders. Samples from eVAE on MNIST and TFD are shown in Fig. 7.
5 RELATED WORK
A number of applications use variational autoencoders as a building block. In Gregor et al. (2015), a generative model for images is proposed in which the generator of the VAE is an attention-based recurrent model that is conditioned on the canvas drawn so far. Eslami et al. (2016) proposes a VAE-based recurrent generative model that describes images as formed by sequentially choosing an object to draw and adding it to a canvas that is updated over time. In Kulkarni et al. (2015), VAEs are used for rendering 3D objects. Conditional variants of VAE are also used for attribute specific image generation (Yan et al., 2015) and future frame synthesis (Xue et al., 2016). All these applications suffer from the problem of model over-pruning and hence have adopted strategies that takes away the clean mathematical formulation of VAE. We have discussed these in § 2.1.
A complementary approach to the problem of model pruning in VAE was proposed in Burda et al. (2015); the idea is to improve the variational bound by using multiple weighted posterior samples. Epitomic VAE provides improved latent capacity even when only single sample is drawn from the posterior.
Methods to increase the flexibility of posterior inference are proposed in (Salimans et al., 2015; Rezende & Mohamed, 2016; Kingma et al., 2016). In Rezende & Mohamed (2016), posterior approximation is constructed by transforming a simple initial density into a complex one with a sequence of invertible transformations. In a similar vein, Kingma et al. (2016) augments the flexibility of the posterior through autoregression over projections of stochastic latent variables. However, the problem of over pruning still persists: for instance, Kingma et al. (2016) enforces a minimum information constraint to ensure that all units are used.
Related is the research in unsupervised sparse overcomplete representations, especially with group sparsity constraints c.f. (Gregor et al., 2011; Jenatton et al., 2011). In the epitomic VAE, we have similar motivations that enable learning better generative models of data.
6 CONCLUSION
This paper introduces Epitomic VAE, an extension of variational autoencoders, to address the problem of model over-pruning, which has limited the generation capability of VAEs in high-dimensional spaces. Based on the intuition that subconcepts can be modeled with fewer dimensions than the full latent space, epitomic VAE models the latent space as multiple shared subspaces that have learned specializations. We show how this model addresses the model over-pruning problem in a principled manner, and present qualitative and quantitative analysis of how eVAE enables increased utilization of the model capacity to model greater data variability. We believe that modeling the latent space as multiple structured subspaces is a promising direction of work, and allows for increased effective capacity that has potential to be combined with methods for increasing the flexibility of posterior inference.
7 ACKNOWLEDGMENTS
We thank the reviewers for constructive comments. Thanks to helpful discussions with Marc’Aurelio Ranzato, Joost van Amersfoort and Ross Girshick. We also borrowed the term ‘epitome’ from an earlier work of Jojic et al. (2003).
8 APPENDIX
8.1 EFFECT OF KL WEIGHT λ ON RECONSTRUCTION
We visualize VAE reconstructions as the KL term weight λ is tuned down to keep latent units active. The top half of each figure are the original digits, and the bottom half are the corresponding reconstructions. While reconstruction performance is good, generation is poor (Fig. 1). This illustrates that VAE learns to model well only regions of the posterior manifold near training samples, instead of generalizing to model well the full posterior manifold.
8.2 EFFECT OF INCREASING LATENT DIMENSION ON RECONSTRUCTION
In § 4.1, Fig. 5 shows the effect of increasing latent dimension on generation for VAE, Dropout VAE, and eVAE models. Here we show the effect of the same factor on reconstruction quality for the models. The top half of each figure are the original digits, and the bottom half are the corresponding reconstructions. As the dimension of the latent variable z increases from 2-d to 20-d, reconstruction becomes very sharp (the best model), but generation degrades (Fig. 5). Dropout VAE has poorer reconstruction but still blurred generation, while eVAE is able to achieve both good reconstruction and generation.
8.3 EVALUATION METRIC FOR GENERATION
There have been multiple approaches for evaluation of variational autoencoders, in particular log-likelihood lower bound and log-density (using the Parzen window estimator, Rifai et al. (2012)). Here we show that for the generation task, log-density is a more appropriate measure than log-likelihood lower bound. Models are trained on binarized MNIST, to be consistent with literature reporting likelihood bounds. The encoder and decoder for all models consist of a single deterministic layer with 500 units.
Table 3 shows the log-likelihood bound and log-density for VAE and eVAE models as the dimensionD of latent variable z is increased. For VAE, as D increases, the likelihood bound improves, but the log-density decreases. Referring to the corresponding generation samples in Fig. 11, we see that sample quality in fact decreases, counter to the likelihood bound but consistent with log-density. The reported VAE bounds and sample quality also matches Figs. 2 and 5 in Kingma & Welling (2014). On the other hand, eVAE log-density first decreases and then improves with larger D. We see that this is also consistent with Fig. 11, where eVAE samples for D = 8 are the most interpretable overall, and D = 48 improves over D = 24 but still has some degenerate or washed out digits. (Note that these models are consistent with Kingma & Welling (2014) but are not the best-performing models reported in our experiments.) Since our work is motivated by the generation task, we therefore use log-density as the evaluation metric in our experiments.
Intuitively, the reason why VAE improves the likelihood bound but generation quality still decreases can be seen in the breakdown of the bound into the reconstruction and KL terms (Table 3 and Fig. 10). The improvement of the bound is due to large improvement in reconstruction, but the KL becomes significantly worse. This has a negative effect on generation, since the KL term is closely related to generation. On the other hand, eVAE reconstruction improves to a lesser extent, but the KL is also not as strongly affected, so generation ability remains stronger overall. As a result of this, simply tuning the KL weight λ in the training objective is insufficient to improve VAE generation, as shown in Fig. 1 in the main paper.

Review:
This paper replaces the Gaussian prior often used in a VAE with a group sparse prior. They modify the approximate posterior function so that it also generates group sparse samples. The development of novel forms for the generative model and inference process in VAEs is an active and important area of research. I don't believe the specific choice of prior proposed in this paper is very well motivated however. I believe several of the conceptual claims are incorrect. The experimental results are unconvincing, and I suspect compare log likelihoods in bits against competing algorithms in nats.
Some more detailed comments:
In Table 1, the log likelihoods reported for competing techniques are all in nats. The reported log likelihood of cVAE using 10K samples is not only higher than the likelihood of true data samples, but is also higher than the log likelihood that can be achieved by fitting a 10K k-means mixture model to the data (eg as done in "A note on the evaluation of generative models"). It should nearly impossible to outperform a 10K k-means mixture on Parzen estimation, which makes me extremely skeptical of these eVAE results. However, if you assume that the eVAE log likelihood is actually in bits, and multiply it by log 2 to convert to nats, then it corresponds to a totally believable log likelihood. Note that some Parzen window implementations report log likelihood in bits. Is this experiment comparing log likelihood in bits to competing log likelihoods in nats? (also, label units -- eg bits or nats -- in table)
It would be really, really, good to report and compare the variational lower bound on the log likelihood!! Alternatively, if you are concerned your bound is loose, you can use AIS to get a more exact measure of the log likelihood. Even if the Parzen window results are correct, Parzen estimates of log likelihood are extremely poor. They possess any drawback of log likelihood evaluation (which they approximate), and then have many additional drawbacks as well.
The MNIST sample quality does not appear to be visually competitive. Also -- it appears that the images are of the probability of activation for each pixel, rather than actual samples from the model. Samples would be more accurate, but either way make sure to describe what is shown in the figure.
There are no experiments on non-toy datasets.
I am still concerned about most of the issues I raised in my questions below. Briefly, some comments on the authors' response:
1. "minibatches are constructed to not only have a random subset of training examples but also be balanced w.r.t. to epitome assignment (Alg. 1, ln. 4)."
Nice! This makes me feel better about why all the epitomes will be used.
2. I don't think your response addresses why C_vae would trade off between data reconstruction and being factorial. The approximate posterior is factorial by construction -- there's nothing in C_vae that can make it more or less factorial.
3. "For C_vae to have zero contribution from the KL term of a particular z_d (in other words, that unit is deactivated), it has to have all the examples in the training set be deactivated (KL term of zero) for that unit"
This isn't true. A standard VAE can set the variance to 1 and the mean to 0 (KL term of 0) for some examples in the training set, and have non-zero KL for other training examples.
4. The VAE loss is trained on a lower bound on the log likelihood, though it does have a term that looks like reconstruction error. Naively, I would imagine that if it overfits, this would correspond to data samples becoming more likely under the generative model.
5/6. See Parzen concerns above. It's strange to train a binary model, and then treat it's probability of activation as a sample in a continuous space.
6. "we can only evaluate the model from its samples"
I don't believe this is true. You are training on a lower bound on the log likelihood, which immediately provides another method of quantitative evaluation. Additionally, you could use techniques such as AIS to compute the exact log likelihood.
7. I don't believe Parzen window evaluation is a better measure of model quality, even in terms of sample generation, than log likelihood.