VAE A PPROXIMATION E RROR : ELBO AND E XPONENTIAL F AMILIES
Anonymous authors
Paper under double-blind review
A BSTRACT
The importance of Variational Autoencoders reaches far beyond standalone generative models — the approach is also used for learning latent representations and can be generalized to semi-supervised learning.
This requires a thorough analysis of their commonly known shortcomings: posterior collapse and approximation errors.
This paper analyzes VAE approximation errors caused by the combination of the ELBO objective and encoder models from conditional exponential families, including, but not limited to, commonly used conditionally independent discrete and continuous models.
We characterize subclasses of generative models consistent with these encoder families.
We show that the ELBO optimizer is pulled away from the likelihood optimizer towards the consistent subset and study this effect experimentally.
Importantly, this subset can not be enlarged, and the respective error cannot be decreased, by considering deeper encoder/decoder networks.
I NTRODUCTION
Variational autoencoders proposed by Kingma & Welling (2014) strive at learning complex data distributions p d ( x ) , x ∈ X in a generative way.
They introduce latent variables z ∈ Z and model the joint distribution as p θ ( x
| z ) p ( z ) , where p ( z ) is a simple distribution that is usually assumed to be known.
The conditional distribution p θ ( x | z ) , called decoder , is modeled in terms of a deep network parametrized by θ ∈ Θ .
Models deﬁned in this way allow to sample from p θ ( x ) =
E p ( z ) p θ ( x | z ) easily, however at the price that computing the posterior p θ ( z | x ) =
p θ ( x | z ) p ( z ) /p
θ ( x ) is usually intractable.
To handle this problem, VAE approximates the posterior p θ ( z | x ) by an amortized inference encoder model q φ ( z | x ) parametrized by φ ∈ Φ .
Given the empirical data distribution p d ( x ) , the model is learned by maximizing the evidence lower bound (ELBO) of the data loglikelihood L ( θ ) =
E p
d log p θ ( x ) .
It can be expressed in the following two equivalent forms:
[Equation]
The ﬁrst form allows for stochastic optimization of ELBO while the second form shows that the gap between log-likelihood and ELBO is exactly the mismatch between the encoder and the posterior.
VAEs constitute a powerful deep learning extension of the expectation maximization approach to handle latent variables.
They are useful not only as generative models but also, e.g ., in semisupervised learning (Kingma et al., 2014; Mattei & Frellsen, 2019).
Furthermore the encoder part constructs an efﬁcient embedding of the data in the latent space, useful in many applications.
The outreach of the VAE approach requires therefore a careful empirical and theoretical analysis of the problems and trade offs involved.
The most important ones are (i) posterior collapse (He et al., 2019;
Lucas et al., 2019; Dai et al., 2018; Dai & Wipf, 2019) and (ii) approximation errors caused by an inappropriate choice of the encoder family.
The VAE approximation error has been studied ( e.g ., Cremer et al. 2018;
Hjelm et al. 2016;
Kim et al. 2018) so far mainly empirically.
The problem also occurs and is well-recognized in the context of variational inference and variational Bayesian inference, where the target posterior distribution is expected to be complex.
It is commonly understood, that the mean ﬁeld approximation of p θ ( z | x ) by q φ ( z | x ) in (1b) signiﬁcantly limits variational Bayesian inference.
In contrast, in VAEs, the decoder may adopt to compensate for the chosen encoder family.
The effect of this coupling, we
[Table]
[Figure]
believe, is not fully understood.
The phenomenon of decoder adopting to the posterior was experimentally observed, e.g ., by Cremer et al.
(2018, Section 5.4), noting that the approximation error is often dominated by the amortization error.
Turner & Sahani (2011, Sec. 1.4) analytically show for linear state space models that simpler variational approximations (such a mean-ﬁeld) can lead to less bias in parameter estimation than more complicated structured approximations.
Similarly, Shu et al.
(2018) view the VAE objective as providing a regularization and show that making the amortized inference model smoother, while increasing the amortization gap, leads to a better generalization.
The common (empirical) understanding of the importance of the gap between the approximate and the true posterior has led to many generalizations of standard VAEs, which achieve impressive practical results, notably, tighter bounds using importance weighting (Burda et al., 2016;
Nowozin, 2018), encoders employing normalizing ﬂows (Rezende & Mohamed, 2015; Kingma et al., 2016), hierarchical and autoregressive encoders (Vahdat & Kautz, 2020;
Sønderby et al., 2016; Ranganath et al., 2016), MRF encoders (Vahdat et al., 2020) and more.
While these extensions mitigate the posterior mismatch problem, they often come at a price of a more difﬁcult training and more expensive inference.
Furthermore, simpler encoders may be of practical interest.
Burda et al.
(2016, Appendix
C) illustrates that IWAE approximate posteriors are less regular and more spread out.
In contrast, factorized encoders provide simple embeddings useful for downstream tasks such as semantic hashing (Chaidaroon & Fang, 2017).
The aim of this paper is to study the approximation error of VAEs and its impact on the learned decoder.
We consider a setting that generalizes many common VAEs, in particular popular models where encoder and decoder are conditionally independent Bernoulli or Gaussian distributions: we assume that both decoder and encoder are conditional exponential families.
We identify the subclass of generative models where the encoder can model the posterior exactly, referred to as consistent
VAEs.
We give a characterization of consistent VAEs revealing that this set in fact does not depend on the complexity of the involved neural networks.
We further show that the ELBO optimizer is pulled towards this set away from the likelihood optimizer.
Specializing the characterization to several common VAE models, we show that the respective consistent models turn out to be RBMlike in many cases.
We experimentally investigate the detrimental effect in one case and show that a simpler but more consistent VAE can perform better in the other.
P ROBLEM S TATEMENT
We adopt the following notion of approximation error.
Consider a generative model class P Θ = { p θ ( x, z ) | θ ∈ Θ } , the encoder class Q Φ = { q φ ( z | x ) | φ ∈ Φ } and the data distribution p d ( x ) .
The maximum likelihood generative model is given by θ ML ∈ argmax θ
∈ Θ E p d ( x ) log p θ ( x ) .
For a decoder with parameters θ we deﬁne its approximation error as the likelihood difference L ( θ ML ) −
L ( θ ) .
Respectively, the VAE approximation error is deﬁned for a given θ as:
[Equation]
In order for this error to become zero, two conditions are necessary and sufﬁcient:
• Parameters ( θ, φ ) must be optimal for the ELBO objective.
• ELBO must be tight at ( θ, φ ) , i e ., L B ( θ, φ ) =
L ( θ ) .
Assuming that the optimality can be achieved, we study the non-tightness gap L ( θ ) − L B ( θ, φ ) .
From (1b) it expresses as E p d (cid:2) D KL ( q φ ( z | x ) (cid:107) p θ ( z | x )) (cid:3) .
It follows that ELBO is tight at ( θ, φ ) iff q φ ( z | x ) ≡ p θ ( z | x ) .
Hence, we deﬁne the consistent set Θ Φ ⊆ Θ as the subset of distributions p θ ( x, z ) whose posteriors are in Q Φ , i.e .,
[Equation]
The KL-divergence in the ELBO objective (1b) can vanish only if θ ∈ Θ Φ .
If the likelihood maximizer θ ML is not contained in Θ Φ , then this KL-divergence pulls the optimizer towards Θ Φ and away from θ ML as illustrated in Fig.
We characterize the consistent set Θ Φ , on which the bound is tight, and show that this set is quite narrow and does not depend on the complexity of the encoder and decoder networks beyond simple 1-layer linear mappings of sufﬁcient statistics.
T HEORETICAL A NALYSIS
We consider a general class of VAEs, where both encoder and decoder are deﬁned as exponential families.
This class includes many common models, in particular Gaussian VAEs and Bernoulli
VAEs with conditional independence assumptions, but also more complex ones, e.g ., where the encoder is a conditional random ﬁeld (Vahdat et al., 2020) 1 .
Assumption 1 (Exponential family VAE) .
Let X and Z be sets of observations and latent variables, respectively.
We consider VAE models deﬁned by
[Equation]
where ν : X → R n and ψ : Z → R m are ﬁxed sufﬁcient statistics of dimensionality n and m ; f θ : Z → R n and g φ : X → R m are the the decoder, resp., encoder networks, depending on learnable parameters θ , resp.
φ ; h : X → R + , h (cid:48) : Z → R + are strictly positive base measures and A , B denote the respective log-partition functions.
Notice that this assumption imposes no restrictions on the nature of random variables x and z .
They can be discrete or continuous, univariate or multivariate.
Similarly, it imposes no restrictions on the complexity of the decoder and encoder networks f θ ( z ) and g φ ( x ) .
Characterization of the consistent set.
In the ﬁrst step of our analysis, we investigate the conditions under which the approximation error of an exponential family VAE can be made exactly zero.
As discussed above, a tight VAE ( θ, φ ) must satisfy ∀ ( x, z )
q φ ( z | x ) =
p θ ( z | x ) , which leads to the following theorem.
Theorem 1.
The consistent set Θ Φ of an exponential family VAE is given by decoders of the form
[Equation]
(5) where W is a n × m matrix and u ∈ R n .
Moreover, the corresponding encoders have the form
[Equation]
where v ∈ R m .
This is a direct consequence of a theorem by Arnold & Strauss (1991) (see Appendix A.1 for more details).
For a tight VAE, Theorem 1 states that the decoder and encoder take the form (5) and (6) with the interaction between x and z parametrized by a single matrix W instead of the complex neural networks with parameters θ, φ .
Thus they turn out to be generalized linear models (GLMs).
The corresponding joint probability distribution is an EF Harmonium (Welling et al., 2005):
[Equation]
Corollary 1.
The subset Θ Φ of consistent models can not be enlarged by considering more complex encoder networks g ( x ) , provided that the afﬁne family W T ν ( x ) can already be represented.
Corollary 2.
Let the decoder network family be afﬁne in ψ ( z ) , i.e .,
f
( z ) =
Wψ ( z ) + a and let the encoder network family g ( x ) include at least all afﬁne maps V ν
( x ) + b .
Then any global optimum of ELBO attains a zero approximation error.
VAEs can escape consistency when they degenerate to an invertible ﬂow.
In practice, VAE models are almost never tight.
It is therefore natural to ask, whether a small VAE posterior mismatch error implies closeness of the optimal decoder to some decoder in the consistent set.
Deﬁnition 1.
A VAE ( p θ , q φ ) is ε -tight for some ε > 0 if E p d ( x ) [ D KL ( q φ ( z | x ) (cid:107) p θ ( z | x ))] ≤ ε .
It turns out that the the KL divergence has a leak, which allows a VAE to approach tightness while not approaching consistency.
In the continuous case an example satisfying ε -tightness with non-linear decoder follows from Dai & Wipf (2019, Theorem 2).
They show for a class of Gaussian VAEs with general neural networks f θ , g φ , that it is possible to build a sequence of network parameters θ t , φ t with the following properties: i) the target distribution is approximated arbitrary well, ii) the posterior mismatch D KL ( q φ t ( z | x ) (cid:107) p θ t ( z | x )) approaches zero and iii) both the encoder and decoder approach deterministic mappings.
The VAE thus approaches a ﬂow model (or invertible neural network) between the data manifold and a subspace of the latent space (Dai & Wipf, 2019).
Clearly, in a general case the ﬂow must be non-linear.
A similar case can be made for discrete variables, see Example A.1.
Non-deterministic nearly-tight VAEs approach consistency.
We would however argue that the mode where the decoder and encoder are nearly-deterministic is not a natural VAE solution.
By making additional assumptions, excluding such deterministic solutions, and restricting ourselves to the ﬁnite space in order to simplify the analysis, we can show that the decoder of an ε -tight VAE does indeed approach a GLM.
Theorem 2.
Let ( p θ , q φ ) be an exponential family VAE (Assumption 1) on a discrete space
X ×
Z with strictly positive encoder q φ ( z | x ) and decoder posterior p θ ( z | x ) , both bounded from below by α > 0 .
If the VAE is ε -tight, then there exists W ∈ R n,m such that the decoder can be approximated by a GLM of the form ˜ p ( x | z ) =
˜ h
( x ) exp (cid:2) (cid:104) ν ( x ) ,
Wψ ( z ) (cid:105) − ˜
A ( z ) (cid:3) with the error
[Equation]
The proof is given in Appendix A.3.
3. C ASES A NALYSIS
This subsection gives a detailed analysis of consistent VAE models in several concrete cases of practical interest.
Diagonal Gaussian VAE Let us consider a Gaussian VAE, as commonly applied to image generation ( e.g ., Dai & Wipf (2019)).
Let X
= R n , Z = R m , p ( x | z ) =
N ( x | µ d ( z ) , σ 2 d I ) , q ( z | x ) =
N ( z | µ e ( x ) , diag( σ 2 e ( x ))) , where µ d , µ e and σ e are neural networks and σ d is a common pixel observation noise parameter.
The decoder has minimal sufﬁcient statistics ν ( x ) = x and the base measure h ( x ) =
N ( x | 0 , σ 2 d I ) .
The encoder has minimal sufﬁcient statistics ψ ( z )
=
( z, z 2 ) , where the square is coordinate-wise.
Theorem 1 implies that a tight optimal VAE has the joint model
[Equation]
for some matrices W , V and vectors a, b, c .
Furthermore, the integral over z must be ﬁnite for all x and therefore x T V + c < 0 must hold for all x ∈ R n .
This is possible only if V = 0 and c < 0 .
The joint distribution is therefore a multivariate Gaussian and the same holds for its marginal p ( x ) .
The neural network µ d ( z ) must degenerate to µ d ( z ) =
σ 2 d · ( Wz + a ) and the two neural networks for the encoder to σ 2 e ( x ) = − 1 / 2 c and µ e ( x ) = − ( W T x + b ) / 2 c , where divisions are coordinate-wise.
VAEs with such simpliﬁed, linear Gaussian encoder-decoder pairs, called "linear
VAEs" (Lucas et al., 2019) are known to be consistent and to match the probabilistic PCA model (Dai et al., 2018; Lucas et al., 2019).
In this context, our Corollary 2 is a generalization of (Lucas et al.,
2019, Lemma 1) showing consistency of linear VAEs, to decoders in any exponential family with natural parameters being a linear mapping of any ﬁxed lifted latent representation ψ ( z ) .
We argue that a joint Gaussian model is too simplistic to generate complex data such as realistic images and that in this case the VAE error is detrimental.
In Section 4.2 we experimentally conﬁrm that optimizing ELBO for a general decoder network µ d causes qualitative and quantitative degradation relative to the ML decoder.
Note that if we allowed σ d to be dependent on z , the resulting joint statistics in ν ⊗ ψ would include terms x 2 z , x 2 z 2 .
The joint distribution would not be Gaussian and may be in fact multi-modal (see Anil Bhattacharayya’s distribution in Arnold et al. 2001).
Bernoulli-MRF VAE Vahdat et al.
(2020) proposed to consider encoders in the Markov Random
Field (MRF) family, in particular encoders of the form q ( z | x ) ∝ exp( (cid:104)
z 1 , V ( x ) z 2 (cid:105) + (cid:104) b
1 ( x ) , z 1 (cid:105) + (cid:104) b
2 ( x ) , z 2 (cid:105) ) , where z 1 , z 2 are two groups of latent variables and interaction weights V , b 1 , b 2 are computed by the encoder network.
In this case q ( z | x ) is itself a (conditional) RBM.
While evaluating q ( z | x ) is difﬁcult, MCMC sampling is efﬁcient.
We assume binary observations x and a conditionally independent Bernoulli decoder family as above.
The decoder thus has sufﬁcient statistics
ν
e = ( x ) and the encoder has ψ = ( z 1 , z 2 , z 1 ⊗ z 2 ) .
Introducing homogeneous constant components x
=
z
10 =
z 20 = 1 , the family of consistent joint distributions can be compactly described as
[Equation]
where the summation in all indices starts from 0 and − W 0 , 0 , 0 is the log-partition function.
This joint model is a higher order MRF with the highest order potentials given by cubic monomials.
Standard Bernoulli VAEs are a special case of the Bernoulli-MRF model, obtained when the interaction weights V are zero.
The joint distribution of such tight optimal VAEs takes the form
[Equation]
which is a restricted Boltzmann machine (RBM).
Since RBMs are well known for being useful in many applications (dimensionality reduction, collaborative ﬁltering, feature learning, topic modeling), we hypothesize that they can make a good baseline for Bernoulli VAEs and furthermore that the effect of pulling the VAE solution towards an RBM may be benign in case of insufﬁcient data.
For example IWAE test likelihood in (Burda et al., 2016) is worse than that of an RBM (Burda et al., 2015) on the Omniglot dataset.
Further debiasing of the IWAE may not improve test likelihood (Nowozin, 2018).
Bernoulli VAE for Semantic Hashing One important application of Bernoulli VAEs is the semantic hashing problem, initially proposed and modeled with RBMs (Salakhutdinov & Hinton, 2009).
The problem is to assign to each document / image a compact binary latent code that can be used for quick retrieval by the nearest neighbor search.
We will detail now a more recent VAE model for text documents (Chaidaroon & Fang, 2017; Shen et al., 2018) and show that it can be tight only in a full posterior collapse.
We correct the encoder so as to allow a larger consistent set and observe that the resulting consistent joint distribution forms a multinomial-Bernoulli RBM.
Let x ∈ N K be word counts in a document with words from a dictionary of size K .
Let z ∈ { 0 , 1 } m be a binary latent code.
Let l = (cid:80) k x k denote the document’s length.
We assume that the document length is independent of the latent topic and its distribution p ( l ) can be learned separately ( e.g ., a log-normal distribution is a good ﬁt).
The decoder is deﬁned using the multinomial distribution model (words in the document are drawn from the same categorical distribution corresponding to its topic):
[Equation]
where f ( z ) is a neural network mapping the latent code to the logits of word occurrence probabilities, A ( η ) = log (cid:80) k
exp( η k ) and h ( x | l ) =
[[ (cid:80) k x k = l ]] (cid:0) l ! / (cid:81) k x k ! (cid:1) is the base measure 2 .
The sufﬁcient statistics are the word counts x .
The prior p ( z ) is assumed uniform Bernoulli.
The encoder is the conditionally independent Bernoulli model, expressed as
[Equation]
[Figure]
where g ( x ) is the encoder network.
Chaidaroon & Fang (2017) experimented with the encoder and decoder design and recommended using TFIDF features instead of raw counts.
First, we note that the inverse document frequency (IDF) is not relevant, since it can be learned by the ﬁrst linear transform in the encoder.
Effectively, the term frequency (TF), given by x/l , is used.
This choice is adopted in later works (Shen et al., 2018; Zamani Dadaneh et al., 2020; Ñanculef et al., 2020).
It might seem reasonable that the latent code modeling the document topic should not depend on the document length, only on the distribution of words in the document.
However, we will argue that this rationale is misleading for stochastic encoders.
We apply Theorem 1 to two groups of variables: observed ( x, l ) and latent z with h ( x, l )
=
h ( x | l ) p ( l ) , ν ( x, l ) = x and ψ ( z ) =
z .
It follows that the consistent joint family is
[Equation]
This however implies that g ( x ) =
Wx + b , i.e . the encoder network must be linear in x .
Consequently, it cannot match a function of word frequencies x/l (as chosen by design) unless W = 0 , i.e . a completely trivial model with an encoder not depending on x .
Such an encoder would imply full posterior collapse.
The corresponding consistent set Θ Φ coincides with the set of collapsed VAEs where the decoder does not depend on the latent variable z in Fig.
We conjecture that the inherent inconsistency of this VAE has a detrimental effect on learning.
If instead, we let the encoder network to access word counts x directly, we obtain that g ( x ) =
Wx + b can form a consistent VAE.
Inspecting this encoder model in more detail, we see that it builds up topic conﬁdence in proportion to the evidence (total word counts), as the true posterior would.
Indeed, the true posterior p ( z | x, l ) satisﬁes the factorization by Bayes’s theorem: p ( z | x, l ) = p ( x | z, l ) p ( z ) /p
( x | l ) .
The prior p ( z ) is constant by design, p ( x | l ) does not vary with z and p ( x | z, l ) factors over all word instances according to (12).
In other words, the coupling between x and z in log p ( z | x, l ) is linear in x .
In Section 4 we study the proposed correction experimentally and show that it enables learning better models under a variety of settings.
E XPERIMENTS 4. A RTIFICIAL E XAMPLE
To start with, we illustrate our ﬁndings on a toy example.
We consider a simple Gaussian mixture model for which we can easily generate samples and compute all necessary quantities including the ELBO objective.
We deﬁne the ground truth model to be p ∗ ( x, z ) =
p ∗ ( z ) p ∗ ( x | z ) , with z ∈ { 1 . . .
4 } , p ∗ ( z ) ≡ 0 .
25 , x ∈ R 2 , p ∗ ( x | z ) =
N ( x | µ ( z ) , σ 2 I ) , i.e . a mixture of four 2D Gaussians.
Fig. 2(a) shows the color-coded posterior distribution p ∗ ( z | x ) .
We assign a color to each component and represent p ∗ ( z | x ) for each pixel x ∈ R 2 by the corresponding mixture of the component colors.
For better interpretability, we illustrate further results by decision maps arg max z p ( z | x ) .
Fig. 2(b) shows the decision map for p ∗ ( z | x ) .
The aim of the experiment is to learn a VAE and to study the inﬂuence of the factorization assumption on the results.
We use the decoder architecture as in the ground truth model — a Gaussian distribution p θ ( x | z ) =
N ( x | θz oh , σ 2 I ) , where z oh is the one-hot (categorical) representation of z , and θ is a 2 × 4 matrix that maps the four latent codes to 2D centers at general locations.
Note that the ground truth model is contained in the chosen decoder family.
Hence, the ML solution is the ground truth decoder p ∗ ( x | z ) .
We restrict the encoder to factor over the binary representation of the code z b ∈ { 0 , 1 } 2 and deﬁne it as q φ ( z b | x ) ∝ exp (cid:104) g φ ( x ) , z b (cid:105) , where g φ ( x ) is implemented as a feed-forward network with two hidden layers, each with 64 units and ReLU activations.
First, we pre-train our factorized encoder by optimizing ELBO and keeping the ground truth decoder ﬁxed.
The next step is to jointly train the encoder and decoder by maximizing ELBO.
Since we are interested how the ELBO objective distorts the likelihood solution, we start with the ground truth decoder and the pre-trained encoder from the previous step.
The ELBO-optimal decoder has to match not only the training data, but also the inexact, factorizing encoder.
The resulting q φ ( z | x ) is shown in Fig. 2(c) and the learned model posterior p θ ( z | x ) ∝ p ( z ) p θ ( x | z ) in Fig. 2(d).
Note that they match each other pretty well, but differ substantially from the ground truth posterior shown in
Fig. 2(b).
The impact of the factorization is clearly visible – one can see two decision boundaries (one for each bit of z b ), which together partition the x -space into four regions, approximating the true posterior.
For comparison, Fig. 2(e) shows the posterior of an RBM trained on the same data.
It is clearly seen that the ELBO optimizer is pulled away from the likelihood optimizer towards an
RBM solution.
Notice also the explanation given in Fig. 2(f).
The approximation error (2) in the decoder caused by using the factorized encoder is only 0 .
03 nats (see Appendix B.1), but the qualitative difference in Fig. 2 appears detrimental.
Summarizing, this simple toy example clearly shows the VAE approximation error caused by the combination of ELBO objective and the factorization assumption for the encoder.
4. G AUSSIAN VAE S FOR C ELEB A I MAGES
The goal and design of this experiment is similar to the previous one.
We ﬁrst deﬁne a ground truth decoder which is used to generate training images.
Then we pre-train an encoder by ELBO keeping the ground truth decoder ﬁxed.
Finally, we train both model parts starting from the ground truth decoder and the pre-trained encoder.
The ground truth generative model is obtained by training a convolutional Generative Adversarial network (GAN) using code of Inkawhich (2017) on the CelebA dataset Liu et al.
We scale and crop all images to 64 × 64 pixels.
In order to get a stochastic decoder, we equip the GAN generator x = d ( z ) , z ∈ R 100 , x ∈ R 64 × 64 × 3 with image noise σ d .
The ground truth generative model is thus deﬁned as p ∗ ( x, z ) =
p ∗ ( z ) p ∗ ( x | z ) , where p ∗ ( z ) =
N ( z | 0 , I ) , p ∗ ( x | z ) =
N ( x | µ d ( z ) , σ 2 d I ) , and σ 2 d is a common noise variance for all pixels and color channels.
We chose σ d = 0 .
(the color values are normalized to [ − 1 , 1] ).
This corresponds to an image noise level, which is just visible, but does not disturb visual perception essentially.
The decoder family of the considered VAE consists of networks with the same architecture as d ( z ) .
This ensures that the ground truth decoder is a likelihood maximizer of the VAE model.
The encoder is deﬁned as q φ ( z | x ) =
N ( x | µ e ( x ) , diag( σ 2 e ( x ))) , where µ e , σ e ∈ R 100 are two outputs of a convolutional neural network with an architecture similar to the architecture of the discriminator used for training the GAN (except the output layer), i.e ., q φ ( z | x ) is a multivariate Gaussian with diagonal covariance matrix whose parameters depend on x .
We pre-train the encoder fully supervised by maximizing its conditional log-likelihood E p ∗ ( x,z ) log q φ ( z | x ) on examples drawn from the ground truth generating model p ∗ ( x, z ) .
The results of pre-training are shown in Fig.
Then we jointly learn the encoder and decoder by maximizing ELBO on x -samples drawn from the ground truth model.
We start the learning with the ground truth decoder p ∗ ( x | z ) and the encoder obtained in the previous step.
Two variants are considered for this training: (i) keeping the image noise σ d ﬁxed and (ii) learning it along with other model parameters.
We evaluate the results quantitatively by computing the Frechet Inception Distances (FID) between the ground truth model p ∗ ( x | z ) and the obtained decoders p θ ( x | z ) using the code of Seitzer (2020).
For this we generate 200k images from each model.
The obtained values are given in Tab. 1.
Fig.
4 shows images generated by the ground truth model and the two learned models.
To conclude, ELBO optimization harms the decoder considerably as clearly seen both from FIDscores and the generated images.
Models with higher ELBO values have worse FID-scores and produce less realistic images.
Experiment
[Table]
optimize encoder and decoder (ELBO, ﬁxed σ d )
[Table]
optimize encoder and decoder (ELBO, learned σ d )
[Table]
ERNOULLI VAE FOR T EXT D OCUMENTS
This experiment compares training of the VAE model for semantic hashing discussed in Section 3. with and without our proposed correction on the 20Newsgroups dataset (Lang & Rennie, 2008).
We describe the dataset, preprocessing and optimization details in Appendix B.2.
We compare three encoders: e1 : linear encoder on word counts (the proposed correction), e2 : deep (2 hidden layers) encoder using word frequencies and e3 : a linear encoder on frequencies.
The encoders are compared across different numbers of latent Bernoulli variables (bits) and different decoder depths.
The decoder depth denotes the number of fully connected hidden ReLU layers (02).
In both the encoder and decoder we use 512 units in hidden layers.
The prior work mainly used linear decoders following the ablation study of Shen et al.
Our experiments also suggest that using deep decoders in combination with longer bit-length leads to a signiﬁcant overﬁtting.
When the decoder is linear, the posterior distribution is tractable and is linear as well, i.e ., the VAE model is equivalent to a Multinomial-Bernoulli RBM.
We experimentally verify that a linear decoder indeed works better in this case.
However, perhaps more surprisingly, we also ﬁnd out that it works better even for non-linear decoders.
Table 2 show the achieved training and test Negative ELBO (NELBO) values.
We observe across all settings that the simple encoder e1 is consistently better than the more complex encoder e2 which in turn is signiﬁcantly better than the linear encoder on frequencies e3.
We conclude that the use of VAEs with deep encoders based on word frequencies (Chaidaroon & Fang, 2017; Shen et al., 2018; Zamani Dadaneh et al., 2020; Ñanculef et al., 2020) is sub-optimal for this dataset.
We do not expect additional regularization techniques proposed in these works to change this conclusion.
We also observe that linear decoders generalize better under 32 and 64 bits compared to more complex decoders, which suffer from overﬁtting.
This implies that in these cases the best encoder-decoder combination is linear, i.e . the basic RBM model.
[Table]
C ONCLUSIONS
We have analyzed the approximation error of VAEs in a general setting, when both the decoder and encoder are exponential families.
This includes commonly used VAE variants as, e [Table] g ., Gaussian
VAEs and Bernoulli VAEs.
We have shown that the subset of generative models consistent with the encoder class is quite restricted: we concisely characterize it as bilinear models on the sufﬁcient statistics of both decoder and encoder, i.e ., RBM-like models.
This connection between tight
VAEs and RBMs / EF Harmoniums was not previously known.
The consistent subset can not be enlarged by using more complex encoder networks as long as encoder’s sufﬁcient statistics remain unchanged.
In combination with the ELBO objective, this causes an approximation error — the
ELBO optimizer is pulled away from the data likelihood optimizer towards this subset.
Moreover, we proved theoretically that close-to-tight VAEs have decoders close to GLMs.
We have shown that the error is detrimental when the consistent subset is too restrictive.
On the other hand it may result in a useful regularization when it contains more expressive models.
In the cases where a lot of data is available and a high quality generative model is of the primary interest, more expressive encoder families are required in addition to large networks.
However, in cases when the learned representations are of interest, models with simple encoders can be used such as in the case of the Text VAE model in our example and their performance can be improved by considering consistency, i.e ., adjusting the encoder and decoder so that the consistent set is large enough.
We restricted this study to exponential families.
While in richer models discussed in the introduction, the approximation error still exists, it is made small by design, and it is less relevant and harder to analyze it theoretically.
One possible open direction where such analysis would make sense is to consider fully factorized non-exponential cases, e.g ., Student-t VAEs (Takahashi et al., 2018) or models satisfying hierarchical or partial factorization (Maalø e et al., 2019).
A P ROOFS A. P ROOF OF T HEOREM
The proof directly follows from the characterization of conditionally speciﬁed joint distributions in the exponential family given by Arnold & Strauss (1991), see (Arnold et al., 2001, Theorem 3):
Theorem A.1 (Arnold & Strauss 1991) .
Let x ∈ X and z ∈ Z be random variables with a strictly positive joint distribution such that both conditional distributions are exponential families with densities
[Equation]
where ν : X → R n and ψ : Z → R m are minimal sufﬁcient statistics, f : Z → R n and g :
X → R m are any mappings, h ( x ) and h (cid:48)
( z ) are base measures, and A and B denote the respective log-partition functions 4 .
Then there exists a matrix W ∈ M ( n + 1 , m + 1) , such that the density of the joint distribution can be represented as
[Equation]
where ν e , ψ e denote the statistics vectors extended with an additional component 1.
A. E XAMPLE OF A DISCRETE VAE APPROACHING A FLOW
Example A.1.
In this example we construct a VAE that can be arbitrary close to tight one, but where the decoder network does not approach a linear map.
Let X = {− 1 , 1 } 2 , Z = {− 1 , 1 } 2 and let p ( z ) be uniform.
Let the decoder be conditionally independent
[Equation]
where π ( z ) denotes the invertible mapping of Z to X given by
[Equation]
If the parameter β is sufﬁciently large, the distribution p ( x | z ) approaches the deterministic distribution δ x
= π ( z ) .
Its posterior therefore also approaches the deterministic distribution δ z
= π − 1 ( x ) (notice that π − 1 = π ).
Let the encoder be the conditionally independent model
q ( z | x ) = exp (cid:2) β (cid:
) z, π − 1 ( x ) (cid:105) − A ( x ) (cid:3) .
This decoder by design approaches δ x = π ( z ) as well and thus the VAE ( p, q ) achieves ε -tightness for sufﬁciently large β .
At the same time the deviation between logarithms of probabilities log q ( z | x ) and log p ( z | x ) grows with β .
A. P ROOF OF T HEOREM
The idea of the proof is to bound the difference between log p ( z | x ) and log q ( z | x ) , which is done by Proposition A.2 and then in the space of log-probabilities to approximate the non-linear mapping f e ( z ) by a linear one as detailed in Proposition A.1.
Choosing the norms and the approximation appropriately allows to construct a bound, which despite the discreteness assumption about the observation space X in Theorem 2 does not depend on its cardinality.
For a ﬁnite set X ⊂ X let H be the | X | -dimensional vector space with the inner product (cid:104) u, v (cid:105)
p d = (cid:80) x ∈ X p d ( x ) u ( x ) v ( x ) , assuming that p d ( x ) > 0 for all x ∈ X .
The respective norm will be denoted as (cid:107) · (cid:107) p d .
Proposition A.1.
Under model Assumption 1, for any ﬁnite X ⊆
X there exists a matrix W ∈ R n,m and a GLM
[Equation]
such that the following inequality holds for all z ∈ Z :
[Equation]
That is, if the RHS is small, the decoder network f ( z ) can be effectively replaced by a linear model
Wψ ( z ) .
Proof.
For clarity, we will omit the dependence of the decoder and encoder on their parameters θ , resp.
φ .
Throughout the proof we will also assume that a single z ∈ Z is ﬁxed.
First, we expand
[Equation]
where A ( z ) = A ( f ( z )) and B ( x ) =
B ( g ( x )) .
We can therefore represent
[Equation]
where
[Equation]
With this representation we have:
[Equation]
Let V be the matrix with rows
ν e ( x ) for all x ∈ X .
Let G be the matrix with rows g e ( x ) for all x ∈ X .
We can rewrite the condition (30) in the form
[Equation]
32b
[Equation]
where ξ ∈ R | X | is the vector of residuals.
Let P be the orthogonal projection onto the range of V in the space H .
Multiplying (32a) by P on the left, we obtain
[Equation]
Because PV = V we obtain
[Equation]
We therefore can consider the decoder network approximation ˜
f e ( z ) = ˜
Wψ e ( z ) , where ˜ W is the solution to the consistent system of linear equations V ˜ W = PG and is independent of z .
We can therefore express
[Equation]
where the inequality holds because P is an orthogonal projection in H .
We obtained that the decoder network f e ( z ) can be approximated by a linear mapping ˜
Wψ e ( z ) such that
[Equation]
Expressing back
[Equation]
We obtain that
[Equation]
where
[Equation]
Proposition A.2.
Under model Assumption 1, let X and Z be discrete (ﬁnite) sets and let z ∈ Z be chosen.
If q ( z | x ) ≥ α and p ( z | x ) ≥ α for all x ∈ X , where X ⊆ X and [Equation]
[Equation]
then
[Equation]
Proof.
Let us denote ε ( x ) =
D KL ( q ( z | x ) (cid:107) p ( z | x )) .
Pinsker’s inequality assures for each x
[Equation]
Substituting S = { z } we obtain
[Equation]
By taking expectation in p d ( x ) on both sides we obtain a variant of Pinsker’s inequality:
[Equation]
Notice that the LHS depends on the given z .
Because all summands are non-negative it follows that
[Equation]
This inequality will be used in several places below
[Equation]
Consider x ∈ X such that p ( z | x ) >
q ( z | x ) .
Then
[Equation]
where the inequality holds because log 2 is monotonously increasing for arguments greater equal than 1 , which is ensured.
Let us now consider x ∈ X such that p ( z | x ) <
q ( z | x ) .
Then
[Equation]
In total, we obtain
[Equation]
Let us denote u ( x ) = | p ( z |
[Equation]
− q [Equation] | α and partition the set X into two parts:
[Equation]
where we chose u 0 ∈ [ √ e − 1 , 5] for reasons to be clariﬁed below.
For both parts, i.e . k
= 1 , 2 , we have
[Equation]
For X 1 , (51) can be further bounded as
[Equation]
where the ﬁrst inequality holds for u 0 ≤ 5 , because in this case log 2 (1+ u ) ≤ log(1+ u 2 ) holds, the second inequality is the Jensen’s inequality for log and the last inequality uses (46) under monotone log .
For X 2 we have the following.
Let V = p d ( X 2 )
= (cid:80) x ∈ X
2 p d ( x ) .
We can express
[Equation]
Using that u < u 2 on X 2 and that log 2 (1 + u ) is monotone for a positive argument
[Equation]
we can bound (53b) as
[Equation]
Further
[Equation]
using that log 2 (1 + v ) is concave on X 2 for v ≥ e −
[Equation]
we have
[Equation]
where in the last step we used (46)
[Equation]
Next we show that V itself is bounded above by ε 2 u 20 α 2 .
It follows from
[Equation]
where the last inequality is again (46).
Now, letting r =
ε
α 2 V ≥ u 20 ≥ 1 we can bound (55c) as follows
[Equation]
Theorem 2 follows by Proposition A.2 and Proposition A.1 with the choice X
= X .
B D ETAILS OF E XPERIMENTAL S ETUP B. A RTIFICIAL E XAMPLE
We give here the achieved likelihood and ELBO values for this experiment.
The negative entropy (cid:80) x p ∗ ( x ) log p ∗ ( x ) of the ground truth model, i.e ., the best reachable data log-likelihood, is − 3 . 65 .
The ELBO of the pre-trained VAE is − 3 . 74 .
During the second step of training, i.e ., the joint learning of the encoder and decoder by ELBO maximization, the ELBO value increases from − 3 .to − 3 .
The data log-likelihood (cid:80) x p ∗ ( x ) log p θ ( x ) drops at the same time from − 3 .
65 to − 3 .
The approximation error (2) in the decoder caused by using the factorized encoder is only 0 .
03 nats, but the qualitative difference between the ground truth model and the ELBO optimizer model shown in Fig.
2 appears detrimental.
B. B ERNOULLI VAE FOR T EXT D OCUMENTS
Dataset In this experiment we used the version of the 20Newsgroups data set (Lang & Rennie, 2008) denoted as “processed” by the authors.
The dataset contains bag-of-words representations of documents and is split into a training set with 11269 documents and a test set with 7505 documents.
We keep only the 10000 most frequent words in the training set, which is a common pre-processing (each of the omitted words occurs not more than in 10 documents).
Optimization To train VAE we used the state-of-the-art unbiased gradient estimator ARM (Yin &
Zhou, 2019) and Adam optimizer with learning rate 0 .
We did not use the test set for parameter selection.
We train for 1000 epochs using 1-sample ARM and then for 500 more epochs using 10 samples for computing each gradient estimate with ARM.
We report the lowest negative ELBO (NELBO) values for the training set and test set during all epochs.
Models The decoder model (12) describes words as independent draws from a categorical distribution speciﬁed by the neural network f ( z ) .
This network respectively has a structure
[Equation]
The input dimension equals to the number of latent bits, the output dimension equals the number of words in the dictionary, 10000.
For decoders with dhidden = 1 , 2 the hidden layers containedunits.
The encoder networks e2 , e3 take on the input word frequencies x/ (cid:80) k x k , the encoder network e takes on input word counts x .
For the deep encoder e2 we used 2 hidden ReLU layers with 512 units each.