Embrace the Gap: VAEs Perform Independent Mechanism Analysis
Abstract
Variational autoencoders (VAEs) are a popular framework for modeling complexdata distributions; they can be efficiently trained via variational inference bymaximizing the evidence lower bound (ELBO), at the expense of a gap to theexact (log-)marginal likelihood.
While VAEs are commonly used for disentangledrepresentation learning, it is unclear why ELBO maximization would yield suchrepresentations, since unregularized maximum likelihood estimation generallycannot invert the data-generating process without additional assumptions.
Yet,
VAEs often succeed at this task.
We seek to elucidate this apparent paradox bystudying nonlinear VAEs in the limit of near-deterministic decoders.
We first provethat, in this regime, the optimal encoder approximately inverts the decoder—acommonly used but unproven conjecture—which we refer to as self-consistency .
Leveraging self-consistency, we show that the ELBO converges to a regularizedlog-likelihood.
This allows VAEs to perform what has recently been termedindependent mechanism analysis (IMA): it adds an inductive bias towards decoderswith column-orthogonal Jacobians, which helps recovering the true latent factors.
The gap between ELBO and log-likelihood is therefore welcome, since it bearsunanticipated benefits for nonlinear representation learning.
In experiments onsynthetic and image data, we show that VAEs uncover the true latent factors whenthe data generating process satisfies the IMA assumption.
Introduction
Latent Variable Models (LVMs) allow to effectively approximate a complex data distribution andto sample from it [3, 45].
Deep LVMs employ a neural network (the decoder or generator ) toparameterize the conditional distribution of the observations given latent variables, which are typicallyassumed to be independent.
However, Maximum Likelihood Estimation (MLE) of the modelparameters is computationally intractable.
In Variational Autoencoders (VAEs) [33, 53], the exactlog-likelihood is substituted with a tractable lower bound, the evidence lower bound (ELBO).
Thisobjective introduces an approximate posterior of the latents given the observations (the encoder ) froma suitable variational distribution whose mean and covariance are parametrized by neural networks.
The encoder is introduced to efficiently train a deep LVM: however, it is not explicitly designed toextract useful representations [15, 55].
Nonetheless, VAEs and their variants are widely used in representation learning [23, 1], wherethey often recover semantically meaningful representations [37, 7, 32, 4].
Our understandingof this empirical success is still incomplete, since (deep) LVMs with independent latents arenonidentifiable from i.i.d.
data [27, 39]; different models fitting the data equally well may yieldarbitrarily different representations, thus rendering the recovery of a ground truth generative modelimpossible.
While auxiliary variables, weak supervision [26, 29, 19, 40, 67, 17], or specific modelconstraints [27, 63, 64, 24, 21] can help identifiability, the mechanism through which the ELBOmay enforce a useful inductive bias remains unclear, despite recent efforts [4, 54, 36, 13, 66].
[Figure]
In this work, we investigate the benefits of optimizing the ELBO for representation learning byanalyzing VAEs in a near-deterministic limit for the conditional distribution parametrized by thenonlinear decoder.
Our first result concerns the encoder’s optimality in this regime.
Previous worksrelied on the intuitive assumption that the encoder inverts the decoder in the optimum [47, 36, 66];we formalize this self-consistency assumption and prove its validity for the optimal variationalposterior in the near-deterministic nonlinear regime.
Using self-consistency, we show that the ELBO tends to a regularized log-likelihood—rather than tothe exact one as conjectured in previous work [47].
The regularization term allows VAEs to performwhat has been termed Independent Mechanism Analysis (IMA) [21]: it encourages column orthogonality of the decoder’s Jacobian.
This generalizes previous findings based on linearizations or approximations of the ELBO [54, 41, 36], and allows us to characterize the gap w.r.t.
the log-likelihood in thedeterministic limit.
Our results elucidate the gap between ELBO and exact log-likelihood as a possiblemechanism through which the ELBO implements a useful inductive bias.
Unlike the unregularizedlog-likelihood, the IMA-regularized objective can help invert the data generating process undersuitable assumptions [21].
We verify this by training VAEs in experiments on synthetic and imagedata, showing that they can recover the ground truth factors when the IMA assumptions are met.
The contributions of this paper can be summarized as follows:
• we characterize and prove self-consistency of VAEs in the near-deterministic regime (i
e., when thedecoder variance tends to zero), justifying its usage in previous works (§ 3.1);• we show that under self-consistency, the ELBO converges to a regularized log-likelihood (§ 3.2),and discuss its possible role as a useful inductive bias in representation learning;• we test the applicability of our theoretical results in experiments on synthetic and image data, andshow that VAEs recover the true latent factors when the IMA assumptions are met (§ 4).
Background
We will connect two unsupervised learning objectives: the ELBO in VAEs and the IMA-regularizedlog-likelihood.
Both stem from LVMs with latent variables z distributed according to a prior p
( z ) ,and a mapping from z to observations x given by a conditional generative model p θ ( x | z ) .
Variational Autoencoders.
Optimizing the data likelihood p θ ( x ) in deep LVMs—i.e., findingdecoder parameters θ maximizing (cid:82) p θ ( x | z ) p 0 ( z ) d z —is intractable in general, so approximateobjectives are required.
Variational approximations [59] replace the true posterior p θ ( z | x ) by anapproximate one, called the variational posterior q ϕ ( z | x ) , which is a stochastic mapping x
(cid:55)→ zwith parameters ϕ .
This allows to evaluate a tractable evidence lower bound (ELBO) [33, 53] of themodel’s log-likelihood that can be defined as
[Equation]
The two terms in (1) are sometimes interpreted as a reconstruction term measuring the sample qualityof the decoder and a regularizer—the Kullback-Leibler Divergence (KL) between the prior and the encoder [34].
The variational approximation trades off computational efficiency with a differencew.r.t.
the exact log-likelihood, which is expressed alternatively as (see [15, 34] and Appx.
A)
[Equation]
where the KL between variational and true posteriors characterizes the gap : if the variational familyof q ϕ ( z | x ) does not include p θ ( z | x ) , the ELBO will be strictly smaller than log p θ ( x ) .
VAEs [33] rely on the variational approximation in (1) to train deep LVMs where neural networksparametrize the encoder q ϕ ( z | x ) and the decoder p θ ( x | z ) .
A common modeling choice constrainsthe variational family of q ϕ ( z | x ) to a factorized Gaussian with posterior means µ ϕ k ( x ) and variancesσ ϕ k ( x ) 2 for the k th factor z k | x , and with a diagonal covariance Σ ϕz | x ; and the decoder to a factorized
Gaussian, conditional on z , with mean f θ ( z ) and an isotropic covariance in d dimensions,
[Equation]
The deterministic limit of VAEs.
The stochasticity of VAEs makes it nontrivial to relate themto generative models with deterministic decoders such as Independent Component Analysis (seeparagraph below), though postulating a deterministic regime (where the decoder precision γ 2 becomesinfinite) is possible.
Interestingly, Nielsen et al.
[47] explored this deterministic limit and arguedthat deterministic VAEs optimize an exact log-likelihood, similar to normalizing flows [52, 48].
Normalizing flows model arbitrarily complex distributions with a simple base distribution p
( z ) anda juxtaposition of deterministic and invertible transformations f θ through the change of variables
[Equation]
The comparison is nontrivial, since VAEs contain an encoder and a decoder, whereas normalizingflows consist of a single architecture.
Nielsen et al.
[47] made this analogy by resorting to whatwe call a self-consistency assumption , stating that the VAE encoder inverts the decoder.
We defineself-consistency in the near-deterministic regime: as the decoder variance goes to zero, i.e. γ → + ∞ .
Definition 1 ((Near-deterministic) self-consistency) .
For a fixed θ , assume that mean decoder f θ is invertible with inverse g θ , and that a map associates each choice of decoder parameters and observation( θ , γ, x ) to an encoder parameter ( θ , γ, x ) (cid:55)→ (cid:98) ϕ
( θ , γ, x ) , we say the VAE is self-consistent wheneverµ (cid:98) ϕ
( x ) → g θ ( x ) and σ (cid:98) ϕ
( x ) 2 → 0 , as γ → + ∞ .
The encoder parameter map reflects the choice of a particular encoder model for each ( θ , γ ) pair: 2in § 3.1, we study this problem by introducing and justifying a particular choice for (cid:98) ϕ
(see also § 5).
This self-consistency assumption appears central to deterministic claims [47, 36], but has not yet beenproven.
In particular, Nielsen et al.
[47] assume that taking the deterministic limit is well-behaved.
However, VAEs’ near-deterministic properties have not been investigated analytically.
Identifiability, ICA, and IMA.
Independent Component Analysis (ICA) [8, 28] models observationsas the mixing of a latent vector z with independent components through a deterministic function f , i.e.,x = f
( z ) , p 0 ( z ) =
(cid:81) i p
( z i ) .
In ICA the focus is on defining conditions under which the originallatent variables can be recovered from observations—i.e., the model is identifiable by design [29].
The goal is to learn an unmixing g θ such that the recovered components y
= g θ ( x ) are estimates of the true ones up to some ambiguities (e.g., permutation and element-wise nonlinear transformations).
Unfortunately, the nonlinear problem is nonidentifiable without further constraints [14, 27]: any two observationally equivalent models can yield components which are arbitrarily entangled, thus making recovery of the ground truth factors impossible.
This is typically shown by suitably constructed counterexamples [27, 39], and it was argued to imply impossibility statements for unsupervised disentanglement [39, 61].
Identifiability can be recovered when auxiliary variables [29, 19, 31, 17]are available, or exploiting a temporal structure in the data [26, 22].
Restrictions on the mixing function class (e.g., linear [8]) are another possibility to recover iden- tifiability [27, 63].
Recently, Gresele et al.
[21] proposed restricting the function class by taking inspiration from the principle of independent causal mechanisms [49], in an approach termed Independent Mechanism Analysis (IMA).
IMA postulates that the latent components influence theobservations “independently”, where influences correspond to the partial derivatives ∂ f θ / ∂z k , andtheir non-statistical independence amounts to an orthogonality condition.
While full identifiability hasnot been characterized for this model class, it was proven to rule out the most common counterexamples to identifiability, and empirically shown to help recover the ground truth through regularizationof the log-likelihood in (4) with an objective L IMA ( f θ , z ) :
= log p θ ( x ) − λ · c IMA ( f θ , z ) , whereλ > 0 and the regularization term c IMA ( f θ , z ) and its expectation C IMA ( f θ , p 0 ) are given by
[Equation]
and termed local (resp. global ) IMA contrast.
When f θ is in the IMA function class (i.e.,
C IMA ( f θ , p 0 ) vanishes), the objective is equal to the log-likelihood; otherwise, it lower bounds it.
Theory
Our theoretical analysis assumes that all the model’s defining densities ( p 0 ( z ) , q ϕ ( z | x ) and p θ ( x | z ) )are factorized.
We also assume a Gaussian decoder, matching common modeling practice in VAEs.
Assumption 1 (Factorized VAE class with isotropic Gaussian decoder and log-concave prior) .
We aregiven a fixed
(i) p 0 ( z ) ∼ (cid:81) k m ( z k ) , with m being smooth and fully supported on R , having a bounded
non-positive second-order, and bounded third-order logarithmic derivatives;
(ii) the encoder and decoder are of the form in (3) , with isotropic decoder covariance 1 /
γ 2 I d ;
(iii) the variational mean and variance encoder classes are universal approximators;
(iv) for all θ , f θ : R d → R d is a bijection with inverse g θ , and both are C 2 with bounded first
and second order derivatives.
Crucially, both the mean encoder and the mean decoder can be nonlinear .
Moreover, the family oflog-concave priors contains the commonly-used Gaussian distribution as a special case.
We studythe near-deterministic decoder regime of such models, where γ → + ∞ .
This regime is expected tomodel data generating processes with vanishing observation noise well—in line with the typical ICAsetting—and is commonly considered in theoretical analyses of VAEs, e.g. in [47] (which additionallyassumes quasi-deterministic encoders), and in [41, 36].
Unlike [47], we consider a large but finiteγ , not at the limit γ = ∞ , where the decoder is fully deterministic.
In fact, for any large but finiteγ , the objective is well-behaved and amenable to theoretical analysis, while the KL-divergence isundefined in the deterministic setting.
The requirement in assumption (iv) deviates from commonpractice in VAEs—where observations are typically higher-dimensional—but it allows to connect
VAEs and exact likelihood methods such as normalizing flows [47] (see also § 5).
Due to considering γ → + ∞ , results are stated in the following “big-O” notation for an integer p :
[Equation]
x
[Equation]
∥
[Equation]
bounded
[Equation]
γ → + ∞ .
3.1 Self-consistency
In this section, we will prove a self-consistency result in the near-deterministic regime.
This restson characterizing optimal variational posteriors (i.e., those minimizing the ELBO gap w.r.t.
thelikelihood) for a particular point x and fixed decoder parameters θ .
Based on (2), any associatedoptimal choice of encoder parameters satisfies
[Equation]
We call self-consistent ELBO the resulting achieved value, denoted
[Equation]
The expression in (7) corresponds to a problem of information projection [9, 45] of p θ ( z | x ) onto theset of factorized Gaussian distributions.
This means that given a variational family, we search for the optimal q ϕ ( z | x ) to minimize the KL to p θ ( z | x ) .
While such information projection problems arewell studied for closed convex sets where they yield a unique minimizer [10], the set projected ontoin our case is not convex (convex combinations of arbitrary Gaussians are not Gaussian), making thisproblem of independent interest.
After establishing upper and lower bounds on the KL divergence(exposed in Prop. 7-8 in Appx.
C.2), we obtain the following self-consistency result.
Proposition 1.
[Self-consistency of near-deterministic VAEs] Under Assumption 1, for all x , θ , asγ →
+ ∞ , there exists at least one global minimum solution of (7).
These solutions satisfy
[Equation]
Prop. 1 states that minimizing the ELBO gap (equivalently, maximizing the ELBO) w.r.t.
the encoderparameters ϕ implies in the limit of large γ that the encoder’s mean µ ϕ ( x ) tends to g θ ( x ) , the imageof x by the inverse decoder.
We can interpret this as the decoder “inverting” the encoder.
Additionally,the variances of the encoder will converge to zero, in line with empirical observations of practitioners.
This is also stated as one property of the polarized regime (cf.
Definition 1 in [54]).
Let us now consider the relevance of this result for training VAEs, i.e., maximizing the expectationof the ELBO for an observed distribution p ( x ) .
While maximization only w.r.t.
ϕ in (7) does notmatch common practice—which is learning θ and ϕ jointly —it models this process in the limit oflarge-capacity encoders.
Indeed, in this case, (7) can be solved for each x as a separate learningproblem, which entails that the following inequality is satisfied for any parameter choice
[Equation]
The joint optimization of encoder and decoder parameters thus reduces to optimizing the subset ofpairs ( θ , (cid:98) ϕ
( x , θ )) , and is equivalent to optimizing the expected self-consistent ELBO, that is
[Equation] [Equation]
This problem reduction is aligned with the original purpose of the ELBO: building a tractable but optimal likelihood approximation.
Namely, (i) ELBO ∗ depends on the same parameters as the likelihood( x , γ and θ ), (ii) its gap KL [ q ϕ ( z | x )
|| p
θ ( z | x )] is minimal.
Clearly, the problem reduction of (11) ismore informative only at the critical points of the VAE loss, since it allows us to compare the optimality of different decoders and Prop. 1 allows us to study this question for near-deterministic decoders.
3.2 Self-consistent ELBO, IMA-regularized log-likelihood and identifiability of VAEs
We want to investigate how the choice of q ϕ ( z | x ) and p θ ( x | z ) implicitly regularizes the Jacobiansof their means µ ϕ ( x ) and f θ ( z ) in the near-deterministic regime.
Exploiting self-consistency, weare able to precisely characterize how this happens: we formalize this in Thm.
Theorem 1.
[VAEs with a near-deterministic decoder approximate the IMA objective] Under
Assumption 1, the variational posterior satisfies
[Equation]
and the self-consistent ELBO (19) approximates the IMA -regularized log-likelihood (6) :
[Equation]
Proof is in Appx.
B. Below, we provide a qualitative argument on the interplay between distributionalassumptions in the VAE and implicit constraints on the decoder’s Jacobian and its inverse.
Modeling assumptions implicitly regularize the mean decoder class f θ under self-consistency.
In the near deterministic regime, p θ ( x ) gets close to the pushforward distribution of theprior by the mean decoder f θ ∗ [ p 0 ( z )] , which can be used to show that the true posteriorp θ ( z | x )
= p θ ( x | z ) p 0 ( z ) /p
θ ( x ) is approximately the pushforward through the inverse mean decoderg θ ∗ [ p θ ( x | z )]
(see Appx.
A for more details).
If we select a given latent z 0 and denote its image byf θ ( z 0 ) , then we can locally linearize g θ by its Jacobian J g θ =
J g θ
( f θ ( z 0 )) , yielding a Gaussianfor the pushforward distribution g θ ∗ [ p θ ( x | z )] with covariance 1 / γ 2 J
[Equation]
As the sufficient statistics of a Gaussian are given by its mean and covariance, the structure of the posterior covariance Σ ϕz | x(which is by design diagonal, cf.
(3)) is crucial for minimizing the gap in (2).
Practically, this impliesthat in the zero gap limit, the covariances of q ϕ ( z | x ) and p θ ( z | x ) should match, i.e., 1 / γ 2 J g θ J T g θ willbe diagonal with entries σ ϕ k ( x ) 2 and therefore J g θ has orthogonal rows.
We can express the decoder
Jacobian via the inverse function theorem as J f θ ( z 0 ) =
J g θ
( f θ ( z 0 ))
− 1 .
As the inverse of a roworthogonal matrix has orthogonal columns, f θ satisfies the IMA principle.
Additionally, we can relatethe variational posterior’s variances to the column-norms of J f θ as σ ϕ k ( x ) 2 = 1 / γ 2 ∥ (cid:2)
J f θ ( z 0 ) (cid:3) : k
∥ − 2 .
The self-consistent ELBO therefore converges to the IMA-regularized log-likelihood (6).
Our argument indicates that minimizing the gap between the ELBO and the log-likelihood encouragescolumn-orthogonality in J f θ by matching the covariances of q ϕ ( z | x ) and g θ ∗ [ p θ ( x | z )] .
Whenq ϕ
( z | x ) =
p θ ( z | x ) , the gap is closed; this is only possible if the decoder is in the IMA class, forwhich c IMA vanishes and the ELBO tends to an exact log-likelihood .
To the best of our knowledge,we are the first to prove this for nonlinear functions, extending related work for linear VAEs [41].
Implications for identifiability of VAEs.
While previous works argued that the VAE objectivefavors decoders with a column-orthogonal Jacobian [54, 36], they did not exactly characterizehow: our result shows that the self-consistent ELBO tends to a regularized log-likelihood, where theregularization term c IMA explicitly enforces this (soft) constraint.
Thus, it possibly explains why VAEsare successful in learning disentangled representations: namely, the IMA function class provablyrules out certain spurious solutions for nonlinear ICA [21], and the IMA-regularized log-likelihoodwas empirically shown to be beneficial in recovering the true latent factors.
Thus, we speak aboutembracing the gap , as its functional form equips VAEs with a useful inductive bias.
While the IMAfunction class has not yet been shown to be identifiabile, such results exist for special cases such asconformal maps (only in d = 2 ) [27], isometries [24] and for closely-related unsupervised nonlinear
ICA models [65].
In the following, we empirically corroborate that VAEs: 1) recover the groundtruth sources when the mixing satisfies IMA, and thereby 2) achieve unsupervised disentanglement.
Experiments
Our experiments serve three purposes: 1) demonstrating that self-consistency holds in practice (§ 4.1);2) showing the relationship of the self-consistent ELBO ∗ , the IMA-regularized and unregularizedlog-likelihood objectives (§ 4.2); and 3) providing empirical evidence that the connection to the IMAfunction class in VAEs can lead to success in learning disentangled representations (§ 4.3).
Moredetails are provided in Appx.
F.
4.1 Self-consistency in practical conditions
Experimental setup.
We use a 3-layer MultiLayer Perceptron (MLP) with smooth Leaky ReLUnonlinearities [20] and orthogonal weight matrices—which intentionally does not belong to the IMA class, as our results are more general.
The 60,000 source samples are drawn from a standard normaldistribution and fed into a VAE composed of a 3-layer MLP encoder and decoder with a Gaussianprior.
We use 20 seeds for each γ 2 ∈ { 1e1; 1e2; 1e3; 1e
1e
Results.
Fig.
2 summarizes our results, featuring the logarithms on each axes.
The left plot showsthat the posterior variances σ ϕ k ( x ) 2 converge to zero with a 1 / γ 2 rate, as predicted by (9).
The centerplot shows that the expression for σ ϕ k ( x ) 2 corresponds to (46) in the optimum of the ELBO bycomparing both sides of the equation.
The right plot shows approximate convergence of the meanencodings µ (cid:98) ϕ
( x ) to g θ ( x ) with a 1 / γ rate (see § 5).
As f θ is not guaranteed to be invertible, we useinstead the optimal encoder and decoder parameters to compare f θ ( µ (cid:98) ϕ ( x )) to x .
4.2 Relationship between ELBO ∗ , IMA-regularized, and unregularized log-likelihoods
of smaller
Experimental setup.
We use an MLP f θ withsquare upper-triangular weight matrices andinvertible element-wise nonlinearities to construct amixing not in the IMA class [21] and fix the VAE decoder to the ground truth such that (4) gives the truedata log-likelihood.
This way, we ensure that theunregularized and IMA-regularized log-likelihoodsdiffer and make the claim of Nielsen et al.
[47] comparable to ours.
With a fixed decoder, the ELBO ∗depends only on ϕ , therefore we only train the encoder with γ 2 values from [1e1; 1e5] (5 seeds each).
Results.
Fig.
4 compares the difference of theestimate of ELBO ∗ and the unregularized/IMAregularized log-likelihoods after convergence overthe whole dataset.
As the decoder and the data arefixed, log p θ ( x ) and C IMA will not change duringtraining, only ELBO ∗ does.
The figure shows thatas γ → + ∞ , ELBO ∗ approaches L IMA ( f θ , z ) , aspredicted by Thm. 1, and not log p θ ( x ) , as stated in [47]—the difference is C IMA .
4.3 Connecting the IMA principle, γ 2 , and disentanglement
Experimental setup (synthetic).
We use 3-dimensional conformal mixings (i.e., the Möbius transform [50]) from the IMA class with uniform ground-truth and prior distributions.
Our results quantifythe relationship of the decoder
Jacobian’s IMA-contrast and identifiability with MCC [25] and showhow this translates to disentanglement—we note that MCC was already used to quantify disentanglement [67, 35].
To determine whether a mixing from the IMA class is beneficial for disentanglement,we apply a volume-preserving linear map after the Möbius transform (using 100 seeds) to makec IMA ̸ = 0 .
Other parameters are the same as in § 4.1, with the exception of picking the best γ 2 = 1e5 .
Results (synthetic).
The left of Fig.
3 empirically demonstrates the benefits of optimizing the
IMA-regularized log-likelihood.
By increasing γ 2 , MCC increases, while c IMA decreases, suggestingthat VAEs in the near-deterministic regime encourage disentanglement by enforcing the IMA
Experimental setup (image).
We train a VAE (not β -VAE) with a factorized Gaussian posteriorand Beta prior on a Sprites image dataset generated using the spriteworld renderer [62] with a Betaground truth distribution.
Similar to [30], we use four latent factors, namely, x- and y-position, colorand size , and omit factors that can be problematic, such as shape (as it is discrete) and rotation (due tosymmetries) [54, 35].
Our choice is motivated by [24, 16] showing that this data-generating processmay approximately satisfy the IMA principle.
Results (image).
The left of Fig.
5 indicates that VAEs can learn the true latent factors and MCCis anticorrelated with c IMA , reinforcing the hypothesis that the data-generating process belongs to the
IMA class.
The center plot compares estimated and true latent factors from the best model (scalingand permutation indeterminacies are removed), whereas the right plot shows the correspondinglatent interpolations—thus, connecting identifiability (measured by MCC) to disentanglement.
Limitations
The near-deterministic regime.
Our theory relies on γ → + ∞ ; this is the regime where posteriorcollapse may be avoided [41], and where calculating the reconstruction loss may be possible evenwithout sampling [36].
However, in practice it may be unclear when γ 2 is large enough.
This seemsto be problem-dependent [54, 41], and possibly tied to the covariance of the observations [57, 56].
Moreover, large values of γ 2 may be harder to optimize due to an exploding reconstruction term in (1).
This may be one explanation for the slight deviation of Fig.
2, right from our theory’s predictions:while convergence of µ ϕ ( x ) to g θ matches the prediction in Prop. 1, its rate is not precisely the onepredicted for the self-consistent ELBO (19).
Another cause could be the encoder’s finite capacity.
Nonetheless, we have experimentally shown that for realistic hyperparameters, VAEs’ behaviormatches the predictions of our theory for the near-deterministic regime.
Dimensionality.
The setting in § 3 requires same dimensionality of z and x .
This matches mostwork on normalizing flows [48] and connecting these to VAEs [47], as well as most of the work onnonlinear ICA [26, 29, 22] (but see, e.g., [31]).
Nonetheless, we could empirically verify that thepredictions of our theory also hold for dimensionality reduction (§ 4.3).
Extending our theory to thissetting could rely on, e.g., [11, 6] and is left for future work.
The ELBO, the self-consistent ELBO, and amortized inference.
There are in principle multipleways to obtain self-consistency (Defn. 1).
Notably, one could simply force the variational mean andvariance encoder maps to behave this way; unlike [36], we model the actual behavior of VAEs trainedunder ELBO maximization, and obtain self-consistency as a result.
For this, we assume that theoptimal encoder, which minimizes the gap between ELBO and log-likelihood, can be learned.
This isnot guaranteed in general, since it requires universal approximation capability of the encoder.
On theother hand, (19) requires unamortized inference to introduce ELBO ∗ , which does not depend on ϕ .
As in practice amortized inference may be used to efficiently estimate a single set of ϕ for all x [58],it can lead to a suboptimal gap to the log-likelihood and discrepancies with our theoretical predictions.
Discussion
On disentanglement in unsupervised VAEs.
It is widely believed that unsupervised VAEs cannotlearn disentangled representations [39, 31], motivating work on models with, e.g., conditionalpriors [31] or sparse decoding [44].
We show that under certain assumptions, ELBO optimization can implement useful inductive biases for representation learning, yielding disentangled representationsin unsupervised VAEs.
However, while our results are formulated for VAEs, some of the mostsuccessful models at disentanglement are modifications thereof—e.g., β -VAEs [23, 4], with anadditional parameter β multiplying the KL in (1).
While they deviate from the information projectionsetting considered in § 3.1, their objectives are equivalent to the ELBO in a sense described in
Appx.
A.3, which allows us to derive convergence to the IMA-regularized likelihood objectivefor γ / √ β → + ∞ .
This encompasses the deterministic limit, but also to the setting β → 0 and γconstant described in [36].
Whether this theoretical regime matches common practice remains anopen question.
Overall, we stress that we uncover one possible mechanism through which VAEsmay achieve disentanglement.
By connecting to IMA [21], we discuss implications on recoveringthe ground truth under suitable assumptions, extending uniqueness results presented in [36].
Wespeculate that our success in disentanglement is probably due to selecting data sets where the mixingis in the IMA class (cf.
[24, 16]), which presumably was not the case in [39].
Characterizing the ELBO gap for nonlinear models.
Thm. 1 characterizes the gap between ELBOand true log-likelihood for nonlinear VAEs, and extends the linear analysis of Lucas et al.
[41]; wealso empirically characterize the gap in the deterministic limit in § 4.2.
An unanticipated consequenceof this result is that—consistent with [41]—VAEs optimize the IMA-regularized log-likelihood in thenear-deterministic limit, and not the unregularized one, as stated in [47].
Extensions to related work.
Several papers discuss the (near-)deterministic regime [47, 54, 36].
Forexample, Nielsen et al.
[47] postulate a deterministic VAE with the encoder inverting the decoder.
Also Kumar and Poole [36] work in that regime, but without justifying the relationship between theencoder and decoder.
Although they show that the choice of p 0 ( z ) and q ϕ ( z | x ) influences uniqueness(by, e.g., ruling out rotations), this does not imply recovering the true latents.
Our approach formalizes(Defn.
1), proves (Prop. 1), and demonstrates the practical feasibility of (§ 4) the near-deterministicregime.
To the best of our knowledge, all previous work relied on the linear case [41] or a (linear)approximation and the evaluation of the ELBO around a point to show the inductive bias on thedecoder Jacobian.
However, our main result (Thm. 1) yields a nonlinear equation where the decoder
Jacobian can be evaluated at any point and is equipped with a convergence bound.
Moreover, theconsistency of VAE estimation for identifiable models [31] requires guarantees on q ϕ ( z | x ) ; our resulthelps proving these.
We discuss extended connections to the literature in Appx.
D and Appx.
E.
Covariance structure and IMA.
We have shown that specific choices for encoder and decodercovariances regularize the decoder Jacobian.
Assuming factorized q ϕ ( z | x ) and isotropic p θ ( x | z ) ,
IMA holds only for the decoder ; since in the other direction the pushforward of q ϕ ( z | x ) through f θhas covariance J f θ ( z ) Σ ϕz | x J f θ ( z ) T , which cannot be simplified in the general case.
Additionallyassuming an isotropic encoder would make IMA hold in both directions and would yield conformal
Jacobians (as both J f θ ( z ) and J g θ ( x ) need to be column-orthogonal).
On the other hand, if theobservation model is non-isotropic, IMA would only hold for the encoder Jacobian.
Covariancestructure has an intuitive meaning: an isotropic p θ ( x | z ) covariance equals having i.i.d.
noise foreach x i .
For data recorded by the same device (e.g., pixels) this may be reasonable, but multi-viewsettings with non-isotropic noise could require different modeling choices.
Conjecture for VAEs with isotropic encoder and decoder covariances.
Following our intuition(Fig. 1), we suspect that an isotropic encoder (the encoder in Thm.
1 has diagonal covariance)
entailsa decoder with conformal Jacobian (i.e., the product of a scalar field and an orthogonal matrix).
Thisis an interesting constraint, especially as there is growing evidence that nonlinear ICA with conformalmixings may be identifiable: the two-dimensional case was proven in [27] and IMA was shown torule out certain spurious solutions for conformal mixings [21].
Based on these, we conjecture:
Conjecture 1 (Unsupervised VAEs with isotropic q ϕ ( z | x ) and p θ ( x | z ) are identifiable) .
Whenq ϕ
( z | x ) and p θ ( x | z ) have (possibly data-dependent) isotropic covariances, i.e., Σ z | x = α ( x ) I d andΣ x
|
z
= β ( z ) I d , and self-consistency holds, then unsupervised VAEs are identifiable.
Conclusion.
We provide a theoretical justification for the widely-used self-consistency assumptionin the near-deterministic regime of small decoder variance.
Using this result, we show that theself-consistent ELBO converges to the IMA-regularized log-likelihood, and not to the unregularizedone.
Thus, we can characterize the gap between ELBO and true log-likelihood and reason about itspossible role as an inductive bias for representation learning in nonlinear VAEs.
We characterize a setof assumptions under which unsupervised VAEs can be expected to disentangle and we demonstratethis behavior in experiments on synthetic and image data.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes] See § 3, § 4, §
(b) Did you describe the limitations of your work?
[Yes] See §
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] SeeAppx. H(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes] We use no data about human subjects.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See § 3(b) Did you include complete proofs of all theoretical results?
[Yes] See § 3, Appx.
B3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main ex-
perimental results (either in the supplemental material or as a URL)?
[Yes] See thesupplementary material
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes] See § 4, Appx.
F(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] See § 4(d)
Did you include the total amount of compute and the type of resources used (e
g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See Appx.
F
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] See § 4 and the
supplementary material
(b) Did you mention the license of the assets?
[Yes] See the supplementary material
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
We included the code and the experiment logs.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]
Appendix
Table of Contents
A Complementary notes
A.1 ELBO decompositions . . . . . . . . . . . . . . . . . . . . . . . . . . .
A.2 Justification of the intuition . . . . . . . . . . . . . . . . . . . . . . . . . . . .
A
A as
T f θ J f θ . . . . . . . . . .
A.5 Assessing the value recovering the true latents . . . . . . . . . . . .
B Main Theoretical Results
B.1 Proof of Proposition 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
B.2 Proof of Theorem 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
C Auxiliary results
C.1 Squared norm statistics . .
C.2 KL divergence bounds .
C.3 Taylor formula-based approximations . .
C.4 Variational posterior variance optimization problem . . . . . . . . . . . . . . .
D Related
work
D.
Implicit inductive biases
in
D.2 (Near)-deterministic VAEs . . . . . . . . . . . . . . . . . . . . . . . . . . . .
E Further remarks on the the IMA–VAE connection
E.1 Linear VAE from Lucas et al.
F Experimental details
F.
The relationship of weight matrix structures and the IMA function class . . . . .
F.2 Self-consistency in practical conditions (§ 4.1) . . . . . . . . . . . . . . . . . .
F.3 Relationship between ELBO ∗ , IMA-regularized, and unregularized log-likelihoods(§ 4.2) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
F.4 Connecting the IMA principle, γ 2 , and disentanglement (§ 4.3) . . . . . . . . .
F.5 Optimality of γ 2 w.r.t.
its MLE . . . . . . . . . . . . . . . . . . . . . . . . . .
G Computational resources
H Societal
impact
I
Notation
Acronyms . .
Nomenclature . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
A Complementary notesA.1 ELBO decompositions
Connection between (1) and (2) .
Here we show how the two decompositions of the ELBOobjective in (1) and (2) can be connected.
We start from equation (2):
[Equation]
By definition of KL-divergence, and applying Bayes rule, we get
[Equation]
We observe that the two terms involving p θ ( x ) cancel, resulting in
[Equation]
which leads to (1) by rearranging the terms:
[Equation]
Expressions for the two terms in equation (1) under Assum.
The above two terms take thefollowing form in our setting.
For the second (“KL”) term, we get
[Equation]
, where H denotes the entropy.
Writing the expression for the entropy of univariate Gaussian variables( 1 / 2 log(2 πσ 2 ) + 1 / 2 ), we have under Assum.
[Equation]
where we introduce the dimension dependent constant κ d = d 2 (log(2 π ) + 1) .
This leads to
[Equation]
The first (“reconstruction”) term, under the isotropic Gaussian decoder of Assum. 1, takes the form
[Equation]
Expression for the gap between ELBO and log-likelihood Let us now write the KL divergencebetween variational and true posteriors, which is the gap appearing in (2).
[Equation]
Using again the expression of the entropy of Gaussian variables, this leads to
[Equation]
such that, using the Bayes formula for the true posterior and Assum.
1, we get
[Equation]
with additive constant c ( x , γ ) = − d 2 (cid:0) log( γ 2 ) + 1 (cid:1)
+ log p θ ( x ) .
Note the log(2 π ) term in theprevious expression cancels with the one coming from the true log posterior.
The analysis of the optima of (16) is non-trivial due to the second term which involves takingexpectations of functions of z w.r.t.
its posterior distribution q ϕ parameterized by µ ϕ and σ ϕ .
Muchof the derivations to obtain our results will revolve around constructing bounds that no longer involvesuch expectations, but instead only depend on µ ϕ and σ ϕ .
A.2 Justification of the intuition
We add here more qualitative details to the statement of subsection 3.2 that the true posterior densityis approximately the pushforward of p θ ( x | z = z 0 ) .
Note that they are not meant to replace a rigoroustreatment, which is deferred to Appx.
B.
As the decoder becomes deterministic, the marginal observed density becomes the pushforward ofthe latent prior by f θ 4 such that
[Equation]
The true posterior is therefore approximately
[Equation]
Conditioning on a given observation x = f θ ( z 0 ) , we getp θ ( z | x =
[Equation]
( z 0 ))
=
p
[Equation]
f θ ( z 0 ) | z ) p
[Equation]
( z ) /p
[Equation]
( x = f θ ( z 0 ))
[Equation] [Equation]
|
[Equation]
) |
[Equation]
Neglecting the variations of the prior relative to those of the posterior (due to near-determinism)
[Equation]
wemake the approximation p
( z ) ≈ p 0 ( z 0 ) such that the above approximation becomes
[Equation]
Using the isotropic Gaussian decoder assumption
[Equation]
we get
[Equation]
In the near-deterministic regime
[Equation]
this posterior distribution should be concentrated in the regionwhere z is close to z 0 , we can then further approximate this density using a Taylor formula
[Equation]
with G =
J g θ
( f θ ( z 0 ))
=
J f θ ( z 0 ) − 1 , which is also matching the expression of the pushforwardof the Gaussian density p θ ( x | z
= z 0 ) by the linearization of g θ around f θ ( z 0 ) (i.e. replacing themapping by its Jacobian at that point, G ).
A.3 A connection between the β parameter of β -VAEs and the decoder precision γ 2
In the context of disentanglement, a commonly used variant of standard VAEs [33] is the β -VAE [7,23, 32, 54, 36].
In this model, an additional parameter β is added to modify the weight of the KLterm in (1), whereas the decoder precision γ 2 is typically set to one [12, 18, 36, 54].
The β -VAEobjective [23] can be written as
[Equation]
The influence of the decoder precision γ 2 and the β parameters on the objective have been related inthe literature, see for example [15, § 2.4.3]—and similar observations can be found in [56, § 3.1].
Under the assumption of a Gaussian decoder, the ELBO from eq.
(1) can be written as (making nowexplicit mention of its decoder parameter γ in parenthesis):
[Equation]
with c ( γ, d ) =
d log γ − d 2 log(2 π ) .
In contrast, the β -VAE objective L β ( x ; θ , ϕ ) (also with explicit mention of γ ) is expressed as:
[Equation] [Equation]
We thus can link this expression to an ELBO as follows:
[Equation]
When we restrict ourselves to common practice, the optimizations of both the ELBO and the L β areperformed with a fixed value of γ (and with fixed β for L β ).
This entails that there is an equivalenceof the solutions resulting from the optimization of each objective, as they differ only by additive andmultiplicative constants.
In particular, given the above expression, the following result is immediate:
Proposition 2.
Let us define the self-consistent β -VAE objective as
[Equation]
Then the following normalized self-consistent β -VAE objective
[Equation] [Equation]
converges to the regularized IMA objective as γ √ β → + ∞ under the same conditions as in Thm.
Proof.
The β -VAE objective is, up to an additive constant and a strictly positive multiplicativeconstant, identical to the ELBO objective.
As a consequence their optimum and the values ofparameters ϕ
and θ at which they are achieved are identical.
This suggests that choosing a fixed β or a β growing as, for example, log γ or even √ γ would lead toself-consistent solutions with the same properties as the vanilla VAE in the deterministic decoderlimit γ → + ∞ , and notably robustness to spurious solutions.
Generalizing Kumar and Poole [36] Moreover, this connection also shows that our result generalizes previous work.
Particularly, considering γ 2 = 1 when expressing the optimal variance analogousto eq.
(55), we can discover the same expression as in [36].
Namely, the objective function has the form (where we now indicate explicitly the dependence on γand emphasize that both γ and β are fixed):
[Equation]
As a consequence, the optimal σ ϕ k ( x ) 2 now includes β compared to (55):
[Equation]
That is, β affects the decoder Jacobian column norms as 1 / γ 2 .
They are not exactly equivalentthough—note the β factor in front of the log-prior terms:
[Equation]
+
[Equation]
Again, as both β and γ are fixed, the additive/multiplicative constants are irrelevant.
Thus, β hasa similar effect on column-orthogonality as 1 / γ 2 .
We formalize this observation in the followingremark:
Remark 1.
β affects the column norms of the decoder Jacobian in the same way as 1 / γ 2 .
One canthink of having γ ′ =
γ/ √ β and tying the tuning of β, γ ′ .
Remark 2 (Generalization of Kumar and Poole [36]) .
Assuming the same conditions as in Thm.
1 andγ 2 = 1 (i.e., a Gaussian decoder with a Hessian H = I d and Gaussian prior with I d as second-orderderivative) with the β -VAE loss, then expressing the optimal posterior covariance with γ / √ β , we get
Eq.
(11) of Kumar and Poole [36],
[Equation]
as a special case.
A.4 c IMA as the (scaled) left KL measure of diagonality of J T f
θ J f θ
In our paper, we used the original definition for c IMA ([21, (8)]), which we restate here:
[Equation]
However, an alternative formulation exists: namely, c IMA ( f θ , z ) can be thought of as the (scaled) left
KL measure of diagonality of the square matrix J f θ ( z ) T J f θ ( z ) , as shown in [2].
That is, we canrewrite the above as [21, (23)]
(with A =
J f θ ( z ) T J f θ ( z ) ):
[Equation]
The importance of eq.
(23) is twofold: i) it provides a theoretical analysis of c IMA as a measure ofthe column-orthogonality of J f θ ( z ) (or, equivalently, the diagonality of J f θ ( z ) T J f θ ( z ) ); andii) it elucidates why c IMA can be used in the dim x ̸ = dim z case, but only to measure columnorthogonality (to exploit the beneficial properties of IMA for identifiability, the theory of IMA needsto be extended to this case).
We leverage ii) as a justification to use c IMA in our image experiments in§ 4.3.
A.5 Assessing the value of c IMA for recovering the true latentsc IMA , given by (6), measures the deviation of the learned decoder from the IMA function class.
As itis positive and unbounded, it is practically relevant to investigate the following question: how muchviolation of the IMA assumption is acceptable to recover the true latents?
Expressed differently, weare interested in whether a threshold can be specified for c IMA ( f θ , z ) to decide whether the true (butunknown) latent factors can be recovered.
Unfortunately, our answer is negative, but this is not specific to IMA theory.
Even in the case oflinear ICA we cannot specify a threshold for the non-Gaussianity of the latent (source) variables toensure that the ground-truth factors can be recovered [27].
We acknowledge that, among others, c IMA being unbounded makes it harder to interpret.
Thus, wego back to first principles to develop an imperfect but hopefully more practical intuition on how toassess the value of c IMA to decide whether the true latents are recovered (if the true mixing is in the
IMA class).
As we already pointed out in Appx.
A.4, c IMA is the left KL measure of diagonalityof J T f θ J f θ [2].
In their analysis, the authors provide closed form solutions expressions to comparedifferent diagonality measures.
For our purposes, [2, Fig. 1] provides an important insight: it shows that in the two-dimensional casec IMA increases nonlinearly in a variable r expressing the degress of diagonality of a matrix (moreprecisely as − log (cid:0) 1 − r 2 (cid:1) , where r = 0 denotes the identity matrix, whereas r = 1 a matrix withparallel columns).
The takeaway for us is that simply comparing two c IMA values can be misleading.
For the two-dimensional case, we could define an expression that is linear in r (but this requires us toaccept that r is a -in some sense- suitable measure of the diagonality of a matrix), namely:
[Equation]
Unfortunately, the d -dimensional case only has a power series formulation; thus, it is nontrivial howto extend the above reasoning.
However, eq.
(24) expresses c IMA as the negative log determinantof a scaled matrix, where the determinant of diag ( A ) −
1 / 2 A diag ( A ) − 1 / 2 is between 0 and 1 ( A =
J f θ ( z ) T J f θ ( z )) .
Thus, we can convert c IMA to the [0; 1] interval as follows:
[Equation] [Equation]
[Equation]
The merit of the above expression is a more natural way to compare values that are normalized tothe [0; 1] interval.
Namely, the original formulation of c IMA may potentially lead to problems, as,e.g., a value of t and 10 t does not mean that one model is ten times better in recovering the truelatents.
Nonetheless, this can be thought (at most) as a small step towards making the analysis of theempirical value of c IMA a useful tool in practical scenarios, where we do not have access to the truelatent factors.
B Main Theoretical Results
B.1 Proof of Proposition 1
We proceed in two steps: first we prove the existence of variational parameters that achieve a globalminimum of the ELBO gap, then we characterize its near-deterministic properties.
We then combinethese results, which rely on specific assumptions, to obtain our main text result under Assum.
We initially use the following milder assumptions than in main text to prove intermediate results.
Assumption 2 (Gaussian Encoder-Gaussian Decoder VAE, minimal properties) .
We are given afixed latent prior and three parameterized classes of R d → R d mappings: the mean decoder classθ (cid:55)→ f θ , and the mean and standard deviation encoder classes, ϕ (cid:55)→ µ ϕ and ϕ (cid:55)→ σ ϕ such that(i) the latent prior has a factorized independent and identically distributed (i.i.d.) densityp 0 ( z ) ∼ (cid:81)
k m ( z k ) , with m smooth fully supported on R , with concave log m ,(ii) conditional on the latent, the decoder has a factorized Gaussian density p θ with mean f θsuch that
[Equation]
(iii) the encoder is factorized Gaussian with posterior mean and variance maps µ ϕ k
( x ) , σ ϕ k ( x ) 2for each component k ,
leading to the factorized posterior density q ϕ
( z | x ) such that
[Equation]
(iv) the mean and variance encoders classes can fit any function,(v) for all possible θ , f θ is a diffeomorphism of R d with inverse g θ
[Equation]
Existence of at least one global minimizer of the gap between true and variational posterior is givenby the following proposition.
Proposition 3 (Existence of global minimum) .
Under Assumption 2.
For a fixed θ assume additionally that g θ is Lipschitz continuous with Lipschitz constant B > 0 , in the sense that
[Equation]
Then there exists at least one choice ( µ ϕ ∈ R d , σ ϕ
[Equation]
R
[Equation]
> 0 ) that achieves the minimum of
KL [ q ϕ ( z | x ) || p
θ ( z | x )] .
Proof.
Using Prop. 7, we have the lower bound
[Equation]
We then notice (see lemma 4) that for all k ,
[Equation]
achieves a global minimum n ( B, γ ) =
− log( B/γ ) + 1 / 2 at σ ϕ k ( x ) =
B/γ .
For arbitrary k 0 , we now 1) lower bound the k ̸ = k 0 terms by n ( B, γ ) ; 2) lower bound and all thelog m terms by their global maximum, which exists by Assum.
1i (log-concave prior); and 3) dropthe non-negative squared norm term, leading to the following weaker lower bound:
[Equation]
The KL divergence is well-defined and finite for any choice of parameters in their domain, thereforeit achieves a particular value K 0 ≥ 0 at one arbitrary selected point of the domain.
Since forall k , the lower bound tends to
Moreover, starting again from the lower bound from Prop. 7,
[Equation]
we now focus on µ ϕ and lower bound all σ ϕ terms.
With this, we get the following weaker lowerbound in terms of µ ϕ :
[Equation]
The lower bound also tends to + ∞ for ∥ µ ϕ ∥ → + ∞ , so there exists a radius R > 0 (possiblydependent on ( γ, x ) )
such that KL [ q ϕ ( z | x ) || p
θ ( z | x )] > K
if ∥ µ ϕ ∥ > R .
As a consequence, the infimum ( ≤ K 0 ) of the minimization problem (7) cannot be achieved outsidethe compact set ( µ ϕ , σ ϕ ) ∈ { µ ϕ ∈ R d : ∥ µ ϕ ∥ ≤ R } ×
[ a, b ] d .
Since the divergence is continuousin ( µ ϕ , σ ϕ ) , there exists a value ( µ (cid:98) ϕ , σ (cid:98) ϕ ) in this compact set achieving the minimum of the KLover the whole parameter domain, and all values achieving this minimum are in this compact set.
For given x , θ and γ > 0 , the variational posterior KL divergence mapping
[Equation]
thus has a minimum, and by smoothness of this mapping, this minimum can be characterized by thevanishing gradient of the KL divergence with respect to the parameters.
Now, let us try to characterizehow this minimum behaves for large γ .
Proposition 4 (Self-consistency of the encoder in the deterministic limit) .
Under Assum.
2, assumeadditionally f θ and g θ are Lipschitz continuous with respective Lipschitz constants C, B > 0 , in thesense that
[Equation] [Equation]
Assume additionally that − log m is quadratically dominated, in the sense that
[Equation]
Then for all x , θ , as γ → + ∞ , any global minimum of (7) satisfies
[Equation] [Equation]
More precisely, for all x ∈ R d , γ >
[Equation] [Equation]
Proof.
We start from the lower bound expression of Prop.
[Equation]
with c ( x , γ ) = − d 2 (cid:0) log( γ 2 ) + 1 (cid:1)
+ log p θ ( x ) .
For any ν ∈ (0 , 1] , we can thus write
[Equation]
Now, from lemma 4 we get
[Equation]
We exploit this lower bound to obtain
[Equation]
g θ ( x ) − µ ϕ (cid:13)(cid:13)
[Equation] [Equation]
ν ) d (cid:88) k
=
[Equation]
ϕ
[Equation]
( x )(cid:35) .
Using the expression of c ( x , γ ) we get
KL [ q ϕ ( z | x ) || p θ
[Equation]
z | x )] ≥ d
[Equation]
log
[Equation]
νB
[Equation]
+ log
[Equation]
−
[Equation]
d
[Equation]
and both the “ d log γ ” as well as “ d / 2 ” terms cancel out such that
KL
[Equation]
Finally, using Prop. 8, the above right hand side is bounded from above by a constant as γ → + ∞ ,and as a consequence, the positive factor of the γ 2 term must vanish (by continuity assumption andits limits note − log m is bounded from below)
[Equation]
This entails that both positive terms it comprises must vanish too.
More precisely, we get the inequality between lower and upper bounds at the optimal solution
[Equation]
which simplifies to
[Equation]
Moreover by continuity assumption and its limits
[Equation]
− log m is bounded from below by− M = − max t log m ( t ) , yielding
[Equation]
2 M (cid:1) + γ 2 2 B − 2 (cid:34)(cid:13)(cid:13)(cid:13) g θ ( x ) − µ (cid:98) ϕ ( x ) (cid:13)(cid:13)(cid:13) 2 + (1 − ν ) d (cid:88) k
= σ (cid:98) ϕ
k
( x )(cid:35) ≤ d (cid:18)2( C 2 − 1) + E + D (cid:20) ∥
g θ ( x ) ∥d +γ(cid:21)(cid:19) such thatγ2 B − 2 (cid:34)(cid:13)(cid:13)(cid:13) g θ ( x ) − µ (cid:98) ϕ
( x ) (cid:13)(cid:13)(cid:13) 2 + (1 − ν ) d (cid:88) k
= σ (cid:98) ϕ
k
( x )(cid:35) ≤ d (cid:18)2( C 2 − 1) + E + D (cid:20) ∥
g θ ( x ) ∥d +γ(cid:21) − (cid:0) log( νB − 2 ) − 2 M (cid:1)(cid:19) and finally
B − 2 (cid:34)(cid:13)(cid:13)(cid:13) g θ ( x ) − µ (cid:98) ϕ
( x ) (cid:13)(cid:13)(cid:13) 2 + (1 − ν ) d (cid:88) k
= σ (cid:98) ϕ
k
( x )(cid:35) ≤ 2 d γ(cid:18)2( C 2 − 1)
+ E + D (cid:20)
∥
g θ ( x ) ∥d +γ(cid:21)
+ M + 1 2 log( B 2 /ν )
(cid:19) (39)
Taking ν = 1 in (39) we get the first intended inequality
[Equation]
Alternatively, (39) implies
[Equation]
Taking a fixed value of ν , say 1 / 2 , we get the second intended inequalityd (cid:88)
[Equation]
We now restate the main text proposition and provide the proof.
Proposition 1.
[Self-consistency of near-deterministic VAEs] Under Assumption 1, for all x , θ , asγ →
+ ∞ , there exists at least one global minimum solution of (7).
These solutions satisfyµ
[Equation]
x
[Equation]
O
[Equation] [Equation]
and σ (cid:98) ϕ
[Equation]
( x )
= O
[Equation] [Equation]
, for all k .
[Equation]
Proof.
We only have to check that Assum.
1 allow fulfilling the following requirements of Prop. 4:
• the Lipschitz continuity requirements in Prop. 4 results from the boundedness of the first
order derivatives of the decoder mean and of its inverse (by using the multivariate Taylortheorem),• concavity of log m , required by Assum.
2, is a direct consequence of non-positivity of the
second-order logarithmic derivative of m in Assum.
1i,• quadratic domination of − log m comes from the boundedness of the second-order logarith-
mic derivative of m (by integrating twice).
Then Prop. 4 follows and the O ( 1 / γ ) convergence of the variational posterior mean of the inverse, aswell as the O ( 1 / γ 2 ) convergence of the variational posterior variance.
Finer approximation of parameter values We now derive a finer result for the convergence of themean, that we will exploit in Thm. 1.
This relies on the existence of an optimum shown by Prop. 3.
At such optimum (cid:98) ϕ we thus have for all k∂
[Equation]
and
[Equation]
We derive the constraints entailed by the first expression:
[Equation]
with
[Equation]
which leads to a set of constraints at optimum
[Equation]
Based on this expression we derive the following result.
Proposition 5.
Under Assum.
1, as γ → + ∞
[Equation]
and
[Equation]
Proof.
We start from the constraints of (40) that we rewrite
[Equation]
We then proceed to approximate the left hand side using a Taylor formula.
Assuming bounded
Hessian components, we can upper and lower bound using third order centered absolute moments ofthe Gaussian as
[Equation]
which we can rewrite (by 1) expanding the norm of the sum; 2) removing constants in the bracket,which lead to zeros after multiplying the zero mean variable and taking the expectation; 3) using
Gaussianity, all centered third order terms vanish.)
[Equation]
Finally computing this integral we get the left hand side as
[Equation]
For the right hand side we get using a Taylor expansion (with notation n : z → log( m ( z
[Equation] [Equation]
Equating the non-negligible terms of the left and right-hand sides we get for each k
[Equation]
such that
[Equation]
where n ′ is applied component-wise.
Because the Jacobian is everywhere invertible (implicitconsequence of Lipschitz assumptions), we can solve for this equations and get
[Equation]
Using again a similar Taylor approximation we get
[Equation]
This equation has the shortcoming of still referring to the posterior mean on both sides.
To fix this,we first note that it implies, by boundedness of the Jacobian, that
[Equation]
By bounding the second-order derivative of the log prior, we get
[Equation]
, which implies
[Equation]
i.e., we obtain an improved convergence rate.
Using this rate and Taylor theorem, we obtain the finalequation by replacing the variational posterior mean by the inverse decoder in (43)
[Equation]
B.2 Proof of Theorem
This will be a corollary of the following result, that uses as a key assumption a rate of O ( 1 / γ 2 ) in theconvergence of the self-consistency equation of the variational mean.
Proposition 6 (VAEs with log-concave factorized prior and close-to-deterministic decoder approximate the IMA objective) .
Under Assum.
1, if additionally the VAE satisfies the followingself-consistency in the deterministic limit
[Equation] [Equation]
(45) then
[Equation]
and the self-consistent ELBO (19) approximates the IMA -regularized log-likelihood (6) :
[Equation]
Proof.
We start from the self-consistent ELBO decomposition as “reconstruction error plus posteriorregularization” terms:
[Equation]
and continue with reformulating both terms, based on Assum.
That is, p 0 is factorized withcomponents i.i.d.
distributed according to a fully supported log-concave density z k ∼ m .
Posterior regularization term Assum.
1 gives us the formula of (14) for this term in the ELBO.
Taking optimal encoder parameters, we get the posterior regularization term for the ELBO ∗
[Equation]
κ
[Equation]
, with κ d = d 2 (log(2 π ) + 1) .
Using the factorized Gaussian encoder and i.i.d.
prior assumptions weget
[Equation]
where we rewrote the distribution p 0 as p
= (cid:81) k m ( z k ) .
Based on the Taylor theorem, with a residual in Lagrange form of n = log m , we have that for all kand u there exists ξ ∈ [ µ ϕ k ( x ) , u ] if u ≥ µ ϕ k ( x ) , or ξ ∈ [ u, µ ϕ k ( x )] if u ≤ µ ϕ k ( x ) such thatn
[Equation]
We assumed that | n (3) | is bounded over R by F , such that
[Equation]
Taking the expectation and using the expression of centered Gaussian absolute moments
[Equation]
As the assumptions entail that optimal posterior variances σ (cid:98)
ϕ k ( x ) 2 get small for γ large (cf.
(45)),this implies the near-deterministic approximation
[Equation]
In addition, using again a Taylor formula and the self-consistency assumption for the mean
[Equation]
Moreover, using again a Taylor formula for n ′′ under boundedness of n (3) and again using theself-consistency assumption for the mean yields
[Equation]
Overall this leads to the approximation of the posterior regularization term
[Equation]
Reconstruction term Now switching to the first (reconstruction) term of the ELBO ∗ , adapting thedecomposition of (15) by using optimal encoder parameters we get
[Equation]
Then in the small encoder noise limit σ k ( x ) 2 ≪ 1 , ∀ k (justified by Prop. 1), we rely on a Taylorapproximation around the posterior mean
z
o = µ ϕ ( x ) based on Lemma 3, which bounds thisapproximation as follows
[Equation]
The linear term in this approximation is easily computed using successively Lemma 1 and Lemma 2to get an expression with the squared column norms of the partial derivatives scaled by the standarddeviations
∂ f θ ∂z k | µ ϕ k
( x ) .
We get
[Equation]
This term can be used as an approximation for the expectation term in the reconstruction loss thanksto the following reverse triangle inequality
[Equation]
σ ϕ k ( x )
 (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) ≤ E q ϕ ( z | x )   (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) x − (cid:32)
f θ ( z ) − d (cid:88) k
= ∂ f θ ∂z k | z
o ( z k − µ ϕ k
( x )) (cid:33)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) 2   , such that the resulting upper bound can be itself bounded as follows
E q
[Equation]

[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
∂z
[Equation]
[Equation]
(cid:33)(cid:13) (cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
  ≤
[Equation]
) 
[Equation]
− d (cid:88)
[Equation] [Equation]
x )
[Equation]
2   .
Each term of the upper bound can be bounded for the optimum encoder parameters: using from leftto right the assumption of (44) and (51), respectively, leading to
[Equation]
Getting back to the whole reconstruction term, using additionally the variance self-consistencyassumption (45), the above shows that we can make the approximation
[Equation]
We can further replace the dependency of the derivatives on the encoder mean using a Taylor formulafor the derivative
[Equation]
such that
[Equation]
ELBO ∗ approximation As a consequence of (50) and (53) the ELBO ∗ becomes
ELBO ∗ ( x ; θ ) = − 1d (cid:88) k
=
[Equation]

[Equation]
where we isolated the terms that depend on parameters σ (cid:98)
ϕ k ( x ) 2 and γ in the approximate objective(cid:92) ELBO ( σ 2 = σ (cid:98) ϕ
( x ) 2 ; x , θ , (cid:98) ϕ ) that we define for arbitrary σ 2 .
(cid:92) ELBO ( σ 2 ; x , θ , (cid:98) ϕ ) =
[Equation]
Where we further break this objective in d components (cid:92) ELBO k ( σ (cid:98)
ϕ k ( x ) 2 ; x , θ , (cid:98) ϕ ) according to theterms of the sum as follows
[Equation]
 and where we note that − n ′′ ≥ 0 due to the log-concavity assumption.
Solving term in k (cid:92) ELBO k ( σ 2 k ) for optimal γ 2 σ ∗ k we get (see lemma 4):
[Equation]
54) and the resulting optimal value (cid:92) ELBO ∗ k ( x , θ , (cid:98) ϕ ) =
(cid:92) ELBO k ( σ ∗ 2 k ; x , θ , (cid:98) ϕ ) is
[Equation]

A Taylor formula around this optimum leads, for some value ξ γ ( x ) lying between σ ∗ 2 k and σ 2 k to(note the first order derivative vanishes, and the second order derivative is upper bounded hence thesecond line)
[Equation]
as a consequence the non-approximate solution for the true optimal ELBO ∗ , as γ grows, must achievea value below this quadratic function, up to a term in O ( 1 / γ 2 ) , and at the same time above (cid:92) ELBO ∗ ,also up to a term in O ( 1 / γ 2 ) .
This entails that it is restricted to a smaller and smaller domain near theapproximate solution and we get
[Equation]
Leading to the approximation of the true objective
ELBO ∗ ( x ; θ ) = − 1d
[Equation]
1 / γ 2 ) , which reduces to
[Equation]
which is the IMA objective.
We now restate the main text theorem and provide its proof.
Theorem 1.
[VAEs with a near-deterministic decoder approximate the IMA objective] Under
Assumption 1, the variational posterior satisfies
[Equation]
and the self-consistent ELBO (19) approximates the IMA -regularized log-likelihood (6) :
[Equation]
Proof.
This is just a corollary of Proposition 6 because Proposition 5 entails through (42) therequired O ( 1 / γ 2 ) rate of convergence for the optimal variational mean in (44), while (45) is fulfilledthrough Prop. 1.
C Auxiliary results
[Figure]
C.1 Squared norm statistics
[Figure]
Lemma 1 (Squared norm variance decomposition) .
For multivariate RV X with mean m
[Equation]
Proof.
[Equation]
hence
[Equation]
This leads to (using that the trace of a scalar is the scalar itself)
[Equation]
because trace [ AB ] = trace [ BA ] we get
[Equation]
Lemma 2 (Trace of transformed unit covariance) .
When the covariance matrix Cov ( ϵ ) is the identity,thentrace
[Equation]
Proof.
For arbitrary matrix A , Cov ( A ϵ ) =
ACov ( ϵ ) A T and thus
[Equation] [Equation]
=
[Equation]
[ A T ACov ( ϵ )] .
Moreover, in our case Cov ( ϵ ) is the identity such that
[Equation]
A ] .
[Equation]
C.2 KL divergence bounds
Proposition 7 (Lipschtiz continuity-based lower bound) .
Assume g θ is Lipschitz continuous with
Lipschitz constant B > 0 , in the sense
[Equation]
Then for any encoder parameter choice
[Equation]
with c ( x , γ ) = − d 2 (cid:0) log( γ 2 ) + 1 (cid:1)
+ log p θ ( x ) .
Proof.
Starting from the KL divergence expression (16),
[Equation]
with additive constant c ( x , γ ) = − d 2 (cid:0) log( γ 2 ) + 1 (cid:1)
+ log p θ ( x ) .
By Lipschitz continuity
[Equation]
using Lemma 1 applied to g θ ( x ) − z , z ∼ q ϕ
( z | x ) we get
KL
[Equation]
Using Jensen’s inequality for − log m (convex by Assum. 1(i)), we get
KL [ q ϕ ( z | x ) || p
θ ( z | x )]
[Equation]
by reordering the terms we finally get
[Equation]
which is the stated KL lower bound.
Proposition 8 (Optimal encoder KL divergence upper bound) .
Assume f θ is Lipschitz continuouswith Lipschitz constant C > 0 , in the sense that∀ z , w ∈ R d :
[Equation]
[Equation]
Assume, − log m is quadratically dominated, in the sense that∃ D > 0 , E > 0 , ∀ u ∈ R , − log m ( u ) ≤ D | u | 2 + E .
Then for the optimal encoder solution of (7)
KL
[Equation]
andlim
[Equation]
Proof.
Starting from the KL divergence expression (16),
[Equation]
with additive constant c ( x , γ ) = −
[Equation]
Let us choose the following posterior (by universal approximation capabilities of the encoder):
[Equation]
Using Lipschitz continuity we get
[Equation]
then, using
[Equation]
we get
[Equation]
using quadratic domination
[Equation]
Using Lemma 1 we get
[Equation]
hence for a parameter (cid:98) ϕ
achieving the minimum divergence we get
[Equation]
As
[Equation]
)) such that the KL divergence for the optimalsolutions is upper bounded by a finite number.
C.3 Taylor formula-based approximations
Lemma 3 (Bound on expectation of multivariate Taylor expansion) .
Assume f : R d
→ R is C 2 andassume z is a multivariate RV on R d with indepedent Gaussian components such that
[Equation]
then for all z o ∈ R d
[Equation]
Proof.
As described in [42, p. 162], for the l -th component of the functionf l ( z ) =
f l ( z o )
+
[Equation]
where the second line puts 1 / 2 of the partial derivatives in matrix form (note it is not exactly the
Hessian as derivatives are taken at different points).
As a consequence
[Equation]
where ∥
H k ∥ 2 is the spectral norm of the matrix and ∥ H k ∥ F is the Frobenious norm 6 leading to thebound
[Equation]
where K is an upper bound on the absolute second order derivatives.
We have ( z k − z ok ) =
σ ϕ k ( x ) ϵ k ,with ϵ multivariate normal, so taking the expectation of the above simplifies to:
[Equation]
Now gathering all components f l to get the squared norm yields:
[Equation]
C.4 Variational posterior variance optimization problem
Lemma 4.
For α > 0 , the function
[Equation]
is strictly convex and achieves its global minimum min
h α
= 12 log α for u ∗ = 1 √ α .
Proof.
Function h
α is stricly convex as a sum of two stricly convex functions.
Its derivative,
[Equation]
thus vanishes only at the minimum for u ∗ = 1 √ α .
We then get that
[Equation]
D Related work
D.1 Implicit inductive biases in the ELBO
Rolinek et al.
[54] reason about the connection to Principal Component Analysis (PCA) in the contextof nonlinear Gaussian VAEs with an isotropic prior and assume that the variational posterior hasdiagonal covariance with distinct singular values .
The authors make it explicit that they investigatethe consequences of optimizing the ELBO.
They locally linearize the decoder to show the inductivebias in VAEs that promotes decoder orthogonality.
Their results hold for β -VAEs, where β shouldbe in the range of satisfying the polarized regime assumption (i.e., when the VAE is close to partialposterior collapse).
The validity of the assumptions (polarized regime and distinct singular values inΣ ϕz | x ) are only experimentally investigated.
The same authors extend their work in [66], completingthe connection to PCA for linear models.
Their experiments, inspired by the connection to PCAfor linear models, show that local perturbations in the data prohibit disentanglement for non-linearmodels.
Nakagawa et al.
[46] builds upon the results of [54] and provides a novel interpretation of
VAEs by introducing implicit variables to express the latent space in terms of an isometric embedding.
Lucas et al.
[41] prove that linear Gaussian VAEs with an isotropic prior give rise to a columnorthogonal decoder and therefore uniquely recover the PCA coordinate axes (not just the correctsubspace, as Probabilistic Principal Component Analysis (PPCA) [60] does), yielding identifiabilityfor Gaussian models—but only when the eigenvalues of the data covariance are distinct.
In theirwork, the decoder variance is shown to be small when avoiding posterior collapse.
More interestingly,the authors derive a formula for the ELBO gap in the linear case that is remarkably similar to the
IMA objective.
We show in Appx.
E.1 that in the limit of a deterministic decoder linear Gaussian
VAEs optimize the IMA objective with λ = 1 .
Dai and Wipf [12] present more general results than[41] since they use affine functions.
Additionally, a connection to Robust PCA [5] is established.
Kumar and Poole [36] generalizes [54], as it admits a variational posterior q ϕ (
z | x ) with blockdiagonal covariance with a uniqueness result for diagonal Σ ϕz | x .
The authors derive a formulafor the optimal Σ ϕz | x [36, Eq. 12], showing that when the decoder Hessian H is diagonal, thedecoder Jacobian will be column-orthogonal even for non-Gaussian decoders.
Their analysis relieson a “concentrated” q ϕ ( z | x ) (i.e., they work in what we term the near-deterministic regime) andsufficiently small values of β —this relationship can be read off from [36, Eq. 12].
Interestingly,the authors also show that rotations of the latents can be ruled out, though they do not connect thedecoder structure (especially, column-orthogonality of its Jacobian) to any specific generative modelfor the data, or to considerations on identifiability of the ground truth sources.
Dai and Wipf [12] use a different setting that turns out to be very interesting to compare to ours.
Themost important is that while we use a factorized Gaussian variational posterior, Dai and Wipf [12]use a non-factorized Gaussian, which leads to major differences.
Broadly construed, Dai and Wipf[12] are able to show in their Theorem 2 (which includes the case of equal latent and observationdimensions matching our setting) that in the deterministic limit, their κ -simple VAE can fit perfectlythe arbitrary observed data (barring few assumptions), while the ELBO gap tends to zero.
The way itis proven relies on a first step where they rely on the Darmois construction [27] to chose the decodermean parameter such that its pushforward is exactly the observation distribution. Then in a secondstep, by an appropriate choice of variational posterior parameters, they show that asymptoticallythe ELBO gap (i.e. the KL divergence between true and variational posteriors) tends to zero in thedeterministic limit.
In contrast, our constraint of factorized variational posterior does not allow the ELBO gap to vanish in the deterministic limit (unless the decoder mean that fits the data perfectly isin the IMA class, which is a very special case, in particular if the Darmois construction is used).
Forthis reason, the proofs and scope of our results are very different: (i) we use information theoreticbounds to show that the encoder inverts the decoder mean (independently from the fact that this onemay or may not fit the data perfectly); (ii) we obtain a rigorous convergence to the IMA regularizedlikelihood, which demonstrates that the gap is not eliminated in the deterministic limit.
Regarding our result for β -VAEs (Prop. 2), the approach of Mathieu et al.
[43] is similar as theauthors show that L β can be expressed in terms of a rescaled ELBO.
The difference is that Mathieuet al.
[43] uses a rescaling of the parameters ϕ , θ , whereas we only scale γ 2 .
D Near deterministic VAEs
Recent work was inspired by the normalizing flow literature and the shortcomings of the stochastic
VAE architecture to propose designs that are (near)-deterministic.
Arguments for this regime rangefrom avoiding posterior collapse (as demonstrated in [41]) to avoiding sampling for the reconstructionloss term [36].
Several papers argued for a similar setting: Dai and Wipf [12] take the limit ofγ → + ∞ (here using γ as the square root of the decoder precision and not the decoder variance asused in [12]) to derive a result relating encoder and decoder properties in this limit in their Theorem5, that has a similar flavor to Prop. 1. In contrast to our nonlinear analysis, this is derived whenoptimizing with respect to both encoder and decoder parameter, and as stated in the previous section,the non-factorized encoder assumptions leads to fundamentally different behavior of the solutionsin the deterministic limit.
Rolinek et al.
[54] refer to the polarized regime (a property of which is thatencoder variances are small, cf.
[54, Definition 1]), Kumar and Poole [36] argue for “concentrated”variational posteriors.
Ghosh et al.
[18] substitute stochasticity with a regularizer on the decoder Jacobian from an intuitive, whereas Kumar et al.
[38] motivate these results from an injectiveflow perspective.
Nielsen et al. [47] also take a normalizing flow perspective to connect VAEs todeterministic models.
Besides benefits of avoiding posterior collapse or possible improvementsduring optimization, this regime serves as a potential connection to the identifiability literature.
E Further remarks on the the IMA–VAE connection
In this section, we elaborate on the connection between VAEs and IMA, by showing that previouswork on linear VAEs can be directly connected to optimizing L IMA .
Our intent with this analysis is toprovide additional insights about the role of γ in a simpler setting.
E.1 Linear VAE from Lucas et al. [41]
We restate the linear VAE model of [41]:
[Equation]
[Equation]
where D is a diagonal matrix, W the decoder and V the encoder weights, µ the mean latentrepresentation.
The authors show that in stationary points, the optimal value for D is
[Equation] [Equation]
If we substitute this expression into the ELBO gap (i.e., the KL between the variational and trueposteriors), we get a similar expression to c IMA —as formalized in Prop. 9.
Proposition 9 (The ELBO converges to L IMA for linear Gaussian VAEs if γ → + ∞ ) .
For linear
Gaussian VAEs, in the limit of γ → ∞ , the ELBO equals the IMA -regularized log-likelihood instationary points with λ = 1 .
Proof.
In [41, Appendix C.2], it is shown that the gap between exact log-likelihood and ELBO forlinear Gaussian VAEs in stationary points reduces to
[Equation]
[Equation]
where W is the decoder weight matrix.
Reformulating the above expression, we arrive at :
[Equation]
Noting that W T W is symmetric with a Singular Value Decomposition (SVD) of UΛU T ( U isorthogonal, Λ ii = ∥
[ W ] : k ∥ 2 ), and I d = UU T ; thus:
[Equation]
Therefore, (70) can be reformulated as the left KL-measure of diagonality [2] of the matrix
U [ Λ + 1 / γ 2 I d ] U T :
[Equation]
which is by definition the local IMA contrast c IMA (cf.
[21, Appendix C.1]).
When γ → + ∞ , theabove expression converges to the left KL-measure of diagonality for W T W , i.e., the local IMAcontrast for the decoder.γ → + ∞ thus means that the ELBO converges to the IMA regularized log-likelihood L IMA withλ = 1 :
[Equation]
which concludes the proof.
Prop. 9, especially (72), gives us intuitive understanding on why and how γ influences how much theorthogonality of W is enforced.
Small γ (high observation noise) means that there is no reason to promote the orthogonality
of the decoder,
as the high noise level (i
e., low-quality fit of x ) will drive (72) towards
diagonality via 1 / γ 2 .
On the other hand, when γ → + ∞ , then the orthogonality of the decoder is promoted.
That
is, the decoder precision γ 2 acts akin to a weighting factor influencing how strong the IMA
principle should be enforced.
We can observe that the ELBO recovers the exact log-likelihood for column-orthogonal W :
Corollary 1 (For column-orthogonal W the ELBO equals the exact log-likelihood) .
When W is inthe form W = OD , then diag (cid:0) W T W (cid:1)
=
W T W =
DO T OD
= D 2 , i.e. the ELBO correspondsto the exact log-likelihood since (72) is zero.
Corollary 1 also implies that γ does not affect the gap between ELBO and exact log-likelihood forcolumn-orthogonal W .
F Experimental details
F.1 The relationship of weight matrix structures and the IMA function class
During the experiments we have used different weight matrices either to ensure that the mixing iswithin or to exclude it from the IMA function class.
Here we summarize our choices also includingthe depth of the network as it can affect the mixing’s place w.r.t.
the IMA function class.
When we use orthogonal weight matrices (§ 4.1,§ 4.2), then a single-layer network is within the
IMA class, but adding more layers with elements-wise nonlinearities will move the MLP outsidethe function class.
When using triangular MLPs (§ 4.2), the network is also outside the IMA class(triangular matrices are orthogonal only when they are diagonal , see also [21, Lemma C.1]).
Notably, Möbius transforms [50] are conformal maps (thus, they are in the IMA class) irrespective ofthe structure of the weight matrix used (cf.
Appx.
F.4 for details).
F.2 Self-consistency in practical conditions (§ 4.1)
For the self-consistency experiments, the mixing is a 3-layer MLP with smooth Leaky ReLU nonlinearities [20] and orthogonal weight matrices—which intentionally does not belong to the IMA class,since our self-consistency result is not constrained to the IMA class.
The 60,000 source samples aredrawn from a standard normal distribution and fed into a VAE composed of a 3-layer MLP encoder anddecoder with a Gaussian prior.
We use 20 seeds for each γ 2 ∈ { 1e1; 1e2; 1e3; 1e4; 1e5 } .
Additionalparameters are described in Tab. 1.
Training is continued until the ELBO ∗ improves on the validationset (we use early stopping [51]), then all metrics are reported for the maximum ELBO ∗ (Fig. 2).
[Table]
F.3 Relationship between ELBO ∗ , IMA-regularized, and unregularized log-likelihoods (§ 4.2)
[Table]
and triangular
mixing is from the IMA-class (Tab. 3), we use a onelayer neural network with orthogonal weight matrices.
The data dimensionality in both cases is two.
Training is carried out using a VAE with a decoderfixed to the ground-truth and separate encoder models for the means and variances of the approximateposterior.
The encoder comprises two three-layerneural networks with ReLU non-linearities and ahidden layer size of 50 .
Due to training instabilitieswhen using a large γ , we train the model by first fixing the mean encoder to the ground-truth inverse of the mixing for the first 30 epochs; thus, only training the variances.
We then train both for the remaining epochs.
Training is stopped after the ELBO ∗ plateaus on the validation set .
A training set of 100 , 000 samples is used, with a validation set andtest set of 30 , 000 and 15 , 000 samples, respectively.
The learning rate is 1e − 4 and the batch size 64 .
We provide additional results when the mixing is from the IMA class (Tab. 3): as C IMA is zero,we expect that both L IMA and the unregularized log-likelihood match.
Indeed, this is what Fig.
6demonstrates.
[Table]
F.4 Connecting the IMA principle, γ 2 , and disentanglement (§ 4.3)
Synthetic data (Möbius transform)
We use 3-dimensional conformal mixings (i.e., the Möbiustransform [50]) from the IMA class with the functional form:
[Equation]
where t , b ∈ R d , W ∈ R d × d , α ∈ R , and ϵ = 2 (to ensure nonlinearity) with d = 3 .
Bothground-truth and prior distributions are uniform to avoid the singularity when z = b .
To determine whether a mixing from the IMA class is beneficial for disentanglement, we apply avolume-preserving linear map after the Möbius transform (using 100 seeds) to construct a mixingoutside of the IMA class.
We fix γ 2 = 1e5 and report further parameters in Tab. 4.
Training iscontinued until the ELBO ∗ improves on the validation set (we use early stopping [51]), then allmetrics are reported for the maximum ELBO ∗ (Fig. 3).
[Table]
Image data (Sprites)
We train a VAE (not β -VAE) with a factorized Gaussian posterior and Betaprior on a Sprites image dataset generated using the spriteworld renderer [62] with a Beta groundtruth distribution.
Similar to [30], we use four latent factors, namely, x- and y-position, color andsize , and omit factors that can be problematic, such as shape (as it is discrete) and rotation (due tosymmetries) [54, 35].
Our choice is motivated by [24, 16] showing that the data-generating processpresumably is in the IMA class.
The architecture both for encoder and decoder consists of fourconvolutional and three linear layers with ReLU nonlinearities (Tab. 5).
Training is continued untilthe ELBO ∗ improves on the validation set (we use early stopping [51]), then all metrics are reportedfor the maximum ELBO ∗ .
[Table]
F.5 Optimality of γ 2 w.r.t.
its MLE
During our experiments, we do not optimize γ 2 , as it is generally the case in the literature [54, 41, 36].
However, as noted by Rybkin et al.
[56], doing so could lead to superior sample quality.
The pricewe need to pay for improved sample generation is a more difficult optimization task (also noted in[57]): including γ 2 as a trainable parameter might require a careful learning rate tuning, and smallerlearning rates can yield suboptimal likelihood values in the beginning [56].
During our experiments, we confirmed that making γ 2 learnable (all else being equal) yields suboptimal results, particularly in terms of MCC.
Thus, we opted for comparing our hyperparametersetting to the maximum likelihood estimate of the decoder variance , as proposed in [56, Eq.
Accomodating the parameter γ 2 instead of the decoder variance, we reformulate the equation as:
[Equation]
i.e., the MLE is the mean squared error between observations x and the decoded mean encodingsf θ (cid:0) µ ϕ ( x ) (cid:1) , where |X| denotes the number of observations.
Interestingly, this is the inverse of thequantity we report on the right plot of Fig. 2.
To compare γ 2MLE (calculated as eq.
and the optimal value of γ 2 we found via grid search fromthe values { 1e1; 1e2; 1e3; 1e4; 1e5 } , we plot the log of both values in Fig.
We can observe thatfor all values except 1e5 , γ 2MLE is larger, sometimes with more than one order of magnitude.
For1e5 , the mean (for the 20 seeds) lie in the range [0 . 8e5; 3 . 8e5] with a mean and standard deviation of2 .
3 ± 0 .
77e5 , indicating that γ 2 = 1e5 and γ 2 MLE are in the same order of magnitude, corroboratingthat we used the optimal setting up to the granularity of our original grid search.
G Computational resources
[Figure]
The self-consistency (§ 4.1), the likelihood comparison (§ 4.2), and the synthetic experiments withthe Möbius transform (§ 4.3, particularly Fig. 3) were ran on a MacBook Pro with a Quad-Core Intel
Core i5 CPU and required approximately nine days.
The Sprites experiments (§ 4.3, particularly
Fig. 5) required approximately four and a half days on an Nvidia RTX 2080 GPU.
H Societal impact
[Figure]
Our paper presents basic research and is mainly theoretical, though the lack of direct connection to aspecific application does not mean that our results could not be used for malevolent purposes.
Weacknowledge that providing a possible mechanism for why unsupervised VAEs can learn disentangledrepresentations can inform specific actors that unsupervised VAEs might be used to extract the truegenerating factors.
Since no auxiliary variables, labels, or conditional distributions are required, thismight lead to a broader use of unsupervised VAEs for trying to learn the true generating factors—including applications with potentially negative societal impact such as extracting features fromimages, video, or text for personal identification; thus, possibly violating the desire of those whointend to remain anonymous.
I Notation
Acronyms
ELBO evidence lower bound
IMA Independent Mechanism Analysisi.i.d.
independent and identically distributed
ICA Independent Component Analysis
KL Kullback-Leibler Divergence
LVM Latent Variable Model
MCC Mean Correlation Coefficient
MLE Maximum Likelihood Estimation
MLP MultiLayer Perceptron
MSE Mean Squared Error
PCA Principal Component Analysis
PPCA Probabilistic Principal Component Analysis
SVD Singular Value Decomposition
VAE Variational Autoencoder
Nomenclature
Independent Mechanism Analysis
C IMA global IMA contrastα scalar field
D general diagonal matrix
O orthogonal matrixy reconstructed sources
L IMA IMA loss functionc IMA local IMA contrast
Variational Autoencoder
V weight matrix of a linear encoder
W weight matrix of a linear decoderµ (cid:98) ϕ
( x ) optimal mean of q ϕ ( z | x )µ ϕ ( x ) mean of q ϕ ( z | x )ϕ parameters of the variational posterior q ϕ
( z | x )σ (cid:98) ϕ
( x ) 2 optimal variance of q ϕ ( z | x )θ parameters of the decoder p θ ( x | z )γ square root of the precision of the VAE decoderΣ ϕz | x covariance matrix of q ϕ ( z | x )
L β β -VAE loss functionf θ decoderg θ inverse decoder(cid:98) ϕ
optimal parameters of the variational posterior q ϕ ( z | x )p ( x ) data distributionp
( z ) latent prior distributionp θ ( z | x ) true posterior distribution of the decoded samples of the VAE, mapping x
(cid:55)→ z ,parametrized by θp θ ( x ) marginal likelihoodp θ ( x | z ) conditional distribution of the decoded samples of the VAE, mapping z (cid:55)→ x , parametrizedby θq ϕ ( z | x ) variational posterior of the VAE, mapping x
(cid:55)→ z parametrized by ϕq (cid:98) ϕ
( z | x ) optimal variational posterior of the VAE, mapping x
(cid:55)→ z parametrized by ϕµ (cid:98)
ϕ k ( x ) optimal mean of q ϕ ( z | x ) in dimension
kµ ϕ k ( x ) mean of q ϕ ( z | x ) in dimension kσ (cid:98) ϕ
k ( x ) 2 optimal
[Table]