Embrace the Gap: VAEs Perform Independent Mechanism Analysis
Abstract
Variational autoencoders (VAEs) are a popular framework for modeling complex data distributions; they can be efficiently trained via variational inference by maximizing the evidence lower bound (ELBO), at the expense of a gap to the exact (log-)marginal likelihood.
While VAEs are commonly used for disentangled representation learning, it is unclear why ELBO maximization would yield such representations, since unregularized maximum likelihood estimation generally cannot invert the data-generating process without additional assumptions.
Yet,
VAEs often succeed at this task.
We seek to elucidate this apparent paradox by studying nonlinear VAEs in the limit of near-deterministic decoders.
We first prove that, in this regime, the optimal encoder approximately inverts the decoder—a commonly used but unproven conjecture—which we refer to as self-consistency .
Leveraging self-consistency, we show that the ELBO converges to a regularized log-likelihood.
This allows VAEs to perform what has recently been termed independent mechanism analysis (IMA): it adds an inductive bias towards decoders with column-orthogonal Jacobians, which helps recovering the true latent factors.
The gap between ELBO and log-likelihood is therefore welcome, since it bears unanticipated benefits for nonlinear representation learning.
In experiments on synthetic and image data, we show that VAEs uncover the true latent factors when the data generating process satisfies the IMA assumption.
Introduction
Latent Variable Models (LVMs) allow to effectively approximate a complex data distribution and to sample from it [3, 48].
Deep LVMs employ a neural network (the decoder or generator ) to parameterize the conditional distribution of the observations given latent variables, which are typically assumed to be independent.
However, Maximum Likelihood Estimation (MLE) of the model parameters is computationally intractable.
In Variational Autoencoders (VAEs) [35, 56], the exact log-likelihood is substituted with a tractable lower bound, the evidence lower bound (ELBO).
This objective introduces an approximate posterior of the latents given the observations (the encoder ) from a suitable variational distribution whose mean and covariance are parametrized by neural networks.
The encoder is introduced to efficiently train a deep LVM: however, it is not explicitly designed to extract useful representations [17, 58].
[Figure]
Nonetheless, VAEs and their variants are widely used in representation learning [25, 1], where they often recover semantically meaningful representations [39, 8, 34, 5].
Our understanding of this empirical success is still incomplete, since (deep) LVMs with independent latents are nonidentifiable from i.i.d.
data [29, 42]; different models fitting the data equally well may yield arbitrarily different representations, thus rendering the recovery of a ground truth generative model impossible.
While auxiliary variables, weak supervision [28, 31, 21, 43, 72, 19], or specific model constraints [29, 67, 68, 26, 23] can help identifiability, the mechanism through which the ELBO may enforce a useful inductive bias remains unclear, despite recent efforts [5, 57, 38, 15, 71].
In this work, we investigate the benefits of optimizing the ELBO for representation learning by analyzing VAEs in a near-deterministic limit for the conditional distribution parametrized by the nonlinear decoder.
Our first result concerns the encoder’s optimality in this regime.
Previous works relied on the intuitive assumption that the encoder inverts the decoder in the optimum [50, 38, 71]; we formalize this self-consistency assumption and prove its validity for the optimal variational posterior in the near-deterministic nonlinear regime.
Using self-consistency, we show that the ELBO tends to a regularized log-likelihood—rather than to the exact one as conjectured in previous work [50].
The regularization term allows VAEs to perform what has been termed Independent Mechanism Analysis (IMA) [23]: it encourages column orthogonality of the decoder’s Jacobian.
This generalizes previous findings based on linearizations or approximations of the ELBO [57, 44, 38], and allows us to characterize the gap w.r.t.
the log-likelihood in the deterministic limit.
Our results elucidate the gap between ELBO and exact log-likelihood as a possible mechanism through which the ELBO implements a useful inductive bias.
Unlike the unregularized log-likelihood, the IMA-regularized objective can help invert the data generating process under suitable assumptions [23].
We verify this by training VAEs in experiments on synthetic and image data, showing that they can recover the ground truth factors when the IMA assumptions are met.
The contributions of this paper can be summarized as follows:
• we characterize and prove self-consistency of VAEs in the near-deterministic regime (i.e., when the decoder variance tends to zero), justifying its usage in previous works (§ 3.1); • we show that under self-consistency, the ELBO converges to a regularized log-likelihood (§ 3.2), and discuss its possible role as a useful inductive bias in representation learning; • we test the applicability of our theoretical results in experiments on synthetic and image data, and show that VAEs recover the true latent factors when the IMA assumptions are met (§ 4).
Background
We will connect two unsupervised learning objectives: the ELBO in VAEs and the IMA-regularized log-likelihood.
Both stem from LVMs with latent variables z distributed according to a prior p
( z ) , and a mapping from z to observations x given by a conditional generative model p θ ( x | z ) .
Variational Autoencoders.
Optimizing the data likelihood p θ ( x ) in deep LVMs—i.e., finding decoder parameters θ maximizing (cid:82) p θ ( x | z ) p 0 ( z ) d z —is intractable in general, so approximate objectives are required.
Variational approximations [63] replace the true posterior p θ ( z | x ) by an approximate one, called the variational posterior q ϕ ( z | x ) , which is a stochastic mapping x
(cid:55)→ z with parameters ϕ .
This allows to evaluate a tractable evidence lower bound (ELBO) [35, 56] of the model’s log-likelihood that can be defined as
[Equation]
The two terms in (1) are sometimes interpreted as a reconstruction term measuring the sample quality of the decoder and a regularizer—the Kullback-Leibler Divergence (KL) between the prior and the encoder [36].
The variational approximation trades off computational efficiency with a difference w.r.t.
the exact log-likelihood, which is expressed alternatively as (see [17, 36] and Appx.
A)
[Equation]
where the KL between variational and true posteriors characterizes the gap : if the variational family of q ϕ ( z | x ) does not include p θ ( z | x ) , the ELBO will be strictly smaller than log p θ ( x ) .
VAEs [35] rely on the variational approximation in (1) to train deep LVMs where neural networks parametrize the encoder q ϕ ( z | x ) and the decoder p θ ( x | z ) .
A common modeling choice constrains the variational family of q ϕ ( z | x ) to a factorized Gaussian with posterior means µ ϕ k ( x ) and variances σ ϕ k ( x ) 2 for the k th factor z k | x , and with a diagonal covariance Σ ϕz | x ; and the decoder to a factorized
Gaussian, conditional on z , with mean f θ ( z ) and an isotropic covariance in d dimensions,
[Equation]
The deterministic limit of VAEs.
The stochasticity of VAEs makes it nontrivial to relate them to generative models with deterministic decoders such as Independent Component Analysis (see paragraph below), though postulating a deterministic regime (where the decoder precision γ 2 becomes infinite) is possible.
Interestingly, Nielsen et al.
[50] explored this deterministic limit and argued that deterministic VAEs optimize an exact log-likelihood, similar to normalizing flows [55, 51].
Normalizing flows model arbitrarily complex distributions with a simple base distribution p 0 ( z ) and a juxtaposition of deterministic and invertible transformations f θ through the change of variables
[Equation]
The comparison is nontrivial, since VAEs contain an encoder and a decoder, whereas normalizing flows consist of a single architecture.
Nielsen et al.
[50] made this analogy by resorting to what we call a self-consistency assumption , stating that the VAE encoder inverts the decoder.
We define self-consistency in the near-deterministic regime: as the decoder variance goes to zero, i.e. γ → + ∞ .
Definition 1 ((Near-deterministic) self-consistency) .
For a fixed θ , assume that mean decoder f θ is invertible with inverse g θ , and that a map associates each choice of decoder parameters and observation ( θ , γ, x ) to an encoder parameter ( θ , γ, x ) (cid:55)→ (cid:98) ϕ
( θ , γ, x ) , we say the VAE is self-consistent whenever µ (cid:98) ϕ
( x ) → g θ ( x ) and σ (cid:98) ϕ
( x ) 2 → 0 , as γ → + ∞ .
The encoder parameter map reflects the choice of a particular encoder model for each ( θ , γ ) pair:in § 3.1, we study this problem by introducing and justifying a particular choice for (cid:98) ϕ
(see also § 5).
This self-consistency assumption appears central to deterministic claims [50, 38], but has not yet been proven.
In particular, Nielsen et al.
[50] assume that taking the deterministic limit is well-behaved.
However, VAEs’ near-deterministic properties have not been investigated analytically.
Identifiability, ICA, and IMA.
Independent Component Analysis (ICA)
[9, 30] models observations as the mixing of a latent vector z with independent components through a deterministic function f , i.e., x = f ( z ) , p 0 ( z ) =
(cid:81) i p
( z i ) .
In ICA the focus is on defining conditions under which the original latent variables can be recovered from observations—i.e., the model is identifiable by design [31].
The goal is to learn an unmixing g θ such that the recovered components y
= g θ ( x ) are estimates of the true ones up to some ambiguities (e.g., permutation and element-wise nonlinear transformations).
Unfortunately, the nonlinear problem is nonidentifiable without further constraints [16, 29]: any two observationally equivalent models can yield components which are arbitrarily entangled, thus making recovery of the ground truth factors impossible.
This is typically shown by suitably constructed counterexamples [29, 42], and it was argued to imply impossibility statements for unsupervised disentanglement [42, 65].
Identifiability can be recovered when auxiliary variables [31, 21, 33, 19] are available, or exploiting a temporal structure in the data [28, 24].
Restrictions on the mixing function class (e.g., linear [9]) are another possibility to recover identifiability [29, 67].
Recently, Gresele et al.
[23] proposed restricting the function class by taking inspiration from the principle of independent causal mechanisms [52], in an approach termed Independent Mechanism Analysis (IMA).
IMA postulates that the latent components influence the observations “independently”, where influences correspond to the partial derivatives ∂ f θ / ∂z k , and their non-statistical independence amounts to an orthogonality condition.
While full identifiability has not been proved for this model class, it was shown to rule out classical families of spurious solutions used as counterexamples to identifiability of unconstrained non-linear ICA [23, 4].
Moroever, Buchholz et al.
[4] further demonstrated local identifiability of this function class.
Also, IMA constraints were empirically shown [23, 62] to help recover the ground truth through regularization of the log-likelihood in (4) with an objective L IMA ( f θ , z ) :
= log p θ ( x ) − λ · c IMA ( f θ , z ) , where λ >and the regularization term c IMA ( f θ , z ) and its expectation C IMA ( f θ , p 0 ) are given by c IMA ( f θ , z ) =
[Equation]
and termed local (resp. global ) IMA contrast.
When f θ is in the IMA function class (i.e.,
C IMA ( f θ , p 0 ) vanishes), the objective is equal to the log-likelihood; otherwise, it lower bounds it.
Theory
Our theoretical analysis assumes that all the model’s defining densities ( p 0 ( z ) , q ϕ ( z | x ) and p θ ( x | z ) ) are factorized.
We also assume a Gaussian decoder, matching common modeling practice in VAEs.
Assumption 1 (Factorized VAE class with isotropic Gaussian decoder and log-concave prior) .
We are given a fixed latent prior and three parameterized classes of R d → R d mappings: the mean decoder class θ (cid:55)→ f θ , and the mean and standard deviation encoder classes, ϕ
(cid:55)→ µ ϕ and ϕ (cid:55)→ σ ϕ s.t.
(i) p 0 ( z ) ∼ (cid:81) k m ( z k ) , with m being smooth and fully supported on R , having a bounded non-positive second-order, and bounded third-order logarithmic derivatives; (ii) the encoder and decoder are of the form in (3) , with isotropic decoder covariance 1 /
γ 2 I d ; (iii) the variational mean and variance encoder classes are universal approximators; (iv) for all θ , f θ : R d → R d is a bijection with inverse g θ , and both are C 2 with bounded first and second order derivatives
Crucially, both the mean encoder and the mean decoder can be nonlinear .
Moreover, the family of log-concave priors contains the commonly-used Gaussian distribution as a special case.
We study the near-deterministic decoder regime of such models, where γ → + ∞ .
This regime is expected to model data generating processes with vanishing observation noise well—in line with the typical
ICA setting—and is commonly considered in theoretical analyses of VAEs, e.g., in [50] (which additionally assumes quasi-deterministic encoders), and in [44, 38].
Unlike Nielsen et al.
[50], we consider a large but finite γ , not at the limit γ = ∞ , where the decoder is fully deterministic.
In fact, for any large but finite γ , the objective is well-behaved and amenable to theoretical analysis, while the
KL-divergence is undefined in the deterministic setting.
The requirement in assumption (iv) deviates from common practice in VAEs—where observations are typically higher-dimensional—but it allows to connect VAEs and exact likelihood methods such as normalizing flows [50] (see also § 5).
Due to considering γ → + ∞ , results are stated in the following “big-O” notation for an integer p :
[Equation]
bounded as
[Equation]
3.1 Self-consistency
In this section, we will prove a self-consistency result in the near-deterministic regime.
This rests on characterizing optimal variational posteriors (i.e., those minimizing the ELBO gap w.r.t.
the likelihood) for a particular point x and fixed decoder parameters θ .
Based on (2), any associated optimal choice of encoder parameters satisfies
[Equation]
We call self-consistent ELBO the resulting achieved value, denoted as
[Equation]
The expression in (7) corresponds to a problem of information projection [10, 48] of p θ ( z | x ) onto the set of factorized Gaussian distributions.
This means that given a variational family, we search for the optimal q ϕ ( z | x ) to minimize the KL to p θ ( z | x ) .
While such information projection problems are well studied for closed convex sets where they yield a unique minimizer [11], the set projected onto in our case is not convex (convex combinations of arbitrary Gaussians are not Gaussian), making this problem of independent interest.
After establishing upper and lower bounds on the KL divergence (exposed in Prop. 7-8 in Appx.
C.2), we obtain the following self-consistency result.
Proposition 1.
[Self-consistency of near-deterministic VAEs] Under Assumption 1, for all x , θ , as γ → + ∞ , there exists at least one global minimum solution of (7).
These solutions satisfy µ (cid:98) ϕ
[Equation]
and σ (cid:98)
[Equation]
( x
[Equation]
2 =
O ( 1 /
[Equation]
2 ) , for all k .
Prop. 1 states that minimizing the ELBO gap (equivalently, maximizing the ELBO) w.r.t.
the encoder parameters ϕ implies in the limit of large γ that the encoder’s mean µ ϕ ( x ) tends to g θ ( x ) , the image of x by the inverse decoder.
We can interpret this as the decoder “inverting” the encoder.
Additionally, the variances of the encoder will converge to zero.
Let us now consider the relevance of this result for training VAEs, i.e., maximizing the expectation of the ELBO for an observed distribution p ( x ) .
While maximization only w.r.t.
ϕ in (7) does not match common practice—which is learning θ and ϕ jointly —it models this process in the limit of large-capacity encoders.
Indeed, in this case, (7) can be solved for each x as a separate learning problem, which entails that the following inequality is satisfied for any parameter choice
[Equation]
The joint optimization of encoder and decoder parameters thus reduces to optimizing the subset of pairs ( θ , (cid:98) ϕ
( x , θ )) , and is equivalent to optimizing the expected self-consistent ELBO, that is
[Equation]
This problem reduction is aligned with the original purpose of the ELBO: building a tractable but optimal likelihood approximation.
Namely, (i) ELBO ∗ depends on the same parameters as the likelihood ( x , γ and θ ), (ii) its gap KL [ q ϕ ( z | x )
|| p
θ ( z | x )] is minimal.
Clearly, the problem reduction of (11) is more informative only at the critical points of the VAE loss, since it allows us to compare the optimality of different decoders and Prop. 1 allows us to study this question for near-deterministic decoders.
3.2 Self-consistent ELBO, IMA-regularized log-likelihood and identifiability of VAEs
We want to investigate how the choice of q ϕ ( z | x ) and p θ ( x | z ) implicitly regularizes the Jacobians of their means µ ϕ ( x ) and f θ ( z ) in the near-deterministic regime.
Exploiting self-consistency, we are able to precisely characterize how this happens: we formalize this in Thm. 1.
Theorem 1.
[VAEs with a near-deterministic decoder approximate the IMA objective] Under
Assumption 1, the variational posterior satisfies
[Equation]
and the self-consistent ELBO (19) approximates the IMA -regularized log-likelihood (6) :
[Equation]
Proof is in Appx.
B. Below, we provide a qualitative argument on the interplay between distributional assumptions in the VAE and implicit constraints on the decoder’s Jacobian and its inverse.
Modeling assumptions implicitly regularize the mean decoder class f θ under self-consistency.
In the near deterministic regime, p θ ( x ) gets close to the pushforward distribution of the prior by the mean decoder f θ ∗ [ p 0 ( z )] , which can be used to show that the true posterior p θ ( z | x ) =
p θ ( x | z ) p 0 ( z ) /p
θ ( x ) is approximately the pushforward through the inverse mean decoder g θ ∗ [ p θ ( x | z )]
(see Appx.
A for more details).
If we select a given latent z 0 and denote its image by f θ ( z 0 ) , then we can locally linearize g θ by its Jacobian J g θ =
J g θ
( f θ ( z 0 )) , yielding a Gaussian for the pushforward distribution g θ ∗ [ p θ ( x | z )] with covariance 1 / γ 2 J g θ J T g θ .
As the sufficient statistics of a Gaussian are given by its mean and covariance, the structure of the posterior covariance Σ ϕz | x (which is by design diagonal, cf.
(3)) is crucial for minimizing the gap in (2).
Practically, this implies that in the zero gap limit, the covariances of q ϕ ( z | x ) and p θ ( z | x ) should match, i.e., 1 / γ 2 J g θ J T g θ will be diagonal with entries σ ϕ k ( x ) 2 and therefore J g θ has orthogonal rows.
We can express the decoder
Jacobian via the inverse function theorem as J f θ ( z 0 ) =
J g θ
( f θ ( z 0 ))
− 1 .
As the inverse of a roworthogonal matrix has orthogonal columns, f θ satisfies the IMA principle.
Additionally, we can relate the variational posterior’s variances to the column-norms of J f θ as σ ϕ k ( x ) 2 = 1 / γ 2 ∥ (cid:2)
J f θ ( z 0 ) (cid:3) : k
∥ − 2 .
The self-consistent ELBO therefore converges to the IMA-regularized log-likelihood (6).
Our argument indicates that minimizing the gap between the ELBO and the log-likelihood encourages column-orthogonality in J f θ by matching the covariances of q ϕ ( z | x ) and g θ ∗ [ p θ ( x | z )] .
When q ϕ ( z | x ) = p θ ( z | x ) , the gap is closed; this is only possible if the decoder is in the IMA class, for which c IMA vanishes and the ELBO tends to an exact log-likelihood .
To the best of our knowledge, we are the first to prove this for nonlinear functions, extending related work for linear VAEs [44].
Implications for identifiability of VAEs.
While previous works argued that the VAE objective favors decoders with a column-orthogonal Jacobian [57, 38], they did not exactly characterize how: our result shows that the self-consistent ELBO tends to a regularized log-likelihood, where the regularization term c IMA explicitly enforces this (soft) constraint.
Thus, it possibly explains why VAEs are successful in learning disentangled representations: namely, the IMA function class provably rules out certain spurious solutions for nonlinear ICA [23], and the IMA-regularized log-likelihood was empirically shown to be beneficial in recovering the true latent factors.
Thus, we speak about embracing the gap , as its functional form equips VAEs with a useful inductive bias.
While the IMA function class has not yet been shown to be identifiabile, such results exist for special cases such as conformal maps ( d = 2 [29], generalized by the very recent work in [4]), isometries [26] and for closely-related unsupervised nonlinear ICA models [69].
In the following, we empirically corroborate that VAEs: 1) recover the ground truth sources when the mixing satisfies
IMA, and thereby 2) achieve unsupervised disentanglement.
Experiments
Our experiments serve three purposes: 1) demonstrating that self-consistency holds in practice (§ 4.1); 2) showing the relationship of the self-consistent ELBO ∗ , the IMA-regularized and unregularized log-likelihood objectives (§ 4.2); and 3) providing empirical evidence that the connection to the IMA function class in VAEs can lead to success in learning disentangled representations (§ 4.3).
More details are provided in Appx.
F.
4.1 Self-consistency in practical conditions
Experimental setup.
We use a 3-layer Multi-Layer Perceptron (MLP) with smooth Leaky ReLU nonlinearities [22] and orthogonal weight matrices—which intentionally does not belong to the IMA class, as our results are more general.
The 60,000 source samples are drawn from a standard normal distribution and fed into a VAE composed of a 3-layer MLP encoder and decoder with a Gaussian prior.
We use 20 seeds for each γ 2 ∈ { 1e1; 1e2; 1e3; 1e4; 1e5 } .
Results.
Fig.
2 summarizes our results, featuring the logarithms on each axes.
The left plot shows that the posterior variances σ ϕ k ( x ) 2 converge to zero with a 1 / γ 2 rate, as predicted by (9).
The center plot shows that the expression for σ ϕ k ( x ) 2 corresponds to (46) in the optimum of the ELBO by comparing both sides of the equation.
The right plot shows approximate convergence of the mean encodings µ (cid:98) ϕ
( x ) to g θ ( x ) with a 1 / γ rate (see § 5).
As f θ is not guaranteed to be invertible, we use instead the optimal encoder and decoder parameters to compare f θ ( µ (cid:98) ϕ ( x )) to x .
4.2 Relationship between ELBO ∗ , IMA-regularized, and unregularized log-likelihoods
over different γ 2 .
Error bars are omitted as they are orders of magnitudes smaller
Experimental setup.
We use an MLP f θ with square upper-triangular weight matrices and invertible element-wise nonlinearities to construct a mixing not in the IMA class [23] and fix the VAE decoder to the ground truth such that (4) gives the true data log-likelihood.
This way, we ensure that the unregularized and IMA-regularized log-likelihoods differ and make the claim of Nielsen et al.
[50] comparable to ours.
With a fixed decoder, the ELBO ∗ depends only on ϕ , therefore we only train the encoder with γ 2 values from [1e1; 1e5] (5 seeds each).
Results.
Fig.
4 compares the difference of the estimate of ELBO ∗ and the unregularized/IMAregularized log-likelihoods after convergence over the whole dataset.
As the decoder and the data are fixed, log p θ ( x ) and C IMA will not change during training, only the ELBO ∗ does.
The figure shows that as γ → + ∞ , ELBO ∗ approaches L IMA ( f θ , z ) , as predicted by Thm. 1, and not log p θ ( x ) , as stated in [50]—the difference is C IMA .
Experimental setup (synthetic).
We use 3-dimensional conformal mixings (i.e., the Möbius transform [53]) from the IMA class with uniform ground-truth and prior distributions.
Our results quantify the relationship of the decoder Jacobian’s IMA-contrast and identifiability with MCC [27] and show how this translates to disentanglement—we note that MCC was already used to quantify disentanglement [72, 37].
To determine whether a mixing from the IMA class is beneficial for disentanglement, we apply a volume-preserving linear map after the Möbius transform (using 100 seeds) to make c IMA ̸ = 0 .
Other parameters are the same as in § 4.1, with the exception of picking the best γ 2 = 1e5 .
Results (synthetic).
The left of Fig.
3 empirically demonstrates the benefits of optimizing the
IMA-regularized log-likelihood.
By increasing γ 2 , MCC increases, while c IMA decreases, suggesting that VAEs in the near-deterministic regime encourage disentanglement by enforcing the IMA principle.
The right plot shows that when the mixing is outside the IMA class, MCC decreases, corroborating the benefits of such mixings for disentanglement.
Experimental setup (image).
We train a VAE (not β -VAE) with a factorized Gaussian posterior and Beta prior on a Sprites image dataset generated using the spriteworld renderer [66] with a Beta ground truth distribution.
Similar to [32], we use four latent factors, namely, x- and y-position, color and size , and omit factors that can be problematic, such as shape (as it is discrete) and rotation (due to symmetries)
Our choice is motivated by [26, 18] showing that this data-generating process may approximately satisfy the IMA principle.
Results (image).
The left of Fig.
5 indicates that VAEs can learn the true latent factors and MCC is anticorrelated with c IMA , reinforcing the hypothesis that the data-generating process belongs to the
IMA class.
The center plot compares estimated and true latent factors from the best model (scaling and permutation indeterminacies are removed), whereas the right plot shows the corresponding latent interpolations—thus, connecting identifiability (measured by MCC) to disentanglement.
Limitations
The near-deterministic regime.
Our theory relies on γ → + ∞ ; this is the regime where posterior collapse may be avoided [44], and where calculating the reconstruction loss may be possible even without sampling [38].
However, in practice it may be unclear when γ 2 is large enough.
This seems to be problem-dependent [57, 44], and possibly tied to the covariance of the observations [60, 59].
Moreover, large values of γ 2 may be harder to optimize due to an exploding reconstruction term in (1).
This may be one explanation for the slight deviation of Fig.
2, right from our theory’s predictions: while convergence of µ ϕ ( x ) to g θ matches the prediction in Prop. 1, its rate is not precisely the one predicted for the self-consistent ELBO (19).
Another cause could be the encoder’s finite capacity.
Nonetheless, we have experimentally shown that for realistic hyperparameters, VAEs’ behavior matches the predictions of our theory for the near-deterministic regime.
Dimensionality.
The setting in § 3 requires equal dimensionality for observations x and latents z , in line with work on normalizing flows [51] and nonlinear ICA [28, 31, 24] (but see, e.g., [33]).
For high-dimensional images, however, it is often assumed that x lives on a lower-dimensional manifold embedded in a higher-dimensional space, where the dimensionality of x is greater than z [13].
While our theoretical results do not cover this case, we observe empirically in Fig.
5 that the predictions of our theory remain accurate when observations are high-dimensional images.
Extending our theory to this setting could leverage ideas explored in, e.g., [13, 12, 7] and is left for future work.
The ELBO, the self-consistent ELBO, and amortized inference.
There are in principle multiple ways to obtain self-consistency (Defn. 1).
Notably, one could simply force the variational mean and variance encoder maps to behave this way; unlike [38], we model the actual behavior of VAEs trained under ELBO maximization, and obtain self-consistency as a result.
For this, we assume that the optimal encoder, which minimizes the gap between ELBO and log-likelihood, can be learned.
This is not guaranteed in general, since it requires universal approximation capability of the encoder.
On the other hand, (19) requires unamortized inference to introduce ELBO ∗ , which does not depend on ϕ .
As in practice amortized inference may be used to efficiently estimate a single set of ϕ for all x [61], it can lead to a suboptimal gap to the log-likelihood and discrepancies with our theoretical predictions.
Discussion
On disentanglement in unsupervised VAEs.
It is widely believed that unsupervised VAEs cannot learn disentangled representations [42, 33], motivating work on models with, e.g., conditional priors [33] or sparse decoding [47].
We show that under certain assumptions, ELBO optimization can implement useful inductive biases for representation learning, yielding disentangled representations in unsupervised VAEs.
However, while our results are formulated for VAEs, some of the most successful models at disentanglement are modifications thereof—e.g., β -VAEs [25, 5], with an additional parameter β multiplying the KL in (1).
While they deviate from the information projection setting considered in § 3.1, their objectives are equivalent to the ELBO in a sense described in
Appx.
A.3, which allows us to derive convergence to the IMA-regularized likelihood objective for γ / √ β → + ∞ .
This encompasses the deterministic limit, and also the setting β → 0 with constant γ described in [38].
Whether this theoretical regime matches common practice remains an open question.
Overall, we stress that we uncover one possible mechanism through which VAEs may achieve disentanglement.
By connecting to IMA [23], we discuss implications on recovering the ground truth under suitable assumptions, extending uniqueness results presented in [38].
We speculate that our success in disentanglement is probably due to selecting data sets where the mixing is in the IMA class (cf.
[26, 18]), which presumably was not the case in [42].
Characterizing the ELBO gap for nonlinear models.
Thm. 1 characterizes the gap between
ELBO and true log-likelihood for nonlinear VAEs, and extends the linear analysis of Lucas et al.
[44] and the results of Dai et al.
[14] in the affine case; we also empirically characterize the gap in the deterministic limit in § 4.2.
An unanticipated consequence of this result is that—consistent with [44]—VAEs optimize the IMA-regularized log-likelihood in the near-deterministic limit, and not the unregularized one, as stated in [50].
Extensions to related work.
Several papers discuss the (near-)deterministic regime [50, 57, 38, 13].
For example, Nielsen et al.
[50] postulate a deterministic VAE with the encoder inverting the decoder.
Also Kumar and Poole [38] work in that regime, but without justifying the relationship between the encoder and decoder.
Although they show that the choice of p 0 ( z ) and q ϕ ( z | x ) influences uniqueness (by, e.g., ruling out rotations), this does not imply recovering the true latents.
Our approach formalizes (Defn. 1), proves (Prop. 1), and demonstrates the practical feasibility of (§ 4) the near-deterministic regime.
To the best of our knowledge, all previous work relied on the linear case [44] or a (linear) approximation and the evaluation of the ELBO around a point to show the inductive bias on the decoder Jacobian.
However, our main result (Thm. 1) yields a nonlinear equation where the decoder
Jacobian can be evaluated at any point and is equipped with a convergence bound.
Moreover, the consistency of VAE estimation for identifiable models [33] requires guarantees on q ϕ ( z | x ) ; our result helps proving these.
Dai and Wipf [13] use a non-factorized Gaussian variational posterior and prove in their Thm. 2 (including the dim x = dim z case) that in the deterministic limit their κ -simple VAE can fit perfectly arbitrary observed data (barring few assumptions), while the ELBO gap tends to zero.
In contrast, we use a factorized variational posterior; this prevents the ELBO gap to vanish in the deterministic limit, except in the special case of a decoder mean in the IMA class fitting the data perfectly.
Dai and Wipf [13] take the limit of γ → + ∞ (here using γ as the square root of the decoder precision and not the decoder variance as used in [13]) to relate encoder and decoder properties in this limit in their Thm. 5, similarly to Prop. 1.
In contrast to our nonlinear analysis, this is derived when optimizing w.r.t.
both encoder and decoder parameters, and with a non-factorized encoder assumption, leading to fundamentally different behavior of the solutions in the deterministic limit.
The work done by Sliwa et al.
[62], simultaneously to ours, showcases an extensive empirical study highlighting that the IMA contrast allows distinguishing true and spurious solutions for a broad range of cases and outperforms standard regularizers such as weight decay.
We discuss extended connections to the literature in Appx.
D and Appx.
E.
Covariance structure and IMA.
We have shown that specific choices for encoder and decoder covariances regularize the decoder Jacobian, such that closing the ELBO gap constrains the decoder to belong to the IMA class.
Following our intuition (Fig. 1), assuming factorized q ϕ ( z | x ) and isotropic p θ ( x | z ) , IMA holds only for the decoder ; since in the other direction the pushforward of q ϕ ( z | x ) through f θ has covariance J f θ ( z ) Σ ϕz | x J f θ ( z ) T , which cannot be simplified in the general case.
Additionally, we conjecture that assuming an isotropic encoder would constrain IMA to hold in both encoding and decoding directions (as both J f θ ( z ) and J g θ ( x ) need to be column-orthogonal), such that the resulting decoder mean is constrained to have orthogonal columns of equal norms, which is a defining property of conformal maps [4].
On the other hand, we conjecture that if the observation model is not isotropic, but the encoder mode is, IMA would only be enforced for the mean encoder
Jacobian, converging to the inverse decoder mean in the deterministic limit.
Implications for recovering the true latent factors using unsupervised VAEs.
Convergence of the
ELBO to the IMA-regularized log-likelihood suggests that unsupervised VAEs may recover the true factors of variation according to current identifiability results of the IMA class [4].
This is based on the following reasoning:
If the ground truth generative model belongs to the IMA class, unsupervised learning of the model with an infinite capacity VAE will, in the deterministic limit, ensure a solution that perfectly fits the data and whose decoder mean is also in the IMA class (by joint optimization of both the likelihood and the regularization term).
Identifiability of the IMA class implies that the VAE will learn the true decoder (up to acceptable ambiguities); then, since self-consistency guarantees that the encoder inverts the decoder, the encoder infers the ground truth generative factors associated to observations.
Although strict identifiability for all functions in the IMA class remains to be proven, three concurrent papers provide guarantees that go towards identifiability:
Leemann et al.
[41] proves identifiability for a subset of the IMA class in the context of concept discovery; Zheng et al.
[70] shows identifiability of nonlinear ICA by assuming a specific sparsity structure of the decoder Jacobian (called structural sparsity ); whereas Buchholz et al.
[4] introduce the concept of local identifiability and proves that IMA is locally identifiable.
Moreover, as mentioned in the above paragraph, we suspect that closing the ELBO gap with an isotropic encoder (while the encoder in Thm.
1 is only constrained to have diagonal covariance) constrains the decoder to be a conformal map.
This is an interesting constraint, as nonlinear ICA with conformal mixings are identifiable: the two-dimensional case was first addressed with some additional constraints in [29], while the general case (in arbitrary dimension) was shown to rule out certain spurious solutions for conformal mixings [23], and finally proven to be identifiable by
Buchholz et al.
[4] in concurrent work.
Hence, we conjecture that given a ground truth generative model with a conformal map from latent to observation space, and an unsupervised VAEs with isotropic Gaussian encoders and decoders, the true latent factors can be recovered.
Conclusion.
We provide a theoretical justification for the widely-used self-consistency assumption in the near-deterministic regime of small decoder variance.
Using this result, we show that the self-consistent ELBO converges to the IMA-regularized log-likelihood, and not to the unregularized one.
Thus, we can characterize the gap between ELBO and true log-likelihood and reason about its role as an inductive bias for representation learning in nonlinear VAEs.
We characterize a set of assumptions under which unsupervised VAEs can be expected to disentangle and we demonstrate this behavior in experiments on synthetic and image data.
Acknowledgments and Disclosure of Funding
The authors thank the anonymous reviewers for their suggestions.
This work was supported by the German Federal Ministry of Education and Research (BMBF): Tübingen AI Center, FKZ: 01IS18039A & 01IS18039B, and by the Machine Learning Cluster of Excellence, EXC number 2064/1 – Project number 390727645.
Wieland Brendel acknowledges financial support via an Emmy
Noether Grant funded by the German Research Foundation (DFG) under grant no.
BR 6382/1-1.
The authors thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for supporting Dominik Zietlow and Patrik Reizinger.
Patrik Reizinger acknowledges his membership in the European Laboratory for Learning and Intelligent Systems (ELLIS) PhD program.
A
Complementary notes
A.1 ELBO decompositions . . . . . . . . . . . . . . . . . . . . . . . . . . .
A.2 Justification of the intuition . . . . . . . . . . . . . . . . . . . . . . . . . . . .
A.3 A connection between the β parameter of β -VAEs and the decoder precision γ 2 .
A.
[Table]
c
[Table]
as the
[Table]
scaled)
left KL measure of diagonality of J T f
θ J f θ . . . . . . . . . .
A.
[Table]
Assessing the value of
[Table]
for recovering the true latents . . . . . . . . . . . .
B Main Theoretical Results
B.1 Proof of Proposition 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
B.2 Proof of Theorem 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
C Auxiliary results
C.1 Squared norm statistics . .
C.
[Table]
KL divergence bounds .
C.3 Taylor formula-based approximations . .
C.4 Variational posterior variance optimization problem . . . . . . . . . . . . . . .
D
Related
work
D.
[Table] [Table]
E Further remarks on the the IMA–VAE connection
E.1 Linear VAE from Lucas et al.
F Experimental details
F.
The relationship of weight matrix structures and the IMA function class . . . . .
F.2 Self-consistency in practical conditions (§ 4.1) . . . . . . . . . . . . . . . . . .
F.3 Relationship between ELBO ∗ , IMA-regularized, and unregularized log-likelihoods (§ 4.2) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
F.4 Connecting the IMA principle, γ 2 , and disentanglement (§ 4.3) . . . . . . . . .
F.5 Optimality of γ 2 w.r.t.
its MLE . . . . . . . . . . . . . . . . . . . . . . . . . .
G Computational resources
H Societal impact
I Notation
Acronyms . .
Nomenclature . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
A Complementary notes A.1 ELBO decompositions
Connection between (1) and (2) .
Here we show how the two decompositions of the ELBO objective in (1) and (2) can be connected.
We start from equation (2):
[Equation]
By definition of KL-divergence
[Equation]
and applying Bayes rule
[Equation]
we get
[Equation]
We observe that the two terms involving p θ ( x ) cancel, resulting in
[Equation]
which leads to (1) by rearranging the terms:
[Equation]
Expressions for the two terms in equation (1) under Assum.
The above two terms take the following form in our setting.
For the second (“KL”) term
[Equation]
we get
[Equation]
, where H denotes the entropy.
Writing the expression for the entropy of univariate Gaussian variables ( 1 / 2 log(2 πσ 2 ) + 1 / 2 ), we have under Assum.
[Equation]
where we introduce the dimension dependent constant κ d = d 2 (log(2 π ) + 1) .
This leads to
[Equation]
The first (“reconstruction”) term, under the isotropic Gaussian decoder of Assum.
[Equation]
takes the form
[Equation]
Expression for the gap between ELBO and log-likelihood Let us now write the KL divergence between variational and true posteriors, which is the gap appearing in (2)
[Equation]
Using again the expression of the entropy of Gaussian variables
[Equation]
this leads to
[Equation]
such that, using the Bayes formula for the true posterior and Assum.
1, we get
[Equation]
with additive constant c ( x , γ ) = − d 2 (cid:0) log( γ 2 ) + 1 (cid:1)
+ log p θ ( x ) .
Note the log(2 π ) term in the previous expression cancels with the one coming from the true log posterior.
The analysis of the optima of (16) is non-trivial due to the second term which involves taking expectations of functions of z w.r.t.
its posterior distribution q ϕ parameterized by µ ϕ and σ ϕ .
Much of the derivations to obtain our results will revolve around constructing bounds that no longer involve such expectations, but instead only depend on µ ϕ and σ ϕ .
A.2 Justification of the intuition
We add here more qualitative details to the statement of subsection 3.2 that the true posterior density is approximately the pushforward of p θ ( x | z = z 0 ) .
Note that they are not meant to replace a rigorous treatment, which is deferred to Appx.
B.
As the decoder becomes deterministic, the marginal observed density becomes the pushforward of the latent prior by f θ 5 such that
[Equation]
The true posterior is therefore approximately
[Equation]
Conditioning on a given observation x = f θ ( z 0 ) , we get p θ ( z | x = f θ ( z 0 ))
=
p θ ( f θ ( z 0 ) | z ) p 0 ( z ) /p
θ ( x = f θ ( z 0 ))
[Equation]
Neglecting the variations of the prior relative to those of the posterior (due to near-determinism), we make the approximation p
( z ) ≈ p 0 ( z 0 ) such that the above approximation becomes
[Equation]
Using the isotropic Gaussian decoder assumption, we get
[Equation]
In the near-deterministic regime, this posterior distribution should be concentrated in the region where z is close to z 0 , we can then further approximate this density using a Taylor formula p θ
[Equation]
|
[Equation]
with G =
J g θ
( f θ ( z 0 ))
=
J f θ ( z 0 ) − 1 , which is also matching the expression of the pushforward of the Gaussian density p θ ( x | z
= z 0 ) by the linearization of g θ around f θ ( z 0 ) (i.e. replacing the mapping by its Jacobian at that point, G ).
A.3 A connection between the β parameter of β -VAEs and the decoder precision γ
In the context of disentanglement, a commonly used variant of standard VAEs [35] is the β -VAE [8, 25, 34, 57, 38].
In this model, an additional parameter β is added to modify the weight of the KL term in (1), whereas the decoder precision γ 2 is typically set to one [13, 20, 38, 57].
The β -VAE objective [25] can be written as
[Equation]
The influence of the decoder precision γ 2 and the β parameters on the objective have been related in the literature, see for example [17, § 2.4.3]—and similar observations can be found in [59, § 3.1].
Under the assumption of a Gaussian decoder, the ELBO from eq.
(1) can be written as (making now explicit mention of its decoder parameter γ in parenthesis):
[Equation]
with c ( γ, d ) =
d log γ − d 2 log(2 π ) .
In contrast, the β -VAE objective L β ( x ; θ , ϕ ) (also with explicit mention of γ ) is expressed as:
[Equation]
We thus can link this expression to an ELBO as follows:
[Equation]
When we restrict ourselves to common practice
[Equation]
the optimizations of both the ELBO and the L β are performed with a fixed value of γ (and with fixed β for L β ).
This entails that there is an equivalence of the solutions resulting from the optimization of each objective
[Equation]
as they differ only by additive and multiplicative constants.
In particular
[Equation]
given the above expression, the following result is immediate:
Proposition 2.
Let us define the self-consistent β -VAE objective as
[Equation]
Then the following normalized self-consistent β -VAE objective
[Equation]
converges to the regularized IMA objective as γ √ β → + ∞ under the same conditions as in Thm.
Proof.
The β -VAE objective is, up to an additive constant and a strictly positive multiplicative constant, identical to the ELBO objective.
As a consequence their optimum and the values of parameters ϕ and θ at which they are achieved are identical.
This suggests that choosing a fixed β or a β growing as, for example, log γ or even √ γ would lead to self-consistent solutions with the same properties as the vanilla VAE in the deterministic decoder limit γ → + ∞ , and notably robustness to spurious solutions.
Generalizing Kumar and Poole [38] Moreover, this connection also shows that our result generalizes previous work.
Particularly, considering γ 2 = 1 when expressing the optimal variance analogous to eq.
(55), we can discover the same expression as in [38].
Namely, the objective function has the form (where we now indicate explicitly the dependence on γ and emphasize that both γ and β are fixed):
[Equation]
As a consequence, the optimal σ ϕ k ( x ) 2 now includes β compared to (55):
[Equation]
That is, β affects the decoder Jacobian column norms as 1 / γ 2 .
They are not exactly equivalent though—note the β factor in front of the log-prior terms:
[Equation]
Again, as both β and γ are fixed, the additive/multiplicative constants are irrelevant.
Thus, β has a similar effect on column-orthogonality as 1 / γ 2 .
We formalize this observation in the following remark:
Remark 1.
β affects the column norms of the decoder Jacobian in the same way as 1 / γ 2 .
One can think of having γ ′ =
γ/ √ β and tying the tuning of β, γ ′ .
Remark 2 (Generalization of Kumar and Poole [38]) .
Assuming the same conditions as in Thm. 1 and γ 2 = 1 (i.e., a Gaussian decoder with a Hessian H = I d and Gaussian prior with I d as second-order derivative) with the β -VAE loss, then expressing the optimal posterior covariance with γ / √ β , we get
Eq.
(11) of Kumar and Poole [38],
[Equation]
as a special case.
A.4 c IMA as the (scaled) left KL measure of diagonality of J T f θ J f θ
In our paper, we used the original definition for c IMA ([23, (8)]), which we restate here:
[Equation]
However, an alternative formulation exists: namely, c IMA ( f θ , z ) can be thought of as the (scaled) left
KL measure of diagonality of the square matrix J f θ ( z ) T J f θ ( z ) , as shown in [2].
That is, we can rewrite the above as [23, (23)]
(with A =
J f θ ( z ) T J f θ ( z ) ):
[Equation]
The importance of eq.
(23) is twofold: i) it provides a theoretical analysis of c IMA as a measure of the column-orthogonality of J f θ ( z ) (or, equivalently, the diagonality of J f θ ( z ) T J f θ ( z ) ); and ii) it elucidates why c IMA can be used in the dim x ̸ = dim z case, but only to measure columnorthogonality (to exploit the beneficial properties of IMA for identifiability, the theory of IMA needs to be extended to this case).
We leverage ii) as a justification to use c IMA in our image experiments in § 4.3.
A.5 Assessing the value of c IMA for recovering the true latents
c IMA , given by (6), measures the deviation of the learned decoder from the IMA function class.
As it is positive and unbounded, it is practically relevant to investigate the following question: how much violation of the IMA assumption is acceptable to recover the true latents?
Expressed differently, we are interested in whether a threshold can be specified for c IMA ( f θ , z ) to decide whether the true (but unknown) latent factors can be recovered.
Unfortunately, our answer is negative, but this is not specific to IMA theory.
Even in the case of linear ICA we cannot specify a threshold for the non-Gaussianity of the latent (source) variables to ensure that the ground-truth factors can be recovered [29].
We acknowledge that, among others, c IMA being unbounded makes it harder to interpret.
Thus, we go back to first principles to develop an imperfect but hopefully more practical intuition on how to assess the value of c IMA to decide whether the true latents are recovered (if the true mixing is in the
IMA class).
As we already pointed out in Appx.
A.4, c IMA is the left KL measure of diagonality of J T f θ J f θ [2].
In their analysis, the authors provide closed form solutions expressions to compare different diagonality measures.
For our purposes, [2, Fig. 1] provides an important insight: it shows that in the two-dimensional case c IMA increases nonlinearly in a variable r expressing the degress of diagonality of a matrix (more precisely as − log (cid:0) 1 − r 2 (cid:1) , where r = 0 denotes the identity matrix, whereas r = 1 a matrix with parallel columns).
The takeaway for us is that simply comparing two c IMA values can be misleading.
For the two-dimensional case, we could define an expression that is linear in r (but this requires us to accept that r is a -in some sense- suitable measure of the diagonality of a matrix), namely:
[Equation]
Unfortunately, the d -dimensional case only has a power series formulation; thus, it is nontrivial how to extend the above reasoning.
However, eq.
(24) expresses c IMA as the negative log determinant of a scaled matrix, where the determinant of diag ( A ) −
1 / 2 A diag ( A ) − 1 / 2 is between 0 and 1 ( A =
J f θ ( z ) T J f θ ( z )) .
Thus, we can convert c IMA to the [0; 1] interval as follows:
[Equation] [Equation]
[Equation]
The merit of the above expression is a more natural way to compare values that are normalized to the [0; 1] interval.
Namely, the original formulation of c IMA may potentially lead to problems, as, e.g., a value of t and 10 t does not mean that one model is ten times better in recovering the true latents.
Nonetheless, this can be thought (at most) as a small step towards making the analysis of the empirical value of c IMA a useful tool in practical scenarios, where we do not have access to the true latent factors.
B Main Theoretical Results B.1 Proof of Proposition
We proceed in two steps: first we prove the existence of variational parameters that achieve a global minimum of the ELBO gap, then we characterize its near-deterministic properties.
We then combine these results, which rely on specific assumptions, to obtain our main text result under Assum.
We initially use the following milder assumptions than in main text to prove intermediate results.
Assumption 2 (Gaussian Encoder-Gaussian Decoder VAE, minimal properties) .
We are given a fixed latent prior and three parameterized classes of R d → R d mappings: the mean decoder class θ (cid:55)→ f θ , and the mean and standard deviation encoder classes, ϕ
(cid:55)→ µ ϕ and ϕ (cid:55)→ σ ϕ such that
(i the latent prior has a factorized independent and identically distributed (i.i.d.)
density p
( z ) ∼ (cid:81) k m ( z k ) , with m smooth fully supported on R , with concave log m , (
ii
) conditional on the latent, the decoder has a factorized Gaussian density p θ with mean f θ such that
[Equation]
(iii) the encoder is factorized Gaussian with posterior mean and variance maps µ ϕ
[Equation]
x ) , σ ϕ
[Equation]
x
for each component k , leading to the factorized posterior density q ϕ
( z | x ) such that
[Equation]
(iv) the mean and variance encoders classes can fit any function, (v) for all possible θ , f θ is a diffeomorphism of R d with inverse g θ .
Existence of at least one global minimizer of the gap between true and variational posterior is given by the following proposition.
Proposition 3 (Existence of global minimum) .
Under Assumption 2.
For a fixed θ assume additionally that g θ is Lipschitz continuous with Lipschitz constant B > 0 , in the sense that
[Equation]
Then there exists at least one choice ( µ ϕ ∈ R d , σ ϕ ∈ R d> 0 ) that achieves the minimum of
KL [ q ϕ ( z | x ) || p
θ ( z | x )] .
Proof.
Using Prop. 7, we have the lower bound
[Equation]
We then notice (see lemma 4) that for all k ,
[Equation]
achieves a global minimum n ( B, γ ) =
− log( B/γ ) + 1 / 2 at σ ϕ k ( x ) =
B/γ .
For arbitrary k 0 , we now 1) lower bound the k ̸ = k 0 terms by n ( B, γ ) ; 2) lower bound and all the log m terms by their global maximum, which exists by Assum.
1i (log-concave prior); and 3) drop the non-negative squared norm term, leading to the following weaker lower bound:
[Equation]
The KL divergence is well-defined and finite for any choice of parameters in their domain, therefore it achieves a particular value K 0 ≥
at one arbitrary selected point of the domain.
Since for all k , the lower bound tends to + ∞ for both σ ϕ k → + ∞ (as the quadratic term dominates the − log term) and σ ϕ k → 0 + , there exist a > b > 0 (possibly dependent on ( γ, x ) )
such that
Moreover, starting again from the lower bound from Prop. 7,
[Equation]
we now focus on µ ϕ and lower bound all σ ϕ terms.
With this, we get the following weaker lower bound in terms of µ ϕ :
[Equation]
The lower bound also tends to + ∞ for ∥ µ ϕ ∥ → + ∞ , so there exists a radius R > 0 (possibly dependent on ( γ, x ) ) such that KL [ q ϕ ( z | x ) || p
θ ( z | x )] > K
if ∥ µ ϕ ∥ > R .
As a consequence, the infimum ( ≤ K 0 ) of the minimization problem (7) cannot be achieved outside the compact set ( µ ϕ , σ ϕ ) ∈ { µ ϕ ∈ R d : ∥ µ ϕ ∥ ≤ R } ×
[ a, b ] d .
Since the divergence is continuous in ( µ ϕ , σ ϕ ) , there exists a value ( µ (cid:98) ϕ , σ (cid:98) ϕ ) in this compact set achieving the minimum of the KL over the whole parameter domain, and all values achieving this minimum are in this compact set.
For given x , θ and γ > 0 , the variational posterior KL divergence mapping
[Equation]
thus has a minimum, and by smoothness of this mapping, this minimum can be characterized by the vanishing gradient of the KL divergence with respect to the parameters.
Now, let us try to characterize how this minimum behaves for large γ
[Equation]
Proposition 4 (Self-consistency of the encoder in the deterministic limit) .
Under Assum.
2, assume additionally f θ and g θ are Lipschitz continuous with respective Lipschitz constants C, B > 0 , in the sense that
[Equation]
Assume additionally that − log m is quadratically dominated, in the sense that
[Equation]
Then for all x
[Equation]
θ , as γ → + ∞
[Equation]
any global minimum of (7) satisfies
[Equation]
More precisely, for all x ∈ R d , γ >
[Equation]
[Table]
[Equation]
and d
[Table]
[Equation]
Proof.
We start from the lower bound expression of Prop.
[Equation]
with c ( x , γ ) = − d
[Equation]
log( γ 2 )
+
[Equation]
+ log p θ ( x ) .
For any ν ∈ (0 , 1] , we can thus write
[Equation]
Now, from lemma 4 we get
[Equation]
We exploit this lower bound to obtain
[Equation]
d
[Equation]
Using the expression of c ( x , γ ) we get
KL
[Equation]
and both the “ d log γ ” as well as “ d / 2 ” terms cancel out such that
[Equation]
Finally, using Prop. 8, the above right hand side is bounded from above by a constant as γ → + ∞ , and as a consequence, the positive factor of the γ 2 term must vanish (by continuity assumption and its limits note − log m is bounded from below)
[Equation]
This entails that both positive terms it comprises must vanish too.
More precisely, we get the inequality between lower and upper bounds at the optimal solution
[Equation]
which simplifies to
[Equation]
−
[Equation]
Moreover by continuity assumption and its limits
[Equation]
− log m is bounded from below by − M = − max t log m ( t ) , yielding
[Equation]
+
γ 2 2 B − 2 (cid:34)(cid:13)(cid:13)(cid:13) g θ ( x ) − µ (cid:98) ϕ
( x ) (cid:13)(cid:13)(cid:13) 2 + (1 − ν ) d (cid:88) k
= σ (cid:98) ϕ
k
( x )(cid:35) ≤ d (cid:18)2( C 2 − 1) + E + D (cid:20) ∥
g θ ( x ) ∥d +γ(cid:21)(cid:19) such that γ
[Equation]
(cid:13)(cid:13)(cid:13)
[Equation]
ν ) d (cid:88) k
= σ (cid:98) ϕ
[Equation]
( x )
[Equation]
(cid:35)
[Equation]
(cid:18)
[Equation] [Equation]
[Equation]
D (cid:20)
[Equation]
( x )
[Equation]
d +γ(cid:21) − (cid:0) log( νB − 2 ) − 2 M (cid:1)(cid:19) and finally
B − 2 (cid:34)(cid:13)(cid:13)(cid:13) g θ ( x )
[Equation]
µ (cid:98) ϕ
( x ) (cid:13)(cid:13)(cid:13) 2 + (1 − ν ) d (cid:88) k
= σ (cid:98)
[Equation]
k ( x )(cid:35)
[Equation]
d
[Equation]
(cid:18)
[Equation] [Equation]
D (cid:20)
[Equation]
( x ) ∥d
[Equation]
γ
[Equation]
(cid:21) + M
[Equation] [Equation]
log( B
[Equation]
/ν ) (cid:19) (39)
Taking ν = 1 in (39) we get the first intended inequality
[Equation]
Alternatively, (39) implies
[Equation]
Taking a fixed value of ν , say 1 / 2 , we get the second intended inequality d (cid:88)
[Equation]
We now restate the main text proposition and provide the proof.
Proposition 1.
[Self-consistency of near-deterministic VAEs] Under Assumption 1, for all x , θ , as γ → + ∞ , there exists at least one global minimum solution of (7).
These solutions satisfy
[Equation]
k
[Equation]
Proof.
We only have to check that Assum.
1 allow fulfilling the following requirements of Prop. 4:
• the Lipschitz continuity requirements in Prop. 4 results from the boundedness of the first order derivatives of the decoder mean and of its inverse (by using the multivariate Taylor theorem), • concavity of log m , required by Assum.
2, is a direct consequence of non-positivity of the second-order logarithmic derivative of m in Assum.
1i, • quadratic domination of − log m comes from the boundedness of the second-order logarithmic derivative of m (by integrating twice).
Then Prop. 4 follows and the O ( 1 / γ ) convergence of the variational posterior mean of the inverse, as well as the O ( 1 / γ 2 ) convergence of the variational posterior variance.
Finer approximation of parameter values We now derive a finer result for the convergence of the mean, that we will exploit in Thm. 1.
This relies on the existence of an optimum shown by Prop. 3.
At such optimum (cid:98) ϕ we thus have for all k
[Equation]
and
[Equation]
We derive the constraints entailed by the first expression:
[Equation]
with
[Equation]
which leads to a set of constraints at optimum
[Equation]
Based on this expression we derive the following result.
Proposition 5.
Under Assum.
1, as γ → + ∞
[Equation]
and
[Equation]
Proof.
We start from the constraints of (40) that we rewrite
[Equation]
We then proceed to approximate the left hand side using a Taylor formula.
Assuming bounded
Hessian components, we can upper and lower bound using third order centered absolute moments of the Gaussian as
[Equation]
which we can rewrite (by 1) expanding the norm of the sum; 2) removing constants in the bracket, which lead to zeros after multiplying the zero mean variable and taking the expectation; 3) using
Gaussianity, all centered third order terms vanish
[Equation]
[Equation]
Finally computing this integral we get the left hand side as
[Equation]
For the right hand side we get using a Taylor expansion (with notation n : z → log( m ( z )) )
[Equation]
Equating the non-negligible terms of the left and right-hand sides we get for each k
[Equation]
such that
[Equation]
where n ′ is applied component-wise.
Because the Jacobian is everywhere invertible (implicit consequence of Lipschitz assumptions), we can solve for this equations and get
[Equation]
[Equation]
Using again a similar Taylor approximation we get
[Equation]
This equation has the shortcoming of still referring to the posterior mean on both sides.
To fix this, we first note that it implies, by boundedness of the Jacobian, that
[Equation]
By bounding the second-order derivative of the log prior, we get
[Equation] [Equation]
, which implies
[Equation]
i.e., we obtain an improved convergence rate.
Using this rate and Taylor theorem, we obtain the final equation by replacing the variational posterior mean by the inverse decoder in (43)
[Equation]
B.2 Proof of Theorem
This will be a corollary of the following result, that uses as a key assumption a rate of O ( 1 / γ 2 ) in the convergence of the self-consistency equation of the variational mean.
Proposition 6 (VAEs with log-concave factorized prior and close-to-deterministic decoder approximate the IMA objective) .
Under Assum.
1, if additionally the VAE satisfies the following self-consistency in the deterministic limit
[Equation] [Equation]
45) then
[Equation]
and the self-consistent ELBO (19) approximates the IMA -regularized log-likelihood (6) :
[Equation]
Proof.
We start from the self-consistent ELBO decomposition as “reconstruction error plus posterior regularization” terms:
[Equation]
and continue with reformulating both terms, based on Assum.
That is, p 0 is factorized with components i.i.d.
distributed according to a fully supported log-concave density z k ∼ m .
Posterior regularization term Assum.
1 gives us the formula of (14) for this term in the ELBO.
Taking optimal encoder parameters, we get the posterior regularization term for the ELBO ∗
[Equation]
with κ d = d 2 (log(2 π ) + 1) .
Using the factorized Gaussian encoder and i.i.d.
prior assumptions we get
[Equation]
where we rewrote the distribution p 0 as p
= (cid:81) k m ( z k ) .
Based on the Taylor theorem, with a residual in Lagrange form of n = log m , we have that for all k and u there exists ξ ∈ [ µ ϕ k ( x ) , u ] if u ≥ µ ϕ k ( x ) , or ξ ∈ [ u, µ ϕ k ( x )] if u ≤ µ ϕ k ( x ) such that n ( u ) =
log( m ( u ))
=
log( m ( µ (cid:98) ϕ
k ( x )))
+ n ′ ( µ (cid:98) ϕ k
[Equation]
x
[Equation]
[Equation]
u − µ [Equation] k
[Equation]
x
[Equation]
We assumed that | n (3) | is bounded over R by F , such that
[Equation]
Taking the expectation and using the expression of centered Gaussian absolute moments
[Equation]
As the assumptions entail that optimal posterior variances σ (cid:98)
[Equation]
k ( x ) 2 get small for γ large (cf.
(45)), this implies the near-deterministic approximation
[Equation]
In addition
[Equation]
using again a Taylor formula and the self-consistency assumption for the mean
[Equation]
Moreover, using again a Taylor formula for n ′′ under boundedness of n (3) and again using the self-consistency assumption for the mean yields
[Equation]
Overall this leads to the approximation of the posterior regularization term
[Equation]
Reconstruction term Now switching to the first (reconstruction) term of the ELBO ∗ , adapting the decomposition of (15) by using optimal encoder parameters we get
[Equation]
Then in the small encoder noise limit σ k ( x ) 2 ≪ 1 , ∀ k (justified by Prop. 1), we rely on a Taylor approximation around the posterior mean z o = µ ϕ ( x ) based on Lemma
[Equation]
which bounds this approximation as follows
[Equation]
The linear term in this approximation is easily computed using successively Lemma 1 and Lemmato get an expression with the squared column norms of the partial derivatives scaled by the standard deviations
∂ f θ ∂z k | µ ϕ k ( x ) .
We get
[Equation]
This term can be used as an approximation for the expectation term in the reconstruction loss thanks to the following reverse triangle inequality
[Equation]
  (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)
[Equation]
(cid:33)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
such that the resulting upper bound can be itself bounded as follows
E
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
≤
[Equation]
Each term of the upper bound can be bounded for the optimum encoder parameters: using from left to right the assumption of (44) and (51), respectively, leading to
[Equation]
Getting back to the whole reconstruction term, using additionally the variance self-consistency assumption (45), the above shows that we can make the approximation
[Equation]
We can further replace the dependency of the derivatives on the encoder mean using a Taylor formula for the derivative
[Equation]
such that
[Equation]
ELBO ∗ approximation As a consequence of (50) and (53) the ELBO ∗ becomes
ELBO ∗ ( x ; θ ) = −
[Equation]
where we isolated the terms that depend on parameters σ (cid:98)
ϕ k ( x ) 2 and γ in the approximate objective (cid:92) ELBO ( σ 2 = σ (cid:98) ϕ
( x ) 2 ; x , θ , (cid:98) ϕ ) that we define for arbitrary σ 2 .
[Equation]
Where we further break this objective in d components (cid:92) ELBO k ( σ (cid:98)
ϕ k ( x ) 2 ; x , θ , (cid:98) ϕ ) according to the terms of the sum as follows
[Equation]
and where we note that − n ′′ ≥
due to the log-concavity assumption.
Solving term in k (cid:92) ELBO k ( σ 2 k ) for optimal γ 2 σ ∗ k we get (see lemma 4):
[Equation]
(54) and the resulting optimal value (cid:92) ELBO ∗ k
( x , θ , (cid:98) ϕ )
= (cid:92) ELBO k ( σ ∗ 2 k ; x , θ , (cid:98) ϕ ) is
[Equation]

A Taylor formula around this optimum leads, for some value ξ γ ( x ) lying between σ ∗ 2 k and σ 2 k to (note the first order derivative vanishes, and the second order derivative is upper bounded hence the second line)
[Equation]
as a consequence the non-approximate solution for the true optimal ELBO ∗ , as γ grows, must achieve a value below this quadratic function, up to a term in O ( 1 / γ 2 ) , and at the same time above (cid:92) ELBO ∗ , also up to a term in O ( 1 / γ 2 ) .
This entails that it is restricted to a smaller and smaller domain near the approximate solution and we get
[Equation]
Leading to the approximation of the true objective
[Equation]
which reduces to
[Equation]
which is the IMA objective.
We now restate the main text theorem and provide its proof.
Theorem 1.
[VAEs with a near-deterministic decoder approximate the IMA objective] Under
Assumption 1, the variational posterior satisfies
[Equation]
and the self-consistent ELBO (19) approximates the IMA -regularized log-likelihood (6) :
[Equation]
Proof.
This is just a corollary of Proposition 6 because Proposition 5 entails through (42) the required O ( 1 / γ 2 ) rate of convergence for the optimal variational mean in (44), while (45) is fulfilled through Prop. 1.
C Auxiliary results C.1 Squared norm statistics
Lemma 1 (Squared norm variance decomposition) .
For multivariate RV X with mean m
[Equation]
Proof.
[Equation]
hence
[Equation]
This leads to (using that the trace of a scalar is the scalar itself)
[Equation]
because trace [ AB ] = trace [ BA ] we get
[Equation]
Lemma 2 (Trace of transformed unit covariance) .
When the covariance matrix Cov ( ϵ ) is the identity, then
[Equation]
Proof.
For arbitrary matrix A
[Equation]
Cov ( A ϵ ) =
ACov ( ϵ ) A T and thus
[Equation]
Moreover, in our case Cov ( ϵ ) is the identity such that
[Equation]
C.2 KL divergence bounds
Proposition 7 (Lipschtiz continuity-based lower bound) .
Assume g θ is Lipschitz continuous with
Lipschitz constant B > 0 , in the sense
[Equation]
Then for any encoder parameter choice
[Equation]
with c ( x , γ ) = − d 2 (cid:0) log( γ 2 ) + 1 (cid:1)
+ log p θ ( x ) .
Proof.
Starting from the KL divergence expression (16),
KL
[Equation]
|
[Equation]
with additive constant c ( x , γ ) = − d 2 (cid:0) log( γ 2 ) + 1 (cid:1)
+ log p θ ( x ) .
By Lipschitz continuity
[Equation]
using Lemma 1 applied to g θ ( x ) − z , z ∼ q ϕ
( z | x ) we get
KL
[Equation]
Using Jensen’s inequality for − log m (convex by Assum. 1(i)), we get
KL [ q ϕ ( z | x ) || p
[Equation]
( z | x )] ≥ −
[Equation]
by reordering the terms we finally get
[Equation]
which is the stated KL lower bound
[Equation]
Proposition 8 (Optimal encoder KL divergence upper bound) .
Assume f θ is Lipschitz continuous with Lipschitz constant C > 0 , in the sense that
[Equation]
Assume
[Equation]
− log m is quadratically dominated
[Equation]
in the sense that
[Equation]
Then for the optimal encoder solution of (7)
[Equation]
and
[Equation]
Proof.
Starting from the KL divergence expression (16),
[Equation]
with additive constant
[Equation]
Let us choose the following posterior (by universal approximation capabilities of the encoder):
[Equation]
Using Lipschitz continuity we get
[Equation]
then, using
[Equation]
we get
[Equation]
using quadratic domination
[Equation]
Using Lemma 1 we get
[Equation]
hence for a parameter (cid:98) ϕ
achieving the minimum divergence we get
[Equation]
As γ → + ∞ , log p θ ( x ) → | J f θ ( g θ ( x )) | − 1 p
( g θ ( x )) such that the KL divergence for the optimal solutions is upper bounded by a finite number.
C.3 Taylor formula-based approximations
Lemma 3 (Bound on expectation of multivariate Taylor expansion) .
Assume f : R d
→ R is C 2 and assume z is a multivariate RV on R d with indepedent Gaussian components such that
[Equation]
then for all z o ∈ R d
[Equation]
Proof.
As described in [45, p. 162], for the l -th component of the function f l
[Equation]
z
[Equation]
z
[Equation]
where the second line puts 1 / 2 of the partial derivatives in matrix form (note it is not exactly the
Hessian as derivatives are taken at different points).
As a consequence
[Equation]
where ∥
H k ∥ 2 is the spectral norm of the matrix and ∥ H k ∥ F is the Frobenious norm 7 leading to the bound
[Equation]
where K is an upper bound on the absolute second order derivatives.
We have ( z k − z ok ) =
σ ϕ k ( x ) ϵ k , with ϵ multivariate normal, so taking the expectation of the above simplifies to:
[Equation]
Now gathering all components f l to get the squared norm yields:
[Equation]
C.4 Variational posterior variance optimization problem
Lemma 4.
For α > 0 , the function
[Equation]
is strictly convex and achieves its global minimum min
h α
= 12 log α for u ∗ = 1 √ α .
Proof.
Function h
α is stricly convex as a sum of two stricly convex functions.
Its derivative,
[Equation]
thus vanishes only at the minimum for u ∗ = 1 √ α .
We then get that
[Equation]
D Related work D.1 Implicit inductive biases in the ELBO
Rolinek et al.
[57] reason about the connection to Principal Component Analysis (PCA) in the context of nonlinear Gaussian VAEs with an isotropic prior and assume that the variational posterior has diagonal covariance with distinct singular values .
The authors make it explicit that they investigate the consequences of optimizing the ELBO.
They locally linearize the decoder to show the inductive bias in VAEs that promotes decoder orthogonality.
Their results hold for β -VAEs, where β should be in the range of satisfying the polarized regime assumption (i.e., when the VAE is close to partial posterior collapse).
The validity of the assumptions (polarized regime and distinct singular values in Σ ϕ z | x ) are only experimentally investigated.
The same authors extend their work in [71], completing the connection to PCA for linear models.
Their experiments, inspired by the connection to PCA for linear models, show that local perturbations in the data prohibit disentanglement for non-linear models.
Nakagawa et al.
[49] builds upon the results of [57] and provides a novel interpretation of
VAEs by introducing implicit variables to express the latent space in terms of an isometric embedding.
Lucas et al.
[44] prove that linear Gaussian VAEs with an isotropic prior give rise to a columnorthogonal decoder and therefore uniquely recover the PCA coordinate axes (not just the correct subspace, as Probabilistic Principal Component Analysis (PPCA) [64] does), yielding identifiability for Gaussian models—but only when the eigenvalues of the data covariance are distinct.
In their work, the decoder variance is shown to be small when avoiding posterior collapse.
More interestingly, the authors derive a formula for the ELBO gap in the linear case that is remarkably similar to the
IMA objective.
We show in Appx.
E.1 that in the limit of a deterministic decoder linear Gaussian
VAEs optimize the IMA objective with λ = 1 .
Dai and Wipf [13] present more general results than [44] since they use affine functions.
Additionally, a connection to Robust PCA [6] is established.
Kumar and Poole [38] generalizes [57], as it admits a variational posterior q ϕ (
z | x ) with blockdiagonal covariance with a uniqueness result for diagonal Σ ϕz | x .
The authors derive a formula for the optimal Σ ϕz | x [38, Eq. 12], showing that when the decoder Hessian H is diagonal, the decoder Jacobian will be column-orthogonal even for non-Gaussian decoders.
Their analysis relies on a “concentrated” q ϕ ( z | x ) (i.e., they work in what we term the near-deterministic regime) and sufficiently small values of β —this relationship can be read off from [38, Eq. 12].
Interestingly, the authors also show that rotations of the latents can be ruled out, though they do not connect the decoder structure (especially, column-orthogonality of its Jacobian) to any specific generative model for the data, or to considerations on identifiability of the ground truth sources.
Dai and Wipf [13] use a different setting that turns out to be very interesting to compare to ours.
The most important is that while we use a factorized Gaussian variational posterior, Dai and Wipf [13] use a non-factorized Gaussian, which leads to major differences.
Broadly construed, Dai and Wipf [13] are able to show in their Theorem 2 (which includes the case of equal latent and observation dimensions matching our setting) that in the deterministic limit, their κ -simple VAE can perfectly fit arbitrary observed data (barring few assumptions), while the ELBO gap tends to zero.
The way it is proven relies on a first step with the Darmois construction [29], chosing the decoder mean parameter such that its pushforward is exactly the observation distribution.
Then in a second step, by an appropriate choice of variational posterior parameters, they show that asymptotically the ELBO gap (i.e., the KL divergence between true and variational posteriors) tends to zero in the deterministic limit.
In contrast, our constraint of factorized variational posterior does not allow the ELBO gap to vanish in the deterministic limit (unless the decoder mean that fits the data perfectly is in the IMA class, which is a very special case; in particular, if the Darmois construction is used).
For this reason, the proofs and scope of our results are very different: (i) we use information theoretic bounds to show that the encoder inverts the decoder mean (independently from the fact that this one may or may not fit the data perfectly); (ii) we obtain a rigorous convergence to the IMA regularized likelihood, which demonstrates that the gap is not eliminated in the deterministic limit.
Regarding our result for β -VAEs (Prop. 2), the approach of Mathieu et al.
[46] is similar as the authors show that L β can be expressed in terms of a rescaled ELBO.
The difference is that Mathieu et al.
[46] uses a rescaling of the parameters ϕ , θ , whereas we only scale γ 2 .
D.2 (Near)-deterministic VAEs
Recent work was inspired by the normalizing flow literature and the shortcomings of the stochastic
VAE architecture to propose designs that are (near)-deterministic.
Arguments for this regime range from avoiding posterior collapse (as demonstrated in [44]) to avoiding sampling for the reconstruction loss term [38].
Several papers argued for a similar setting: Dai and Wipf [13] take the limit of γ → + ∞ (here using γ as the square root of the decoder precision and not the decoder variance as used in [13]) to derive a result relating encoder and decoder properties in this limit in their Theorem 5, that has a similar flavor to Prop. 1.
In contrast to our nonlinear analysis, this is derived when optimizing with respect to both encoder and decoder parameter, and as stated in the previous section, the non-factorized encoder assumptions leads to fundamentally different behavior of the solutions in the deterministic limit.
Rolinek et al.
[57] refer to the polarized regime (a property of which is that encoder variances are small, cf.
[57, Definition 1]), Kumar and Poole [38] argue for “concentrated” variational posteriors.
Ghosh et al.
[20] substitute stochasticity with a regularizer on the decoder
Jacobian from an intuitive, whereas Kumar et al.
[40] motivate these results from an injective flow perspective.
Nielsen et al.
[50] also take a normalizing flow perspective to connect VAEs to deterministic models.
Besides benefits of avoiding posterior collapse or possible improvements during optimization, this regime serves as a potential connection to the identifiability literature.
E Further remarks on the the IMA–VAE connection
In this section, we elaborate on the connection between VAEs and IMA, by showing that previous work on linear VAEs can be directly connected to optimizing L IMA .
Our intent with this analysis is to provide additional insights about the role of γ in a simpler setting.
E.1 Linear VAE from Lucas et al. [44]
We restate the linear VAE model of [44]:
[Equation]
where D is a diagonal matrix, W the decoder and V the encoder weights, µ the mean latent representation.
The authors show that in stationary points, the optimal value for D is
[Equation]
If we substitute this expression into the ELBO gap (i.e., the KL between the variational and true posteriors), we get a similar expression to c IMA —as formalized in Prop. 9.
Proposition 9 (The ELBO converges to L IMA for linear Gaussian VAEs if γ → + ∞ ) .
For linear
Gaussian VAEs, in the limit of γ → ∞ , the ELBO equals the IMA -regularized log-likelihood in stationary points with λ = 1 .
Proof.
In [44, Appendix C.2], it is shown that the gap between exact log-likelihood and ELBO for linear Gaussian VAEs in stationary points reduces to
[Equation]
where W is the decoder weight matrix.
Reformulating the above expression, we arrive at :
[Equation]
Noting that W T W is symmetric with a Singular Value Decomposition (SVD) of UΛU T ( U is orthogonal, Λ ii = ∥
[ W ] : k ∥ 2 ), and I d = UU T ; thus:
[Equation]
Therefore, (70) can be reformulated as the left KL-measure of diagonality [2] of the matrix
U [ Λ + 1 / γ 2 I d ] U T :
[Equation] [Equation]
which is by definition the local IMA contrast c IMA (cf.
[23, Appendix C.1]).
When γ → + ∞ , the above expression converges to the left KL-measure of diagonality for W T W , i.e., the local IMA contrast for the decoder.
γ → + ∞ thus means that the ELBO converges to the IMA regularized log-likelihood L IMA with λ = 1 :
[Equation]
which concludes the proof.
Prop. 9, especially (72), gives us intuitive understanding on why and how γ influences how much the orthogonality of W is enforced.
Small γ (high observation noise) means that there is no reason to promote the orthogonality of the decoder, as the high noise level (i
e., low-quality fit of x ) will drive (72) towards diagonality via 1 / γ 2 . 2.
On the other hand, when γ → + ∞ , then the orthogonality of the decoder is promoted.
That is, the decoder precision γ 2 acts akin to a weighting factor influencing how strong the IMA principle should be enforced.
We can observe that the ELBO recovers the exact log-likelihood for column-orthogonal W :
Corollary 1 (For column-orthogonal W the ELBO equals the exact log-likelihood) .
When W is in the form W = OD , then diag (cid:0) W T W (cid:1)
=
W T W =
DO T OD
= D 2 , i.e. the ELBO corresponds to the exact log-likelihood since (72) is zero.
Corollary 1 also implies that γ does not affect the gap between ELBO and exact log-likelihood for column-orthogonal W .
F Experimental details F.1 The relationship of weight matrix structures and the IMA function class
During the experiments we have used different weight matrices either to ensure that the mixing is within or to exclude it from the IMA function class.
Here we summarize our choices also including the depth of the network as it can affect the mixing’s place w.r.t.
the IMA function class.
When we use orthogonal weight matrices (§ 4.1,§ 4.2), then a single-layer network is within the
IMA class, but adding more layers with elements-wise nonlinearities will move the MLP outside the function class.
When using triangular MLPs (§ 4.2), the network is also outside the IMA class (triangular matrices are orthogonal only when they are diagonal , see also [23, Lemma C.1]).
Notably, Möbius transforms [53] are conformal maps (thus, they are in the IMA class) irrespective of the structure of the weight matrix used (cf.
Appx.
F.4 for details).
F.2 Self-consistency in practical conditions (§ 4.1)
For the self-consistency experiments, the mixing is a 3-layer MLP with smooth Leaky ReLU nonlinearities [22] and orthogonal weight matrices—which intentionally does not belong to the IMA class, since our self-consistency result is not constrained to the IMA class.
The 60,000 source samples are drawn from a standard normal distribution and fed into a VAE composed of a 3-layer MLP encoder and decoder with a Gaussian prior.
We use 20 seeds for each γ 2 ∈ { 1e1; 1e2; 1e3; 1e4; 1e5 } .
Additional parameters are described in Tab. 1.
Training is continued until the ELBO ∗ improves on the validation set (we use early stopping [54]), then all metrics are reported for the maximum ELBO ∗ (Fig. 2).
[Table]
F.3 Relationship between ELBO ∗ , IMA-regularized, and unregularized log-likelihoods (§ 4.2)
[Table]
over different γ with an IMA-class mixing
For the experiments comparing the ELBO ∗ , IMAregularized, and unregularized log-likelihoods, data is generated by mixing points from a standard Gaussian prior using an invertible neural network.
When the mixing is not in the IMA-class (Tab. 2), we use a two-layer neural network with sigmoid nonlinearites and triangular weight matrices.
When the mixing is from the IMA-class (Tab. 3), we use a onelayer neural network with orthogonal weight matrices.
The data dimensionality in both cases is two.
Training is carried out using a VAE with a decoder fixed to the ground-truth and separate encoder models for the means and variances of the approximate posterior.
The encoder comprises two three-layer neural networks with ReLU non-linearities and a hidden layer size of 50 .
Due to training instabilities when using a large γ , we train the model by first fixing the mean encoder to the ground-truth inverse of the mixing for the first 30 epochs; thus, only training the variances.
We then train both for the remaining epochs.
Training is stopped after the ELBO ∗ plateaus on the validation set .
A training set of 100 , 000 samples is used, with a validation set and test set of 30 , 000 and 15 , 000 samples, respectively.
The learning rate is 1e − 4 and the batch size 64 .
We provide additional results when the mixing is from the IMA class (Tab. 3): as C IMA is zero, we expect that both L IMA and the unregularized log-likelihood match.
Indeed, this is what Fig.demonstrates.
[Table]
F.4 Connecting the IMA principle, γ 2 , and disentanglement (§
Synthetic data (Möbius transform)
We use 3-dimensional conformal mixings (i
[Table]
e., the Möbius transform [53]) from the IMA class with the functional form:
[Equation]
where t , b ∈ R d , W ∈ R d × d , α ∈ R , and ϵ = 2 (to ensure nonlinearity) with d = 3 .
Both ground-truth and prior distributions are uniform to avoid the singularity when z = b .
To determine whether a mixing from the IMA class is beneficial for disentanglement, we apply a volume-preserving linear map after the Möbius transform (using 100 seeds) to construct a mixing outside of the IMA class.
We fix γ 2 = 1e5 and report further parameters in Tab. 4.
Training is continued until the ELBO ∗ improves on the validation set (we use early stopping [54]), then all metrics are reported for the maximum ELBO ∗ (Fig. 3)
[Table] [Table]
Image data (Sprites)
We train a VAE (not β -VAE) with a factorized Gaussian posterior and Beta prior on a Sprites image dataset generated using the spriteworld renderer [66] with a Beta ground truth distribution.
Similar to [32], we use four latent factors, namely, x- and y-position, color and size , and omit factors that can be problematic, such as shape (as it is discrete) and rotation (due to symmetries)
Our choice is motivated by [26, 18] showing that the data-generating process presumably is in the IMA class.
The architecture both for encoder and decoder consists of four convolutional and three linear layers with ReLU nonlinearities (Tab. 5).
Training is continued until the ELBO ∗ improves on the validation set (we use early stopping [54]), then all metrics are reported for the maximum ELBO ∗ .
[Table]
F.5 Optimality of γ 2 w.r.t. its MLE
During our experiments, we do not optimize γ 2 , as it is generally the case in the literature [57, 44, 38].
However, as noted by Rybkin et al.
[59], doing so could lead to superior sample quality.
The price we need to pay for improved sample generation is a more difficult optimization task (also noted in [60]): including γ 2 as a trainable parameter might require a careful learning rate tuning, and smaller learning rates can yield suboptimal likelihood values in the beginning [59].
During our experiments, we confirmed that making γ 2 learnable (all else being equal) yields suboptimal results, particularly in terms of MCC.
Thus, we opted for comparing our hyperparameter setting to the maximum likelihood estimate of the decoder variance , as proposed in [59, Eq.
Accomodating the parameter γ 2 instead of the decoder variance, we reformulate the equation as:
[Equation]
i.e., the MLE is the mean squared error between observations x and the decoded mean encodings f θ (cid:0) µ ϕ ( x ) (cid:1) , where |X| denotes the number of observations.
Interestingly, this is the inverse of the quantity we report on the right plot of Fig. 2.
To compare γ 2MLE (calculated as eq.
and the optimal value of γ 2 we found via grid search from the values { 1e1; 1e2; 1e3; 1e4; 1e5 } , we plot the log of both values in Fig.
We can observe that for all values except 1e5 , γ 2MLE is larger, sometimes with more than one order of magnitude.
For 1e5 , the mean (for the 20 seeds) lie in the range [0 . 8e5; 3 . 8e5] with a mean and standard deviation of 2 . 3 ± 0 .
77e5 , indicating that γ 2 = 1e5 and γ 2 MLE are in the same order of magnitude, corroborating that we used the optimal setting up to the granularity of our original grid search.
G Computational resources
The self-consistency (§ 4.1), the likelihood comparison (§ 4.2), and the synthetic experiments with the Möbius transform (§ 4.3, particularly Fig. 3) were ran on a MacBook Pro with a Quad-Core Intel
Core i5 CPU and required approximately nine days.
The Sprites experiments (§ 4.3, particularly
Fig. 5) required approximately four and a half days on an Nvidia RTX 2080 GPU.
H Societal impact
Our paper presents basic research and is mainly theoretical, though the lack of direct connection to a specific application does not mean that our results could not be used for malevolent purposes.
We acknowledge that providing a possible mechanism for why unsupervised VAEs can learn disentangled representations can inform specific actors that unsupervised VAEs might be used to extract the true generating factors.
Since no auxiliary variables, labels, or conditional distributions are required, this might lead to a broader use of unsupervised VAEs for trying to learn the true generating factors— including applications with potentially negative societal impact such as extracting features from images, video, or text for personal identification; thus, possibly violating the desire of those who intend to remain anonymous.
I Notation Acronyms
ELBO evidence lower bound
IMA Independent Mechanism Analysis i.i.d.
independent and identically distributed
ICA Independent Component Analysis
KL Kullback-Leibler Divergence
LVM Latent Variable Model
MCC Mean Correlation Coefficient
MLE Maximum Likelihood Estimation
MLP Multi-Layer Perceptron
MSE Mean Squared Error
PCA Principal Component Analysis
PPCA Probabilistic Principal Component Analysis
SVD Singular Value Decomposition
VAE Variational Autoencoder
NomenclatureIndependentMechanism Analysis
C IMA global IMA contrast α scalar field
D general diagonal matrix
O orthogonal matrix y reconstructed sources
L IMA IMA loss function c IMA local IMA contrast
Variational Autoencoder
V weight matrix of a linear encoder
W weight matrix of a linear decoder µ (cid:98) ϕ
( x ) optimal mean of q ϕ ( z | x ) µ ϕ ( x ) mean of q ϕ ( z | x ) ϕ parameters of the variational posterior q ϕ
( z | x ) σ (cid:98) ϕ
( x ) 2 optimal variance of q ϕ ( z | x ) θ parameters of the decoder p θ ( x | z ) γ square root of the precision of the VAE decoder Σ ϕz | x covariance matrix of q ϕ ( z | x )
L β β -VAE loss function f θ decoder g θ inverse decoder (cid:98) ϕ optimal parameters of the variational posterior q ϕ ( z | x ) p ( x ) data distribution p
( z ) latent prior distribution p θ
( z | x ) true posterior distribution of the decoded samples of the VAE, mapping x
(cid:55)→ z , parametrized by θ p θ ( x ) marginal likelihood p θ ( x | z ) conditional distribution of the decoded samples of the VAE, mapping z (cid:55)→ x , parametrized by θ q ϕ
( z | x ) variational posterior of the VAE, mapping x (cid:55)→ z parametrized by ϕ q (cid:98) ϕ
( z | x ) optimal variational posterior of the VAE, mapping x
(cid:55)→ z parametrized by ϕ µ (cid:98)
ϕ k ( x ) optimal mean of q ϕ ( z | x ) in dimension k µ ϕ k
( x ) mean of q ϕ ( z | x ) in dimension k σ (cid:98)
ϕ
k ( x ) 2 optimal variance of q ϕ ( z | x ) in dimension k σ ϕ k ( x ) 2 variance of q ϕ ( z | x ) in dimension k g θ inverse decoder component
H Hessian matrix
I d d -dimensional identity matrix
J Jacobian matrix Σ covariance matrix
x observation vector z latent vector X observation space d dimensionality of the observation space X
z latent single component