Learning Manifold Dimensions with Conditional Variational Autoencoders
Abstract
Although the variational autoencoder (VAE) and its conditional extension (CVAE) are capable of state-of-the-art results across multiple domains, their precise behavior is still not fully understood, particularly in the context of data (like images) that lie on or near a low-dimensional manifold.
For example, while prior work has suggested that the globally optimal VAE solution can learn the correct manifold dimension, a necessary (but not sufﬁcient) condition for producing samples from the true data distribution, this has never been rigorously proven.
Moreover, it remains unclear how such considerations would change when various types of conditioning variables are introduced, or when the data support is extended to a union of manifolds (e.g., as is likely the case for MNIST digits and related).
In this work, we address these points by ﬁrst proving that VAE global minima are indeed capable of recovering the correct manifold dimension.
We then extend this result to more general CVAEs, demonstrating practical scenarios whereby the conditioning variables allow the model to adaptively learn manifolds of varying dimension across samples.
Our analyses, which have practical implications for various CVAE design choices, are also supported by numerical results on both synthetic and real-world datasets.
Introduction
Variational autoencoders (VAE) [6, 14] and conditional variants (CVAE) [17] are powerful generative models that produce competitive results in various domains such as image synthesis [21, 5, 13], natural language processing [16], time-series forecasting [9, 19], and trajectory prediction [8].
As a representative example, when equipped with an appropriate deep architecture, VAE models have recently achieved state-of-the-art performance generating large-scale images [11].
And yet despite this success, there remain VAE/CVAE behaviors in certain regimes of interest where we lack a precise understanding or a supporting theoretical foundation.
In particular, when the data lie on or near a low-dimensional manifold, as occurs with real-world images [12], it is meaningful to have a model that learns the manifold dimension correctly.
The latter can provide insight into core properties of the data and be viewed as a necessary, albeit not sufﬁcient, condition for producing samples from the true distribution.
Although it has been suggested in prior work [4, 3] that a VAE model can learn the correct manifold dimension when globally optimized, this has only been formally established under the assumption that the decoder is linear or afﬁne [2].
And the potential ability to learn the correct manifold dimension becomes even more nuanced when a conditioning variable is introduced.
In this regard, a set of discrete conditions (e.g.,
MNIST image digit labels) may correspond with different “slices” through the data space, with each inducing a manifold with varying dimension (intuitively, the manifold dimension of images labelled “1” is likely smaller than those of “5”).
Alternatively, it is possible to have data expand fully in the ambient space but lie on a low-dimensional manifold when continuous conditional variables are present.
Such a situation can be trivially constructed by simply treating some data dimensions, or transformations thereof, as the conditioning variable.
In both scenarios, the role of CVAE models remains under-explored.
Moreover, unresolved CVAE properties in the face of low-dimensional data structure extend to practical design decisions as well.
For example, there has been ongoing investigation into the choice between a ﬁxed VAE decoder variance and a learnable one [4, 10, 15, 18, 3], an issue of heightened signiﬁcance when conditioning variables are involved.
And there exists similar ambiguity regarding the commonly-adopted strategy of sharing weights between the prior and encoder/posterior in CVAEs [7, 17].
Although perhaps not obvious at ﬁrst glance, in both cases these considerations are inextricably linked to the capability of learning data manifold dimensions.
Against this backdrop our paper makes the following contributions:
(i)
In Section 2.1 we provide the ﬁrst demonstration of general conditions under which VAE global minimizers provably learn the correct data manifold dimension.
(ii)
We then extend the above result in Section 2.2 to address certain classes of CVAE models with either continuous or discrete conditioning variables, the latter being associated with data lying on a union of manifolds.
(iii) Later, Section 3 investigates common CVAE model designs and training practices, including the impact of strategies for handling γ and the impact of weight sharing between condition prior and posterior networks.
(iv) Section 4 supports our theoretical conclusions and analysis with numerical experiments on both synthetic and real-world datasets.
Learning the Dimension of Data Manifolds
We begin with the deﬁnition of data:
Deﬁnition 1 (Data lying on a Manifold) Suppose r and d are two positive integers with r < d .
Then X is a simple r -Riemannian manifold embedded in R d when there exists a diffeomorphism ϕ between X and R r .
Speciﬁcally, for every x ∈ X , there exists a u = ϕ ( x ) ∈ R r , where ϕ is invertible and both ϕ and ϕ − 1 are differentiable.
Given inﬁnite observable variables x ∈ X , where X is a r -dimensional manifold embedded in R d whose deﬁnition is in 1, and latent variable z ∈ Z κ ⊆ R κ which serves as a low-dimensional representation, a VAE model is built to approximate the ground truth measure with a parametric density p θ ( x ) = (cid:82) p θ ( x | z ) p ( z ) dz , where prior p ( z ) =
N (0 , I ) .
Denote the ground-truth measure on
X as ω gt .
The probability mass of an inﬁnitesimal dx on the manifold is ω gt ( dx ) and (cid:82) X ω gt ( dx )
= 1 .
The canonical cost of VAE model is a bound on the average negative log-likelihood
[Equation]
Before starting our main theorems, we ﬁrst convert the similar idea from [4] and formalize a κ -simple
VAE model with explicit Gaussian and Lipschitz assumptions, as well as the way of parameterizations.
Deﬁnition 2 ( κ -simple VAE)
A κ -simple VAE is deﬁned as a VAE model with dim[ z ] = κ latent dimensions, the Gaussian encoder q φ ( z | x ) =
N ( z | µ z ( x ; φ ) , diag { σ 2 z ( x ; φ ) } ) , the Gaussian decoder p θ ( x | z ) =
N ( x | µ x ( z ; θ ) , γI ) , and the prior p ( z ) =
N ( z | 0 , I ) .
Here γ > 0 is a trainable scalar in θ .
The mean functions of the encoder and the decoder, i.e. µ z ( x ; φ ) and µ x ( z ; θ ) are arbitrary
L -Lipschitz continuous functions.
2.1 Learning Manifold Dimension in VAE
We start by analyzing the dimension of manifold learning in VAE models.
As it is pointed out in [4], if γ of a VAE model is trainable and not constrained, γ must be arbitrarily small but explicitly nonzero to minimize the cost and reconstruct data from X .
Here we deﬁne active latent dimensions as the dimensions that have a decaying encoder variance related to γ .
Deﬁnition 3 (Active Latent Dimensions in VAE)
In a κ -simple VAE, let { θ ∗ γ , φ ∗ γ } denote any global optimal κ -simple VAE solutions as a function of any ﬁxed γ , then for any j = 1 , . . .
, κ , a dimension j of latent variable z is deﬁned as an active dimension if the optimal encoder variance σ z ( x ;
φ ∗ γ ) 2 j → 0 as γ → 0 .
Theorem 1 (Learning the Data Manifold Dimension Using VAEs) Suppose training data lies on a r -dimensional manifold embedded in R d .
Then for any κ ≥ r , when a κ -simple VAE model achieves its any global optimums, we have
(i) L ( θ ∗ γ , φ ∗ γ ) =
( d − r ) log γ + O (1) , and (ii)
The number of active latent dimensions almost surely equals r , and (iii)
The optimal variance of each active dimension satisﬁes σ z ( x ; φ ∗ γ ) 2 j = O ( γ ) .
Theorem 4 aims to state that when the decoder is an L-Lipschitz continuous function and learns the ground truth diffeomorphism, the number of active dimensions determines intrinsic dimensionality of the generative model in data space.
A rigorous proof of Theorem 4 is provided in the appendices.
As a summary, our proof follows this path: ﬁrst, we prove the existence of active dimensions and the rate by contradiction.
The conclusion is that there must exist at least r active dimensions whose encoder variances going to zero at a rate of O ( γ ) for reconstruction tasks, otherwise the reconstruction term will grow at a rate of O ( 1 γ ) as γ goes to zero.
Next, we get an upper bound and a lower bound of ELBO and show that their difference is a constant, both in the form of ( d − r ) log γ + O (1) .
Last, we get the exact number of active dimensions by showing that adding unnecessary active dimensions decreases the coefﬁcient of log γ , which may increase loss value.
This result implies that we can explicitly obtain the manifold dimension and the state of convergence by viewing the performance of σ 2 z and γ .
Also, both in theory and experiments, the ratio of the norm term in reconstruction and γ is a constant, i.e. the data dimensions d .
Speciﬁcally, when a VAE model converges to its optimum, γ will go to zero, as well as r dimensions of σ 2 z , which comes from the optimal γ ∗ =
arg min γ ˜ L ( θ, φ )
=
L 2 d E ε ∼ N (0 ,I ) [ || σ z ( x ) 1: r ε || 2 ] .
To achieve the optimal solution, the active latent dimensions play a crucial role in reconstruction and the inactive ones help to push KL regularization factor to its minimum along these dimensions.
By deﬁnition, inactive dimensions are with a variance such that lim γ → 0 σ z ( x ; φ ∗ γ ) =
C > 0 , where C is some constant.
When such latent dimensions are greater than d − r , no matter how many inactive dimensions there are, the model cannot perfectly reconstruct even a single manifold dimension.
Note that previous work [4] has demonstrated that global minima of VAE models can achieve zero reconstruction error for all samples lying on the data manifold.
However, it was not previously demonstrated in a general setting that this perfect reconstruction was possible using a minimal number of active latent dimensions, and hence, it is conceivable for generated samples involving a larger number of active dimensions to stray from this manifold.
In contrast, to achieve perfect reconstruction using the minimal number of active latent dimensions, as we will demonstrate under the stated assumptions, implies that generated samples must also lie on the manifold.
The noisy signals from inactive dimensions are blocked by the decoder and therefore cannot produce deviations from the manifold.
2.2 Learning Manifold Dimension in Conditional VAE
In this section, we extend the analysis on κ -simple VAE models to κ -simple CVAE models.
We ﬁrst the manifold with condition.
Speciﬁcally, suppose X is a Riemannian manifold and there exists a set
C where for each x ∈ X , there is a corresponding c ∈ C , and we call that c is the condition of x .
Deﬁnition 4 ( κ -simple CVAE)
For any x ∈
X and its corresponding c ∈
C , a κsimple CVAE is an extension of κ -simple VAE with the Gaussian encoder q φ ( z | x, c ) =
N ( z | µ z ( x, c ; φ ) , diag { σ 2 z ( x, c ; φ ) } ) , the Gaussian decoder p θ ( x | z, c ) =
N ( x | µ x ( z, c ; θ ) , γI ) , and the prior p θ ( z | c ) =
N ( z | µ z ( c ; θ ) , diag { σ 2 z ( c ; θ ) } ) .
µ z ( x, c ; φ ) and µ x ( z, c ; θ ) are arbitrarily LLipschitz continuous, while µ z ( c ; θ ) can be arbitrary function.
The canonical CVAE cost with conditional variable being c is
[Equation]
When prior is independent of c , i.e. p ( z | c )
= p ( z ) , the cost still holds.
In a CVAE model, besides samples x ∈ X , the conditioning variable c may also provide information relevant to x .
Thus we deﬁne the effective dimension to measure the contribution of the conditioning variable.
Deﬁnition 5 (Effective Dimension of the Conditioning Variable)
For any x ∈ X and its corresponding c ∈ C , we call integer t the number of effective dimensions of c , if 1) there exists a function g : C
→ R t and exists t dimensions of ϕ ( x ) , denoted as ϕ ( x ) t ,
such that g ( c ) =
ϕ ( x ) t , where ϕ is the diffeomorphism in Deﬁnition 1, and 2) there doesn’t exist such a function g for t
+ 1 .
Further, let
C t ⊂ C be the subset for all c with effective dimensions of t .
Deﬁnition 5 shows that any c ∈ C t can reconstruct at most t ≤ r dimensions of the manifold.
Also, we extend our deﬁnition of active latent dimensions to CVAE.
Deﬁnition 6 (Active Latent Dimensions in CVAE)
In a κ -simple CVAE, denote diag { σ z p ( c ; θ ) } and diag { σ z q ( x, c ; φ ) } as the variance of the prior and encoder.
For any j = 1 , . . .
, κ , a dimension j of latent variable z is deﬁned as an active dimension if the ratio σ z q ( x, c ; φ ∗ γ ) 2 j /σ z p ( c ; θ ∗ γ ) 2 j → 0 as γ → 0 .
As special cases, for VAE or CVAE with a standard Gaussian prior we have σ z p ( x ) 2 j = 1 , and that indicates Deﬁnition 3 is a special case of Deﬁnition 6.
Next, we extend Theorem 4 to CVAE models, which describes how the manifold dimension relates to conditioning variables.
Theorem 2 (Learning Manifold Dimension in CVAE)
Suppose X is a r -dimensional manifold embedded in R d with its corresponding condition c ∈ C t .
Then for any κ ≥ r , when a κ -simple
CVAE model achieves its global optimum, we have
(i) L c ( θ ∗ γ , φ ∗ γ )
=
( d − r + t ) log γ + O (1) , and (ii)
The number of active latent dimensions almost surely equals r − t , and (iii) The variance of each active dimension satisﬁes σ z q ( x, c ; φ ∗ γ )
j /σ z p ( c ; θ ∗ γ )
j = O ( γ ) .
This theorem illustrates the signiﬁcance of conditioning variable that it can push the cost to a smaller value than a VAE model without condition, which is favorable.
It also points out that the condition will reduce the number of active latent dimensions without disrupting the data ﬁt.
It worth mentioning that even though the data is not on the manifold initially, once a conditioning variable with a positive effective dimension is incorporated, the conditioned data then become on a manifold, which can be induced directly in our theorem.
Pushing further, we have the following result describing CVAE’s capacity in learning data lies on a union of manifolds.
Corollary 2.1 (Adaptive Active Dimension) Suppose there exist x 1 , x 2 ∈ X with r -dimensional manifold and their corresponding conditioning variables c 1 ∈ C t 1 and c 2 ∈ C t 2 , i.e. c 1 , c 2 have t 1 and t 2 effective dimensions respectively.
Given a CVAE model with an optimal solution { θ ∗ , φ ∗ } , we have the number of active dimensions of z
This result indicates that CVAE can adaptively learn different data manifolds under conditioning variables with different effective dimensions.
When c is discrete, it is equivalent to training separate
VAEs where each model corresponds to a discrete condition value.
On the other hand, collecting all possible conditions in a single CVAE model enables it to compress the dimension dynamically.
For example, c describes the set of manifolds that share the same dimension, thus for samples in each set, the corresponding manifold dimensions are the same.
When c is continuous, conditioning variables may contain intact information of certain manifold dimensions, which substitute for corresponding active dimensions.
In Section 4, we demonstrate numerical evidence of this behavior.
On Common Model Design Choices
In this section, we review some model designs and training practices that are commonly seen in prior works but could potentially negatively impact the model convergence and performance, especially within the present context of learning data manifold dimensionality.
3.1 On the Equivalence of Conditioned and Unconditioned Prior
In prior works, it is popular to design a parameterized prior p θ ( z | c ) 2 in CVAE [24, 7, 8, 20, 1], which is a trainable module alongside the encoder q φ ( z | x, c ) and the decoder p θ ( x | z, c ) .
Remark 1 (Remove conditioning variable from the Prior) Consider a κ -simple CVAE model with prior p θ ( z | c ) , encoder q φ ( z | x, c ) and decoder p θ ( x | z, c ) .
We can always ﬁnd another κsimple CVAE model with prior p ( z ) ∼ N (0 , I ) , encoder q φ (cid:48)
( z | x, c ) and decoder p θ (cid:48) ( x | z, c ) , such that L c ( θ, φ ) =
L c ( θ (cid:48) , φ (cid:48) ) .
Remark 1 indicates that a conditioned prior is not necessary.
Speciﬁcally, as shown in its proof, we can always explicitly convert an existing κ -simple CVAE model with prior p θ ( z | c ) into another κ -simple
CVAE model with prior being N (0 , I ) .
In practice, when we have the ﬂexibility in architecture design, it is equivalent to use a standard Gaussian prior or a parameterized one.
3.2 The Initial γ Impacts Model Convergence and Dimension Learning
Recall in Theorem 4, we have L ( θ ∗ γ , φ ∗ γ ) =
( d − r ) log γ + O (1) , and that implies when L ( θ ∗ γ , φ ∗ γ ) approaches to −∞ , γ is expected to be closer to 0.
This behavior makes γ important in CVAE training.
A direct conclusion is that the model would be compromised if γ is set to be a ﬁxed positive constant, as analysed and empirically studied in literature [4, 10, 15, 18, 3].
Although a model converging towards global optimum would push γ to 0 regardless of its initial value, in practice, we observe that this initial value of γ still signiﬁcantly impacts the model performance.
In the cost function, we have
[Equation]
By this formulation we see that γ acts as the weight between data likelihood and KL-divergence.
A large γ downgrades the weight for data likelihood, and encourages the model to optimize the
KL-divergence term ﬁrst, and vice versa.
Making γ a learnable parameter is equivalent to adaptively balancing the cost.
When the initial value is set to be on an extreme end, the model is at risk of falling into a local minimum, as empirically in Section 4.4.
3.3 Weight Sharing Keeps Sequence Model from Global Optimal Convergence
The cost of VAE balances the three modules the encoder, decoder and prior with the two terms, the data likelihood and KL-divergence.
It has been argued in literature [3] that such a cost might lead to unstable training dynamics.
Therefore, one widely adapted trick[17] in practice is to share model weights between prior and posterior.
Weight sharing is conducted in various forms.
For data from a general domain where x and c are differently distributed, the weight sharing couples with the design in which the posterior takes in x and c as two input and process differently at the beginning, and the prior processes c as its input with the same module and weight in the posterior.
In this section we take the sequential data as a representation for the weight sharing issue because it takes a more thorough approach of weight sharing.
Sequential data poses the task of using early parts of a sequence to predict the later parts.
Formulating in CVAE, we have c being the early parts, and that conditions the to-predict later parts x .
Assuming x is a sequential data, i.e. { x l } with l > 0 denoted as the index, and each x l is paired with its condition c = x <l .
As a result, the encoder becomes q φ ( z l | x ≤ l ) , and the prior becomes p ( z l | x <l ) for any l greater than the observed sequence length.
It has been proposed in literature [7, 8] to share weights between the encoder and prior, i.e. p ( z l | x <l ) =
q φ ( z l | x <l ) .
Theorem 3 (Weight Sharing Compromises Performance of Sequential Modeling)
By sharing weights between the encoder and prior, the conditional VAE model has its cost being
[Equation]
Then we have L c ( θ, φ ) = Ω(1) for any θ and φ .
Theorem 6 implies that a fully weight-sharing pair of prior and encoder prevents the model from achieving its global optimum.
As the numerical study in Section 4.5, this weight sharing strategy could severely constraint the performance.
Experiments
In this section we present numerical results to support our analysis.
We ﬁrst verify model behavior in the synthetic environment for a controllable set of experiments, and extend to real-world datasets to further validate our conclusions.
4.1 Dataset and Metrics
Synthetic Dataset
The basic data samples are generated from a mixture of a ﬁnite number of
Gaussian distributions.
Speciﬁcally, we ﬁrst generate a 5-dimensional categorical selecting vector from uniform distribution, and normalize it with norm equaling one.
Then we generate two rdimensional parameter vectors µ u and log σ u from a standard Gaussian distribution in R r .
By repeating the second step 5 times, we get 5 independent Gaussian distributions.
Our synthetic data u ∈ R r is generated by selecting one of the 5 candidate distributions with a probability of the selection vector, and next generate samples from the selected distribution.
To get the manifold X in R d , we design a function h :
R r
→ R d , and the data sample x = h ( u ) .
In our experiments, h ( u ) =
Sigmoid ( G x u ) , where G x ∈ R d × r is randomly initialized controlled by a certain random seed.
The conditioning variable c is transformed by another transform h (cid:48)
( u 1: t ) =
G (cid:48)
u 1: t , where
G (cid:48)
∈ R t × t .
Given a ﬁxed seed, the generation results is controlled by parameters r , d and t .
For all the experiments, the training size of synthetic data is 100,000.
Real-world dataset Besides synthetic dataset, we also investigate the model behavior on
MNIST[23] and Fashion MNIST [22], two image datasets 4 .
These two datasets contain x with complex mappings between the manifold and ambient space.
At the same time, they are not too complex thus the training are made reasonably easy with common neural architecture, otherwise our focus could be driven to speciﬁc architecture choices other than VAE and CVAE model behavior.
By carefully tuning the pair ( x, c ) , we can control how c impacts the model.
Metrics The main metric that we use to evaluate VAEs and CVAEs is the negative Evidence
Lower BOund (ELBO), as in Eq 1 and Eq 2, and note that it is reported on convention and not for comparison.
There are also auxiliary metrics that we use to compare and diagnose models, including
Active Dimensions (AD), Reconstruction error E || x − µ x ( z ) || 2 (Recon), the KL-divergence ( KL ), and the data variance γ .
[Table]
[Table]
4.2 Manifold Dimensions Learning in VAE
Here
[Table]
we provide direct numerical support to Theorem 4.
Speciﬁcally
[Table]
we show that the model captures the correct dimensions of manifold regardless of the ambient data dimensions.
To indicate how the active dimension is determined, we provide an example in Section B of the appendix.
Although in numerical experiments a variance cannot be exact zero, we can still observe from a converged model that each variance of encoder is either close to 1, or close to zero.
With d = κ = 20 and varying r
[Table]
we demonstrate that the VAE model can learn the active latent dimensions when γ converges to a small value, regardless of the value of r .
The results are showed in
Table 1 and all the numbers are values at convergence.
Experimental results on MNIST and Fashion MNIST are reported in Table 2.
Without knowing the true manifold dimensions, a small latent dimension discourages the model from catching up the information for every dimension, while once given enough latent dimensions, the number of active dimensions equals to data’s intrinsic dimension and the rest is for the regularization.
The two parts of latent dimensions cooperate to push the VAE model to its global optimum.
[Table] [Table]
4.3 CVAE Adaptive Active Dimensions Varied With c
In this section
[Table]
we design experiments to show that a CVAE model can learn the dimension of manifold with respect to the effective latent dimensions of the condition.
Further, it can also learn the adaptive active dimension on a union of manifolds
[Table]
By Theorem 5 and Remark
[Table]
a CVAE model with an unconditioned prior and condition c ∈ C t is equivalent to a general one in design, whose active dimensions equals r − t .
In this experiments, we investigate the behavior of a CVAE model when data lies on a single manifold and its relation to condition is static.
Speciﬁcally, we set c to be the ﬁrst t dimensions of u , i.e. take G (cid:48) as I t , and let d = κ = 20 , r = 10 .
Table 3 shows the results.
On MNIST and Fashion MNIST, we train CVAEs with the class label of each image as the conditioning variable, and the results with κ = 32 are in Table 4.
By comparing it to Table 2, we notice that the
CVAE model for MNIST has a lower active dimension and better ELBO, while for FashionMNIST the CVAE performs similarly.
This implies that when data points in MNIST are conditioning on the label, they lie on a lower-dimensional manifold, while for data points in FashionMNIST where among classes the manifold dimensions are similar, the class label doesn’t reduce the manifold dimension.
This characterization of the dataset complexity between classes is consistent with their intuitive visual complexity.
We also provide an example of the encoder variance on the MNIST dataset in Section B of the appendix.
Discrete Condition To verify Corollary 2.1, we train a CVAE model on d -dimensional data that lie on a union of manifolds with different dimensions, with the conditioning variable c being an indicator of the source manifold.
Table 5 shows that the model successfully captures the active latent dimensions with an attention layer in the decoder, while without the attention layer the model learns a static active dimension of 4.
The result indicates two aspects of CVAE: 1) CVAE models have capacity to capture a union of manifolds simultaneously and converge on each manifold (indicated by -ELBO); 2) In practice, we need careful model architecture design to help the model to distinguish manifolds.
In this experiment d = 20 and κ = 40 .
Continuous Condition When c is a continuous variable, we show that the model can also adaptively learn the active dimensions.
In our setting, a continuous condition c ∈ C t contains t -dimensional information of samples, thus under different conditions the data may lie on different manifolds.
We set r = 12 , d = 20 and κ = 20 , then we set the ﬁrst 2 , 4 , 6 , 8 , 10 dimensions of u as the conditions.
Since the model needs dimension of c to be consistent, we padding them to R 10 with 0.
With the attention layer, the model can use the condition to collapse certain dimensions instead of activating all the possible dimensions and increasing unnecessary loss in KL term.
The results are included in
Table 6.
[Table] [Table]
4.4 Initialization of γ Impacts Model Training
Here we check how the initialization of γ would impact model convergence, as mentioned in
Section 3.2.
Our results are in Table 7.
With different initial log γ values, we observe the convergence of VAE is affected signiﬁcantly, as a result, the model can’t capture the correct manifold dimension.
In comparison
[Table]
CVAE models are less sensitive, as CVAEs have more parameters and the conditioned data lie on a lower-dimension manifold thus it actually poses an easier task.
Here the CVAE model with an unparameterized prior p ( z ) shows that in practice, given enough model capacity and good training parameters, one can directly train an unparameterized CVAE and achieve similar performance, instead of starting from training a parameterized one.
Still, when log γ = 20 , they converge to local optima.
From Table 7, we can conclude that in practice the performance at convergence heavily depends on both model architecture design and hyperparameters.
Results with less careful architecture or hyper-parameters may not reﬂect the true global optimal.
In this experiment, d = 20 , κ = 20 , r = 10 .
In CVAE, t = 5 .
4.5 Shared Weights Between Encoder and Prior
The experiments in this section is to support Theorem 6, i.e. sharing weights between the encoder and prior is not a satisfactory way for training in sequential data, we generate a sequence by
AutoRegressive-Moving-Average (ARMA) model with ﬁxed parameters, then use rolling windows to make our samples.
Speciﬁcally, the window size is 5, i.e. each sample we use the past 5 points as our condition, and reconstruct the sixth one which is viewed as current sample.
The results in Tableshow that when weight shared, the norm term of reconstruction cannot go down, which may result in a restriction in the capacity of the model, and this observation is coherent to the key contradiction used in our proof of Theorem 6.
[Table] [Table]
Conclusion
In this paper we provide insights on the behavior of VAEs and CVAEs for data lying on lowdimensional manifolds.
In the theoretic analysis we show that VAE has the ability to learn the manifold dimension.
More interestingly, we extend the conclusion to Conditional VAE that works with conditioned data.
This additional conditioning term impacts the relation between data and manifold, as a result it also affects the model design choices in practice.
With the theoretic results, we further examine some design choices.
A dedicated section of experiments supports our statements and conclusions numerically on both synthetic and real-world data.
Finally, we believe this work can in general help to improve the design and diagnose techniques of VAEs and CVAEs.
Limitations Our work may still have limitations in empirical veriﬁcation for real-world datasets at a larger scale or from other modalities than images.
Although our contributions are theoretically grounded in practice it still takes non-trivial efforts to design model architectures that both align with our guideline and ﬁt the data.
A Details on model architecture and computation resources
Synthetic Dataset For the experiments with VAE models, both the encoder and decoder are deﬁned as a Multi Layer Perceptron (MLP) with a single hidden layer.
For all the experiments with CVAE models except Sec 4.5, the encoder ﬁrst processes conditioning variable c via a MLP, then concatenate the output and sample x as the input of another MLP, and the decoder processes c in the same way then use a MLP to decode the latent variable.
In Sec 4.5, both the encoder and decoder are LSTM with one hidden layer.
Real Dataset Both of the encoder and decoder use two ResNet Blocks to process MNIST/ Fashion
MNIST images.
The encoder block is made up of two ConvNet in residual function and a 1 ∗
ConvNet for the shortcut to reshape.
The decoder block uses one ConvNet in the residual function and inverse ConvNet for the shortcut.
Resources We conduct our experiments on an Amazon Web Services g4dn.12xlarge
EC2 instance, which provides 4 T4 GPUs.
We estimate that the time to run through all experiments in this paper once would cost 20 GPU-hours.
The research activity for this paper cost around 100 GPU-hours in total.
B Example of the encoder variance
To show the active dimensions visually, here we report the encoder variances both on synthetic data and MNIST dataset.
[Table]
[Table]
C Proof of TheoremRecap of Theorem
Theorem 4 (Learning the Data Manifold Dimension Using VAEs) Suppose training data lies on a r -dimensional manifold embedded in R d .
Then for any κ ≥ r , when a κ -simple VAE model achieves its any global optimums, we have
(i) L ( θ ∗ γ , φ ∗ γ ) =
( d − r ) log γ + O (1) , and (ii)
The number of active latent dimensions almost surely equals r , and
(iii)
The optimal variance of each active dimension satisﬁes σ z ( x ; φ ∗ γ ) 2 j = O ( γ ) .
Summary of the Proof In this section, we deﬁne three categories based on the number of active dimensions and the rate of their encoder variance.
Note that any possible VAE optimum has to fall into the following three categories: the number of active dimensions whose encoder variance σ 2 z ( x, φ ∗ γ )
=
O ( γ ) is either greater than r , equal to r or less than r .
The proof’s logic ﬂow is:
When the number of active dimensions whose encoder variance σ 2 z ( x ) =
O ( γ ) is less than r , the reconstruction error will increase at a rate of O ( 1 γ ) , thus the cost cannot reach the optimum.
This is proven in Sec C.1; 2.
When such dimensions’ number equals r , the cost is exactly ( d − r ) log γ + O (1) .
The corresponding proof is in Sec C.2; 3.
When such dimensions’ number is greater than r , denoted as m > r , the cost is ( d − m ) log γ + O (1) >
( d − r ) log γ + O (1) .
It is showed in Sec C.3.
C.1 The number of active dimensions whose encoder variance σ 2 z ( x ) = O ( γ ) is less than r
The main idea is to link the gap between a large σ z and large reconstruction error.
For a given z 0 , µ x ( z 0 ) will equal some x 0 such that || x 0 − µ x ( z 0 ) ||
2 = 0 .
But for other choices from X where x (cid:54) = x 0 , we have || x − µ x ( z 0 ) ||
2 > 0 leading to the positive expectation term (cid:82) z q ( z | x ) || x − µ x ( z ) ||
2 dz .
To minimize such positive error, we need to lower the density q ( z | x ) where x (cid:54) = x 0 , which is a function of σ z .
Suppose that the number of active dimensions whose encoder variance σ 2 z ( x ) =
O ( γ ) is less than r .
In this section, we will show that under this assumption the model can’t reach its global optimum, i.e.
L − ∞ .
Remind that the cost of VAE is
[Equation]
We have
[Equation]
Following the two facts:
Lebesgue measure on the real numbers is σ -ﬁnite. 2.
z ∈ R κ and x ∈ X , where X is a r -dimensional manifold embedded in R d .
and referring to Fubini’s theorem, we can switch the integration order of ω gt ( dx ) and dz .
Assume the components of z ∈ R κ is permutable.
For a r -dimensional manifold, we can always use the ﬁrst r dimensions of z to get ϕ ( x ) , i.e. once given r-dimensional information, there always exists a decoder, s
[Equation]
t. µ x ( z 1: r ) = µ x ( z ) .
Denote by µ z ( x ) 1: r and σ 2 z ( x ) 1: r the mean and covariance matrix of the ﬁrst r dimension of z .
After switching the integration order, we have
[Equation]
C.1.1 Analyze
the density with respect
to
σ
z ( x ) 1: r and z 1: r − µ z ( x )
r
Next, for the integral over X in the ﬁrst term in Eq 5, we examine a certain z 1: r ∈ Z r and view it as a constant.
Since µ x is a deterministic function, µ x ( z 1: r ) is also constant.
The log-density on z 1: r is
[Equation]
Take the derivative of σ 2 z , we have
[Equation]
When σ 2 z is smaller than ( z 1: r − µ z ( x ) 1: r )( z 1: r − µ z ( x ) 1: r ) T , the second term’s rate is larger.
Thus the density is monotonically increasing when σ 2 z ≺ ( z 1: r − µ z ( x ) 1: r )( z 1: r − µ z ( x ) 1: r ) T and monotonically decreasing
when σ 2 z (cid:31) ( z 1: r − µ z ( x ) 1: r )( z 1: r − µ z ( x ) 1: r ) T .
Note that µ x is LLipschitz continuous, so we have L | z 1: r − µ
[Equation]
x
[Equation]
−
[Equation]
The equality comes from the fact that we can choose optimal µ z and µ x , s.t.
µ x ( µ z ( x ) 1: r ) = x .
Now we can divide x ∈ X into four cases and we assume all the four disjoint cases exist when analyzing, otherwise the integration over corresponding domain is 0 which would not affect our result.
The four cases are as follows:
[Equation]
µ
[Equation]
X 2 ( z
1: r )
= { x : σ 2 z ( x ) 1: r ≺ ( z 1: r − µ z ( x ) 1: r )( z 1: r − µ z ( x ) 1: r ) T } ∩ { x : ||
z 1: r − µ z ( x ) 1: r || < + ∞ } 3.
[Equation] [Equation]
We have X 1 ( z 1: r ) ∪X 2 ( z 1: r ) = { x : σ 2 z ( x ) 1: r ≺ ( z 1: r − µ z ( x ) 1: r )( z 1: r − µ z ( x ) 1: r ) T } and X 3 ( z 1: r ) ∪ X 4 ( z 1: r ) = { x : σ 2 z ( x ) 1: r (cid:23) ( z 1: r − µ z ( x ) 1: r )( z 1: r − µ z ( x ) 1: r ) T } .
Thus X 1 ( z 1: r ) ∪ X 2 ( z 1: r ) ∪ X 3 ( z 1: r ) ∪ X 4 ( z 1: r ) cover the whole space of x related to z 1: r , i.e. X ( z 1: r ) .
[Equation]
i
[Equation]
+ ∞ }
Denote σ 2 l as the lower bound of σ 2 z ’s eigenvalues, which cannot approach 0 by our assumption.
σ z < + ∞ .
The integral over X 1 ( z 1: r )
[Equation]
will approach 0 as ||
z 1: r − µ z ( x ) 1: r || = + ∞ .
Thus (cid:82) z
[Equation]
r 0 dz =
[Equation]
Denote N = max x {|| z 1: r − µ z ( x ) 1: r || 2 } and X α 2 ( z 1: r ) = { x : || x − µ x ( z 1: r ) || 2 > α } ∩ X 2 ( z 1: r ) , where α 0 .
If for any α , X α 2 ( z 1: r ) = ∅ , we have for all x ∈ X 2 ( z 1: r ) , x
= µ x ( z 1: r ) .
However, if µ x ( x ) r ∈ X 2 ( z 1: r ) , i.e. µ x ( z 1: r ) satisﬁes σ 2 z ( µ x ( z 1: r ))
≺ ( z 1: r − µ z ( µ x ( z 1: r )))
( z 1: r − µ z ( µ x ( z 1: r ))) T .
We can ﬁnd a pair of µ x and µ z , e.g. identity mapping, s.t. µ z ( µ x ( z 1: r )) =
z 1: r and σ 2 z ( µ x ( z 1: r )) < 0 , which is impossible.
Thus µ x ( z 1: r ) / ∈ X 2 ( z 1: r ) and X 2 ( z 1: r ) = ∅ .
Thus, once
X 2 ( z 1: r ) (cid:54) = ∅ , there exists an α , s.t.
X
α 2 ( z 1: r ) (cid:54) = ∅ .
The integral over X 2 ( z 1: r )
[Equation]
N
The last inequality comes from the fact that (cid:36) ( X α 2 ( z 1: r )) ≥ 1 where (cid:36) is a counting measure and the non-negativity of the second term.
[Equation]
In this case the density is monotonically decreasing with σ z .
Since σ z (cid:54) =
+ ∞ , denote σ 2 u as the upper bound of the eigenvalues of σ 2 z .
It can also bound ||
z 1: r − µ z ( x ) 1: r
|| 2 .
Use the same strategy in ( ii ) , deﬁne X α (cid:48) 3 ( z 1: r )
= { x : || x − µ x ( z 1: r )
|| 2 > α (cid:48) }
∩ X 3 ( z 1: r ) .
If X 3 ( z 1: r ) (cid:54) = ∅ , we have (cid:90)
[Equation]
In this case the density is monotonically decreasing with σ z
[Equation]
and the dominant factor is
[Equation]
Since σ z is arbitrarily large
[Equation]
it is obvious that (cid:112) | σ 2 z | > tr ( σ 2 z ) ≥ ||
z 1: r − µ z ( x ) 1: r || 2 ≥ 1 L 2 || x − µ x
( z 1: r )
|| 2 .
Note that | · | = det ( · ) .
The integral over X 4 ( z 1: r )
[Equation]
will approach 0 as σ 2 z → ∞
[Equation]
C.1.
Analyze the existence of the above cases and
get a lower bound
We have X
=
X
1 ∪ X 2 ∪ X 3 ∪ X 4 , where X i = ∪ z 1: r X
i
z 1: r ) , i = 1 , 2 , 3 , 4 .
In X 1 ∪ X 4 , the integral is 0.
To get a lower bound of Eq 5, we need to prove X 2 ∪ X 3 (cid:54) = ∅ , i.e. there must exists z 1: r such that x ∈ { σ 2 z ( x ) 1: r < ∞} ∩ {|| z 1: r − µ z ( x ) 1: r || < ∞} exists.
For σ z ( x ) r , if σ z ( x ) r = ∞ , then in the KL term the trace tr ( σ 2 z ( x ) 1: r ) = + ∞ which cannot be offset by − log | σ 2 z ( x ) 1: r | .
Thus to minimize loss, σ z < ∞ .
For || z 1: r − µ z ( x ) 1: r || < ∞ , with L -Lipschitz continuity, for any z ∗ 1: r , we can ﬁnd a x ∗ ∈ X , s.t.
|| z ∗ 1: r − µ z ( x ∗ ) 1: r || = 0 .
Denote U δ ( x ) r as a neighborhood of x with the radius of δ .
For any x ∈ U δ ( x ∗ ) , we have || µ z ( x ) 1: r − z ∗ 1: r
||
=
|| µ z ( x ) 1: r − µ z ( x ∗ ) 1: r || ≤
L || x − x ∗ || ≤ Lδ
So U δ ( x ∗ ) ⊂
X 2 ∪ X 3 .
To get a positive lower bound, we need to prove there exists x (cid:48) and δ , s.t.
the image of µ z ( x (cid:48) ) for x ∈ U δ ( x (cid:48) ) is with positive measure.
If for any x ∈ U δ ( x 1 ) , µ z ( x ) 1: r = z 11: r , and for any x ∈ U δ ( x 2 ) , µ z ( x ) 1: r = z 21: r , which satisfy δ <
|| x 2 − x 1 || ≤ 32 δ , and z 11: r (cid:54) =
z 21: r .
Note that can always choose a larger δ to get such pair of { x 1 , x 2 } .
Then there exists x 3 ∈ U δ ( x 1 ) ∩ U δ ( x 2 ) , µ z ( x 3 ) should equals z 1 1: r and z 2 1: r simultaneously which is impossible.
Thus, there must exists x ∗ , s.t.
µ z ( U δ ( x ∗ )) has a positive measure.
With the existence of x ∗ , such that U δ ( x ∗ ) ⊂
X 2 ∪ X 3 and positive measured µ z ( U δ ( x ∗ )) , we have
[Equation]
Here denote C = (cid:82) z 1: r min { ασ rl e − 1 2 σ − 2 r l
[Equation]
} dz 1: r for simplicity.
C.1.3 Analyze the rate of the lower bound
The ﬁrst term Cγ grows at a rate of O( 1 γ ).
Because σ 2 z is at a lower rate than γ , we have
[Equation]
and the right part decreases at a rate of log 1 γ .
When γ → 0 , O ( 1 γ ) >
O (log 1 γ ) , which means the increase from reconstruction term cannot be offset by the decrease from the KL term.
Moreover, from the fact that O ( 1 γ ) >
O (log 1 γ ) , when γ is small enough, the loss is monotonically decreasing with γ .
Therefore, when γ → 0 , the lower bound cannot approach −∞ , which means at this case, the model can never achieve optimum.
Thus, there must exist some active dimensions whose variance σ 2 z ( x ) satisﬁes σ 2 z ( x ) =
O ( γ ) as γ → 0 to reach the global optimum.
We can learn from the expression of C that as long as the number of such active dimensions whose encoder variance σ 2 z ( x ) =
O ( γ ) exceeds r , as γ approaches zero, the reconstruction term is at most at the rate of O (1) .
Next, we will show that when there exist at least r such active dimensions, the VAE model’s optimum is achievable.
C.2 The number of
active dimensions whose encoder variance σ 2 z ( x ) =
O ( γ ) equals r
In this section, we get an upper bound and a lower bound and show that both case the cost is ( d − r ) log γ + O (1) .
C.2.1 An Upper Bound of ELBO
Get an upper bound by Lipschitz We can write z = µ z ( x ) + ε ∗ σ z ( x ) , where ε ∼ N (0 , I ) .
Since decoder mean function µ x ( z ; θ ) is L -Lipschitz continuous, we have:
[Equation]
where the ﬁrst equality comes from the fact that we can choose optimal encoder-decoder pairs such that µ x ( µ z ( x ) 1: r )
=
x .
Take it into L ,
[Equation]
γ
[Equation]
We get an upper bound of L , denoted as ˜ L .
Analysis of the Upper Bound ˜ L Now we only pay attention to the upper bound ˜ L and try to prove that it is at a rate of O (( d − r ) log γ ) .
We can get implicit optimal values of ˜ L : γ ∗ and σ ∗ ( x ) 21: r by taking the derivative of ˜ L separately.
We have optimal γ ∗
[Equation]
and
[Equation]
we have the optimal variance of ˜ L :
[Equation]
[Equation]
It shows that 1 √ γ σ ∗ z ( x ) 1: r = O (1) when it reaches the optimal value.
Take the optimal values into ˜ L , then we get ˜ L as a function of γ ∗ and σ ∗ z ( x ) 1: r :
[Equation]
Deﬁne { λ i } κi =1 as the eigenvalues of σ z ( x ) .
Denote ˜ r as the number of { λ i } κi =
r +
that will go toas γ ∗ → 0 .
We have
[Equation]
To minimize Eq 12, we want ˜ r to be as small as possible and at the best equals 0 which is achievable.
Since the rest κ − r − ˜ r = κ − r dimensions are irrelevant to γ
[Equation]
at least will not approach 0 when γ → 0 , we can view them as constants.
We have the loss equals
[Equation]
C.2.2 A Lower Bound of ELBO
From C.1 we have analyzed the loss performance when there are less than r active dimensions whose variance goes to zero at a rate no lower than γ .
In this part, we focus on the case that r latent dimensions are such active dimensions whose encoder variance goes to zero at a rate of O ( γ ) .
Without loss of generality, we assume the ﬁrst r latent dimensions satisfy σ 2 z ( x ) 1: r = O ( γ ) as γ → 0 .
We have
[Equation]
The inequalities come from the fact that the norm term is non-negative and the active dimensions’ rate is no less than γ .
For the last equality, we can use the strategy in Eq 12.
To minimize the lower bound, there should not be any active dimensions in these r + 1 : κ dimensions.
We get an upper bound and a lower bound at the same rate, i.e. log γ with r active latent dimensions.
Therefore, the original loss is also with r active dimensions.
We have the optimal cost for each x equals
[Equation]
So far, we have get the conclusion in Thm 4 about the form of ELBO when γ → 0 , as well as the number and rate of active dimensions.
Next, we show that the number of active dimensions can’t be greater than r .
C.3 When the number of active dimensions is greater than r
Denote now there are m active dimensions and m > r .
From Eq 11, in this case ˜ r
= m − r , and the loss is
[Equation]
since here lim γ → 0 λ i = 0 , − 2 (cid:80) r +˜ r i = r +1 log
λ
i is monotonically increases as ˜ r increases at the rate of Ω(log 1 γ ) .
For the reconstruction term, it is unaffected since we only use the ﬁrst r latent dimensions for reconstruction.
Therefore, the loss will increase at the rate of Ω(log 1 γ ) , which is larger than the loss when m = r .
In conclusion, only when the number of active dimensions equals r , and these active dimensions’ encoder variance σ 2 z ( x ) =
O ( γ ) as γ → 0 , the optimal cost is ( d − r ) log γ + O (1) .
D Proof of TheoremRecap of Theorem
Theorem 5 (Learning Manifold Dimension in CVAE)
Suppose X is a r -dimensional manifold embedded in R d with its corresponding condition c ∈ C t .
Then for any κ ≥ r , when a κ -simple
CVAE model achieves its global optimum, we have
(i) L c ( θ ∗ γ , φ ∗ γ )
=
( d − r + t ) log γ + O (1) , and (ii)
The number of active latent dimensions almost surely equals r − t , and (iii) The variance of each active dimension satisﬁes σ z q ( x, c ; φ ∗ γ )
j /σ z p ( c ; θ ∗ γ ) 2 j = O
( γ ) .
Summary of Proof In this section, we ﬁrst make the proof when p θ ( z | c ) =
p ( z ) , i.e. a parameter free prior, and then extend to the case when the prior involves the conditioning variable.
The logic ﬂow is as follows:
The prior is independent of c (a)
Following the same proof idea in Theorem 1, when the number of active dimensions whose encoder variance σ 2 z q ( x, c )
=
O ( γ ) is less than r − k , the reconstruction error will grow at a rate of O ( 1 γ ) .
This is proven in Sec D.1.1; (b) In Sec D.1.2 we show that both the upper bound and the lower bound are ( d − r + t ) log γ and the exact number of active dimensions is r − t 2.
The prior is a function of c (a) Since involving c in the prior will not affect the reconstruction term, we extend the conclusion in Sec D.1.1 to the general case; (b) Show that both the upper bound and the lower bound are ( d − r + t ) log γ and the exact number of active dimensions is r − t .
The proof is in Sec D.2.
Under CVAE setting, we ﬁrst make some denotations for proof.
Since the encoder, prior, and decoder share the same condition c , the model has ﬂexibility to use part of each c from the three networks.
Denote t as the number of effective dimensions of c , and k as the number of effective dimensions of c used in the decoder p θ ( x | z, c ) , i.e. there exists a pair of encoder and decoder, s.t. µ x ( c ) =
ϕ − 1 ( u 1: k ) , where 0 ≤ k ≤ t and is a learnable parameter.
The encoder and prior use the rest t − k effective dimensions, i.e. µ x ( µ z ( c )) =
ϕ − 1 ( u k +1: t ) , and this part of information will be included in the latent variable z .
D.1 When the prior is independent of c , i.e. p θ ( z | c )
= p ( z )
In this case, we can write the cost as:
[Equation]
D.1.
The number of active dimensions whose encoder variance σ 2 z q ( x, c ) =
O ( γ ) is less than r − k
Following the same proof idea in Theorem 1, assume there is no active dimension in σ z ( x ) .
For the reconstruction term, it is equivalent to reconstruct a ( r − k ) -dimensional manifold.
Thus we can ﬁnd an lower bound of the cost
[Equation]
where C
[Equation]
=
[Equation]
The ﬁrst term C (cid:48) γ grows at a rate of O( 1 γ ).
Because σ 2 z is at a lower rate than γ , we have
[Equation]
and the right part decreases at a rate of log 1 γ .
When γ → 0 , O ( 1 γ ) >
O (log 1 γ ) , which means the increase from reconstruction term cannot be offset by the decrease from the KL term.
Moreover, from the fact that O ( 1 γ ) >
O (log 1 γ ) , when γ is small enough, the loss is monotonically decreasing with γ .
Therefore, when γ → 0 , the lower bound cannot approach −∞ , which means at this case, the model can never achieve optimum.
Thus, there must exist some active dimensions whose variance satisﬁes σ 2 z q ( x, c )
i
=
O
( γ ) , i = 1 , . .
, κ as γ → 0 to reach the global optimum, and it is showed in C (cid:48)
that as long as the number of such active dimensions exceeds r − k , as γ approaches zero, the reconstruction term is at most at the rate of O (1) .
Next, we will show that when there exist at least r − k such active dimensions, the CVAE model’s optimum is achievable.
D.1.2 Bounds of CVAE cost
The upper bound We have z 1: t − k
= µ z q ( x, c ) 1: t − k + σ z q ( x, c ) 1: t − k ε 1 and z t − k
+1: r = µ z q ( x, c ) t − k +1: r − k
+ σ z q ( x ) t − k +1: r − k ε 2 , where ε 1 ∼ N (0 , I t − k ) , ε 2 ∼ N (0 , I r − t ) .
The loss is:
[Equation]
Denote σ 2 z q ( x, c ) 1: t − k as σ 2 z q ( c ) for simplicity
[Equation]
and denote the upper bound of loss as L uc .
Take the derivative of σ z q ( c ) and σ z q ( x, c ) t − k
+1: r − k separately.
Because the diagonal elements in σ z q are independent
[Equation]
we can make both achieve optimum.
We have:
[Equation]
To minimize L uc ( γ, k ) , the optimal k is t , thus the upper bound is:
[Equation] [Equation]
The lower bound We have show that there must be at least r − k active dimension at a rate of
O ( γ ) , otherwise the loss will increase at a rate of O ( γ ) .
We can get a lower bound
[Equation]
Denote the lower bound as L lc ( γ, k ) , to minimize it, we have k = t , thus the lower bound is
[Equation]
Both L uc and L lc are at a rate of O (log γ ) , we come to the conclusion that the ELBO is ( d − r + t ) log γ + O (1) and the number of active dimensions is r − t
when p θ ( z | c ) = p ( z ) .
D.2 The general case
Deﬁne a trainable parametric prior of z , i.e. z ∼ N ( µ z p ( c ) , σ 2 z p ( c )) .
Since involving c in the prior doesn’t affect the reconstruction term, we have the conclusion in Section D.1.1 that there are at least r − k active latent dimensions at a rate of O ( γ ) .
Without loss of generality, we assume the ﬁrst r − k dimension of σ 2 z q ( x, c ) , i.e. σ 2 z q ( x, c ) 1: r − k
= O ( γ ) .
The upper bound
The loss is:
[Equation] [Equation]
Since we can only control k dimensions of the prior when training
[Equation]
take the derivative of µ z p ( c ) 1: k and σ z p ( c ) 1: k , we have
[Equation]
Let them achieve the optimal values.
The loss becomes
[Equation]
Eq 24 shows that if we have a ﬂexible enough prior, there are t − k latent dimensions that won’t provide any loss both in reconstruction and kl term.
To minimize Eq 24, σ 2 z q ( c ) ∗ = 0 and σ 2 z q ( x ) ∗ t − k
+1: r − k =
γ IL 2 .
Let them be the optimums, and view the terms that are irrelevant with γ when it approachesas constants, we have
[Equation]
The lower bound to get the lower bound, we have
[Equation]
[Equation] [Equation]
z p
[Equation]
1: κ
[Equation]
The last inequality comes from the conclusion that there are at least r − k active dimensions at a rate of O ( γ ) and the loss is monotonously increase with γ .
In this case k can be any integer in [0 , t ] , thus we cannot determine how many dimensions are used by the encoder and decoder separately.
But no matter what value k is, the cost of CVAE is
[Equation]
E Proof of TheoremRecap of Theorem
Theorem 6 (Weight Sharing Compromises Performance of Sequential Modeling)
By sharing weights between the encoder and prior, the conditional VAE model has its cost being
[Equation]
Then we have L c ( θ, φ ) = Ω(1) for any θ and φ .
Summary of Proof In this section, we ﬁrst deﬁne a space of sequences, and then separate the sequences into two categories according to the performance of the KL term.
In Sec E.1, we analyze the case when the Kl term equals O (log 1 γ ) , and in Sec E.2, the rate of KL term is higher than
O (log 1 γ ) .
In both categories, we prove that the whole cost cannot go to −∞ .
Let θ ∗ , φ ∗ = arg min θ,φ L c ( θ, φ ) .
Deﬁne S
⊂ X as the set of the sequences, and the sequence is deﬁned as { x l } ∞ l =1 ∈ S .
Consider when l equals to a constant l 0 , we have the prior as q φ ∗ ( z | x <l 0 ) , and encoder as q φ ∗ ( z | x ≤ l 0 ) .
Next, consider l = l 0 + 1 , we have the prior as q φ ∗ ( z | x ≤ l 0 ) , which is exactly the same as the encoder at l = l 0 , and the encoder as q φ ∗ ( z | x ≤ l 0 +1 ) .
The cost function at these two points are
[Equation]
and
L
[Equation]
respectively.
We don’t include the sequences within which all the samples share the same values into consideration in this theorem.
Next, we separate the sequences with varied values into two cases.
E.1 KL term is at a rate of O (log 1 γ ) when γ → 0 .
Denote log q φ ( z | x ≤ l ) − log q φ ( z | x <l ) =
f l ( γ ) =
O (log 1 γ ) .
In this setting, we analyze the reconstruction term
[Equation]
Similarly we have
[Equation]
With the condition that log q φ ( z | x ≤ l ) − log q φ ( z | x <l ) =
f l ( γ ) =
O (log 1 γ ) , we have
KL [ q φ ( z | x ≤ l ) ||
q φ ( z | x <l )]
=
E q φ ( z | x ≤ l ) [log q φ ( z | x ≤
[Equation]
That shows KL term is either small than a constant or goes to inﬁnity at a slower rate than rate of log 1 γ when γ → 0 .
We can also get q φ
[Equation]
z | x <l ) ≥
[Equation]
) , from which we have
[Equation]
Together we have
[Equation]
For any l 0 = 1 , 2 , . . .
and all z ∈ Z
[Equation]
we have the following cases:
For any z ∈ Z 1 , µ ∗ x ( z ) =
x l 0 and µ ∗ x ( z ) (cid:54) =
x l 0 +1 .
We have || x l 0 − µ ∗ x
( z ) || 2 γ → ∞ at a rate of O ( 1 γ ) .
For any z ∈ Z 2 , µ ∗ x ( z ) =
x l 0 +1 and µ ∗ x ( z ) (cid:54) =
x l 0 .
We have || x l 0+1 − µ ∗ x ( z )
||
2 γe fl 0( γ ) → ∞ at a rate of O ( 1 γe fl 0( γ ) ) .
For any z ∈ Z 3 , µ ∗ x ( z ) (cid:54) =
x l 0 and µ ∗ x ( z ) (cid:54) =
x l 0 +1 .
Both cases above cause the norm term equal Ω(1) .
With the setting
[Equation]
the lower bound of reconstruction term is
[Equation]
If ( i ) Z 1 ∪ Z 3 is zero measured
[Equation]
( ii ) for any z ∈ Z 1 ∪ Z
[Equation]
q φ ∗ ( z | x ≤ l 0 +1 ) = 0 or ( iii )
|| x l
[Equation] [Equation]
µ ∗ x ( z ) || 2 approaches zero, all the density will collapse to a zero-measured set, which may cause over-ﬁtting.
In this case, there must exist a sequence { x l } i 0 , in which (cid:80) ∞
l
[Equation]
1 (cid:82)
[Equation]
q
[Equation]
( z | x
[Equation]
+1 )
|| x l +
[Equation]
− µ ∗ x ( z ) || 2 dz ≥ C , where C is a constant, otherwise all the sequences { x l } i ∈ S, i = 1 , 2 , . .
share the same values.
Then, for Eq 33, there must exist a constant C (cid:48) , such that
[Equation]
Thus, the lower bound of the cost is
[Equation]
When γ goes to zero, O ( 1 γ ) >
O (log 1 γ ) .
We get the conclusion that L c ( θ, φ ) =
(cid:82) X Ω(
) ω gt ( dx ) for any θ and φ .
E.2 KL term goes to inﬁnity at a rate higher than O (log 1 γ ) .
In this case, we have
[Equation]
Thus it can only happen when there are some dimensions where σ 2 z ( x ≤ l 0 ) is active while σ 2 z ( x ≤ l 0 +1 ) is not, which indicate that tr ( σ − 2 z ( x ≤ l 0 ) σ 2 z ( x ≤ l 0 +1 ))
→ ∞ at a rate of Ω( 1 γ ) .
We have −
[Equation]
Because
[Equation]
−∞ at a rate of
O
[Equation]
log γ ) while tr
[Equation]
→ ∞ at a rate of Ω( 1 γ ) , the whole loss will go to inﬁnity.
In summary, in both cases, when summing over l , L c ( θ, φ ) will go to inﬁnity.
F Proof of Corollary 2. Recap of Corollary 2.
Corollary 2.1 (Adaptive Active Dimension) Suppose there exists conditioning variables c 1 ∈ C tand c 2 ∈ C t 2 , i.e. c 1 , c 2 have t 1 and t 2 effective dimensions respectively.
Given a CVAE model with optimal solution { θ ∗ , φ ∗ } , we have the number of active dimensions of z 1 ∼ q φ ∗ ( z | x, c 1 ) equals to r − t 1 , and that of z 2 ∼ q φ ∗ ( z | x, c 2 ) equals to r − t 2 .
Under the setting of Theorem 5, the CVAE model is arbitrarily complex.
Denote X t 1 and X t 2 , where
X
[Equation]
∪ X
Then for any x ∈ X t 1 and x ∈ X t 2 .
Denote t = t 1 ω ( X t 1 ) + t 2 ω ( X t 2 ) , which is a weighted-average of t 1 and t 2 .
We have
Thus we have
[Equation]
Thus the active dimension of z 1 ∼ q φ ∗ ( z | x, c 1 ) equals r − t 1 , and that of z 2 ∼ q φ ∗ ( z | x, c 2 ) equals r − t 2 .
G Justiﬁcation of Remark
Consider a κ -simple CVAE model with encoder q φ ( z | x, c ) , prior p θ ( z | c ) and decoder p θ ( x | z, c )
[Equation]
Further
[Equation]
let µ q ( x, c ) , σ q ( x, c ) be the distributional parameters for z ∼ q φ ( z | x, c ) , µ p ( c ) , σ p ( c ) be the distributional parameters for z ∼ p θ ( z | c ) .
Name this model as M
[Equation]
and we have its cost with regard to ( θ, φ ) being
[Equation] [Equation]
φ
[Equation]
=
[Equation]
Next, we construct another κ -simple CVAE, M (cid:48) , with a standard Gaussian prior, only using computation modules in M .
Speciﬁcally, the new prior, decoder, and encoder are deﬁned as:
• Prior: p (cid:48)
( z (cid:48) )
=
N (0 , I ) • Decoder: p (cid:48)
( x | z (cid:48) , c ) =
p θ ( x | z (cid:48) ∗ σ p ( c ) + µ p ( c ) , c ) • Encoder: q (cid:48)
( z | x, c ) =
N ( µ (cid:48) q , σ (cid:48) q ) , where – µ (cid:48) q
= ( µ q ( x, c ) − µ p ( c )) /σ
p ( c ) – σ (cid:48) q
= σ q ( x, c ) /σ p ( c )
With M (cid:48) deﬁned, we are going to show that it has the exact same cost value as the above one during training, i.e. L ( M (cid:48) ; θ, φ ) =
L ( M ; θ, φ ) , and the generated data distribution during generation, i.e. p (cid:48) ( x | z (cid:48) , c ) N ( z (cid:48) ; 0 , I ) ≡ p θ ( x | z, c ) p θ ( z | c ) .
During training, we have z (cid:48) ∼ N ( µ (cid:48) q , σ (cid:48) q ) , thus z (cid:48) ∗ σ p ( c ) +
µ p ( c ) ∼ N (( µ q ( x, c ) − µ p ( c )) /σ p ( c ) ∗ σ p ( c ) + µ p ( c ) , σ q ( x, c ) /σ p ( c ) ∗ σ p ( c ))
=
N ( µ q ( x, c ) , σ q ( x, c )) .
Thus, we have
[Equation]
Besides, we also have
[Equation]
In terms of generation equivalence
[Equation]
for any z (cid:48) p ∼ N (0 , I )
[Equation]
we have
[Equation]
Therefore we conclude that M (cid:48) and M share the same cost value,
i
[Equation]
e. L c ( M (cid:48) ; θ, φ ) =
L c ( M
[Equation]
θ, φ ) , and equivalent data generation distributions.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] (c) Did you discuss any potential negative societal impacts of your work?
[No] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] See Appendix. 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] See Sectionand Appendix.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[No] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See Appendix. 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] (b) Did you mention the license of the assets?
[Yes] See Section(c)
Did you include any new assets either in the supplemental material or as a URL?
[No] (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[No] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[No]
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]