Learning Manifold Dimensions with Conditional Variational Autoencoders
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Although the variational autoencoder (VAE) and its conditional extension (CVAE)are capable of state-of-the-art results across multiple domains, their precise behavioris still not fully understood, particularly in the context of data (like images) thatlie on or near a low-dimensional manifold.
For example, while prior work hassuggested that the globally optimal VAE solution can learn the correct manifolddimension, a necessary (but not sufﬁcient) condition for producing samples fromthe true data distribution, this has never been rigorously proven.
Moreover, itremains unclear how such considerations would change when various types ofconditioning variables are introduced, or when the data support is extended to aunion of manifolds (e.g., as is likely the case for MNIST digits and related).
Inthis work, we address these points by ﬁrst proving that VAE global minima areindeed capable of recovering the correct manifold dimension.
We then extendthis result to more general CVAEs, demonstrating practical scenarios whereby theconditioning variables allow the model to adaptively learn manifolds of varyingdimension across samples.
Our analyses are also supported by numerical resultson both synthetic and real-world datasets.
Introduction
VAEs [6, 14] and CVAEs [17] are poweful generative models as they have achieved competitiveresults in various domains, for example, image synthesis [21, 5, 13], natural language processing [16],time-series forecasting [9, 19], trajectory prediction [8] etc.
Recently, it has been shown that it alsoachieves state-of-the-art performance for large-scale images given the right architecture [11].
Besides the success, VAEs and CVAEs still have observed behaviors that need precise understandingor theoretical analysis.
In particular, when the data lie on or near a low-dimensional manifold, it ismeaningful to have a model that learns the dimension of manifold correctly, since it is a necessary butnot sufﬁcient condition for the model to produce samples from the true data distribution.
In general data real-world [12], and the dimension can help the generation quality as well as the understanding of data itself.
Although it is suggested in priorwork [4, 3] that a VAE model can learn the correct manifold dimension when globally optimized, thishas only been proven under an assumption that the decoder is linear or approximately linear [2, 4].
Moreover, the ability to learn the correct manifold dimension is less clear when a conditioningvariable is introduced.
As a set of discrete conditions may correspond to different “slices” of dataand each could lie on a manifold with different dimension.
For example, in MNIST one may expectthe manifold dimension of images labelled “1” is smaller than those of “5”.
It is also possible tohave data expand fully in the ambient space but lie on a low-dimensional manifold when conditioned.
Such an example can be on purposely constructed by directly selecting some data dimensions andtake corresponding manifold dimensions as the conditioning variable.
In addition, the above theoretical properties of VAEs and CVAEs not only help our understandingon the precise behavior on paper, but also directly relevant to the analysis on some common modeldesign choices and assumptions in practice.
For example, there has been debate about the choicebetween a ﬁxed data variance and a learnable one [4, 10, 15, 18, 3].
Another commonly adaptedstrategy is to share weights for prior and encoder in CVAEs [7, 17].
It may not be obvious, but thesedesign choices are relevant to the ability of learning the manifold dimensions.
Our paper raises and investigates the following four unresolved issues for VAEs and CVAEs:
(i)
Besides the extensive observation of VAE being able to learn the correct data manifolddimension, there is no rigorous proof of it yet.(ii)
It remains unclear how to extend the above observation of VAE to conditional VAE, i
e. howare the manifolds related with conditioning variables.(iii)
There has no discussion on the situation where data lies on a union of manifolds.(iv)
The active practices on VAE and CVAE models emerges various model assumptions and
design choices, it remains to be discussed that how these assumptions and choices impactthe learning of correct manifold dimensions.
In Section 2, we ﬁrst formally prove that a VAE model can learn the correct data manifold dimension,then extend it to the conditional VAE.
In Section 3, some popular model design choices are discussedand analyzed.
Section 4 supports our theoretical conclusions with numerical experiments on bothsynthetic and real-world datasets.
Learning the Dimension of Data Manifold
We begin with the deﬁnition of data:
Deﬁnition 1 (Data lying on a Manifold) Suppose r and d are two positive integers and r < d , Xis a simple r -Riemannian manifold embedded in R d when there exists a diffeomorphism ϕ between
X and R r .
Speciﬁcally, for every x ∈ X , there exists u = ϕ ( x ) ∈ R r , where ϕ − 1 exists and both ϕand ϕ − 1 are invertible and differentiable. Given inﬁnite observable variables x ∈ X , where X is a r -dimensional manifold embedded in 62 R d whose deﬁnition is in 1, and latent variable z ∈ Z κ ⊆ R κ which serves as a low-dimensionalrepresentation, a VAE model is built to approximate the ground truth measure with a parametricdensity p θ ( x ) = (cid:82) p θ ( x | z ) p ( z ) dz , where prior p ( z ) = N (0 , I ) .
Denote the ground-truth measure on X as ω gt .
The probability mass of an inﬁnitesimal dx on the manifold is ω gt ( dx ) and (cid:82) X ω gt ( dx )
= 1 .
The canonical cost of VAE model is a bound on the average negative log-likelihood
[Equation]
Before starting our main theorems, we ﬁrst convert the similar idea from [4] and formalize a κ -simple
VAE model with explicit Gaussian and Lipschitz assumptions, as well as the way of parameterizations.
Deﬁnition 2 ( κ -simple VAE)
A κ -simple VAE is deﬁned as a VAE model with dim[ z ] = κ latentdimensions, the Gaussian encoder q φ ( z | x ) =
N ( z | µ z ( x ; φ ) , diag { σ 2 z ( x ; φ ) } ) , the Gaussian decoderp θ ( x | z ) = N ( x | µ x ( z ; θ ) , γI ) , and the prior p ( z ) =
N ( z | 0 , I ) .
Here γ > 0 is a trainable scalarin θ .
The mean functions of the encoder and the decoder, i.e. µ z ( x ; φ ) and µ x ( z ; θ ) are arbitrary L -Lipschitz continuous functions.
2.1 Learning Manifold Dimension in VAE
We start by analyzing the dimension of manifold learning in VAE models referred to issue (i).
As it ispointed out in [4], if γ of a VAE model is trainable and not constrained, γ must be arbitrarily smallbut explicitly nonzero to minimize the cost and reconstruct data from X .
Here we deﬁne active latentdimensions as the dimensions that have a decaying encoder variance related to γ .
Deﬁnition 3 (Active Latent Dimensions in VAE)
In a κ -simple VAE, let { θ ∗ γ , φ ∗ γ } denote anyglobal optimal κ -simple VAE solutions as a function of any ﬁxed γ , then for any j = 1 , . . .
, κ ,a dimension j of latent variable z is deﬁned as an active dimension if the optimal encoder varianceσ z ( x ; φ ∗ γ ) 2 j → 0 as γ → 0 .
Theorem 1 (Learning the Data Manifold Dimension Using VAEs) Suppose training data lies ona r -dimensional manifold embedded in R d .
Then for any κ ≥ r , when a κ -simple VAE model achievesits any global optimums, we have
(i) L ( θ ∗ γ , φ ∗ γ ) =
( d − r ) log γ[-
-] + O (1) , and
(ii)
The number of active latent dimensions almost surely equals r and
(iii)
The optimal variance of each active dimension satisﬁes σ z ( x ; φ ∗ γ ) 2 j = O ( γ ) .
A rigorous proof of Theorem 1 is provided in the appendices.
As a summary, our proof follows thispath: ﬁrst, we prove the existence of active dimensions and the rate by contradiction.
The conclusionis that there must exist at least r active dimensions whose encoder variances going to zero at a rateof O ( γ ) for reconstruction tasks, otherwise the reconstruction term will grow at a rate of O ( 1 γ ) as γgoes to zero.
Next, we get an upper bound and a lower bound of ELBO and show that their differenceis a constant, both in the form of ( d − r ) log γ + O (1) .
Last, we get the exact number of activedimensions by showing that adding unnecessary active dimensions decreases the coefﬁcient of log γ ,which may increase loss value.
This result address issue (i) in Section 1.
It implies that we can explicitly obtain the manifolddimension and the state of convergence by viewing the performance of σ 2 z and γ .
Also, both intheory and experiments, the ratio of the norm term in reconstruction and γ is a constant, i.e. thedata dimensions d .
Speciﬁcally, when a VAE model converges to its optimum, γ will go to zero,as well as r dimensions of σ 2 z .
To achieve the optimal solution, the active latent dimensions playa crucial role in reconstruction and the inactive ones help to push KL regularization factor to itsminimum along these dimensions.
By deﬁnition, inactive dimensions are with a variance such thatlim γ → 0 σ z ( x ; φ ∗ γ ) =
C > 0 , where C is some constant.
When such latent dimensions are greaterthan d − r , no matter how many inactive dimensions there are, the model cannot perfectly reconstructeven a single manifold dimension.
2.2 Learning Manifold Dimension in Conditional VAE
In this section, we extend the analysis on κ -simple VAE models to κ -simple CVAE models.
We ﬁrstthe manifold with condition.
Speciﬁcally, suppose X is a Riemannian manifold and there exists a set
C where for each x ∈ X , there is a corresponding c ∈ C , and we call that c is the condition of x .
Deﬁnition 4 ( κ -simple CVAE)
For any x ∈ X and its corresponding c ∈ C , a κsimple CVAE is an extension of κ -simple VAE with the Gaussian encoder q φ ( z | x, c ) =
N ( z | µ z ( x, c ; φ ) , diag { σ 2 z ( x, c ; φ ) } ) , the Gaussian decoder p θ ( x | z, c ) = N ( x | µ x ( z, c ; θ ) , γI ) , andthe prior p θ ( z | c ) =
N ( z | µ z ( c ; θ ) , diag { σ 2 z ( c ; θ ) } ) . µ z ( x, c ; φ ) and µ x ( z, c ; θ ) are arbitrarily LLipschitz continuous, while µ z ( c ; θ ) can be arbitrary function.
The canonical CVAE cost with conditional variable being c is
[Equation]
When prior is independent of c , i.e. p ( z | c )
= p ( z ) , the cost still holds.
In a CVAE model, besides samples x ∈ X , the conditioning variable c may also provide informationrelevant to x .
Thus we deﬁne the effective dimension to measure the contribution of the conditioningvariable.
Deﬁnition 5 (Effective Dimension of the Conditioning Variable)
For any x ∈ X and its corresponding c ∈ C , we call integer t the number of effective dimensions of c , if 1) there exists a functiong : C → R t and exists t dimensions of ϕ ( x ) , denoted as ϕ ( x ) t , such that g ( c ) =
ϕ ( x ) t , where ϕ is the diffeomorphism in Deﬁnition 1, and 2) there doesn’t exist such a function g for t + 1 .
Further, let
C t ⊂ C be the subset for all c with effective dimensions of t .
Deﬁnition 5 shows that any c ∈ C t can reconstruct at most t ≤ r dimensions of the manifold.
Also, we extend our deﬁnition of active latent dimensions to CVAE.
Deﬁnition 6 (Active Latent Dimensions in CVAE)
In a κ -simple CVAE, denote diag { σ z p ( c ; θ ) }and diag { σ z q ( x, c ; φ ) } as the variance of the prior and encoder.
For any j = 1 , . . .
, κ , a dimensionj of latent variable z is deﬁned as an active dimension if the ratio σ z q ( x, c ; φ ∗ γ ) [-
-]2 j /σ z p ( c ; θ ∗ γ ) 2 j → 0 asγ → 0 .
As special cases, for VAE or CVAE with a standard Gaussian prior we have σ z p ( x ) 2 j = 1 , and thatindicates Deﬁnition 3 is a special case of Deﬁnition 6.
Next, we extend Theorem 1 to CVAE models, which describes how the manifold dimension relates toconditioning variables.
Theorem 2 (Learning Manifold Dimension in CVAE)
Suppose X is a r -dimensional manifoldembedded in R d with its corresponding condition c ∈ C t .
Then for any κ ≥ r , when a κ -simple CVAE model achieves its global optimum, we have
(i) L c ( θ ∗ γ , φ ∗ γ ) =
( d − r + t ) log γ + O (1) , and
(ii) The
number of active latent dimensions almost
surely equals r − t , and
(iii)
The variance of each active dimension satisﬁes σ z q ( x, c {+; φ ∗ γ ) 
+}
j /σ z p {+( c ; θ ∗ γ ) 
+}
j = O ( γ ) .
This theorem addresses issue (ii) in Section 1.
It shows the signiﬁcance of conditioning variable thatit can push the cost to a smaller value than a VAE model without condition, which is favorable.
It alsopoints out that the condition will reduce the number of active latent dimensions without disruptingthe data ﬁt.
It worth mentioning that even though the data is not on the manifold initially, once aconditioning variable with a positive effective dimension is incorporated, the conditioned data thenbecome on a manifold, which can be induced directly in our theorem.
Moreover, to address issue (iii) in Section 1, we have the following results describing CVAE’scapacity in learning data lies on a union of manifolds.
Corollary 2.1 (Adaptive Active Dimension) Suppose there exist x 1 , x 2 ∈ X with r -dimensionalmanifold and their corresponding conditioning variables c 1 ∈ C t 1 and c 2 ∈ C t 2 , i.e. c 1 , c 2 have t 1 andt 2 effective dimensions respectively.
Given a CVAE model with an optimal solution { θ ∗ , φ ∗ } , we havethe number of active dimensions of z 1 ∼ q φ ∗ ( z | x, c 1 ) equals to r − t 1 , and that of z 2 ∼ q φ ∗ ( z | x, c 2 )equals to r − t 2 .
This result indicates that CVAE can adaptively learn different data manifolds under conditioningvariables with different effective dimensions.
When c is discrete, it is equivalent to training separate
VAEs where each model corresponds to a discrete condition value.
On the other hand, collecting allpossible conditions in a single CVAE model enables it to compress the dimension dynamically.
Forexample, c describes the set of manifolds that share the same dimension, thus for samples in each set,the corresponding manifold dimensions are the same.
When c is continuous, conditioning variablesmay contain intact information of certain manifold dimensions, which substitute for correspondingactive dimensions.
In Section 4, we demonstrate numerical evidence of this behavior.
On Common Model Design Choices
In this section, we review some model designs and training practices that are commonly seen inprior works but could potentially negatively impact the model convergence and performance.
Thefollowing discussions can be seen as important steps towards resolving issue (iv) in Section 1.
3.1 On the Equivalence of Conditioned and Unconditioned Prior
In prior works, it is popular to design a parameterized prior p θ ( z | c ) 1 in CVAE [24, 7, 8, 20, 1], whichis a trainable module alongside the encoder q φ ( z | x, c ) and the decoder p θ ( x | z, c ) .
Remark 1 (Remove conditioning variable from the Prior) Consider a κ -simple CVAE modelwith prior p
θ ( z | c ) , encoder q φ ( z | x, c ) and decoder p θ ( x | z, c ) .
We can always ﬁnd another κsimple CVAE model with prior p ( z ) ∼ N (0 , I ) , encoder q φ (cid:48)
( z | x, c ) and decoder p θ (cid:48) ( x | z, c ) , suchthat L c ( θ, φ ) =
L c ( θ (cid:48) , φ (cid:48) ) .
Remark 1 indicates that a conditioned prior is not necessary.
Speciﬁcally, as shown in its proof, we canalways explicitly convert an existing κ -simple CVAE model with prior p θ ( z | c ) into another κ -simple
CVAE model with prior being N (0 , I ) .
In practice, when we have the ﬂexibility in architecturedesign, it is equivalent to use a standard Gaussian prior or a parameterized one.
3.2 The Initial γ Impacts Model Convergence and Dimension Learning
Recall in Theorem 1, we have L ( θ ∗ γ , φ ∗ γ ) =
( d − r ) log γ + O (1) , and that implies when L ( θ ∗ γ , φ ∗ γ )approaches to −∞ , γ is expected to be closer to 0.
This behavior makes γ important in CVAEtraining.
A direct conclusion is that the model would be compromised if γ is set to be a ﬁxed positiveconstant, as analysed and empirically studied in literature [4, 10, 15, 18, 3].
Although a model converging towards global optimum would push γ to 0 regardless of its initial value, in practice, we observe that this initial value of γ still signiﬁcantly impacts the model performance.
In the cost function, we have
[Equation]
By this formulation we see that γ acts as the weight between data likelihood and KL-divergence.
A large γ downgrades the weight for data likelihood, and encourages the model to optimize the
KL-divergence term ﬁrst, and vice versa.
Making γ a learnable parameter is equivalent to adaptivelybalancing the cost.
When the initial value is set to be on an extreme end, the model is at risk of fallinginto a local minimum, as empirically in Section 4.4.
3.3 Weight Sharing Keeps Model from Global Optimal Convergence
The cost of VAE balances the three modules the encoder, decoder and prior with the two terms, thedata likelihood and KL-divergence.
It has been argued in literature [3] that such a cost might lead tounstable training dynamics.
Therefore, one widely adapted trick[17] in practice is to share modelweights between prior and posterior.
Weight sharing is conducted in various forms.
For data from a general domain where x and c aredifferently distributed, the weight sharing couples with the design in which the posterior takes in xand c as two input and process differently at the beginning, and the prior processes c as its input withthe same module and weight in the posterior.
In this section we take the sequential data as a representation for the weight sharing issue because ittakes a more thorough approach of weight sharing.
Sequential data poses the task of using early partsof a sequence to predict the later parts.
Formulating in CVAE, we have c being the early parts, andthat conditions the to-predict later parts x .
Assuming x is a sequential data, i.e. { x l } with l > 0 denoted as the index, and each x l is paired withits condition c = x <l .
As a result, the encoder becomes q φ ( z l | x ≤ l ) , and the prior becomes p ( z l | x <l )for any l greater than the observed sequence length.
It has been proposed in literature [7, 8] to shareweights between the encoder and prior, i.e. p ( z l | x <l ) =
q φ ( z l | x <l ) .
Theorem 3 (Weight Sharing Compromises Performance of Sequential Modeling)
By sharingweights between the encoder and prior, the conditional VAE model has its cost being
[Equation]
Then we have L c ( θ, φ ) = [-(cid:82) X Ω(
 ) ω gt ( dx )-]Ω(1) for any θ and φ .
Theorem 3 implies that a fully weight-sharing pair of prior and encoder prevents the model fromachieving its global optimum.
As the numerical study in Section 4.5, this weight sharing strategycould severely constraint the performance.
Experiments
In this section we present numerical results to support our analysis.
We ﬁrst verify model behavior inthe synthetic environment for a controllable set of experiments, and extend to real-world datasets tofurther validate our conclusions.
4.1 Dataset and Metrics
Synthetic Dataset
The basic data samples are generated from a mixture of a ﬁnite number of Gaussian distributions.
Speciﬁcally, we ﬁrst generate a 5-dimensional categorical selecting vectorfrom uniform distribution, and normalize it with norm equaling one.
Then we generate two rdimensional parameter vectors µ u and log σ u from a standard Gaussian distribution in R r .
Byrepeating the second step 5 times, we get 5 independent Gaussian distributions.
Our synthetic datau ∈ R r is generated by selecting one of the 5 candidate distributions with a probability of theselection vector, and next generate samples from the selected distribution.
To get the manifold Xin R d , we design a function h :
R r → R d , and the data sample x = h ( u ) .
In our experiments,h ( u ) =
Sigmoid ( G x u ) , where G x ∈ R d × r is randomly initialized controlled by a certain randomseed.
The conditioning variable c is transformed by another transform h (cid:48)
( u 1: t ) =
G (cid:48)
u 1: t , where G (cid:48) ∈ R t × t .
Given a ﬁxed seed, the generation results is controlled by parameters r , d and t .
For allthe experiments, the training size of synthetic data is 100,000.
Real-world dataset Besides synthetic dataset, we also investigate the model behavior on
MNIST[23] and Fashion MNIST [22], two image datasets 2 .
These two datasets contain x withcomplex mappings between the manifold and ambient space.
At the same time, they are not toocomplex thus the training are made reasonably easy with common neural architecture, otherwise ourfocus could be driven to speciﬁc architecture choices other than VAE and CVAE model behavior.
Bycarefully tuning the pair ( x, c ) , we can control how c impacts the model.
Metrics The main metric that we use to evaluate VAEs and CVAEs is the negative Evidence Lower BOund (ELBO), as in Eq 1 and Eq 2, and note that it is reported on convention and not forcomparison.
There are also auxiliary metrics that we use to compare and diagnose models, including Active Dimensions (AD), Reconstruction error E || x − µ x ( z ) || 2 (Recon), the KL-divergence ( KL ),and the data variance γ .
4.2 Manifold Dimensions Learning in VAE
Here, we provide direct numerical support to Theorem 1.
Speciﬁcally, we show that the modelcaptures the correct dimensions of manifold regardless of the ambient data dimensions.
To indicatehow the active dimension is determined, we provide an example in Section B of the appendix.
Although in numerical experiments a variance cannot be exact zero, we can still observe from aconverged model that each variance of encoder is either close to 1, or close to zero.
With d = κ = 20 and varying r , we demonstrate that the VAE model can learn the active latentdimensions when γ converges to a small value, regardless of the value of r .
The results are showed in
Table 1 and all the numbers are values at convergence.
Experimental results on MNIST and Fashion MNIST are reported in Table 2.
Without knowing thetrue manifold dimensions, a small latent dimension discourages the model from catching up theinformation for every dimension, while once given enough latent dimensions, the number of activedimensions equals to data’s intrinsic dimension and the rest is for the regularization.
The two parts oflatent dimensions cooperate to push the VAE model to its global optimum. [Table]
[Table]
4.3 CVAE Adaptive Active Dimensions Varied With c
In this section [Table] we design experiments to show that a CVAE model can learn the dimension ofmanifold with respect to the effective latent dimensions of the condition.
Further, it can also learn theadaptive active dimension on a union of manifolds.
By Theorem 2 and Remark 1, a CVAE model with an unconditioned prior and condition c ∈ C t isequivalent to a general one in design, whose active dimensions equals r − t .
In this experiments, weinvestigate the behavior of a CVAE model when data lies on a single manifold and its relation tocondition is static.
Speciﬁcally, we set c to be the ﬁrst t dimensions of u , i.e. take G (cid:48) as I t , and letd = κ = 20 , r = 10 .
Table 3 shows the results.
On MNIST and Fashion MNIST, we train CVAEs with the class label of each image as the conditioning variable [Table] and the results with κ = 32 are in Table 4.
By comparing it to Table 2, we notice that the
CVAE model for MNIST has a lower active dimension and better ELBO, while for FashionMNISTthe CVAE performs similarly.
This implies that when data points in MNIST are conditioning on thelabel, they lie on a lower-dimensional manifold, while for data points in FashionMNIST where amongclasses the manifold dimensions are similar, the class label doesn’t reduce the manifold dimension.
This characterization of the dataset complexity between classes is consistent with their intuitive visual
[Table] [Table]
complexity. We also provide an example of the encoder variance on the MNIST dataset in Section Bof the appendix.
Discrete Condition To verify Corollary 2.1, we train a CVAE model on d -dimensional data thatlie on a union of manifolds with different dimensions, with the conditioning variable c being anindicator of the source manifold.
Table 5 shows that the model successfully captures the active latentdimensions with an attention layer in the decoder [Table] while without the attention layer the model learnsa static active dimension of 4.
The result indicates two aspects of CVAE: 1) CVAE models havecapacity to capture a union of manifolds simultaneously and converge on each manifold (indicated by-ELBO); 2) In practice, we need careful model architecture design to help the model to distinguishmanifolds.
In this experiment d = 20 and κ = 40 .
Continuous Condition When c is a continuous variable, we show that the model can also adaptivelylearn the active dimensions.
In our setting, a continuous condition c ∈ C t contains t -dimensionalinformation of samples, thus under different conditions the data may lie on different manifolds.
Weset r = 12 , d = 20 and κ = 90 , then we set the ﬁrst 2 , 4 , 6 , 8 , 10 dimensions of u as the conditions.
Since the model needs dimension of c to be consistent, we padding them to R 10 with 0.
With theattention layer, the model can use the condition to collapse certain dimensions instead of activatingall the possible dimensions and increasing unnecessary loss in KL term.
The results are included in
Table 6.
4.4 Initialization of γ Impacts Model Training
Here we check how the initialization of γ would impact model convergence, as mentioned in
Section 3.2.
Our results are in Table 7.
With different initial log γ values, we observe the convergenceof VAE is affected signiﬁcantly, as a result, the model can’t capture the correct manifold dimension.
In comparison, CVAE models are less sensitive, as CVAEs have more parameters and the conditioned [Table] [Table] [Table]
data lie on a lower-dimension manifold thus it actually poses an easier task.
Still, when log γ = 20 ,they converge to local optima.
In this experiment, d = 20 , κ = 20 , r = 10 .
In CVAE, t = 5 .
4.5 Shared Weights Between Encoder and Prior
The experiments in this section is to support Theorem 3, i.e. sharing weights between the encoderand prior is not a satisfactory way for training in sequential data, we generate a sequence by
AutoRegressive-Moving-Average (ARMA) model with ﬁxed parameters, then use rolling windows tomake our samples.
Speciﬁcally, the window size is 5, i.e. each sample we use the past 5 points as ourcondition, and reconstruct the sixth one which is viewed as current sample.
The results in Table 8show that when weight shared, the norm term of reconstruction cannot go down, which may result ina restriction in the capacity of the model, and this observation is coherent to the key contradictionused in our proof of Theorem 3.
Conclusion
In this paper we provide insights on the behavior of VAEs and CVAEs for data lying on lowdimensional manifolds.
In the theoretic analysis, we show that VAE has the ability to learn themanifold dimension.
More interestingly, we extend the conclusion to Conditional VAE that workswith conditioned data.
This additional conditioning term impacts the relation between data andmanifold, as a result it also affects the model design choices in practice.
With the theoretic results,we further examine some design choices.
A dedicated section of experiments supports our statementsand conclusions numerically on both synthetic and real-world data.
Finally, we believe this work canin general help to improve the design and diagnose techniques of VAEs and CVAEs.
Limitations Our work may still have limitations in empirical veriﬁcation for real-world datasetsat a larger scale or from other modalities than images.
Although our contributions are theoreticallygrounded, in practice it still takes non-trivial efforts to design model architectures that both align withour guideline and ﬁt the data. [Table]
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[No](d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]2.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes](b) Did you include complete proofs of all theoretical results?
[Yes] See Appendix.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] See Section 4and Appendix.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[No]
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See Appendix.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes] See Section
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[No]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[No]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
[N/A](b)
Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
[N/A]