Learning Manifold Dimensions with Conditional Variational Autoencoders
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Although the variational autoencoder (VAE) and its conditional extension (CVAE)are capable of state-of-the-art results across multiple domains, their precise behavioris still not fully understood, particularly in the context of data (like images) thatlie on or near a low-dimensional manifold.
For example, while prior work hassuggested that the globally optimal VAE solution can learn the correct manifolddimension, a necessary (but not sufﬁcient) condition for producing samples fromthe true data distribution, this has never been rigorously proven.
Moreover, itremains unclear how such considerations would change when various types ofconditioning variables are introduced, or when the data support is extended to aunion of manifolds (e.g., as is likely the case for MNIST digits and related).
Inthis work, we address these points by ﬁrst proving that VAE global minima areindeed capable of recovering the correct manifold dimension.
We then extendthis result to more general CVAEs, demonstrating practical scenarios whereby theconditioning variables allow the model to adaptively learn manifolds of varyingdimension across samples.
Our analyses are also supported by numerical resultson both synthetic and real-world datasets.
Introduction
VAEs [7, 15] and CVAEs [18] are poweful generative models as they have achieved competitiveresults in various domains, for example, image synthesis [22, 6, 14], natural language processing [17],time-series forecasting [10, 20], trajectory prediction [9] etc.
Recently, it has been shown that it alsoachieves state-of-the-art performance for large-scale images given the right architecture [12].
Besides the success, VAEs and CVAEs still have observed behaviors that need precise understandingor theoretical analysis.
In particular, when the data lie on or near a low-dimensional manifold, it ismeaningful to have a model that learns the dimension of manifold correctly, since it is a necessary butnot sufﬁcient condition for the model to produce samples from the true data distribution.
In general world 13], and the can help the generation quality as well as the understanding of data itself.
Although it is suggested in priorwork [4, 3] that a VAE model can learn the correct manifold dimension when globally optimized, thishas only been proven under an assumption that the decoder is linear or approximately linear [2, 4].
Moreover, the ability to learn the correct manifold dimension is less clear when a conditioningvariable is introduced.
As a set of discrete conditions may correspond to different “slices” of data andeach could like on a manifold with different dimension.
For example, in MNIST one may expect themanifold dimension of an image of “1” is smaller than that of “5”.
It is also possible to have datathat expand fully in the ambient space but become on a low-dimensional manifold when conditioned.
Such an example can be on purposely constructed by directly selecting some data dimensions thatcorrespond to certain manifold dimensions as the conditioning variable.
In addition, the above theoretical properties of VAEs and CVAEs not only help our understandingon the precise behavior on paper, but also directly relevant to the analysis on some common modeldesign choices and assumptions in practice.
For example, there has been debate about the choicebetween a ﬁxed data variance and a learnable one [4, 11, 16, 19, 3].
Another commonly adaptedstrategy is to share weights for prior and encoder in CVAEs [8, 18].
It may not be obvious, but thesedesign choices are relevant to the ability of learning the manifold dimensions.
Our paper raises and investigates the following four unresolved issues for VAEs and CVAEs:
(i)
Besides the extensive observation of VAE being able to learn the correct data manifolddimension, there is no rigorous proof of it yet.(ii)
It remains unclear how to extend the above observation of VAE to conditional VAE, i
e. howare the manifolds related with conditioning variables. (iii)
There has no discussion on the situation where data lies on a union of manifolds.(iv)
The active practices on VAE and CVAE models emerges various model assumptions and
design choices, it remains to be discussed that how these assumptions and choices impactthe learning of correct manifold dimensions.
In Section 2, we ﬁrst formally prove that a VAE model can learn the correct data manifold dimension,then extend it to the conditional VAE.
In Section 3, some popular model design choices are discussedand analyzed.
Section 4 supports our theoretical conclusions with numerical experiments on bothsynthetic and real-world datasets.
Learning the Dimension of Data Manifold
Given inﬁnite observable variables x ∈ X , where X is a r -dimensional manifold embedded in R dand latent variable z ∈ Z κ ⊆ R κ which serves as a low-dimensional representation, a VAE model isbuilt to approximate the ground truth measure with a parametric density p θ ( x ) = (cid:82) p θ ( x | z ) p ( z ) dz ,where prior p ( z ) = N (0 , I ) .
We assume that X is a simple Riemannian manifold, i.e. there exists a diffeomorphism ϕ between
X and R r .
Speciﬁcally, for every x ∈ X , there exists u = ϕ ( x ) ∈ R r .
Denote the ground-truthmeasure on X as ω gt .
The probability mass of an inﬁnitesimal dx on the manifold is ω gt ( dx ) and 63 (cid:82)
X ω gt ( dx ) = 1 .
The canonical cost of VAE model is a bound on the average negative log-likelihood
L
[Equation]
[Equation]
Before starting our main theorems, we ﬁrst convert the similar idea from [4] and formalize a κ -simple
VAE model with explicit Gaussian and Lipschitz assumptions, as well as the way of parameterizations.
Deﬁnition 1 ( κ -simple VAE)
A κ -simple VAE is deﬁned as a VAE model with dim[ z ] = κ latent dimensions, the Gaussian encoder q φ ( z | x ) =
N ( z | µ z ( x ) , σ 2 z ( x )) , and the Gaussian decoderp θ ( x | z ) =
N ( x | µ x ( z ) , γI ) .
Here γ > 0 is a tunable scalar.
The mean functions of the encoder andthe decoder, i.e. µ z ( x ) and µ x ( z ) are sufﬁciently complex and L -Lipschitz continuous, while thevariance of the encoder σ 2 z ( x ) is a diagonal matrix.
2.1 Learning Manifold Dimension in VAE
We start by analyzing the dimension of manifold learning in VAE models referred to issue (i).
As it ispointed out in [5], if γ of a VAE model is trainable and not constrained, γ must be arbitrarily smallbut explicitly nonzero to minimize the cost and reconstruct data from X .
Here we deﬁne active latentdimensions to indicate the latent dimensions that are used for reconstruction tasks and have someproperties related to γ .
Deﬁnition 2 (Active Latent Dimensions in VAE)
In a κ -simple VAE, let γ 0 be an arbitrarily smallpositive value and satisﬁes γ 0 ≤ γ . For any j = 1 , . . .
, κ , a dimension j of latent variable z isdeﬁned as an active dimension if the encoder variance σ z ( x ) 2 j = O ( f ( γ 0 )) as γ 0 → 0 , where f isany function that goes to zero when γ 0 goes to zero.
Theorem 1 (Learning the Data Manifold Dimension Using VAEs) Suppose r < d .
Let{ θ ∗ γ 0 , φ ∗ γ 0 } denote the global optimal κ -simple VAE solution under γ = γ 0 .
Then for any κ ≥ r ,when a κ -simple VAE model achieves its global optimum, we have
(i) L ( θ ∗ γ 0 , φ ∗ γ 0 ) =
(cid:82) X ( d − r ) log γ {+
+} + O (1) ω gt ( dx ) can be uniquely achieved, and
(ii)
The number of active latent dimensions almost surely equals r , and
(iii)
The variance of each active dimension satisﬁes σ z ( x ) 2 j = O ( γ 0 ) .
A rigorous proof of Theorem 1 is provided in the appendices.
As a summary, our proof followsthis path: ﬁrst, we prove the existence of active dimensions and the rate by contradiction.
Theconclusion is that there must exist at least r active dimensions going to zero at a rate of O ( γ ) ,otherwise the reconstruction term will grow at a rate of O ( 1 γ ) as γ goes to zero.
Next, we get an upperbound and a lower bound of ELBO and show that their difference is a constant, both in the form of( d − r ) log γ + O (1) .
Last, we get the exact number of active dimensions by showing that addingunnecessary active dimensions decreases the coefﬁcient of log γ , which may increase loss value.
This result address issue (i) in Section 1.
It implies that we can explicitly obtain the manifolddimension and the state of convergence by viewing the performance of σ 2 z and γ .
Also, both intheory and experiments, the ratio of the norm term in reconstruction and γ is a constant, i.e. the datadimensions d .
Speciﬁcally, when a VAE model converges to its optimum, γ will go to zero, as well asr dimensions of σ 2 z .
To achieve the optimal solution, the active latent dimensions play a crucial rolein reconstruction and the inactive ones help to push KL regularization factor to its minimum alongthese dimensions.
2.2 Learning Manifold Dimension in Conditional VAE
In this section, we extend the analysis on κ -simple VAE models to κ -simple CVAE models.
Deﬁnition 3 ( κ -simple CVAE)
For c ∈ C , where C is the space of all the possible conditions, aκ -simple CVAE is deﬁned as a Conditional VAE model with dim[ z ] = κ latent dimensions, the
Gaussian encoder q φ ( z | x, c ) = N ( z | µ z ( x, c ) , σ 2 z ( x, c )) , and the Gaussian decoder p θ ( x | z, c ) = N ( x | µ x ( z, c ) , γI ) .
Here γ > 0 is a tunable scalar.
The mean functions of the encoder and thedecoder, i.e. µ z ( x, c ) and µ x ( z, c ) are L -Lipschitz continuous, while the variance of the encoderσ 2 z ( x, c ) is a diagonal matrix.
The canonical CVAE cost with conditional variable being c is
[Equation]
When prior is independent of c , i.e. p ( z | c )
= p ( z ) , the cost still holds.
In a CVAE model, besides samples x ∈ X , the conditioning variable c may also provide informationrelevant to x .
Thus we deﬁne the effective dimension to measure the contribution of the conditioningvariable.
Deﬁnition 4 (Effective Dimension of the Conditioning Variable)
For any c ∈ C , we call integert the number of effective dimensions of c , if 1) there exists a function g : C → R t , such thatg ( c ) =
ϕ ( x ) t , where ϕ ( x ) t ∈ R t , and 2) there doesn’t exist such a function g for t + 1 .
Further, let
C t ⊂ C be the subset for all c with effective dimensions of t .
Deﬁnition 4 shows that any c ∈ C t can reconstruct at most t ≤ r dimensions of the manifold.
Also, we extend our deﬁnition of active latent dimensions to CVAE.
Deﬁnition 5 (Active Latent Dimensions in CVAE)
In a κ -simple CVAE, let γ 0 be an arbitrarilysmall positive value and satisﬁes γ 0 ≤ γ . Denote σ z p ( x, c ) and σ z q ( x, c ) as the variance of theprior and encoder.
For any j = 1 , . . .
, κ , a dimension j of latent variable z is deﬁned as an activedimension if the ratio σ z q ( x, c ) {+
+} j /σ z p ( x, c ) 2 j = O ( f ( γ 0 )) as γ 0 → 0 , where f is any function thatgoes to zero when γ 0 goes to zero.
As special cases, for VAE or CVAE with a standard Gaussian prior we have σ z p ( x ) 2 j = 1 , and thatindicates Deﬁnition 2 is a special case of Deﬁnition 5.
Next, we extend Theorem.1 to CVAE models, which describes how the manifold dimension relates toconditioning variables.
Theorem 2 (Learning Manifold Dimension in CVAE)
Suppose c ∈ C t and r < d . Let { θ ∗ γ 0 , φ ∗ γ 0 }denote the global optimal κ -simple CVAE solution under γ = γ 0 .
Then for any κ ≥ r , when aκ -simple CVAE model achieves its global optimum, we have
[Equation]
can be uniquely achieved(
[Equation]
The number of active latent dimensions
almost surely equals r − t , and(
iii)
The variance of each active dimension satisﬁes σ z q x c [-; φ ∗ γ ) 
-][Equation]
/σ [Equation] p [-( c ; θ ∗ γ ) 
-][Equation]
x , c ) [Equation] = O (
γ
This theorem addresses issue (ii) in Section 1.
It shows the signiﬁcance of conditioning variable thatit can push the cost to a smaller value than a VAE model without condition, which is favorable.
It alsopoints out that the condition will reduce the number of active latent dimensions without disruptingthe data ﬁt.
It worth mentioning that even though the data is not on the manifold initially, once aconditioning variable with a positive effective dimension is incorporated, the conditioned data thenbecome on a manifold, which can be induced directly in our theorem.
Moreover, to address issue (iii) in Section 1, we have the following results describing CVAE’scapacity in learning data lies on a union of manifolds.
Corollary 2.1 (Adaptive Active Dimension) Suppose there exists conditioning variables c 1 ∈ C t 1and c 2 ∈ C t 2 , i.e. c 1 , c 2 have t 1 and t 2 effective dimensions respectively.
Given a CVAE model withoptimal solution { θ ∗ , φ ∗ } , we have the number of active dimensions of z 1 ∼ q φ ∗ ( z | x, c 1 ) equals tot 1 , and that of z 2 ∼ q φ ∗ ( z | x, c 2 ) equals to t 2 .
This result indicates that CVAE can adaptively learn different data manifolds under conditioningvariables with different effective dimensions.
When c is discrete, it is equivalent to training separate
VAEs where each model corresponds to a discrete condition value.
On the other hand, collecting allpossible conditions in a single CVAE model enables it to compress the dimension dynamically.
Forexample, c describes the set of manifolds that share the same dimension, thus for samples in each set,the corresponding manifold dimensions are the same.
When c is continuous, conditioning variablesmay contain intact information of certain manifold dimensions, which substitute for correspondingactive dimensions.
In Section 4, we demonstrate numerical evidence of this behavior.
On Common Model Design Choices
In this section, we review some model designs and training practices that are commonly seen inprior works but could potentially negatively impact the model convergence and performance.
Thefollowing discussions can be seen as important steps towards resolving issue (iv) in Section 1.
3.1 On the Equivalence of Conditioned and Unconditioned Prior
In prior works, it is popular to design a parameterized prior p θ ( z | c ) in CVAE [25, 8, 9, 21, 1], whichis a trainable module alongside the encoder q θ ( z | x, c ) and the decoder p θ ( x | z, c ) .
Remark 1 (Remove conditioning variable from the Prior) Consider a κ -simple CVAE modelwith prior p
θ ( z | c ) , encoder q φ ( z | x, c ) and decoder p θ ( x | z, c ) .
We can always ﬁnd another κsimple CVAE model with prior p ( z ) ∼ N (0 , I ) , encoder q φ (cid:48)
( z | x, c ) and decoder p θ (cid:48) ( x | z, c ) , suchthat L c ( θ, φ ) =
L c ( θ (cid:48) , φ (cid:48) ) .
Remark 1 indicates that a conditioned prior is not necessary.
Speciﬁcally, as shown in its proof, we canalways explicitly convert an existing κ -simple CVAE model with prior p θ ( z | c ) into another κ -simple
CVAE model with prior being N (0 , I ) .
In practice, when we have the ﬂexibility in architecturedesign, it is equivalent to use a standard Gaussian prior or a parameterized one.
3.2 The Initial γ Impacts Model Convergence and Dimension Learning
Recall in Theorem 1, we have L ( θ ∗ γ , φ ∗ γ )
= (cid:82) X ( d − r ) log γ + O (1) ω gt ( dx ) , and that implies when L ( θ ∗ γ , φ ∗ γ ) approaches to −∞ , γ is expected to be closer to 0.
This behavior makes γ important in CVAE training.
A direct conclusion is that the model would be compromised if γ is set to be a ﬁxedpositive constant, as analysed and empirically studied in literature [4, 11, 16, 19, 3].
Although a model converging towards global optimum would push γ to 0 regardless of its initial value, in practice, we observe that this initial value of γ still signiﬁcantly impacts the model performance.
In the cost function, we have
[Equation]
By this formulation we see that γ acts as the weight between data likelihood and KL-divergence.
A large γ downgrades the weight for data likelihood, and encourages the model to optimize the
KL-divergence term ﬁrst, and vice versa.
Making γ a learnable parameter is equivalent to adaptivelybalancing the cost.
When the initial value is set to be on an extreme end, the model is at risk of fallinginto a local minimum, as empirically in Section 4.4.
3.3 Weight Sharing Keeps Model from Global Optimal Convergence
The cost of VAE balances the three modules with the two terms, the data likelihood and KL-divergence.
It has been argued in literature [3] that such a cost might lead to unstable training dynamics.
Therefore,one widely adapted trick[18] in practice is to share model weights between prior and posterior.
Weight sharing is conducted in various forms.
For data from a general domain where x and c aredifferently distributed, the weight sharing couples with the design in which the posterior takes in xand c as two input and process differently at the beginning, and the prior processes c as its input withthe same module and weight in the posterior.
In particular, sequential data modeling takes a more thorough approach of weight sharing.
Sequentialdata poses the task of using early parts of a sequence to predict the later parts.
Formulating in CVAE,we have c being the early parts, and that conditions the to-predict later parts x .
Assuming x is a sequential data, i.e. { x l } with l > 0 , and each x l is paired with its condition c = x <l .
As a result, the encoder becomes q φ ( z l | x ≤ l ) , and the prior becomes p ( z l | x <l ) for any l greater thanthe observed sequence length.
It has been proposed in literature [8, 9] to share weights between theencoder and prior, i.e. p ( z l | x <l ) =
q φ ( z l | x <l ) .
Theorem 3 (Weight Sharing Compromises Performance of Sequential Modeling)
By sharingweights between the encoder and prior, the conditional VAE model has its cost being
[Equation]
Then we have L c ( θ, φ ) = {+(cid:82) X Ω(
 ) ω gt ( dx )+} for any θ and φ .
Theorem 3 implies that a fully weight-sharing pair of prior and encoder prevents the model fromachieving its global optimum.
As the numerical study in Section 4.5, this weight sharing strategycould severely constraint the performance.
Experiments
In this section we present numerical results to support our analysis.
We ﬁrst verify model behavior inthe synthetic environment for a controllable set of experiments, and extend to real-world datasets tofurther validate our conclusions.
4.1 Dataset and Metrics
Synthetic Dataset
The basic data samples are generated from a mixture of a ﬁnite number of Gaussian distributions.
Speciﬁcally, we ﬁrst generate a 5-dimensional categorical selecting vector from [Table] uniform distribution, and normalize it with norm equals one.
Then we generate two r -dimensionalparameter vectors µ u and log σ u from a standard Gaussian distribution in R r .
By repeating thesecond step 5 times, we get 5 independent Gaussian distributions.
Our synthetic data u ∈ R r isgenerated by selecting one of the 5 candidate distributions with a probability of the selection vector,and next generate samples from the selected distribution.
To get the manifold X in R d , we design afunction h :
R r → R d , and the data sample x = h ( u ) .
In our experiments, h ( u ) =
Sigmoid ( G x u ) ,where G x ∈ R d × r is randomly initialized controlled by a certain random seed.
The conditioningvariable c is transformed by another transform h (cid:48)
( u 1: t ) =
G (cid:48)
u 1: t , where G (cid:48) ∈ R t × t .
Given a ﬁxedseed, the generation results is controlled by parameters r , d , t and κ .
Real-world dataset Besides synthetic dataset, we also investigate the model behavior on
MNIST[24] and Fashion MNIST [23], two image datasets 1 .
These two datasets contain x withcomplex mappings between the manifold and ambient space.
At the same time, they are not toocomplex thus the training are made reasonably easy with common neural architecture, otherwise ourfocus could be driven to speciﬁc architecture choices other than VAE and CVAE model behavior.
Bycarefully tuning the pair ( x, c ) , we can control how c impacts the model.
Metrics The main metric that we use to evaluate VAEs and CVAEs is the Negative Log-Likelihood(NLL), − log p ( x ) , which is bounded by Evidence Lower BOund (ELBO), as in Eq 1 and Eq 2.
Inthe tables we compute ELBO and report them under the name NLL.
There are also auxiliary metricsthat we use to compare and diagnose models, including Active Dimensions (AD), Reconstructionerror E || x − µ x ( z ) || 2 (Recon), the KL-divergence ( KL ), and the data variance γ .
4.2 Manifold Dimensions Learning in VAE
Here, we provide direct numerical support to Theorem 1.
Speciﬁcally, we show that the modelcaptures the correct dimensions of manifold regardless of the ambient data dimensions.
[Table]
[Table]
With d = κ = 20 and varying r [Table] we demonstrate that the VAE model can learn the active latentdimensions when γ converges to a small value, regardless of the value of r .
The results are showed in
Table 1 and all the numbers are values at convergence.
When the latent dimension κ is no less thanthe intrinsic manifold dimension, the number of active dimensions is exactly r and a small NLL isachievable.
Experimental results on MNIST and Fashion MNIST are reported in Table 2.
Without knowing thetrue manifold dimensions, a small latent dimension discourages the model from catching up theinformation for every dimension, while once given enough latent dimensions, the number of activedimensions equals to data’s intrinsic dimension and the rest is for the regularization.
The two parts oflatent dimensions cooperate to push the VAE model to its global optimum.
4.3 CVAE Adaptive Active Dimensions Varied With c
In this section, we design experiments to show that a CVAE model can learn the dimension ofmanifold with respect to the effective latent dimensions of the condition.
Further, it can also learn theadaptive active dimension on a union of manifolds.
By Theorem 2 and Remark 1, a CVAE model with an unconditioned prior and condition c ∈ C t isequivalent to a general one in design, whose active dimensions equals r − t .
In this experiments, weinvestigate the behavior of a CVAE model when data lies on a single manifold and its relation tocondition is static.
Speciﬁcally, we set c to be the ﬁrst t dimensions of u , and let d = κ = 20 , r = 10 .
Table 3 shows the results.
On MNIST and Fashion MNIST, we train CVAEs with the class label of each image as the conditioning variable, and the results with κ = 32 are in Table 4.
By comparing it to Table 2, we notice that the
CVAE model for MNIST has a lower active dimension and better NLL, while for FashionMNIST the CVAE performs similarly.
This implies that when data points in MNIST are conditioning on the label,they lie on a lower-dimensional manifold, while for data points in the more complex FashionMNISTthe class label doesn’t reduce the manifold dimension.
This characterization of the dataset complexityis consistent with their intuitive visual complexity.
[Table] [Table]
[Table] [Table]
Discrete Condition To verify Corollary 2.1, we train a CVAE model on d -dimensional data thatlie on a union of manifolds with different dimensions, with the conditioning variable c being anindicator of the source manifold.
Table 5 shows that the model successfully captures the active latentdimensions with an attention layer in the decoder, while without the attention layer the model learnsa static active dimension of 4.
The result indicates two aspects of CVAE: 1) CVAE models havecapacity to capture a union of manifolds simultaneously and converge on each manifold (indicatedby NLL); 2) In practice, we need careful model architecture design to help the model to distinguishmanifolds.
Continuous Condition When c is a continuous variable, we show that the model can also adaptivelylearn the active dimensions.
In our setting, a continuous condition c ∈ C t contains t -dimensionalinformation of samples, thus under different conditions the data may lie on different manifolds.
Weset r = 12 , d = 20 and κ = 90 , then we set the ﬁrst 2 , 4 , 6 , 8 , 10 dimensions of u as the conditions.
Since the model needs dimension of c to be consistent, we padding them to R 10 with 0.
With theattention layer, the model can use the condition to collapse certain dimensions instead of activatingall the possible dimensions and increasing unnecessary loss in KL term.
The results are included in
Table 6.
4.4 Initialization of γ Impacts Model Training
Here we check how the initialization of γ would impact model convergence, as mentioned in
Section 3.2.
Our results are in Table 7.
By initializing log γ to different values, we observe theconvergence of VAE is affected signiﬁcantly, as a result, the model can’t capture the correct manifolddimension.
In comparison, CVAE models are less sensitive, as CVAEs have more parameters and theconditioned data lie on a lower-dimension manifold thus it actually poses an easier task.
Still, whenlog γ = 20 , they converge to local optima.
4.5 Shared Weights Between Encoder and Prior
The experiments in this section is to support Theorem 3, i.e. sharing weights between the encoderand prior is not a satisfactory way for training in sequential data, we generate a sequence by
AutoRegressive-Moving-Average (ARMA) model with ﬁxed parameters, then use rolling windows tomake our samples.
Speciﬁcally, the window size is 5, i.e. each sample we use the past 5 points as ourcondition, and reconstruct the sixth one which is viewed as current sample.
The results in Table 8show that when weight shared, the norm term of reconstruction cannot go down, which may result ina restriction in the capacity of the model, and this observation is coherent to the key contradictionused in our proof of Theorem 3. [Table] [Table]
Conclusion
In this paper we provide insights on the behavior of VAEs and CVAEs for data lying on lowdimensional manifolds.
In the theoretic analysis, we show that VAE has the ability to learn themanifold dimension.
More interestingly, we extend the conclusion to Conditional VAE that workswith conditioned data.
This additional conditioning term impacts the relation between data andmanifold, as a result it also affects the model design choices in practice.
With the theoretic results,we further examine some design choices.
A dedicated section of experiments supports our statementsand conclusions numerically on both synthetic and real-world data.
Finally, we believe this work canin general help to improve the design and diagnose techniques of VAEs and CVAEs.
Limitations Our work may still have limitations in empirical veriﬁcation for real-world datasetsat a larger scale or from other modalities than images.
Although our contributions are theoreticallygrounded, in practice it still takes non-trivial efforts to design model architectures that both align withour guideline and ﬁt the data.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[No] (d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] See Appendix.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] See Section and Appendix.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[No]
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See Appendix.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes] See Section
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[No]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
No
If you crowdsourcing or conducted research with human subjects.
(a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]