Improving Variational Autoencoders with Density Gap-based Regularization
Anonymous Author(s)
AffiliationAddress email
Abstract
Variational autoencoders (VAEs) are one of the powerful unsupervised learningframeworks in NLP for latent representation learning and latent-directed generation.
The classic optimization goal of VAEs is to maximize the Evidence Lower
Bound (ELBo), which consists of a conditional likelihood for generation and a negative Kullback-Leibler (KL) divergence for regularization.
In practice, optimizing
ELBo often leads the posterior distribution of all samples converge to the samedegenerated local optimum, namely posterior collapse or KL vanishing .
There areeffective ways proposed to prevent posterior collapse in VAEs, but we observe thatthey in essence make trade-offs between posterior collapse and hole problem , i.e.,mismatch between the aggregated posterior distribution and the prior distribution.
To this end, we introduce new training objectives to tackle both two problemsthrough a novel regularization based on the probabilistic density gap between theaggregated posterior distribution and the prior distribution.
Through experimentson language modeling, latent space visualization and interpolation, we show thatour proposed method can solve both problems effectively and thus outperforms theexisting methods in latent-directed generation.
To the best of our knowledge, weare the first to jointly solve the hole problem and the posterior collapse.
Introduction
As one of the powerful likelihood-based generative models, variational autoencoders (VAEs) [22, 32]are designed for probabilistic modeling directed by continuous latent variables, which is successfullyapplied in many NLP tasks, e.g., paraphrase generation [8, 20], dialogue generation [45, 14], machinetranslation [34, 12] and data augmentation [43, 39].
One of the major advantages of VAEs is theflexible latent representation space which enables easy manipulation of high-level semantics oncorresponding representations, e.g., guided sentence generation with interpretable vector operators.
Despite the attractive theoretical strengths, VAEs are observed to suffer from a well-known problemnamed posterior collapse or KL vanishing [22], an optimum state of VAEs when the posteriordistribution contains little information about the corresponding datapoint, which is particularlyobvious when strong auto-regressive decoders are implemented [46, 4].
Another challenge for VAEs is the hole problem , the state when the aggregated (approximate) posteriorfails to fit the prior distribution, and thus the inference from prior becomes no longer suitable todescribe the global data distribution [33], which can lead to poor generation quality in VAEs [1, 25].
In this work, we take systematic experiments on VAEs for text generation to study the posteriorcollapse and the hole problem in existing methods.
We demonstrate that VAEs with specific networkstructures [10, 45] or modified training strategies [6, 13] have limited effect on solving posteriorcollapse, while VAEs with hard restrictions [9, 41, 46] or weakened KL regularization [18, 21] on
Meanwhile
BN can solve posterior collapse effectively at the cost of bringing hole problem , i.e., mismatch between the aggregated posterior and the
Our proposed DG-VAE intends to solve both problems through a novel regularization based on density gap .
Complete illustrations are shown in Appendix G. the posterior can solve posterior collapse effectively at the expense of introducing hole problem, asillustrated in Figure 1.
On that basis, we hypothesize that those two problems stem from the conflict between the KL regularization in ELBo and the function definition of the prior.
As such we propose a novel regularizationto substitute the KL regularization in ELBo for VAEs, which is based on the density gap betweenthe aggregated posterior and the prior.
We provide theoretical proof that our method in essencemaximizes the ELBo as well as the mutual information between the input and the latent variable.
In terms of Gaussian distribution-based VAEs, we further propose the corresponding marginalregularization on each dimension respectively, and we prove it in essence maximizes the ELBo aswell as the sum of mutual information between the input and the latent variable on all dimensions.
To validate our methods in practice, we take experiments on language modeling, latent-guidedgeneration and latent space visualization.
We demonstrate that our methods form latent spaces bothactive and consistent with the prior, and thus generate smoother sentences from latent interpolation.
Background and Related Work2.1 VAEs and ELBo
VAEs are proposed to perform efficient inference and learning in directed probabilistic models, wherethe random generation process consists of two steps: (1) sample a latent value z from the prior p θ ( z ) ;(2) generate a datapoint x from the conditional distribution p θ ( x | z ) .
As the true posterior p θ ( z | x ) isintractable, a recognition model q ϕ ( z | x ) is introduced to approximate the true posterior [22].
Specifically, VAEs represent an observation x as a latent distribution q ϕ ( z | x ) , from which latentvariables are sampled to direct the reconstruction of x .
As the optimization goal of VAEs, the Evidence
Lower Bound (ELBo) is composed of a Kullback-Leibler (KL) divergence for regularization onposterior and a log likelihood for reconstruction conditioned on posterior.
The ELBo in fact forms alower bound on the marginal likelihood given prior latent variables [22], as illustrated in Eq.
[Equation]
2.2 The Aggregated Posterior Distribution
The aggregated (approximate) posterior q ϕ ( z ) refers to the expectation of approximate posteriordistribution on data distribution, as defined in Eq.
2, where the distribution of observation x is represented by the discrete distribution of datapoints in the dataset X = { x n } Nn =1 , q ϕ ( n ) ≡ 1 N .
[Equation]
2.3 Posterior Collapse
Intuitively, the KL divergence in ELBo (i.e., D KL ( q ϕ ( z | x ) ∥
p θ ( z )) ) encourages the approximateposterior of every single datapoint to be close to the the prior [22].
This intends to ensure the priorcan depict the latent distribution over data distribution, but it can also lead to posterior collapse when
D KL ( q ϕ ( z | x ) ∥ p θ ( z )) has much stronger force on q ϕ
( z | x ) than E q ϕ
( z | x ) [log p θ ( x | z )] does, whichleads that q ϕ ( z | x ) ≈
p θ ( z ) ∀ x , the sampled latent variable z ∼ q ϕ ( z | x ) contains much more noisethan useful information about x [27], and thus the decoder p θ ( x | z ) becomes insensitive to z [46].
Early works to solve posterior collapse attribute it to the difficulty in optimizing ELBo, so theirmethods mainly focus on the training strategies [6, 13, 24].
Some works put emphasis on semanticlearning of z through specific model structures [10, 45, 29, 17].
Instead of Gaussian distribution, vMFVAE [9, 41] adopts von Mises-Fisher distribution for latent variables, which restricts the posteriorlatent in a hyperspherical space and forms a constant KL divergence.
BN-VAE [46] restricts theposterior distribution through a batch normalization layer with a fixed scale γ , so as to guarantee apositive lower bound of the KL divergence.
β -VAE [18] directly changes the weight (denoted as β )of KL term inside ELBo, while free-bits [21] changes the KL term inside ELBo to a hinge loss term.
In contrast, we hypothesize the posterior collapse is due to the conflict between the KL regularizationin ELBo and the function definition of the prior, and tackle it through replacing the KL regularizationin ELBo with a novel regularization on the aggregated posterior.
2.4 Hole Problem
Formally, the hole problem refers to the phenomenon that the aggregated posterior distributionq ϕ ( z ) fails to fit into the prior distribution p θ ( z ) .
Inferences located in the holes (i.e., areas withmismatch between density in q ϕ ( z ) and p θ ( z ) ) are observed to generate images that are obscure andcorrupted [1], or sentences with problematic word matching and content against commonsense [25].
The hole problem of VAEs for image generation is observed in several studies, commonly ascribed tothe limited expressivity of the prior and tackled by increasing the flexibility of the prior via hierarchical priors [23], auto-regressive models [16], a mixture of encoders [37], normalizing flows [40],resampled priors [3] and energy-based models [1].
In contrast, we observe that the vanilla VAEs(with static prior) for text generation has no hole problem, but it arises when existing methods areapplied to solve posterior collapse.
Therefore our work is targeted at solving posterior collapse andavoiding hole problem at the same time, for VAEs with static prior.
2.5 Regularization on Aggregated Posterior
As the approximate posterior is introduced to approximate the true posterior, i.e., q ϕ ( z | x ) ≈ p θ ( z | x ) ,the aggregated posterior should be close to the prior as a result, i.e., q ϕ ( z ) ≈
p θ ( z ) .
From this pointof view, several works are proposed to replace KL regularization (on posterior for single datapoint) in
ELBo with regularization on the aggregated posterior, which can be summarized as Eq. 3, where Dis the divergence (or discrepancy) between two distributions.
[Equation]
Among them, Adversarial Auto-Encoder (AAE) [28] adopts the Generative Adversarial Network(GAN) [15] framework to regularize the aggregated posterior through Jensen–Shannon divergence
D JS ; Wasserstein Auto-Encoder (WAE) [36, 2] regularizes the aggregated posterior through minimizing Maximum Mean Discrepancy (MMD);
Implicit VAE with Mutual Information regularization(iVAE MI ) regularizes the aggregated posterior through the dual form of KL divergence D KL on thebasis of Implicit VAE (iVAE)
Those methods have a same weakness that their approximationsof the divergence between two continuous distributions are depicted by merely sampling sets fromthe distributions, which involves noise from random sampling and can hardly be zero even for the same distribution.
In contrast, our method approximates the divergence between two continuousdistributions in the perspective of their mismatch in PDFs, which we quantify through density gapthat can be zero if and only if they are the same, which we describe in section 3.
We validate our method in comparison with those methods through experiments on a synthetic dataset,the details and results of which are demonstrated in Appendix A.
Methodology
Density Gap-based Discrepancy One of the straight manifestations of holes in latent space is themismatch of probabilistic density between q ϕ ( z ) and p θ ( z ) , we quantify this mismatch at a specificposition z in the latent space through DG ( θ , ϕ ; z ) , which we refer to as Density Gap .
Here we onlyconsider z ∈ { z | q ϕ
( z ) > 0 } .
We assume q ϕ ( z ) and p θ ( z ) are differentiable and p θ ( z ) > 0 .
[Equation]
It can be inferred that the expectation of DG ( θ , ϕ ; z ) on q ϕ ( z ) equals to the KL divergence betweenq ϕ
( z ) and p θ ( z ) , as illustrated in Eq.
5, which is a strict divergence
, i.e., E z ∼ q ϕ
( z ) [ DG ( θ , ϕ ; z )]
=
iff q ϕ
( z ) =
p θ ( z ) .
[Equation]
So, we can approximate and optimize D KL ( q ϕ ( z ) ∥
p θ ( z )) via Monte Carlo, as illustrated in Eq.
6,where
z s idd ∼ q ϕ ( z ) denotes the s th random sample from the aggregated posterior distribution.
[Equation]
It should be noticed that D KL ( q ϕ ( z ) ∥
p θ ( z )) approximated by this is an overall divergence,as it considers the posterior distribution of all datapoints as a whole, instead of averaging
D KL ( q ϕ ( z | x ) ∥ p θ ( z )) across all datapoints as ELBo does.
On that basis, we can implement L D ( θ , ϕ ; x ) with D = D KL , which is equivalent to replacing the
KL term in ELBo with D KL ( q ϕ ( z ) ∥ p θ ( z )) approximated by Eq. 6.
According to the decomposition(illustrated in Eq. 7) of KL term in ELBo given by Hoffman et al.
[19], maximizing L D KL ( θ , ϕ ; x n )on the whole dataset is equivalent to maximizing ELBo as well as I q ϕ ( n, z )
[ n, z ] , the mutual information of z and n in their joint distribution q ϕ ( n, z ) , as illustrated in Eq. 8.
[Equation]
where I q ϕ ( n, z )
[ n, z ] =
E q ϕ ( n
[Equation]
z ) [log q ϕ ( n
[Equation]
z
[Equation]
q ϕ
[Equation]
q ϕ
[Equation]
z
[Equation] [Equation]
q ϕ
[Equation]
n
[Equation] [Equation]
Optimization on Mini-Batch Theoretically attractive as it is, maximizing L D KL ( θ , ϕ ; x n ) approximated by DG ( θ , ϕ ; z s ) is undesirable for training VAEs on large datasets, because the probabilisticdensity of q ϕ ( z ) at z needs computation across the whole dataset and is changing in every training step.
In practice, training deep networks such like VAEs commonly adopts mini-batch gradientdescent, where only a small subset of the dataset is used for calculating gradients and updatingparameters in each iteration step.
So, a practicable way is to aggregate the posterior of datapoints inside a mini-batch B = { x n } | B | n =1 ,as stated in Eq.
9, where z n,m is the m th sample from the posterior of datapoint x n .
Here stratifiedsampling (i.e., sampling S =
| B | × M samples from q ϕ ,B ( z ) through sampling M samples fromq ϕ
( z | x n ) for each datapoint x n ∈ B ) is used to ensure a steady Monte Carlo approximation, and thereparameterization trick [22] is applied to ensure a differentiable output.
[Equation]
n
[Equation]
M M
[Equation]
m
[Equation]
DG ( θ , ϕ , B ; z n,m ) , where z n,m idd ∼ q ϕ ( z | x n ) (9)
Through this approximation, we can implement L D KL ( θ , ϕ , B ; x n ) that regularizes q ϕ ,B ( z ) towardsp θ ( z ) for a mini-batch B , as stated in Eq. 10.
[Equation]
It should be noticed that the mutual information term I q ϕ ( n, z )
[ n, z ] is different in Eq.
8 (on the wholedataset) and Eq.
10 (on a mini-batch), because the range of discrete variable n is from 1 to N in Eq.
8but 1 to | B | in Eq.
Consequently, I q ϕ ( n, z ) [ n, z ] has a upper bound of H ( n )
= log N in Eq.
8 but
H ( n ) = log | B | in Eq.
In other words, maximizing I q ϕ ( n, z ) [ n, z ] in Eq.
8 intends to distinguishz of x n from that of N − 1 other datapoints while it is limited to | B | − 1 other datapoints in Eq.
Marginal Regularization for More Mutual Information As described above, approximatingand optimizing D KL ( q ϕ ,B ( z ) ∥
p θ ( z )) is practicable but has limited effect.
Empirically, Gaussiandistribution-based VAEs trained by Eq.
10 still have limited active units, which means the encodedlatent variable z still collapses to the prior on most dimensions, where it provides little information.
To activate z on all dimensions, we propose to regularize q ϕ ,B ( z ) towards p θ ( z ) on each dimensionrespectively, i.e., regularize the marginal distribution on each dimension, as illustrated in Eq.
11,where
z i ∈ R is the i th component of z ∈ R Dim and the corresponding probability density functionsare of the marginal distributions on the i th dimension; z n,m,i is the i th component of the m th samplefrom the posterior of datapoint x n .
[Equation]
M
[Equation]
In Gaussian distribution-based VAEs, the marginal distributions of z on different dimensions areindependent, i.e., q ϕ ( z | x n ) = (cid:81) Dimi q ϕ ( z i | x n ) and p θ ( z ) = (cid:81)
Dimi p θ ( z i ) , so their KL divergencecan be decomposed as D KL ( q ϕ ( z | x n ) ∥
p θ ( z ))
= (cid:80) Dimi D KL ( q ϕ ( z i | x n ) ∥
p θ ( z i )) .
Thus, we can infer the decomposition of D KL,mrg ( q ϕ ,B ( z ) ∥
p θ ( z )) through Eq.
[Equation]
So, maximizing L D KL,mrg ( θ , ϕ , B ; x n ) derived from Eq.
11 is equivalent to maximizing ELBo aswell as the mutual information of n and z i for each dimension i respectively, as stated in Eq. 13.
Inother words, it intends to distinguish z i of x n from that of | B |− 1 other datapoints for each dimensioni respectively.
We refer to our models based on this Density Gap-based regularization as DG-VAEs.
[Equation]
) [ n,
[Equation]
Aggregation Size for Ablation As discussed above, the size of mini-batch | B | sets a upperbound of the mutual information term
I q ϕ ( n, z ) [ n, z ] (or I q ϕ ( n, z i ) [ n, z i ] ).
To validate this impact,we further extend DG-VAEs through dividing the mini-batch into non-overlapping subsets B = 168 (cid:83)
Ci =1 b i , s.t.
b j ∩ b i = ∅ iff
i ̸
=
j , calculating and optimizing the KL divergence over each subset, i.e.,1 C (cid:80) Ci =1
| b i | (cid:80) | b i | j =
[ L D KL,mrg ( θ , ϕ , b i ; x n )] , where C denotes the number of subsets and | b i | =
| B | Cis the size of those subsets, which we refer to as the aggregation size and denote as | b | for simplification.
It can be inferred that the DG-VAE with | b | = 1 is equivalent to the vanilla VAE trained by ELBo,except that it approximates the KL term through Monte Carlo instead of integration.
Extension to von Mises-Fisher Distribution-based VAEs Besides the commonly used Gaussiandistribution-based VAEs, we also consider von Mises-Fisher (vMF) distribution-based VAEs.
Asthe decomposition q ϕ ( z )
= (cid:81) Dimi q ϕ
( z i ) is not established for latent variables following vMFdistribution (i.e., z ∼ vMF ( µ , κ ) ), marginal regularization for vMF-VAEs may not be interpretable,so we only implement Eq.
10 in vMF-VAEs.
We refer to those extensions as DG-vMF-VAEs.
Experiments
4.1 Experimental Setup
Datasets We consider four public available datasets commonly used for VAE-based languagemodeling task in our experiments: Yelp [42], Yahoo [42, 44], a downsampled version of Yelp [35](we denote this as Short-Yelp) and a downsampled version of SNLI [5, 24].
The statistics of thosedatasets are illustrated in Table 1.
It can be viewed that Yelp and Yahoo contain long sentences while
Short-Yelp and SNLI contain short sentences.
Baselines We consider a wide range of VAEs for solving posterior collapse in text generation, wherethe hyperparameters are set according to Zhu et al.
• VAEs with modified training strategies (i
e., KL annealing): VAE with linear KL annealingin the first 10 epochs (default) [6]; VAE with linear KL annealing for 10 epochs at the startof every 20 epochs (cyclic-VAE) [13];
• VAEs with specific model structures: VAE with additional Bag-of-Words loss (bowVAE) [45], and VAE with skip connection from the latent variable z to the vocabulary
classifier for generation (skip-VAE) [10];
[Table] [Table]
• VAEs with hard restrictions on the posterior distribution: δ -VAE with the committed rateδ = 0 .
15 [31]; BN-VAEs with the scale of BN layer γ ∈ { 0 . 6 , 0 .
[Table]
-VAEs with the distribution’s concentration κ ∈ { 13 , 25 , 50 , 100 , 200 } [9, 41];• VAEs with weakened KL regularization: FB-VAEs (free-bits) with the target KLλ KL ∈ { 4 , 9 , 16 , 25 , 36 , 49 } [21]; β -VAEs with the weight of KL term β ∈{ 0 . 0 , 0 . 1 , 0 . 2 , 0 . 4 , 0 . 8 } [18].
Configurations We completely follow Zhu et al.
[46] in models’ backbone structures, data preprocessing and training procedure, which we describe in details in Appendix B.
4.2 Language Modeling
We evaluate the performance of our methods and the baselines on language modeling, where thefollowing metrics are reported: the prior log likelihood priorLL ( θ ) and the posterior log likelihoodpostLL ( θ , ϕ ) for generation quality; the KL term in ELBo KL ( ϕ ) , the mutual information MI ( ϕ )of z and n and the number of active units AU ( ϕ ) [7] for posterior collapse; the number of consistentunits CU ( ϕ ) (we propose) for hole problem.
The corresponding expressions and explanations are inthe Appendix C.
We illustrate part of the results on Yahoo in Table 2 and all results on all datasets in Appendix D. Itcan be observed that: (1) models with modified training strategies or specific model structures canalleviate the problem of posterior collapse but has limited effect according to MI ( ϕ ) and AU ( ϕ ) ;(2) models with hard restrictions or weakened KL regularization on the posterior can solve posteriorcollapse better through harder restrictions or further weakening according to the increase of KL ( ϕ ) ,
MI ( ϕ ) and AU ( ϕ ) , but the decrease of CU ( ϕ ) indicates that their posterior latent spaces tend to beincreasingly inconsistent with that of prior; (3) in contrast, our proposed DG-VAE has the similar performance to the vanilla VAE when | b | = 1 , and with the increase of | b | , it can solve the posteriorcollapse effectively and avoid the hole problem at the same time.
It can also be viewed that with the increase of MI ( ϕ ) or KL ( ϕ ) , postLL ( θ , ϕ ) tends to increasewhile priorLL ( θ ) tends to decrease, as the decoder θ becomes more dependent on the encoder ϕ .
Wefurther plot the curves of priorLL ( θ ) and postLL ( θ , ϕ ) for models with different hyperparametersin Figure 2, where we can observe that DG-VAEs make better trade-offs than BN-VAEs and β -VAEsdo on short datasets and perform competitively to BN-VAEs and FB-VAEs on long datasets.
We also compare the performances of DG-vMF-VAEs with vMF-VAEs under different settings ofκ .
As they have the same KL ( ϕ ) while AU ( ϕ ) and CU ( ϕ ) are inappropriate to report for vMFdistributions, we only plot their curves of priorLL ( θ ) and postLL ( θ , ϕ ) in Figure 3.
It can beobserved that DG-vMF-VAEs outperform vMF-VAEs in most cases.
4.3 Visualization of the Posterior
To further investigate the posterior in latent space of those models, we visualize the aggregatedposterior distribution and the posterior centers distribution on the 2 most active dimensions, i.e., thetwo dimensions with the highest V ar x ∼ X
[ E q ϕ ( z | x ) [ z ]] , as depicted in Figure 4.
set.
Illustrations for more datasets, more models and more dimensions are shown in Appendix G.
Here we can observe that BN-VAEs, FB-VAEs and β -VAEs can better solve the posterior collapsewith harder restrictions or further weakening, but meanwhile they are faced with different kind of mismatch between the aggregated posterior distribution and the prior distribution, i.e., hole problem.
In contrast, with the increase of aggregation size | b | , DG-VAE can better solve posterior collapse andavoid hole problem at the meantime.
4.4 Interpolation Study
One of the main advantages of VAEs over normal language models (e.g., GPT-2[30]) is that VAEsembed datapoints into a continuous latent space and enables latent-guided generation.
We evaluatethis ability through interpolation, where the models encode two sentences x a and x b as their posteriorcenters, i.e., z a = E q ϕ
( z | x a )
[ z ] and z b = E q ϕ ( z | x b )
[ z ] , and decode the variables between them, i.e.,z λ =
z a · (1 − λ )
+ z b · λ, λ ∈ { 0 . 0 , 0 . 1 , . .
The interpolated sentences are wished to besemantically smooth and meaningful, which we evaluate through the average Rouge-L F1-score [26],as stated in Eq.
14, where F lcs denotes the F1-score of Longest Common Subsequence (LCS).
Weplot the curves of Rouge-L F1-score and λ for models on Yahoo dataset in Figure 5 and those curveson other datasets in the Appendix E.
[Equation]
As shown in Figure 5, the average F1-score of LCS tend to be lower in the middle than at the ends,which indicates that generated sentences tend to be not smooth at the middle, which correspondsto the phenomenon of generation near to holes observed in previous work [25].
The vanilla VAEperforms the worst as it suffers from posterior collapse, and only generates the same plain sentence;meanwhile, DG-VAE outperforms BN-VAEs, FB-VAEs and β -VAEs in the quality of interpolationon Yahoo dataset as it can solve posterior collapse and avoid hole problem at the same time.
In summary, the existing methods for solving posterior collapse in VAEs either have limited effect,or can effectively solve posterior collapse at the cost of introducing hole problem.
In contrast, ourproposed DG-VAE can effectively solve posterior collapse and avoid hole problem at the same time,which is demonstrated by the posterior centers spread in latent space and the aggregated posterior inconsistent with the prior.
Furthermore, our proposed DG-VAE outperforms the existing methods inthe quality of latent-guided generation due to these improvements in latent space.
Discussion
Conclusion In this work, we take systematic experiments to demonstrate the posterior collapse andhole problem in existing continuous VAEs for text generation.
To solve both problems at the sametime, we propose a density gap-based regularization on the aggregated posterior to replace the KLregularization in ELBo, and prove it in essence maximizes the ELBo as well as the mutual informationbetween the latent and the input.
Experiments on real-world datasets prove the effectiveness of ourmethod on solving both problems and its improvement on latent-guided generation.
Limitation & Future work Both the theory and the ablation study show that the effectiveness ofour proposed method depends on the aggregation size | b | , which is still limited by the batch sizeduring training.
Therefore a promising future direction is to find a solution to break this limit, suchlike the memory bank mechanism in contrastive learning [38].
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
Yes] We propose a novel regularization for VAEs to solveboth posterior
collapse
and hole problem.
(b) Did you describe the limitations of your work?
[Yes] See Section 5.
(c) Did you discuss any potential negative societal impacts of your work?
[No]
(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] We state the
full set of assumptions in the first paragraph of Section 3.
(b) Did you include complete proofs of all theoretical results?
[Yes] We provide completeproofs for all theoretical results in Section 3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] We include the
code, scripts for downloading data and instructions for reproduction in the supplemental
material.
(b) Did you specify all the training details (e
g., data splits, hyperparameters, how theywere chosen)?
[Yes] See Appendix B.
(c) Did you report error bars (e
g., with respect to the random seed after running experi-
ments multiple times)?
[No](d) Did you include the total amount of compute and the type of resources used (e
g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See Appendix B.
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] See Section 4.
(b) Did you mention the license of the assets?
[No]
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[No]
If you used crowdsourcing or conducted
research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]