On Neural Network Pruning’s Effect on Generalization
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Practitioners frequently observe that pruning improves model generalization.
Along-standing hypothesis attributes such improvement to model size reduction.
However, recent studies on over-parameterization characterize a new model sizeregime, in which larger models achieve better generalization.
A contradiction ariseswhen pruning is applied to over-parameterized models – while theory predicts thatreducing size harms generalization, pruning nonetheless improves it.
Motivated bysuch a contradiction, we re-examine pruning’s effect on generalization empirically.
We demonstrate that pruning’s generalization-improving effect cannot be fullyaccounted for by weight removal.
Instead, we ﬁnd that pruning can lead to betteroptimization, improving training loss.
We ﬁnd that pruning can also lead to strongerregularization, mitigating the harmful effect of noisy examples.
We empiricallydemonstrate that better optimization through extending training time or strongerregularization through reducing model width alone cannot explain the full extentof pruning’s generalization-improving effects.
Introduction
Neural network pruning techniques remove unnecessary weights to reduce the memory footprintand computational requirements of a model.
In practice, practitioners can remove a large fraction(often 80-90%) of weights without impacting generalization, measured by test error [LeCun et al.
,1990, Han et al., 2015b, Frankle and Carbin, 2019]. While recent pruning research [Han et al. ,2015b, Frankle and Carbin, 2019, Renda et al., 2020] focuses on reducing model footprint, improvinggeneralization has been a core design objective for earlier works on pruning [LeCun et al., 1990, Hassibi et al., 1993]; recent pruning literatures also frequently note that it improves generalization[Han et al., 2015a, Frankle and Carbin, 2019].
How does pruning improve generalization?
An enduring hypothesis claims that pruning beneﬁtsgeneralization by reducing the number of weights, and thus model size.
1 We refer to this hypothesisas the size-reduction hypothesis .
However, much has evolved since the proposal of this hypothesis.
Firstly, pruning algorithms grew increasingly complex.
State-of-the-art pruning algorithm [Rendaet al., 2020] prunes weights iteratively .
Each iteration consists of weights removal , where the algorithm zeros out a set of unimportant weights, followed by retraining , where the algorithm replays
[Figure]
the original learning rate schedule.
Across multiple iterations, such a retraining scheme effectivelyadopts a cyclic learning rate schedule, which may positively inﬂuence generalization [Smith, 2017].
Secondly, the classical bias-variance trade-off implies that when the model size decreases, generalization may improve.
The size-reduction hypothesis is based on pruning’s effect in this bias-variancetrade-off regime, predicting that decreasing model size may improve generalization.
However, recentwork on the double descent phenomenon [Belkin et al., 2019, Nakkiran et al., 2020] reveals thatbeyond this classical bias-variance trade-off regime lies an over-parameterized regime where modelsare large enough to achieve near-zero training error.
In this regime, bigger models achieve bettergeneralization[Belkin et al., 2019, Nakkiran et al., 2020].
Practitioners often apply modern pruningalgorithms to models in this over-parameterized regime; in this regime, while theory predicts thatreducing model size should harm generalization, pruning nevertheless improves it.
This contradictioncasts doubt on the size-reduction hypothesis.
The evolution of pruning algorithms and the renewed theoretical understanding of the role of overparameterization motivate us to re-examine pruning’s generalization-improving effect empirically.
A New Direction.
Our ﬁrst result is that the size-reduction hypothesis does not fully explainpruning’s effect on generalization.
Figure 1a compares the generalization of a family of sparsemodels of different sizes generated by a state-of-the-art pruning algorithm [Renda et al., 2020] andthe generalization of a family of models generated by the same algorithm, except modiﬁed to nolonger remove weights.
The interventions remaining for the augmented pruning algorithm to generatedifferent models are the number of training epochs and the cyclic learning rate schedule.
The results show that for any sparse VGG16 model generated by the pruning algorithm with morethan 1% of its original weights, its generalization is indistinguishable from that of the model generatedby the augmented algorithm that does not remove weights but is trained for the same number ofepochs.
These results demonstrate that the size-reduction hypothesis does not provide a completeaccount of pruning’s effect on generalization.
Our Approach.
We instead develop an alternative explanation for pruning’s beneﬁcial impact ongeneralization through an analysis of pruning’s effect on each training example as measured by thechange in its training loss before and after pruning.
We further evaluate the extent to which a trainingexample inﬂuences generalization by using an approximate measure – the L2 distance between thepredicted probabilities and one hot labels early in training, called the EL2N score, developed by Paulet al.
With our analysis, we uncover the following results.
Better Optimization.
The left plot in Figure 1b presents the change in training loss after pruningfor the training examples as grouped by percentiles of their EL2N scores for a VGG16 on CIFAR-10.
The results show that pruning improves generalization by decreasing training loss, with a particularemphasis on those examples with a high inﬂuence on generalization (i.e, high EL2N score).
We referto pruning’s ability to reduce training loss as better optimization .
Stronger Regularization.
The right plot in Figure 1b also presents the change in training loss ofexamples as grouped by percentiles of their EL2N scores, but for the same benchmark but with thepruning algorithm conﬁgured to produce a smaller, sparse model.
In this setting, the results show thatpruning can also improve generalization by increasing training loss, also with a similar, particularemphasis on highly inﬂuential examples.
While it is perhaps counter-intuitive that increasing training loss, particularly on inﬂuential examples,may improve generalization, we hypothesize that it is due to an observation that Paul et al.
[2021]made: in image datasets, highly inﬂuential training examples include a small fraction that haveambiguous or erroneous labels, training on which impairs generalization.
Therefore, it is insteadbetter for the model to not ﬁt them.
We examine this hypothesis by introducing random label noise into the training data.
We ﬁnd thatfor models with the best generalization, pruning increases training loss on these noisy examples,while still preserving the training loss of examples from the original training data.
The result is anoverall improvement in generalization for the sparse model.
We use the term stronger regularizationto describe pruning’s effect to avoid ﬁtting noisy examples.
Implications.
It has been a long-standing hypothesis that pruning improves generalization byreducing the number of weights count reduction (via minimum description length principle [Grünwald,2007] or Vapnik–Chervonenkis theory [Vapnik, 1999]).
Our empirical observations suggest that thetheory for pruning should focus on the two effects of modern pruning algorithms: better optimization,and stronger regularization.
To understand the implication of pruning’s regularization effect ongeneralization, our results suggest that it is important to quantify the heterogeneous effect of pruningacross different dataset subgroups.
Contributions
We ﬁnd that pruning leads to better optimization , improving model training loss.2.
We ﬁnd that pruning also leads to stronger regularization , mitigating the detrimental effectof noisy examples to generalization.
We demonstrate that our deconstruction of pruning’s effect into optimization and regulariza- tion cannot be further simpliﬁed — alternative explanations such as increased training time
and reduced model size fail to explain pruning’s generalization-improving effect fully.
Preliminaries
We study Iterative Magnitude Pruning (IMP) with learning-rate rewinding [Frankle and Carbin, 2019,
Frankle et al., 2020].
This algorithm consists of four steps: (1) Train a neural network to completion.(2) Remove a speciﬁed fraction of the smallest-magnitude weights.
(3) Reset the learning rate to itsvalue in an earlier epoch t [Frankle et al., 2020] and (4) Repeat steps 1-3 iteratively until the modelreaches a desired overall level of sparsity.
Terminology.
We refer to models without any weights removed as dense models ; in contrast, werefer to models with weights removed by pruning as sparse models .
We use the term optimal sparsemodel to refer to the model with the sparsity that achieves the best validation error.
We call suchsparsity the optimal sparsity .
We deﬁne generalization as classiﬁcation error on the test set.
Experimental Methodology.
We use standard architectures: LeNet [Lecun et al., 1998], VGG-16[Simonyan and Zisserman, 2015], ResNet-20, ResNet-32 and ResNet-50 [He et al., 2016], and trainon benchmarks (MNIST, CIFAR-10, CIFAR-100, ImageNet) using standard hyperparameter settings,following Frankle and Carbin [2019], Frankle et al.
[2020], Wang et al.
Also following [Frankle and Carbin, 2019, Frankle et al., 2020], we set the t hyperparameter of IMP to t = 0 for LeNet-MNIST benchmark and t = 10 for the others.
Appendix A presents our hyperparametersettings in full detail.
We report all results by running the same experiment three times with distinct random seeds.
Weconclude that any real-valued results of two experiments differ if and only if the mean differencebetween three independent runs of respective experiments is at least one standard deviation awayfrom zero, otherwise we say that the results match .
We run all experiments on TPUs with PyTorch[Paszke et al., 2019] using the OpenLTH library [Frankle and Carbin, 2019]..
[Figure]
Generalization Improvement from Better Optimization
It is well-appreciated that not all training examples have the same effect on training and generalization(see a vast literature on data pruning [Toneva et al., 2019, Paul et al., 2021], fairness [Calders et al. ,2009, Hardt et al., 2016], and many more).
Recent work by Hooker et al.
[2019] illuminates thatpruning has varying effects on subgroups of test examples.
In this section, we identify which subgroups of training examples are affected by pruning, and thespecial properties of these subgroups: it is the subgroups that are most difﬁcult for the dense model tolearn.
We then connect these examples to effects on generalization via leave-subgroup-out retrainingand empirically show that the subgroups of training examples that are most affected by pruning alsotend to have the highest impact on generalization.
Method.
Paul et al.
[2021] propose a metric (EL2N) that scores individual training examples basedon their error early in training.
Paul et al.
[2021] shows that the EL2N score correlates with thedifﬁculty of individual examples, which he measures with how frequently the model misclassiﬁesthe example during training [Toneva et al., 2019, Paul et al., 2021].
For a range of models, Paulet al.
[2021] present empirical evidence that the EL2N score of a training example correlates with itsinﬂuence on generalization.
The authors further draw theoretical connections between EL2N scoreand inﬂuence on generalization We present details of EL2N score calculation in Appendix A.3.
To measure the change in optimization due to pruning, we partition the training set into M subgroups,each with a different range of EL2N scores.
For each subgroup, we then compute the average changein training loss that pruning causes on examples in the subgroup.
We compute the change in trainingloss by subtracting the training loss of the dense model from that of the sparse model.
A negativevalue indicates that pruning improves the subgroup training loss.
We measure training loss at twosparsities of interest: the sparsity that achieves the best generalization, and the highest sparsity thatstill attains better generalization than the dense model.
In Appendix B, we present pruning’s effect on subgroup training loss at a wider range of sparsity levels.
We pick M = 20 because it is the largestvalue of M that enables us to clearly present per-subgroup training loss change.
Results.
We visualize pruning’s effect on subgroup training loss in Figure 2.
Our results consistentlyshow that pruning most signiﬁcantly affects examples with the highest range of EL2N scores.
Theaverage magnitude of pruning’s effect on examples with top 20% EL2N scores is 3.5-71x times thaton examples with bottom 80% EL2N scores.
Figure 2a shows that at the sparsity that achieves the best generalization, pruning’s effect on allsubgroups of the training dataset is to reduce their training loss.
This conﬁrms our expectation thatpruning leads to better optimization, which may improve generalization.
The largest training loss reduction happens to the examples with high EL2N scores, which have beenlinked to high importance on generalization [Paul et al., 2021].
In Appendix D, we empirically validatethe importance of pruning-affected examples to generalization: excluding 20% of pruning-affectedexamples hurts generalization more than excluding a random subset of the same size.
Generalization Improvement from Stronger Regularization
Pruning to the highest sparsity that improves generalization (Figure 2b) displays a similar trend aspruning to the optimal sparsity (Figure 2a) – generalization improves with an overall reduction intraining loss, except for CIFAR10 dataset, where pruning improves generalization while at the sametime causing the sparse models to avoid ﬁtting all subgroups of examples.
Speciﬁcally, for CIFAR10ResNet20 at 83.22% sparsity (Figure 2b, second from left) and CIFAR10-VGG16 at 99.08% sparsity(Figure 2b, third from left), the change in training loss on most subgroups is positive, indicatingworse training loss.
While pruning always increases training loss when it removes a large enoughfraction of model weights, it is surprising to observe the accompanied generalization improvement.
The second and third plots of Figure 2b show that the loss increase is especially pronounced on aparticular subgroup of examples with high EL2N scores, that are slow to learn and have an outsizedeffect on generalization.
Paul et al.
[2021] observed that although examples with high EL2N scoresare predominately beneﬁcial for model generalization, there exist a small fraction of noisy examples,such as ones with ambiguous or erroneous labels, that have high EL2N scores and exert a signiﬁcantinﬂuence on model generalization, albeit in a harmful way.
Here we zoom in on the effect of pruning on noisy examples.
By injecting random label noise, wehave control over which examples are noisy.
Method.
We inject p % random label noise by selecting p % examples uniformly at random andchanging the label of each example to one of the other labels in the dataset sampled uniformly atrandom.
We refer to the data that was not affected as clean data .
We sample and ﬁx the randomlabels before each independent run of an experiment.
We report the following results for models trained on datasets with random label noise: (1) pruning’simpact on generalization versus sparsity (Figure 3); (2) pruning’s impact on training loss for theclean data and random label data(Figure 4); 4 (3) pruning’s impact on training loss for subgroups ofexamples, each with a different range of EL2N scores (Figure 5).
Generalization results.
As seen in Figure 3, in the presence of random label noise, the generalization of the optimal sparse models is better than that of the dense models on 11 out of 13 benchmarksand matches that of the dense models on the rest of the benchmarks.
Our results provide evidencethat pruning to the optimal sparsity improves generalization in the presence of random label noise.
The extent of pruning’s generalization improvement grows as the portion of random label noiseincreases.
For example, on the CIFAR10-VGG16 benchmark, pruning to the optimal sparsityimproves generalization by 5.2%, 16.1% and 34.6% on 15%, 30% and 60% label noise.
Clean versus noisy data.
Figure 4 shows that pruning from a dense model to a sparse one,the average loss on the noisy examples is initially reduced (for example, training loss on noisy data reduces when pruning the CIFAR10-ResNet20 model to more than 10% weighs remaining), suggesting better optimization.
As the fraction of weights remaining drops further, pruning leads tostronger regularization – the loss on noisy data starts to go up: the gap between the average trainingloss of clean versus noisy examples widens up (for example, training loss on noisy data increases when pruning the CIFAR10-ResNet20 model to less than 10% weighs remaining), and eventuallydiminishes as the sparsity level increases further
In particular, for a range of sparsities generalization improves over the dense model the better than those with randomized labels Taking the dense model the
60% label noise an example this model incurs the
Pruning
Change in loss by subgroups.
Similarly, as in Figure 2, we report the change in loss on subgroupsof examples with different EL2N scores, but this time in the presence of label noise.
As shown in
Figure 5, pruning to the optimal sparsity increases the loss of high EL2N examples, contrary to whatwe observe in Figure 2 when training on clean data only.
Note that random label examples have high
EL2N scores [Paul et al., 2021].
Intuitively, noisy examples can signiﬁcantly harm generalization,and a model that increases the training loss on noisy examples may improve generalization.
In Appendix E, we empirically validate that excluding examples misclassiﬁed by the sparse modelsfrom the training dataset improves dense model generalization similar to pruning.
Conclusion.
In this section, using random label noise, we enhance the two effects of pruning: betteroptimization and stronger regularization.
At relatively low pruning levels, regularization effects arerelatively small, and better optimization is observed via reduced loss of slow-to-learn (high EL2Nscore) examples.
As the pruning level increases, regularization effects strengthen, as seen from theincrease in training loss on noisy examples.
It is worth emphasizing that across all sparsities weobserve a highly nonuniform effect of pruning-induced regularization across EL2N score subgroupson both standard datasets and datasets with random label noise.
Crucially, the optimal optimization–regularization trade-off depends on the noise levels in the dataset.
On datasets with few noisy examples, Figure 2 show that at the optimal sparsity, better optimizationand negligible regularization is the key to good generalization, as indicated by reduced training loss.
On datasets with injected random label noise, the optimal sparsity is the one at which pruning-inducedregularization effects increase noisy data loss.
The Effects of Optimization and Regularization in Isolation
We have identifed two key components in pruning’s effect on generalization: better optimization,and stronger regularization.
These two components intersect with two components of the pruningalgorithm itself: increased training time and weight reduction, respectively.
In this section we studythese two components in isolation, concluding that neither of them alone can explain the effect ofpruning on generalization across all sparsities.
In Appendix K, we also evaluate, via ablation studies,the individual contribution of design choices of pruning algorithm itself to generalization.
5.1 Extended Training Time
In sufﬁciently overparameterized models, Nakkiran et al.
[2020] observed that extending the trainingtime results in smaller training and test error simultaneously.
Pruning also effectively extends thetraining time.
It is natural to ask what happens to generalization if the weight removal part is removedfrom pruning, and only the retraining part remains.
To answer, we train the dense models exactlyas done in the pruning algorithm, for the same number of gradient steps and learning rate schedule,but without removing any weights.
The comparison between the generalization of dense models wetrain with the said dense training algorithm and the generalization of sparse models we obtain withpruning shows that longer training time alone fails to explain pruning’s effect on generalization.
Method.
We study the effect of training dense models for extra gradient steps on generalization,without removing any weights.
When training the dense models, we use the same learning rateschedule as pruning, replaying the original learning rate schedule designed for the dense modelmultiple times, effectively adopting a cyclic learning rate schedule.
We refer to this training algorithmas extended dense training .
We report and compare the test errors of models that we obtain withpruning and extended dense training algorithm.
Results.
We plot test errors of models that we obtain using extended dense training and pruning as a function of training epochs in Figure 6.
Extended dense training and pruning have similargeneralization in the standard dataset regime.
However, extended dense training underperformspruning on 9 out of the 13 benchmarks that include random label noise. On LeNet, ResNet20, Table
Appendix M. VGG16, ResNet32 and ResNet50 benchmarks with random label noise, extended dense trainingachieves test errors that are worse than pruning by 0 (matching) to 2.6%, 0.4 to 14.5%, 3.6 to 34.2%,0 (matching) to 4.8% and -0.3%, respectively.
Conclusion.
At low sparsities, generalization of dense models is indistinguishable from generalization of sparse models.
However, at high sparsities, generalization of dense models no longermatches generalization of sparse models, especially in the presence of substantive random label noise.
Therefore, longer training time alone does not account for pruning’s generalization-improving effectin the presence of substantive random label noise.
5.2 Size-Reduction
Pruning reduces model size by reducing the number of weights in a given model.
We study the effectsof size-reduction on generalization alone without increasing training time.
We do so by comparingpruning with model width down-scaling, since both techniques reduce the number of weights in amodel.
At optimal sparsities on datasets without random label noise, in Section 1 we show that sizereduction is not necessary to replicate generalization improvements; in this section, we show thatsize reduction is also not sufﬁcient to replicate generalization improvement.
Method.
We reduce model size by down-scaling the width of a dense model: we train a sequenceof dense models, where next model in the sequence has 80% of the width of the model preceding it.
We then compare generalization of models we obtain with pruning versus width down-scaling.
Results.
We plot the test errors of models we obtain with pruning and width down-scaling, usingsquare and circle-shaped markers respectively in Figure 7.
6 Width down-scaling under-performspruning on 4 out of 5 benchmarks on the original training dataset without random label noise.
When training with random label noise, width down-scaling has a strong regularization effect, andsigniﬁcantly improves generalization, similarly to pruning.
Despite this improvement, a gap remainsbetween the minimum test error achieved by pruning and width down-scaling: pruning does better on LeNet, ResNet20, VGG16, ResNet32 and ResNet50 benchmarks by -4 to 0.4%, 0% (matching), 1.5to 1.8%, 0 (matching) to 2.1%, and 0.36% in test error, respectively.
Conclusion.
We conclude that size reduction alone, through down-scaling model width, cannotexplain changes in generalization due to pruning.
In particular, at optimal sparsity levels pruningachieves better generalization on both standard datasets, and datasets with random label noise.
Related Work
Pruning.
Pruning has an extended history: early works found that pruning enhances model interpretability and generalization [Mozer and Smolensky, 1988, LeCun et al., 1990, Hassibi et al., 1993].
Recently, motivated by the advent of deep neural network models, pruning is used to reduce thestorage and computational demand of such models [Han et al., 2015b, 2016, Liu et al., 2017].
For ourwork, pruning methods can be categorized by 1) time of onset, 2) structuredness and granularity, and 3) iterative or one-shot.
We choose to study pruning methods that prune only after the completion ofdense model training; such pruning methods often produce models with the best accuracy/parametercount trade-off compared with alternative ones with an earlier time of onset, such as pruning beforetraining [Frankle et al., 2021].
Similarly, we focus on pruning individual elements in weight tensors inan unstructured manner; alternatively, structured, coarse-grained pruning methods impose additionalconstraints on pruning, and therefore may lead to inferior pruning outcomes.
Our study concernsstate-of-the-art iterative pruning methods [Frankle and Carbin, 2019, Renda et al., 2020], whereweights are removed gradually in multiple iterations, with retraining interspersed between iterationsto repair the damaged networks; compared with its one-shot counterpart, iterative pruning methodscan prune models to more extreme sparsity without destabilizing training [Frankle et al., 2020].
Pruning’s Effect on Generalization.
Pruning literature often remarks on pruning’s beneﬁcialimpact on model generalization [LeCun et al., 1990, Han et al., 2015b, Frankle and Carbin, 2019].
This intriguing effect has sparked interest in examining pruning as a technique to gain additionalgeneralization beneﬁts beyond common regularization methods: Giles and Omlin [1994] demonstrated that pruning can improve the generalization of recurrent neural network models better thanweight decay.
Thimm and Fiesler [1995], Augasta and Kathirvalavakumar [2013] compared theeffect on the generalization of several contemporary pruning techniques.
Recently, Bartoldson et al.[2020] discovered a positive correlation between pruning stability and model generalization, andattributed this correlation to an increased ﬂatness of minima due to pruning.
Compared with priorliterature, our work differs in scope and methodology.
Our object of study involves a distinct class ofstate-of-the-art iterative unstructured-pruning methods, and we derive novel characterizations of theunderlying mechanism by examining pruning’s impact model training.
Pruning’s Effect on Data.
Our work contributes to a growing line of work [Hooker et al., 2019, Liebenwein et al., 2021] investigating pruning’s effect on examples beyond prediction error.
Hookeret al.
[2019] discovered that model compression can have a disproportionately large impact on theprediction performance of under-represented long-tail of the data distribution.
Liebenwein et al.[2021] studied the performance and stopping criterion for pruning using metrics such as out-ofdistribution generalization and resilience to noise, and ﬁnd that the maximum weight sparsity thatpreserves prediction performance varies signiﬁcantly across different tasks.
To the best of ourknowledge, our work is the ﬁrst to examine pruning’s impact on examples in the training set .
Closing Discussion
Societal Impact.
Our work examines pruning’s effect on generalization.
However, we measuregeneralization using test error and Hooker et al.
[2019] show that test error may be an imperfectcharacterization of model prediction quality due to a lack of consideration of fairness and equity.
Concretely, our work does not address the potential disproportionate impact of model pruning on labelcategories.
Our work must therefore be interpreted within the context of the community’s current andfuture understanding of pruning’s potential contribution to systemic bias, especially against minoritygroups that may be underrepresented in existing training datasets.
Limitations.
Our work is empirical in nature.
Though we validate each of our claims with extensiveexperiment results using 5 dataset-model architecture combinations, each with 2-4 random labelnoise levels, culminating in 18 independent benchmarks, we recognize that our claims may still notgeneralize to all circumstances due to the empirical nature of our study.
Conclusion.
We show that the long-standing size-reduction hypothesis attributing pruning’s generalization beneﬁts to its reduction of the size of a model does not fully explain the relationshipbetween pruning and generalization.
Inspired by prior studies [Hooker et al., 2019, Liebenwein et al. ,2021] that demonstrate pruning’s non-uniform impact on examples, we developed a new analysis tointerpret pruning’s impact on individual examples by assessing its impact on an example’s trainingloss in relation to the example’s inﬂuence on generalization.
With this analysis, we ﬁnd that pruning leads to both better optimization, improving model training loss, as well as stronger regularization, mitigating the harmful effect of noisy examples togeneralization.
Both aspects work to improve generalization of a model.
Our ﬁndings along with our novel analytical technique advance the empirical understanding ofpruning as a learning algorithm.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
You are strongly encouraged to include a justiﬁcation to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .
• Did you include the license to the code and datasets?
[No] The code and the data areproprietary.•
Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] Section (c) Did you discuss any potential negative societal impacts of your work?
[Yes] Section
(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]2.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A](b)
Did you include complete proofs of all theoretical results?
[N/A]3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[No] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes] Appendix A
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[Yes] Section
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] Appendix A
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] Section
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
[N/A]