On Neural Network Pruning’s Effect on Generalization
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Practitioners frequently observe that pruning improves model generalization.
Along-standing hypothesis attributes such improvement to model size reduction.
However, recent studies on over-parameterization characterize a new model sizeregime, in which larger models achieve better generalization.
A contradiction ariseswhen pruning is applied to over-parameterized models – while theory predicts thatreducing size harms generalization, pruning nonetheless improves it.
Motivated bysuch a contradiction, we re-examine pruning’s effect on generalization empirically.
We demonstrate that pruning’s generalization-improving effect cannot be fullyaccounted for by weight removal.
Instead, we ﬁnd that pruning can lead to bettertraining, improving model training loss.
We ﬁnd that pruning can also lead tostronger regularization, mitigating the harmful effect of noisy examples.
Pruningextends model training time and reduces model size, which improves trainingand strengthens regularization respectively.
We empirically demonstrate that bothfactors are essential to explaining pruning’s beneﬁts to generalization fully.
Introduction
Neural network pruning techniques remove unnecessary weights to reduce the memory footprintand computational requirements of a model.
In practice, practitioners can remove a large fraction(often 80-90%) of weights without impacting generalization, measured by test error [25, 16, 10].
While recent pruning research [16, 10, 37, 5, 24, 32, 9, 50, 3, 27, 47, 38, 28, 29, 41, 39, 7, 51, 43]focuses on reducing model footprint, improving generalization has been a core design objective forearlier works on pruning [25, 19]; recent pruning literatures also frequently note that it improvesgeneralization [15, 10].
How does pruning improve generalization?
An enduring hypothesis claims that pruning beneﬁtsgeneralization by reducing the number of weights and thus model size.
1 We refer to this hypothesisas the size-reduction hypothesis .
However, much has evolved since the proposal of this hypothesis.
Firstly, pruning algorithms grew increasingly complex.
State-of-the-art pruning algorithm [37] prunesweights iteratively .
Each iteration consists of weights removal , where the algorithm zeros out a set ofunimportant weights, followed by retraining , where the algorithm replays the original learning rateschedule.
Across multiple iterations, such a retraining scheme effectively adopts a cyclic learningrate schedule, which may positively inﬂuence generalization [42].
[Figure]
Secondly, the classical bias-variance trade-off implies that when the model size decreases, generalization may improve.
The size-reduction hypothesis is based on pruning’s effect in this bias-variancetrade-off regime, predicting that decreasing model size may improve generalization.
However, recentwork on the double descent phenomenon [4, 34] reveals that beyond this classical bias-variance tradeoff regime lies an over-parameterized regime where models are large enough to achieve near-zerotraining error.
In this regime, bigger models achieve better generalization [4, 34].
Practitioners oftenapply modern pruning algorithms to models in this over-parameterized regime; in this regime, whiletheory predicts that reducing model size should harm generalization, pruning nevertheless improvesit.
This contradiction casts doubt on the size-reduction hypothesis.
The evolution of pruning algorithms and the renewed theoretical understanding of the role of overparameterization motivate us to re-examine pruning’s generalization-improving effect empirically.
A New Direction.
Our ﬁrst result is that the size-reduction hypothesis does not fully explainpruning’s effect on generalization.
Figure 1a compares the generalization of a family of sparse modelsof different sizes generated by a state-of-the-art pruning algorithm [37] and the generalization of afamily of models generated by the same algorithm, except modiﬁed to no longer remove weights.
The interventions remaining for the augmented pruning algorithm to generate different models arethe number of training epochs and the cyclic learning rate schedule.
The results show that for any sparse VGG16 model generated by the pruning algorithm with morethan 1% of its original weights, its generalization is indistinguishable from that of the model generatedby the augmented algorithm that does not remove weights but is trained for the same number ofepochs.
These results demonstrate that the size-reduction hypothesis does not provide a completeaccount of pruning’s effect on generalization.
Our Approach.
We instead develop an alternative explanation for pruning’s beneﬁcial impact ongeneralization through an analysis of pruning’s effect on each training example as measured by thechange in its training loss before and after pruning.
We further evaluate the extent to which a trainingexample inﬂuences generalization by using an approximate measure – the L2 distance between thepredicted probabilities and one hot labels early in training, called the EL2N score, developed by Paulet al.
With our analysis, we uncover the following results.
Better Training.
The left plot in Figure 1b presents the change in training loss after pruning forthe training examples as grouped by percentiles of their EL2N scores for a VGG16 on CIFAR-10.
The results show that pruning improves generalization by decreasing training loss, with a particular2 In Appendix C we show this phenomenon for 4 additional benchmarks, and other pruning algorithms.
emphasis on those examples with a high inﬂuence on generalization (i.e, high EL2N score).
We referto pruning’s ability to reduce training loss as better training .
Stronger Regularization.
The right plot in Figure 1b also presents the change in training lossof examples as grouped by percentiles of their EL2N scores, for the same benchmark but with thepruning algorithm conﬁgured to produce a smaller, sparse model.
In this setting, the results show thatpruning can also improve generalization by increasing training loss, also with a similar, particularemphasis on highly inﬂuential examples.
While it is perhaps counter-intuitive that increasing training loss, particularly on inﬂuential examples,may improve generalization, we hypothesize that it is due to an observation that Paul et al.
[36] made:in image datasets, highly inﬂuential training examples include a small fraction that have ambiguousor erroneous labels, training on which impairs generalization.
Therefore, it is instead better for themodel to not ﬁt them.
We examine this hypothesis by introducing random label noise into the training data.
We ﬁnd thatfor models with the best generalization, pruning increases training loss on these noisy examples,while still preserving the training loss of examples from the original training data.
The result is anoverall improvement in generalization for the sparse model.
We use the term stronger regularizationto describe pruning’s effect to avoid ﬁtting noisy examples.
Implications.
It has been a long-standing hypothesis that pruning improves generalization by reducing the number of weights (via minimum description length principle [14] or Vapnik–Chervonenkistheory [46]).
Our empirical observations suggest that the theory for pruning should focus on the twoeffects of modern pruning algorithms: better training, and stronger regularization.
To understand theimplication of pruning’s regularization effect on generalization, our results suggest that it is importantto quantify the heterogeneous effect of pruning across different dataset subgroups.
Contributions
We ﬁnd that pruning leads to better training , improving
model training loss
We ﬁnd that pruning also leads to stronger regularization ,
mitigating the detrimental effect
of noisy examples to generalization.
We demonstrate that our deconstruction of pruning’s effect into training and regularizationcannot be further simpliﬁed — alternative explanations such as increased training time andreduced model size only partially explain pruning’s generalization
improving effect
Preliminaries
We study Iterative Magnitude Pruning (IMP) with learning-rate rewinding [10, 11].
This algorithmconsists of four steps: (1) Train a neural network to completion.
(2) Remove a speciﬁed fraction ofthe smallest-magnitude weights.
(3) Reset the learning rate to its value in an earlier epoch t [11] and(4)
Repeat steps 1-3 iteratively until the model reaches a desired overall level of sparsity.
Terminology.
We refer to models without any weights removed as dense models ; in contrast, werefer to models with weights removed by pruning as sparse models .
We use the term optimal sparsemodel to refer to the model with the sparsity that achieves the best validation error.
We call suchsparsity the optimal sparsity .
We deﬁne generalization as classiﬁcation error on the test set.
Experimental Methodology.
We use standard architectures: LeNet [26], VGG-16 [40], ResNet-20,
ResNet-32 and ResNet-50 [20], and train on benchmarks (MNIST, CIFAR-10, CIFAR-100, ImageNet)using standard hyperparameter settings, following Frankle and Carbin [10], Frankle et al.
[11], Wanget al.
Also following [10, 11], we set the t hyperparameter of IMP to t = 0 for LeNet-MNISTbenchmark and t = 10 for the others.
Appendix A presents our hyperparameter settings in full detail.
We report all results by running the same experiment three times with distinct random seeds.
Weconclude that any real-valued results of two experiments differ if and only if the mean differencebetween three independent
[Figure]
Generalization Improvement from Better Training
It is well-appreciated that not all training examples have the same effect on training and generalization(see a vast literature on data pruning [45, 36], fairness [6, 18], and many more).
Recent work by
Hooker et al.
[21] illuminates that pruning has varying effects on subgroups of test examples.
In this section, we identify which subgroups of training examples are more affected by pruning, andthe special properties of these subgroups: it is the subgroups that are most difﬁcult for the densemodel to learn.
We then connect these examples to effects on generalization via leave-subgroup-outretraining and empirically show that the subgroups of training examples that are most affected bypruning also tend to have the highest impact on generalization.
Method.
Paul et al.
[36] propose a metric (EL2N) that scores individual training examples based ontheir error early in training.
Paul et al.
[36] shows that the EL2N score correlates with the difﬁcultyof individual examples, which they measure with how frequently the model misclassiﬁes the exampleduring training [45, 36].
For a range of models, Paul et al.
[36] present empirical evidence that
the
EL2N score of a training example correlates with its inﬂuence on generalization.
The authors furtherdraw theoretical connections between EL2N score and inﬂuence on generalization We present detailsof EL2N score calculation in Appendix A.3.
To measure the change in (cross entropy) loss on training examples due to pruning, we partition thetraining set into M subgroups, each with a different range of EL2N scores.
For each subgroup, wethen compute the average change in training loss that pruning causes on examples in the subgroup.
We compute the change in training loss by subtracting the training loss of the dense model from thatof the sparse model.
A negative value indicates that pruning improves the subgroup training loss.
Wemeasure training loss at two sparsities of interest: the sparsity that achieves the best generalization,and the highest sparsity that still attains better generalization than the dense model.
In Appendix B,we present pruning’s effect on subgroup training loss at a wider range of sparsities.
We pick M = 20as
it is the largest value of M that enables us to clearly present per-subgroup training loss change.
Results.
We visualize pruning’s effect on subgroup training loss in Figure 2.
Our results consistentlyshow that pruning most signiﬁcantly affects examples with the highest range of EL2N scores.
Theaverage magnitude of pruning’s effect on examples with top 20% EL2N scores is 3.5-71x times thaton examples with bottom 80% EL2N scores.
Figure 2a shows that at the sparsity that achieves the best generalization, pruning’s effect on allsubgroups of the training dataset is to reduce their training loss.
This conﬁrms our expectation thatpruning leads to better training, which may improve generalization.
The largest training loss reduction happens to the examples with high EL2N scores, which havebeen linked to high importance on generalization [36].
In Appendix D, we empirically validatethe importance of pruning-affected examples to generalization: excluding 20% of pruning-affectedexamples hurts generalization more than excluding a random subset of the same size.
Generalization Improvement from Stronger Regularization
Pruning to the highest sparsity that improves generalization (Figure 2b) displays a similar trend aspruning to the optimal sparsity (Figure 2a) – generalization improves with an overall reduction intraining loss, except for CIFAR10 dataset, where pruning improves generalization while at the sametime causing the sparse models to avoid ﬁtting all subgroups of examples.
Speciﬁcally, for CIFAR10ResNet20 at 83.22% sparsity (Figure 2b, second from left) and CIFAR10-VGG16 at 99.08% sparsity(Figure 2b, third from left), the change in training loss on most subgroups is positive, indicatingworse training loss.
While pruning always increases training loss when it removes a large enoughfraction of model weights, it is surprising to observe the accompanied generalization improvement.
The second and third plots of Figure 2b show that the loss increase is especially pronounced on aparticular subgroup of examples with high EL2N scores, that are slow to learn and have an outsizedeffect on generalization.
Paul et al.
[36] observed that although examples with high EL2N scores arepredominately beneﬁcial for model generalization, there exist a small fraction of noisy examples,such as ones with ambiguous or erroneous labels, that have high EL2N scores and exert a signiﬁcantinﬂuence on model generalization, albeit in a harmful way.
Here we zoom in on the effect of pruning on noisy examples.
By injecting random label noise, wehave control over which examples are noisy.
Method.
We inject p % random label noise by selecting p % examples uniformly at random andchanging the label of each example to one of the other labels in the dataset sampled uniformly atrandom.
We refer to the data that was not affected as clean data .
We sample and ﬁx the randomlabels before each independent run of an experiment.
We report the following results for models trained on datasets with random label noise: (1) pruning’simpact on generalization versus sparsity (Figure 3); (2) pruning’s impact on training loss for theclean data and random label data(Figure 4); 4 (3) pruning’s impact on training loss for subgroups ofexamples, each with a different range of EL2N scores (Figure 5).
[Figure]
Generalization results.
As seen in Figure 3, in the presence of random label noise, the generalization of the optimal sparse models is better than that of the dense models on 11 out of 13 benchmarksand matches that of the dense models on the rest of the benchmarks.
Our results provide evidencethat pruning to the optimal sparsity improves generalization in the presence of random label noise.
The extent of pruning’s generalization improvement grows as the portion of random label noiseincreases.
For example, on the CIFAR10-VGG16 benchmark, pruning to the optimal sparsityimproves generalization by 5.2%, 16.1% and 34.6% on 15%, 30% and 60% label noise
Clean versus noisy data.
Figure 4 shows that pruning from a dense model to a sparse one,the average loss on the noisy examples is initially reduced (for example, training loss on noisydata reduces when pruning the CIFAR10-ResNet20 model to more than 10% weighs remaining),suggesting better training.
As the fraction of weights remaining drops further, pruning leads tostronger regularization – the loss on noisy data starts to go up: the gap between the average trainingloss of clean versus noisy examples widens up (for example, training loss on noisy data increaseswhen pruning the CIFAR10-ResNet20 model to less than 10% weighs remaining), and eventuallydiminishes as the sparsity level increases further.
In particular, for a range of sparsities wheregeneralization improves over the dense model, the sparse model ﬁts clean examples signiﬁcantlybetter than those with randomized labels.
Taking the dense VGG16 model trained on the CIFAR10dataset with 15% - 60% label noise as an example, we see that this model incurs 0.03 - 0.13 highertraining loss on the partition with injected random label noise than on the partition without.
Pruningto the optimal level of sparsity expands this gap to 1.38 - 2.97.
[Figure]
Change in loss by subgroups.
Similarly, as in Figure 2, we report the change in loss on subgroupsof examples with different EL2N scores, but this time in the presence of label noise.
As shown in
Figure 5, pruning to the optimal sparsity increases the loss of high EL2N examples, contrary to whatwe observe in Figure 2 when training on clean data only.
Note that random label examples have high
EL2N scores [36].
Intuitively, noisy examples can signiﬁcantly harm generalization, and a modelthat increases the training loss on noisy examples may improve generalization.
In Appendix E, weempirically validate that excluding examples misclassiﬁed by the sparse models from the trainingdataset improves dense model generalization similar to pruning.
Conclusion.
In this section, using random label noise, we enhance the two effects of pruning:better training and stronger regularization.
At relatively low pruning levels, regularization effects arerelatively small, and better training is observed via reduced loss of slow-to-learn (high EL2N score)examples.
As the pruning level increases, regularization effects strengthen, as seen from the increasein training loss on noisy examples.
It is worth emphasizing that across all sparsities we observe ahighly nonuniform effect of
Crucially, the optimal training–regularization trade-off depends on the noise levels in the dataset.
On datasets with few noisy examples, Figure 2 show that at the optimal sparsity, better training andnegligible regularization is the key to good generalization, as indicated by reduced training loss.
Ondatasets with injected random label noise, the optimal sparsity is the one at which pruning-inducedregularization effects increase noisy data loss.
Isolating the Effects of Extending Training Time and Reducing Model Size
We have identifed two key components in pruning’s effect on generalization: better training, andstronger regularization.
These two components intersect with two components of the pruningalgorithm itself: increased training time and weight reduction, respectively.
In this section we studythese two components in isolation, concluding that neither of them alone can explain the effect ofpruning on generalization across all sparsities.
In Appendix K, we also evaluate, via ablation studies,the individual contribution of design choices of pruning algorithm itself to generalization.
5.1 Extended Training Time
In sufﬁciently overparameterized models, Nakkiran et al.
[34] observed that extending the trainingtime results in smaller training and test error simultaneously.
Pruning also effectively extends thetraining time.
It is natural to ask what happens to generalization if the weight removal part is removedfrom pruning, and only the retraining part remains.
To answer, we train the dense models exactlyas done in the pruning algorithm, for the same number of gradient steps and learning rate schedule,but without removing any weights.
The comparison between the generalization of dense models wetrain with the said dense training algorithm and the generalization of sparse models we obtain withpruning shows that longer training time alone fails to explain pruning’s effect on generalization.
Method.
We study the effect of training dense models for extra gradient steps on generalization,without removing any weights.
When training the dense models, we use the same learning rateschedule as pruning, replaying the original learning rate schedule designed for the dense modelmultiple times, effectively adopting a cyclic learning rate schedule.
We refer to this training algorithmas extended dense training .
We report and compare the test errors of models that we obtain withpruning and extended dense training algorithm.
Results.
We plot test errors of models that we obtain using extended dense training and pruningas a function of training epochs in Figure 6.
Extended dense training and pruning have similargeneralization in the standard dataset regime, where pruning improves training at optimal sparsity.
Indeed, in Appendix.
M, we show that longer training time similarly leads to better training.
However,extended dense training underperforms pruning on 9 out of the 13 benchmarks that include randomlabel noise.
On LeNet, ResNet20, VGG16, ResNet32 and ResNet50 benchmarks with random labelnoise, extended dense training achieves test errors that are worse than pruning by 0 (matching)
to2.6%, 0.4 to 14.5%, 3.6 to 34.2%, 0 (matching) to 4.8% and -0.3%, respectively.
Conclusion.
At low sparsities, generalization of dense models is indistinguishable from generalization of sparse models.
However, at high sparsities, generalization of dense models no longer matches generalization of sparse models, especially in the presence of substantive random label noise.
Therefore, longer training time alone does not account for pruning’s generalization-improving effectin the presence of substantive random label noise.
5.2 Size-Reduction
Pruning reduces model size by reducing the number of weights in a given model.
We study the effectsof size-reduction on generalization alone without increasing training time.
We do so by comparingpruning with model width down-scaling, since both techniques reduce the number of weights in amodel.
At optimal sparsities on datasets without random label noise, in Section 1 we show that sizereduction is not necessary to replicate generalization improvements; in this section, we show thatsize reduction is also not sufﬁcient to replicate generalization improvement.
Method.
We reduce model size by down-scaling the width of a dense model: we train a sequenceof dense models, where next model in the sequence has 80% of the width of the model preceding it.
We then compare generalization of models we obtain with pruning versus width down-scaling.
Results.
We plot the test errors of models we obtain with pruning and width down-scaling, usingsquare and circle-shaped markers respectively in Figure 7.
6 Width down-scaling under-performspruning on 4 out of 5 benchmarks on the original training dataset without random label noise.
Whentraining with random label noise, width down-scaling improves generalization similar to pruning.
In
Appendix M, we show that size reduction leads to stronger regularization.
Despite this improvement,a gap remains between the minimum test error achieved by pruning and width down-scaling: pruningdoes better on LeNet, ResNet20, VGG16, ResNet32 and ResNet50 benchmarks by -4 to 0.4%, 0%(matching), 1.5 to 1.8%, 0 (matching) to 2.1%, and 0.36% in test error, respectively.
Conclusion.
We conclude that size reduction alone, through down-scaling model width, cannotexplain changes in generalization due to pruning.
In particular, at optimal sparsity levels pruningachieves better generalization on both standard datasets, and datasets with random label noise.
Related Work
Pruning.
Pruning has an extended history: early works found that pruning enhances model interpretability and generalization [33, 25, 19].
Recently, motivated by the advent of deep neuralnetwork models, pruning is used to reduce the storage and computational demand of such models[16, 17, 31, 5, 24, 32, 9, 50, 3, 27, 47, 38, 28, 29, 41, 39, 7, 51, 43].
For our work, pruning methodscan be categorized by 1) time of onset, 2) structuredness and granularity, and 3) iterative or one-shot.
We choose to study pruning methods that prune only after the completion of dense model training; such pruning methods often produce models with the best accuracy/parameter count trade-offcompared with alternative ones with an earlier time of onset, such as pruning before training [12].
Similarly, we focus on pruning individual elements in weight tensors in an unstructured manner;alternatively, structured, coarse-grained pruning methods impose additional constraints on pruning,and therefore may lead to inferior pruning outcomes.
Our study concerns state-of-the-art iterative pruning methods [10, 37], where weights are removed gradually in multiple iterations, with retraininginterspersed between iterations to repair the damaged networks; compared with its one-shot counterpart, iterative pruning methods can prune models to more extreme sparsity without destabilizingtraining [11].
Pruning’s Effect on Generalization.
Pruning literature often remarks on pruning’s beneﬁcialimpact on model generalization [25, 16, 10].
This intriguing effect has sparked interest in examiningpruning as a technique to gain additional generalization beneﬁts beyond common regularizationmethods: Giles and Omlin [13] demonstrated that pruning can improve the generalization of recurrent neural network models better than weight decay.
Thimm and Fiesler [44], Augasta and
Kathirvalavakumar [1] compared the effect on the generalization of several contemporary pruningtechniques.
Recently, [2] discovered a positive correlation between pruning stability and modelgeneralization, where pruning stability refers to the drop in test accuracy immediately followingweights removal.
The authors attributed this correlation to an increased ﬂatness of minima due topruning.
Compared with prior literature, our work differs in scope and methodology.
Our objectof study involves a distinct class of state-of-the-art iterative unstructured-pruning methods, and wederive novel characterizations of the underlying mechanism by examining pruning’s impact modeltraining.
Pruning’s Effect on Data.
Our work contributes to a growing line of work [21, 30] investigatingpruning’s effect on examples beyond prediction error.
Hooker et al.
[21] discovered that modelcompression can have a disproportionately large impact on the prediction performance of underrepresented long-tail of the data distribution.
Liebenwein et al.
[30] studied the performance andstopping criterion for pruning using metrics such as out-of-distribution generalization and resilienceto noise, and ﬁnd that the maximum weight sparsity that preserves prediction performance variessigniﬁcantly across different tasks.
To the best of our knowledge, our work is the ﬁrst to examinepruning’s impact on examples in the training set .
Closing Discussion
Societal Impact.
Our work examines pruning’s effect on generalization.
However, we measuregeneralization using test error and Hooker et al.
[21] show that test error may be an imperfectcharacterization of model prediction quality due to a lack of consideration of fairness and equity.
Concretely, our work does not address the potential disproportionate impact of model pruning on labelcategories.
Our work must therefore be interpreted within the context of the community’s current andfuture understanding of pruning’s potential contribution to systemic bias, especially against minoritygroups that may be underrepresented in existing training datasets.
Limitations.
Our work is empirical in nature.
Though we validate each of our claims with extensiveexperiment results using 5 dataset-model architecture combinations, each with 2-4 random labelnoise levels, culminating in 18 independent benchmarks, we recognize that our claims may still notgeneralize to all circumstances due to the empirical nature of our study.
Conclusion.
We show that the long-standing size-reduction hypothesis attributing pruning’s generalization beneﬁts to its reduction of the size of a model does not fully explain the relationshipbetween pruning and generalization.
Inspired by prior studies [21, 30] that demonstrate pruning’snon-uniform impact on examples, we developed a new analysis to interpret pruning’s impact onindividual examples by assessing its impact on an example’s training loss in relation to the example’sinﬂuence on generalization.
With this analysis, we ﬁnd that pruning leads to both better training, improving model training loss,as well as stronger regularization, mitigating the harmful effect of noisy examples to generalization.
Both aspects work to improve generalization of a model.
Our ﬁndings along with our novel analytical technique advance the empirical understanding ofpruning as a learning algorithm.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
You are strongly encouraged to include a justiﬁcation to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .
• Did you include the license to the code and datasets?
[No] The code and the data areproprietary.•
Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’scontributions and scope?
[Yes](b) Did you describe the limitations of your work?
[Yes] Section 7(c) Did you discuss any potential negative societal impacts of your work?
[Yes] Section 7(d)
Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]2.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A](b)
Did you include complete proofs of all theoretical results?
[N/A]3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[No](b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes] Appendix A
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[Yes] Section
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] Appendix A
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] Section
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
[N/A]