The Effects of Regularization and Data Augmentation are Class Dependent
Anonymous Author(s)
AffiliationAddress email
Abstract
Regularization is a fundamental technique to prevent over-fitting and to improvegeneralization performances by constraining a model’s complexity.
Current Deep Networks heavily rely on regularizers such as Data-Augmentation (DA) or weightdecay, and employ structural risk minimization, i.e. cross-validation, to selectthe optimal regularization hyper-parameters.
In this study, we demonstrate thattechniques such as DA or weight decay produce a model with a reduced complexitythat is unfair across classes. The optimal amount of DA or weight decay foundfrom cross-validation over all classes leads to disastrous model performances onsome classes e.g. on Imagenet with a resnet50, the “barn spider” classification testaccuracy falls from 68% to 46% only by introducing random crop DA during training.
Even more surprising, such performance drop also appears when introducinguninformative regularization techniques such as weight decay.
Those results demonstrate that our search for ever increasing generalization performance —averagedover all classes and samples— has left us with models and regularizers that silentlysacrifice performances on some classes.
This scenario can become dangerous whendeploying a model on downstream tasks e.g. an Imagenet pre-trained resnet50deployed on INaturalist sees its performances fall from 70% to 30% on class #8889when introducing random crop DA during the Imagenet pre-training phase.
Thoseresults demonstrate that finding a correct measure of a model’s complexity withoutclass-dependent preference remains an open research question.
Introduction
Machine learning and deep learning aim at learning systems to solve as accurately as possible agiven task at hand [LeCun et al., 1998, Bishop and Nasrabadi, 2006, Jordan and Mitchell, 2015].
This process often takes the form of (i) being given a finite dataset, a (differentiable) loss function,and a performance measure, (ii) splitting the dataset into train/valid/test sets to optimizing thesystem’s parameters e.g. from gradient updates of the loss on the train set while cross-validatinghyper-parameters using the valid set, and (iii) assessing the system’s performance on the test set.
As the training set is finite, and the optimal design of the system is unknown, it is common toemploy regularization during the optimization phase to reduce over-fitting
[Tikhonov, 1943,
Tihonov, 1963] i.e. to decrease the system’s performance gap between train set and test set samples[Simard et al., 1991, Chapelle et al., 2000, Bottou, 2012, Neyshabur et al., 2014].
Central to ourstudy is the fact that hyper-parameter selection is done via cross-validation by maximizing the validset performance with ad-hoc statistics e.g. the average accuracy over all samples for classes inclassification tasks.
Cross-validation commonly involves many different types of regularization along with their “strengths”[Goodfellow et al., 2016, He et al., 2021].
Most variants of regularization take one of two forms
Current Deep Learning t e s t e
[Figure]
r
[Figure]
best average test error of δ with poor generalization on class
Ideal Complexity Measure
[Figure]
model complexity best average test error of γ ≪ δ with similar inter-class generalization agreement on the optimal complexity
[Figure]
Data-Augmentation (DA) and weight-decay.
DA is a data-driven and informed regularization strategythat artificially increase the number of training samples [Shorten and Khoshgoftaar, 2019].
Asopposed to most explicit regularizers e.g. Tikhonov regularization [Krogh and Hertz, 1991], alsodenoted as weight decay, DA’s regularization is implicit as it is not a function of a model’s parameter,but a function of the training samples [Neyshabur et al., 2014, Hernández-García and König, 2018,
LeJeune et al., 2019]; although some DA strategies can be turned into explicit regularizers Balestrieroet al.
Nevertheless, a key distinction between DA and weight decay is that DA tends torequires more domain knowledge to be successful than weight decay.
Most —if not all— of currentstate-of-the-art employ such regularizers [Huang et al., 2018, Chen et al., 2020b, Liu et al., 2021, Tanand Le, 2021, Liu et al., 2022].
In this paper, we will demonstrate that when cross-validation is employed to select the regularization settings maximizing the validation performance, a significant bias is introduced into thetrained model: the regularized model exhibits strong per-class favoritism i.e. while the averagetest performance is improved, it is at the cost of producing a model with significant performancedrop on some of the classes as illustrated in the schematic of fig.
For readers familiar withstatistical estimation results e.g. the bias-variance trade-off [Kohavi et al., 1996, Von Luxburg and
Schölkopf, 2011] or bayesian estimation e.g. Tikhonov regularization [Box and Tiao, 2011, Gruber,2017], it should not be surprising that regularization produces bias (more details and backgroundprovided in appendix A).
In fact, it is beneficial to introduce bias through regularization if it results ina significant reduction of the estimator variance —when one minimizes the average empirical risk.
However, the potentially dangerous effect of regularization that this study brings forward is that thebias introduced by regularization is class-dependent, including on transfer learning tasks .
To thoroughly validate this observation, we propose a variety of controlled experiments in section 2.
First, we carefully quantify the impact of DA, weight decay and dropout on the per-class performanceof a model in sections 2.1 and 2.2, demonstrating that current deep learning finds itself in thescenario depicted on the left of fig.
Then, we consider the task of transfer learning in section 2.3where it is again possible to identify again a per-class bias on the target dataset even though theregularization was applied on a different (source) training set.
This latter scenario is particularlyrelevant in current times where it is common to deploy a large pre-trained model on a variety of tasksand raise an important issue: selecting the —on average— best performing pre-trained model canlead to catastrophic individual class performance even on for different downstream tasks.
Our next section 3 will aim at exploring possible explanations and solutions.
First, we will providea brief theoretical justification on why and when DA can be the cause of model bias (section 3.1) regardless of the task and data at hand.
This will shed light to a first possible issue: the DA parametersthat make the transformed input preserve its label information vary depending on the class underlyingstatistics.
In short, DA silently introduces class-imbalance in the training set.
We propose a dedicatedanalysis of the label-preserving property of DA on different classes and models in section 3.2.
Wethen take on the task of searching for a possible solution by first reviewing known theoretical studiesquantifying the interplay between regularization and bias in section 3.3.
Lastly, we propose somesolutions of our own in section 3.4 built from the gained insights of section 3.2 using label-distillationand adaptive DA.
All the codebase used to train the various models and to generate the figures will be publicly released uponcompletion of the review process along with the pre-trained weights of all the models.
For the reviewers,we currently provide in the supplementary material all summary statistics in .json files that are sufficient toreproduce the figures of the study and further explore the interplay between regularization and class-dependentbias.
Maximizing the Average Model Performance by Cross-Validation SilentlyProduce Poor Final Performances on a Minority of the Classes
The first part of our focuses on providing extensive empirical validation of fig.
We precisely quantifythe amount of class-dependent bias caused by DA in various realistic scenarios in section 2.1; andextend our analysis to weight decay and dropout in section 2.2.
Finally, we conclude by demonstratinghow the bias introduced by regularization transfers to downstream tasks e.g. when deploying an Imagenet ( source ) trained model on the INaturalist ( target ) dataset in section 2.3; that scenario iskey as it demonstrates the potential harm of selecting the best performing model —on average— onthe source dataset which could turn out to also be the most biased model on the target dataset classof interest.
In fact, it is crucial to remember that regularization, or any other form of structural riskminimization, improves generalization performances by increasing the bias of the estimator so thatthe estimator’s variance is decreased by a greater amount.
However, nothing guarantees the fairnessof this bias i.e. for it to be equally distributed amongst the dataset classes.
2.1 Precisely Measuring the Per-Class Effect of Data-Augmentation with Controlled
This section aims at quantifying precisely the amount of downward or upward per-class performanceshift that came as a result from using DA. We thus propose a sensitivity analysis by training a largecollection of models with varying DA policies to precisely assess the relation between DA andclass-dependent model bias.
DA samples have been known to sometimes disregard the semantic information of the originalsamples [Krizhevsky et al., 2012].
Nevertheless, DA remains applied universally, and fearlesslyacross tasks and datasets [Shorten and Khoshgoftaar, 2019] as it provides significant performanceimprovements, even in semi-supervised and unsupervised settings Guo et al.
[2018], Xie et al.
Misra and Maaten [2020].
First, we propose in fig.
2 a sensitivity analysis by training the same architecture on Imagenet with varying DA policies.
In particular, we consider a given DA (randomcrop in this case) and we vary the support of the parameter α which represents how much of theoriginal image is kept in the crop.
We train DNs using α ∈ [100 , τ ] with τ varying from 100 to 8 andfor each case, we report our metrics averaged over 20 trained models.
We observe a clear relationbetween increase in the strength of the DA, increase in the average test accuracy overall classes,and decrease in some per-class test accuracies .
For example, on a resnet50 Imagenet setting, theaccuracy on the “academic gown” class goes from 62% to 40% steadily as τ decreases.
We defer thesame experiment but using weight decay in the next section 2.2. To further convey our claim, we now propose a formal statistical test [Neyman and Pearson, 1933, Fisher, 1955] on the hypothesis that the per-class accuracy is significantly lower when DA isapplied for those classes. A test of significance is a formal procedure for comparing observeddata with a claim or hypothesis.
In our case we aim to test if the mean accuracy on class yof a model trained without DA is {+greater than the one obtained with DA. Due to the stochasticoptimization process, this accuracy is a random variable even when the dataset is not changed. Hencewe define that random variable by Accu y ( X ) and our null hypothesis by H 
 = E+}
[Accu y ( X )] < 121 E
[Accu y ( X DA )] with X the original dataset and X DA the DA dataset using random crop withα ∈ [100 , 8] .
A one-sided t-test is used and the form that does not assume equal variances is knownas Welch’s t-test [Welch, 1947] with statistic t
= ˆ µ ( X 1 ) − ˆ µ ( X 2 ) √
ˆ σ 2 ( X 1 ) − ˆ σ 2 ( X 2 ) , and with degrees of freedomν = (cid:16) ˆ σ 2 ( X 1 ) N+ ˆ σ 2 ( X 2 ) N 2 (cid:17) 2 (cid:16) ˆ σ 4 ( X 1 ) N 21 ( N 2 − 1) + ˆ σ 4 ( X 2 )
N 22 ( N 1 − 1) (cid:17) − 1 .
We obtain that there is enough evidence toreject H 0 with 95% confidence for 4 .
4% of all the classes, and with 99% confidence for 2 .
1% of allthe classes. Hence there is sufficient evidence to say that the per-class test accuracies is not increasedwhen introducing DA for 4 . 4% of the 1000 Imagenet classes. Note that one could formulate thisproblem in term of average treatment effect, where the treatment is the application of DA and theoutcome is the accuracy on class y of the trained model.
Doing so, one could measure the per-samplebias from the Conditional Treatment Effect, however we limit ourselves to a per-class study andleave such fine-grained analysis for future work. The next section 2.2 proposes to reproduce thoseexperiments but considering weight decay as a regularizer instead of DA. 2.2 Uninformed Weight Decay and Dropout Also Create Class-Dependent Model Bias We quantified in section 2.1 how much per-class bias was produced by DA.
As per the argumentsgiven in sections 3.1 and 3.2, it would be natural to assume that what makes DA responsible forcreating class-dependent bias in DNs is our misfortune in defining correct augmentation policies,and thus, that other regularization techniques that are uninformed e.g. weight decay would behavedifferently. The goal of this section is to demonstrate that weight decay also suffers from the sameclass-dependent behavior indicating that designing a regularizer for DNs that is fair across classesmight require novel innovative solutions. The basic formulation of weight-decay consists in having any loss to be minimized L and add to it anadditional term denoted as γ ∥ θ ∥ 22 where θ collects the model’s parameters and γ is the strength of theregularization (proportional to the restriction on the trained model complexity). In general, we do not incorporate the bias term(s) within θ as this would directly remove the ability of the model to learnthe natural class prior [Hastie et al., 2009].
As a result, and throughout this study, we consider θ toincorporate all the DN parameters except for the ones of batch-normalization layers, as commonlydone in deep learning [Leclerc et al., 2022].
We report in fig.
3 the per-class performance of aresnet50 trained on Imagenet with varying weight decay coefficient γ (as was done for DA in fig.
2)and we observe that different classes have different test accuracy sensitivities to variations in γ .
Somewill see their generalization performance increase, while others will have decreasing generalizationperformances.
That is, even for uninformative regularizers such as weight decay a per-class biasis introduced, reducing performances for some of the classes .
Although weight-decay is one ofthe most popular regularizer that is uninformed on the data and task at hand, recent studies havedemonstrated that techniques such as model pruning —which can be seen as a post-training modelcomplexity reduction i.e. regularization— also produce increased bias towards under-representedfeatures [Hooker et al., 2019, 2020].
We further confirm such findings in figs.
8 and 9 for dropoutwhere the same per-class trend is observed.
The next section 2.3 proposes to study the scenario of introducing a pre-trained model, on a differentdownstream task. We will demonstrate that even in this setting, the class-dependent effect of ourcurrent regularizers remains present and unfair towards specific downstream classes.
2.3 The Class-Dependent Bias Transfers to Other Downstream Tasks
The last experiment we propose is to quantify the amount of class-dependent bias that transfers toother downstream tasks, a common situation in transfer learning and in system deployment to thereal world [Pan and Yang, 2009].
We thus want to measure how regularization applied during thepre-training phase on a source dataset impacts the per-class accuracy of that model on the targetdataset.
In order to keep the setting similar to section 2.1, we adopt a resnet50 model with random crop DA.
That model is pre-trained on Imagenet dataset (source) with varying value of τ (random crop lowerbound) and then, the trained model is transferred to the INaturalist dataset [Van Horn et al., 2018](target) that consists of 10,000 classes.
When transferring the model to INaturalist, the parametersare kept frozen, and only a linear classifier is trained on top of it.
We report in fig.
4 the performanceof the trained models with varying τ on different INaturalist classes.
We observe once again thatthe best resnet50 —on average— is not necessarily the one that should be deployed as there exists astrong per-class bias that varies with τ .
As a result, picking the best performing model from a sourcedataset to a target dataset, might leave the pipeline to perform poorly since that model might also bethe one that is the most biased against the class of interest in the target dataset.
This result should motivate the design of novel regularizers that do not reduce performances betweenclasses at different regimes.
Additionally, due to the cost of training multiple models with varyingregularization settings, one might wonder on the possible alternative solutions to detect trends suchas shown in fig.
4 only when given a single pre-trained model.
Understanding Why and When Regularization Produces Models WithClass-Dependent Preferences
The first part of our study from section 2 empirically validated that current regularization methods donot produce the same amount of complexity control over different classes, resulting in some classesbecoming poorly predictable from a model that has been trained with regularization hyper-parametersmaximizing the average valid set performances.
We now provide in sections 3.1 and 3.2 someintuition on why DA can be a source of bias regardless of the task, dataset and model at hand.
Then,section 3.3 reviews existing works trying to confront regularization and model bias, and as we willsee, there does not exist yet a unified solution nor analysis that not only explains but also correctsthe behavior observed in section 2. Nevertheless, we attempt in section 3.4 to explore a few possiblesolutions.
3.1 A Data-Augmentation Policy That is Not Label-Preserving For All Classes Will CreateClass-Imbalance Performances
To provide a simple explanation on how DA causes bias in a trained model, we propose the followingderivation that holds for any signal e.g. timeseries, images, videos.
Without loss of generality [Huiand Belkin, 2020] we will consider here the ℓ 2 loss, although the same derivation carries out with anydesired metric.
Dataset notations.
Given a sample x ∈ X with X ⊂ R D , we consider y ≜ f ∗ ( x ) to be theground-truth target value.
Hence our hope is to learn an approximator f
θ that is as close as possible tof ∗ everywhere in X , although we only observe a finite training set X ≜ { ( x 1 , y 1 ) , . . .
, ( x N , y N ) } .
Data-Augmentation notations.
Additionally, one employs a DA policy T : R D × K (cid:55)→ R D suchthat given a transformation parameter α ∈ K , T α ( x ) produces the transformed view of x .
Often,one also defines a density p on K that helps in sampling transformation parameters that are a prioriknown to be the most useful.
Theorem 1.
Whenever the transformations produced by T α , ∀ α do not respect the level-set of f ∗ , andwhenever the model has enough capacity to minimize the training loss, the DA will create irreduciblebias in f θ as in
[Equation]
The main idea of the proof, provided in appendix B, is to show that if a transformation does not movesamples on the level-set of the true function (left-hand-side of eq.
(1)), then f θ will learn a differentlevel (since it has 0 training error), and thus ∥ f ∗ − f θ ∥ > 0 i.e. f θ is biased regardless of the trainingset.
Whenever the left-hand-side of eq.
(1) is 0 , the DA is denoted as label-preserving [Cui et al., 2015,
Taylor and Nitschke, 2018].
From the above, we see that unless the target y associated to T α ( x ) ismodified accordingly to encode the shift in the target function level-set produced by T α , any DA that is not label-preserving will introduce a bias .
Some DAs propose to incorporate labeltransformation i.e. not only x but also y is augmented to better inform on the uncertainty that hasbeen added into T θ ( x ) .
This is for example the case for MixUp [Zhang et al., 2017], ManifoldMixUp[Verma et al., 2019], CutMix [Yun et al., 2019] and their extensions.
Our goal in the next section 3.2 is to demonstrate how DAs such as random crop, color jittering,or CutOut are only label preserving for some values of α that vary with the sample class.
As aconsequence, while the use of the DA improves the average test performance, it is at the cost of asignificant reduction in performance for some of the [-classes.3.
-]classes.
3.2 The Same Data-Augmentation can be Label-Preserving or Not Between Different ClassesIntroducing Class-Imbalance During Training
In the previous section 3.1 we provided a general argument on the sufficient conditions for DA toproduce a biased model.
We hope in this section to provide a more concrete example that appliesto current DN training.
To that end, we will demonstrate that a DA can be label-preserving or notdepending on the sample’s class, hence, since the same DA policy is employed for all classes, the augmented dataset will exhibit a class-imbalance in favor of the classes for which the DA is mostlabel-preserving.
To measure by how much a given DA, T α , is label-preserving, we propose to take 6 popular architectures that are pre-trained on Imagenet [Deng et al., 2009] from the official PyTorch [Paszkeet al., 2019] repository, and to evaluate their accuracy performances for varying DA settings (topof fig. 5).
We observe that when considering the dataset as a whole, it is possible to identify a DAregime as a function of α for which the amount of information present in T α ( x ) becomes insufficientto predict the correct label on average. But more interestingly, we also take the per-class accuracyperformance (bottom of fig. 5) and observe that for some classes, any level of transformation α canproduce augmented samples with enough information to be correctly classified, while other classessee their samples become unpredictable as soon as T α moves away from the identity mapping. Notethat we report test set performances ensuring that the observed performances are not due to ad-hocmemorization.
To further ensure that the observed relation between label-preservation, sample class, and amountof transformation α is sound, we provide in fig.
6 the per-class test accuracy on different models,all exhibit the same trends.
In short, we identify that when creating an augmented dataset byapplying the same DA across classes, the number of per-class samples that actually containenough information about their true labels will become largely imbalance between classes, evenif the original dataset was balanced .
Any model trained on the augmented dataset will thus focuson the classes for which the DA is the most label-preserving.
[Figure]
Beyond the above intuitive and natural understand we obtained in term of data-augmentation, thereexists theoretical studies that we propose to summarize below3.3 Do Existing Studies Provide Answers Into the Inter-Play Between Regularization and
Class-Dependent Bias?
We explored in section 3.2 a possible explanation of the class-dependent bias we observed in section 2hinting at the need to use class-dependent DA.
We propose here to summarize existing studies thatare attempting to better understand the impact of DA and regularization in general onto a model’sperformance and bias.
Then, section 3.4 will explore their application.
Input independent DA exacerbates the bias already present in a dataset.
Especially relevant toour results is a recent result of Xu et al.
In this work, it was theorized that when the underlyingdataset contains inherent biases, training on the original data is more effective than employing an i.i.d.
DA policy, i.e. applying the same random augmentation to all samples/classes, to produce an unbiasedmodel.
In short, the DA exacerbates the already present biases and makes the trained model furtheraway from the unbiased optimum.
The difficulty of this result lies in defining bias for real images.
As per our experiments from section 3, we observe that bias can take many form e.g. one class mightnaturally represent its object always under the same angle.
This is particularly true say for boatswhich are rarely captured upside-down.
Hence, simply having classes with different natural statisticscould be enough for Xu et al.
[2020] to prohibit the use of DA in current datasets.
Furthermore,
Raghunathan et al.
[2020] obtained a surprising result combining both DA and regularization.
Inthat case, it was found that the minimum norm interpolant on the original + DA dataset could have alarger standard error than the minimum norm interpolant on the original dataset alone, even whenusing label-preserving DA.
However, this phenomenon only occurs as long as the model remainsover-parametrized, even when considering the original+DA dataset.
Input dependent DA can reduce sample/class bias.
Recall that section 3.2 brought forward onepossible explanation on how DA can be the source of training-set by introduce class-imbalance due tothe same DA being label-preserving for some classes and not for others.
From this, a direct solutionwould be to adapt the “strength”.
This solution, formalized in Xu et al.
[2020], consists in measuringthe bias of a model and adapt the DA policy accordingly to correct it.
This has been done in differentflavors e.g. in McLaughlin et al.
[2015], Iosifidis and Ntoutsi [2018], Jaipuria et al.
Onelimitation of this direction is that it requires to estimate a model bias and adapt the DA accordinglywhich can be challenging for large scale models.
Learned DA e.g. from a GAN can produce even more class-dependent bias.
One natural extensionof the hands-on adaptivity of a DA to a measured model bias would be to learn a DA to maximize amodel’s performance, for example.
This line of work has led to many learn DA policies e.g. using
Generative Adversarial Networks [Hu and Li, 2019].
Yet, it has been shown that learning a DA tomaximize some aggregated measure of performance will produce even more bias in a model [Huand Li, 2019].
In fact, and as per the controlled experiments from section 2, the learned DA willentirely disregard a minority of the classes if it means that the produced DA can drastically improveperformances on all others, effectively maximizing the average performances.
Inherent tradeoff between DA and model robustness.
In addition to the implication of DAinto bias, there exists an intertwined relationship between DA and model robustness.
In fact, evenassuming the use of perfectly adapted DAs, there exists an inherent tradeoff between accuracy and robustness that holds even in the infinite data limit [Tsipras et al., 2018, Fawzi et al., 2018, Zhang et al.,
For example, Min et al.
[2021] proved in the robust linear classification regime that (i) moredata improves generalization in a weak adversary regime, (ii) more data can improve generalizationup to a point where additional data starts to hurt generalization in a medium adversary regime, andthat (iii) more data immediately decreases generalization error in a strong adversary regime.
3.4 Class-Dependent Data-Augmentation
Seems Insufficient for Performance Recovery
We observed in section 2 that DA could lead to disastrous per-class performances on a minority ofclasses.
We now propose to implement one solution from section 3.3 that consists in simply notapplying the harmful DA to the classes suffering from it.
Solving the per-class impact of DA is crucial even beyond the per-class harmful effects.
In fact, asan illustrative example, a standard resnet50 on Imagenet reaches 77 .
14% top-1 with a random croplower bound of 8% .
Using cross-validation we could reach 77 . 29 with a random crop lower bound of10% on all classes.
Yet, and most interestingly, if one were able to get the best per-class performance,one could reach 79 .
We thus explore one solution from section 3.3 that consists in not applyingthe random crop DA onto the classes whose test performances degrade with it.
We report in fig.
7how such strategy is not sufficient to guarantee that the class performance is not impacted.
In fact, weobserve that applying this DA only to the classes that benefit from it is enough to bias the model anddegrade the performances on some classes at the same pace than when applying the DA to all classesunconditionally.
As a result, we observe that no readily and easily implemented solution provides us with a strategy toprevent deep learning to fall into the scenario depicted on the left of fig.
Those findings howevermotivate the search of novel model complexity controls that is fair among classes.
From a moretheoretical viewpoint, it might also be possible to better understand if even such a fair per-class modelcomplexity could exist, which is not clear as natural image classes tend to have inherently differentstatistics.
Conclusions
We proposed in this study to understand the impact of regularization, in particular data-augmentationand weight decay, into the final performances of a deep network.
We obtained that the use ofregularization increases the average test performances at the cost of significant performance drops onsome specific classes.
By focusing on maximizing aggregate performance statistics we have producedlearning mechanisms that can be potentially harmful, especially in transfer learning tasks.
In fact,we have also observed that varying the amount of regularization employed during pre-training of aspecific dataset impacts the per-class performances of that pre-trained model on different downstreamtasks e.g. going from Imagenet to INaturalist.
Lastly, commonly prescribed solutions e.g. classdependent data-augmentation do not seem to help indicating that the sole use of an augmentation onsome classes is enough to bias the model on all classes.
Hence, there remains a vast research area toexplore in order to turn deep learning model selection from the current regime to a more ideal one(left to right of fig. 1).
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[N/A]
We do not propose a novel
method that requires such discussion
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] Wecarefully described why the observations we have made can be dangerous for realworld applications
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] Our only
“theoretical result” consists in a formal statistical test for which we precisely describe our settings and statistics
(b) Did you include complete proofs of all theoretical results?
[N/A]
No theoretical result
requiring proofs was provided
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main exper-
imental results (either in the supplemental material or as a URL)?
[Yes] We include
summary statistics in the supplementary material which is enough to validate our claims.
The full codebase and all the saved models will be released upon completion
of the review process (this includes almost a thousand pre-trained resnet50s)
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes] We specify within each caption the key experimental setups,
additional training details are provided in the appendix
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[Yes] we averaged all of our runs over a significant amount of
realizations (20) and perform a formal statistical test on the significance of our results
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] We provide this in the beginning of
the appendix.
Note that we will be providing the saved models and summary statistics
to make the results easily reproducible even without GPU or computation ressources.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] For Imagenet and
INaturalist
(b) Did you mention the license of the assets?
[Yes] the full code will be released with a
NC by NC license on GitHub
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes] we include summary result files that contain all the statistics to reproduce the
main claims of the paper in the supplementary material.
The full codebase along with
pre-trained models will be released upon completion of the review process.
[Table] (d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A] We are only using the Imagenet and INaturalist datasets
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A] We are only using the Imagenet and INaturalist
datasets
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
[N/A]