Models Out Of Line: A Fourier Lens On Distribution Shift Robustness
Anonymous Author(s)
Abstract
Improving the accuracy of deep neural networks (DNNs) on out-of-distribution(OOD) data is critical to an acceptance of deep learning (DL) in real world applications.
It has been observed that accuracies on in-distribution (ID) versus OOD datafollow a linear trend and models that outperform this baseline are exceptionally rare(and referred to as “effectively robust”).
Recently, some promising approaches have been developed to improve OOD robustness: model pruning, data augmentation,and ensembling or zero-shot evaluating large pretrained models.
However, therestill is no clear understanding of the conditions on OOD data and model properties that are required to observe effective robustness.
We approach this issue by conducting a comprehensive empirical study of diverse approaches that are known to impact OOD robustness on a broad range of natural and synthetic distributionshifts of CIFAR-10 and ImageNet.
In particular, we view the “effective robustness puzzle" through a Fourier lens and ask how spectral properties of both models and OOD data influence the corresponding effective robustness.
We find this Fourier lens offers some insight into why certain robust models, particularly those from the
CLIP family, achieve OOD robustness.
However, our analysis also makes clear thatno known metric is consistently the best explanation (or even a strong explanation)of
OOD robustness. Thus, to aid future research into the OOD puzzle, we addressthe gap in publicly-available models with effective robustness by introducing a set of pretrained models— RobustNets —with varying levels of effective robustness.
Introduction
Deep learning (DL) holds great promise for solving difficult real-world problems in domains such ashealthcare, autonomous driving and cyber-physical systems.
The major roadblock in adopting DLfor real-world applications is that real-world data often deviates from the training data due to noise,corruptions or other changes in distribution that may have temporal or spatial causes [3, 14, 29, 31].
DL models are known to be highly brittle under such distribution shifts, which raises the risk ofmaking incorrect predictions with harmful consequences.
Designing approaches to learn modelsthat achieve both high accuracy on in-distribution data and high robustness to distribution shifts isparamount to the safe adoption of DL in real-world applications.
Although a performance drop under distribution shifts is expected, several recent works [1, 23] have noticed an intriguing phenomenon: accuracy on in-distribution (ID) versus accuracy on distributionshifted data obeys a linear trend across a wide range of models, corruption benchmarks, and predictiontasks.
Unfortunately, as nearly all current DL models bear a substantial loss under distribution shifts,this linear relationship implies that our current training strategies are insufficient to bridge thegap between ID and out-of-distribution (OOD) performance.
However, despite being exceedingly rare, models that break this linear trend to obtain effective robustness [1] exist—such models defy explanation by the wisdom that OOD accuracy and ID accuracy should correlate strongly, they are “out of line”.
Identifying when and why such models arise is key to overcoming OOD brittleness in DL models.
Related research efforts have focused on improving OOD robustness via new techniques such asmodel pruning [7], data augmentation [15], and ensembling large pre-trained models [25, 34].
Whilethese efforts have created significant excitement in the ML robustness community, it remains unclear(a) under which conditions on OOD data these methods can improve effective robustness (ER), and (b)which underlying model properties make a DNN effectively robust.
A more in-depth understandingof these phenomena is likely to help in bridging the gap between ID and OOD performance.
Towards achieving this goal, we carry out a comprehensive empirical study leveraging models trainedusing each of the aforementioned state-of-the-art approaches to improving OOD robustness.
Weanalyze the impact of these robustness improvement methods on various model architectures, twoclean datasets (CIFAR-10 and ImageNet), and a broad range of natural and synthetic distributionshifts.
In particular, we view the OOD robustness puzzle through a Fourier lens and ask how spectralproperties of OOD data and models influence their effective robustness.
In this process, we designnew metrics that capture Fourier sensitivity of models as test data moves farther away from thetraining data manifold.
We make the following contributions:
• A new perspective on the state of OOD robustness: across diverse models, metrics, and distribution
shifts, we observe that no one single metric rules them all, not even in-distribution accuracy.
This accents the potential need for a multi-faceted approach to understanding OOD robustness,
in contrast to ID generalization where single properties like model flatness can enjoy predictive
success across an array of models [18].
• Design of Fourier sensitivity metrics: our new metrics quantify various model properties, including
spectral ones that can explain the effective robustness of ensembled CLIP models [34] better than
ID accuracy.•
The first collection of publicly-available effectively robust models: to help crack the OOD robustness puzzle, we make available (upon publication) a dataset of CIFAR- 10 models with effectiverobustness that stems from training under a variety of data augmentation and pruning schemes.
Related work
Distribution shift fragility is a well-documented phenomenon among neural networks [28, 29, 14, 13,22] in which a model trained on an ID dataset performs markedly worse when evaluated on OOD data,even when humans find the OOD data just as easy to classify.
These OOD accuracy gaps often followa linear relationship between ID and OOD accuracy, but the slope and intercept of the linear trendlinevaries depending on the specific ID and OOD datasets [22, 10].
In our work, we focus on two popularimage classification benchmark ID datasets, CIFAR-10 [20] and ImageNet [5], and two OOD testdatasets for each, CIFAR-10.1 [28], CIFAR-10-C [14], ImageNetV2 [29], and ImageNet-C [13].
While most models follow the linear ID-OOD trendlines of Miller et al.
[22], some models are“effectively robust" and appear above the trendline, with OOD accuracy higher than expected given
ID accuracy [1].
These robust models can arise through various methods: by pruning a model as asort of regularization [7], by training with data augmentation intended to mimic the expected OODtest data [15], or by pretraining on a larger and more diverse dataset and evaluating zero-shot [25],partially finetuning [1], or interpolating the weights of a zero-shot and a finetuned model [34].
Although as a community we are beginning to uncover robust training methods, it remains a mysterywhy these particular methods achieve robustness on particular distribution shifts.
We study thismystery via a Fourier lens based on prior work involving both image frequency and function frequencyanalyses.
Ortiz-Jimenez et al.
[24], Yin et al. [36], and Sun et al.
[33] study model sensitivity to Fourierperturbations of the input images and analyze how different data augmentations produce different
Fourier sensitivities (and correspondingly robustness to image perturbations of different frequencies).
Another line of research focuses on spectral bias [9, 2, 35, 26], wherein models prioritize learning lowfrequency functions over the input space.
This bias towards low function frequencies can also affectrobustness and is influenced by training hyperparameters like data augmentation and weight decay [9].
Among the model properties we study as potential predictors of OOD accuracy are ID accuracy
[22],model Jacobian norm [16], the linear pixel-space interpolation metrics of Fridovich-Keil et al.
[9],and our own Fourier amplitude and phase interpolation metrics.
We study these metrics over a wide range of robust and nonrobust models, including sparse models from [7] on CIFAR-10 and pretrainedmodels from CLIP [25, 34] and RobustBench [4] on ImageNet.
Methods
Datasets.
We consider two standard image classification tasks: CIFAR-10 [20] and ImageNet [5].
CIFAR-10 consists of low-resolution (32 by 32 pixels) color images from 10 animal and objectclasses, divided into 50000 training and 10000 test images; ImageNet consists of higher-resolution(cropped to 224 by 224 pixels) color images from 1000 animal and object classes, with more than amillion training images and 50000 validation images.
For each of these "in-distribution" (ID) datasetswe consider two "out of distribution" (OOD) datasets: one defined by a set of synthetic corruptions ofthe original test/validation images (CIFAR-10-C [14] and ImageNet-C [13], respectively) and anotherdefined by a re-collection of new test images following as closely as possible the original datasetcreation procedure (CIFAR-10.1 [28] and ImageNetV2 [29]).
Models.
In our CIFAR-10 experiments we use 3 different model architectures: Conv8 [8], ResNet18[12], and VGG16 [32].
For each model architecture, we consider a range of model pruning strategiesthat have been studied for their effect on OOD robustness [7, 17] as some of these pruned modelswere demonstrated to have higher OOD robustness than dense models.
Namely, lottery-ticket stylepruning methods [8, 30, 27, 6] were able to provide robustness gains on CIFAR-10-C [7].
We outlinethe various pruning techniques and sparsity levels we utilized in detail in Section 4.2.
In our ImageNet experiments we use 28 standard (nonrobust) pretrained models from Torchvision [21],5 pretrained models from the ImageNet-C leaderboard on RobustBench [4], and 11 models obtainedby robustness-enhancing weight-space interpolation [34] between zero-shot and ImageNet-finetuned CLIP ViT-B/16 [25].
Previously-proposed metrics: ID accuracy, Jacobian norm, pixel interpolation.
Because OODrobustness is a complex and multi-faceted problem, we make use of multiple previously-proposedmodel measurements and apply them to study effectively robust models.
To the best of our knowledge,our work is the first application of most of these metrics (except for in-distribution accuracy) to thestudy of effective robustness.
Specifically, we evaluate four previously-proposed metrics that havebeen observed to correlate with some notion of generalization: accuracy on unseen in-distributiondata ("ID accuracy")
[22], Jacobian norm [16], and within-class and between-class pixel interpolationhigh frequency fraction [9].
These latter metrics are analogous to the high frequency fractions wecompute for amplitude and phase interpolation (described below), except that interpolating pathssimply interpolate in pixel space between two images, where the two images are from either thesame class ("within-class") or different classes ("between-class")
Prior work observed thatin-distribution generalization typically improves as within-class high frequency fraction decreasesand between-class high frequency fraction increases [9].
Reducing the norm of the Jacobian of modeloutputs with respect to input data can push the decision boundary away from the training points,providing robustness to random perturbations of the input data [16].
Thus, models with smaller Jacobian norms may perform better on corrupted/shifted data.
We estimate the norm of the Jacobianusing a random-projection-based approach [16]; further details can be found in the supplement.
Fourier interpolation metrics.
We also introduce novel Fourier interpolation metrics to capturemore information about model behavior that may help answer the robustness puzzle.
Inspiredby the pixel-space interpolation procedure of Fridovich-Keil et al.
[9], we propose evaluating thesmoothness of a model’s predictions along image paths that perturb the Fourier amplitude or phaseinformation of one test image towards another, while preserving the Fourier phase or amplitude(respectively) information of the original image.
The intuition for this type of interpolation is thatmost of the semantic content of an image is contained within its Fourier phases, which encodestructural information like edges.
Accordingly, we can perturb the Fourier amplitude of an imagewithout destroying its semantic meaning, and we can reliably destroy semantic content by perturbing
Fourier phase.
Example Fourier amplitude and phase interpolating paths on ImageNet are shown in
Figure 1; example paths on CIFAR-10 are included in the appendix.
More precisely, we compute an interpolating path by first randomly selecting two images x 0 andx 1 from the ID test/validation set.
We perform standard preprocessing on each image, which involves cropping to a standard size (for ImageNet) and normalizing each pixel by the training mean andstandard deviation (for both CIFAR-10 and ImageNet).
We then compute the two-dimensionaldiscrete Fourier transform (DFT) of each image and separate the complex-valued results intoamplitude and phase components a i and p i for i ∈ { 0 , 1 } .
To construct an image along an amplitudeinterpolating path, we retain the original phase p 0 and interpolate the low-frequency amplitude as(1 − λ ) a 0 + λ a 1 (while preserving the high frequency amplitude), where λ ∈ [0 , 1] , and producean image via the inverse discrete Fourier transform.
Phase interpolation is defined analogously;we retain the original amplitude and interpolate the low-frequency phase.
We choose a frequencythreshold for what is considered low frequency for interpolation based on visual inspection to ensurethat amplitude interpolation preserves semantic meaning and phase interpolation destroys it.
Weinterpolate the lowest 40% of image frequencies for both amplitude and phase on CIFAR-10, thelowest 20% of image frequencies for phase on ImageNet, and all image frequencies for amplitudeon ImageNet.
We choose 100 evenly-spaced λ values to define each path, and choose 5000 randompaths for CIFAR-10 and 1000 for ImageNet.
The images in Figure 1 are rescaled for visualization(essentially undoing the initial pixel-wise normalization).
We evaluate each model on each path, producing a probability vector over the available classesfor each image along the path.
We compute two metrics based on these path predictions: highfrequency fraction (HFF) and consistent distance (CD).
For high frequency fraction, we take theone-dimensional DFT of the 100 model predictions along the path, average the Fourier amplitudesamong all the classes, and compute the fraction of the total amplitude above a frequency threshold.
This metric is always between 0 and 1 (usually between 0.1 and 0.3).
The higher the average highfrequency fraction , the more sensitive a model is to the given Fourier perturbation of its input images.
We also consider consistent distance , which is defined as the index of the first image along the paththat the model classifies differently (by highest softmax prediction) than the original image.
Thismetric is always between 1 and 100, and is often at least 40.
Consistent distance is intended as a moreintuitive metric to capture the same notion of robustness to corruptions of image Fourier content.
Inparticular, Fourier amplitude corruptions change image statistics but do not hamper human semanticidentification; a more robust model in this sense should have a lower high frequency fraction and ahigher consistent distance on average compared to a less robust model.
Results
We are interested in understanding when (under what conditions on the distribution shift) and why(via model properties) neural networks can exhibit effective robustness.
We perform a systematicstudy of three tunable "knobs" that are available to the neural network practitioner and have beenshown to impact OOD robustness: pruning [7], data augmentation [15], and weight ensembling [34].
For each knob, we evaluate a benchmark set of pretrained CIFAR-10 or ImageNet models on theoriginal in-distribution test set, several OOD test sets, and a suite of model property metrics capturing local smoothness (via model Jacobian norm), frequency response to pixel-space interpolation withinand between classes [9], and Fourier amplitude and phase sensitivity (via high frequency fraction andconsistent distance ).
Representative results from our analysis of these three knobs are presented in Sections 4.2, 4.3, and 4.4.
For each dataset we consider one natural distribution [-shifts: CIFAR-10.
-]shift (CIFAR-10.1 or
ImageNetV2) as well as six synthetic corruptions (from CIFAR-10-C or ImageNet-C) comprised oftwo low-frequency corruptions, two mid-frequency corruptions, and two high-frequency corruptions(in that order).
Full results on all the corruptions in CIFAR-10-C and ImageNet-C are deferredto the supplement. In all figures, we show 95% confidence intervals around each measurement(Clopper-Pearson for accuracy measurements and Gaussian bounds for averages of other metrics). We begin by comparing the spectral properties of different distribution shifts in Section 4.1.
In thefollowing sections we find that the relationship between model properties and OOD robustness isdependent upon these spectral statistics of the specific distribution shift. This analysis highlights thedifficulty of using a single model property to understand OOD performance but also indicates thatour proposed metrics may play an important role in solving the OOD puzzle. 4.1 Spectral characterization of distribution shifts To explore the behavior of models on OOD data, it is beneficial to characterize the nature of the
OOD data with respect to the ID test data. Specifically, considering the difference in power spectraldensity (PSD) between each of the OOD test sets and the original ID test set allows us to categorizeeach OOD test set as being concentrated on low, mid, or high frequency perturbations with respectto the ID test set. We perform this characterization for CIFAR-10.1 and each of the 15 corruptionsin CIFAR-10-C; computational details for PSD are provided in the appendix. Figure 2 shows thedistribution shift PSDs for CIFAR-10.1 together with two low (brightness, contrast), mid (defocusblur, pixelate), and high (gaussian noise, impulse noise) frequency CIFAR-10-C corruptions.
Of note,this PSD characterization illustrates that the OOD shift encountered on CIFAR-10.1 data is composedof low-frequency information, much like the brightness and contrast CIFAR-10-C corruptions.
[Figure]
4.2 When and why does model pruning confer effective robustness?
Diffenderfer et al.
[7] demonstrated the nuanced behavior that model pruning has on OOD robustness,specifically with respect to the distribution shift from CIFAR-10 to CIFAR-10-C.
Notably, prunedmodels are capable of providing superior OOD robustness compared to dense, or unpruned, models.
Furthermore, the effect, either positive or negative, and degree of robustness arising from modelpruning is dependent on both the model architecture and the pruning algorithm.
It remains unknownwhich properties of these pruned models contribute to their robustness on OOD data.
In an effort to better understand these findings, we further investigate this behavior by consideringthe three categories of pruning used in [7]: traditional (fine-tuning [11], gradual magnitude pruning[37]), rewinding lottery-tickets (weight-rewinding [8], learning-rate rewinding [30]), and initializationlottery-tickets (edgepopup [27], biprop [6]).
Note that we often abbreviate "lottery tickets" as LT.
We provide details on each of the pruning techniques in the supplement.
For each pruning strategyand each architecture, we prune models to 50%, 60%, 80%, 90%, and 95% sparsity.
For all pruningmethods, pruning is performed in an unstructured (i.e., individual weights are pruned) and globalmanner (i.e., prune to a given sparsity across the entire network).
Additionally, for traditionaland initialization LTs, pruning was performed in a layerwise fashion (i.e., where each networklayer is pruned to the given sparsity level).
In Figure 3 we investigate when (on which types ofdistribution shifts) and in Table 1 we study why (via corresponding model property measurements)these differently-pruned CIFAR-10 models achieve their varying degrees of robustness. [Figure] [Table]
Because different model architectures respond differently to different pruning methods, in eachsubplot of Figure 3 we color-code models by their architecture and fit a separate probit-domain
OOD vs. ID accuracy regression line to each architecture, so that each regression captures a rangeof different pruning amounts (what fraction of weights are pruned).
We display these regressionswith opacity proportional to the fit quality R 2 , so that regression lines that fit the data well are darkand easily seen, while lower-quality linear fits are more transparent.
In the corresponding Tables 1,each reported slope ( m ) and R 2 value is the average of the corresponding values among the threearchitectures we consider.
For ease of reading, we bold average R 2 values above 0.5, and amongthese entries we highlight the best (bright green) and second-best (faded green) R 2 values for eachpruning method on each OOD dataset.
Although these results are nuanced and raise additional openquestions, we summarize a few key takeaways from our CIFAR-10 model pruning experiments:
• As reported in Miller et al.
[22], ID accuracy is often a strong predictor of OOD accuracy; thistrend holds across different pruning methods.
However, the linear trend breaks down for mid andhigh frequency image corruptions like pixelate, Gaussian noise, and impulse noise.
• Even for low-frequency corruptions, different model architectures might lie on either the same ordifferent robustness lines (as evidenced by comparing brightness and contrast).
• Initialization LT pruning confers robustness to high-frequency corruptions and can induce a negativecorrelation between ID and OOD accuracy for these corruptions.• Jacobian norm is rarely a good predictor of OOD accuracy, but it does correlate negatively (lowernorm is better) for models pruned by rewinding methods on mid-frequency corruptions.•
As reported in [9], lower high frequency fraction HFF (i
e. smoother predictions) for within-classpixel interpolation is a good predictor of OOD accuracy, at least for low-frequency distributionshifts where ID accuracy is also predictive.• Amplitude and phase interpolation are sometimes predictive of OOD accuracy, particularly forinitialization LT pruning, and in the expected direction of correlation (a better model should havelower HFF and higher CD).
However, even these metrics fail to explain most of the robustnessbehavior for high (and some mid) frequency distribution shifts on CIFAR-10.
4.3 When and why does data augmentation confer effective robustness?
A common practice to protect against distribution shift is to train with data augmentation [15].
Accordingly, we revisit each of the CIFAR-10 models considered in Section 4.2 with three differentvariations of data augmentation used during training: no augmentation (clean), AugMix [15], and
Gaussian noise augmentation [19].
In Figure 4 we investigate when (on which types of distribution shifts) and in Table 2 we study why (via corresponding model property measurements) thesedifferently-augmented CIFAR-10 models achieve varying degrees of robustness.
As in Figure 3 and
Table 1, we fit a separate regression line to each model architecture and average the resulting slopeand R 2 values.
The models that contribute to each regression line therefore share an architecture andan augmentation strategy, but vary by pruning method and amount.
Our findings in this experimentecho those in Section 4.2, but we make the following additional observations:
• The closer the match between the training data and the OOD evaluation data in terms of theirspectral profile, the better ID accuracy, within-class pixel interpolation, and amplitude interpolation HFF and CD are at predicting OOD accuracy.
This is evident from the high R 2 values forthese metrics when predicting defocus blur accuracy for models trained with AugMix, and whenpredicting Gaussian noise or impulse noise accuracy for models trained with Gaussian noise.•
The closer the match (in terms of PSD) between the training data and the OOD data, the better the OOD accuracy.
This unsurprising result is evident from the higher accuracies of AugMix-trainedmodels on contrast and defocus blur corruptions and the higher accuracies of Gaussian-noise-trainedmodels on Gaussian noise and impulse noise corruptions.
4.4 CLIP: Weight ensembling a robust model on ImageNet
Perhaps the most stunning example of OOD robustness for image classification is CLIP [25], whichshowed that a large model pretrained on a massive dataset and evaluated zero-shot on ImageNetachieves effective robustness on many OOD test sets simultaneously.
Andreassen et al.
[1] foundthat finetuning these robust models towards ImageNet improves ID accuracy on ImageNet buterodes effective robustness, and Wortsman et al.
[34] offered a "best of both worlds" solution byinterpolating in weight space between a pretrained zero-shot model and its finetuned counterpart,improving ID accuracy on ImageNet without sacrificing robustness.
Accordingly, we study a set of11 CLIP models obtained by interpolating the pretrained and finetuned versions of a CLIP ViT-B/16model [25, 34] with various relative weights on the two models.
We compare this CLIP model setto a benchmark set of 28 standard (nonrobust)
ImageNet pretrained models from Torchvision [21]as well as 5 pretrained models from the RobustBench [4] ImageNet-C leaderboard; these modelstypically use data augmentation to improve corruption performance but are not nearly as effectivelyrobust as the CLIP models.
We make the following observations in Figure 5 and Table 3:
• Although in-distribution accuracy is a reasonable predictor of OOD accuracy for most models and
distribution shifts, it performs markedly worse when predicting OOD accuracy for CLIP models(and a couple of effectively robust RobustBench models on some OOD shifts).
• Fourier interpolation metrics, particularly HFF and CD for amplitude interpolation, are strong predictors (often R 2 ≥ 0 . 9 ) of OOD performance for CLIP models across OOD shifts.
Thisobservation helps answer why CLIP models are more robust: they are less sensitive to perturbations of Fourier amplitude, which preserve image semantics for humans but confuse nonrobust models.
[Figure] [Table]
Discussion
Our work is only the beginning of a true understanding of what makes models effectively robust todistribution shifts.
For example, Section 4.2 showed different types of model pruning can conferrobustness to different distribution shifts, and while some of this robustness can be explained byimproved behavior in response to amplitude and phase perturbations, much remains an open mystery.
As we saw in Section 4.3 (and has been noted by prior work [36, 15]), models can achieve robustnessto some distribution shifts by training with data augmentations designed to imitate the expected OODdata.
This approach can be a powerful way to mitigate expected test-time distribution shifts but leavesopen the question of how to prepare for the unexpected.
We learned in Section 4.4 that one way in which CLIP models achieve OOD robustness is by reducingtheir sensitivity to perturbations of Fourier amplitude, perhaps making them more attuned to humanperceptions of semantic meaning.
This finding provides an exciting opportunity for future work to both confirm our results across additional robust models and incorporate these Fourier interpolation metricsinto training paradigms, for instance as regularizers, to explicitly encode robustness in future models [Figure] [Figure] [Table]
Our results provide a new perspective on the effective robustness problem by showing that no onesingle metric rules them all.
This dictates the need for a multi-faceted analysis to understand OODrobustness, which is in sharp contrast to the in-distribution setting where a single property such asmodel flatness can reasonably explain generalization performance.
RobustNets benchmark.
To aid further research into understanding, modeling, and designing for
OOD robustness, we will publish a database of our pretrained models on CIFAR-10 (with diverseeffective robustness) with the full range of model architectures, data augmentations, and pruning typeswe study.
The models we study on ImageNet are already available publically.
We will also publishthe ID and OOD accuracy values for each model, and our code used to evaluate Jacobian norm andhigh frequency fraction and consistent distance for Fourier amplitude and phase interpolation, so thatfuture researchers can apply it to study new and more robust models as they become available.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] Limitations are included in the
discussion section.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A](b)
Did you include complete proofs of all theoretical results?
[N/A]3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[No] Code will be
made public upon acceptance.
We do also provide sufficient details in the text that
anyone could reproduce our code.
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes] Most of these are provided in the methods section, and some are
in the supplement.
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[Yes] We use 95% confidence intervals in all of our figures.
(d) Did you include the total amount of compute and the type of resources used (e.g.,type of GPUs, internal cluster, or cloud provider)?
[No] Our experiments did require
substantial compute to evaluate all of our models on all of the interpolation paths, but
we use pretrained models so we do not require any compute for training.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] We cite the creators
of all the models and datasets we use.
(b) Did you mention the license of the assets?
[No] Licenses for individual assets can be
found by visiting the websites of the corresponding creators.
(c) Did you include any new assets either in the supplemental material or as a URL?
[No] We do not include new assets in the submission; however, we will release our
RobustNets database of pretrained models evaluated in this paper upon acceptance.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]