Models Out Of Line: A Fourier Lens On Distribution Shift Robustness
Abstract
Improving the accuracy of deep neural networks on out-of-distribution (OOD) data is critical to an acceptance of deep learning in real world applications.
It has been observed that accuracies on in-distribution (ID) versus OOD data follow a linear trend and models that outperform this baseline are exceptionally rare (and referred to as “effectively robust”).
Recently, some promising approaches have been developed to improve OOD robustness: model pruning, data augmentation, and ensembling or zero-shot evaluating large pretrained models.
However, there still is no clear understanding of the conditions on OOD data and model properties that are required to observe effective robustness.
We approach this issue by conducting a comprehensive empirical study of diverse approaches that are known to impact
OOD robustness on a broad range of natural and synthetic distribution shifts of
CIFAR-10 and ImageNet.
In particular, we view the “effective robustness puzzle” through a Fourier lens and ask how spectral properties of both models and OOD data correlate with OOD robustness.
We ﬁnd this Fourier lens offers some insight into why certain robust models, particularly those from the CLIP family, achieve
OOD robustness.
However, our analysis also makes clear that no known metric is consistently the best explanation of OOD robustness.
Thus, to aid future research into the OOD puzzle, we address the gap in publicly-available models with effective robustness by introducing a set of pretrained CIFAR-10 models— RobustNets — with varying levels of OOD robustness.
Introduction
Deep learning (DL) holds great promise for solving difﬁcult real-world problems in domains such as healthcare, autonomous driving, and cyber-physical systems.
The major roadblock in adopting DL for real-world applications is that real-world data often deviates from the training data due to noise, corruptions or other changes in distribution that may have temporal or spatial causes [3, 20, 41, 44].
DL models are known to be highly brittle under such distribution shifts, which raises the risk of making incorrect predictions with harmful consequences.
Designing approaches to learn models that achieve both high accuracy on in-distribution data and high robustness to distribution shifts is paramount to the safe adoption of DL in real-world applications.
Several recent works [1, 35] have noticed an intriguing phenomenon: accuracy on in-distribution (ID) versus accuracy on distribution-shifted data obeys a linear trend across a wide range of models, corruption benchmarks, and prediction tasks.
Unfortunately, as nearly all current DL models bear a substantial loss under distribution shifts, this linear relationship implies that our current training strategies are insufﬁcient to bridge the gap between ID and out-of-distribution (OOD) performance.
However, despite being exceedingly rare, models that break this linear trend to obtain effective robustness [1] exist—such models defy explanation by the wisdom that OOD accuracy and ID accuracy should correlate strongly: they are “out of line”.
Identifying when and why such models arise is key to overcoming OOD brittleness in DL models.
Related research efforts have focused on improving OOD robustness via new techniques such as model pruning [8], data augmentation [5, 21], and ensembling large pre-trained models [37, 52].
While these efforts have created signiﬁcant excitement in the ML robustness community, it remains unclear (a) under which conditions on OOD data these methods can improve effective robustness (ER), and (b) which underlying model properties make a DNN effectively robust.
A more in-depth understanding of these phenomena is likely to help in bridging the gap between ID and OOD performance.
Towards achieving this goal, we carry out a comprehensive empirical study leveraging models trained using each of the aforementioned state-of-the-art approaches to improving OOD robustness.
We analyze the impact of these robustness improvement methods on various model architectures, two clean datasets (CIFAR-10 and ImageNet), and a broad range of natural and synthetic distribution shifts.
In particular, we view the OOD robustness puzzle through a Fourier lens and ask how spectral properties of OOD data and models inﬂuence their effective robustness.
In this process, we design new metrics that capture Fourier sensitivity of models as test data moves farther away from the training data manifold.
We make the following contributions:
• A new perspective on the state of OOD robustness: across diverse models, metrics, and distribution shifts, we observe that no one single metric rules them all, not even in-distribution accuracy.
This accents the potential need for a multi-faceted approach to understanding OOD robustness, in contrast to ID generalization where single properties like model ﬂatness can enjoy predictive success across an array of models [28].
• Design of Fourier sensitivity metrics: our new metrics quantify various model properties, including spectral ones that strongly correlate with the OOD robustness of ensembled CLIP models [52], explaining this robustness much better than ID accuracy.
• A public collection of effectively robust models: to help crack the OOD robustness puzzle, we make available (at https://github.com /sarafridov/RobustNets )
a RobustNets dataset of CIFAR10 models with effective robustness that stems from training under a variety of data augmentation and pruning schemes.
Related work
Distribution shift fragility is a well-documented phenomenon among neural networks [40, 41, 20, 19, 34] in which a model trained on an ID dataset performs markedly worse when evaluated on OOD data, even when humans ﬁnd the OOD data just as easy to classify.
These
OOD accuracy gaps often follow a linear relationship between ID and OOD accuracy, but the slope and intercept of the linear trendline varies depending on the speciﬁc ID and OOD datasets [34, 15].
In our work, we focus on two popular image classiﬁcation benchmark ID datasets, CIFAR-10 [30] and ImageNet [6], and an array of OOD benchmarks for each.
While most models follow the linear ID-OOD trendlines of Miller et al.
[34], some models are “effectively robust" and appear above the trendline, with OOD accuracy higher than expected given
ID accuracy [1].
These models are quite rare, and thus far have only been shown to arise through a few training paradigms: by pruning a model as a sort of regularization [8], by training with data augmentation intended to mimic the expected OOD test data [5, 21], or by pretraining on a larger and more diverse dataset and evaluating zero-shot [37], partially ﬁnetuning [1], or interpolating the weights of a zero-shot and a ﬁnetuned model [52].
Although as a community we are beginning to uncover robust training methods, it remains a mystery why these particular methods achieve robustness on particular distribution shifts.
We study this mystery via a Fourier lens based on prior work involving both image frequency and function frequency analyses.
Ortiz-Jimenez et al.
[36], Yin et al.
[55], and Sun et al.
[47] study model sensitivity to Fourier perturbations of the input images and analyze how different data augmentations produce different Fourier sensitivities (and correspondingly robustness to image perturbations of different frequencies).
Chen et al.
[4] showed that exchanging amplitude and phase information between images (as a regularization scheme) has connections to robustness as well.
Another line of research focuses on spectral bias [13, 2, 54, 38], wherein models prioritize learning low frequency functions over the input space.
This bias towards low function frequencies can also affect robustness and is inﬂuenced by training hyperparameters like data augmentation and weight decay [13].
Among the model properties we study as potential predictors of OOD accuracy are ID accuracy [34], model Jacobian norm [24], the linear pixel-space interpolation metrics of Fridovich-Keil et al.
[13], and our own Fourier amplitude and phase interpolation metrics.
We study these metrics over a wide range of robust and nonrobust models, including sparse models from [8] on CIFAR-10 and pretrained models from CLIP [37, 52] and RobustBench [5] on ImageNet.
Methods
Datasets.
We consider two standard image classiﬁcation tasks: CIFAR-10 [30] and ImageNet [6].
For each of these "in-distribution" (ID) datasets we consider a variety of "out of distribution" (OOD) datasets, both natural and synthetic.
For CIFAR-10 we consider CIFAR-10.1 [40] and CIFAR-10-C [20]; for ImageNet we consider ImageNetV2 [41], ImageNet-C [19], ImageNet-R [23], ImageNet-A [9], ImageNet-Sketch [50], and ObjectNet [10].
Models.
Our CIFAR-10 experiments use 3 different model architectures: Conv8 [12], ResNet [17], and VGG16 [45].
For each model architecture, we consider a range of model pruning strategies that have been studied for their effect on OOD robustness [8, 25] as some of these pruned models were demonstrated to have higher OOD robustness than dense models.
Namely, lottery-ticket style pruning methods [12, 42, 39, 7] were able to provide robustness gains on CIFAR-10-C [8].
We outline the various pruning techniques and sparsity levels we utilized in detail in Section 4.1.
In our ImageNet experiments we use 28 standard (nonrobust) pretrained models from Torchvision [33], 5 pretrained models from the ImageNet-C leaderboard on RobustBench [5], and 33 models obtained by robustness-enhancing weight-space interpolation [52] between zero-shot and ImageNetﬁnetuned CLIP ViT-B/16, ViT-B/32, and ViT-L/14 [37].
More details on the ImageNet models we use from Torchvision and RobustBench can be found in Section A.1.
Previously-proposed metrics: ID accuracy, Jacobian norm, pixel interpolation.
Because OOD robustness is a complex and multi-faceted problem, we make use of multiple previously-proposed model measurements and apply them to study effectively robust models.
To the best of our knowledge, our work is the ﬁrst application of most of these metrics (except for in-distribution accuracy) to the study of effective robustness.
Speciﬁcally, we evaluate four previously-proposed metrics that have been observed to correlate with some notion of generalization: accuracy on unseen in-distribution data ("ID accuracy") [34], Jacobian norm [24], and within-class and between-class pixel interpolation high frequency fraction [13].
These latter metrics are analogous to the high frequency fractions we compute for amplitude and phase interpolation (described below), except that interpolating paths simply interpolate in pixel space between two images, where the two images are from either the same class ("within-class") or different classes ("between-class")
Prior work observed that in-distribution generalization typically improves as within-class high frequency fraction decreases and between-class high frequency fraction increases [13].
Reducing the norm of the Jacobian of model outputs with respect to input data can push the decision boundary away from the training points, providing robustness to random perturbations of the input data [24].
Thus, models with smaller Jacobian norms may perform better on corrupted/shifted data.
We estimate the norm of the
Jacobian using a random-projection-based approach [24]; further details can be found in Section A.4.
Fourier interpolation metrics.
We also introduce novel Fourier interpolation metrics to capture more information about model behavior that may help answer the robustness puzzle.
Inspired by the pixel-space interpolation procedure of Fridovich-Keil et al.
[13], we propose evaluating the smoothness of a model’s predictions along image paths that perturb the Fourier amplitude or phase information of one test image towards another, while preserving the Fourier phase or amplitude (respectively) information of the original image.
The intuition for this type of interpolation is that most of the semantic content of an image is contained within its Fourier phases, which encode structural information like edges.
Accordingly, we can perturb the Fourier amplitude of an image without destroying its semantic meaning, and we can reliably destroy semantic content by perturbing
Fourier phase.
Example Fourier amplitude and phase interpolating paths on ImageNet are shown in
Figure 1; example paths on CIFAR-10 and further computational details are included in Section A.3.
Note that the human visual system is robust to both types of corruption to some extent (see [51]).
More precisely, we compute an interpolating path by ﬁrst randomly selecting two images x 0 and xfrom the ID test/validation set.
We perform standard preprocessing on each image, which involves cropping to a standard size (for ImageNet) and normalizing each pixel by the training mean and standard deviation (for both CIFAR-10 and ImageNet).
We then compute the two-dimensional discrete Fourier transform (DFT) of each image and separate the complex-valued results into amplitude and phase components a i and p i for i 2 { 0 , 1 } .
To construct an image along an amplitude interpolating path, we retain the original phase p 0 and interpolate the low-frequency amplitude as (1   ) a [-
-]0 +  a(while preserving the high frequency amplitude), where  2 [0 , 1] , and produce an image via the inverse discrete Fourier transform.
Phase interpolation is deﬁned analogously; we retain the original amplitude and interpolate the low-frequency phase.
We interpolate the lowest 40% of image frequencies for both amplitude and phase on CIFAR-10, the lowest 20% of image frequencies for phase on ImageNet, and all image frequencies for amplitude on ImageNet.
We choose 100 evenly-spaced  values to deﬁne each path, and choose 5000 random paths for CIFAR-10 and 7000 for ImageNet.
The images in Figure 1 are rescaled for visualization (essentially undoing the initial pixel-wise normalization).
We evaluate each model on each path, producing a probability vector over the available classes for each image along the path.
We compute two metrics based on these path predictions: high frequency fraction (HFF) and consistent distance (CD).
For high frequency fraction, we take the one-dimensional DFT of the 100 model predictions along the path, average the Fourier amplitudes among all the classes, and compute the fraction of the total amplitude above a frequency threshold (10% of the maximum frequency).
This metric is always between 0 and 1 (usually between 0.1 and 0.3).
The higher the average high frequency fraction , the more sensitive a model is to the given
Fourier perturbation of its input images.
We also consider consistent distance , which is deﬁned as the index of the ﬁrst image along the path that the model classiﬁes differently (by highest softmax prediction) than the original image.
This metric is always between 1 and 100, and is often at least 40.
Consistent distance is intended as a more intuitive metric to capture the same notion of robustness to corruptions of image Fourier content.
In particular, Fourier amplitude corruptions change image statistics but do not hamper human semantic identiﬁcation; a more robust model in this sense should have a lower high frequency fraction and a higher consistent distance on average compared to a less robust model.
Results
We are interested in understanding when (under what conditions on the distribution shift) and why (via model properties) neural networks can exhibit effective robustness.
Although our experiments are correlational rather than causal, they provide evidence for and against various hypotheses regarding why effective robustness might emerge.
We hope that future work will make use of our proposed metrics and released model database RobustNets to answer the causal questions raised by our experiments, perhaps following Dziugaite et al.
We perform a systematic study of three model interventions that have been shown to impact
OOD robustness: pruning [8], data augmentation [5, 21], and weight ensembling [52].
For each intervention, we evaluate a benchmark set of pretrained CIFAR-10 and/or ImageNet models on the original in-distribution test set, several OOD test sets, and a suite of model property metrics capturing local smoothness (via model Jacobian norm), frequency response to pixel-space interpolation within and between classes [13], and Fourier amplitude and phase sensitivity (via high frequency fraction and consistent distance ).
Representative results from our analysis of these three interventions are presented in Sections 4.1, 4.2, 4.3, and A.10.
For each dataset we consider a mixture of natural and synthetic distribution shifts: CIFAR-10.
[40] and CIFAR-10-C [20], and ImageNet-V2 [41],
ImageNet-R [23], ObjectNet [10], ImageNet-A
[9], ImageNet-Sketch [50], and ImageNet-C [19].
Since CIFAR-10-C and ImageNet-C are each composed of 15 synthetic corruptions ranging from low to high frequency, we include results on representative corruptions of each frequency band in the main text and defer results on the remaining corruptions to the appendix.
In all ﬁgures, we show 95% conﬁdence intervals around each measurement (Clopper-Pearson for accuracy measurements and Gaussian bounds for averages of other metrics).
4.1 When and why does model pruning confer effective robustness?
Diffenderfer et al.
[8] demonstrated the nuanced behavior that model pruning has on OOD robustness, speciﬁcally with respect to the distribution shift from CIFAR-10 to CIFAR-10-C.
Notably, pruned models are capable of providing superior OOD robustness compared to dense, or unpruned, models.
Furthermore, the effect, either positive or negative, and degree of robustness arising from model pruning is dependent on both the model architecture and the pruning algorithm.
It remains unknown which properties of these pruned models contribute to their robustness on OOD data.
In an effort to better understand these ﬁndings, we further investigate this behavior by considering the three categories of pruning used in Diffenderfer et al.
[8]: traditional [16, 57], rewinding lotterytickets [12, 42], and initialization lottery-tickets [39, 7].
Note that we often abbreviate "lottery tickets" as LT.
We provide details on each of the pruning techniques in Section A.2.
For each pruning strategy and each architecture, we prune models to 50%, 60%, 80%, 90%, and 95% sparsity.
For all pruning methods, pruning is performed in an unstructured (i.e., individual weights are pruned) and global manner (i.e., prune to a given sparsity across the entire network).
Additionally, for traditional and initialization LTs, pruning was performed in a layerwise fashion (i.e., where each network layer is pruned to the given sparsity level).
In Figure 2 we investigate when (on which types of distribution shifts) and in Table 1 we study why (via corresponding model property measurements) these differently-pruned CIFAR-10 models achieve their varying degrees of robustness.
Because different model architectures respond differently to different pruning methods, in each subplot of Figure 2 we color-code models by their architecture and ﬁt a separate probit-domain
OOD vs. ID accuracy regression line to each architecture, so that each regression captures a range of different pruning amounts (what fraction of weights are pruned).
We display these regressions with opacity proportional to the ﬁt quality R 2 , so that regression lines that ﬁt the data well are dark and easily seen, while lower-quality linear ﬁts are more transparent.
In the corresponding Tables 1, each reported slope ( m ) and R 2 value is the average of the corresponding values among the three architectures we consider.
For ease of reading, we bold average R 2 values above 0.5, and among these entries we highlight the best (bright green) and second-best (faded green)
R 2 values for each pruning method on each OOD dataset.
Although these results are nuanced and raise additional open questions, we summarize a few key takeaways from our CIFAR-10 model pruning experiments:
• As reported in Miller et al.
[34], ID accuracy is often a strong predictor of OOD accuracy; this trend holds across different pruning methods.
However, the linear trend breaks down for mid and high frequency image corruptions like pixelate, Gaussian noise, and impulse noise.
• Even for low-frequency corruptions, different model architectures might lie on either the same or different robustness lines (as evidenced by comparing brightness and contrast).
• When trained without data augmentation, higher effective robustness is present on several mid- and high-frequency corruptions by non-residual architectures (i
e. Conv8 and VGG16)
(see Figures
and 8).
However, when training with data augmentation, less variation in effective robustness is realized by different architectures (see Figures 3 and 9).
[Figure]
[Table]
• Initialization LT pruning confers robustness to high-frequency corruptions and can induce a negative correlation between ID and OOD accuracy for these corruptions
• Jacobian norm is rarely a good predictor of OOD accuracy, but it does correlate negatively (lower norm is better) for models pruned by rewinding methods on mid-frequency corruptions.
• As reported in Fridovich-Keil et al.
[13], lower high frequency fraction HFF (i.e. smoother predictions) for within-class pixel interpolation is a good predictor of OOD accuracy, at least for low-frequency distribution shifts where ID accuracy is also predictive.
• Amplitude and phase interpolation are sometimes predictive of OOD accuracy, particularly for initialization LT pruning, and in the expected direction of correlation (a better model should have lower HFF and higher CD).
However, even these metrics fail to explain most of the robustness behavior for high (and some mid) frequency distribution shifts on CIFAR-10.
Please see Figure 10 for visualizations of the data ﬁt associated with the most predictive metric for each distribution shift and pruning approach.
4.2 When and why does data augmentation confer effective robustness?
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table] [Figure] [Table] A common practice to protect against distribution shift is to train with data augmentation [5, 21].
Accordingly, we revisit each of the CIFAR-10 models considered in Section 4.1 with three different variations of data augmentation used during training: no augmentation (clean), AugMix [21], and
Gaussian noise augmentation [29].
In Figure 3 we investigate when (on which types of distribution shifts) and in Table 2 we consider candidate explanations for why (via corresponding model property measurements) these differently-augmented CIFAR-10 models achieve varying degrees of robustness.
As in Figure 2 and Table 1, we ﬁt a separate regression line to each model architecture and average the resulting slope and R 2 values.
The models that contribute to each regression line therefore share an architecture and an augmentation strategy, but vary by pruning method and amount.
Our ﬁndings in this experiment echo those in Section 4.1, but we make the following additional observations:
• The closer the match between the training data and the OOD data in terms of their spectral proﬁle (see Section A.5 for visualizations), the better ID accuracy, within-class pixel interpolation, and amplitude interpolation HFF and CD are at predicting OOD accuracy.
This is evident from the high R 2 values for these metrics when predicting defocus blur accuracy for models trained with AugMix, and when predicting Gaussian or impulse noise accuracy for models trained with Gaussian noise.
• The closer the spectral match between the training data and the OOD data, the better the OOD accuracy.
This unsurprising result is evident from the higher accuracies of AugMix-trained models on contrast and defocus blur corruptions and the higher accuracies of Gaussian-noise-trained models on Gaussian noise and impulse noise corruptions.
Please see Figure 11 for visualizations of the data ﬁt associated with the most predictive metric for each distribution shift and data augmentation approach.
4.3 CLIP: Weight ensembling a robust model on ImageNet
Perhaps the most stunning example of OOD robustness for image classiﬁcation is CLIP [37], which showed that a large model pretrained on a massive dataset and evaluated zero-shot on ImageNet achieves effective robustness on many OOD test sets simultaneously.
Andreassen et al.
[1] found that ﬁnetuning these robust models towards ImageNet improves ID accuracy on ImageNet but erodes effective robustness, and Wortsman et al.
[52] offered a "best of both worlds" solution by interpolating in weight space between a pretrained zero-shot model and its ﬁnetuned counterpart, improving ID accuracy on ImageNet without sacriﬁcing robustness.
Accordingly, we study a set of 33 CLIP models obtained by interpolating the pretrained and ﬁnetuned versions of three CLIP ViT models (B/16, B/32, and L/14) [37, 52] with various relative weights on the two models.
We compare this CLIP model set to a benchmark set of 28 standard (nonrobust)
ImageNet pretrained models from
Torchvision [33] as well as 5 pretrained models from the RobustBench [5] ImageNet-C leaderboard; these models typically use data augmentation to improve corruption performance but are not nearly as effectively robust as the CLIP models.
We make the following observations in Figure 4 and Table 3:
• Although in-distribution accuracy is a reasonable predictor of OOD accuracy for most models and distribution shifts, it performs markedly worse when predicting OOD accuracy for CLIP models (and a couple of effectively robust RobustBench models on some OOD shifts).
• Fourier interpolation metrics for both amplitude and phase interpolation are strong predictors (often R 2  0 . 9 ) of OOD performance for CLIP models across OOD shifts.
This suggests a hypothesis for why CLIP models are more robust: they are less sensitive to perturbations of Fourier amplitude and phase [51], which preserve image semantics for humans but confuse nonrobust models.
We also consider the ability of these model metrics to predict robustness in ImageNet models trained with various data augmentation approaches, which can produce effective robustness on many of the distribution shifts we consider [5].
Figures 5 and 13 show the most predictive metric’s performance for these non-CLIP models from Figure 4.
We again ﬁnd that, although no single metric is best across all distribution shifts, our proposed Fourier interpolation metrics are often most predictive, outperforming in-distribution accuracy.
See Appendix A.10 for full results and discussion.
Discussion
Our work is only the beginning of a full understanding of what makes models effectively robust to distribution shifts, starting with the domain of image classiﬁcation where some models have demonstrated effective robustness.
For example, Section 4.1 showed different types of model pruning can confer robustness to different distribution shifts, and while some of this robustness can be explained by improved behavior in response to amplitude and phase perturbations, much remains an open mystery.
As we saw in Section 4.2 (and has been noted by prior work [55, 21]), models can achieve robustness to some distribution shifts by training with data augmentations designed to imitate the expected OOD data.
This approach can be a powerful way to mitigate expected test-time distribution shifts but leaves open the question of how to prepare for the unexpected.
We learned in Section 4.3 that more robust
CLIP models are less sensitive to perturbations of Fourier amplitude and phase, perhaps making them more attuned to human perceptions of semantic meaning.
This ﬁnding provides an exciting opportunity for future work to conﬁrm our results with both additional out-of-line models and causal experiments; and to incorporate our Fourier interpolation metrics into training paradigms, for instance as regularizers or data augmentation strategies, to explicitly encode robustness in future models
[Figure]
[Figure]
[Table]
Our results provide a new perspective on the effective robustness problem by showing that no one single metric rules them all.
This dictates the need for a multi-faceted analysis to understand OOD robustness, which is in sharp contrast to the in-distribution setting where a single property such as model ﬂatness can reasonably explain generalization performance
[Figure]
RobustNets benchmark.
The robustnesses of our analyzed CIFAR-10 models remain out-of-line.
To track and aid progress on effective robustness prediction, we release our code as well as a database of our model weights and metric values at https://github [Figure] [Table] [Figure]
[Table]
[Figure]
[Table]
[Table] Acknowledgments
We would like to thank Jonathan Frankle, Mitchell Wortsman
[Figure]
and Ludwig Schmidt for helpful discussion and pointers to resources.
SFK acknowledges support from NSF GRFP.
This work was performed under the auspices of the U.S. Department of Energy by Lawrence Livermore National
Laboratory under Contract DE-AC52-07NA27344 and was supported by the LLNL-LDRD Program under Project No. 2022-SI-004 (LLNL-CONF-835574)
[Figure]
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] Limitations are included in the discussion section.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[N/A] (b) Did you include complete proofs of all theoretical results?
[N/A] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] Code and pretrained models are available at https://github.com/sarafridov/RobustNets .
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] Most of these are provided in the methods section, and some are in the supplement.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] We use 95% conﬁdence intervals in all of our ﬁgures.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[No] Our experiments did require substantial compute to evaluate all of our models on all of the interpolation paths, but we use pretrained models so we do not require any compute for training.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] We cite the creators of all the models and datasets we use.
(b) Did you mention the license of the assets?
[No] Licenses for individual assets can be found by visiting the websites of the corresponding creators.
(c) Did you include any new assets either in the supplemental material or as a URL?
[No] We release our RobustNets database of pretrained models evaluated in this paper.
(d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]