Explicit Tradeoffs between Adversarial and Natural Distributional Robustness
Anonymous Author(s)
AffiliationAddress email
Abstract
Several existing works study either adversarial or natural distributional robustnessof deep neural networks separately.
In practice, however, models need to enjoy bothtypes of robustness to ensure reliability.
In this work, we bridge this gap and showthat in fact, explicit tradeoffs exist between adversarial and natural distributionalrobustness.
We first consider a simple linear regression setting on Gaussian datawith disjoint sets of core and spurious features.
In this setting, through theoreticaland empirical analysis, we show that (i) adversarial training with ℓ 1 and ℓ 2 normsincreases the model reliance on spurious features; (ii) For ℓ ∞ adversarial training,spurious reliance only occurs when the scale of the spurious features is largerthan that of the core features; (iii) adversarial training can have an unintendedconsequence in reducing distributional robustness, specifically when spuriouscorrelations are changed in the new test domain.
Next, we present extensiveempirical evidence, using a test suite of twenty adversarially trained modelsevaluated on five benchmark datasets (ObjectNet, RIVAL10, Salient ImageNet-1M,
ImageNet-9, Waterbirds), that adversarially trained classifiers rely on backgroundsmore than their standardly trained counterparts, validating our theoretical results.
We also show that spurious correlations in training data (when preserved in thetest domain) can improve adversarial robustness, revealing that previous claimsthat adversarial vulnerability is rooted in spurious correlations are incomplete.
Introduction
Despite continuously improving upon state of the art accuracy on various benchmarks, deep imageclassifiers remain brittle to distribution shifts, suffering massive performance drops when evaluatedon non-i.i.d.
data.
For example, the accuracy of object detectors trained on ImageNet [12] reducesby 40-45% on ObjectNet [5], where images are taken within households at various viewpoints androtations.
The reliance of deep models on spurious features , which correlate with class labels but areirrelevant to the true labeling function [29], is one cause of poor model robustness, as performancedegrades when spurious correlations are broken.
Indeed, model reliance on spurious features liketexture [17] and background [68, 6] is well documented.
Of greater concern, deep models in safetycritical applications such as detection of pneumonia [71] and COVID-19 [11] have been observed torely on hospital specific spurious markers, causing poor generalization to new hospitals.
Adversarial examples [62, 18] pose another troubling distribution shift, where imperceptible inputperturbations can cause model accuracy to drop to zero.
Many works have been proposed to improvethe adversarial robustness of deep models [35, 73, 51, 58, 36, 10, 65, 54], including the widelypopular adversarial training , where inputs are augmented via adversarial attack during training[39].
While spurious correlation robustness has also attracted lots of attention [2, 29, 38, 49], thetwo problems are most often considered independently, despite both being essential to the safe andreliable deployment of deep models in the wild.
The increased reliance on spurious features leads to worse distributional robustness .
In the few works that do consider adversarial and spurious correlation robustness in tandem, theprevailing argument is that the origin of adversarial vulnerability is that the model focuses on spuriouscorrelations that can be manipulated [76, 72, 25].
However, recently, noise-based analyses on
RIVAL10 [41] and Salient ImageNet-1M [60] datasets suggest adversarial training may actuallyincrease model sensitivity to spurious features; a result that is both counter-intuitive and in directcontrast to existing ideology.
To better understand this observation, we first appeal to a simple linear regression problem on
Gaussian data with disentangled core and spurious features.
In this setting, we theoretically show
• Adversarial training under ℓ 2 and ℓ 1 norms increase model reliance on spurious features,as using spurious features forces an attacker to spread its budget over additional features.• For ℓ ∞ adversarial training, increased spurious feature reliance only occurs when the scale ofthe spurious feature is larger than that of the core features.
That is, spurious features are usedwhen perturbations that corrupt core features are too small to disrupt spurious correlations.
• Due to increased spurious feature reliance, there is an explicit tradeoff between adversarialand distributional robustness.
Specifically, we show that adversarial training decreases model
robustness to distribution shifts in the test domain where spurious correlations are broken.
To validate our theory, we evaluate twenty models adversarially trained using ℓ 2 and ℓ ∞ projected gradient descent [50].
Specifically, we inspect performance on multiple spurious robustness benchmarksover synthetic (ImageNet-9 [68], Waterbirds [49]) and real (RIVAL10 [41], Salient ImageNet-1M[60], ObjectNet [5]) datasets.
Figure 1 summarizes our experiments, where we find that adversariallytrained models consistently show greater sensitivity to spurious features compared to standardlytrained baselines, with the effect more dramatic for ℓ 2 adversarial training than ℓ ∞ 1 .
Finally, we showthat the presence of spurious correlations in training data (when preserved in test domain) can improveadversarial robustness, with stronger spurious correlations leading to greater accuracy under attack.
Our work combines two prevalent but often separately considered notions of robustness, yieldingsurprising theoretically-derived and empirically-supported results.
We hope our contributions grantinsight to both adversarial and distributional robustness communities, and emphasize the need forholisitic evaluations of model robustness.
Review of Literature
Adversarial Robustness.
Since adversarial examples were first observed in deep models [62, 18], thephenomenon has been extensively studied.
New attacks [8, 34, 42, 15] and defenses [35, 43, 40, 37]are introduced frequently, in a game of cat and mouse where the attacker generally has the upperhand [3].
Certified defenses seek to break this cycle by offering provable robustness guarantees
Arguably the most popular defense is adversarial training [39] where images areaugmented with adversarial perturbations during training, amounting to a min max optimization.
Natural Distributional Robustness.
In contrast to synthetic adversarial perturbations, many worksseek to characterize the robustness of deep models to naturally occurring distribution shifts, forinstance due to common corruptions (noise, blur, etc) [22] or changes in rendition [21, 63].
[30]compiles ten benchmarks of realistic distribution shifts over diverse applications (medical, economic,etc).
Many algorithms have been proposed to improve out-of-distribution robustness [38, 49, 2, 33],though in comprehensive evaluations, their gains over empirical risk minimization are marginal,as they often only hold for certain distribution shifts [70, 21].
[70] identifies diversity and correlationshifts as two key dimensions to OOD robustness; our work focuses on the latter.
Spurious Correlations.
Solely optimizing for accuracy leads deep models to rely on any patterns predictive of class in the training domain.
This includes spurious features, which are irrelevant to the truelabeling function.
Natural image datasets are riddled with spurious features [31, 59].
Spurious featurereliance becomes problematic under distribution shifts that break their correlation to class labels:sidewalk segmentation struggles in the absence of cars [56], familiar objects cannot be recognized inunfamiliar poses [1] or uncommon settings [27, 5], etc.
A natural and ubiquitous spurious correlationin vision is image backgrounds, observed in numerous prior works to be leveraged by models forclassification [68, 41] and object detection [47].
Spurious correlations also relate to algorithmicbiases [13, 17], with implications for fairness [19, 7, 9, 28], reflecting the importance of this issue.
Accordingly, many works seek to improve spurious correlation robustness.
Families of approachesinclude optimizing for worst group accuracy [49, 23, 48, 74, 38], learning invariant latent spaces[46, 2], appealing to meta-learning [45] or causality [44, 4].
Our work does not focus on mitigationmethods, but instead sheds insight on how optimizing for a different notion of robustness (i.e. adversarial) affects spurious feature reliance, and consequently, natural distributional robustness.
Generally,models trained under ERM are believed to have a propensity to use spurious features, especiallywhen they are easy to learn, due to bias of learners (algorithmic and human) to absorb simple featuresfirst [55] and take shortcuts [16].
However, recent work suggests that core features are still learnedunder ERM even when spurious features are favored, and simple finetuning on data without thespurious feature can efficiently reduce spurious feature reliance without full model retraining [29].
Unintended Outcomes of Adversarial Training.
Adversarial training achieves improved accuracyunder attack, but comes at the cost of standard accuracy, with multiple works provably demonstratingthis tradeoff [73, 14].
Notably, [26] inspires our theoretical analysis, though we focus on the effectof adversarial training on out-of-distribution (OOD) robustness to spurious correlation shifts, ratherthan its effect on standard accuracy.
A more positive outcome is that adversarial training leadsto perceptually aligned gradients [53], with applications to model debugging [61, 66], and further,transfer learning on adversarially robust features yields better accuracy on downstream tasks comparedto features learned from standard training [50], despite having lower accuracy on the original task.
To our knowledge, robustness to adversarial and natural distribution shifts have not been studiedin tandem.
However, spurious correlations are at times mentioned with adversarial robustness,usually in claims that the origin of adversarial vulnerability is in model’s focus on (imperceptible)spurious features [76, 72, 25].
Our results create tension with the contrapositive of their argument,as we show that mitigating adversarial vulnerability (via adversarial training) results in increasedspurious feature reliance.
We do this analytically in a simple linear regression setting (Section3), and empirically on multiple benchmarks, with an emphasis on natural spurious features (i.e.,backgrounds) in our experiments (Section 3).
Further, we even demonstrate a case where the presenceof a spurious feature leads to improved adversarial robustness (Section 4.3)).
We note that the spuriousfeatures we observe to be positively associated with adversarial robustness may be distinct fromthose that prior works claim contribute to adversarial vulnerability.
However, our result of adversarialtraining leading to increased spurious feature reliance (of any kind) is novel and contrary to commonunderstanding.
Given the critical nature of adversarial and spurious correlation robustness for modelsecurity, reliability, and fairness, the significance of our result in revealing potential misconceptionson the interplay of these two crucial modes of robustness should not be understated.[41] and [60] recently observed decreased core sensitivity on a handful of ℓ 2 adversarially trainedmodels, in spirit with our findings, but with no explanation.
We offer the first rigorous analysisof this counterintuitive phenomenon, evaluating 5 to 10 times as many models in 5 times as manysettings.
More importantly, we theoretically prove that adversarial training increases spurious feature reliance, contributing novel fundamental insight as to how optimizing for adversarial robustness canlead to reduced robustness to natural distribution shits, uncovering important effects like the norm ofadversarial training and the scale of spurious features at play.
Theoretical Analysis on Linear models
We begin by analysing the effects of adversarial training on a simple linear regression model.
Consider the model Y = ⟨ X, θ opt ⟩ + W where X ∈ R m is the input variable, θ opt ∈ R m is theoptimal parameter, ⟨ ., .
⟩ represents the inner product and W ∈ R is a noise variable.
We assumethat the input variables follow a multivariate Gaussian distribution N (0 , Σ) where Σ ∈ R m × m is thecovariance matrix and further assume that W is sampled from the Gaussian distribution N (0 , σ 2 w ) .
We assume that the set of features [ m ] consists of two groups, the core features C and the spuriousfeatures S .
Without loss of generality, we assume that C = { 1 , . . .
, p } and S
= { p + 1 , . . .
, m } .
Weassume that the optimal parameter θ opt has non-zero entries on the set C only.
This implies that theoutput depends on the input only through the core features and conditioned on the core features, it isindependent of the spurious ones.
More formally, we assume that Y ⊥ X S | X C where X S and X Crepresent the core and spurious subsets of the input, respectively.
For loss functions, we define the standard loss function as L ( θ ) =
E (cid:2) ( y − ⟨ X, θ ⟩ ) 2 (cid:3) , , and theadversarial loss function as
[Equation]
where p is the attack norm and ϵ represents the norm budget.
We first show an equivalent form of (1) that is more amenable to analysis.
Theorem 1.
Assume that Y = ⟨
X, θ opt ⟩ + W where W ∼ N (0 , σ 2 w ) is independent of X andθ opt ∈ R m is a fixed parameter.
Assume further that X follows the distribution N (0 , Σ) and defineσ 2 θ as ( θ − θ opt ) T Σ( θ − θ opt ) + σ 2 w .
The loss function (1) is equivalent to
[Equation]
where c 1 = (cid:113) 2 π < 1 , c 2 = 1 − c 21 and ∥ .
∥ q is the dual norm of ∥ .
∥ p , i.e, 1 p + 1 q = 1 .
Furthermore,the above formulation is convex in θ .
The above result is similar to Proposition 3.2 in [26] which provides characterization results for theℓ 2 norm.
The key distinction of our results is providing a simple convex formulation of the robustminimization problem, allowing the results to be easily generalized for an arbitrary ℓ p norm.
Thetheorem shows that the optimal value (cid:98) θ minimizing the adversarial losss L p,ϵ ( θ ) is not θ opt and ingeneral, may be non-zero on the set of spurious features S .
This means that adversarial trainingdirects the model towards using the spurious correlations in order to increase robustness.
The proof of the theorem is provided in the Appendix.
The main structure of the proof is similar tothat of [26].
We first show that the inner maximization problem of (1) can be rewritten as
[Equation]
This allows us to rewrite (1) as
[Equation]
where for ( a ) we have used the fact that Y = ⟨ X, θ ⟩ + W .
Using the fact that X and W are Gaussian,we show that this is equal to σ 2 θ + ϵ 2 ·
∥ θ ∥ 2 q + 2 · c 1 · ϵ · ∥ θ ∥ q · σ θ which we can further simplify to(2) with some algebraic manipulation.
Next, using standard techniques for analysing composition of convex functions, we show that σ 2 θ and ( c 1 σ θ + ϵ · ∥ θ
∥ q ) 2 are both convex in θ , implying that L p,ϵ ( θ )is convex in θ as well.
Using the convex formulation in Theorem 1, we evaluate the linear model for different values ofparameters to better understand the reliance of the model on spurious features.
In our experiments,we consider a simple model with 5 features, the first two of which are core features.
We let η be a parameter controlling the correlation degree between the core and spurious features,with larger values corresponding to higher correlation.
For the distribution of the data, define thematrix (cid:101) Q and Q matrices as
[Equation]
Note that Q is obtained by normalizing the rows of (cid:101) Q .
Each row of the Q matrix corresponds to aninput feautre.
We let Σ take the value QQ T .
This is equivalent to sampling X from the distribution
Q N (0 , I ) .
Throughout our experiments, we set σ w = 0 .
For an arbitrary vector θ , we define its Norm Fraction over Spurious feaures NFS ( θ ) as
[Equation]
Intuitively, NFS measures the degree to which a model relies on spurious features.
For our firstexperiment, we perform adversarial training with varying parameter ϵ to obtain a predictor (cid:98) θ andevaluate its NFS.
We consider different choices of the ℓ p norm as well as the correlation parameter η
The results can be seen in Figure 2.
[Figure]
178 As seen in Figure 2, for the ℓ 1 and ℓ 2 norms, increasing the adversary’s budget causes the modelto rely more on the spurious features.
To understand why this happens, it is helpful to consider agame theoretic perspective: If the model only looks at the core features, then the adversary will onlyneed to perturb these features.
Thus, even though the spurious features are normally less suitable forprediction, they now have the advantage of being less perturbed.
Assuming ϵ is large enough, themodel will be better off looking at these features in forming its prediction.
Of course, if the modelonly looks at the spurious features, core features will become even more informative as they wouldbe unperturbed as well.
In the game’s equilibrium, the model would use both the core and spuriousfeatures, relying more on the spurious features with increased values of ϵ .
Interestingly, this reasoning does not always apply for the ℓ ∞ norm.
Indeed, if the model were toonly look at the core features, the adversary may still perturb the spurious features with no extra cost.
This is because the ℓ ∞ norm only measures the maximum perturbation in each feature and as long asthe perturbations on the spurious features are not larger than the perturbations on the core features, the norm would not change.
The results of Figure 2c support this as the adversary does not rely onthe spurious features for its predictions.
Importantly, however, in some cases the adversary may require a larger budget to perturb the spurious features compared to core features.
Indeed, if we were to scale a feature by multiplicationwith a large number, then the adversary would require a larger budget to perturb that feature asthe budget for that feature has effectively decreased.
We can therefore expect that for the ℓ ∞norm, scaling up the spurious features would cause the model to rely on them for prediction.
[Figure]
Figure 3 shows that this is indeed the case.
The figure showsthe norm fraction over spurious (NFS) after scaling the spuriousfeatures for different values of ϵ .
The scaling is done by multiplying the rows of Q corresponding to the spurious featuresby a scaling parameter.
As seen in Figure 3, larger values ofthe scaling parameter, as well as larger values of ϵ , cause themodel to become more reliant on the spurious features.
Next, we measure the effect of using spurious features on themodel’s distributional robustness.
To do this, we train twoadversarial models.
The first model, which we denote by “core”,uses only the core features while the second model, denoted by“total”, uses all of the features.
We then simulate a distributionshift that breaks the spurious correlations by adding random
Gaussian noise to the spurious rows of matrix Q defined in (4).
Specifically, for each entry of Q in a spurious row, we add anoise sampled from N (0 , σ 2 Q ) where σ Q is a noise parameter.
We use a fixed value of η = 0 .
and vary the parameter ϵ , thenorm p used in adversarial training as well as the variance of the Gaussian noise.
The results areshown in Figure 4.
[Figure]
We observe that the clean (i.e. in distribution; σ Q = 0 ) loss of the “core" model may be higher orlower than that of the “total" model, but in both cases, the total model is consistently more vulnerableto distributional shifts resulted from breaking spurious correlations.
Empirical Evidence
We now demonstrate increased spurious feature reliance in adversarially trained models over multiplebenchmarks.
We evaluate models on two backbones (ResNet18, ResNet50) adversarially trained on
ImageNet [12] using two norms ( ℓ 2 , ℓ ∞ ) under five attack budgets (denoted ϵ ) per norm, resulting ina 2 × 2 × 5 = 20 model test suite, as well as standardly trained baselines.
See appendix for details.
4.1 AT hurts Natural Distributional Robustness only when Spurious Correlations are broken
We first show reduced distributional robustness of adversarial models occurs specifically in caseswhere natural spurious correlations are broken.
We appeal to the ImageNet-C [22] and ObjectNet
[5] OOD benchmarks.
ImageNet-C augments ImageNet samples with common corruptions like noiseor blurring, distorting both core and spurious features equally.
Crucially, these corruptions do notbreak spurious correlations.
On the other hand, ObjectNet is formed by having workers captureimages of common household objects (including samples from 113 classes of ImageNet) in theirhomes .
Thus, only spurious features are affected.
Namely, ObjectNet introduces distribution shifts inbackground, rotation, and viewpoint.
We plot accuracies on these benchmarks in figure 5.
Recall that adversarially trained models have lower standard accuracy, which can confound ouranalysis, so we compare the drop in OOD accuracy to the drop in ImageNet accuracy across ourmodel suite.
Observe that the ratio of ImageNet-C accuracy to ImageNet accuracy is roughlyconstant across models.
However, the ratio of ObjectNet accuracy to ImageNet accuracy is lowerfor adversarially trained models.
Therefore, even after controlling for reduced standard accuracy,the distributional robustness of adversarially trained models is worse than that of standard models.
Importantly, this effect does not hold for distribution shifts that maintain spurious correlations,indicating that the reduced distributional robustness is due to increased spurious feature reliance.
4.2 Reduced Core Sensitivity, and Difference in the Effect of ℓ 2 & ℓ ∞ Adversarial Training
We now directly quantify sensitivity to core features via RIVAL10 and Salient ImageNet-1M datasets[41, 60].
The premise of this analysis is that model sensitivity to an input region can be quantifiedby the drop in accuracy due to corrupting that region [59].
[41] introduced the noise-based metricrelative foreground sensitivity ( RFS ), which is the gap between accuracy drops due to background and foreground noise, normalized so to allow for comparisons across models with varying generalnoise robustness.
RIVAL10 object segmentations allow for RFS computation.
Analogously, relativecore sensitivity ( RCS ) is computed using Salient ImageNet-1M’s soft segmentations of core inputregions.
A key distinction between the two metrics is that RCS is computed directly on pretrainedmodels performing the original 1000 -way ImageNet classification task, while RFS first requiresmodels to be finetuned on the much coarser 10 -way classification task of RIVAL10.
Also, Salient
ImageNet-1M includes all ImageNet images , while RIVAL10 only consists of 20 ImageNet classes.
Figure 6 shows a decrease in RFS and RCS as the attack budget ϵ seen during ℓ 2 adversarial trainingrises.
Thus, adversarial training reduces core feature sensitivity relative to spurious featuresensitivity .
Notably, this effect does not hold for models adversarially trained with attacks under theℓ ∞ norm for RCS , though it does for RFS .
Alluding to our theoretical result, we conjecture that in
Salient ImageNet-1M, the scales of the spurious features are much smaller than in RIVAL10, due tothe diversity of images and finer grain of classes.
That is, a smaller perturbation is needed to alter aspurious feature so that it correlates with an incorrect class when there are 1000 classes than whenthere are only 10 classes with generally disparate backgrounds.
4.3 Adversarial Training Increases Background Reliance in Synthetic Datasets
Now, we take a closer look at the reliance of adversarially trained models on the contextual spuriousfeature of backgrounds via the synthetic datasets ImageNet-9 [68] and Waterbirds [49].
Bothdatasets use segmentations to superimpose objects over varying backgrounds, detailed below.
ImageNet-9 (IN-9) organizes a subset of ImageNet into nine superclasses.
Multiple validation sets exist for IN-9, where backgrounds or foregrounds are altered; we use M IXED -S AME and M IXED -R
AND .
In both sets, original backgrounds are swapped out for new ones.
Crucially, in M IXED -S AME , the newbackgrounds are taken from other instances within the same class , while M IXED -R AND uses randombackgrounds .
The metric, Background Gap , is the difference in model accuracy on M IXED -S AMEand M IXED -R
AND (i.e. drop due to breaking spurious background correlation).
Figure 7 shows that the background gap for our test suite of twenty robust ResNets and two standardlytrained baselines.
Nine out of the ten ℓ 2 adversarially trained models have larger background gapsthan the standard baselines, while the same is true for five out of the ten ℓ ∞ adversarially trainedmodels.
When considering relative gaps (i.e. as a percent of the accuracy on M IXED -S AME ), theincrease in gap becomes even more dramatic, with the ℓ 2 adversarially trained ResNet50 for ϵ = 5having a 18 . 9% relative drop, compared to the 9 .
7% relative drop for the standardly trained ResNet50.
Waterbirds combines foregrounds from Caltech Birds [64] and backgrounds from SUN Places [67].
The task is binary classification of landbirds and waterbirds.
The majority group ( 95% of training samples) consists of landbirds over land backgrounds and waterbirds over water backgrounds.
Theminority group breaks this spurious correlation, placing landbirds over water backgrounds, and viceversa.
The test set is evenly split between these groups.
We train only a final linear layer atop thefrozen feature extractors (so that models remain adversarially robust) for each of our models on the
Waterbirds training set for ten epochs, saving the model with highest validation accuracy.
Figure 8 shows majority and minority group accuracies, and the gap between them, for our test suiteof models.
Again, we see increased gaps for robust models, with 100% of ℓ 2 and 60% of ℓ ∞ modelsrespectively having larger gaps than the standardly trained baseline on the corresponding backbone.
In both benchmarks, breaking the background spurious correlation causes a more significant dropin performance for adversarially trained models than standardly trained models, indicating thatadversarial training led to increased reliance on backgrounds.
The observed affect is stronger for ℓ 2adversarially trained models than ℓ ∞ ones.
Further, the gaps grow near monotonically with ϵ .
4.4 Reverse Effect: Presence of Spurious Correlations Can Improve Adversarial Robustness
Finally, we show evidence that is directly at odds with theclaim that spurious features lead to adversarial vulnerability.
We train ResNet18s on CIFAR10 [32] with a spurious featureinjected.
Namely, images have all values in one color channelslightly increased.
The majority group consists of red-shiftedimages from classes 0 − 4 and green shifted images from classes5 − 9 , while the minority group has reverse color-shifts.
Theparameter ρ is the ratio between majority and minority groupsize, controlling the strength of the spurious correlation (higherρ means stronger spurious correlation; ρ = 1 : 1 means thespurious feature has no predictive power).
We then evaluate theaccuracy of the trained model under adversarial attack on ani.i.d.
test set ( spurious feature retained , no distribution shift).
Figure 9 visualizes the results.
Not surprisingly, the clean accuracy is higher for models trained on data with higher ρ , as the spurious feature is more predictive forhigher ρ .
In fact, the added predictive influence the spurious feature leads to better accuracy underattack , with the gap between highest and lowest ρ values growing up to four fold compared to the baseline gap in clean accuracy.
Thus, using a spurious feature can improve adversarial robustness.
Despitebeing on a contrived example, this experiment shows that, while some spurious correlations may causeadversarial vulnerability, others do the opposite : the picture is more nuanced than previously assumed.
A Limitations of Theorem 1
While Theorem 1 provides an understanding of the tradeoff between adversarial and natural distributional robustness, there are some limitations.
Firstly, the results consider a setting where the core andspurious features are completely disentangled, i.e, they each represent different parts of the input.
Inpractice, spurious features may be entangled with the core features (e.g., the color of an image mayrepresent a spurious feature.)
In addition, our results mainly consider the goal of adversarial trainingas we focus on the expected loss L p,ϵ ( θ ) , rather than its finite-sample variant.
This is because evenfor an ℓ 2 adversary, characterizing the finite-sample behaviour of adversarial training is difficult andrequires careful assumptions on the asymptotic behaviour of the parameters (e.g., see Theorem 3.3 in[26]).
We leave exploring these directions to future work.
Even so, we believe our theoretical resultsare of interest to the community since disjoint features already capture a wide variety of spuriouscorrelations, e.g., background correlations, as well as examples where a spurious object is present inthe image.
The main goal of our theoretical analysis is to show the existence of explicit tradeoffsbetween adversarial and distributional robustness and build practical insights using those results.
B Societal Impact
Our work touches on two important notions of robustness for the safe and fair deployment of deepmodels in the wild.
We hope our results lead to careful analysis of all modes of robustness, and theinterplay between them, before deep models are used in sensitive applications.
While our resultscreate tension with some previous works [72, 76, 25], we stress that we do not wish to diminish theirwork; instead, we hope our work reveals the vast nuance associated with spurious correlations, whichcan help and hurt models in various ways.
Lastly, we release all code to encourage future work.
C Varying the Number of Core and Spurious Features
In this section, we further analyze the plateauing behaviour of the performance of the linear modelobserved in Figure 2 for different values of core and spurious features with the ℓ 1 and ℓ 2 adversarialtraining.
We first focus on the ℓ 1 case.
We consider different values for the number of core features c and totalfeatures m and measure NFS for different values of adversarial budget ϵ as in Figure 2.
The matrix Σis constructed using Equation (4) as before, with modified number of rows and columns based onthe values of c, p .
Similarly, θ opt is constructed as before, with the core coordinates set to 1 and thespurious coordinates set to 0 .
The value of η is fixed at 0 .
The results are shown in Figure 10.
As shown in the Figure, when using m total features and c core features, NFS plateaus at m − cm forlarge values of ϵ .
Intuitively, this is because of the structure of the optimization problem (2).
Recallthat when using the ℓ 1 norm, the value of q in (2) equals ∞ .
As such, adversarial training tries to finda parameter θ that has a low ℓ ∞ norm and is “close” (as measured by σ θ ) to θ opt .
The ℓ ∞ penaltyencourages values of θ that are uniform across the coordinates.
Since there are m − c spuriousfeatures and m total features, this leads to models that have an NFS value of m − cm .
We further repeat the above experiment with ℓ 2 norm.
The results are shown in Figure 11.
As seenin the figure, we see the same qualitative results as in Figure 2, with higher values of NFS whenincreasing numbers of spurious features.
D Proof of Theorem 1
Proof.
We first claim that
[Equation]
To see why this holds, note that for all δ satisfying ∥
δ ∥
p ≤ ϵ ,
[Equation]
[Figure] [Figure]
where ( a ) follows from the triangle inequality and ( b ) follows from Hölder’s inequality.
With asuitable choice of δ , we can achieve equality for ( b ) .
As ∥ θ ∥ q = ∥− θ ∥ q , at least one of { δ, − δ }would further achieve equality for ( a ) .
As maximizing | .
| is equivalent to maximizing ( . ) 2 , (3) isproved.
Given (3), we can rewrite (1) as
[Equation]
Where for ( a ) we have used the fact that Y = ⟨ X, θ ⟩ + W .
Define v θ as ⟨ X, θ − θ opt ⟩ + W .
As X was assumed to be sampled from N (0 , Σ) , v θ is distributedas N (0 , σ 2 θ ) .
It follows that
[Equation]
where for ( a ) we have used the fact that E (cid:2)(cid:12)(cid:12) N
[Equation]
σ
[Equation]
= c 1 · σ .
This proves (2) as claimed.
As for convexity, σ θ is convex in θ since it can be written as (cid:13)(cid:13)(cid:13)
[Σ 12 ( θ − θ opt ) , σ w ] (cid:13)(cid:13)(cid:13) 2 where [ ., . ]denotes the vector stacking operation.
As c 1 σ θ + ϵ · ∥ θ ∥ q is always positive and x → x 2 is convexand increasing for x ≥ 0 , this implies that ( c 1 σ θ + ϵ · ∥ θ ∥ q ) 2 is convex as well.
Finally c 2 σ 2 θ isconvex as c 2 > 0 and therefore (1) is convex in θ .
E Additional Details on Reverse Effect (Section 4.3)
Our final empirical observation is that the presence of a spurious feature (in both training and testdistributions) can lead to increased adversarial robustness.
This more directly creates tension withclaims that adversarial vulnerability is born out of spurious feature reliance.
We refer to this as the‘reverse effect’, in relation to our primary empirical and theoretical finding that adversarial trainingincreases spurious feature reliance.
We now elaborate on the experimental setup discussed in Section4.3, reproduce the results with a different spurious feature, and finally appeal to ImageNet-9 todemonstrate this effect using a more realistic spurious feature (i.e. backgrounds).
E.1 Experimental Setup
Overview.
We inject spurious correlations to the CIFAR10 dataset.
Based on the class label,we adjust half the images (i.e. with class label from 5 to 9) to shift in one direction with highprobability.
For example, a dog image is made greener with probability 0 .
95 , corresponding toa majority-to-minority group ratio of ρ = 19 : 1 .
With probability 0 .
05 , we shift in the otherdirection (e.g. make redder).
We then standardly train a ResNet18 from scratch on the datasetwith the spurious feature injected for the 10-way CIFAR classification task.
Importantly, weevaluate robust accuracy with the spurious feature retained, and then compare adversarial robustness of models trained under data with different strengths of the injected spurious correlation.
the presence of a spurious correlation in training data makes
Figure 9 and 14 show that for two distinct spurious features(color and lighting), robust accuracy is higher when the spurious correlation is stronger.
Notably, the gain is larger than thegain in standard accuracy.
Intuitively, we see that relying onthe predictive power of the spurious feature is helpful for standard accuracy, and especially for acccuracy under adversarialattack.
Despite being irrelevant to the true labeling function,the spurious feature can improve model performance, andindeed even lead to better adversarial robustness.
Details.
Color shift is achieved by increasing all pixel intensities along one channel by 0 .
Lighting shift is achievedby simply scaling an input by 1 . 25 to make brighter or 0 .
75to make darker.
All images are clamped to remain in the[0 , 1] pixel range after spurious feature injection.
Modelsare trained for 20 epochs using an Adam optimizer with alearning rate of 0 .
001 and weight decay of 1 e − 4 .
E.2 Leveraging ImageNet-9
We now demonstrate the observed reverse effect on the higher resolution ImageNet-9 dataset, leveraging the natural and ubiquitous spurious feature of backgrounds.
We finetune pretrained modelson M IXED -S AME and M IXED -R AND separately, and evaluate each model’s accuracy under attackon the same split that they were trained over.
Further, we leverage the adversarially trained modelsfrom test suite in this experiment.
This way, accuracy under attack is more informative, as the modelsare trained to expect attacks (i.e. we are not imposing any distribution shifts that would lead tounexpected model behavior).
Along this vain, we attack each backbone with the same norm and ϵthat it was pretrained over.
Figure 15 shows the gain in accuracy for the models trained and evaluated on M IXED -S
AME comparedto
those using M IXED -R
AND .
We see that the presence of background correlations increases bothstandard and robust accuracy for all models (i.e. gains are positive).
Further, gains in accuracyunder attack are larger than gains in standard accuracy in nearly all cases.
Thus, it seems like theadded predictive power of the spurious background feature has a significantly nontrivial impacton improving adversarial robustness, contradicting many existing arguments on the link betweenspurious correlations and adversarial vulnerability.
[Table]
[Table]
F Adversarially Robust Model Test Suite (Section 3)F.1 Model Details
We utilize the treasure trove of open-source adversarially trained models, contributed by [50],accessible at https://github.com/Microsoft/robust-models-transfer .
For completeness,we now provide details on the models we use, though we refer readers to Appendix A.1 of the originaltext, where the information we share now is sourced.
Training All models were trained on ImageNet in batches of 512 samples, using SGD optimizer withmomentum of 0 . 9 and weight decay of 1 e − 4 , for a total of 90 epochs, with learning rate droppingby a factor of 10 every 30 epochs.
The standard procedure of [39] was performed to adversariallytrain models, using 3 projected gradient descent steps with a step size 23 ϵ for the attack budget ϵ .
Selected Models We focus our empirical study on the ResNet architecture [20] because ofits wide spread popularity.
Specifically, we study ResNet18s and ResNet50s that are adversarially trained under the ℓ 2 norm, for ϵ ∈ { 0 . 25 , 0 . 5 , 1 , 3 , 5 } , and ℓ ∞ norm, for ϵ ∈{ 0 . 5 / 255 , 1 / 255 , 2 / 255 , 4 / 255 , 8 / 255 } , as well as standardly trained baselines.
Table 1 shows the standard accuracies for these models.
Note that we at times compare between theℓ 2 and ℓ ∞ adversarially trained models (e.g. figure 6).
We acknowledge that direct comparisons arechallenging because the threat model under which adversarial robustness is optimized for are different.
However, we note that standard accuracies of the i th ℓ 2 AT model is roughly the same as that of the i thℓ ∞ AT model, suggesting that those models lie in similar points of the accuracy-robustness tradeoff.
Additional Models.
We extend our analysis to other architectures.
We replicate all pretrained-modelexperiments on the Wide ResNet50 (2x) backbones, for which we have checkpoints for each ofthe five ϵ values for both ℓ 2 and ℓ ∞ norms.
We also inspect MobileNetv2 [52], DenseNet161 [24],
ResNeXt5050_32x4d [69], ShuffleNet [75], and VGG16_bn [57].
For each of these five architectures,we compare an ℓ 2 adversarially trained model with ϵ = 3 to a standardly trained baseline.
F.2 Experimental Details
ObjectNet and ImageNet-C [5, 22].
We report raw accuracies under noise, blur, and digitalcorruption types for ImageNet-C, as opposed to relative corruption error.
For ObjectNet, we map
ImageNet predictions to the set of 113 overlapping classes in ObjectNet.
RIVAL10 ( RFS ) and
Salient ImageNet-1M ( RCS )
RFS computation requires finetuning a final linear layerover fixed features for the coarse-grained ten way RIVAL10 classification.
RCS operates on modelsoff the shelf, directly inspecting accuracies over ImageNet classes (and samples, with region-basednoise corruption).
ImageNet-9 and Waterbirds [68, 49].
ImageNet-9 accuracies are obtainedby mapping off-the-shelf model predictions to the nine coarse labels deterministically.
Waterbirdsrequires finetuning, which we do over fixed features.
For RIVAL10 and Waterbirds finetuning, weuse Adam with learning rate of 1 e − 4 and weight decay of 1 e − 5 for 20 and 15 epochs respectively.
F.3 Results on Extended Model Test Suite
We now corroborate all our empirical findings on new backbones, expanding our analysis to 21 newmodels (including 10 AT WideResNet50s over both ℓ 2 and ℓ ∞ norms) over six architectures.
WideResNets.
We corroborate all our empirical findingson ResNet18s and ResNet50s on the WideResNet50 (2x)architecture.
Figure 16 shows that accuracy drop in ATmodels is more severe on distribution shifts that breakspurious correlations (ObjectNet), unlike the accuracydrop due to corruption of both core and spurious features (ImageNet-C), which can likely be explained bythe reduced standard accuracy of AT models.
Figure 17 shows reduced sensitivity to core andforeground regions for AT models.
Again, the effect ismore pronounced for ℓ 2 adversarially training and forlarger ϵ .
Also, we again see that decrease in RCS is lessconsistent than the drop in RFS .
We conjecture thatthe diversity and fine-grain Salient ImageNet classification task reduces the strength of spuriouscorrelations present in the data, thus diluting our observed effects of adversarial training on spuriousfeature reliance.
Lastly, figure 18 shows the drop in accuracy due to breaking spurious background correlations islarger for AT models.
Indeed, the absolute background gap (IN-9) for the WideResNet50 underℓ 2 AT with ϵ = 5 is 50% larger than the gap for the standardly trained baseline.
We note that theabsolute gaps are smaller in some cases.
We believe the lower standard accuracy of AT models maycontribute to this, as there is less accuracy to drop from.
Nonetheless, it is intriguing that in somecases, ℓ ∞ adversarial training seems to reduce spurious feature reliance; while our theory explainshow a spurious feature can be completely ignored under ℓ ∞ training, it does not explain cases wherespurious feature reliance is reduced compared to standard training.
We believe this is an interestingdirection for future work.
Other backbones.
We now show results for ten other models, half of which are ℓ 2 adversariallytrained with ϵ = 3 , while the others are standardly trained.
Figure 19 summarizes our results,corroborating each of our empirical findings.
[Figure]