Explicit Tradeoffs between Adversarial and Natural Distributional Robustness
Abstract
Several existing works study either adversarial or natural distributional robustness of deep neural networks separately.
In practice, however, models need to enjoy both types of robustness to ensure reliability.
In this work, we bridge this gap and show that in fact, explicit tradeoffs exist between adversarial and natural distributional robustness.
We first consider a simple linear regression setting on Gaussian data with disjoint sets of core and spurious features.
In this setting, through theoretical and empirical analysis, we show that (i) adversarial training with ℓ 1 and ℓ 2 norms increases the model reliance on spurious features; (ii) For ℓ ∞ adversarial training, spurious reliance only occurs when the scale of the spurious features is larger than that of the core features; (iii) adversarial training can have an unintended consequence in reducing distributional robustness, specifically when spurious correlations are changed in the new test domain.
Next, we present extensive empirical evidence, using a test suite of twenty adversarially trained models evaluated on five benchmark datasets (ObjectNet, RIVAL10, Salient ImageNet-1M,
ImageNet-9, Waterbirds), that adversarially trained classifiers rely on backgrounds more than their standardly trained counterparts, validating our theoretical results.
We also show that spurious correlations in training data (when preserved in the test domain) can improve adversarial robustness, revealing that previous claims that adversarial vulnerability is rooted in spurious correlations are incomplete.
Introduction
Despite continuously improving upon state of the art accuracy on various benchmarks, deep image classifiers remain brittle to distribution shifts, suffering massive performance drops when evaluated on non-i.i.d.
data.
For example, the accuracy of object detectors trained on ImageNet [12] reduces by 40-45% on ObjectNet [5] where images are taken within households at various viewpoints and rotations.
The reliance of deep models on spurious features , which correlate with class labels but are irrelevant to the true labeling function [30], is one cause of poor model robustness, as performance degrades when spurious correlations are broken.
Indeed, model reliance on spurious features like texture [17] and background [69, 6] is well documented.
Of greater concern, deep models in safety critical applications such as detection of pneumonia [72] and COVID-19 [11] have been observed to rely on hospital specific spurious markers, causing poor generalization to new hospitals.
Adversarial examples [63, 18] pose another troubling distribution shift, where imperceptible input perturbations can cause model accuracy to drop to zero.
Many works have been proposed to improve the adversarial robustness of deep models [36, 74, 52, 59, 37, 10, 66, 55], including the widely popular adversarial training , where inputs are augmented via adversarial attack during training [40].
While spurious correlation robustness has also attracted lots of attention [2, 30, 39, 50], the 36th Conference on Neural Information Processing Systems (NeurIPS 2022).
The increased reliance on spurious features leads to worse distributional robustness .
two problems are most often considered independently, despite both being essential to the safe and reliable deployment of deep models in the wild.
In the few works that do consider adversarial and spurious correlation robustness in tandem, the prevailing argument is that the origin of adversarial vulnerability is that the model focuses on spurious correlations that can be manipulated [77, 73, 26].
However, recently, noise-based analyses on
RIVAL10 [42] and Salient ImageNet-1M [61] datasets suggest adversarial training may actually increase model sensitivity to spurious features; a result that is both counter-intuitive and in direct contrast to existing ideology.
To better understand this observation, we first appeal to a simple linear regression problem on
Gaussian data with disentangled core and spurious features.
In this setting, we theoretically show
• Adversarial training under ℓ 2 and ℓ 1 norms increase model reliance on spurious features, as using spurious features forces an attacker to spread its budget over additional features.
• For ℓ ∞ adversarial training, increased spurious feature reliance only occurs when the scale of the spurious feature is larger than that of the core features.
That is, spurious features are used when perturbations that corrupt core features are too small to disrupt spurious correlations.
• Due to increased spurious feature reliance, there is an explicit tradeoff between adversarial and distributional robustness.
Specifically, we show that adversarial training decreases model robustness to distribution shifts in the test domain where spurious correlations are broken.
To validate our theory, we evaluate twenty models adversarially trained using ℓ 2 and ℓ ∞ projected gradient descent [51].
Specifically, we inspect performance on multiple spurious robustness benchmarks over synthetic (ImageNet-9 [69], Waterbirds [50]) and real (RIVAL10 [42], Salient ImageNet-1M [61], ObjectNet [5]) datasets.
Figure 1 summarizes our experiments, where we find that adversarially trained models consistently show greater sensitivity to spurious features compared to standardly trained baselines, with the effect more dramatic for ℓ 2 adversarial training than ℓ ∞ 1 .
Finally, we show that the presence of spurious correlations in training data (when preserved in test domain) can improve adversarial robustness, with stronger spurious correlations leading to greater accuracy under attack.
Our work combines two prevalent but often separately considered notions of robustness, yielding surprising theoretically-derived and empirically-supported results.
We hope our contributions grant insight to both adversarial and distributional robustness communities, and emphasize the need for holistic evaluations of model robustness.
Review of Literature
Adversarial Robustness.
Since adversarial examples were first observed in deep models [63, 18], the phenomenon has been extensively studied.
New attacks [8, 35, 43, 15] and defenses [36, 44, 41, 38] are introduced frequently, in a game of cat and mouse where the attacker generally has the upper hand [3].
Certified defenses seek to break this cycle by offering provable robustness guarantees [59, 37, 52, 10].
Arguably the most popular defense is adversarial training [40] where images are augmented with adversarial perturbations during training, amounting to a min max optimization.
Natural Distributional Robustness.
In contrast to synthetic adversarial perturbations, many works seek to characterize the robustness of deep models to naturally occurring distribution shifts, for instance due to common corruptions (noise, blur, etc) [22] or changes in rendition [21, 64].
[31] compiles ten benchmarks of realistic distribution shifts over diverse applications (medical, economic, etc).
Many algorithms have been proposed to improve out-of-distribution robustness [39, 50, 2, 34], though in comprehensive evaluations, their gains over empirical risk minimization are marginal, as they often only hold for certain distribution shifts [71, 21].
[71] identifies diversity and correlation shifts as two key dimensions to OOD robustness; our work focuses on the latter.
Spurious Correlations.
Solely optimizing for accuracy leads deep models to rely on any patterns predictive of class in the training domain.
This includes spurious features, which are irrelevant to the true labeling function.
Natural image datasets are riddled with spurious features [32, 60].
Spurious feature reliance becomes problematic under distribution shifts that break their correlation to class labels: sidewalk segmentation struggles in the absence of cars [57], familiar objects cannot be recognized in unfamiliar poses [1] or uncommon settings [28, 5], etc.
A natural and ubiquitous spurious correlation in vision is image backgrounds, observed in numerous prior works to be leveraged by models for classification [69, 42] and object detection [48].
Spurious correlations also relate to algorithmic biases [13, 17], with implications for fairness [19, 7, 9, 29], reflecting the importance of this issue.
Accordingly, many works seek to improve spurious correlation robustness.
Families of approaches include optimizing for worst group accuracy [50, 24, 49, 75, 39], learning invariant latent spaces [47, 2], appealing to meta-learning [46] or causality [45, 4].
Our work does not focus on mitigation methods, but instead sheds insight on how optimizing for a different notion of robustness (i.e. adversarial) affects spurious feature reliance, and consequently, natural distributional robustness.
Generally, models trained under ERM are believed to have a propensity to use spurious features, especially when they are easy to learn, due to bias of learners (algorithmic and human) to absorb simple features first [56] and take shortcuts [16].
However, recent work suggests that core features are still learned under ERM even when spurious features are favored, and simple finetuning on data without the spurious feature can efficiently reduce spurious feature reliance without full model retraining [30].
Unintended Outcomes of Adversarial Training.
Adversarial training achieves improved accuracy under attack, but comes at the cost of standard accuracy, with multiple works provably demonstrating this tradeoff [74, 14, 23].
Notably, [27] inspires our theoretical analysis, though we focus on the effect of adversarial training on out-of-distribution (OOD) robustness to spurious correlation shifts, rather than its effect on standard accuracy.
A more positive outcome is that adversarial training leads to perceptually aligned gradients [54], with applications to model debugging [62, 67], and further, transfer learning on adversarially robust features (despite being less accurate on the original task) yields better accuracy on downstream tasks compared to features learned from standard training [51].
To our knowledge, robustness to adversarial and natural distribution shifts have not been studied in tandem.
However, spurious correlations are at times mentioned with adversarial robustness, usually in claims that the origin of adversarial vulnerability is in model’s focus on (imperceptible) spurious features [77, 73, 26].
Our results create tension with the contrapositive of their argument, as we show that mitigating adversarial vulnerability (via adversarial training) results in increased spurious feature reliance.
We do this analytically in a simple linear regression setting (Section 3), and empirically on multiple benchmarks, with an emphasis on natural spurious features (i.e., backgrounds) in our experiments (Section 3).
Further, we even demonstrate a case where the presence of a spurious feature leads to improved adversarial robustness (Section 4.3)).
We note that the spurious features we observe to be positively associated with adversarial robustness may be distinct from those that prior works claim contribute to adversarial vulnerability.
However, our result of adversarial training leading to increased spurious feature reliance (of any kind) is novel and contrary to common understanding.
Given the critical nature of adversarial and spurious correlation robustness for model security, reliability, and fairness, the significance of our result in revealing potential misconceptions on the interplay of these two crucial modes of robustness should not be understated.
[42] and [61] recently observed decreased core sensitivity on a handful of ℓ 2 adversarially trained models, in spirit with our findings, but with no explanation.
We offer the first rigorous analysis of this counterintuitive phenomenon, evaluating 5 to 10 times as many models in 5 times as many settings.
More importantly, we theoretically prove that adversarial training increases spurious feature reliance, contributing novel fundamental insight as to how optimizing for adversarial robustness can lead to reduced robustness to natural distribution shits, uncovering important effects like the norm of adversarial training and the scale of spurious features at play.
Theoretical Analysis on Linear models
We begin by analysing the effects of adversarial training on a simple linear regression model.
Consider the model Y = ⟨ X, θ opt ⟩ + W where X ∈ R m is the input variable, θ opt ∈ R m is the optimal parameter, ⟨ ., .
⟩ represents the inner product and W ∈ R is a noise variable.
We assume that the input variables follow a multivariate Gaussian distribution N (0 , Σ) where Σ ∈ R m × m is the covariance matrix and further assume that W is sampled from the Gaussian distribution N (0 , σ 2 w ) .
We assume that the set of features [ m ] consists of two groups, the core features C and the spurious features S .
Without loss of generality, we assume that C = { 1 , . . .
, p } and S
= { p + 1 , . . .
, m } .
We assume that the optimal parameter θ opt has non-zero entries on the set C only.
This implies that the output depends on the input only through the core features and conditioned on the core features, it is independent of the spurious ones.
More formally, we assume that Y ⊥ X S | X C where X S and X C represent the core and spurious subsets of the input, respectively.
For loss functions, we define the standard loss function as L ( θ ) =
E (cid:2) ( y − ⟨ X, θ ⟩ ) 2 (cid:3) , , and the adversarial loss function as
[Equation]
where p is the attack norm and ϵ represents the norm budget.
We first show an equivalent form of (1) that is more amenable to analysis.
Theorem 1.
Assume that Y = ⟨
X, θ opt ⟩ + W where W ∼ N (0 , σ 2 w ) is independent of X and θ opt ∈ R m is a fixed parameter.
Assume further that X follows the distribution N (0 , Σ) and define σ 2 θ as ( θ − θ opt ) T Σ( θ − θ opt ) + σ 2 w .
The loss function (1) is equivalent to
[Equation]
where c 1 = (cid:113) 2 π < 1 , c 2 = 1 − c 21 and ∥ .
∥ q is the dual norm of ∥ .
∥ p , i.e, 1 p + 1 q = 1 .
Furthermore, the above formulation is convex in θ .
The above result is similar to Proposition 3.2 in [27] which provides characterization results for the ℓ 2 norm.
The key distinction of our results is providing a simple convex formulation of the robust minimization problem, allowing the results to be easily generalized for an arbitrary ℓ p norm.
The theorem shows that the optimal value (cid:98) θ minimizing the adversarial loss L p,ϵ ( θ ) is not θ opt and in general, may be non-zero on the set of spurious features S .
This means that adversarial training directs the model towards using the spurious correlations in order to increase robustness.
The proof of the theorem is provided in the Appendix.
The main structure of the proof is similar to that of [27].
We first show that the inner maximization problem of (1) can be rewritten as
[Equation]
This allows us to rewrite (1) as
[Equation]
where for ( a ) we have used the fact that Y = ⟨ X, θ ⟩ + W .
Using the fact that X and W are Gaussian, we show that this is equal to σ 2 θ + ϵ 2 ·
∥ θ ∥ 2 q + 2 · c 1 · ϵ · ∥ θ ∥ q · σ θ which we can further simplify to
(2) with some algebraic manipulation.
Next, using standard techniques for analysing composition of convex functions, we show that σ 2 θ and ( c 1 σ θ + ϵ · ∥ θ
∥ q ) 2 are both convex in θ , implying that L p,ϵ ( θ ) is convex in θ as well.
Using the convex formulation in Theorem 1, we evaluate the linear model for different values of parameters to better understand the reliance of the model on spurious features.
In our experiments, we consider a simple model with 5 features, the first two of which are core features.
We let η be a parameter controlling the correlation degree between the core and spurious features, with larger values corresponding to higher correlation.
For the distribution of the data, define the matrix (cid:101) Q and Q matrices as
[Equation]
Note that Q is obtained by normalizing the rows of (cid:101) Q .
Each row of the Q matrix corresponds to an input feature.
We let Σ take the value QQ T .
This is equivalent to sampling X from the distribution
Q N (0 , I ) .
Throughout our experiments, we set σ w = 0 .
For an arbitrary vector θ , we define its Norm Fraction over Spurious features NFS ( θ ) as
[Equation]
Intuitively, NFS measures the degree to which a model relies on spurious features.
For our first experiment, we perform adversarial training with varying parameter ϵ to obtain a predictor (cid:98) θ and evaluate its NFS.
We consider different choices of the ℓ p norm as well as the correlation parameter η
The results can be seen in Figure 2.
[Figure]
the adversarially trained model on spurious features as measured by NFS (see Equation (5)) for different choices of ℓ p norm and different values of adversarial budget ϵ and spurious correlation parameter η .
As seen in Figure 2, for the ℓ 1 and ℓ 2 norms, increasing the adversary’s budget causes the model to rely more on the spurious features.
To understand why this happens, it is helpful to consider a game theoretic perspective: If the model only looks at the core features, then the adversary will only need to perturb these features.
Thus, even though the spurious features are normally less suitable for prediction, they now have the advantage of being less perturbed.
Assuming ϵ is large enough, the model will be better off looking at these features in forming its prediction.
Of course, if the model only looks at the spurious features, core features will become even more informative as they would be unperturbed as well.
In the game’s equilibrium, the model would use both the core and spurious features, relying more on the spurious features with increased values of ϵ .
Interestingly, this reasoning does not always apply for the ℓ ∞ norm.
Indeed, if the model were to only look at the core features, the adversary may still perturb the spurious features with no extra cost.
This is because the ℓ ∞ norm only measures the maximum perturbation in each feature and as long as the perturbations on the spurious features are not larger than the perturbations on the core features, the norm would not change.
The results of Figure 2c support this as the adversary does not rely on the spurious features for its predictions.
Importantly, however, in some cases the adversary may require a larger budget to perturb the spurious features compared to core features.
Indeed, if we were to scale a feature by multiplication with a large number, then the adversary would require a larger budget to perturb that feature as the budget for that feature has effectively decreased.
We can therefore expect that for the ℓ ∞ norm, scaling up the spurious features would cause the model to rely on them for prediction.
Figure 3 shows that this is indeed the case.
The figure shows the norm fraction over spurious (NFS) after scaling the spurious features for different values of ϵ .
The scaling is done by multiplying the rows of Q corresponding to the spurious features by a scaling parameter.
As seen in Figure 3, larger values of the scaling parameter, as well as larger values of ϵ , cause the model to become more reliant on the spurious features.
Next, we measure the effect of using spurious features on the model’s distributional robustness.
To do this, we train two adversarial models.
The first model, which we denote by “core”, uses only the core features while the second model, denoted by “total”, uses all of the features.
We then simulate a distribution shift that breaks the spurious correlations by adding random
Gaussian noise to the spurious rows of matrix Q defined in (4).
Specifically, for each entry of Q in a spurious row, we add a noise sampled from N (0 , σ 2 Q ) where σ Q is a noise parameter.
We use a fixed value of η = 0 .
and vary the parameter ϵ , the norm p used in adversarial training as well as the variance of the Gaussian noise.
The results are shown in Figure 4.
[Figure]
features on distributional robustness.
Each figures compares two models, one using only the core features, and another using all of the features (denoted by “core” and “total” respectively).
For the ℓ 1 and ℓ 2 norms, the scale of the spurious features is 1 while for the ℓ ∞ norm, the scale is set to 3.
We observe that the clean (i.e. in distribution; σ Q = 0 ) loss of the “core" model may be higher or lower than that of the “total" model, but in both cases, the total model is consistently more vulnerable to distributional shifts resulted from breaking spurious correlations.
Empirical Evidence
We now demonstrate increased spurious feature reliance in adversarially trained models over multiple benchmarks.
We evaluate models on two backbones (ResNet18, ResNet50) adversarially trained on
ImageNet [12] using two norms ( ℓ 2 , ℓ ∞ ) under five attack budgets (denoted ϵ ) per norm, resulting in a 2 × 2 × 5 = 20 model test suite, as well as standardly trained baselines.
See appendix for details.
4.1 AT hurts Natural Distributional Robustness only when Spurious Correlations are broken
We first show reduced distributional robustness of adversarial models occurs specifically in cases where natural spurious correlations are broken.
We appeal to the ImageNet-C [22] and ObjectNet
[5] OOD benchmarks.
ImageNet-C augments ImageNet samples with common corruptions like noise or blurring, distorting both core and spurious features equally.
Crucially, these corruptions do not break spurious correlations.
On the other hand, ObjectNet is formed by having workers capture images of common household objects (including samples from 113 classes of ImageNet) in their homes .
Thus, only spurious features are affected.
Namely, ObjectNet introduces distribution shifts in background, rotation, and viewpoint.
We plot accuracies on these benchmarks in figure 5.
Recall that adversarially trained models have lower standard accuracy, which can confound our analysis, so we compare the drop in OOD accuracy to the drop in ImageNet accuracy across our model suite.
Observe that the ratio of ImageNet-C accuracy to ImageNet accuracy is roughly constant across models.
However, the ratio of ObjectNet accuracy to ImageNet accuracy is lower for adversarially trained models.
Therefore, even after controlling for reduced standard accuracy, the distributional robustness of adversarially trained models is worse than that of standard models.
Importantly, this effect does not hold for distribution shifts that maintain spurious correlations, indicating that the reduced distributional robustness is due to increased spurious feature reliance.
4.2 Reduced Core Sensitivity, and Difference in the Effect of ℓ 2 & ℓ ∞ Adversarial Training
We now directly quantify sensitivity to core features via RIVAL10 and Salient ImageNet-1M datasets [42, 61].
The premise of this analysis is that model sensitivity to an input region can be quantified by the drop in accuracy due to corrupting that region [60].
[42] introduced the noise-based metric relative foreground sensitivity ( RFS ), which is the gap between accuracy drops due to background and foreground noise, normalized so to allow for comparisons across models with varying general noise robustness.
RIVAL10 object segmentations allow for RFS computation.
Analogously, relative core sensitivity ( RCS ) is computed using Salient ImageNet-1M’s soft segmentations of core input regions.
A key distinction between the two metrics is that RCS is computed directly on pretrained models performing the original 1000 -way ImageNet classification task, while RFS first requires models to be finetuned on the much coarser 10 -way classification task of RIVAL10.
Also, Salient
ImageNet-1M includes all ImageNet images , while RIVAL10 only consists of 20 ImageNet classes.
Figure 6 shows a decrease in RFS and RCS as the attack budget ϵ seen during ℓ 2 adversarial training rises.
Thus, adversarial training reduces core feature sensitivity relative to spurious feature sensitivity .
Notably, this effect does not hold for models adversarially trained with attacks under the ℓ ∞ norm for RCS , though it does for RFS .
Alluding to our theoretical result, we conjecture that in
Salient ImageNet-1M, the scales of the spurious features are much smaller than in RIVAL10, due to the diversity of images and finer grain of classes.
That is, a smaller perturbation is needed to alter a spurious feature so that it correlates with an incorrect class when there are 1000 classes than when there are only 10 classes with generally disparate backgrounds.
4.3 Adversarial Training Increases Background Reliance in Synthetic Datasets
Now, we take a closer look at the reliance of adversarially trained models on the contextual spurious feature of backgrounds via the synthetic datasets ImageNet-9 [69] and Waterbirds [50].
Both datasets use segmentations to superimpose objects over varying backgrounds, detailed below.
validation sets exist for IN-9, where backgrounds or foregrounds are altered; we use M IXED -S AME and M IXED -R
AND .
In both sets, original backgrounds are swapped out for new ones.
Crucially, in M IXED -S AME , the new backgrounds are taken from other instances within the same class , while M IXED -R AND uses random backgrounds .
The metric, Background Gap , is the difference in model accuracy on M IXED -S AME and M IXED -R
AND (i.e. drop due to breaking spurious background correlation).
Figure 7 shows that the background gap for our test suite of twenty robust ResNets and two standardly trained baselines.
Nine out of the ten ℓ 2 adversarially trained models have larger background gaps than the standard baselines, while the same is true for five out of the ten ℓ ∞ adversarially trained models.
When considering relative gaps (i.e. as a percent of the accuracy on M IXED -S AME ), the increase in gap becomes even more dramatic, with the ℓ 2 adversarially trained ResNet50 for ϵ
=having a 18 . 9% relative drop, compared to the 9 .
7% relative drop for the standardly trained ResNet50.
Waterbirds combines foregrounds from Caltech Birds [65] and backgrounds from SUN Places [68].
The task is binary classification of land birds and water birds.
The majority group ( 95% of training samples) consists of land birds over land backgrounds and water birds over water backgrounds.
The minority group breaks this spurious correlation, placing land birds over water backgrounds, and vice versa.
The test set is evenly split between these groups.
We train only a final linear layer atop the frozen feature extractors (so that models remain adversarially robust) for each of our models on the
Waterbirds training set for ten epochs, saving the model with highest validation accuracy.
Figure 8 shows majority and minority group accuracies, and the gap between them, for our test suite of models.
Again, we see increased gaps for robust models, with 100% of ℓ 2 and 60% of ℓ ∞ models respectively having larger gaps than the standardly trained baseline on the corresponding backbone.
In both benchmarks, breaking the background spurious correlation causes a more significant drop in performance for adversarially trained models than standardly trained models, indicating that adversarial training led to increased reliance on backgrounds.
The observed affect is stronger for ℓadversarially trained models than ℓ ∞ ones.
Further, the gaps grow near monotonically with ϵ .
4.4 Reverse Effect: Presence of Spurious Correlations Can Improve Adversarial Robustness
Finally, we show evidence that is directly at odds with the claim that spurious features lead to adversarial vulnerability.
We train ResNet18s on CIFAR10 [33] with a spurious feature injected.
Namely, images have all values in one color channel slightly increased.
The majority group consists of red-shifted images from classes 0 − 4 and green shifted images from classes 5 − 9 , while the minority group has reverse color-shifts.
The parameter ρ is the ratio between majority and minority group size, controlling the strength of the spurious correlation (higher ρ means stronger spurious correlation; ρ = 1 : 1 means the spurious feature has no predictive power).
We then evaluate the accuracy of the trained model under adversarial attack on an i.i.d.
test set ( spurious feature retained , no distribution shift).
Figure 9 visualizes the results.
Not surprisingly, the clean accuracy is higher for models trained on data with higher ρ , as the spurious feature is more predictive for higher ρ .
In fact, the added predictive influence the spurious feature leads to better accuracy under attack , with the gap between highest and lowest ρ values growing up to four fold compared to the baseline gap in clean accuracy.
Thus, using a spurious feature can improve adversarial robustness.
Despite being on a contrived example, this experiment shows that, while some spurious correlations may cause adversarial vulnerability, others do the opposite : the picture is more nuanced than previously assumed.
Acknowledgements
This project was supported in part by NSF CAREER AWARD 1942230, HR001119S (GARD), ONR YIP award N00014-22-1-2271, Army Grant No. W911NF2120076, the NSF award
CCF2212458, an AWS Machine Learning Research Award.
A Societal Impact
Our work touches on two important notions of robustness for the safe and fair deployment of deep models in the wild.
We hope our results lead to careful analysis of all modes of robustness, and the interplay between them, before deep models are used in sensitive applications.
While our results create tension with some previous works [73, 77, 26], we stress that we do not wish to diminish their work; instead, we hope our work reveals the vast nuance associated with spurious correlations, which can help and hurt models in various ways.
B Limitations of Theorem
While Theorem 1 provides an understanding of the tradeoff between adversarial and natural distributional robustness, there are some limitations.
Firstly, the results consider a setting where the core and spurious features are completely disentangled, i.e, they each represent different parts of the input.
In practice, spurious features may be entangled with the core features (e.g., the color of an image may represent a spurious feature.)
In addition, our results mainly consider the goal of adversarial training as we focus on the expected loss L p,ϵ ( θ ) , rather than its finite-sample variant.
This is because even for an ℓ 2 adversary, characterizing the finite-sample behaviour of adversarial training is difficult and requires careful assumptions on the asymptotic behaviour of the parameters (e.g., see Theorem 3.3 in [27]).
We leave exploring these directions to future work.
Even so, we believe our theoretical results are of interest to the community since disjoint features already capture a wide variety of spurious correlations, e.g., background correlations, as well as examples where a spurious object is present in the image.
The main goal of our theoretical analysis is to show the existence of explicit tradeoffs between adversarial and distributional robustness and build practical insights using those results.
C Varying the Number of Core and Spurious Features
In this section, we further analyze the plateauing behaviour of the performance of the linear model observed in Figure 2 for different values of core and spurious features with the ℓ 1 and ℓ 2 adversarial training.
We first focus on the ℓ 1 case.
We consider different values for the number of core features c and total features m and measure NFS for different values of adversarial budget ϵ as in Figure 2.
The matrix Σ is constructed using Equation (4) as before, with modified number of rows and columns based on the values of c, p .
Similarly, θ opt is constructed as before, with the core coordinates set to 1 and the spurious coordinates set to 0 .
The value of η is fixed at 0 .
The results are shown in Figure 10.
As shown in the Figure, when using m total features and c core features, NFS plateaus at m − cm for large values of ϵ .
Intuitively, this is because of the structure of the optimization problem (2).
Recall that when using the ℓ 1 norm, the value of q in (2) equals ∞ .
As such, adversarial training tries to find a parameter θ that has a low ℓ ∞ norm and is “close” (as measured by σ θ ) to θ opt .
The ℓ ∞ penalty encourages values of θ that are uniform across the coordinates.
Since there are m − c spurious features and m total features, this leads to models that have an NFS value of m − c m .
We further repeat the above experiment with ℓ 2 norm.
The results are shown in Figure 11.
As seen in the figure, we see the same qualitative results as in Figure 2, with higher values of NFS when increasing numbers of spurious features.
D Proof of Theorem
Proof.
We first claim that
[Equation]
To see why this holds, note that for all δ satisfying ∥
δ ∥
p ≤ ϵ ,
[Equation]
[Figure] [Figure]
where ( a ) follows from the triangle inequality and ( b ) follows from Hölder’s inequality.
With a suitable choice of δ , we can achieve equality for ( b ) .
As ∥ θ ∥ q = ∥− θ ∥ q , at least one of { δ, − δ } would further achieve equality for ( a ) .
As maximizing | .
| is equivalent to maximizing ( . ) 2 , (3) is proved.
Given (3), we can rewrite (1) as
[Equation]
Where for ( a ) we have used the fact that Y = ⟨ X, θ ⟩ + W .
Define v θ as ⟨ X, θ − θ opt ⟩ + W .
As X was assumed to be sampled from N (0 , Σ) , v θ is distributed as N (0 , σ 2 θ ) .
It follows that
[Equation]
where for ( a ) we have used the fact
that E (cid:2)(cid:12)(cid:12) N (0 , σ 2 (cid:12)(cid:12)(cid:3) =
c 1 · σ .
This proves (2) as claimed
[Equation]
As for convexity
[Equation]
σ θ is convex in θ since it can be written as (cid:13)(cid:13)(cid:13)
[Σ 12 ( θ − θ opt ) , σ w ] (cid:13)(cid:13)(cid:13) 2 where [ ., . ] denotes the vector stacking operation.
As c 1 σ θ + ϵ · ∥ θ ∥ q is always positive and x → x 2 is convex and increasing for x ≥ 0 , this implies that ( c 1 σ θ + ϵ · ∥ θ ∥ q ) 2 is convex as well.
Finally c 2 σ 2 θ is convex as c 2 >
and therefore (1) is convex in θ .
E Additional Details on Reverse Effect (Section 4.3)
Our final empirical observation is that the presence of a spurious feature (in both training and test distributions) can lead to increased adversarial robustness.
This more directly creates tension with claims that adversarial vulnerability is born out of spurious feature reliance.
We refer to this as the ‘reverse effect’, in relation to our primary empirical and theoretical finding that adversarial training increases spurious feature reliance.
We now elaborate on the experimental setup discussed in Section 4.3, reproduce the results with a different spurious feature, and finally appeal to ImageNet-9 to demonstrate this effect using a more realistic spurious feature (i.e. backgrounds).
E.1 Experimental Setup
Overview.
We inject spurious correlations to the CIFAR10 dataset.
Based on the class label, we adjust half the images (i.e. with class label from 5 to 9) to shift in one direction with high probability.
For example, a dog image is made greener with probability 0 .
95 , corresponding to a majority-to-minority group ratio of ρ = 19 : 1 .
With probability 0 .
05 , we shift in the other direction (e.g. make redder).
We then standardly train a ResNet18 from scratch on the dataset with the spurious feature injected for the 10-way CIFAR classification task.
Importantly, we evaluate robust accuracy with the spurious feature retained, and then compare adversarial robustness of models trained under data with different strengths of the injected spurious correlation.
Figure 9 and 14 show that for two distinct spurious features (color and lighting), robust accuracy is higher when the spurious correlation is stronger.
Notably, the gain is larger than the gain in standard accuracy.
Intuitively, we see that relying on the predictive power of the spurious feature is helpful for standard accuracy, and especially for accuracy under adversarial attack.
Despite being irrelevant to the true labeling function, the spurious feature can improve model performance, and indeed even lead to better adversarial robustness.
Details.
Color shift is achieved by increasing all pixel intensities along one channel by 0 .
Lighting shift is achieved by simply scaling an input by 1 . 25 to make brighter or 0 .to make darker.
All images are clamped to remain in the [0 , 1] pixel range after spurious feature injection.
Models are trained for 20 epochs using an Adam optimizer with a learning rate of 0 .
001 and weight decay of 1 e − 4 .
E.2 Leveraging ImageNet-
We now demonstrate the observed reverse effect on the higher resolution ImageNet-9 dataset, leveraging the natural and ubiquitous spurious feature of backgrounds.
We finetune pretrained models on M IXED -S AME and M IXED -R AND separately, and evaluate each model’s accuracy under attack on the same split that they were trained over.
Further, we leverage the adversarially trained models from test suite in this experiment.
This way, accuracy under attack is more informative, as the models are trained to expect attacks (i.e. we are not imposing any distribution shifts that would lead to unexpected model behavior).
Along this vain, we attack each backbone with the same norm and ϵ that it was pretrained over.
Figure 15 shows the gain in accuracy for the models trained and evaluated on M IXED -S AME compared to those using M IXED -R
AND .
We see that the presence of background correlations increases both standard and robust accuracy for all models (i.e. gains are positive).
Further, gains in accuracy under attack are larger than gains in standard accuracy in nearly all cases.
Thus, it seems like the added predictive power of the spurious background feature has a significantly nontrivial impact on improving adversarial robustness, contradicting many existing arguments on the link between spurious correlations and adversarial vulnerability.
[Table]
[Table]
F Adversarially Robust Model Test Suite (Section 3) F.1 Model Details
We utilize the treasure trove of open-source adversarially trained models, contributed by [51], accessible at https://github.com/Microsoft/robust-models-transfer .
For completeness, we now provide details on the models we use, though we refer readers to Appendix A.1 of the original text, where the information we share now is sourced.
Training All models were trained on ImageNet in batches of 512 samples, using SGD optimizer with momentum of 0 .
9 and weight decay of 1 e − 4 , for a total of 90 epochs, with learning rate dropping by a factor of 10 every 30 epochs.
The standard procedure of [40] was performed to adversarially train models, using 3 projected gradient descent steps with a step size 23 ϵ for the attack budget ϵ .
Selected Models We focus our empirical study on the ResNet architecture [20] because of its wide spread popularity.
Specifically, we study ResNet18s and ResNet50s that are adversarially trained under the ℓ 2 norm, for ϵ ∈ { 0 . 25 , 0 . 5 , 1 , 3 , 5 } , and ℓ ∞ norm, for ϵ ∈ { 0 . 5 / 255 , 1 / 255 , 2 / 255 , 4 / 255 , 8 / 255 } , as well as standardly trained baselines.
Table 1 shows the standard accuracies for these models.
Note that we at times compare between the ℓ 2 and ℓ ∞ adversarially trained models (e.g. figure 6).
We acknowledge that direct comparisons are challenging because the threat model under which adversarial robustness is optimized for are different.
However, we note that standard accuracies of the i th ℓ 2 AT model is roughly the same as that of the i th ℓ ∞ AT model, suggesting that those models lie in similar points of the accuracy-robustness tradeoff.
Additional Models.
We extend our analysis to other architectures.
We replicate all pretrained-model experiments on the Wide ResNet50 (2x) backbones, for which we have checkpoints for each of the five ϵ values for both ℓ 2 and ℓ ∞ norms.
We also inspect MobileNetv2 [53], DenseNet161 [25],
ResNeXt5050_32x4d [70], ShuffleNet [76], and VGG16_bn [58].
For each of these five architectures, we compare an ℓ 2 adversarially trained model with ϵ = 3 to a standardly trained baseline.
F.2 Experimental Details
ObjectNet and ImageNet-C [5, 22].
We report raw accuracies under noise, blur, and digital corruption types for ImageNet-C, as opposed to relative corruption error.
For ObjectNet, we map
ImageNet predictions to the set of 113 overlapping classes in ObjectNet.
RIVAL10 ( RFS ) and
Salient ImageNet-1M ( RCS ) [42, 61].
RFS computation requires finetuning a final linear layer over fixed features for the coarse-grained ten way RIVAL10 classification.
RCS operates on models off the shelf, directly inspecting accuracies over ImageNet classes (and samples, with region-based noise corruption).
ImageNet-9 and Waterbirds [69, 50].
ImageNet-9 accuracies are obtained by mapping off-the-shelf model predictions to the nine coarse labels deterministically.
Waterbirds requires finetuning, which we do over fixed features.
For RIVAL10 and Waterbirds finetuning, we use Adam with learning rate of 1 e − 4 and weight decay of 1 e − 5 for 20 and 15 epochs respectively.
F.3 Results on Extended Model Test Suite
We now corroborate all our empirical findings on new backbones, expanding our analysis to 21 new models (including 10 AT WideResNet50s over both ℓ 2 and ℓ ∞ norms) over six architectures.
WideResNets.
We corroborate all our empirical findings on ResNet18s and ResNet50s on the WideResNet50 (2x) architecture.
Figure 16 shows that accuracy drop in AT models is more severe on distribution shifts that break spurious correlations (ObjectNet), unlike the accuracy drop due to corruption of both core and spurious features (ImageNet-C), which can likely be explained by the reduced standard accuracy of AT models.
Figure 17 shows reduced sensitivity to core and foreground regions for AT models.
Again, the effect is more pronounced for ℓ 2 adversarially training and for larger ϵ .
Also, we again see that decrease in RCS is less consistent than the drop in RFS .
We conjecture that the diversity and fine-grain Salient ImageNet classification task reduces the strength of spurious correlations present in the data, thus diluting our observed effects of adversarial training on spurious feature reliance.
Lastly, figure 18 shows the drop in accuracy due to breaking spurious background correlations is larger for AT models.
Indeed, the absolute background gap (IN-9) for the WideResNet50 under ℓ 2 AT with ϵ = 5 is 50% larger than the gap for the standardly trained baseline.
We note that the absolute gaps are smaller in some cases.
We believe the lower standard accuracy of AT models may contribute to this, as there is less accuracy to drop from.
Nonetheless, it is intriguing that in some cases, ℓ ∞ adversarial training seems to reduce spurious feature reliance; while our theory explains how a spurious feature can be completely ignored under ℓ ∞ training, it does not explain cases where spurious feature reliance is reduced compared to standard training.
We believe this is an interesting direction for future work.
Other backbones.
We now show results for ten other models, half of which are ℓ 2 adversarially trained with ϵ = 3 , while the others are standardly trained.
Figure 19 summarizes our results, corroborating each of our empirical findings.
[Figure]