T ESTING R OBUSTNESS A GAINST U NFORESEEN A DVERSARIES
Anonymous authors
Paper under double-blind review
A BSTRACT
Most existing adversarial defenses only measure robustness to L p adversarial attacks.
Not only are adversaries unlikely to exclusively create small L p perturbations, adversaries are unlikely to remain ﬁxed.
Adversaries adapt and evolve their attacks; hence adversarial defenses must be robust to a broad range of unforeseen attacks .
We address this discrepancy between research and reality by proposing a new evaluation framework called ImageNet - UA .
Our framework enables the research community to test ImageNet model robustness against attacks not encountered during training.
To create ImageNet - UA ’s diverse attack suite, we introduce a total of four novel adversarial attacks.
We also demonstrate that, in comparison to ImageNet - UA , prevailing L ∞ robustness assessments give a narrow account of adversarial robustness.
By evaluating current defenses with
ImageNet - UA , we ﬁnd they provide little robustness to unforeseen attacks.
We hope the greater variety and realism of ImageNet - UA enables development of more robust defenses which can generalize beyond attacks seen during training.
I NTRODUCTION
Neural networks perform well on many datasets (He et al., 2016) yet can be consistently fooled by minor adversarial distortions (Goodfellow et al., 2014).
The research community has responded by quantifying and developing adversarial defenses against such attacks (Madry et al., 2017), but these defenses and metrics have two key limitations.
First, the vast majority of existing defenses exclusively defend against and quantify robustness to
L p -constrained attacks (Madry et al., 2017; Cohen et al., 2019;
Raff et al., 2019; Xie et al., 2018).
Though real-world adversaries are not L p constrained (Gilmer et al., 2018) and can attack with diverse distortions (Brown et al., 2017; Sharif et al., 2019), the literature largely ignores this and evaluates against the L p adversaries already seen during training (Madry et al., 2017; Xie et al., 2018), resulting in optimistic robustness assessments.
The attacks outside the L p threat model that have been proposed (Song et al., 2018; Qiu et al., 2019; Engstrom et al., 2017; Evtimov et al., 2017;
Sharif et al., 2016) are not intended for general defense evaluation and suffer from narrow dataset applicability, difﬁculty of optimization, or fragility of auxiliary generative models.
Second, existing defenses assume that attacks are known in advance (Goodfellow, 2019) and use knowledge of their explicit form during training (Madry et al., 2017).
In practice, adversaries can deploy unforeseen attacks not known to the defense creator.
For example, online advertisers use attacks such as perturbed pixels in ads to defeat ad blockers trained only on the previous generation of ads in an ever-escalating arms race (Tram`er et al., 2018).
However, current evaluation setups implicitly assume that attacks encountered at test-time are the same as those seen at train-time, which is unrealistic.
The reality that future attacks are unlike those encountered during training is akin to a train-test distribution mismatch—a problem studied outside of adversarial robustness (Recht et al., 2019; Hendrycks & Dietterich, 2019)—but now brought to the adversarial setting.
The present work addresses these limitations by proposing an evaluation framework ImageNet - UA to measure robustness against unforeseen attacks.
ImageNet - UA assesses a defense which may have been created with knowledge of the commonly used L ∞ or L 2 attacks with six diverse attacks (four of which are novel) distinct from L ∞ or L 2 .
We intend these attacks to be used at test-time only and not during training.
Performing well on ImageNet - UA thus demonstrates generalization to a diverse set of distortions not seen during defense creation.
While ImageNet - UA
P r e v i ou s A tt ac k s
L
[Figure]
Elastic
[Figure]
N
[Figure]
does not provide an exhaustive guarantee over all conceivable attacks, it evaluates over a diverse unforeseen test distribution similar to those used successfully in other studies of distributional shift (Rajpurkar et al., 2018;
Hendrycks & Dietterich, 2019; Recht et al., 2019).
ImageNet - UA works for ImageNet models and can be easily used with our code available at https://github.com/ anon-submission-2020/anon-submission-2020 .
Designing ImageNet - UA requires new attacks that are strong and varied, since real-world attacks are diverse in structure.
To meet this challenge, we contribute four novel and diverse adversarial attacks which are easily optimized.
Our new attacks produce distortions with occlusions, spatial similarity, and simulated weather, all of which are absent in previous attacks.
Performing well on
ImageNet - UA thus demonstrates that a defense generalizes to a diverse set of distortions distinct from the commonly used L ∞ or L 2 .
With ImageNet - UA , we show weaknesses in existing evaluation practices and defenses through a study of 8 attacks against 48 models adversarially trained on ImageNet-100, a 100-class subset of ImageNet.
While most adversarial robustness evaluations use only L ∞ attacks, ImageNet - UA reveals that models with high L ∞ attack robustness can remain susceptible to other attacks.
Thus,
L ∞ evaluations are a narrow measure of robustness, even though much of the literature treats this evaluation as comprehensive (Madry et al., 2017; Qian & Wegman, 2019; Schott et al., 2019; Zhang et al., 2019).
We address this deﬁciency by using the novel attacks in ImageNet - UA to evaluate robustness to a more diverse set of unforeseen attacks.
Our results demonstrate that L ∞ adversarial training, the current state-of-the-art defense, has limited generalization to unforeseen adversaries, and is not easily improved by training against more attacks.
This adds to the evidence that achieving robustness against a few train-time attacks is insufﬁcient to impart robustness to unforeseen test-time attacks (Jacobsen et al., 2019; Jordan et al., 2019; Tram`er & Boneh, 2019).
In summary, we propose the framework ImageNet - UA to measure robustness to a diverse set of attacks, made possible by our four new adversarial attacks.
Since existing defenses scale poorly to multiple attacks (Jordan et al., 2019; Tram`er & Boneh, 2019), ﬁnding defense techniques which generalize to unforeseen attacks is crucial to create robust models.
We suggest ImageNet - UA as a way to measure progress towards this goal.
R ELATED W ORK
Adversarial robustness is notoriously difﬁcult to correctly evaluate (Papernot et al., 2017; Athalye et al., 2018a).
To that end, Carlini et al.
(2019a) provide extensive guidance for sound adversarial robustness evaluation.
By measuring attack success rates across several distortion sizes and using a broader threat model with diverse differentiable attacks, ImageNet - UA has several of their recommendations built-in, while greatly expanding the set of attacks over previous work on evaluation.
[Figure]
AdversariallyOptimizedJPEG
Randomly
[Figure]
Fog
AdversariallyOptimizedFog
[Figure]
AdversariallyOptimizedSnow
Randomly
[Figure] Gabor [Figure] We are only aware of a few prior works which evaluate on unforeseen attacks in speciﬁc limited circumstances.
Wu et al.
(2020) evaluate against physically-realizable attacks from Evtimov et al.
(2017) and Sharif et al.
(2016), though this limits the threat model to occlusion attacks on narrow datasets.
Outside of vision, Pierazzi et al.
(2020) proposes constraining attacks by a more diverse set of problem-space constraints in diverse domains such as text and malware or source code generation; however, even in this framework, analytically enumerating all such constraints is impossible.
Within vision, prior attacks outside the L p threat model exist, but they lack the general applicability and fast optimization of ours.
Song et al.
and Qiu et al.
(2019) attack using variational autoencoders and StarGANs, respectively, resulting in weaker attacks which require simple image distributions suitable for VAEs and GANs.
Engstrom et al.
(2017) apply Euclidean transformations determined by brute-force search.
Zhao et al.
(2019) use perceptual color distances to align human perception and L 2 perturbations.
Evtimov et al.
(2017) and Sharif et al.
(2016) attack stop signs and face-recognition systems with carefully placed patches or modiﬁed eyeglass frames, requiring physical object creation and applying only to speciﬁc image types.
N EW A TTACKS FOR A B ROADER T HREAT M ODEL
There are few diverse, easily optimizable, plug-and-play adversarial attacks in the current literature; outside of Elastic (Xiao et al., 2018), most are L p attacks such as L ∞ (Goodfellow et al., 2014), L(Szegedy et al., 2013; Carlini & Wagner, 2017), L 1 (Chen et al., 2018).
We rectify this deﬁciency with four novel adversarial attacks: JPEG, Fog, Snow, and Gabor.
Our attacks are differentiable and fast, while optimizing over enough parameters to be strong.
We show example adversarial images in Figure 1 and compare stochastic and adversarial distortions in Figure 2.
Our novel attacks provide a range of test-time adversaries visually and semantically distinct from
L ∞ and L 2 attacks.
Namely, they cause distortions with large L ∞ and L 2 norm, but result in images that are perceptually close to the original.
These attacks are intended as unforeseen attacks not used during training, allowing them to evaluate whether a defense can generalize from L ∞ or L 2 to a more varied set of distortions than current evaluations.
Though our attacks are not exhaustive, performing well against them already demonstrates robustness to occlusion, spatial similarity, and simulated weather, which are absent from previous evaluations.
Our attacks create an adversarial image x (cid:48) from a clean image x with true label y .
Let model f map images to a softmax distribution, and let (cid:96) ( f ( x ) , y ) be the cross-entropy loss.
Given a target class y (cid:48) (cid:54) =
y , our attacks attempt to ﬁnd a valid image x (cid:48)
such that (1) the attacked image x (cid:48) is obtained by applying a distortion (of size controlled by a parameter ε ) to x , and (2) the loss (cid:96) ( f ( x (cid:48) ) , y (cid:48) ) is minimized.
An unforeseen adversarial attack is a white- or black-box adversarial attack unknown to the defense designer which does not change the true label of x according to an oracle or human.
3. F OUR N EW U NFORESEEN A TTACKS JPEG.
JPEG applies perturbations in a JPEG-encoded space of compressed images rather than raw pixel space.
More precisely, JPEG compression is a linear transform JPEG which applies colorspace conversion, the discrete cosine transform, and then quantization.
Our JPEG attack imposes the L ∞ -constraint [Equation] on the attacked image x (cid:48) . We optimize z = JPEG ( x (cid:48) ) under this constraint to ﬁnd an adversarial perturbation in the resulting frequency space.
The perturbed frequency coefﬁcients are quantized, and we then apply a right-inverse of JPEG to obtain the attacked image x (cid:48) in pixel space.
We use ideas from Shin & Song (2017) to make this differentiable.
The resulting attack is conspicuously distinct from L p attacks.
Fog.
Fog simulates worst-case weather conditions.
Robustness to adverse weather is a safety critical priority for autonomous vehicles, and Figure 2 shows Fog provides a more rigorous stress-test than stochastic fog (Hendrycks & Dietterich, 2019).
Fog creates adversarial fog-like occlusions by adversarially optimizing parameters in the diamond-square algorithm (Fournier et al., 1982) typically used to render stochastic fog effects.
This algorithm starts with random perturbations to the four corner pixels of the image.
At step t , it iteratively perturbs pixels at the centers of squares and diamonds formed by those pixels perturbed at step t − 1 .
The perturbation of a step t pixel is the average of the neighboring step t − 1 perturbations plus a parameter value which we adversarially optimize.
We continue this process until all pixels have been perturbed; the outcome is a fog-like distortion to the original image.
Snow.
Snow simulates snowfall with occlusions of randomly located small image regions representing snowﬂakes.
Because the distortions caused by snowﬂakes are not differentiable in their locations, we instead place occlusions representing snowﬂakes at randomly chosen locations and orientations and adversarially optimize their intensities. This choice results in a fast, differentiable, and strong attack.
Compared to synthetic stochastic snow (Hendrycks & Dietterich, 2019), our adversarial snow is faster and includes snowﬂakes at differing angles.
Figure 2 shows adversarial snow exposes model weaknesses more effectively than the easier stochastic, average-case snow.
Gabor.
Gabor spatially occludes the image with visually diverse Gabor noise Lagae et al.
Gabor noise is a form of band-limited anisotropic procedural noise which convolves a parameter mask with a Gabor kernel which is a product of a Gaussian kernel and a harmonic kernel. We choose the Gabor kernel randomly and adversarially optimize the parameters of the mask starting from a sparse initialization.
We apply spectral variance normalization (Co et al., 2019) to the resulting distortion and add it to the input image to create the attack.
3. I MPROVING E XISTING A TTACKS
Elastic modiﬁes the attack of Xiao et al.
; it warps the image by distortions x (cid:48)
=
Flow ( x, V ) , where V : { 1 , . .
, 224 } 2 → R 2 is a vector ﬁeld on pixel space, and Flow sets the value of pixel ( i, j ) to the bilinearly interpolated original value at ( i, j ) + V
( i, j ) .
We construct V by smoothing a vector ﬁeld W by a Gaussian kernel (size 25 × 25 , σ ≈ 3 for a 224 × 224 image) and optimize W under (cid:107) W ( i, j ) (cid:107) ∞
≤ ε for all i, j .
The resulting attack is suitable for large-scale images.
The other three attacks are L 1 , L 2 , L ∞ attacks, but we improve the L 1 attack.
For L ∞ and L 2 constraints, we use randomly-initialized projected gradient descent (PGD), which applies gradient descent and projection to the L ∞ and L 2 balls (Madry et al., 2017).
Projection is difﬁcult for L 1 , and previous L 1 attacks rely on computationally intensive methods for it (Chen et al., 2018; Tram`er & Boneh, 2019).
We replace PGD with the Frank-Wolfe algorithm (Frank & Wolfe, 1956), which [Figure] optimizes a linear function instead of projecting at each step (pseudocode in Appendix D).
This makes our L 1 attack more principled than previous implementations.
ImageNet - UA : M EASURING R OBUSTNESS TO U NFORESEEN A TTACKS
We propose the framework ImageNet - UA and its CIFAR-10 analogue CIFAR - 10 - UA to measure and summarize model robustness while fulﬁlling the following desiderata: (1) defenses should be evaluated against a broad threat model through a diverse set of attacks, (2) defenses should exhibit generalization to attacks not exactly identical to train-time attacks, and (3) the range of distortion sizes used for an attack must be wide enough to avoid misleading conclusions caused by overly weak or strong versions of that attack (Figure 3).
The ImageNet - UA evaluation framework aggregates robustness information into a single measure, the mean Unforeseen Adversarial Robustness ( mUAR ).
The mUAR is an average over six different attacks of the Unforeseen Adversarial Robustness ( UAR ), a metric which assesses the robustness of a defense against a speciﬁc attack by using a wide range of distortion sizes.
UAR is normalized using a measure of attack strength, the ATA , which we now deﬁne.
Adversarial Training Accuracy ( ATA ).
The Adversarial Training Accuracy ATA ( A, ε ) estimates the strength of an attack A against adversarial training (Madry et al., 2017), one of the strongest known defense methods.
For a distortion size ε , it is the best adversarial test accuracy against A achieved by adversarial training against A .
We allow a possibly different distortion size ε (cid:48) during training, since this can improves accuracy, and we choose a ﬁxed architecture for each dataset.
For ImageNet-100, we choose ResNet-50 for the architecture, and for CIFAR-10 we choose ResNet56.
When evaluating a defense with architecture other than ResNet-50 or ResNet-56, we recommend using ATA values computed with these architectures to enable consistent comparison.
To estimate
ATA ( A, ε ) in practice, we evaluate models adversarially trained against distortion size ε (cid:48) for ε (cid:48) in a large range (we describe this range at this section’s end).
UAR : Robustness Against a Single Attack.
The UAR , a building block for the mUAR , averages a model’s robustness to a single attack over six distortion sizes ε 1 , . .
. , ε 6 chosen for each attack (we describe the selection procedure at the end of this section).
It is deﬁned as
[Equation]
where Acc ( A, ε k , M ) is the accuracy Acc ( A, ε k , M ) of a model M after attack
A at distortion size ε k .
The normalization in (1) makes attacks of different strengths more commensurable in a stable way.
We give values of ATA ( A, ε k ) and ε k for our attacks on ImageNet-100 and CIFAR-10 in
Tables 4 and 5 (Appendix B), allowing computation of UAR of a defense against a single attack with six adversarial evaluations and no adversarial training.
mUAR : Mean Unforeseen Attack Robustness.
We summarize a defense’s performance on
ImageNet - UA with the mean Unforeseen Attack Robustness ( mUAR ), an average of UAR scores
L ∞ L 2 L 1 J P E G E l a s t i c F og S n o w G a b o r
Adversarial Attack
None
L ∞ L
L
JPEG
Elastic
Fog
Snow
Gabor
A d v e r s a r i a ll y
T r a i n e d
D e f e n s e 7 17 22 0 31 16 1088 42 15 14 49 20 3780
Defense Robustness Under Diﬀerent Attacks
StandardTraining ε =ε =ε =ε = 8 ε = 16 ε =
Performance of Defenses Adversarially Trained Against L ∞
UAR ( L [Figure]
) mUAR for the L 1 , Elastic, JPEG, Fog, Snow, and Gabor attacks: mUAR := 1 6 (cid:104) UAR ( L 1 )+ UAR ( Elastic )+ UAR ( JPEG )+ UAR ( Fog )+ UAR ( Snow )+ UAR ( Gabor ) (cid:105) .
Our measure mUAR estimates robustness to a broad threat model containing six unforeseen attacks at six distortion sizes each, meaning high mUAR requires generalization to several held-out attacks.
In particular, it cannot be achieved by the common practice of engineering defenses to a single attack, which Figure 4 shows does not necessarily provide robustness to different attacks.
Our four novel attacks play a crucial role in mUAR by allowing us to estimate robustness to a sufﬁciently large set of adversarial attacks.
As is customary when studying train-test mismatches and distributional shift, we advise against adversarially training with these six attacks when evaluating ImageNet - UA to preserve the validity of mUAR , though we encourage training with other attacks. Distortion Sizes. We explain the ε (cid:48) values used to estimate ATA and the choice of ε 1 , . . .
, ε 6 used to deﬁne UAR .
This calibration of distortion sizes adjusts for the fact (Figure 3) that adversarial robustness against an attack may vary drastically with distortion size.
Further, the relation between distortion size and attack strength varies between attacks, so too many or too few ε k values in a certain range may cause an attack to appear artiﬁcially strong or weak according to UAR .
We choose distortion sizes between ε min and ε max as follows.
The minimum distortion size ε min is the largest ε for which the adversarial accuracy of an adversarially trained model at distortion size ε is comparable to that of a model trained and evaluated on unattacked data (for ImageNet-100, within 3 of 87).
The maximum distortion size ε max is the smallest ε which either reduces adversarial accuracy of an adversarially trained model at distortion size ε below 25 or yields images confusing humans (adversarial accuracy can remain non-zero in this case).
As is typical in recent work on adversarial examples (Athalye et al., 2018b; Evtimov et al., 2017;
Dong et al., 2019; Qin et al., 2019), our attacks can be perceptible at large distortion sizes.
We make this choice to reﬂect perceptibility of attacks in real world threat models per Gilmer et al.
For ATA , we evaluate against models adversarially trained with ε (cid:48) increasing geometrically from ε min to ε max by factors of 2 .
We then choose ε k as follows:
We compute ATA at ε increasing geometrically from ε min to ε max by factors of 2 and take the size- 6 subset whose ATA values have minimum (cid:96) 1 -distance to the ATA values of the L ∞ attack in Table 4 (Appendix B.1).
For example, for Gabor, ( ε min , ε max ) =
(6 . 25 , 3200) , so we compute ATAs at the 10 values ε = 6 . 25 , . .
Viewing size- 6 subsets of the ATAs as vectors with decreasing coordinates, we select ε k for Gabor corresponding to the vector with minimum (cid:96) 1 -distance to the ATA vector for L ∞ .
L ∞ training, the most popular defense, provides less robustness than L 2 training.
Comparing the highest mUAR achieved to individual UAR values in Figure 4 indicates a large robustness gap.
[Table]
[Table]
N EW I NSIGHTS F ROM ImageNet - UA
We use ImageNet - UA to assess existing methods for adversarial defense and evaluation.
First,
ImageNet - UA reveals that L ∞ trained defenses fail to generalize to different attacks, indicating substantial weakness in current L ∞ adversarial robustness evaluation.
We establish a baseline for ImageNet - UA using L 2 adversarial training which is difﬁcult to improve upon by adversarial training alone.
Finally, we show non-adversarially trained models can still improve robustness on
ImageNet - UA over standard models and suggest this as a direction for further inquiry.
5. E XPERIMENTAL S ETUP
We adversarially train 48 models against the 8 attacks from Section 3 and evaluate against targeted attacks.
We use the CIFAR-10 and ImageNet-100 datasets for ImageNet - UA and CIFAR - 10 - UA .
ImageNet-100 is a 100 -class subset of ImageNet-1K (Deng et al., 2009) containing every tenth class by WordNet ID order; we use a subset of ImageNet-1K due to the high compute cost of adversarial training.
We use ResNet-56 for CIFAR-10 and ResNet-50 from torchvision for ImageNet-
(He et al., 2016).
We provide training hyperparameters in Appendix A.
To adversarially train against attack A , at each mini-batch we select a uniform random (incorrect) target class for each training image.
For maximum distortion size ε , we apply targeted attack A to the current model with distortion size ε (cid:48) ∼ Uniform(0 , ε ) and take a SGD step using only the attacked images.
Randomly scaling ε (cid:48) improves performance against smaller distortions.
We train on 10 -step attacks for attacks other than Elastic, where we use 30 steps due to a harder optimization.
For L p , JPEG, and Elastic, we use step size ε/ √ steps; for Fog, Gabor, and Snow, we use step size (cid:112) 0 .
001 / steps because the latent space is independent of ε .
These choices have optimal rates for non-smooth convex functions (Nemirovski & Yudin, 1978; 1983).
We evaluate on 200 -step targeted attacks with uniform random (incorrect) target, using more steps for evaluation than training per best practices (Carlini et al., 2019b).
Figure 4 summarizes ImageNet-100 results.
Full results for ImageNet-100 and CIFAR-10 are in
Appendix E and robustness checks to random seed and attack iterations are in Appendix F.
5. ImageNet - UA R EVEALS W EAKNESSESS IN L ∞ T RAINING AND T ESTING
We use ImageNet - UA to reveal weaknesses in the common practices of L ∞ robustness evaluation and L ∞ adversarial training.
We compute the mUAR and UAR ( L ∞ ) for models trained against the L ∞ attack with distortion size ε and show results in Figure 5.
For small ε ≤ 4 , mUAR and
[Table]
UAR ( L ∞ ) increase together with ε .
For larger ε ≥ 8 , UAR ( L ∞ ) continues to increase with ε , but the mUAR decreases, a fact which is not apparent from L ∞ evaluation.
The decrease in mUAR while UAR ( L ∞ ) increases suggests that L ∞ adversarial training begins to heavily ﬁt L ∞ distortions at the expense of generalization at larger distortion sizes.
Thus, while it is the most commonly used defense procedure, L ∞ training may not lead to improvements on other attacks or to real-world robustness.
Worse, L ∞ evaluation against L ∞ adversarial training at higher distortions indicates higher robustness.
In contrast, mUAR reveals that L ∞ adversarial training at higher distortions in fact hurts robustness against a more diverse set of attacks.
Thus, L ∞ evaluation gives a misleading picture of robustness.
This is particularly important because L ∞ evaluation is the most ubiquitous measure of robustness in deep learning (Goodfellow et al., 2014; Madry et al., 2017; Xie et al., 2018).
5. L IMITS OF A DVERSARIAL T RAINING FOR ImageNet - UA
We establish a baseline on ImageNet - UA using L 2 adversarial training but show a signiﬁcant performance gap even for more sophisticated existing adversarial training methods.
To do so, we evaluate several adversarial training methods on ImageNet - UA and show results in Table 1.
Our results show that L 2 trained models outperform L ∞ trained models and have signiﬁcantly improved absolute performance, increasing mUAR from 14 . 0 to 50 .
7 compared to an undefended model.
The individual UAR values in Figure 7 (Appendix E.1) improve substantially against all attacks other than Fog, including several (Elastic, Gabor, Snow) of extremely different nature to L 2 .
This result suggests pushing adversarial training further by training against multiple attacks simultaneously via joint adversarial training (Jordan et al., 2019; Tram ` er & Boneh, 2019) detailed in
Appendix C. Table 2 shows that, despite using twice the compute of L 2 training, ( L ∞ , L 2 ) joint training only improves the mUAR from 50 .
7 to 50 .
We thus recommend L 2 training as a baseline for ImageNet - UA , though there is substantial room for improvement compared to the highest UAR s against individual attacks in Figure 4, which are all above 80 and often above 90 .
5. ImageNet - UA R OBUSTNESS THROUGH N ON -A DVERSARIAL D EFENSES
We ﬁnd that methods can improve robustness to unforeseen attacks without adversarial training.
Table 3 shows mUAR for SqueezeNet (Iandola et al., 2017), ResNeXts (Xie et al., 2016), and ResNets.
For ImageNet-1K models, we mask 900 logits to predict ImageNet-100 classes.
A popular defense against average case distortions (Hendrycks & Dietterich, 2019) is Stylized ImageNet (Geirhos et al., 2019), which modiﬁes training images using image style transfer in hopes of making networks rely less on textural features.
Table 3 shows it provides some improvement on
ImageNet - UA .
More recently, Lopes et al.
(2019) propose to train against Gaussian noise applied to small image patches, improving the mUAR by 3% over the ResNet-50 baseline.
The second largest mUAR improvement comes from training a ResNeXt on approximately 1 billion images (Mahajan et al., 2018).
This three orders of magnitude increase in training data yields a 5.4% mUAR increase over a vanilla ResNeXt baseline.
Finally, Hendrycks et al.
(2020) create AugMix, which randomly mixes stochastically generated augmentations.
Although AugMix did not use random nor adversarial noise, it improves robustness to unforeseen attacks by 10%.
These results imply that defenses not relying on adversarial examples can improve ImageNet - UA performance.
They indicate that training on more data only somewhat increases robustness on ImageNet - UA , unlike many other robustness benchmarks (Hendrycks & Dietterich, 2019;
Hendrycks et al., 2019) where more data helps tremendously (Orhan, 2019).
While models with lower clean accuracy (e.g., SqueezeNet and ResNet-18) have higher UAR ( L ∞ ) and UAR ( L 2 ) than many other models, there is no clear difference in mUAR .
Last, these non-adversarial defenses have minimal cost to accuracy on clean examples, unlike adversarial defenses.
Much remains to explore, and we hope non-adversarial defenses will be a promising avenue toward adversarial robustness.
C ONCLUSION
This work proposes a framework ImageNet - UA to evaluate robustness of a defense against unforeseen attacks .
Because existing adversarial defense techniques do not scale to multiple attacks, developing models which can defend against attacks not seen at train-time is essential for robustness.
Our results using ImageNet - UA show that the common practice of L ∞ training and evaluation fails to achieve or measure this broader form of robustness.
As a result, it can provide a misleading sense of robustness.
By incorporating our 4 novel and strong adversarial attacks,
ImageNet - UA enables evaluation on the diverse held-out attacks necessary to measure progress towards robustness more broadly.
A T RAINING HYPERPARAMETERS
For ImageNet-100, we trained on machines with 8 NVIDIA V100 GPUs using standard data augmentation (He et al., 2016).
Following best practices for multi-GPU training (Goyal et al., 2017), we ran synchronized SGD for 90 epochs with batch size 32 × 8 and a learning rate schedule with“warm-up” epochs and a decay at epochs 30, 60, and 80 by a factor of 10.
Initial learning rate after warm-up was 0 .
1 , momentum was 0 .
9 , and weight decay was 10 − 4 .
For CIFAR-10, we trained on a single NVIDIA V100 GPU for 200 epochs with batch size 32 , initial learning rate 0 .
1 , momentum 0 .
9 , and weight decay 10 − 4 .
We decayed the learning rate at epochs 100 and 150.
B C ALIBRATION OF ImageNet - UA AND CIFAR - 10 - UA B. C ALIBRATION FOR ImageNet - UA
Calibrated distortion sizes and ATA values are in Table 4.
B. C ALIBRATION FOR CIFAR - 10 - UA
The ε calibration procedure for CIFAR-10 was similar to that used for ImageNet-100.
We started with small ε min values and increased ε geometrically with ratio 2 until adversarial accuracy of an adversarially trained model dropped below 40.
Note that this threshold is higher for CIFAR-10 than
ImageNet-100 because there are fewer classes.
The resulting ATA values for CIFAR-10 are shown in Table 5.
C J OINT ADVERSARIAL TRAINING
Our joint adversarial training procedure for two attacks A and A (cid:48) is as follows.
At each training step, we compute the attacked image under both A and A (cid:48) and backpropagate with respect to gradients induced by the image with greater loss.
This corresponds to the “max” loss of Tram ` er & Boneh (2019).
We train ResNet-50 models for ( L ∞ , L 2 ) , ( L ∞ , L 1 ) , and ( L ∞ , Elastic ) on ImageNet-100.
Table 6 shows training against ( L ∞ , L 1 ) is worse than training against L 1 at the same distortion size and performs particularly poorly at large distortion sizes.
Table 7 shows joint training against
[Table] [Table]
[Table] [Table]
( L ∞ , Elastic ) also performs poorly, never matching the UAR score of training against Elastic at moderate distortion size ( ε = 2 ).
D T HE F RANK -W OLFE A LGORITHM
We chose to use the Frank-Wolfe algorithm for optimizing the L 1 attack, as Projected Gradient
Descent would require projecting onto a truncated L 1 ball, which is a complicated operation.
In contrast, Frank-Wolfe only requires optimizing linear functions g (cid:62) x over a truncated L 1 ball; this can be done by sorting coordinates by the magnitude of g and moving the top k coordinates to the boundary of their range (with k chosen by binary search).
This is detailed in Algorithm 1.
E F ULL EVALUATION RESULTS E. F ULL EVALUATION RESULTS AND ANALYSIS FOR I MAGE N ET -
We show the full results of all adversarial attacks against all adversarial defenses for ImageNetin Figure 6.
These results also include L 1 -JPEG and L 2 -JPEG attacks, which are modiﬁcations of the JPEG attack applying L p -constraints in the compressed JPEG space instead of L ∞ constraints.
Full UAR scores are provided for ImageNet-100 in Figure 7.
E. F ULL EVALUATION RESULTS AND ANALYSIS FOR CIFAR-
We show the results of adversarial attacks and defenses for CIFAR-10 in Figure 8.
We experienced difﬁculty training the L 2 and L 1 attacks at distortion sizes greater than those shown and have omitted those runs, which we believe may be related to the small size of CIFAR-10 images.
Full UAR values for CIFAR-10 are shown in Figure 9.
F R OBUSTNESS OF OUR RESULTS F. R EPLICATION
We replicated our results for the ﬁrst three rows of Figure 6 with different random seeds to see the variation in our results.
As shown in Figure 10, deviations in results are minor.
[Figure]
=
[Figure]
P
[Figure]
=
[Figure]
G
[Figure]
J P E G =
L - J P E G =
[Figure]
[Figure]
J P E G =
[Figure]
L
[Figure]
J P E G =
[Figure]
L
[Figure]
J P E G =
[Figure]
L
[Figure]
J P E G =
[Figure]
J P E G =
[Figure]
[Figure]
J P E G =
[Figure]
L [Figure] J P E G =
[Figure]
L
[Figure]
J P E G =
[Figure]
L
[Figure] J P E G =
[Figure]
[Figure]
J P E G =
[Figure]
L
[Figure]
J P E G =
[Figure]
L
[Figure]
J P E G =
[Figure]
L
[Figure]
J P E G =
[Figure]
L 1 - J P E G =
[Figure]
L 1 - J P E G =
[Figure]
L 1 - J P E G =
[Figure]
L 1 - J P E G =
[Figure]
L 1 - J P E G =
[Figure]
L
[Figure]
J P E G =
[Figure]
E l a
[Figure]
t
[Figure]
c =
[Figure]
E
[Figure]
a s t i c =
[Figure]
E
[Figure]
i c = 1 E l a
[Figure]
t i c =
[Figure]
E
[Figure]
a
[Figure]
t i c =
[Figure]
E l a
[Figure]
t i c =
[Figure]
[Table]
N m a a
i
L 2 = L 2 =
L 2 = L 2 =
L 2 = L 2 =
L 1 = 9562 .
L 1 =
L 1 = 38250 .
L 1 = L 1 =
L 1 = L 1 =
L - J P E G = 0 . L - J P E G = 0 .
L - J P E G = 0 . L - J P E G = 0 .
L - J P E G = 0 .
L - J P E G = L - J P E G =
L 2 - J P E G = L 2 - J P E G =
L 2 - J P E G = L 2 - J P E G =
L 2 - J P E G = L 2 - J P E G =
L 2 - J P E G = L 2 - J P E G =
L 1 - J P E G =
L 1 - J P E G = L 1 - J P E G =
L 1 - J P E G = L 1 - J P E G =
L 1 - J P E G = L 1 - J P E G =
L 1 - J P E G = L 1 - J P E G =
L 1 - J P E G = L 1 - J P E G =
E l a s t i c = 0 . E l a s t i c = 0 .
E l a s t i c
=
E l a s t i c
=
E l a s t i c
=
E l a s t i c
=
E l a s t i c
=
F o g = F o g =
F o g = F o g =
F o g = F o g =
F o g = F o g =
F o g = F o g =
G a b o r = 6 .
G a b
o r = 12 .
G a b
o r =
G a b
o r =
G a b
o r =
G a b
o r =
G a b
o r =
G a b
o r =
G a b
o r =
G a b
o r =
S n o w = 0 . S n o w = 0 .
S n o w = 0 . S n o w = 0 .
S n o w = 0 .
S n o w = S n o w =
S n o w = S n o w =
S n o w = 25 10 057110 061290
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
F i gu r e 6 :
A cc u r ac y
o f a dv e r s a r i a l a tt ac k
( c o l u m n ) a g a i n s t a dv e r s a r i a ll y t
r a i n e d m od e l (r o w ) on
I m a g e N e t100 .
: 4: x
6: g ← ∇
8: s
S do 14:
15: b
←
16: else 17: b i ← − i 18: end if 19: end for 20:
M k ← (cid:80) i ∈
S k | b i | { Compute L perturbation of moving k largest coordinates. } 21: k ∗ ← max { k | M k ≤ ρ } { Choose largest k satisfying L 1 constraint.
} 22: 23: { Compute ˆ x maximizing g (cid:62) x over the L 1 ball. } 24: for i = 1 , . . .
, d do 25: if i ∈ S k ∗ then 26: ˆ x i ← x i + b i 27: else if i = s k ∗ +
then 28: ˆ x i ← x i + ( ρ − M k ∗ ) sign( g i ) 29: else 30: ˆ x i ← x i 31: end if 32: end for 33: x ( t ) ← (1 − 1 t ) x
( t − 1) + 1 t ˆ x { Average ˆ x with previous iterates } 34: end for 35: ¯ x ← x ( T )
F. C ONVERGENCE
We replicated the results in Figure 6 with 50 instead of 200 steps to see how the results changed based on the number of steps in the attack.
As shown in Figure 11, the deviations are minor.
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure] [Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
P
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
P E
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
E
[Figure]
[Table]
t
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
E l a s t i c
=
N o r m a l t r a i n i n g
L =
L = L =
L = L =
L =
L 2 = L 2 =
L 2 = L 2 =
L 2 = L 2 =
L 2 = L 2 =
L 2 = L 2 =
L 1 = L 1 =
L 1 = L 1 =
L 1 = L 1 =
L 1 = L 1 =
L 1 =
L - J P E G = 0 . L - J P E G = 0 .
L - J P E G = 0 . L - J P E G = 0 .
L - J P E G = 0 .
L - J P E G =
L 2 - J P E G = 0 . L 2 - J P E G = 0 .
L 2 - J P E G = 0 . L 2 - J P E G = 0 .
L 2 - J P E G = L 2 - J P E G =
L 2 - J P E G = L 2 - J P E G =
L 1 - J P E G =
L 1 - J P E G = L 1 - J P E G =
L 1 - J P E G = L 1 - J P E G =
L 1 - J P E G = L 1 - J P E G =
L 1 - J P E G = L 1 - J P E G =
L 1 - J P E G = L 1 - J P E G =
E l a s t i c = 0 . E l a s t i c = 0 .
E l a s t i c = 0 .
E l a s t i c
=
E l a s t i c
=
E l a s t i c
=
E l a s t i c
=
E l a s t i c
= 59009149143853900 21009283528961200 9000 8937092892522907390
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
F i gu r e 8 :
A cc u r ac y
o f a dv e r s a r i a l a tt ac k
( c o l u m n ) a g a i n s t a dv e r s a r i a ll y t
r a i n e d m od e l (r o w ) on
C
I F
A R10 .
[Table]
[Figure] [Figure]
[Table]
t
[Table] [Table]
A tt a c k
[Table]
s a
[Table]
[Figure] [Figure]
[Table]
N o r m a l t r a i n i
16 L
L 2 = L 2 =
L 2 = L 2 =
L 2 = L 2 =
L 1 = 9562 .
L 1 =
L 1 = 38250 .
L 1 = L 1 =
L 1 = L 1 =
L - J P E G = 0 . L - J P E G = 0 .
L - J P E G = 0 . L - J P E G = 0 .
L - J P E G = 0 .
L - J P E G = L - J P E G =
L 2 - J P E G = L 2 - J P E G =
L 2 - J P E G = L 2 - J P E G =
L 2 - J P E G = L 2 - J P E G =
L 2 - J P E G = L 2 - J P E G =
L 1 - J P E G =
L 1 - J P E G = L 1 - J P E G =
L 1 - J P E G = L 1 - J P E G =
L 1 - J P E G = L 1 - J P E G =
L 1 - J P E G = L 1 - J P E G =
L 1 - J P E G = L 1 - J P E G =
E l a s t i c = 0 . E l a s t i c = 0 .
E l a s t i c
=
E l a s t i c
=
E l a s t i c
=
E l a s t i c
=
E l a s t i c
=
F o g = F o g =
F o g = F o g =
F o g = F o g =
F o g = F o g =
F o g = F o g =
G a b o r = 6 .
G a b
o r = 12 .
G a b
o r =
G a b
o r =
G a b
o r =
G a b
o r =
G a b
o r =
G a b
o r =
G a b
o r =
G a b
o r =
S n o w = 0 .
S n o w = 0 . S n o w = 0 .
S n o w = 0 . S n o w = 0 .
S n o w = S n o w =
S n o w = S n o w =
S n o w = 28 20 058130
[Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table]
Adversarial accuracy
F i gu r e 11 :
R e p li ca o f F i gu r
ew
it
h 50 s t
e p s i n s t ea d o f a t e v a l u a ti on ti m e .
D e v i a ti on s i n r e s u lt s a r
e
m i no r .