Increasing Conﬁdence in Adversarial Robustness Evaluations
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Hundreds of defenses have been proposed to make deep neural networks robustagainst minimal (adversarial) input perturbations.
However, only a handful of thesedefenses held up their claims because correctly evaluating robustness is extremelychallenging: Weak attacks often fail to ﬁnd adversarial examples even if theyunknowingly exist, thereby making a vulnerable network look robust.
In this paper, we propose a test to identify weak attacks, and thus weak defenseevaluations.
Our test slightly modiﬁes a neural network to guarantee the existenceof an adversarial example for every sample.
Consequentially, any correct attackmust succeed in breaking this modiﬁed network.
For eleven out of thirteen previously-published defenses, the original evaluationof the defense fails our test, while stronger attacks that break these defenses passit.
We hope that attack unit tests — such as ours — will be a major component infuture robustness evaluations and increase conﬁdence in an empirical ﬁeld that iscurrently riddled with skepticism.
Introduction
Suppose that someone presents you with a purported proof that P (cid:54) = NP .
The proof is long, complicated,and difﬁcult to follow.
How would you go about checking if this proof is correct?
One cumbersome way would be to directly refute the proof’s claim, e.g., to demonstrate that actually
P = NP by designing an algorithm that solves 3-SAT in polynomial time.
While this would deﬁnitelyrefute the proof, it is likely orders of magnitude more difﬁcult than simply showing that the proof isincorrect.
Accordingly, researchers typically refute incorrect proofs by studying proofs line-by-line,until they identify some major ﬂaw in the reasoning.
Evaluating adversarial defenses is somewhat similar to proving a theorem: It is difﬁcult to estimate therobustness of a defense correctly (just as it is difﬁcult to write down a correct proof), but relatively easyto severely overestimate its robustness (just as it is easy to write down a wrong proof).
This contrastswith many other areas of machine learning, where performance evaluations are often trivial — forexample, by computing accuracy on some held-out test set.
However, evaluating defense robustnessnecessarily involves reasoning over all possible adversaries and showing none can succeed.
Thatis, a defense evaluation aims to prove that something is impossible.
As a result, despite signiﬁcantevaluation effort, most published defenses are later broken by stronger attacks [10, 3, 11, 38, 14].
In this light, we here argue for viewing defense proposals as theorem statements, and the corresponding evaluations as proofs.
The purpose of a defense evaluation, then, is to provide a convincing andrigorous argument that the defense is correct.
Currently, for an adversary to claim to have a “break”of a defense, it is necessary to actually produce the adversarial examples that cause the model tomake an error — analogous to refuting a complexity-theoretic impossibility result by producing an
[Figure]
efﬁcient algorithm.
We argue that this is not how things should work.
A valid refutation of a theoremwould be to say “there is a ﬂaw in your proof on line 9”.
This is because the null hypothesis for atheorem is that it is false, just as the null hypothesis for a defense should be that it is not robust.
Unfortunately, for defenses against adversarial examples, outside of studying the actual code usedto evaluate (i.e., attack) the defense, there are few opportunities to identify ﬂawed evaluations byreading a research paper .
Indeed, current best practices for identifying ﬂawed evaluations are limitedto looking for artifacts that indicate something has gone (terribly) wrong — for example, that anattack fails even when it is allowed to construct unbounded perturbations [11].
We develop a new active robustness test to complement existing (passive) tests [11, 26].
Our testdesigns a new task that is solvable by any sufﬁciently strong attack.
This test can be seen as a “unittest” that the evaluation methodology is correct.
Our test purposefully injects adversarial examplesinto a defense and then checks if the attack used to evaluate the defense is able to ﬁnd them.
Ifthe attack fails this test, we know that the attack is too weak to distinguish between a robust and anon-robust defense, and thus the evaluation should not be trusted.
Returning to our earlier theorem analogy, showing that a defense evaluation fails our test should beviewed similarly as showing a ﬂaw in part of a theorem’s proof.
Note that this does not necessarilyimply that the theorem is not correct (a.k.a. that the defense is not effective), but it should not give usany conﬁdence that it is.
We show that our test would have potentially identiﬁed eleven out of thirteen weak evaluations foundin peer-reviewed papers.
We hope that our testing methodology can become a standard component offuture defense evaluations.
To this end, defenses with exceptionally novel or different techniques,training algorithms, or architectures, may need to develop their own tailored version of our activeunit test, in order to provide convincing evidence that the defense evaluation is indeed correct.
Background
Adversarial Examples Adversarial examples contain imperceptible perturbations that change thedecision of a deep neural network in arbitrary directions [4, 37].
Since they can manipulate thebehavior of a model, they are seen as a security concern for machine learning applications.
To ﬁndadversarial examples for a network, one looks for inputs changing the output of the network whilebeing close (under some norm) to the original data sample.
There are a number of methods to solvethis optimization problem and to attack a network.
Adversarial attacks can be divided into whitebox methods that use gradient information about the model [e.g., 20, 10, 6, 13, 14], and black boxmethods that only use the output of the network [e.g., 5, 17, 1, 23, 2].
Defenses With an increasing awareness of the risk posed by adversarial examples, a vast numberof defenses have been proposed to increase adversarial robustness.
For example, some defenses rely on additional input pre-processing steps [e.g., 16], some introduce architectural changes [e.g.,41], and others propose methods for detecting adversarial examples [e.g., 21].
However, most ofthese defenses eventually turned out to be ineffective against stronger attacks [3, 38].
Until now onlyadversarial training [20] and its variants [e.g., 27, 28, 15] stood the test of time and could not becircumvented.
A different approach to defend classiﬁers against adversarial perturbations are certiﬁeddefenses which give a theoretical guarantee of the classiﬁer’s robustness.
However, the robustness ofthese approaches does not yet reach that of adversarial training [40, 12, 19].
Challenges in Evaluating Defenses Properly evaluating the robustness of a model against adversarial examples is non-trivial and there are many potential pitfalls [11].
The critical issue is that whena defense is shown to be robust to a speciﬁc attack, this either means that the model is truly robust, orthat the attack is suboptimal (see Figure 1A).
Possible reasons for an attack to be suboptimal includemechanisms in the model that (often unintentionally) hinder the attack’s optimization process or poorchoices of hyperparameters [3].
Examples of the former include defenses built around non-continuousactivation functions [e.g., 41] or relying on vanishing gradients [e.g., 36].
To address the latter issue,prior work has developed attacks that alleviate the need to manually tune hyperparameters [14].
But as we will show, these attacks are not guaranteed to work well for any model.
While the latterissue can be counteracted by using adaptive attacks [38] that are adjusted to a speciﬁc model’sidiosyncrasies, it remains non-trivial to detect suboptimal attacks in the ﬁrst place.
Previous worksuggested guidelines for evaluating the adversarial robustness of a model [11] and developed (passive)indicator values hinting at a failed evaluation [26].
Speciﬁcally, Pintor et al.
[26] hook various metricsinto the execution of gradient-based attacks to check for failure cases such as vanishing gradients.
Our work goes beyond these passive failure indicators and argues that researchers should activelydemonstrate that their adversarial attack is sufﬁciently strong, and that their empirical ﬁndings can,thus, be trusted.
Our proposed approach also has the advantage of being agnostic to the type of attackbeing used in an evaluation (e.g., gradient-based, decision-based, transfer-based, etc.).
Active Attack Evaluation Tests
The evaluation of a defense against adversarial attacks becomes more reliable — and the estimatedrobustness more correct — if the attack is sufﬁciently strong.
The strength of an attack is not anabsolute value but depends on the defense it is meant to evaluate, as various defense mechanismshinder speciﬁc attacks [3].
Thus, for a new defense, one needs to demonstrate that the attack proposedto evaluate it is appropriate.
In this section, we propose a test that measures the adequacy of adefense’s evaluation, and is thereby able to warn researchers of potentially unreliable robustnessclaims.
To empirically demonstrate the robustness of a defense for some clean input x c , one runs an attackand shows that it fails to ﬁnd an adversarial example x adv within distance d ( x c , x adv ) ≤ (cid:15) .
But can wereally be sure there are no adversarial examples in the (cid:15) ball if the attack fails?
Since the attack cannotguarantee this, there might exist stronger attacks that do ﬁnd adversarial examples (see Figure 1A).
We propose a test that enables researchers to check whether an attack is too weak to support theirrobustness claims.
In our test, we build a new defense that is as similar as possible to the original,but where we intentionally inject an adversarial example x adv . 1 After building the new (by deﬁnitionvulnerable) defense, we measure its robustness by running the original evaluation method, andchecking whether an adversarial example is found.
If the originally used attack fails to ﬁnd anyadversarial example for the modiﬁed defense, we should not expect it to properly estimate therobustness of the original defense either.
3.1 Test for Classiﬁers with Linear Classiﬁcation Readouts
We begin by describing how to construct the modiﬁed vulnerable model for a classiﬁer f that consistsof a feature extractor f ∗ followed by a linear classiﬁcation head.
Any standard neural networkarchitecture falls into this category: the feature extractor f ∗ comprises every layer except the last, andthe linear classiﬁcation head is the ﬁnal logit projection layer.
Our evaluation methodology keeps thefeature extractor f ∗ unchanged to avoid changing the fundamental behavior of the model, but replaces this idea a completely different context, namely to unit- the classiﬁcation readout with a newly trained module.
This module is trained on a new, speciallyconstructed dataset which allows us to reliably create a classiﬁer where — by design — there existsat least one adversarial example for each sample.
A simpliﬁed pseudocode implementation of ourtest is shown in Algorithm 1; all missing functions used there are deﬁned in Appendix A.2. Algorithm 1 Binarization Test for Classiﬁers with Linear Classiﬁcation Readouts. Missing functions are deﬁned in Appendix A.2.
input: test samples X test , feature extractor f ∗ of original classiﬁer, number of inner/boundary samples N i and N b , distance (cid:15) , sampling functions for data from the inside/boundary of the (cid:15) -ball.
function B INARIZATION T EST ( f ∗ , X test , N b , N i , (cid:15) )
attack_successful
= [] random_attack_successful = [] for all x c ∈ X test do h =
CreateBinaryClassiﬁer( f ∗ , x c , (cid:15) )
# evaluate robustness of binary classiﬁer attack_successful.insert (RunAttack( h, x c ))
random_attack_successful.insert (RunRandomAttack( h, x c ))
ASR =
Mean( attack_successful )
RASR =
Mean( random_attack_successful ) return ASR,
RASR end function function C REATE B INARY C LASSIFIER ( f ∗ , x c ) # draw input samples around clean example
X i = { x c } ∪ { SampleInnerPoint( x c , (cid:15) ) } 1 ,...,N i X b = { SampleBoundaryPoint( x c , (cid:15) ) } 1 ,...,N b
# get features for images
F i = [Equation] f [Equation]
[Equation]
i }
F b = { f ∗ ( x ) [Equation]
x
[Equation]
X b } # deﬁne labels & create labeled dataset [Equation] [Equation] end function
In detail, for each test sample x c our test consists of the following steps:
Creating the Dataset Initially, we create two collections of input samples which are perturbedversions of x c
[Equation]
1) which are sets of points (randomly sampled) from the inside and the boundary of the (cid:15) -ball, respectively, with size
N i , N b > 0 .
Further, ξ ∈ (0 , 1) controls the margin between the inner X i and theboundary set X b .
Decreasing ξ effectively increases the gap between inner and boundary points, thus,making it easier to distinguish between the two sets of samples.
Next, for every sample in each of the two sets, obtain the feature representation of the penultimatelayer of f ,
[Equation]
Training the Vulnerable Classiﬁer Now, we train a linear (binary) discriminator g that distinguishes samples from F i and F b , i.e., it distinguishes between mildly perturbed images — in theinterior of the (cid:15) -ball — and some more strongly perturbed images — on the boundary of the (cid:15) -ball. {+
+}
We want to make sure there exists at least one sample within the threat model’s (cid:15) -ball that g classiﬁesdifferently than the clean original sample.
Due to the construction of the dataset, we can guaranteethis by ensuring that g achieves a perfect accuracy on these two sets.
If this is not possible for anoriginal sample x c , we cannot apply the test and, hence, skip the sample 3 .
Combining the original classiﬁer’s feature extractor f ∗ with the binary discriminator g yields a newclassiﬁer h =
g ◦ f ∗ that maps samples to a binary decision.
Most importantly, for this new classiﬁereach boundary sample X b acts as an (cid:15) -bounded adversarial example x adv for the clean sample x c .
Evaluating the Vulnerable Classiﬁer We evaluate two properties of this new classiﬁer h :1.
The efﬁcacy of the used evaluation method/adversarial attack.
For this, one uses the originaladversarial attack to attack the modiﬁed model h for the clean sample x c and records whetheran adversarial sample x ∗ within the allowed (cid:15) -ball is found.
When calculated and averaged overmultiple samples, we call this value the test score .
The difﬁculty of the test
To assess this, we use a model-agnostic attack, namely a purelyrandomized one.
We attack the modiﬁed classiﬁer h by randomly sampling approximately asmany additional data points from within the (cid:15) -ball around the clean sample x c as the adversarialattack queries the model, e.g. for an N -step PGD attack [20] use N additional random samples.
Finally, one tests whether at least one of them turns out to be an adversarial perturbation for h .
Averaging over multiple samples yields the random attack success rate (R-ASR).
Note that if the classiﬁer f does not use a linear classiﬁcation readout, one has to modify the testslightly: Instead of using a linear readout for g one needs to use the same type of mechanism thatwas used originally.
While this modiﬁcation is conceivable for various mechanisms, e.g., k-nearestneighbors classiﬁcation [35], there might be architectures for which this is harder or impossible, e.g.,classiﬁcation through likelihood estimations based on generative models [32, 45].
3.2 Tests for Models Leveraging Detectors
Detection defenses use an additional algorithm that detects and rejects adversarial examples [21].
Asuccessful attack, thus, has to fool both the classiﬁer and the detector, and we have to probe both inour active test.
As earlier, we assume that the classiﬁer can be divided into a feature encoder and linear readout.
We make no assumptions on the architecture of the detector, as a wide variety of designs have beenproposed in the literature.
We deﬁne two tests: a regular and an inverted test.
Any reliable evaluationmethod must pass both.
A pseudocode deﬁnition of the proposed tests is given as Algorithm 2 in the
Appendix.
Regular Test Adversarial examples for a detection defense need to change the classiﬁer’s outputwhile also remaining undetected.
Thus, to still guarantee that there exists a valid adversarial example,we need to change the construction of the binary classiﬁer slightly to take into account the detector.
Speciﬁcally, we modify the set X b of boundary points such that none of these samples gets rejectedby the detector.
In practice, we enforce this with rejection sampling, by redrawing boundary pointsuntil
we ﬁnd a point that is undetected.
Note, that we make no modiﬁcations to the detector, sincethis might require non-trivial optimization of the detector’s parameters.
Some adversarial attacks for detector defenses (e.g., feature matching [30]) assume access to referencedata samples that belong to a different class but are not adversarial and, thus, do not get rejected.
Inour setting this can be realized by randomly sampling data points outside the (cid:15) ball.
Thus, we create anew collection
[Equation]
for which the binary classiﬁer must predict the same class as for the boundary samples X b .
Here,
N r ≥ 0 and η > 1 . 0 control the number of samples and how far outside of the (cid:15) ball they are located.
Again, as for X b , we need to ensure that none of these samples get detected.
By training the linearreadout on X i , X b and X r
we guarantee that there exists at least one undetected adversarial samplewithin the (cid:15) -ball around x c , and at least N r samples outside the (cid:15) -ball that are also undetected.
Inverted Test Since the detector was tuned for a classiﬁcation problem different from the oneposed in the regular test above, it might not work well and rarely reject samples.
Thus, a potentialissue with the regular test is that a weak attack might pass the test even though the attack completelyignores the detector.
Indeed, many evaluations of detector defenses consider weak attacks that areoblivious to the presence of the detector [9].
Thus, an attack passing the test may not be sufﬁcient totell us that the attack is actually strong enough to successfully target the detector.
To this end, we introduce a second inverted test that inverts the attack’s goal: Instead of ﬁndingadversarial samples that do not get detected, the goal is now to ﬁnd an adversarial example thatdoes get detected.
Since any detector that claims non-zero robustness must detect some adversarialexamples, we use these to construct the set of boundary samples X b .
Finally, we only need to negatethe decision of the detection algorithm before proceeding exactly as for the previously described test.
Passing both the regular as well as the inverted test is a necessary condition for an adequate adversarialattack.
In fact, this indicates that the attack is not agnostic to the detector but properly takes it intoaccount.
In contrast, passing only one of the tests indicates that only the classiﬁer and not the detectoris directly targeted.
Evaluation
We now show that our binarization test could have revealed insufﬁciently strong evaluations in elevenout of thirteen previously peer-reviewed and published defenses.
Of these eleven defenses, nine hadalready been broken by subsequent attacks — thereby conﬁrming that the original evaluation wasindeed too weak.
For the other two defenses that fail our test, closer inspection of both defenses’evaluations and code indeed reveal serious issues, and we show that the defenses are broken bystronger attacks.
All of these defenses assume an (cid:96) ∞ threat model.
The speciﬁc design choices forthe tests adapted to each defense can be found in Appendix B.
Defenses without Detectors We analyze eight defenses which use a classiﬁer with a linear classiﬁcation readout [8, 41, 43, 22, 25, 33, 31, 44].
We additionally apply our test to two defenses that do not use a simple linear readout to performclassiﬁcation.
However, it is straightforward to adapt the binarization test deﬁned in Algorithm 1for these classiﬁer architectures.
The defense by Verma and Swami [39] leverages an ensemble ofreadouts.
For our test, we therefore also train an ensemble of binary readouts.
The classiﬁer of
Pang et al.
[24] learns to map images to pre-deﬁned class-prototype vectors, and then uses nearestneighbour classiﬁcation.
We reﬂect this in the test by using two of the class prototypes and associatingthem with the inner and boundary samples, respectively.
Then we re-train a linear layer mappingfrom features to class prototypes.
In fact, we are the ﬁrst to show that the defense by Sarkar et al.
[31] is less robust than originallyreported, as suggested by the fact that it fails our test.
All of the above defenses except those by
Sarkar et al.
[31] were known to be ﬂawed and have been circumvented before.
Defenses with Detectors We investigate three published defenses that aim to detect adversarialperturbations.
Following previous work [7], we analyze each defense in a setting where it achieves afalse positive rate of 5 %.
While the detection algorithm proposed by Roth et al.
[29] runs statisticaltests on the classiﬁer’s conﬁdence, Shan et al.
[34] and Yang et al.
[42] analyze earlier activations ofthe classiﬁer.
The ﬁrst two defenses have been broken before [38, 7] while the latter had not beenindependently re-evaluated yet.
1 Initial Evaluation of Not-previously-broken Defenses
We begin by investigating the two recent and not yet broken defenses.
Here, we are interested inseeing whether the original robustness evaluations pass our binarization test.
While a positive resultwould increase conﬁdence in the defenses’ claims, a negative outcome would cast doubts.
Sarkar et al.
[31] The original evaluation of this defense fails our test with a test score of 0 . 04 .
That is, the original attack only ﬁnds an adversarial example in the modiﬁed binary classiﬁer 4% of
[Figure]
al
[Figure]
the time — even though at least one adversarial example is guaranteed to exist for every test sample.
This is strong evidence that the attack is weak and thus the robustness claim likely overestimated.
Upon investigation, we found a ﬂaw in the original evaluation’s code: The statistics of the batchnormalization layers are not frozen during evaluation, which changes the behavior of the modelduring the attack.
Properly freezing these layers at inference and increasing the number of PGD stepsfrom 20 to 75 yields a perfect score ( 1 . 0 ) in our binarization test.
Moreover, this updated evaluationmethodology reduces the robust accuracy to ≤ 1 % down from the originally reported 60 .
15 % and,thus, effectively breaks the defense.
Yang et al.
For this detector-based defense, we ﬁnd that the attack used in the originalevaluation is agnostic to the detector and only targets the classiﬁer.
Consequentially, this attack failsboth the regular and the inverted binarization test with a low score of 0 .
26 and 0 . 63 , respectively.
We thus create a new adversarial attack based on PGD that combines two objectives: (1) fool theclassiﬁer by maximizing the adversarial loss and (2) stay undetected by matching the features of anon-adversarial sample as much as possible (a feature matching attack [30, 38]).
This adaptive attackachieves a nearly perfect score of 0 .
99 in the test and reduces the robust accuracy of the defense fromthe originally reported 99 % down to below 12 %.
4.2 Interpreting Test Results for Weak and Strong Attacks
Since eleven of the considered defenses have already been broken before, and we showed howto break the remaining two, we now have access to both a ﬂawed and a well-working adversarialevaluation method for each defense.
This allows us to compare how these attacks perform in terms ofboth the estimated robust accuracy and the score on the binarization test.
We visualize the results in
Figure 2.
For eleven out of the thirteen considered defenses, our proposed test would have ﬂaggedtheir evaluation as insufﬁcient: the original attacks’ test performance is substantially below a perfectscore (i.e., < 0 . 95 ).
Furthermore, the test scores improve for all defenses when replacing the originalevaluation code with an improved attack (the defense of Sen et al.
[33] is an exception, as the originalevaluation already obtains a perfect test score due to an integration bug, as described below).
Explaining the False Positives Our test incorrectly lets two defense evaluations that had bugs pass(see red markers in Figure 2).
When investigating these failure cases in more detail, we ﬁnd thatthe original attack used by Sen et al.
[33] is not ﬂawed or incorrectly implemented per se, but it is not used correctly.
Namely, the attack generates adversarial examples with respect to the classiﬁer’spredicted label, instead of the ground-truth label.
As a result, for some misclassiﬁed samples theattack actually corrects the classiﬁer’s mistake!
By switching to an attack that correctly targets theground-truth label, we reduce the robust accuracy drastically.
Our test is unable to catch such a mistake: By design, our test constructs a binary classiﬁer with 100%accuracy (and thus the classiﬁer’s predicted label is always equal to the ground-truth).
If we view ourproposed test as a unit test for an attack, then the type of bug in the above evaluation is akin to anintegration bug , where the (correct) attack is called with incorrect parameters.
For the defense by Zhang and Wang [43] we notice, a high R-ASR value ( > 0 .
75 ) that we could notdecrease further.
Thus, for this defense, our binarized classiﬁer is too easy to attack.
We hypothesizethat by increasing the number of inner samples
N i substantially, the test might become hard enoughto indicate sub-optimal evaluations for this defense.
Explaining the Suboptimal Re-evaluations There are also four defenses for which the improvedattacks still fail our test, even though their test performance is better than for the original attacks.
The authors of the improved attack for Pang et al.
[25] note that while this attack already breaks thedefense, one could improve the attack further [38].
For the defenses by Mustafa et al.
[22] and Zhangand Xu [44], the improved attacks are not adaptive attacks but part of AutoAttack’s attack collection[14].
Although these attacks were sufﬁcient to drastically reduce the measured robustness of thedefenses (see Appendix, Figure 4), they are likely not optimal attacks for these defenses.
While theattack [7] used for the re-evaluation breaks the defense by Shan et al.
[34], the imperfect test scorehints at the possibility of an even more potent and yet-to-be-discovered attack.
[Figure]
4.3 Hardness of the Test
To put the performance that an adversarial attack achieves in the binarization test into perspective wequantify the hardness of the test using the previously introduced random attack success rate (R-ASR).
Comparing it to the test result of the attack allows us to deduce how effective the attack is in ﬁndingadversarial examples for the model in question.
There are several parameters and design choices relevant for our test that inﬂuence its hardness.
Forone, by increasing N i
we train the binary discriminator on a larger number of different non-adversarialpoints which increases the robustness of the discriminator and, thus, makes the test harder.
Conversely,by increasing N b we plant a larger number of adversarial examples for the discriminator within the(cid:15) -ball, making the test simpler.
Even with a large but ﬁnite number of training samples, there is no unique solution for the binarydiscriminator but instead a set of valid solutions.
While all of these classiﬁers have perfect accuracyon the training set, they differ in how close the decision boundary is placed to the boundary samples.
The closer the decision boundary is placed to the boundary samples, the smaller the volume of validadversarial examples and, thus, the harder the test becomes.
The effect of the the decision boundary’scloseness on the test’s hardness is illustrated in Figure 3 for the defense of Mustafa et al.
Here,placing the boundary closer to the boundary samples decreases both the R-ASR as well as the ASRof two sub-optimal attacks while that of a better suited attack stays robustly at 1 . 0 .
On the one hand, a test that is too easy has no predictive power about the attack’s efﬁcacy (since anyattack might trivially pass it), while on the other hand, a test that is too challenging might actuallyunderestimate the attack’s true performance (i.e., ﬁnding an adversarial example in the modiﬁeddefense might be harder than in the original defense, and thus even a strong attack against the defensemight fail the test if it is tuned too aggressively).
Therefore, one needs to tune the test’s hardness to areasonable level.
We recommend the following procedure to adjust the test’s hardness: To ensure we do not overestimatethe test performance (since this is the more dangerous direction), we start with a conﬁguration thatmakes the test as hard as possible.
Then, decrease the hardness until the adversarial attack in questionreaches an (almost) perfect ASR.
Note that if there is no conﬁguration that yields a near-perfect ASR,then the attack did not pass the test and one should be skeptical of the attack’s ability to properlyestimate the classiﬁer’s robustness.
Finally, compare the ASR with the R-ASR (the success rate of arandom attack): If the ASR is not substantially higher — or is even lower — than the R-ASR, this isstrong evidence that the attack performs poorly.
If instead the gap is large, the attack has passed thisnecessary test and might be powerful enough to properly estimate the classiﬁer’s robustness.
Discussion & Conclusion
This paper made a case for active tests to evaluate adversarial robustness.
The goal of an active test isto provide compelling evidence that an attack has sufﬁcient power to evaluate a classiﬁer’s robustness.
We presented such a test for defenses using linear classiﬁcation readouts and showed how to adaptthis test for different defense mechanisms such as detector-based defenses.
The type of test proposedin this work acts as a necessary condition for robustness evaluations, i.e., an attack that fails the testwill most likely overestimate the defense’s robustness.
While we have presented a potential test that could help defense authors demonstrate sufﬁcient powerof their adversarial evaluation, our tests are not meant to be comprehensive and directly applicableto every possible defense.
For example, our tests are primarily designed to work for defenses thatuse linear classiﬁcation readout layers.
If a defense were to have a different classiﬁcation layerinstead, such as a k-Nearest Neighbor classiﬁer, then our tests would need to be modiﬁed accordingly.
Consequentially, defense authors should aim to develop their own attack unit tests, depending on theparticular claims made.
Our test could have revealed weak evaluations in eleven out of thirteen previously peer-reviewed (andsubsequently broken) defenses.
We are thus optimistic that active tests can improve the reliability offuture publications in the ﬁeld of adversarial robustness.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] We discuss the limitations of our
proposed method in Section 5.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes]
Our
work has to do with adversarial examples but our work focuses on improving defenses
against adversarial attacks and not creating better attacks.
Thus, our work has a positive
societal impact since it makes machine learning models more reliable.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
This paper
presents an empirical method to validate the results of a robustness evaluation method.
Therefore, there are no theoretical results in the paper.
(b) Did you include complete proofs of all theoretical results?
[N/A] See the response
above.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] We included
the implementation of our proposed test as well as the code to reproduce the results for
the defenses investigated in this work.
(b) Did you specify all the training details (e
g., data splits, hyperparameters,
how they
were chosen)?
[Yes] A detailed overview of the experimental details including the
hyperparameters used is given in Section B
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[No] We follow standard practice when reporting results
regarding the adversarial robustness of neural networks, i.e., we run the evaluation on a
sufﬁciently large sample size such that the overall result does not vary substantially
across runs.
(d) Did you include the total amount of compute and the type of resources used (e.g., type
of GPUs, internal cluster, or cloud provider)?
[Yes] A description of the used hardware
and total amount of compute is presented in section B.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] Since our study inves-
tigates thirteen published defenses against adversarial attacks, we used the respective
open-source reference implementation for our experiments.
We cited all authors and
included their original licenses in our source code.
We also cited the authors of the
dataset used to evaluate the defenses on.
(b) Did you mention the license of the assets?
[Yes] We include the license of the defenses’
code in the submitted code.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
We include the code for our test as well as the modiﬁcations done to the defenses’
reference implementations to run the test in the supplemental material.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[Yes] In this work we are only using data that was explicitly made
available for research and cited the authors.
No further data was used, curated or
collected.
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[N/A] Since the dataset used in this work is a standard
dataset (CIFAR-10) we do not discuss the aforementioned issues.
However, due to its
low image resolution
, the dataset is unlikely to contain problematic content.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
[Table]
applicable?
[N/A]
We did not conduct any research with human participants.
[Table]
(b) Did you describe any potential participant risks, with links to Institutional Review
[Table]
Board (IRB) approvals, if applicable?
[N/A]
We did not conduct any research with
[Table]
human participants.
[Table]
(c) Did you include the estimated hourly wage paid to participants and the total amount
[Table]
spent on participant compensation?
[N/A]
We did not conduct any research with human
[Table]
participants.
[Table]