E
XPLOITING E XCESSIVE I NVARIANCE CAUSED BY
N ORM -B
OUNDED A DVERSARIAL R OBUSTNESS
Anonymous authors
Paper under double-blind review
A BSTRACT
Adversarial examples are malicious inputs crafted to cause a model to misclassify them.
In their most common instantiation, “perturbation-based” adversarial examples introduce changes to the input that leave its true label unchanged, yet result in a different model prediction.
Conversely, “invariance-based” adversarial examples insert changes to the input that leave the model’s prediction unaffected despite the underlying input’s label having changed.
So far, the relationship between these two notions of adversarial examples has not been studied, we close this gap
We demonstrate that solely achieving perturbation-based robustness is insufﬁcient for complete adversarial robustness.
Worse, we ﬁnd that classiﬁers trained to be (cid:96) p -norm robust are more vulnerable to invariance-based adversarial examples than their undefended counterparts.
We construct theoretical arguments and analytical examples to justify why this is the case.
We then illustrate empirically that the consequences of excessive perturbation-robustness can be exploited to craft new attacks.
Finally, we show how to attack a provably robust defense — certiﬁed on the MNIST test set to have at least 87% accuracy (with respect to the original test labels) under perturbations of (cid:96) ∞ -norm below ε = 0 .
4 — and reduce its accuracy (under this threat model with respect to an ensemble of human labelers) to 60% with an automated attack, or just 12% with human-crafted adversarial examples.
I NTRODUCTION
Research on adversarial examples is motivated by a spectrum of questions.
These range from the security of models deployed in the presence of real-world adversaries, to the need to capture limitations of representations and their (in)ability to generalize (Gilmer et al., 2018a).
The broadest accepted deﬁnition of an adversarial example is “an input to a ML model that is intentionally designed by an attacker to fool the model into producing an incorrect output” (Goodfellow & Papernot, 2017).
Many formal deﬁnitions of adversarial examples were introduced since their initial discovery (Szegedy et al., 2013; Biggio et al., 2013).
In a majority of work, adversarial examples are formalized as adding a perturbation δ to some test example x to obtain an input x ∗ that produces an incorrect model outcome.
1 We refer to this class of malicious inputs as perturbation-based adversarial examples .
To enable concrete progress, the adversary’s capabilities may optionally be constrained by placing a bound on the maximum perturbation δ added to the original input.
The goal of this constraint is to ensure that semantics of the input are left unaffected by the perturbation δ .
In the computer vision domain, (cid:96) p norms have grown to be the default metric to measure semantic similarity.
This led to a series of proposals for increasing the robustness of models to perturbation-based adversaries that operate within the constraints of an (cid:96) p ball.
These include robust optimization (Madry et al., 2017), explicit regularization of a model’s Lipschitz constant (Cisse et al., 2017), or a variety of techniques to build models that are provably robust to small (cid:96) p perturbations (Wong & Kolter, 2018;
Raghunathan et al., 2018; Zhang et al., 2019).
In this paper, we show that optimizing a model’s robustness to l p -bounded perturbations is not only insufﬁcient to address the lack of generalization identiﬁed via adversarial examples, but also potentially harmful .
Intuitively, as (cid:96) p distances are only crude approximations to the true visual
[Figure]
similarity in a given task, over-optimizing a model’s robustness to (cid:96) p -bounded perturbations also renders the model invariant to actual semantics of the underlying task.
Excessive invariance of a model to class semantics can give rise to a new class of adversarial perturbations called invariance adversarial examples (Jacobsen et al., 2019).
These are perturbations that successfully change the human-assigned ground-truth label of an input, while keeping the model’s classiﬁcation of the perturbed input unchanged (for example, we take an image of a ‘1’ digit and perturb it—without changing the model’s classiﬁcation—until a human would agree that the digit now represents a ‘7’).
See Figure 1 for an intuitive illustration of this phenomenon.
Whereas previous efforts (e.g., (Jacobsen et al., 2019)) explored different notions of adversarial examples independently, our work is the ﬁrst to expose a complex relationship between perturbationbased and invariance-based adversarial examples.
Speciﬁcally, we introduce analytical constructions and empirical evidence that shows that increasing a model’s robustness to perturbation-based adversarial examples can increase the model’s vulnerability to invariance-based adversarial examples.
We formally show how to construct a model that is robust to perturbation-based adversarial examples but not to invariance-based adversarial examples.
We further demonstrate how an imperfect model for the adversarial spheres task proposed by Gilmer et al.
(2018b) is either vulnerable to perturbationbased or invariance-based attacks—depending on whether the point attacked is on the inner or outer sphere.
Hence, these two types of adversarial examples are needed to fully account for model failures.
Finally, we empirically demonstrate with a new attack how an adversary can exploit the excessive invariance of certain models.
We introduce the ﬁrst algorithm that crafts invariance-based adversarial examples for the (cid:96)
and (cid:96) ∞ norms, and show that many common models disagree with human labelers on these examples.
In particular, our algorithm breaks a provably-robust defense on MNIST (Zhang et al., 2019).
This model is certiﬁed to have 87% test-accuracy under l ∞ -perturbations of radius ε = 0 .
Yet, on our automatically-generated invariance-based adversarial examples, the model only agrees with the human-assigned label in 60% of the cases; when we manually craft invariance adversarial examples we reduce accuracy to just 12% as determined by an ensemble of humans.
P ERTURBATION - AND I NVARIANCE - BASED A DVERSARIAL EXAMPLES
In order to make precise statements about adversarial examples, we begin with two deﬁnitions.
Deﬁnition 1 (Perturbation-based Adversarial Examples) .
Let G denote the i -th layer, logit or argmax of the classiﬁer.
A Perturbation-based adversarial example (or perturbation adversarial) x ∗ ∈ R d corresponding to a legitimate test input x ∈ R d fulﬁlls:
Created by adversary: x ∗ ∈ R
d is created by an algorithm A : R d → R d with x (cid:55)→ x ∗ .
Perturbation of output: (cid:107) G ( x ∗ ) − G ( x ) (cid:107) >
δ and O ( x ∗ )
=
O ( x ) , where perturbation δ > 0 is set by the adversary and O : R d → { 1 , . .
, C } denotes the oracle .
Furthermore, x ∗ is (cid:15) -bounded if (cid:107) x − x ∗ (cid:107) < (cid:15) , where (cid:107) · (cid:107) is a norm on R d and (cid:15) > 0 .
Property (i) allows us to distinguish perturbation adversarial examples from points that are misclassiﬁed by the model without adversarial intervention.
Furthermore, the above deﬁnition incorporates also adversarial perturbations designed for hidden features as in (Sabour et al., 2016), while usually the decision of the classiﬁer D (argmax-operation on logits) is used as the perturbation target.
Our deﬁnition also identiﬁes (cid:15) -bounded perturbation-based adversarial examples (Goodfellow et al., 2015) as a speciﬁc case of unbounded perturbation-based adversarial examples.
However, our analysis primarily considers the latter, which correspond to the threat model of a stronger adversary.
Deﬁnition 2 (Invariance-based Adversarial Examples) .
Let G denote the i -th layer, logit or argmax of the classiﬁer.
An invariance-based adversarial example (or invariance adversarial) x ∗ ∈ R d corresponding to a legitimate test input x ∈ R d fulﬁlls:
Created by adversary: x ∗ ∈ R
d is created by an algorithm A : R d → R d with x (cid:55)→ x ∗ .
Lies in pre-image of x under G : G ( x ∗ )
=
G ( x ) and O ( x ) (cid:54) =
O ( x ∗ ) ,
where O : R d → { 1 , . . .
, C } denotes the oracle .
As a consequence, D ( x ) = D ( x ∗ ) also holds for invariance-based adversarial examples, where
D is the output of the classiﬁer.
Intuitively, adversarial perturbations cause the output of the classiﬁer to change, while the oracle would still label the new input x ∗ in the original source class.
Whereas perturbation-based adversarial examples exploit the classiﬁer’s excessive sensitivity in task-irrelevant directions , invariance-based adversarial examples explore the classiﬁer’s pre-image to identify excessive invariance in task-relevant directions : its prediction is unchanged while the oracle’s output differs.
Brieﬂy put, perturbation-based and invariance-based adversarial examples are complementary failure modes of the learned classiﬁer.
R OBUSTNESS TO P ERTURBATION - BASED A DVERSARIAL E XAMPLES C AN C AUSE I NVARIANCE - BASED V ULNERABILITIES
We now investigate the relationship between the two adversarial example deﬁnitions from Section 2.
So far, it has been unclear whether solving perturbation-based adversarial examples implies solving invariance-based adversarial examples, and vice versa.
In the following, we show that this relationship is intricate and developing models robust in one of the two settings only would be insufﬁcient.
In a general setting, invariance and stability can be uncoupled.
For example, consider a linear classiﬁer with matrix A .
The perturbation-robustness is tightly related to forward stability (largest singular value of A ).
On the other hand, the invariance-view relates to the stability of the inverse (smallest singular value of A ) and to the null-space of A .
As largest and smallest singular values are uncoupled for general matrices A , the relationship between both viewpoints is likely non-trivial in practice.
3. B UILDING OUR I NTUITION WITH E XTREME U NIFORM C ONTINUITY
In the extreme, a classiﬁer achieving perfect uniform continuity would be a constant classiﬁer.
Let
D : R n → [0 , 1] C denote a classiﬁer with D ( x ) =
y ∗ for all x ∈ R d .
As the classiﬁer maps all inputs to the same output y ∗ , there exist no x ∗ , such that D ( x ) (cid:54) =
D ( x ∗ ) .
Thus, the model is trivially perturbation-robust (at the expense of decreased utility).
On the other hand, the pre-image of y ∗ under D is the entire input space, thus D is arbitrarily vulnerable to invariance-based adversarial examples.
Because this toy model is a constant function over the input domain, no perturbation of an initially correctly classiﬁed input can change its prediction.
[Figure]
This trivial model illustrates how one not only needs to control sensitivity but also invariance alongside accuracy to obtain a robust model.
Hence, we argue that the often-discussed tradeoff between accuracy and robustness (see Tsipras et al.
for a recent treatment) should in fact take into account at least three notions: accuracy, sensitivity, and invariance.
This is depicted in Figure 1.
In the following, we present arguments as for why this insight can also extend to almost perfect classiﬁers.
3. C OMPARING I NVARIANCE - BASED AND P ERTURBATION - BASED R OBUSTNESS
We now show how the analysis of perturbation-based and invariance-based adversarial examples can uncover different model failures.
To do so, we consider the synthetic adversarial spheres problem of Gilmer et al.
(2018b).
The goal of this synthetic task is to distinguish points from two cocentric spheres (class 1: (cid:107) x (cid:107)
= R 1 and class 2: (cid:107) x (cid:107) 2 = R 1 ) with different radii R 1 and R 2 .
The dataset was designed such that a robust (max-margin) classiﬁer can be formulated as:
[Equation]
Our analysis considers a similar, but slightly sub-optimal classiﬁer in order to study model failures in a controlled setting:
[Equation]
which computes the norm of x from its ﬁrst d − n cartesian-coordinates and outputs -1 (resp.
+1)
for the inner (resp.
outer) sphere.
The bias b is chosen based on ﬁnite training set (see Appendix A).
Even though this sub-optimal classiﬁer reaches nearly 100 % on ﬁnite test data, the model is imperfect in the presence of adversaries that operate on the manifold (i.e., produce adversarial examples that remain on one of the two spheres but are misclassiﬁed).
Most interestingly, the perturbation-based and invariance-based approaches uncover different failures (see Appendix A for details on the attacks):
• Perturbation-based: All points x from the outer sphere (i
e., (cid:107) x (cid:107) 2 = R 2 ) can be perturbed to x ∗ , where O ( x ) =
D ( x ) (cid:54) =
D ( x ∗ ) while staying on the outer sphere (i.e., (cid:107) x ∗ (cid:107) 2 = R 2 ).
• Invariance-based: All points x from the inner sphere ( (cid:107) x (cid:107) 2 = R 1 ) can be perturbed to x ∗ , where D ( x ) =
D ( x ∗ ) (cid:54) =
O ( x ∗ ) , despite being in fact on the outer sphere after the perturbation has been added (i.e., (cid:107) x ∗ (cid:107) 2 = R 2 ).
In Figure 2, we plot the mean accuracy over points sampled either from the inner or outer sphere, as a function of the norm of the adversarial manipulation added to create perturbation-based and invariance-based adversarial examples.
This illustrates how the robustness regime differs signiﬁcantly between the two variants of adversarial examples.
Therefore, by looking only at perturbation-based
(respectively invariance-based) adversarial examples, important model failures may be overlooked.
This is exacerbated when the data is sampled in an unbalanced fashion from the two spheres: the inner sphere is robust to perturbation adversarial examples while the outer sphere is robust to invariance adversarial examples (for accurate models).
I NVARIANCE - BASED A TTACKS IN P RACTICE
We now show that our argument is not limited to the analysis of synthetic tasks, and give practical automated attack algorithms to generate invariance adversarial examples.
We elect to study the only dataset for which robustness is considered to be nearly solved under the (cid:96) p norm threat model:
MNIST (Schott et al., 2019).
We show that MNIST models trained to be robust to perturbation-based adversarial examples are less robust to invariance-based adversarial examples.
As a result, we show that while perturbation adversarial examples may not exist within the (cid:96) p ball around test examples, adversarial examples still do exist within the (cid:96) p ball around test examples.
Why MNIST?
The MNIST dataset is typically a poor choice of dataset for studying adversarial examples, and in particular defenses that are designed to mitigate them (Carlini et al., 2019).
In large part this is due to the fact that MNIST is signiﬁcantly different from other vision classiﬁcation problems (e.g., features are quasi-binary and classes are well separated in most cases).
However, the simplicity of MNIST is why studying (cid:96) p -norm adversarial examples was originally proposed as a toy task to benchmark models (Goodfellow et al., 2015).
However, several years later, it is now argued that training MNIST classiﬁers whose decision is constant in an (cid:96) p -norm ball around their training data provides robustness to adversarial examples (Schott et al., 2019; Madry et al., 2017; Wong &
Kolter, 2018; Raghunathan et al., 2018).
MNIST is the only dataset for which robustness to adversarial examples is considered even remotely close to being solved (Schott et al., 2019) and researchers working on (provable) robustness to adversarial examples have moved on to other, larger vision datasets such as CIFAR-10 (Madry et al., 2017;
Wong et al., 2018) or ImageNet (Lecuyer et al., 2018;
Cohen et al., 2019).
This section argues that, contrary to popular belief, MNIST is far from being solved.
We show why robustness to (cid:96) p -norm perturbation-based adversaries is insufﬁcient, even on MNIST, and why defenses with unreasonably high uniform continuity can harm the performance of the classiﬁer and make it more vulnerable to other attacks exploiting this excessive invariance.
4. A TOY WORST - CASE : BINARIZED MNIST CLASSIFIER
[Figure]
To give an initial constructive example, consider a MNIST classiﬁer which binarizes (by thresholding at, e.g., 0.5) all of its inputs before classifying them with a neural network.
As (Tramèr et al., 2018;
Schott et al., 2019) demonstrate, this binarizing classiﬁer is highly (cid:96) ∞ -robust, because most perturbations in the pixel space do not actually change the (thresholded) feature representation.
However, this binary classiﬁer will have trivial invariance-based adversarial examples.
Figure 8 shows an example of this attack.
Two images which are dramatically different to a human (e.g., a digit of a one and a digit of a four) can become identical after pre-processing the images with a thresholding function at 0 .
5 (as examined by, e.g.,
Schott et al.
With this toy worst-case discussed, we now turn to the full MNIST evaluation.
4. G ENERATING M ODEL - AGNOSTIC I NVARIANCE - BASED A DVERSARIAL E XAMPLES
In the following, we build on existing invariance-based attacks (Jacobsen et al., 2019; Behrmann et al., 2018; Li et al., 2019) to propose a model-agnostic algorithm for crafting invariance-based adversarial examples.
That is, our attack algorithm generates invariance adversarial examples that cause a human to change their classiﬁcation, but where most models, not known by the attack algorithm, will not change their classiﬁcation.
[Figure]
Our algorithm for generating invariance-based adversarial examples is simple, albeit tailored to work speciﬁcally on datasets where comparing images in pixel space is meaningful, like MNIST.
Begin with a source image, correctly classiﬁed by both the oracle evaluator (i.e., a human) and the model.
Next, try all possible afﬁne transformations of training data points whose label is different from the source image, and ﬁnd the target training example which—once transformed—has the smallest distance to the source image.
Finally, construct an invariance-based adversarial example by perturbing the source image to be “more similar” to the target image under the (cid:96) p metric considered.
In Appendix B, we describe instantiations of this algorithm for the (cid:96) 0 and (cid:96) ∞ norms.
Figurevisualizes the sub-steps for the (cid:96) 0 attack, which are described in details in Appendix B.
The underlying assumption of this attack is that small afﬁne transformations are less likely to cause an oracle classiﬁer to change its label of the underlying digit than (cid:96) p perturbations.
In practice, we validate this hypothesis with a human study in Section 4.3.
Before evaluating this attack, we make one deﬁnition we will use to discuss the accuracy of
Deﬁnition 3 (Successful (cid:15) -bounded Invariance Attack) .
For a given test example x assigned label y , a successful invariance attack is a perturbed example x ∗ such that the oracle classiﬁcation O ( x ∗ ) (cid:54) =
y and (cid:107) x − x ∗ (cid:107) <
(cid:15) , where (cid:107) · (cid:107) is a norm on R d and (cid:15) >
In practice, we obtain the oracle classiﬁcation by asking an ensemble of human labelers to label the point x ∗ ; if more than some fraction of them agree on the label (throughout this section, 70% ) and that label is different from the original, we call the attack successful.
Note that success or failure is independent of any machine learning model: it only has to do with whether or not the underlying label has actually changed according to the (human) oracle.
4. E VALUATION
Attack analysis.
We generate 1000 adversarial examples using each of the two above approaches on examples randomly drawn from the MNIST test set.
Our attack is slow, with the alignment process taking (amortized) minutes per example.
We performed no optimizations of this process and expect it could be improved.
The mean (cid:96) 0 distortion required is 25.9 (with a median of 25).
The (cid:96) ∞ adversarial examples always use the full budget of 0 . 3 or 0 .
4 and take a similar amount of time to generate; most of the cost is again dominated by ﬁnding the nearest (transformed) training image.
Human Study.
We randomly selected 100 examples from the MNIST test set and createinvariance-based adversarial examples under the (cid:96) 0 norm and (cid:96) ∞ norm, as described above.
We then conduct a human study to evaluate whether or not these invariance adversarial examples indeed are successful as deﬁned earlier, i.e., whether humans agree that the label has been changed.
We additionally hand-crafted 50 invariance adversarial examples under the speciﬁc norms.
The process to create these examples was quite simple: we built a minimal image editor that enabled us change images at a pixel level under a given (cid:96) p constraint.
One author then modiﬁed 50 random test examples in the way that they perceived as changing the underlying class.
We presented 40 human evaluators with these 100 images, half of which were natural unmodiﬁed MNIST digits, and the remaining half were distributed randomly between (cid:96) 0 or (cid:96) ∞ invariance adversarial examples. [Figure]
Results.
For the clean (unmodiﬁed) test images, 98 of the 100 examples were labeled correctly by all human evaluators.
The other 2 images were labeled correctly by over 90% of human evaluators.
Our (cid:96) 0 attack is highly effective:
For 55 of the 100 examples at least 70% of human evaluator who saw that digit assigned it the same way, with a different label from the original test label.
Humans only agreed with the original test label (with the same 70% threshold) on 34 of the images, while they did not form a consensus on the remaining 18 examples.
The (much simpler) (cid:96) ∞ attack is less effective: with a distortion of 0 .
3 the oracle label changed 21% of the time and with 0 .
4 the oracle label changed 37% of the time.
We summarize results in Table 5 (a).
In Figure 5 (b) we show sample invariance adversarial examples.
To simplify the analysis in the following section, we split our generated invariance adversarial examples into two sets: the successes and the failures, as determined by whether the plurality decision by humans was different than or equal to the human label.
We only evaluate the models on the subset of invariance adversarial examples that caused the humans to switch their classiﬁcation.
Model Evaluation.
Given oracle ground-truth labels for each of the images (as decided by humans), we report how often models agree with the human-assigned label.
Table 1 summarizes this analysis.
For the invariance adversarial examples we report model accuracy only on the successful attacks.
Every classiﬁer labeled all successful (cid:96) ∞ invariance adversarial examples incorrectly (with one exception where the (cid:96) 2 PGD-trained classiﬁer Madry et al.
(2017) labeled one of the invariance adversarial examples correctly).
Despite this fact, PGD adversarial training and Analysis by Synthesis
Schott et al.
(2019) are two of the state-of-the-art (cid:96) ∞ perturbation-robust classiﬁers.
The situation is more complex for the (cid:96) 0 -invariance adversarial examples.
In this setting, the models which achieve higher (cid:96) 0 perturbation-robustness result in lower accuracy on this new invariance test set.
For example, Bafna et al.
(2018) develops a (cid:96) 0 perturbation-robust classiﬁer that relies on the sparse Fourier transform.
This perturbation-robust classiﬁer is substantially weaker to invariance adversarial examples, getting only 38% accuracy compared to a baseline classiﬁer’s 54% accuracy.
Breaking Certiﬁed Defenses.
Our invariance attacks are sufﬁciently strong that they constitute a break of some certiﬁed defenses.
For example, Zhang et al.
(2019) develop a certiﬁed defense to (cid:96) ∞ adversarial examples which proves that the accuracy on the test set when perturbed by ε = 0 .
4 is at least 87% .
When we run their pre-trained model on all 100 of our ε = 0 .
4 invariance adversarial examples we ﬁnd it has a 96% “accuracy” (i.e., it matches the original test label 96% of the time).
However, when we compare the accuracy of this model compared to the new labels as assigned by an ensemble of humans, the accuracy is just 63% .
That is, while the proof in the paper is mathematically correct it does not actually deliver 87% accuracy on any new test input perturbed by ε = 0 .
4 : humans would have changed their classiﬁcation in many of these settings.
Worse, for the 50 adversarial examples we crafted by hand, the model disagrees with the human ensemble 88% of the time: it has just 12% accuracy.
Agreement between model and humans, for successful invariance adversarial examples
[Table] [Table]
4. N ATURAL I MAGES
While the previous discussion focused on synthetic (Adversarial Spheres) and simple tasks like
MNIST [Table] similar phenomena may arise in natural images.
In Figure [Table] we show two different (cid:96)perturbations of the original image (left).
The perturbation of the middle image is nearly imperceptible and thus the classiﬁer’s decision should be robust to such changes.
On the other hand, the image on the right went through a semantic change (from tennis ball to a strawberry) and thus the classiﬁer should be sensitive to such changes (even though this case is ambiguous due to two objects in the image).
However, in terms of the (cid:96) 2 norm the change in the right image is even smaller than the imperceptible change in the middle.
Hence, making the classiﬁer robust within this (cid:96) 2 norm-ball will make the classiﬁer vulnerable to invariance-based adversarial examples like the semantic changes in the right image. A similar argument can be made for other norms. For instance Co et al. (2018) show that a perturbation of magnitude 16/255 in (cid:96) ∞ , can sufﬁce to give an image of a cat the appearance to be printed on a shower curtain (both are classes in Imagenet).
However, we leave studying this direction in more detail to future work.
[Figure]
C ONCLUSIONS
Training models robust to perturbation-based adversarial examples should not be treated as equivalent to learning models robust to all adversarial examples.
While most of the research has focused on perturbation-based adversarial examples that exploit excessive classiﬁer sensitivity , we show that the reverse viewpoint of excessive classiﬁer invariance should also be taken into account when evaluating robustness.
Furthermore, other unknown types of adversarial examples may exist: it remains unclear whether or not the union of perturbation and invariance adversarial examples completely captures the full space of evasion attacks.
Consequences for (cid:96) p -norm evaluation and certiﬁed defenses.
Our invariance-based attacks are able to ﬁnd (non-perturbation-based) adversarial examples within the (cid:96) p ball on classiﬁers that were trained to be robust to (cid:96) p -norm perturbation-based adversaries.
As a consequence of this analysis, researchers should carefully set the radii of (cid:96) p -balls when measuring robustness to norm-bounded perturbation-based adversarial examples.
Robustness to small epsilons can indeed lead to a better trade-off between perturbation- and invariance-based robustness (Engstrom et al., 2019).
However, due to the misalignment between p-norms with the "true perceptual distance metric" of the oracle, training against larger epsilon (as is common on MNIST for instance, and we expect will soon become true on larger datasets) will inevitably lead to erroneous invariances and hence to vulnerability with respect to invariance-based adversarial examples.
Furthermore, setting a consistent radius across all of the data may be difﬁcult: we ﬁnd in our experiments that some class pairs are more easily attacked than others by invariance-based adversaries.
In particular, we ﬁnd that recent proposals that claim exceptionally high (cid:96) 0 and (cid:96) ∞ norm-bounded robustness are not meaningfully correct.
Given the claimed perturbation budget, it is possible to generate invariance adversarial examples that fool these defenses.
Indeed, another by-product of our study is to showcase the importance of human studies when the true label of candidate adversarial inputs becomes ambiguous and cannot be inferred algorithmically.
Defenses to adversarial examples on MNIST must stop blindly increasing the distortion threshold and claiming robustness, and in particular for (cid:96) ∞ distortion of ε = 0 . 4 .
Any claim of robustness greater than 60% accuracy is not meaningfully correct because at this distortion level, it is possible to generate examples where humans change their classiﬁcation.
Invariance.
Our work conﬁrms ﬁndings reported recently in that it surfaces the need for mitigating undesired invariance in classiﬁers.
The cross-entropy loss as well as architectural elements such as
ReLU activation functions have been put forward as possible sources of excessive invariance (Jacobsen et al., 2019; Behrmann et al., 2018).
Other work has pointed out a trade-off as a consequence of robustness to perturbations in different frequency spectra (Yin et al., 2019).
However, more work is needed to develop quantitative metrics for invariance-based robustness.
One promising architecture class to control invariance-based robustness are invertible networks (Dinh et al., 2014) because, by construction, they cannot build up any invariance until the ﬁnal layer (Jacobsen et al., 2018; Behrmann et al., 2019).
A D ETAILS ABOUT A DVERSARIAL S PHERES E XPERIMENT
In this section we provide details about the Adversarial Spheres (Gilmer et al., 2018b) experiment.
First the bias b is chosen, such that the classiﬁer D is the max-margin classiﬁer on the (ﬁnite) training set X (assuming separability: l ≤ u ):
[Equation]
Second, the attacks are designed such that the adversarial examples x ∗ stay on the data manifold (two concentric spheres).
In particular, following steps are taken:
Perturbation-based: All points x from the outer sphere (i.e., (cid:107) x (cid:107)
= R 2 ) can be perturbed to x ∗ , where O ( x ) =
D ( x ) (cid:54) =
D ( x ∗ ) , while staying on the outer sphere (i.e., (cid:107) x ∗ (cid:107)
=
R 2 ) via following steps:
Perturbation of decision: x ∗ 1 ,...,d − n = a ( x 1 ,...,d − n ) , where scaling a > 0 is chosen such that (cid:107) x ∗ 1 ,...,d − n (cid:107) 2 < b 2.
Projection to outer sphere: x ∗ d − n,...,d
=
c ( x d − n,...,d ) , where scaling c > 0 is chosen such that (cid:107) x ∗ d − n,...
,d (cid:107)
= (cid:113) R 2 2 − (cid:107) x ∗ 1 ,...,d − n (cid:107) {+
+}
For points x from the inner sphere, this is not possible if b > R 1 .
Invariance-based: All points x from the inner sphere ( (cid:107) x (cid:107) 2 = R 1 ) can be perturbed to x ∗ , where
D ( x ) =
D ( x ∗ ) (cid:54) =
O ( x ∗ ) , despite being in fact on the outer sphere after the perturbation has been added (i.e., (cid:107) x ∗ (cid:107) 2 = R 2 ) via following steps:
Fixing the used dimensions: x ∗ 1 ,...,d − n = x 1 ,...,d − n 2.
Perturbation of unused dimensions:
x ∗ d − n,...,d
= a ( x d − n,...,d ) , where scaling a > 0 is chosen such that (cid:107) x ∗ d − n,...
,d (cid:107)
= (cid:113) R 22 − (cid:107) x ∗ 1 ,...,d − n (cid:107)
For points x from the outer sphere, this is not possible if b > R 1 .
B D ETAILS ABOUT M ODEL - AGNOSTIC I NVARIANCE - BASED A TTACKS
Here, we give details about our model-agnostic invariance-based adversarial attacks on MNIST.
Generating (cid:96)
-invariant adversarial examples.
Assume we are given a training set X consisting of labeled example pairs ( x, y ) .
As input our algorithm accepts an example ˆ x with oracle label
O (ˆ x ) =
ˆ y .
Image ˆ x with label ˆ y = 8 is given in Figure 4 (a).
Deﬁne S =
{ x : ( x, y ) ∈ X , y (cid:54) =
ˆ y } , the set of training examples with a different label.
Now we deﬁne T to be the set of transformations that we allow: rotations by up to 20 degrees, horizontal or vertical shifts by up to 6 pixels (out of 28), shears by up to 20% , and re-sizing by up to 50% .
Now, we generate the new augmented training set X ∗ =
{ ( t ( x ) , y, t ) : t ∈ T , ( x
, y ) ∈ X} .
By assumption, each of these examples is labeled correctly by the oracle.
In our experiments, we verify the validity of this assumption through a human study and omit any candidate adversarial example that violates this assumption.
Finally, we search for
[Equation]
By construction, we know that ˆ x and x ∗ are similar in pixel space but have a different label.
Figure(b-c) show this step of the process.
Next, we introduce a number of reﬁnements to make x ∗ be “more similar” to ˆ x .
This reduces the (cid:96) 0 distortion introduced to create an invariance-based adversarial example—compared to directly returning x ∗ as the adversarial example.
First, we deﬁne δ =
| ˆ x − x ∗ | > 0 .
5 where the absolute value and comparison operator are taken element-wise.
Intuitively, δ represents the pixels that substantially change between x ∗ and ˆ x .
We choose 0 .
5 as an arbitrary threshold representing how much a pixel changes before we consider the change “important”.
This step is shown in Figure 4 (d).
Along with δ containing the useful changes that are responsible for changing the oracle class label of ˆ x , it also contains irrelevant changes that are superﬁcial and do not contribute to changing the oracle class label.
For example, in Figure 4 (d) notice that the green cluster is the only semantically important change; both the red and blue changes are not necessary.
To identify and remove the superﬁcial changes, we perform spectral clustering on δ .
We compute δ i by enumerating all possible subsets of clusters of pixel regions.
This gives us many possible potential adversarial examples x ∗ i
=
ˆ x + δ i .
Notice these are only potential because we may not actually have applied the necessary change that actually changes the class label.
We show three of the eight possible candidates in Figure 4.
In order to alleviate the need for human inspection of each candidate x ∗ i to determine which of these potential adversarial examples is actually misclassiﬁed, we follow an approach from Defense-GAN Samangouei et al.
(2018) and the Robust
Manifold Defense Ilyas et al.
(2017): we take the generator from a GAN and use it to assign a likelihood score to the image.
We make one small reﬁnement, and use an AC-GAN Mirza & Osindero (2014) and compute the class-conditional likelihood of this image occurring.
This process reduces (cid:96)distortion by 50% on average.
As a small reﬁnement, we ﬁnd that initially ﬁltering X by 20% least-canonical examples makes the attack succeed more often.
Generating (cid:96) ∞ -invariant adversarial examples.
Our approach for generating (cid:96) ∞
-invariant examples follows similar ideas as for the (cid:96) 0 case, but is conceptually simpler as the perturbation budget can be applied independently for each pixel (as we will see, our (cid:96) ∞ attack is however less effective than the (cid:96) 0 one, so further optimizations may prove useful).
We build an augmented training set X ∗ as in the (cid:96) 0 case.
Instead of looking for the closest nearest neighbor for some example ˆ x with label
O (ˆ x ) =
ˆ y , we restrict our search to examples ( x ∗ , y ∗ , t ) ∈
X ∗ with speciﬁc target labels y ∗ , which we’ve empirically found to produce more convincing examples (e.g., we always match digits representing a 1 , with a target digit representing either a 7 or a 4 ).
We then simply apply an (cid:96) ∞ -bounded perturbation (with (cid:15) = 0 . 3 ) to ˆ x by interpolating with x ∗ , so as to minimize the distance between ˆ x and the chosen target example x ∗ .
C I NVARIANCE - BASED A DVERSARIAL E XAMPLES FOR B INARIZED MNIST
[Figure]
( l o g s c a l e )
D C OMPLETE S ET OF 100 I NVARIANCE A DVERSARIAL E XAMPLES
Below we give the 100 randomly-selected test images along with the invariance adversarial examples that were shown during the human study.
D. O RIGINAL I MAGES D.
{+(cid:96) 
 I NVARIANCE A DVERSARIAL E XAMPLES D.+}
(cid:96) ∞
I NVARIANCE
A
DVERSARIAL
E
XAMPLES