O N THE G EOMETRY OF A DVERSARIAL E XAMPLES
Anonymous authors
Paper under double-blind review
A BSTRACT
Adversarial examples are a pervasive phenomenon of machine learning models where seemingly imperceptible perturbations to the input lead to misclassiﬁcations for otherwise statistically accurate models.
We propose a geometric framework, drawing on tools from the manifold reconstruction literature, to analyze the high-dimensional geometry of adversarial examples.
In particular, we highlight the importance of codimension : for low-dimensional data manifolds embedded in high-dimensional space there are many directions off the manifold in which to construct adversarial examples.
Adversarial examples are a natural consequence of learning a decision boundary that classiﬁes the low-dimensional data manifold well, but classiﬁes points near the manifold incorrectly.
Using our geometric framework we prove (1) a tradeoff between robustness under different norms, (2) that adversarial training in balls around the data is sample inefﬁcient, and (3) sufﬁcient sampling conditions under which nearest neighbor classiﬁers and ball-based adversarial training are robust.
I NTRODUCTION
Deep learning at scale has led to breakthroughs on important problems in computer vision (Krizhevsky et al.
(2012)), natural language processing (Wu et al.
(2016)), and robotics (Levine et al.
Shortly thereafter, the interesting phenomena of adversarial examples was observed.
A seemingly ubiquitous property of machine learning models where perturbations of the input that are imperceptible to humans reliably lead to conﬁdent incorrect classiﬁcations (Szegedy et al.
(2013); Goodfellow et al.
What has ensued is a standard story from the security literature: a game of cat and mouse where defenses are proposed only to be quickly defeated by stronger attacks (Athalye et al.
This has led researchers to develop methods which are provably robust under speciﬁc attack models (Madry et al.
; Wong & Kolter (2018); Sinha et al.
Raghunathan et al.
As machine learning proliferates into society, including security-critical settings like health care (Esteva et al.
(2017)) or autonomous vehicles (Codevilla et al.
, it is crucial to develop methods that allow us to understand the vulnerability of our models and design appropriate counter-measures.
In this paper, we propose a geometric framework for analyzing the phenomenon of adversarial examples.
We leverage the observation that datasets encountered in practice exhibit low-dimensional structure despite being embedded in very high-dimensional input spaces.
This property is colloquially referred to as the “Manifold Hypothesis”: the idea that low-dimensional structure of ‘real’ data leads to tractable learning.
We model data as being sampled from class-speciﬁc low-dimensional manifolds embedded in a high-dimensional space.
We consider a threat model where an adversary may choose any point on the data manifold to perturb by (cid:15) in order to fool a classiﬁer.
In order to be robust to such an adversary, a classiﬁer must be correct everywhere in an (cid:15) -tube around the data manifold.
Observe that, even though the data manifold is a low-dimensional object, this tube has the same dimension as the entire space the manifold is embedded in.
Our analysis argues that adversarial examples are a natural consequence of learning a decision boundary that classiﬁes all points on a low-dimensional data manifold correctly, but classiﬁes many points near the manifold incorrectly.
The high codimension , the difference between the dimension of the data manifold and the dimension of the embedding space, is a key source of the pervasiveness of adversarial examples.
Our paper makes the following contributions.
First, we develop a geometric framework, inspired by the manifold reconstruction literature, that formalizes the manifold hypothesis described above and our attack model.
Second, we highlight the role codimension plays in vulnerability to adversarial
[Figure]
Intuitively, the decision axis captures an optimal decision boundary between the data manifolds.
It’s optimal in the sense that each point on the decision axis is as far away from each data manifold as possible.
Notice that in the ﬁrst example, the decision axis coincides with the maximum margin line.
examples.
As the codimension increases, there are an increasing number of directions off the data manifold in which to construct adversarial perturbations.
Prior work has attributed vulnerability to adversarial examples to input dimension (Gilmer et al.
This is the ﬁrst work that investigates the role of codimension in adversarial examples.
Interestingly, we ﬁnd that different classiﬁcation algorithms are less sensitive to changes in codimension.
Third, we apply this framework to prove the following results: (1) we show that the choice of norm to restrict an adversary is important in that there exists a tradeoff between being robust to different norms: we present a classiﬁcation problem where improving robustness under the (cid:107)·(cid:107) ∞ norm requires a loss of Ω(1 − 1 / √ d ) in robustness to the (cid:107)·(cid:107) 2 norm; (2) we show that a common approach, training against adversarial examples drawn from balls around the training set, is insufﬁcient to learn robust decision boundaries with realistic amounts of data; and (3) we show that nearest neighbor classiﬁers do not suffer from this insufﬁciency, due to geometric properties of their decision boundary away from data, and thus represent a potentially robust classiﬁcation algorithm.
Finally we provide experimental evidence on synthetic datasets and
MNIST that support our theoretical results.
R ELATED W ORK
This paper approaches the problem of adversarial examples using techniques and intuition from the manifold reconstruction literature.
Both ﬁelds have a great deal of prior work, so we focus on only the most related papers here.
2. A DVERSARIAL E XAMPLES
Some previous work has considered the relationships between adversarial examples and high dimensional geometry.
Franceschi et al.
(2018) explore the robustness of classiﬁers to random noise in terms of distance to the decision boundary, under the assumption that the decision boundary is locally ﬂat.
The work of Gilmer et al.
(2018) experimentally evaluated the setting of two concentric under-sampled 499 -spheres embedded in R 500 , and concluded that adversarial examples occur on the data manifold.
In contrast, we present a geometric framework for proving robustness guarantees for learning algorithms, that makes no assumptions on the decision boundary.
We carefully sample the data manifold in order to highlight the importance of codimension ; adversarial examples exist even when the manifold is perfectly classiﬁed.
Additionally we explore the importance of the spacing between the constituent data manifolds and sampling requirements for learning algorithms.
Wang et al.
(2018) explore the robustness of k -nearest neighbor classiﬁers to adversarial examples.
In the setting where the Bayes optimal classiﬁer is uncertain about the true label of each point, they show that k -nearest neighbors is not robust if k is a small constant.
They also show that if k ∈ Ω( √ dn log n ) , then k -nearest neighbors is robust.
Using our geometric framework we show a complementary result: in the setting where each point is certain of its label, 1 -nearest neighbors is robust to adversarial examples.
The decision and medial axes deﬁned in Section 3 are maximum margin decision boundaries.
Hard margin SVMs deﬁne deﬁne a linear separator with maximum margin, maximum distance from the training data (Cortes & Vapnik (1995)).
Kernel methods allow for maximum margin decision boundaries that are non-linear by using additional features to project the data into a higher-dimensional feature space (Shawe-Taylor & Cristianini (2004)).
The decision and medial axes generalize the notion of maximum margin to account for the arbitrary curvature of the data manifolds.
There have been attempts to incorporate maximum margins into deep learning (Sun et al.
; Liu et al.
(2016); Liang et al.
; Elsayed et al.
(2018)), often by designing loss functions that encourage large margins at either the output (Sun et al.
or at any layer (Elsayed et al.
In contrast, the decision axis is deﬁned on the input space and we use it as an analysis tool for proving robustness guarantees.
2. M ANIFOLD R ECONSTRUCTION
Manifold reconstruction is the problem of discovering the structure of a k -dimensional manifold embedded in R d , given only a set of points sampled from the manifold.
A large vein of research in manifold reconstruction develops algorithms that are provably good : if the points sampled from the underlying manifold are sufﬁciently dense, these algorithms are guaranteed to produce a geometrically accurate representation of the unknown manifold with the correct topology.
The output of these algorithms is often a simplicial complex , a set of simplices such as triangles, tetrahedra, and higher-dimensional variants, that approximate the unknown manifold.
In particular these algorithms output subsets of the Delaunay triangulation, which, along with their dual the Voronoi diagram, have properties that aid in proving geometric and topological guarantees (Edelsbrunner & Shah (1997)).
The ﬁeld ﬁrst focused on curve reconstruction in R 2 (Amenta et al.
(1998)) and subsequently in R(Dey & Kumar (1999)).
Soon after algorithms were developed for surface reconstruction in R 3 , both in the noise-free setting (Amenta & Bern (1999); Amenta et al.
and in the presence of noise (Dey & Goswami (2004)).
We borrow heavily from the analysis tools of these early works, including the medial axis and the reach.
However we emphasize that we have adapted these tools to the learning setting.
To the best of our knowledge, our work is the ﬁrst to consider the medial axis under different norms.
In higher-dimensional embedding spaces (large d ), manifold reconstruction algorithms face the curse of dimensionality .
In particular, the Delaunay triangulation, which forms the bedrock of algorithms in low-dimensions, of n vertices in R d can have up to Θ( n (cid:100) d/ 2 (cid:101) ) simplices.
To circumvent the curse of dimensionality, algorithms were proposed that compute subsets of the Delaunay triangulation restricted to the k -dimensional tangent spaces of the manifold at each sample point (Boissonnat & Ghosh (2014)).
Unfortunately, progress on higher-dimensional manifolds has been limited due to the presence of so-called “sliver” simplices, poorly shaped simplices that cause in-consistences between the local triangulations constructed in each tangent space (Cheng et al.
; Boissonnat & Ghosh (2014)).
Techniques that provably remove sliver simplices have prohibitive sampling requirements (Cheng et al.
(2000); Boissonnat & Ghosh (2014)).
Even in the special case of surfaces ( k = 2 ) embedded in high dimensions ( d > 3 ), algorithms with practical sampling requirements have only recently been proposed (Khoury & Shewchuk (2016)).
Our use of tubular neighborhoods as a tool for analysis is borrowed from Dey et al.
and Khoury & Shewchuk (2016).
In this paper we are interested in learning robust decision boundaries, not reconstructing the underlying data manifolds, and so we avoid the use of Delaunay triangulations and their difﬁculties entirely.
In Section 6 we present robustness guarantees for two learning algorithms in terms of a sampling condition on the underlying manifold.
These sampling requirements scale with the dimension of the underlying manifold k , not with the dimension of the embedding space d .
T HE G EOMETRY OF D ATA
We model data as being sampled from a set of low-dimensional manifolds embedded in highdimensional space.
A k -dimensional manifold M ⊂ R
d is a topological space with the deﬁning property that, at every point x ∈ M , there exists a neighborhood U ⊂ M containing x such that there exists a homeomorphism h : U → R k
(continuous bijection with continuous inverse).
Informally M locally “looks like” R k , even though its global topology may be very different from R k .
A sphere is an example of a 2 -manifold; in a small enough region around each point, a sphere looks like R 2 .
The special case of 1 -manifolds are called curves and 2 -manifolds surfaces , and for clarity we will use the simplest word hereafter.
The codimension of M is d − k , the difference between the dimension of the manifold and the dimension of the embedding space.
At x ∈ M , the tangent space T x M at x is a k -dimensional linear subspace while the normal space N x M is a ( d − k )
-dimensional linear subspace.
In this language, the “Manifold Hypothesis” states that in practice, data is often sampled from manifolds with high codimension.
In this paper we are primarily interested in the classiﬁcation problem.
Thus we model data as being sampled from C distinct manifolds {M 1 , . . .
, M C } , one for each class.
When we wish to refer to the entire space from which a dataset is sampled, we refer to
M = ∪ i ∈
[ C ] M i .
A ﬁnite sample
X ∈ R n × d of size n with labels y ∈
[ C ] n is sampled from M if each sample X
i (the i th row of X ) lies on or near M y i , the data manifold corresponding to class y i .
What we mean by “near” will be deﬁned rigorously below.
Consider a (cid:107) · (cid:107) p -ball B centered at some point c ∈ R d and imagine growing B by increasing the radius.
For nearly all starting points c , the ball B eventually intersects one, and only one , of the
M i ’s.
Thus the nearest point to c on M , in the norm (cid:107) · (cid:107) p , lies on M i .
(Note that the nearest point on
M i need not be unique.)
Deﬁne the decision axis Λ p of M as the set of points c such that the boundary of B intersects two or more of the M i , but the interior of B does not intersect any of the manifolds; that is the decision axis Λ p is the set of points that have two or more closest points, in the norm (cid:107) · (cid:107) p , on distinct manifolds .
See Figure 1.
The decision axis is inspired by the medial axis, which was ﬁrst proposed by Blum (1967) in the context of image analysis and subsequently modiﬁed for the purposes of curve and surface reconstruction by Amenta et al.
We have modiﬁed the deﬁnition to account for multiple constituent manifolds each of a different class and have renamed our variant in order to avoid confusion in the future.
The decision axis Λ p can intuitively be thought of as a decision boundary that is optimal in the following sense.
First, Λ p separates the constituent manifolds when they do not intersect (Lemma 7).
Second, each point of Λ p is as far away from the constituent manifolds as possible in the norm (cid:107) · (cid:107) p .
As shown in the leftmost example in Figure 1, in the case of two linearly separable circles of equal radius, the decision axis Λ 2 is exactly the line that separates the data with maximum margin.
For arbitrary manifolds, Λ p generalizes the notion of maximum margin to account for the arbitrary curvature of the constituent manifolds.
Let T ⊂ R d be any set.
The reach rch p ( T ; M ) of M is deﬁned as inf x ∈M ,y ∈ T (cid:107) x
− y (cid:107) p .
When
M is compact, the reach is achieved by the point on M that is closest to T under the (cid:107) · (cid:107) p norm.
We will drop
M from the notation when it is understood from context.
Finally, an (cid:15) - tubular neighborhood of M is deﬁned as M (cid:15),p = { x ∈ R d : inf y ∈M (cid:107) x − y (cid:107) p ≤ (cid:15) } .
That is,
M (cid:15),p is the set of all points whose distance to M under the metric induced by (cid:107) · (cid:107) p is less than (cid:15) .
Note that while
M is k -dimensional, M (cid:15),p is always d -dimensional.
Tubular neighborhoods are how we rigorously deﬁne adversarial examples.
Consider a classiﬁer f : R d → [ C ] for M .
An (cid:15) - adversarial example is a point x ∈ M (cid:15),pi such that f ( x ) (cid:54) =
i .
A classiﬁer f is robust to all (cid:15) -adversarial examples when f correctly classiﬁes not only M , but all of M (cid:15),p .
Thus the problem of being robust to adversarial examples is rightly seen as one of generalization .
In this paper we will be primarily concerned with exploring the conditions under which we can provably learn a decision boundary that correctly classiﬁes M (cid:15),p .
When (cid:15) < rch p Λ p , the decision axis Λ p is one decision boundary that correctly classiﬁes M (cid:15),p (Corollary 9).
Throughout the remainder of the paper we will drop the p in M (cid:15),p from the notation, instead writing M (cid:15) ; the norm will always be clear from context.
The geometric quantities deﬁned above can be deﬁned more generally for any distance metric d ( · , · ) .
In this paper we will focus exclusively on the metrics induced by the norms (cid:107) · (cid:107) p for p > 0 .
The decision axis under (cid:107) · (cid:107) 2 is in general not identical to the decision axis under (cid:107) · (cid:107) ∞ .
In Section 4 we will prove that since Λ 2 is not identical to Λ ∞ there exists a tradeoff in the robustness of any decision boundary between the two norms.
1.0 ϵ 0.
1.0 Accuracy
Robustness Tradeoff with FGSM Attack d = 2 d =d =0.
1.0 ϵ 0.
1.0 Accuracy
Robustness
[Figure]
d = 2 d =d =
A P ROVABLE T RADEOFF IN R OBUSTNESS B ETWEEN N ORMS
Schott et al.
(2018) explore the vulnerability of robust classiﬁers to attacks under different norms.
In particular, they take the robust pretrained classiﬁer of Madry et al.
(2018) which was trained to be robust to (cid:107) · (cid:107) ∞ -perturbations and subject it to (cid:107) · (cid:107) 0 and (cid:107) · (cid:107) 2 attacks.
They show that accuracy drops to 0% under (cid:107)·(cid:107)
attacks and to 35% under (cid:107)·(cid:107) 2 .
Here we explain why poor robustness under the norm (cid:107) · (cid:107) 2 should be expected.
We say a decision boundary D f for a classiﬁer f is (cid:15) -robust in the (cid:107) · (cid:107) p norm if (cid:15) < rch p D f .
In words, starting from any point x ∈ M , a perturbation η x must have p -norm greater than rch p D f to cross the decision boundary.
The most robust decision boundary to (cid:107) · (cid:107) p -perturbations is Λ p .
In Theorem 1 we construct a learning setting where Λ 2 is distinct from Λ ∞ .
Thus, in general, no single decision boundary can be optimally robust in all norms .
Theorem 1.
Let S 1 , S 2 ⊂ R
d
+1 be two concentric d -spheres with radii r 1 < r 2 respectively.
Let
S = S 1 ∪ S 2 and let Λ 2 , Λ ∞ be the (cid:107)·(cid:107) 2 and (cid:107)·(cid:107) ∞ decision axes of S .
Then Λ 2 (cid:54) = Λ ∞ .
Furthermore rch 2 Λ ∞ ∈ O (rch 2 Λ 2 / √ d ) .
From Theorem 1 we conclude that the minimum distance from S 1 to Λ ∞ under the (cid:107) · (cid:107) 2 norm is upper bounded as rch 2 Λ ∞ ∈ O
(rch 2 Λ 2 / √ d ) .
If a classiﬁer f is trained to learn Λ ∞ , an adversary, starting on S 1 , can construct an (cid:107) · (cid:107) 2 adversarial example for a perturbation as small as O (1 / √ d ) .
Thus we should expect f to be less robust to (cid:107) · (cid:107) 2 -perturbations.
Figure 2 veriﬁes this result experimentally.
The proof of Theorem 1 is provided in Appendix A
We expect that Λ 2 (cid:54) =
Λ ∞ is the common case in practice.
For example, Theorem 1 extends immediately to concentric cylinders and intertwined tori by considering 2 -dimensional planar crosssections.
In general, we expect that Λ 2 (cid:54) =
Λ ∞ in situations where a 2 -dimensional cross-section with M has nontrivial curvature.
Theorem 1 is important because, even in recent literature, researchers have attributed this phenomena to overﬁtting.
Schott et al.
(2018) state that “the widely recognized and by far most successful defense by Madry et al.
(1) overﬁts on the L ∞ metric (it’s highly susceptible to L 2 and L 0 perturbations)” (emphasis ours).
We disagree; the Madry et al.
(2018) classiﬁer performed exactly as intended.
It learned a decision boundary that is robust under (cid:107) · (cid:107) ∞ , which we have shown is quite different from the most robust decision boundary under
[Equation]
Interestingly, the proposed models of Schott et al.
(2018) also suffer from this tradeoff.
Their model
ABS has accuracy 80% to (cid:107) · (cid:107) 2 attacks but drops to 8% for (cid:107) · (cid:107) ∞ .
Similarly their model ABS Binary has accuracy 77% to (cid:107) · (cid:107) ∞
attacks
but drops to 39% for (cid:107) · (cid:107) 2 attacks.
We reiterate, in general, no single decision boundary can be optimally robust in all norms.
[Figure]
X (cid:15) IS A P OOR M ODEL OF M (cid:15)
Madry et al.
(2018) suggest training a robust classiﬁer with the help of an adversary which, at each iteration, produces (cid:15) -perturbations around the training set that are incorrectly classiﬁed.
In our notation, this corresponds to learning a decision boundary that correctly classiﬁes X (cid:15) =
{ x ∈ R d : (cid:107) x − X i (cid:107) 2 ≤ (cid:15) for some training point
X i } .
We believe this approach is insufﬁciently robust in practice, as X (cid:15) is often a poor model for
M (cid:15) .
In this section, we show that the volume vol X (cid:15) is often a vanishingly small percentage of vol M (cid:15) .
These results shed light on why the ball-based learning algorithm L deﬁned in Section 6 is so much less sample-efﬁcient than nearest neighbor classiﬁers.
In Section 7.1 we experimentally verify these observations by showing that in high-dimensional space it is easy to ﬁnd adversarial examples even after training against a strong adversary.
For the remainder of this section we will consider the (cid:107) · (cid:107) 2 norm.
Theorem 2.
Let M ⊂ R d be a k -dimensional manifold embedded in R d such that vol k M < ∞ .
Let
X ⊂ M be a ﬁnite set of points sampled from M .
Suppose that (cid:15) ≤ rch 2 Ξ where Ξ is the medial axis of
M , deﬁned as in Dey (2007).
Then the percentage of M (cid:15) not covered by X (cid:15) is lower bounded by vol
[Equation]
For any ﬁxed ﬁnite training set size | X | , ﬁxed k and δ , Equation 1 approaches 0 as d → ∞ .
In high codimension, even moderate under-sampling of M leads to a signiﬁcant loss of coverage of M (cid:15) because the volume of the union of balls centered at the samples shrinks faster than the volume of M (cid:15) .
Theorem 2 states that in high codimensions the fraction of M (cid:15) covered by X (cid:15) goes to 0 .
Almost nothing is covered by X (cid:15) for training set sizes that are realistic in practice.
Thus X (cid:15) is a poor model of M (cid:15) , and high classiﬁcaiton accuracy on X (cid:15) does not imply high accuracy in M (cid:15) .
The proof of Theorem 2 is given in Appendix A.
Note that an alternative way of deﬁning the ratio vol X (cid:15) / vol M (cid:15) is as vol ( X (cid:15)
∩ M (cid:15) ) /
vol M (cid:15) .
This is equivalent in our setting since X ⊂ M and so X (cid:15)
⊂ M (cid:15) .
For the remainder of the section we provide intuition for Theorem 2 by considering the special case of k -dimensional planes.
Deﬁne Π = { x ∈ R d : (cid:96) ≤ x 1 , . . .
, x k ≤ µ and x k +1 = . . .
= x d = 0 } ; that is Π is a subset of the x [Equation]
x k -plane bounded between the coordinates [ (cid:96), µ ] .
A δ -cover of a manifold M in the norm (cid:107) · (cid:107) 2 is a ﬁnite set of points X such that for every x ∈ M there exists X i such that (cid:107) x
− X
i (cid:107) 2 ≤ δ .
It is easy to construct an explicit δ -cover X of Π : place sample points at the vertices of a regular grid, shown in Figure 3 by the black vertices.
The centers of the cubes of this regular grid, shown in blue in Figure 3, are the furthest points from the samples.
The distance from the vertices of the grid to the centers is √ k ∆ / 2 where ∆ is the spacing between points along an axis of the grid.
To construct a δ -cover we need √ k ∆ / 2 = δ which gives a spacing of ∆ = 2 δ/ √ k .
The size of this sample is | X | = (cid:16) √ k ( µ − (cid:96) )
2 δ (cid:17) k .
Note that | X | scales exponentially in k , the dimension of Π , not in d , the dimension of the embedding space.
Recall that Π δ is the δ -tubular neighborhood of Π .
The δ -balls around X , which comprise X δ , cover Π and so any robust approach that guarantees correct classiﬁcation within X δ will achieve perfect accuracy on Π .
However, we will show that X δ covers only a vanishingly small fraction of Π δ .
Let
B δ denote the d -ball of radius δ centered at the origin.
An upper bound on the volume of X δ is vol
[Equation]
Next we bound the volume vol Π δ from below.
Intuitively, a lower bound on the volume can be derived by placing a ( d − k ) -dimensional
ball in the normal space at each point of Π and integrating the volumes.
Figure 3 (Right) illustrates the lower bound argument in the case of k = 1 , d = 2 .
[Equation]
Combining Equations 2 and 3 gives an upper bound on the percentage of Π δ that is covered by X (cid:15) .
[Equation]
Notice that the factors involving δ and ( µ − (cid:96) ) cancel.
Figure 4 (Left) shows that this expression approaches 0 as the codimension ( d − k ) of Π increases.
Suppose we set δ = 1 and construct a 1 -cover of Π .
The number of points necessary to cover Π with balls of radius 1 depends only on k , not the embedding dimension d .
However the number of points necessary to cover the tubular neighborhood Π 1 with balls of radius 1 increases depends on both k and d .
In Theorem 3 we derive a lower bound on the number of samples necessary to cover Π 1 .
Theorem 3.
Let Π be a bounded k -ﬂat as described above, bounded along each axis by (cid:96) < µ .
Let n denote the number of samples necessary to cover the 1 -tubular neighborhood Π 1 of Π with (cid:107) · (cid:107) 2 -balls of radius 1 .
That is let n be the minimum value for which there exists a ﬁnite sample X of size n such that Π⊂
[Equation]
X B ( x, 1) =
X 1 .
Then
[Equation]
Written asymptotically, n ≥ Ω (cid:0) d k/ 2 (cid:1) .
Theorem 3 states that, in general, it takes many fewer samples to accurately model M than to model
M (cid:15) .
Figure 4 (Right) compares the number of points necessary to construct a 1 -cover of Π with the lower bound on the number necessary to cover Π 1 from Theorem 3.
The number of points necessary to cover Π 1 increases as Ω (cid:0) d k/ 2 (cid:1) , scaling polynomially in d and exponentially in k .
In contrast, the number necessary to construct a 1 -cover of Π remains constant as d increases, depending only on k .
The proof of Theorem 3 is given in Appendix A.
Our lower bound of Ω (cid:0) d k/ 2 (cid:1) samples is similar to the work of Schmidt et al.
(2018) who prove that, in the simple Gaussian setting, robustness requires as much as Ω( √ d ) more samples.
Their arguments are statistical while ours are geometric.
Approaches that produce robust classiﬁers by generating adversarial examples in the (cid:15) -balls centered on the training set do not accurately model M (cid:15) , and it will take many more samples to do so.
If the method behaves arbitrarily outside of the (cid:15) -balls that deﬁne X (cid:15) , adversarial examples will still exist and it will likely be easy to ﬁnd them.
The reason deep learning has performed so well on a variety of tasks, in spite of the brittleness made apparent by adversarial examples, is because it is much easier to perform well on M than it is to perform well on M (cid:15) .
[Figure]
P ROVABLY R OBUST C LASSIFIERS
Adversarial training, the process of training on adversarial examples generated in a (cid:107)·(cid:107) p -ball around the training data, is a very natural approach to constructing robust models (Goodfellow et al.
Madry et al.
In our notation this corresponds to training on samples drawn from X (cid:15) for some (cid:15) .
While natural, we show that there are simple settings where this approach is much less sample-efﬁcient than other classiﬁcation algorithms, if the only guarantee is correctness in X (cid:15) .
Deﬁne a learning algorithm L with the property that, given a training set X ⊂ M sampled from a manifold
M , L outputs a model f L such that for every x ∈ X with label y , and every ˆ x ∈ B ( x, rch p Λ p ) , f L (ˆ x ) =
f L ( x ) =
y .
Here B ( x, r ) denotes the ball centered at x of radius r in the relevant norm.
That is, L learns a model that outputs the same label for any (cid:107) · (cid:107) p -perturbation of x up to rch p Λ p as it outputs for x .
L is our theoretical model of adversarial training (Goodfellow et al.
(2014); Madry et al.
Theorem 4 states that L is sample inefﬁcient in high codimensions.
Theorem 4.
There exists a classiﬁcation algorithm
A that, for a particular choice of M , correctly classiﬁes M (cid:15) using exponentially fewer samples than are required for L to correctly classify M (cid:15) .
Theorem 4 follows from Theorems 5 and 6.
In Theorems 5 and 6 we will prove that a nearest neighbor classiﬁer f nn is one such classiﬁcation algorithm.
Nearest neighbor classiﬁers are naturally robust in high codimensions because the Voronoi cells of X are elongated in the directions normal to M when X is dense (Dey (2007)).
Recall that a δ -cover of a manifold M in the norm (cid:107)·(cid:107) p is a ﬁnite set of points X such that for every x ∈ M there exists X i such that (cid:107) x
− X
i (cid:107) p ≤ δ .
Theorem 5 gives a sufﬁcient sampling condition for f
L to correctly classify M (cid:15) for all manifolds M .
Theorem 5 also provides a sufﬁcient sampling condition for a nearest neighbor classiﬁer f nn to correctly classify
M (cid:15) , which is substantially less dense than that of f L .
Thus different classiﬁcation algorithms have different sampling requirements in high codimensions.
Theorem 5.
Let M ⊂ R d be a k -dimensional manifold and let (cid:15) < rch p Λ p for any p .
Let f nn be a nearest neighbor classiﬁer and let f L be the output of a learning algorithm L as described above.
Let X nn , X L ⊂ M denote the training sets for f nn and L respectively.
We have the following sampling guarantees:
If X nn is a δ -cover for δ ≤ 2(rch p Λ p − (cid:15) )
then f nn correctly classiﬁes M (cid:15) .
If X L is a δ -cover for δ ≤ rch p Λ p − (cid:15) then f L correctly classiﬁes M (cid:15) .
The bounds on δ in Theorem 5 are sufﬁcient, but they are not always necessary.
There exists manifolds where the bounds in Theorem 5 are pessimistic, and less dense samples corresponding to larger values of δ would sufﬁce.
In Theorem 6 we show a setting where bounds on δ similar to those in
Theorem 5 are necessary .
In this setting, the difference of a factor of 2 in δ between the sampling requirements of f nn and f L leads to an exponential gap between the sizes of X nn and X L necessary to achieve the same amount of robustness.
[Figure]
Consider two subsets of k -ﬂats Π 1 , Π 2 , as deﬁned in Section 5, where Π 1 lies in the subspace x d =and Π 2 lies in the subspace x d = 1 ; thus rch 2 Λ 2 = 1 .
In the (cid:107) · (cid:107) 2 norm we can show that the gap in Theorem 5 is necessary for Π
=
Π
∪ Π 2 .
Furthermore the bounds we derive for δ -covers for Π for both f nn and f L are tight.
Combined with well-known properties of covers, we get that the ratio | X L | / | X nn | is exponential in k .
Theorem 6.
Let Π
= Π 1 ∪ Π 2 as described above.
Let X nn , X L ⊂ Π be minimum training sets necessary to guarantee that f nn and f L correctly classify M (cid:15) .
Then we have that
[Equation]
We have shown that both L and nearest neighbor classiﬁers learn robust decision boundaries when provided sufﬁciently dense samples of M .
However there are settings where nearest neighbors is exponentially more sample-efﬁcient than L in achieving the same amount of robustness.
We experimentally verify these theoretical results in Section 7.1.
Proofs for all of the results in this section are provided in Appendix A.
E XPERIMENTS 7. H IGH C ODIMENSION R EDUCES R OBUSTNESS
Section 5 suggests that as the codimension increases it should become easier to ﬁnd adversarial examples.
To verify this, we introduce two synthetic datasets, C IRCLES and P LANES , which allow us to carefully vary the codimension while maintaining dense samples.
The C IRCLES dataset consists of two concentric circles in the x 1 - x 2 -plane, with rch 2 Λ 2 = 1 .
We densely sample 1000 random points on each circle for both the training and the test sets.
The P LANES dataset consists of two 2 -dimensional planes, the ﬁrst in the x d = 0 and the second in x d = 2 , so that rch 2 Λ 2 = 1 .
We sample the training set at the vertices of the grid described in Section 5, and the test set at the centers of the grid cubes, the blue points in Figure 3.
Further details are provided in Appendix E and visualizations in Appendix H.
We consider two attacks, the fast gradient sign method (FGSM)
(Goodfellow et al.
and the basic iterative method (BIM) (Kurakin et al. (2016)) under (cid:107) · (cid:107) 2 .
We use the implementations provided in the cleverhans library (Papernot et al.
Further implementation details are provided in Appendix
E. Our experimental results are averaged over 20 retrainings of our model architecture, using Adam (Kingma & Ba (2015)).
Further implementation details are provided in Appendix E.
Figure 5(Left, Center) shows FGSM and BIM attacks on the C IRCLES dataset as we vary the codimension.
For both attacks we see a steady decrease in robustness as we increase the codimension, on average.
Madry et al.
(2018) propose training against a PGD adversary to improve robustness.
Sectionsuggests that this should be insufﬁcient to guarantee robustness, as X (cid:15) is often a poor model for
0.5 ϵ
[Figure] [Figure]
1.0 Accuracy
[Figure]
- NN 0.
0.5 ϵ 0.
1.0 Accuracy
[Figure]
[Figure] [Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
M (cid:15) .
We train against a PGD adversary with (cid:15)
= 1 under (cid:107) · (cid:107) 2 -perturbations on the P LANES dataset.
Figure 5 (Right) shows that it is still easy to ﬁnd adversarial examples for (cid:15) < 1 and that as the codimension increases we can ﬁnd adversarial examples for decreasing values of (cid:15) .
In contrast, nearest neighbor achieves perfect robustness for all epsilon on this data (see Appendix B. for details).
7. MNIST
To explore performance on a more realistic dataset, we compared nearest neighbors with robust and natural models on MNIST.
We considered two attacks: BIM under l ∞ norm against the natural and robust models as well as a custom attack against nearest neighbors.
Each of these attacks are generated from the MNIST test set.
Architecture details can be found in Appendix E. Figure 6 (Left) shows that nearest neighbors is substantially more robust to BIM attacks than the naturally trained model.
Figure 6 (Center) shows that nearest neighbors is comparable to the robust model up to (cid:15)
= 0 .
3 , which is the value for which the robust model was trained.
After (cid:15) = 0 . 3 , nearest neighbors is substantially more robust to BIM attacks than the robust model.
At (cid:15) = 0 .
5 , nearest neighbors maintains accuracy of 78% to adversarial perturbations that cause the accuracy of the robust model to drop to 0% .
In Appendix B.2 we provide a similar result for FGSM attacks.
Figure 6 (Right) shows the performance of nearest neighbors and the robust model on adversarial examples generated for nearest neighbors.
The nearest neighbor attacks are generated as follows: iteratively ﬁnd the k nearest neighbors and compute an attack direction by walking away from the neighbors in the true class and toward the neighbors in other classes.
We ﬁnd that nearest neighbors is able to be tricked by this approach, but the robust model is not.
This indicates that the errors of these models are distinct and suggests that ensemble methods may effectively get the best of both worlds.
Additionally, a closer investigation shows strong qualitative differences between the BIM adversarial examples and the examples generated for nearest neighbors.
Appendix J argues that the adversarial examples that fool nearest neighbor line up better with human intuition.
C ONCLUSION
We have presented a geometric framework for proving robustness guarantees for learning algorithms.
Our framework is general and can be used to describe the robustness of any classiﬁer.
We have shown that no single model can be simultaneously robust to attacks under all norms and that nearest neighbor classiﬁers are theoretically more sample efﬁcient than adversarial training.
Most importantly, we have highlighted the role of codimension in contributing to adversarial examples and veriﬁed our theoretical contributions with experimental results.
We believe that a geometric understanding of the decision boundaries learned by deep networks will lead to both new geometrically inspired attacks and defenses.
In Appendix C we provide a novel gradient-free geometric attack in support of this claim.
Finally we believe future work into the geometric properties of decision boundaries learned by various optimization procedures will provide new techniques for black-box attacks.
A O MITTED P ROOFS A. A UXILIARY L EMMAS
Lemma 7.
Let M 1 , M 2 ⊂ R d be k -dimensional manifolds such that M∩M 2 = ∅ .
Let Λ p be their decision axis for any p and let γ : [0 , 1] → R d be any path
such that γ (0) ∈
M 1 and γ (1) ∈ M 2 .
Then γ ∩ M (cid:54) = ∅ , that is γ must cross the decision axis.
Proof.
Deﬁne f 1 , f 2 : [0 , 1] → R as f 1 ( t ) =
d ( γ ( t ) , M 1 ) and f 2 ( t ) =
d ( γ ( t ) , M 2 ) .
Consider the function g ( t ) =
f 1 ( t ) − f 2 ( t ) .
Since M 1 ∩ M 2 = ∅ and γ starts on M 1 and terminates on M 2 the function g (0) < 0 and g (1) > 0 .
Then, since g is continuous, the Intermediate Value Theorem implies that there exists t 1 ∈ [0 , 1] such that g ( t 1 ) = 0 .
Thus d ( γ ( t 1 ) , M 1 ) =
d ( γ ( t 1 ) , M 2 ) , which implies that γ ( t 1 ) is on the decision axis Λ .
Theorem 8.
Let f be any classiﬁer on M = M 1 ∪ M 2 .
The maximum accuracy achievable, assuming a uniform distribution, on
M
[Equation]
is
[Equation]
Proof.
It is clearly optimal to classify points in vol( M (cid:15) 1 \ M (cid:15) 2 ) as class 1 and to classify points in vol( M (cid:15) 2 \ M (cid:15) 1 ) as class 2 .
Such a classiﬁer can only be wrong when points lie in this intersection.
For points in this intersection, the probability of a misclassiﬁcation is 12 for any classiﬁcation that f makes.
Thus, the probability of misclassiﬁcation is
[Equation]
Corollary 9.
For (cid:15) < rch p (Λ p ; M ) there exists a decision boundary that correctly classiﬁes M (cid:15) .
Proof.
For (cid:15) < rch p Λ p , M (cid:15) ∩
Λ p = ∅
and so Λ p is one such decision boundary.
A. P ROOF OF T HEOREM
Proof.
The decision axis under (cid:107) · (cid:107) 2 , Λ 2 , is just the d -sphere with radius ( r 1 + r 2 ) / 2 .
However, Λ ∞ is not identical to Λ 2 in this setting; in fact most Λ ∞ of approaches S 1 as d increases.
The geometry of a (cid:107) · (cid:107) ∞ -ball B ∆ centered at m ∈ R d with radius ∆ is that of a hypercube centered at m with side length 2∆ .
To ﬁnd a point on Λ ∞ we place B ∆ tangent to the north pole q of S 1 so that the corners of B ∆ touch S 2 .
The north pole has coordinate representation q = (0 , . .
. , 0 , r 1 ) , the center m = (0 , . .
. , 0 , r 1 + ∆) , and a corner of B ∆ can be expressed as p = (∆ , . . .
, ∆ , r 1 + 2∆) .
Additionally we have the constraint that (cid:107) p (cid:107) 2 =
r 2 since p ∈ S 2 .
Then we can solve for ∆ as
[Equation]
where the last step follows from the quadratic formula and the fact that ∆ > 0 .
For ﬁxed r 1 , r 2 , the value ∆ scales as O (1 / √ d ) .
It follows that rch 2 Λ ∞ ∈ O (rch 2 Λ 2 / √ d ) .
A. P ROOF OF T HEOREM
Proof.
Assuming the balls centered on the samples in X are disjoint we get the upper bound
[Equation]
[Equation]
This is identical to the reasoning in Equation 2.
The medial axis Ξ of M is deﬁned as the closure of the set of all points in R d that have two or more closest points on M in the norm (cid:107) · (cid:107) 2 .
The medial axis Ξ is similar to the decision axis Λ 2 , except that the nearest points do not need to be on distinct constituent manifolds.
For (cid:15) ≤ rch 2 Ξ , we have the lower bound
[Equation]
Combining Equations 8 and 9 gives the result.
A. P ROOF OF T HEOREM
Proof.
We ﬁrst construct an upper bound by generously assuming that the balls centered at the samples are disjoint.
That is
[Equation]
To guarantee that Π 1 ⊂ ∪ x ∈ X B [Equation] x, 1) = X 1 we set the left hand side of Equation 10 equal to 1 and solve for n .
[Equation]
The last inequality follows from Equation 3.
Setting δ = 1 gives the result.
Taking the Taylor series gives the asymptotic result.
A. P ROOF OF T HEOREM
Proof.
We begin by proving (1).
Let q ∈ M (cid:15) be any point in M (cid:15) .
Suppose without loss of generality that q ∈ M (cid:15)i for some class i .
The distance d ( q, M j ) from q to any other data manifold
M j , and thus any sample on M j , is lower bounded by d ( q, M j ) ≥ 2 rch p Λ p − (cid:15) .
It is then both necessary and sufﬁcient that there exists a x ∈ M i such that d ( q, x ) < 2 rch p Λ p − (cid:15) for f nn ( q ) = i .
(Necessary since a properly placed sample on
M j can achieve the lower bound on d ( q, M j ) .)
The distance from q to the nearest sample x on
M i is d ( q, x ) ≤ (cid:15) + δ for some δ > 0 .
The question is how large can we allow δ to be and still guarantee that f nn correctly classiﬁes
[Equation]
We need
[Equation]
which implies that δ ≤ 2(rch p Λ p − (cid:15) ) .
It follows that a δ -cover with δ = 2(rch p Λ p − (cid:15) ) is sufﬁcient, and in some cases necessary, to guarantee that f nn correctly classiﬁes M (cid:15) .
Next we prove (2).
As before let q ∈ M (cid:15)i .
It is both necessary and sufﬁcient for q ∈ B rch p Λ p ( x ) for some sample x ∈ M i to guarantee that f L ( q )
=
i , by deﬁnition of L .
The distance to the nearest sample x on
M i is d ( q, x ) ≤ (cid:15) + δ for some δ > 0 .
Thus it sufﬁces that δ ≤ rch p Λ p − (cid:15) .
A. P ROOF OF T HEOREM
Proof.
Let q ∈ Π (cid:15) 1 .
Since Π 1 is ﬂat, the distance to from q to the nearest sample x ∈ Π 1 is bounded as (cid:107) q − x (cid:107) 2 ≤ √ (cid:15) 2 + δ 2 .
For f nn ( q ) = 1 we need that (cid:107) q − x (cid:107) 2 ≤ 2 − (cid:15) , and so it sufﬁces that δ ≤ 2 √ 1 − (cid:15) .
In this setting, this is also necessary; should δ be any larger a property placed sample on Π 2 can claim q in its Voronoi cell.
Similarly for f L ( q ) = 1 we need that (cid:107) q − x (cid:107) 2 ≤ 1 , and so it sufﬁces that δ ≤ √ 1 − (cid:15) 2 .
In this setting, this is also necessary; should δ be any larger, q lies outside of every (cid:107) · (cid:107) 2 -ball B 1 ( x ) and so
L is free to learn a decision boundary that misclassiﬁes q .
Let N ( δ, M ) denote the size of the minimum δ -cover of M .
Since Π is ﬂat (has no curvature) and since the intersection of Π with a d -ball centered at a point on Π is a k -ball, a standard volume argument can be applied in the afﬁne subspace aﬀ Π to conclude that N ( δ, Π)
∈ Θ (cid:0) vol k Π /δ k (cid:1) .
So we have
[Equation]
Since Π is constant in both settings, the factor vol k Π as well as the constant factors hidden by Θ( · ) cancel.
(Note that we are using the fact that Π 1 , Π 2 have ﬁnite k -dimensional volume.)
The inequality follows from the fact that the expression (1 + (cid:15) ) − k/ 2 is monotonically decreasing on the interval [0 , 1] and takes value 2 − k/ 2 at (cid:15) = 1 .
B A DDITIONAL E XPERIMENTS
We present additional experiments to support our theoretical predictions.
We reproduce the results of Section 7 using different optimization algorithms (Section B.1) and attack methods (Section B.2).
These additional experiments are consistent with our conclusions in Section 7.
Additionally we provide evidence that adversarial perturbations lie mostly in the directions of the normal space (Section B.3).
We show that a nearest neighbor classiﬁer is robust in high codimensions (Section B.4).
Finally we show that increasing the sampling density substantially does not notably improve the robustness of adversarial training (Section B.5).
B. R EPRODUCING R ESULTS USING SGD
In Section 7.1 we showed that increasing the codimension reduces the robustness of the decision boundaries learned by Adam on C IRCLES .
In Figure 7 we reproduce this result using SGD.
Again we see that as we increase the codimension the robustness decreases.
SGD presents with much less variances than Adam, which we attribute to implicit regularization that has been observed for SGD (Soudry et al.
[Figure]
Next we consider the adversarial training procedure of Madry et al.
(2018) using SGD instead of
Adam.
We note that the authors of Madry et al.
(2018) use Adam in their own experiments.
Figureshows that the result is consist with the result in Section 7.1.
Again SGD presents with lower variance.
[Figure]
B. R EPRODUCING R ESULTS USING FGSM
In Section 7.1 we evaluated the robustness of nearest neighbors against BIM attacks under the (cid:107)·(cid:107) ∞ on MNIST.
In Figure 9 we evaluate the robustness of nearest neighbors against FGSM attacks under the (cid:107) · (cid:107) ∞ on MNIST.
We use the naturally pretrained (natural) and adversarially pretrained (robust) convolutional models provided by Madry et al.
Figure 9 (Left) shows that nearest neighbors is substantially more robust to FGSM attacks than the naturally trained model.
Figure 9 (Right) shows that nearest neighbors is comparable to the robust model up to (cid:15)
= 0 .
3 , which is the value for which the robust model was trained.
After (cid:15) = 0 . 3 , nearest neighbors is substantially more robust to
FGSM attacks than the robust model.
At (cid:15) = 0 .
5 , nearest neighbors maintains accuracy of 78% to adversarial perturbations that cause the accuracy of the robust model to drop to 39% .
B. A DVERSARIAL P ERTURBATIONS ARE IN THE D IRECTIONS N ORMAL TO THE D ATA M ANIFOLD
Let η x be an adversarial perturbation generated by FGSM with (cid:15)
= 1 at x ∈ M .
Note that the adversarial example is constructed as ˆ x = x + η x .
In Figure 10 we plot a histogram of the angles ∠ ( η x , N x M ) between η x and the normal space N x M for the C IRCLES dataset in codimensions 1 , 10 , 100 , and 500 .
In codimension 1 , 88% of adversarial perturbations make an angle of less than 10 ◦ with the normal space.
Similarly in codimension 10 , 97% , in codimension 100 , 96% , and in
[Figure]
[Figure] [Figure]
[Figure] [Figure]
RobustnessofNaturalModelvsNN
Natural 1 - NN 0.
0.5 ϵ 0.
[Figure]
Accuracy
RobustnessofRobustModelvsNN
[Figure]
- NN codimension 500 , 93% .
As Figure 10 shows, nearly all adversarial perturbations make an angle less than 20 ◦ with the normal space.
Our results are averaged over 20 retrainings of the model using
SGD.
Throughout this paper we’ve argued that high codimension is a key source of the pervasiveness of adversarial examples.
Figure 10 shows that adversarial perturbations are well aligned with the normal space.
When the codimension is high, there are many directions normal to the manifold and thus many directions in which to construct adversarial perturbations
[Figure]
Frequency
[Figure]
B. N EAREST N EIGHBORS IS R OBUST IN H IGH C ODIMENSION
In Section 7.1 we showed that the robustness of learned decision boundaries decreased as the codimension increased.
In Figure 11 we repeat the experiment in Figure
[Figure]
in which we measured the robustnesss of our neural network models to FGSM attacks as the codimension increased.
We repeat this experiment using nearest neighbors to classify the adversarial examples generated by FGSM.
Figure 11 shows that nearest neighbors is robust even when the codimension is high, as long as the low-dimensional data manifold is well sampled.
This is a consquence of the fact that the Voronoi cells of the samples are elongated in the directions normal to the data manifold when the sample is dense.
[Figure]
[Figure]
B. S AMPLING M ORE D ENSELY
The P LANES dataset is sampled so that the trianing set is a 1 -cover of the underlying planes, which requires 450 sample points.
Figure 13 shows the results of increasing the sampling density to a 0 .
5cover (1682 samples) and a 0 .
25 -cover (6498 samples).
Increasing the sampling density improves the robustness of adversarial training at the same codimension and particularly in low-codimension.
However adversarial training with a substantially larger training set does not produce a classiﬁer as robust as a nearest neighbor classiﬁer on a much smaller training set.
Nearest neighbors is much more sample efﬁcient than adversarial training, as predicted by Theorem 5 and experimentally veriﬁed in Section B.4.
C A G RADIENT -F REE G EOMETRIC A TTACK
Most current attacks rely on the gradient of the loss function at a test sample to ﬁnd a direction towards the decision boundary.
Partial resistance against such attacks can be achieved by obfuscating the gradients, but Athalye et al.
(2018) showed how to circumvent such defenses.
Brendel et al.
(2018) propose a gradient-free attack for (cid:107) · (cid:107) 2 , that starts from a misclassifed point and walks toward the original point.
In this section we propose a gradient-free attack that only requires oracle access to a model, meaning we only query the model for a prediction.
Consider a point x ∈ X test and the (cid:107) · (cid:107) p -ball B
r ( x )
1.0 ϵ 0.
1.0 Accuracy 1 - cover codim = 1 codim = 10 codim =codim =0.
1.0 ϵ 0.
1.0 Accuracy 0.5 - cover codim = 1 codim = 10 codim =codim =0.
1.0 ϵ 0.
1.0 Accuracy 0.25 - cover codim = 1 codim = 10 codim =codim =0.
1.0 ϵ 0.
1.0 Accuracy
AdverarialTrainingwithBIMAttack ( Adam ) codim = 1 codim = 10 codim =codim =
Increasing the sampling density improves robustness at the same codimension.
However even training on a signiﬁcantly denser training set does not produce a classiﬁer as robust as a nearest neighbor classiﬁer on a much sparser training set, centered at x of radius r .
To construct an adversarial perturbation η x ∈ B r ( x ) , giving an adversarial example ˆ x = x + η x , we project every point in X test onto B r ( x ) and query the oracle for a prediction for each point.
If y ∈ X test projected to a point y (cid:48) that the model classiﬁed differently than x , we take η x = y (cid:48)
− x , otherwise η x = 0 .
This incredibly simple attack reduces the accuracy of the pretrained robust model of Madry et al.
(2018) for (cid:107) · (cid:107) ∞ and (cid:15) = 0 .
3 to 90 .
6% , less than two percent shy of the current SOTA for whitebox attacks, 88 .
79% (Zheng et al.
Simple datasets, such as C IRCLES and P LANES , allow us to diagnose issues in learning algorithms in settings where we understand how the algorithm should behave.
For example Athalye et al.
(2018) state that the work of Madry et al.
(2018) does not suffer from obfuscated gradients.
In Appendix D we provide evidence that Madry et al.
(2018) does suffer from the obfuscated gradients problem, failing one of Athalye et al.
(2018)’s criteria for detecting obfuscated gradients.
D T HE M ADRY D EFENSE S UFFERS FROM O BFUSCATED G RADIENTS
Athalye et al.
identiﬁed the problem of “obfuscated gradients”, a type of a gradient masking (Papernot et al.
that many proposed defenses employed to defend against adversarial examples.
They identiﬁed three different types of obfuscated gradients: shattered gradients, stochastic gradeints, and exploding/vanishing gradients.
They examined nine recently proposed defenses, concluded that seven suffered from at least one type of obfuscated gradient, and showed how to circumvent each type of obfuscated gradient and thus each defense that employed obfuscated gradients.
Regarding the work of Madry et al.
(2018), Athalye et al.
(2018) stated “We believe this approach does not cause obfuscated gradients”.
They note that “our experiments with optimization based attacks do succeed with some probability”.
In this section we provide evidence that the defense of
Madry et al.
(2018) does suffer from obfuscated gradients, speciﬁcally shattered gradients.
Shattered gradients occur when a defence causes the gradient ﬁeld to be “nonexistent or incorrect” (Athalye et al.
Speciﬁcally we provide evidence that the defense of Madry et al.
(2018) works by shattering the gradient ﬁeld of the loss function around the data manifolds.
In Figure 14 (Left) we show the normalized gradient ﬁeld of the loss function for a network trained on a 2-dimensional version of our P LANES dataset using the adversarial training procedure of Madry et al.
(2018) with a PGD adversary.
While the gradients have meaningful directions, Figure 14 (Left) shows that magnitude of the gradient ﬁeld is nearly 0 everywhere around the data manifolds, which are at y = 0 and y = 2 .
The only notable gradients are near the decision axis which is at y = 1 .
One criteria that Athalye et al.
(2018) propose for identifying obfuscated gradients is whether onestep attacks perform better than iterative attacks.
The reason this criteria is useful for identifying obfuscated gradients is because one-step attacks like FGSM ﬁrst normalize the gradient, ignoring its magnitude, then take as large of a step as allowed in the direction of the normalized gradient.
So long as the gradient on the manifold points towards the decision boundary, FGSM will be effective at ﬁnding an adversarial example.
[Figure] In Figure 15 we show the adversarial examples generated using PGD (left), FGSM (center), and BIM (right) for (cid:15) =
1 starting at the test set for the P LANES dataset.
FGSM produces adversarial examples at the decision axis y
= 1 , exactly where we would expect.
Notice that all of the adversarial perturbation is normal to the data manifold, suggesting that the gradient on the manifold points towards the decision boundary.
However the adversarial examples produced by PGD lie closer to the manifold from which the example was generated.
PGD splits the total perturbation between both the normal and the tangent spaces of the data manifold, as shown by the arrows in Figure 15.
This suggests that, when trained adversarially, the network learned a gradient ﬁeld that has small but correct gradients on the data manifold, but gradients that curve in the tangent directions immediately off the manifold .
Lastly notice that BIM, another iterative method, also produces adversarial examples that are near the decision axis.
Athalye et al.
(2018) cite success with iterative based optimization procedures as evidence against obfuscated gradients.
However BIM also ignores the magnitude of the gradient, as it simply applies FGSM iteratively.
The network has learned a gradient ﬁeld that is overﬁt to the particulars of the PGD attack.
BIM successfully navigates this gradient ﬁeld, while PGD does not.
While the network is robust to PGD attacks at test time, it is less robust to FGSM and BIM attacks.
[Figure]
E I MPLEMENTATION D ETAILS
In Section 7 we introduced two synthetic datasets, C IRCLES and P LANES .
The C IRCLES dataset consists of two concentric circles, the ﬁrst with radius r 1 = 1 and the second with radius r 2 = 3 , so that the rch = 1 .
The P LANES dataset consists of two 2 -dimensional planes, the ﬁrst in the subspace deﬁned by x d = 0 , and the second in x d = 2 , so that rch = 1 .
The ﬁrst two axis of both planes are bounded as − 10 ≤ x 1 , x 2 ≤ 10 , while x 3 = . . .
=
x
d − 1 = 0 .
Both planes are sampled as described in Section 5, so that X 1 covers the underlying planes, where X is the training set.
We consider three attacks, FGSM, BIM, and PGD, primarily under the (cid:107) · (cid:107) 2 norm.
For the iterative attacks BIM and PGD, we set the number of iterations to 30 with a step size of (cid:15) step = 0 .
05 per iteration.
Our controlled experiments on synthetic data consider a fully connected network with 1 hidden layer, 100 hidden units, and ReLU activations.
This model architecture is more than capable of representing a nearly perfect robust decision boundary for both C IRCLES and P LANES , the latter of which is linearly separable.
We set the learning rate for Adam as α = 0 . 1 , which we found to work best for our datasets.
The parameters for the exponential decay of the ﬁrst and second moment estimates were set to β 1 = 0 .
9 and β 2 = 0 .
We set the learning rate for SGD as α = 0 .
1 and decrease the learning rate by a factor of 10 every 100 epochs.
We train all of our models forepochs, following Wilson et al.
All of our experiments are implemented using PyTorch.
When comparing against a published result we use publicly available repositories, if able.
For the robust loss of Wong & Kolter (2018), we use the code provided by the authors 2 .The provided implementation 3 of the adversarial training procedure of Madry et al.
(2018) considers a PGD adversary with (cid:107)·(cid:107) ∞ -perturbations.
We reimplemented their adversarial training procedure for (cid:107)·(cid:107) 2 -perturbations following their implementation and using the PGD attack implemented in the cleverhans library (Papernot et al.
The models of Madry et al.
(2018) consist of two convolutional layers with 32 and 64 ﬁlters respectively, each followed by 2 × 2 max pooling.
After the two convolutional layers, there are two fully connected layers each with 1024 hidden units.
F V OLUME A RGUMENTS FOR d -S PHERES
Let S ⊂ R
d
+1 be a unit d -sphere embedded in R d +1 .
The volume of S (cid:15) is given by
[Equation]
where Γ denotes the gamma function.
Let X ⊂ S be a ﬁnite sample of size n of S .
The set X (cid:15) is the set of all (cid:15) perturbations of points in X under the norm (cid:107) · (cid:107) 2 .
How well does X (cid:15) approximate
S (cid:15) as a function of n, d and (cid:15) ?
To answer this question we upper bound the ratio vol X (cid:15) / vol S (cid:15) by generously assuming that the balls B ( X i , (cid:15) ) are disjoint.
The resulting upper bound is
[Equation]
In Figure 16 we show three different views of this bound.
In Figure 16 (Left) we set n = 10 12 and plot four different values of (cid:15) ; in each case the percentage of volume of S (cid:15) that is covered by X (cid:15) quickly approaches 0 .
Similarly, in Figure 16 (Center), if we ﬁx (cid:15) = 1 and plot four different values of n , in each case we have the same result.
Finally in Figure 16 (Right) we plot a lower bound on number of samples necessary to cover S (cid:15) by X (cid:15) for four different values of (cid:15) ; in each case the number of samples necessary grows exponentially with the dimension.
G V ORONOI D IAGRAMS AND D ELAUNAY T RIANGULATIONS
Let X ⊂ R d be a ﬁnite set of n points.
The Voronoi diagram of X , denoted Vor X , under the metric d ( · , · ) is a subdivision of R d into n cells where each cell is deﬁned as
[Equation]
1.0 Ratio ϵ = 0.
ϵ = 0.2 ϵ = 0.
ϵ
= 0.
50 Dimension 0.2 0.
1.0 Ratio ϵ = 0.
ϵ = 0.2 ϵ = 0.
ϵ
= 0.
1.0 Ratio n = 10 4 n = 10n = 10 8 n = 10     50 Dimension 0.2 0.
1.0 Ratio ϵ = 0.
ϵ = 0.2 ϵ = 0.
ϵ
= 0.
2 × 104 × 106 × 108 × {+
+} SampleSize ϵ = 0.
 ϵ = 0.2 ϵ = 0.
ϵ
{+= 0. 50 Dimension 0.2 0. 1.0 Ratio ϵ = 0.
+} ϵ = 0.2 ϵ = 0.
ϵ
= 0.
In words, the Voronoi cell Vor v of v ∈ X is the set of all points in R d that are closer to v than any other sample point u (cid:54)
= v in X .
The Voronoi diagram is then deﬁned as the set of all Voronoi cells,
Vor X = { Vor v : v ∈ X } .
When d ( · , · ) is induced by the norm (cid:107) · (cid:107) 2 , the Voronoi cells are convex.
See Figure 17.
The Delaunay triangulation of X , denoted Del X is a triangulation of the convex hull of X into d -simplices.
Every d -simplex τ ∈ Del X , as well as every lower-dimensional face of τ , has the deﬁning property that there exists an empty circumscribing ball B such that the vertices of τ lie on the boundary of B and the interior of B is free from any points in X .
See Figure 17.
This empty circumscribing ball property of Delaunay triangulations implies many desirable properties that are useful in mesh generation (Cheng et al.
and manifold reconstruction (Edelsbrunner & Shah (1997)).
The Delaunay triangulation of a point set always exists, but is not unique in general.
There exists a well known duality between the Voronoi diagram and the Delaunay triangulation of
X .
For every j -dimensional face σ ∈ Vor X there exist a dual ( d − j ) -dimensional simplex denoted σ ∗ ∈
Del X whose d − j + 1 vertices are the d − j + 1 vertices of X whose Voronoi cells intersect at σ .
In particular, every d -cell of Vor X is dual to the vertex of Del X that generates that cell, and every ( d − 1) -face of Vor X is dual to an edge of Del X .
A nearest neighbor classiﬁer f nn given a query point q simply returns the class of the point in X that generated the Voronoi cell in which q lies.
Thus the decision boundary of f nn is the union of ( d − 1) and lower dimensional Voronoi faces.
Furthermore, when X is a dense sample of a manifold M , the Voronoi cells are well known to be elongated in the directions normal to M Dey (2007).
This fact underlies many of our results.
H V ISUALIZATION OF D ATASETS
In Figure 18 we provide visualizations of our two synthetic datasets, C IRCLES (left) and P LANES (right).
I V ISUALIZATION OF D ECISION B OUNDARIES
In Figure 19 we provide visualizations of the decision boundaries learned by (a-d) our fully connected network architecture with cross entropy loss for various optimization procedures and various training lengths, (e) our fully connected network architecture trained using the robust loss of Wong & Kolter (2018) for (cid:107) · (cid:107) ∞ -perturbations, and (f) a nearest neighbor classiﬁer for (cid:107) · (cid:107) 2 on the train- ing set.
Speciﬁcally we train on the C IRCLES dataset, embedded in R 3 .
The training set is entirely contained in the xy -plane.
We then visualize cross sections of the decision boundary for various values of z ∈ [ − 5 , 5] .
We color points labeled as in the same class as the outer circle with the color blue and points labeled as in the same class as the inner circle as orange.
Figure 19 shows the cross sections of the decision boundaries, averaged over 20 retrainings.
The visualization shows how various optimization algorithms learn decision boundaries that extend into the normal directions where no data is provided.
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
boundaries that extend into the normal directions in which there is no data provided.
We average the decision boundary over 20 retrainings, so faded results indicated how frequently a point was labeled a speciﬁc class.
J
C OMPARING N EAREST N EIGHBOR A DVERSARIAL E XAMPLES WITH BIM
A TTACK
[Figure]