Original: M ANIFOLD A SSUMPTION AND D EFENSES A GAINST A DVERSARIAL P ERTURBATIONS
Revised: M ANIFOLD A SSUMPTION AND D EFENSES A GAINST A DVERSARIAL P ERTURBATIONS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: In the adversarial perturbation problem of neural networks, an adversary starts with a neural network model F and a point x that F classiﬁes correctly, and identiﬁes another point x (cid:48) , which is nearby x , that F classiﬁes incorrectly .
Revised: In the adversarial perturbation problem of neural networks, an adversary starts with a neural network model F and a point x that F classiﬁes correctly, and identiﬁes another point x (cid:48) , which is nearby x , that F classiﬁes incorrectly .

Original: In this paper we consider a defense method that is based on the semantics of F .
Revised: In this paper we consider a defense method that is based on the semantics of F .

Original: Our starting point is the common manifold assumption , which states that natural data points lie on separate low dimensional manifolds for different classes.
Revised: Our starting point is the common manifold assumption , which states that natural data points lie on separate low dimensional manifolds for different classes.

Original: We then make a further postulate which states that (a good model) F is conﬁdent on natural points on the manifolds, but has low conﬁdence on points outside of the manifolds, where a natural measure of “conﬁdent behavior” is (cid:107) F ( x ) (cid:107) ∞ (i.e. how conﬁdent F is about its prediction).
Revised: We then make a further postulate which states that (a good model) F is conﬁdent on natural points on the manifolds, but has low conﬁdence on points outside of the manifolds, where a natural measure of “conﬁdent behavior” is (cid:107) F ( x ) (cid:107) ∞ (i.e. how conﬁdent F is about its prediction).

Original: Under this postulate, an adversarial example becomes a point that is outside of the low dimensional manifolds which F has learned, but is still close to at least one manifold under some distance metric.
Revised: Under this postulate, an adversarial example becomes a point that is outside of the low dimensional manifolds which F has learned, but is still close to at least one manifold under some distance metric.

Original: Therefore, defending against adversarial perturbations becomes embedding an adversarial point back to the nearest manifold where natural points are drawn from.
Revised: Therefore, defending against adversarial perturbations becomes embedding an adversarial point back to the nearest manifold where natural points are drawn from.

Original: We propose algorithms to formalize this intuition and perform a preliminary evaluation.
Revised: We propose algorithms to formalize this intuition and perform a preliminary evaluation.

Original: Noting that the effectiveness of our method depends on both how well F satisﬁes the postulate and how effective we can conduct the embedding, we use a model trained recently by Madry et al., as the base model , and use gradient based optimization , such as the Carlini-Wagner attack (but now they are used for defense), as the embedding procedure .
Revised: Noting that the effectiveness of our method depends on both how well F satisﬁes the postulate and how effective we can conduct the embedding, we use a model trained recently by Madry et al., as the base model , and use gradient based optimization , such as the Carlini-Wagner attack (but now they are used for defense), as the embedding procedure .

Original: Our preliminary results are encouraging: The base model wrapped with the embedding procedure achieves almost perfect success rate in defending against attacks that the base model fails on, while retaining the good generalization behavior of the base model.
Revised: Our preliminary results are encouraging: The base model wrapped with the embedding procedure achieves almost perfect success rate in defending against attacks that the base model fails on, while retaining the good generalization behavior of the base model.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: In the adversarial perturbation problem of neural networks, an adversary starts with a neural network model F (we assume that F ends with a softmax layer, which is a common assumption in the literature), and a point x that F classiﬁes correctly, and identiﬁes another point x (cid:48) , which is nearby x , that F classiﬁes incorrectly .
Revised: In the adversarial perturbation problem of neural networks, an adversary starts with a neural network model F (we assume that F ends with a softmax layer, which is a common assumption in the literature), and a point x that F classiﬁes correctly, and identiﬁes another point x (cid:48) , which is nearby x , that F classiﬁes incorrectly .

Original: Szegedy et al.
Revised: Szegedy et al.

Original: (2013) ﬁrst noticed the vulnerability of existing (deep) neural networks to adversarial perturbations, which is a somewhat surprising phenomenon given the great generalization capability of these networks.
Revised: (2013) ﬁrst noticed the vulnerability of existing (deep) neural networks to adversarial perturbations, which is a somewhat surprising phenomenon given the great generalization capability of these networks.

Original: Since then, a line of research (see, for example, Goodfellow et al.
Revised: Since then, a line of research (see, for example, Goodfellow et al.

Original: (2014); Papernot et al.
Revised: (2014); Papernot et al.

Original: (2016b)
Revised: (2016b)

Original: ; Miyato et al.
Revised: ; Miyato et al.

Original: ; Madry et al.
Revised: ; Madry et al.

Original: (2017)) has been devoted to harden neural networks against adversarial perturbation.
Revised: (2017)) has been devoted to harden neural networks against adversarial perturbation.

Original: However, while modest progress has been made, until now there is still a large margin in successfully defending against more advanced attacks, such as Carlini-Wagner attacks Carlini & Wagner (2017).
Revised: However, while modest progress has been made, until now there is still a large margin in successfully defending against more advanced attacks, such as Carlini-Wagner attacks Carlini & Wagner (2017).

Original: In this paper we propose a defense method that is based on the semantics of a neural network.
Revised: In this paper we propose a defense method that is based on the semantics of a neural network.

Original: Our starting point is the common manifold assumption (Zhu & Goldberg (2009); man) made in several classic unsupervised and semi-supervised learning algorithms, which states that natural data points lie on separate low dimensional manifolds for different classes.
Revised: Our starting point is the common manifold assumption (Zhu & Goldberg (2009); man) made in several classic unsupervised and semi-supervised learning algorithms, which states that natural data points lie on separate low dimensional manifolds for different classes.

Original: Recent work, for example Basri & Jacobs (2016), has given intriguing evidence that neural networks can efﬁciently represent low dimensional manifold data.
Revised: Recent work, for example Basri & Jacobs (2016), has given intriguing evidence that neural networks can efﬁciently represent low dimensional manifold data.

Original: Based on this assumption, we make a further postulate , which states that a good model is conﬁdent on natural points on the manifolds, but has low conﬁdence on points outside of the manifolds.
Revised: Based on this assumption, we make a further postulate , which states that a good model is conﬁdent on natural points on the manifolds, but has low conﬁdence on points outside of the manifolds.

Original: To this end, a natural measure of conﬁdence of a model F at a point x is to simply measure how conﬁdent F is about its prediction, that is, (cid:107) F ( x ) (cid:107) ∞ .
Revised: To this end, a natural measure of conﬁdence of a model F at a point x is to simply measure how conﬁdent F is about its prediction, that is, (cid:107) F ( x ) (cid:107) ∞ .

Original: With these assumptions, the adversarial perturbation problem can thus be understood as follows: the manifold assumption indicates that the natural data points are far from permeating the entire space, thus while we may have learned perfectly for data points from the natural low dimensional manifolds, an adversary may still be able to perturb a point slightly to make it out of the natural manifolds, and render the model’s behavior “unpredictable.”
Revised: With these assumptions, the adversarial perturbation problem can thus be understood as follows: the manifold assumption indicates that the natural data points are far from permeating the entire space, thus while we may have learned perfectly for data points from the natural low dimensional manifolds, an adversary may still be able to perturb a point slightly to make it out of the natural manifolds, and render the model’s behavior “unpredictable.”

Original: Moreover, our postulate indicates that this “unpredictable behavior” can be understood as the model having low conﬁdence in its prediction.
Revised: Moreover, our postulate indicates that this “unpredictable behavior” can be understood as the model having low conﬁdence in its prediction.

Original: Therefore, a natural plan to tackle adversarial perturbation problem is to try to embed an adversarial point, which is distinguishable from natural points as the model has low conﬁdence at it, back to the natural manifolds where the model has higher conﬁdence.
Revised: Therefore, a natural plan to tackle adversarial perturbation problem is to try to embed an adversarial point, which is distinguishable from natural points as the model has low conﬁdence at it, back to the natural manifolds where the model has higher conﬁdence.

Original: We propose two embedding objective functions for this purpose: (1) δ -Most
Revised: We propose two embedding objective functions for this purpose: (1) δ -Most

Original: Conﬁdent Neighbor objective ( MCN δ ), where given a point x , and a radius parameter δ > 0 , we compute MCN δ ( F, x ) =
Revised: Conﬁdent Neighbor objective ( MCN δ ), where given a point x , and a radius parameter δ > 0 , we compute MCN δ ( F, x ) =

Original: arg max z ∈ N ( x,δ ) (cid:107) F ( z ) (cid:107) ∞ where N ( x, δ ) denotes the δ -ball around x with respect to some norm (cid:107) · (cid:107) , and (2) p -Nearest Conﬁdent Neighbor objective ( NCN p ), where given a parameter p ∈ (0 , 1) , we compute NCN p ( F, x ) =
Revised: arg max z ∈ N ( x,δ ) (cid:107) F ( z ) (cid:107) ∞ where N ( x, δ ) denotes the δ -ball around x with respect to some norm (cid:107) · (cid:107) , and (2) p -Nearest Conﬁdent Neighbor objective ( NCN p ), where given a parameter p ∈ (0 , 1) , we compute NCN p ( F, x ) =

Original: arg min z (cid:107) z − x (cid:107) subject to (cid:107) F ( z ) (cid:107) ∞ ≥ p .
Revised: arg min z (cid:107) z − x (cid:107) subject to (cid:107) F ( z ) (cid:107) ∞ ≥ p .

Original: With these two objectives, the end to end prediction thus becomes
Revised: With these two objectives, the end to end prediction thus becomes

Original: F ( MCN δ ( F, x )) or F ( NCN p ( F, x )) .
Revised: F ( MCN δ ( F, x )) or F ( NCN p ( F, x )) .

Original: A notable feature of these defense objectives is that they are semantic :
Revised: A notable feature of these defense objectives is that they are semantic :

Original: They only fail when F has a conﬁdent but wrong prediction in the neighborhood.
Revised: They only fail when F has a conﬁdent but wrong prediction in the neighborhood.

Original: However, by our postulate this is infeasible because an adversarial point must be near to at least one natural manifold where F has higher conﬁdence.
Revised: However, by our postulate this is infeasible because an adversarial point must be near to at least one natural manifold where F has higher conﬁdence.

Original: Therefore, if the natural manifolds are indeed well separated as indicated by the manifold assumption, we can successfully defend the attacks.
Revised: Therefore, if the natural manifolds are indeed well separated as indicated by the manifold assumption, we can successfully defend the attacks.

Original: Guided by our method, we construct algorithms to compute MCN δ and NCN p and give a preliminary evaluation over CIFAR10.
Revised: Guided by our method, we construct algorithms to compute MCN δ and NCN p and give a preliminary evaluation over CIFAR10.

Original: Note that the effectiveness of our method depends on two components: (1) To which degree the base model F satisﬁes our postulate, and (2) Effectiveness of solving the embedding objectives.
Revised: Note that the effectiveness of our method depends on two components: (1) To which degree the base model F satisﬁes our postulate, and (2) Effectiveness of solving the embedding objectives.

Original: As a result, we use a deep residual network recently trained by Madry et al.
Revised: As a result, we use a deep residual network recently trained by Madry et al.

Original: Madry et al.
Revised: Madry et al.

Original: (2017), as base model, and use gradient based optimization, such as Carlini-Wagner attacks Carlini & Wagner (2017) (which, however, are now used for defense) to solve MCN δ or
Revised: (2017), as base model, and use gradient based optimization, such as Carlini-Wagner attacks Carlini & Wagner (2017) (which, however, are now used for defense) to solve MCN δ or

Original: NCN p .
Revised: NCN p .

Original: Our empirical results are encouraging: (1) It achieves almost perfect success rate in defending against attacks that the base model fail at, and (2) It also retains the good generalization behavior of the base model.
Revised: Our empirical results are encouraging: (1) It achieves almost perfect success rate in defending against attacks that the base model fail at, and (2) It also retains the good generalization behavior of the base model.

Original: There are two interpretations of the encouraging success of our method: First, from a purely perspective, our results advocate that a good model should have the property that it is conﬁdent only on the structure it has learned (for example the natural manifolds), and does not claim conﬁdence beyond it .
Revised: There are two interpretations of the encouraging success of our method: First, from a pure robustness perspective, our results advocate that a good model should have the property that it is conﬁdent only on the structure it has learned (for example the natural manifolds), and does not claim conﬁdence beyond it .

Original: Second, perhaps more importantly, adversarial perturbation can naturally coexist with good generalization , and thus may not be surprising after all.
Revised: Second, perhaps more importantly, adversarial perturbation can naturally coexist with good generalization , and thus may not be surprising after all.

Original: In fact, even if one has learned perfectly over the underlying natural manifolds, adversarial perturbation may still exist in abundance, and our method can be viewed as given a post-processing to reduce points from outside (but near) the manifolds to the natural manifolds one has learned.
Revised: In fact, even if one has learned perfectly over the underlying natural manifolds, adversarial perturbation may still exist in abundance, and our method can be viewed as given a post-processing to reduce points from outside (but near) the manifolds to the natural manifolds one has learned.

Original: This gives rise to intriguing yet challenging questions, such as trying to learn apriori how to embed the entire space to manifolds that one is conﬁdent about rather than relying on a post-processing embedding procedure, which we leave as future research.
Revised: This gives rise to intriguing yet challenging questions, such as trying to learn apriori how to embed the entire space to manifolds that one is conﬁdent about rather than relying on a post-processing embedding procedure, which we leave as future research.

Original:  P RIOR W ORK
Revised:  P RIOR W ORK

Original: Szegedy et al.
Revised: Szegedy et al.

Original: ﬁrst observed the susceptibility of deep neural networks to adversarial perturbations.
Revised: ﬁrst observed the susceptibility of deep neural networks to adversarial perturbations.

Original: Since then, a large body of work have been devoted to study hardening neural networks for this problem (a small set of work in this direction is Goodfellow et al.
Revised: Since then, a large body of work have been devoted to study hardening neural networks for this problem (a small set of work in this direction is Goodfellow et al.

Original: Papernot et al.
Revised: Papernot et al.

Original: (2016b)
Revised: (2016b)

Original: ; Miyato et al.
Revised: ; Miyato et al.

Original: Simultaneously, another line of work have been devoted to devise more effective or efﬁcient attacks (a small set of work in this direction is Moosavi-Dezfooli et al.
Revised: Simultaneously, another line of work have been devoted to devise more effective or efﬁcient attacks (a small set of work in this direction is Moosavi-Dezfooli et al.

Original: Papernot et al.
Revised: Papernot et al.

Original: (2016a)
Revised: (2016a)

Original: ; Carlini & Wagner (2017)).
Revised: ; Carlini & Wagner (2017)).

Original: Unfortunately, there still seems to be a large margin for the defense methods to defend against more sophisticated attacks, such as Carlini-Wagner attacks Carlini & Wagner (2017).
Revised: Unfortunately, there still seems to be a large margin for the defense methods to defend against more sophisticated attacks, such as Carlini-Wagner attacks Carlini & Wagner (2017).

Original: For example, while the recent robust residual network constructed by Madry et al.
Revised: For example, while the recent robust residual network constructed by Madry et al.

Original: (2017) achieves encouraging robustness results on MNIST, on
Revised: (2017) achieves encouraging robustness results on MNIST, on

Original: CIFAR10 the accuracy against a strong adversary can be as low as 45 .
Revised: CIFAR10 the accuracy against a strong adversary can be as low as 45 .

Original: A dominant defense approach is to add to the training objective function an appropriate “robustness component,” and hope that the learning algorithm automatically learns to be robust.
Revised: A dominant defense approach is to add to the training objective function an appropriate “robustness component,” and hope that the learning algorithm automatically learns to be robust.

Original: Our method is fundamentally different and tries to directly exploit the structure a model has learned.
Revised: Our method is fundamentally different and tries to directly exploit the structure a model has learned.

Original: Our method has some remote similarities with a recent proposal by Lu et al.
Revised: Our method has some remote similarities with a recent proposal by Lu et al.

Original: (2017), which tries to exploit features deep in the network (that is “semantics” of a model) to detect adversarial examples.
Revised: (2017), which tries to exploit features deep in the network (that is “semantics” of a model) to detect adversarial examples.

Original: However, beyond detecting adversarial attacks, our work takes a step further to compute correct prediction on the adversarial examples.
Revised: However, beyond detecting adversarial attacks, our work takes a step further to compute correct prediction on the adversarial examples.

Original: Moreover, our work is built upon the common manifold assumption made in semi-supervised and unsupervised learning, and we are not aware of any previous work that has explored this direction.
Revised: Moreover, our work is built upon the common manifold assumption made in semi-supervised and unsupervised learning, and we are not aware of any previous work that has explored this direction.

Original: We note that the manifold assumption has guided the design of several classic dimensionality reduction algorithm, unsupervised and semi-supervised learning algorithms (for example, see Basri & Jacobs (2016) and references therein).
Revised: We note that the manifold assumption has guided the design of several classic dimensionality reduction algorithm, unsupervised and semi-supervised learning algorithms (for example, see Basri & Jacobs (2016) and references therein).

Original: An intriguing recent effort is to demonstrate that neural networks are effective devices to represent low dimensional manifold data (for example, Rifai et al.
Revised: An intriguing recent effort is to demonstrate that neural networks are effective devices to represent low dimensional manifold data (for example, Rifai et al.

Original: (2011); Shaham et al.
Revised: (2011); Shaham et al.

Original: ; Basri & Jacobs (2016)).
Revised: ; Basri & Jacobs (2016)).

Original: Our work is directly inspired by these work, and tries to argue that if manifold assumption indeed holds then it is somewhat not surprising to see the coexistence of good generalization and adversarial perturbation problem.
Revised: Our work is directly inspired by these work, and tries to argue that if manifold assumption indeed holds then it is somewhat not surprising to see the coexistence of good generalization and adversarial perturbation problem.

Original:  P RELIMINARIES
Revised:  P RELIMINARIES

Original: We only need minimal background to set up discussions in this paper.
Revised: We only need minimal background to set up discussions in this paper.

Original: Following previous work Carlini & Wagner (2017); Papernot et al.
Revised: Following previous work Carlini & Wagner (2017); Papernot et al.

Original: (2016b)
Revised: (2016b)

Original: we deﬁne F to be a neural network after the softmax layer.
Revised: we deﬁne F to be a neural network after the softmax layer.

Original: With this notation, the ﬁnal classiﬁcation is then C F ( x ) =
Revised: With this notation, the ﬁnal classiﬁcation is then C F ( x ) =

Original: arg max i F ( x ) i where F ( x ) i gives the conﬁdence of the network in classifying x for class
Revised: arg max i F ( x ) i where F ( x ) i gives the conﬁdence of the network in classifying x for class

Original: i .
Revised: i .

Original: We use Z ( x ) to denote part of F except the softmax layer.
Revised: We use Z ( x ) to denote part of F except the softmax layer.

Original: That is, Z ( x ) computes the logits to be fed into the softmax function.
Revised: That is, Z ( x ) computes the logits to be fed into the softmax function.

Original: We use C to denote the class of all labels.
Revised: We use C to denote the class of all labels.

Original:  P OSTULATE AND A LGORITHMS
Revised:  P OSTULATE AND A LGORITHMS

Original: Our starting point is the common manifold assumption that guides the design of several unsupervised and semi-supervised learning algorithms, which states that natural data points of different classes come from separated low-dimensional manifolds (Zhu & Goldberg (2009); man).
Revised: Our starting point is the common manifold assumption that guides the design of several unsupervised and semi-supervised learning algorithms, which states that natural data points of different classes come from separated low-dimensional manifolds (Zhu & Goldberg (2009); man).

Original: Intuitively, we would expect that a good learning algorithm should learn a model that behaves well on these manifolds.
Revised: Intuitively, we would expect that a good learning algorithm should learn a model that behaves well on these manifolds.

Original: On the other hand, what about points that are not on these manifolds?
Revised: On the other hand, what about points that are not on these manifolds?

Original: Naturally, we would want that the model behaves badly on those points, because those do not belong to data generation.
Revised: Naturally, we would want that the model behaves badly on those points, because those do not belong to data generation.

Original: Our postulate gives a concrete version of this intuition and states that:
Revised: Our postulate gives a concrete version of this intuition and states that:

Original: Postulate:
Revised: Postulate:

Original: (A good model) F is conﬁdent on natural points drawn from the manifolds, but has low conﬁdence on points outside of the manifolds.
Revised: (A good model) F is conﬁdent on natural points drawn from the manifolds, but has low conﬁdence on points outside of the manifolds.

Original: Essentially, the “conﬁdence” in the postulate provides a means to distinguish between “good” and “bad” behaviors of a model.
Revised: Essentially, the “conﬁdence” in the postulate provides a means to distinguish between “good” and “bad” behaviors of a model.

Original: A natural measure of conﬁdence of a model F at a point x is to simply measure (cid:107) F ( x ) (cid:107) ∞ , that is, how conﬁdent F is about its prediction.
Revised: A natural measure of conﬁdence of a model F at a point x is to simply measure (cid:107) F ( x ) (cid:107) ∞ , that is, how conﬁdent F is about its prediction.

Original: With this measure our postulate thus states the following: (1) (cid:107) F ( x ) (cid:107) ∞ is large for a point x that is drawn from the manifolds, and (2) (cid:107) F ( x ) (cid:107) ∞ is small otherwise.
Revised: With this measure our postulate thus states the following: (1) (cid:107) F ( x ) (cid:107) ∞ is large for a point x that is drawn from the manifolds, and (2) (cid:107) F ( x ) (cid:107) ∞ is small otherwise.

Original: We now give a view of adversarial examples under our postulate.
Revised: We now give a view of adversarial examples under our postulate.

Original: The manifold assumption indicates that natural data points are far from permeating the entire space.
Revised: The manifold assumption indicates that natural data points are far from permeating the entire space.

Original: Therefore adversarial examples can be viewed as points that are “perturbed” to be out of the manifolds that generate natural data points, but they are still close to at least one underlying manifold under certain distance metric.
Revised: Therefore adversarial examples can be viewed as points that are “perturbed” to be out of the manifolds that generate natural data points, but they are still close to at least one underlying manifold under certain distance metric.

Original: Moreover, with our postulate it is the case that a good model F must be unconﬁdent there.
Revised: Moreover, with our postulate it is the case that a good model F must be unconﬁdent there.

Original: Therefore, a natural plan to tackle adversarial points is to try to embed them back to the manifolds where the model has higher conﬁdence.
Revised: Therefore, a natural plan to tackle adversarial points is to try to embed them back to the manifolds where the model has higher conﬁdence.

Original: In particular, we note that there are two components for this idea to work: First, we need the model F , which is the base model we start with, to be trained in a way to satisfy the postulate so that it will not be conﬁdent on “bad” points outside the manifolds.
Revised: In particular, we note that there are two components for this idea to work: First, we need the model F , which is the base model we start with, to be trained in a way to satisfy the postulate so that it will not be conﬁdent on “bad” points outside the manifolds.

Original: Second, we need an effective, and preferably efﬁcient, embedding procedure so that we can go back to the natural manifold.
Revised: Second, we need an effective, and preferably efﬁcient, embedding procedure so that we can go back to the natural manifold.

Original: Objective Functions.
Revised: Objective Functions.

Original: We now give two natural objective functions for such embedding:
Revised: We now give two natural objective functions for such embedding:

Original: Deﬁnition 1 ( Most Conﬁdent Neighbor Objective ) .
Revised: Deﬁnition 1 ( Most Conﬁdent Neighbor Objective ) .

Original: Let F be a model, (cid:107) · (cid:107) be a norm, and δ >be a real number.
Revised: Let F be a model, (cid:107) · (cid:107) be a norm, and δ >be a real number.

Original: The δ -Most Conﬁdent Neighbor objective (or simply MCN δ objective) computes:
Revised: The δ -Most Conﬁdent Neighbor objective (or simply MCN δ objective) computes:

Original: [Equation]
Revised: [Equation]

Original: where N ( x , δ ) is the δ -ball around x with respect to (cid:107) · (cid:107) , and (cid:107) · (cid:107) ∞ denotes the L ∞ -norm.
Revised: where N ( x , δ ) is the δ -ball around x with respect to (cid:107) · (cid:107) , and (cid:107) · (cid:107) ∞ denotes the L ∞ -norm.

Original: Deﬁnition 2 ( Nearest Conﬁdent Neighbor Objective ) .
Revised: Deﬁnition 2 ( Nearest Conﬁdent Neighbor Objective ) .

Original: Let F be a model, (cid:107) · (cid:107) be a norm, and p ∈ (0 , 1) be a real number.
Revised: Let F be a model, (cid:107) · (cid:107) be a norm, and p ∈ (0 , 1) be a real number.

Original: The p - Nearest Conﬁdent Neighbor objective (or simply NCN p objective) computes:
Revised: The p - Nearest Conﬁdent Neighbor objective (or simply NCN p objective) computes:

Original: [Equation]
Revised: [Equation]

Original:  to 
Revised: subject to (cid:107) [Equation]

Original: [Equation]
Revised: [Equation]

Original: where (cid:107) · (cid:107) ∞ denotes the L ∞ -norm.
Revised: where (cid:107) · (cid:107) ∞ denotes the L ∞ -norm.

Original: In words, MCN says that one should consider a reasonable δ -neighborhood of a point and view the most conﬁdent point in that ball as a good point that lies on a natural manifold; meanwhile, NCN says that one should consider the nearest neighbor that achieves certain conﬁdence threshold as a point that lies on a natural manifold.
Revised: In words, MCN says that one should consider a reasonable δ -neighborhood of a point and view the most conﬁdent point in that ball as a good point that lies on a natural manifold; meanwhile, NCN says that one should consider the nearest neighbor that achieves certain conﬁdence threshold as a point that lies on a natural manifold.

Original: Under our postulate, solutions of these objectives are thus preferable to be used for prediction.
Revised: Under our postulate, solutions of these objectives are thus preferable to be used for prediction.

Original: Duality.
Revised: Duality.

Original: We note that the optimization problems for 1 and 2 are essentially dual to each other:
Revised: We note that the optimization problems for 1 and 2 are essentially dual to each other:

Original: In the MCN objective, we have a ﬁxed radius of the neighborhood to search and one ﬁnds the point that achieves highest conﬁdence.
Revised: In the MCN objective, we have a ﬁxed radius of the neighborhood to search and one ﬁnds the point that achieves highest conﬁdence.

Original: In NCN objective, instead of ﬁxing the neighborhood radius, one starts with a conﬁdence threshold p , and ﬁnds the nearest point that achieves that conﬁdence.
Revised: In NCN objective, instead of ﬁxing the neighborhood radius, one starts with a conﬁdence threshold p , and ﬁnds the nearest point that achieves that conﬁdence.

Original: In fact, we note that the Lagrange dual of these two objective functions share the same form.
Revised: In fact, we note that the Lagrange dual of these two objective functions share the same form.

Original: To see this, we note that the Lagrange dual problems of 1 and 2 are given as
Revised: To see this, we note that the Lagrange dual problems of 1 and 2 are given as

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: respectively.
Revised: respectively.

Original: Therefore, if one sets µ = 1 λ , then these two problems are equivalent to the following uniﬁed optimization problem
Revised: Therefore, if one sets µ = 1 λ , then these two problems are equivalent to the following uniﬁed optimization problem

Original: [Equation]
Revised: [Equation]

Original: Intuitively, this objective encourages small distances of z from x while large conﬁdence of F at z .
Revised: Intuitively, this objective encourages small distances of z from x while large conﬁdence of F at z .

Original: Algorithms.
Revised: Algorithms.

Original: We now give algorithms that instantiate our discussion so far.
Revised: We now give algorithms that instantiate our discussion so far.

Original: In particular, we give algorithms to solve MCN and NCN objectives for embedding purpose.
Revised: In particular, we give algorithms to solve MCN and NCN objectives for embedding purpose.

Original: To start with, we note that from an algorithmic point of view, our approach essentially tries to wrap around a learned model with a “shell” to handle points that are “slightly” out of the natural manifolds it may have learned.
Revised: To start with, we note that from an algorithmic point of view, our approach essentially tries to wrap around a learned model with a “shell” to handle points that are “slightly” out of the natural manifolds it may have learned.

Original: There are many possible shells that can be instantiated for this purpose to exploit the effectiveness-efﬁciency tradeoff.
Revised: There are many possible shells that can be instantiated for this purpose to exploit the effectiveness-efﬁciency tradeoff.

Original: For example, a valid such shell is to sample in the neighborhood and do a weighted majority vote.
Revised: For example, a valid such shell is to sample in the neighborhood and do a weighted majority vote.

Original: We are interested in experimenting with these shells, which may shed light on what a deep network has learned.
Revised: We are interested in experimenting with these shells, which may shed light on what a deep network has learned.

Original: We thus ﬁrst give some general deﬁnitions.
Revised: We thus ﬁrst give some general deﬁnitions.

Original: We begin with model shell .
Revised: We begin with model shell .

Original: Deﬁnition 3 ( Model Shell ) .
Revised: Deﬁnition 3 ( Model Shell ) .

Original: Let F be a model.
Revised: Let F be a model.

Original: A model shell G is an algorithm that takes as its input F and a point x ∈ R d to classify, and outputs a softmax layer
Revised: A model shell G is an algorithm that takes as its input F and a point x ∈ R d to classify, and outputs a softmax layer

Original: G ( F, x ) .
Revised: G ( F, x ) .

Original: In this paper we only consider the case where F is a model that is chosen apriori and ﬁxed.
Revised: In this paper we only consider the case where F is a model that is chosen apriori and ﬁxed.

Original: In this case, the resulting model for classiﬁcation is G ( F, · ) , which maps a feature vector to a softmax layer.
Revised: In this case, the resulting model for classiﬁcation is G ( F, · ) , which maps a feature vector to a softmax layer.

Original: In this scenario, we say that F is the base model of the model shell G , and denote G ( F, · ) as G [ F ] .
Revised: In this scenario, we say that F is the base model of the model shell G , and denote G ( F, · ) as G [ F ] .

Original: Model shell captures the intuition that we want to wrap around an existing model with another algorithm that can exploit semantics of the base model.
Revised: Model shell captures the intuition that we want to wrap around an existing model with another algorithm that can exploit semantics of the base model.

Original: In this paper, we only examine model shells with a restricted structure, which we call factor model shells .
Revised: In this paper, we only examine model shells with a restricted structure, which we call factor model shells .

Original: We deﬁne search factor ,
Revised: We deﬁne search factor ,

Original: Deﬁnition 4 ( Search Factor ) .
Revised: Deﬁnition 4 ( Search Factor ) .

Original: A search factor H is an algorithm that takes as its input a model F , and a point x ∈ R d to classify, and output another point x (cid:48) ∈ R d .
Revised: A search factor H is an algorithm that takes as its input a model F , and a point x ∈ R d to classify, and output another point x (cid:48) ∈ R d .

Original: If F is chosen and ﬁxed apriori we say that F is the base model of the search factor H , and denote H ( F, · ) as H [ F ] .
Revised: If F is chosen and ﬁxed apriori we say that F is the base model of the search factor H , and denote H ( F, · ) as H [ F ] .

Original: Deﬁnition 5 ( Factor Model Shell ) .
Revised: Deﬁnition 5 ( Factor Model Shell ) .

Original: A model shell G is said to be a factor model shell if G can be written as a composition of F ◦ H where H is a search factor.
Revised: A model shell G is said to be a factor model shell if G can be written as a composition of F ◦ H where H is a search factor.

Original: In other words, G ( F, x ) works asG
Revised: In other words, G ( F, x ) works as

Original: [Equation]
Revised: [Equation]

Original: A factor model shell captures our intuitions from MCN or NCN objectives.
Revised: A factor model shell captures our intuitions from MCN or NCN objectives.

Original: That is, we ﬁrst apply a search factor H to ﬁnd another feature point x (cid:48)
Revised: That is, we ﬁrst apply a search factor H to ﬁnd another feature point x (cid:48)

Original: =
Revised: =

Original: H ( F, x ) , and then apply the base model F on x (cid:48) to produce a ﬁnal prediction.
Revised: H ( F, x ) , and then apply the base model F on x (cid:48) to produce a ﬁnal prediction.

Original: As a result, a factor model shell for MCN δ objective should compute the ﬁnal prediction as:
Revised: As a result, a factor model shell for MCN δ objective should compute the ﬁnal prediction as:

Original: [Equation]
Revised: [Equation]

Original: and accordingly for NCN p objective:
Revised: and accordingly for NCN p objective:

Original: [Equation]
Revised: [Equation]

Original: Semantic defenses.
Revised: Semantic defenses.

Original: We note that exact computations of (6) and (7) provide “semantic defenses:”
Revised: We note that exact computations of (6) and (7) provide “semantic defenses:”

Original: That is, the ﬁnal prediction is wrong only if the model has a conﬁdent but wrong prediction in the neighborhood.
Revised: That is, the ﬁnal prediction is wrong only if the model has a conﬁdent but wrong prediction in the neighborhood.

Original: Under our postulate this is impossible because an adversarial point must be near to a manifold where the model has high conﬁdence.
Revised: Under our postulate this is impossible because an adversarial point must be near to a manifold where the model has high conﬁdence.

Original: We note that a line of work shows that a deep network can have high conﬁdent predictions on random noises Nguyen et al.
Revised: We note that a line of work shows that a deep network can have high conﬁdent predictions on random noises Nguyen et al.

Original: , we discuss these phenomena later in Section 6.
Revised: , we discuss these phenomena later in Section 6.

Original: Non-differentiability of MCN or NCN model shells.
Revised: Non-differentiability of MCN or NCN model shells.

Original: We note that there are additional beneﬁts taking the MCN or NCN approach.
Revised: We note that there are additional beneﬁts taking the MCN or NCN approach.

Original: That is, the resulting factor model shells, ( F ◦ MCN δ ) and ( F ◦ NCN p ) are non-differentiable .
Revised: That is, the resulting factor model shells, ( F ◦ MCN δ ) and ( F ◦ NCN p ) are non-differentiable .

Original: This will render existing attacks, which are all based on the assumption that the output softmax layer is differentiable with respect to the input feature vector x 1 fail to work directly to attack the end-to-end model F ◦ MCN δ or F ◦ NCN p .
Revised: This will render existing attacks, which are all based on the assumption that the output softmax layer is differentiable with respect to the input feature vector x 1 fail to work directly to attack the end-to-end model F ◦ MCN δ or F ◦ NCN p .

Original: In particular, we note that MCN δ ( F, x ) and NCN p ( F, x ) are non-differentiable given that F is a deep neural network.
Revised: In particular, we note that MCN δ ( F, x ) and NCN p ( F, x ) are non-differentiable given that F is a deep neural network.

Original: This, in particular, implies the compositions, namely F ◦ MCN δ and F ◦ NCN p , are non-differentiable as well.
Revised: This, in particular, implies the compositions, namely F ◦ MCN δ and F ◦ NCN p , are non-differentiable as well.

Original: This forces attacks to either attack the base model F and then transfer attacks to the model shell, or has to ﬁnd some differentiable approximations in order to perform attacks, or use derivative-free optimization to attack the model shell directly.
Revised: This forces attacks to either attack the base model F and then transfer attacks to the model shell, or has to ﬁnd some differentiable approximations in order to perform attacks, or use derivative-free optimization to attack the model shell directly.

Original: Solving objective functions by Gradient-based optimization.
Revised: Solving objective functions by Gradient-based optimization.

Original: We now give one concrete search factor to solve (1) and (2).
Revised: We now give one concrete search factor to solve (1) and (2).

Original: To start with, we note that the optimization for solving MCN δ (1) is nothing but for each label t ∈ C we try to ﬁnd
Revised: To start with, we note that the optimization for solving MCN δ (1) is nothing but for each label t ∈ C we try to ﬁnd

Original: [Equation]
Revised: [Equation]

Original: and then compute z ∗ =
Revised: and then compute z ∗ =

Original: arg max z t F ( z t ) t .
Revised: arg max z t F ( z t ) t .

Original: Now, however, we can solve (8) using any preferred gradient-based optimization (for example, projected gradient descent Nocedal & Wright (2006)).
Revised: Now, however, we can solve (8) using any preferred gradient-based optimization (for example, projected gradient descent Nocedal & Wright (2006)).

Original: This thus gives the following factor model shell:
Revised: This thus gives the following factor model shell:

Original: [Equation]
Revised: [Equation]

Original: is natural for neural networks because neural networks are typically composition of differentiable functions.
Revised: is natural for neural networks because neural networks are typically composition of differentiable functions.

Original: With ﬁrst order information, attacks can “relatively accu- rately” follow the gradient direction to modify the image feature so as to increase the conﬁdence of the model in incorrect labels and thus produce adversarial perturbations.
Revised: With ﬁrst order information, attacks can “relatively accu- rately” follow the gradient direction to modify the image feature so as to increase the conﬁdence of the model in incorrect labels and thus produce adversarial perturbations.

Original: Input: x a feature vector, δ > 0 a real parameter, a base model F , any gradient-based optimization algorithm O to solve the constrained optimization problem deﬁned in (8).
Revised: Input: x a feature vector, δ > 0 a real parameter, a base model F , any gradient-based optimization algorithm O to solve the constrained optimization problem deﬁned in (8).

Original: 1: function MCNOracleShell( x , δ, F ) 2: for t C do 3: z t ← O ( x , F, t ) 4: return arg t ∈C F ( z t )
Revised: MCNOracleShell

Original: Remark 1 ( Turning an adversarial attack to a defense ) .
Revised: Remark 1 ( Turning an adversarial attack to a defense ) .

Original: In particular, we note that solving (8) is similar to a targeted adversarial attack where we want to modify x so as to increase the conﬁdence in label t .
Revised: In particular, we note that solving (8) is similar to a targeted adversarial attack where we want to modify x so as to increase the conﬁdence in label t .

Original: Therefore one can instantiate O as any adversarial attack .
Revised: Therefore one can instantiate O as any adversarial attack .

Original: Speciﬁcally, one can any strong attack proposed in the Carlini-Wagner Carlini & Wagner (2017), denoted by CW , which takes as input a feature vector x , a model F , and a label t , and output z t that arg max i F (CW( x , F, t ))
Revised: Speciﬁcally, one can any strong attack proposed in the Carlini-Wagner Carlini & Wagner (2017), denoted by CW , which takes as input a feature vector x , a model F , and a label t , and output z t that arg max i F (CW( x , F, t ))

Original: =
Revised: =

Original: t , and use it as O .
Revised: t , and use it as O .

Original: Remark 2 ( Local vs. global optimal ) .
Revised: Remark 2 ( Local vs. global optimal ) .

Original: We note that gradient-based optimization may only ﬁnd local optimal in the neighborhood.
Revised: We note that gradient-based optimization may only ﬁnd local optimal in the neighborhood.

Original: However, global optimal may not be needed for our method to work, as long as local minima are good.
Revised: However, global optimal may not be needed for our method to work, as long as local minima are good.

Original: For example, let l be the correct class, and denote
Revised: For example, let l be the correct class, and denote

Original: S l = { z : z ∈ N ( x, δ ) , z is a local maximal of (cid:107) F ( x ) (cid:107) ∞ , (cid:107) F ( z ) (cid:107) ∞
Revised: S l = { z : z ∈ N ( x, δ ) , z is a local maximal of (cid:107) F ( x ) (cid:107) ∞ , (cid:107) F ( z ) (cid:107) ∞

Original: =
Revised: =

Original: F ( z ) l } .
Revised: F ( z ) l } .

Original: That is S l contains all the local maximal for label l .
Revised: That is S l contains all the local maximal for label l .

Original: Moreover, deﬁne
Revised: Moreover, deﬁne

Original: S − l = { z : z ∈ N ( x, δ ) , z is a local maximal of (cid:107) F ( x ) (cid:107) ∞ , (cid:107) F ( z ) (cid:107) ∞ (cid:54)
Revised: S − l = { z : z ∈ N ( x, δ ) , z is a local maximal of (cid:107) F ( x ) (cid:107) ∞ , (cid:107) F ( z ) (cid:107) ∞ (cid:54)

Original: = F ( z ) l } .
Revised: = F ( z ) l } .

Original: That is S − l contains all the local maximal that is not for label l , then local optimal works as soon as inf z ∈S
Revised: That is S − l contains all the local maximal that is not for label l , then local optimal works as soon as inf z ∈S

Original: [Equation]
Revised: [Equation]

Original: We can also take a similar path to solve NCN p objective with standard numeric optimization.
Revised: We can also take a similar path to solve NCN p objective with standard numeric optimization.

Original: Note that (cid:107) F ( z ) (cid:107) ∞ ≥ p is equivalent to ∃ t, F ( z ) t ≥ p .
Revised: Note that (cid:107) F ( z ) (cid:107) ∞ ≥ p is equivalent to ∃ t, F ( z ) t ≥ p .

Original: Therefore, we can solve
Revised: Therefore, we can solve

Original: [Equation]
Revised: [Equation]

Original: for every t , and then compute z ∗ = arg min z t (cid:107)
Revised: for every t , and then compute z ∗ = arg min z t (cid:107)

Original: z t − x (cid:107) .
Revised: z t − x (cid:107) .

Original: By a similar reasoning as the MCN case above, z ∗ is the solution to problem 2.
Revised: By a similar reasoning as the MCN case above, z ∗ is the solution to problem 2.

Original: Note that problem 9 is a non-convex optimization problem since F ( z ) t is non-convex and thus it remains hard to ﬁnd a global optimal solution.
Revised: Note that problem 9 is a non-convex optimization problem since F ( z ) t is non-convex and thus it remains hard to ﬁnd a global optimal solution.

Original: Nevertheless, it makes sense to ﬁnd local solutions.
Revised: Nevertheless, it makes sense to ﬁnd local solutions.

Original: For example, a natural approach is to turn solving the original constrained optimization problem into solving a series of unconstrained optimization.
Revised: For example, a natural approach is to turn solving the original constrained optimization problem into solving a series of unconstrained optimization.

Original: More speciﬁcally, let us deﬁne the quadratic penalty function
Revised: More speciﬁcally, let us deﬁne the quadratic penalty function

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original:  where [ y ] − denotes max { y, 0 } .
Revised: ) where [ y ] − denotes max { y, 0 } .

Original: Then we can solve the unconstrained optimization min Q ( z , α
Revised: Then we can solve the unconstrained optimization min Q ( z , α

Original: [Equation]
Revised: [Equation]

Original: t ) for a series of different α , and terminates once we ﬁnd a satisfying solution.
Revised: t ) for a series of different α , and terminates once we ﬁnd a satisfying solution.

Original: The intuition is that i) solving the unconstrained optimization is easy, ii) by increasing the coefﬁcient α , we can force the minimizer of the penalty function closer to the feasible region F ( z ) ≥ p , , and iii) for well chosen α , the minimizer of the penalty function is close enough to that of problem 9.
Revised: The intuition is that i) solving the unconstrained optimization is easy, ii) by increasing the coefﬁcient α , we can force the minimizer of the penalty function closer to the feasible region F ( z ) ≥ p , , and iii) for well chosen α , the minimizer of the penalty function is close enough to that of problem 9.

Original: This is well known as the augmented Lagrangian method Bertsekas (1996).
Revised: This is well known as the augmented Lagrangian method Bertsekas (1996).

Original: There are other classic constrained optimization algorithms such as interior methods and sequential quadratic programming Nocedal &
Revised: There are other classic constrained optimization algorithms such as interior methods and sequential quadratic programming Nocedal &

Original: Wright (2006).
Revised: Wright (2006).

Original: More recently, there is also some work Bienstock & Michalka (2014) speciﬁcally focusing on optimizing convex objective over non-convex constraints, into which problem 9 falls as well.
Revised: More recently, there is also some work Bienstock & Michalka (2014) speciﬁcally focusing on optimizing convex objective over non-convex constraints, into which problem 9 falls as well.

Original: In short, there are quite a few methods can be exploited to solve problem 9.
Revised: In short, there are quite a few methods can be exploited to solve problem 9.

Original: Having that being said, the algorithm to solve the original NCN p is summarized below as Algorithm 2.
Revised: Having that being said, the algorithm to solve the original NCN p is summarized below as Algorithm 2.

Original: Algorithm 2 Solving NCN p objective.
Revised: Algorithm 2 Solving NCN p objective.

Original:  P RELIMINARY E VALUATION
Revised:  P RELIMINARY E VALUATION

Original: In this section we perform a preliminary empirical evaluation of our approach against strong adversarial attacks.
Revised: In this section we perform a preliminary empirical evaluation of our approach against strong adversarial attacks.

Original: Note that there are three basic components in a defense with our method: (1) the base model we choose, (2) the embedding procedure we choose to wrap around the base model, and (3) the attack we choose to challenge the defense.
Revised: Note that there are three basic components in a defense with our method: (1) the base model we choose, (2) the embedding procedure we choose to wrap around the base model, and (3) the attack we choose to challenge the defense.

Original: As a result, there are three key empirical questions we aim to answer:
Revised: As a result, there are three key empirical questions we aim to answer:

Original: How susceptible is the chosen base model to the chosen attack?
Revised: How susceptible is the chosen base model to the chosen attack?

Original: Clearly, we want that the chosen attack is strong enough to break even a “good” base model on many points.
Revised: Clearly, we want that the chosen attack is strong enough to break even a “good” base model on many points.

Original: How effective does our approach improve robustness of the base model?
Revised: How effective does our approach improve robustness of the base model?

Original: Ideally, we want to see a signiﬁcant improvement of robustness for a good base model.
Revised: Ideally, we want to see a signiﬁcant improvement of robustness for a good base model.

Original: Will our approach change the generalization behavior of the base model?
Revised: Will our approach change the generalization behavior of the base model?

Original: Since our method will change the way how the model predicts, we need to justify if there is little or no change in generalization.
Revised: Since our method will change the way how the model predicts, we need to justify if there is little or no change in generalization.

Original: A summary of our ﬁndings are the following:
Revised: A summary of our ﬁndings are the following:

Original: We choose a recent robust model trained by Madry et al.
Revised: We choose a recent robust model trained by Madry et al.

Original: Madry et al.
Revised: Madry et al.

Original: (2017) as the base model, and use a strong attack by Carlini Wagner (denoted by CW attack) Carlini & Wagner (2017) as the attack method.
Revised: (2017) as the base model, and use a strong attack by Carlini Wagner (denoted by CW attack) Carlini & Wagner (2017) as the attack method.

Original: We use CIFAR10 Krizhevsky (2009) as the dataset to evaluate the robustness and generalization.
Revised: We use CIFAR10 Krizhevsky (2009) as the dataset to evaluate the robustness and generalization.

Original: As reported by the paper, Madry et al.
Revised: As reported by the paper, Madry et al.

Original: ’s model is fairly susceptible to CW attack over CIFAR10, and the accuracy against CW attack can be as low as 45 .
Revised: ’s model is fairly susceptible to CW attack over CIFAR10, and the accuracy against CW attack can be as low as 45 .

Original: Our evaluation conﬁrms this susceptibility and indicates a signiﬁcant room for improving robustness .
Revised: Our evaluation conﬁrms this susceptibility and indicates a signiﬁcant room for improving robustness .

Original: For model shell, we choose MCNOracleShell , and use again CW attack as indicated by our discussion in Remark 1.
Revised: For model shell, we choose MCNOracleShell , and use again CW attack as indicated by our discussion in Remark 1.

Original: We ﬁnd that this embedding procedure signiﬁcantly improves the robustness of the base model over the small batch of points in our evaluation.
Revised: We ﬁnd that this embedding procedure signiﬁcantly improves the robustness of the base model over the small batch of points in our evaluation.

Original: In fact, CarliniWagerShell retains robustness for points that are already robust with the base model, but also successfully defend all attacks that can succeed on the base model.
Revised: In fact, CarliniWagerShell retains robustness for points that are already robust with the base model, but also successfully defend all attacks that can succeed on the base model.

Original: Therefore, it reduces the attack success rate from 30% to 0% .
Revised: Therefore, it reduces the attack success rate from 30% to 0% .

Original: Finally, applying CW attack as the embedding procedure incurs almost no effect on generalization.
Revised: Finally, applying CW attack as the embedding procedure incurs almost no effect on generalization.

Original: In fact, there is only one such point that the embedding procedure changes the prediction from correct to wrong.
Revised: In fact, there is only one such point that the embedding procedure changes the prediction from correct to wrong.

Original: We note that the model has very low conﬁdence ( ∼ 50% , which is much lower than other correct predictions) on the correct prediction, which may indicate that this image is difﬁcult to classify.
Revised: We note that the model has very low conﬁdence ( ∼ 50% , which is much lower than other correct predictions) on the correct prediction, which may indicate that this image is difﬁcult to classify.

Original: We give more discussions on this point later in this section.
Revised: We give more discussions on this point later in this section.

Original: Limitations of our experiments.
Revised: Limitations of our experiments.

Original: Unfortunately due to the complexity of the Madry et al.’s residual network model, our embedding procedure (which is CW attack in our experiments) takes a long time to compute z t in the MCNOracleShell (Algorithm (1)).
Revised: Unfortunately due to the complexity of the Madry et al.’s residual network model, our embedding procedure (which is CW attack in our experiments) takes a long time to compute z t in the MCNOracleShell (Algorithm (1)).

Original: Therefore in the current evaluation we only sample a small batch of points to test.
Revised: Therefore in the current evaluation we only sample a small batch of points to test.

Original: Speciﬁcally, we generate a random permutation of images in the test set, go one by one through the permutation, and test our method only on test images that the model correctly classiﬁes .
Revised: Speciﬁcally, we generate a random permutation of images in the test set, go one by one through the permutation, and test our method only on test images that the model correctly classiﬁes .

Original: In our experiments, we have evaluated our method onpoints which the base model classiﬁes correctly.
Revised: In our experiments, we have evaluated our method onpoints which the base model classiﬁes correctly.

Original: While this is a deﬁnite limitation of our current experiments, we believe that our method has shown some early promise in tackling the important problem of adversarial perturbation.
Revised: While this is a deﬁnite limitation of our current experiments, we believe that our method has shown some early promise in tackling the important problem of adversarial perturbation.

Original: We have made our model and code publicly available in an anonymous repository (cod) for better visibility (in fact our method is also easy to set up once a base model is ready).
Revised: We have made our model and code publicly available in an anonymous repository (cod) for better visibility (in fact our method is also easy to set up once a base model is ready).

Original: We hope that our work can trigger more thoughts in exploiting manifold assumption to enhance model robustness.
Revised: We hope that our work can trigger more thoughts in exploiting manifold assumption to enhance model robustness.

Original: Finally, we believe that the efﬁciency issue of an embedding procedure will be eventually resolved once we get the right robustness method.
Revised: Finally, we believe that the efﬁciency issue of an embedding procedure will be eventually resolved once we get the right robustness method.

Original: So far we have been trying to control the parameters of CW attack to accelerate the embedding and it already gives signiﬁcant improvements over the vanilla version (without sacriﬁcing robustness and generalization).
Revised: So far we have been trying to control the parameters of CW attack to accelerate the embedding and it already gives signiﬁcant improvements over the vanilla version (without sacriﬁcing robustness and generalization).

Original: Also, there are some faster attack methods that seem to give more reasonable running time, which we plan to integrate into our code.
Revised: Also, there are some faster attack methods that seem to give more reasonable running time, which we plan to integrate into our code.

Original: All in all, accelerating embedding while retaining good robustness and generalization seems to be interesting avenue to 
Revised: All in all, accelerating embedding while retaining good robustness and generalization seems to be interesting avenue to pursue.

Revised: We now give more details of our experiments.

Original: Experimental Setup We use CIFAR10 dataset Krizhevsky (2009) for our experiments.
Revised: Experimental Setup We use CIFAR10 dataset Krizhevsky (2009) for our experiments.

Original: CIFAR dataset consists of 60000 color images of objects where 50000 are for training and 10000 are for testing.
Revised: CIFAR dataset consists of 60000 color images of objects where 50000 are for training and 10000 are for testing.

Original: Each image is of size 32 × 32 pixels, and there are ten labels { 0 , 1 , . . .
Revised: Each image is of size 32 × 32 pixels, and there are ten labels { 0 , 1 , . . .

Original: , 9 } for classiﬁcation.
Revised: , 9 } for classiﬁcation.

Original: We use a model recently trained by Madry et al.
Revised: We use a model recently trained by Madry et al.

Original: (2017) as base model.
Revised: (2017) as base model.

Original: We port their model to
Revised: We port their model to

Original: Keras so that the original Carlini-Wagner attack code (cw-) can be directly applied.
Revised: Keras so that the original Carlini-Wagner attack code (cw-) can be directly applied.

Original: We do so by using the same architecture and copy the weights of the trained model from Madry et al.
Revised: We do so by using the same architecture and copy the weights of the trained model from Madry et al.

Original: (mad).
Revised: (mad).

Original: As we mentioned brieﬂy before, we use Carlini-Wager attacks ( CW attack) for two purposes: (1) as an attack to challenge the defense.
Revised: As we mentioned brieﬂy before, we use Carlini-Wager attacks ( CW attack) for two purposes: (1) as an attack to challenge the defense.

Original: In this case, we use the original attack for attacks that are as strong as possible.
Revised: In this case, we use the original attack for attacks that are as strong as possible.

Original: We apply non-targeted attacks (namely changing classiﬁcation is the purpose) with L ∞ norm (i.e. try to ﬁnd a perturbation of minimal L ∞ norm).
Revised: We apply non-targeted attacks (namely changing classiﬁcation is the purpose) with L ∞ norm (i.e. try to ﬁnd a perturbation of minimal L ∞ norm).

Original: We enforce a norm bound ε = 8255 and a perturbation is valid if and only if it both changes the classiﬁcation and its L ∞ norm is within the bound.
Revised: We enforce a norm bound ε = 8255 and a perturbation is valid if and only if it both changes the classiﬁcation and its L ∞ norm is within the bound.

Original: Note that in the original work of Madry et al.
Revised: Note that in the original work of Madry et al.

Original: the norm bound used is ε = 8 . 0 .
Revised: the norm bound used is ε = 8 . 0 .

Original: We use the current bound because we normalize the images when applying the CW attack.
Revised: We use the current bound because we normalize the images when applying the CW attack.

Original: (2) as an embedding procedure for instantiating MCNOracleShell .
Revised: (2) as an embedding procedure for instantiating MCNOracleShell .

Original: We call the resulting shell CarliniWagnerShell 2 .
Revised: We call the resulting shell CarliniWagnerShell 2 .

Original: Experimental Methodology
Revised: Experimental Methodology

Original: We ﬁrst compute a random permutation of the test images, and then go through the permutation one by one.
Revised: We ﬁrst compute a random permutation of the test images, and then go through the permutation one by one.

Original: For one image, we discard it if the base model makes a wrong prediction.
Revised: For one image, we discard it if the base model makes a wrong prediction.

Original: Otherwise, we do the following tests: (1) Susceptibility of the base model to
Revised: Otherwise, we do the following tests: (1) Susceptibility of the base model to

Original: CW attack at this point.
Revised: CW attack at this point.

Original: The success/failure of the CW attack on each point is recorded for later comparisons.
Revised: The success/failure of the CW attack on each point is recorded for later comparisons.

Original: (2) Robustness.
Revised: (2) Robustness.

Original: There are two cases: (i) the model is already robust to CW attack where then we expect CarlinWagnerShell retains this robustness, and (ii) the model is vulnerable to CW attack, where we then expect CarliniWagnerShell to harden the model.
Revised: There are two cases: (i) the model is already robust to CW attack where then we expect CarlinWagnerShell retains this robustness, and (ii) the model is vulnerable to CW attack, where we then expect CarliniWagnerShell to harden the model.

Original: (3) Generalization.
Revised: (3) Generalization.

Original: We also measure whether CarliniWagnerShell produces correct prediction on the point that the base model predicts correctly.
Revised: We also measure whether CarliniWagnerShell produces correct prediction on the point that the base model predicts correctly.

Original: This reﬂects whether the model shell retains the generalization.
Revised: This reﬂects whether the model shell retains the generalization.

Original: Experimental results.
Revised: Experimental results.

Original: Among 30 points which the base model classiﬁes correctly, we ﬁnd the base model is susceptible to CW for 9 of them (thus the attack success rate is 30% ).
Revised: Among 30 points which the base model classiﬁes correctly, we ﬁnd the base model is susceptible to CW for 9 of them (thus the attack success rate is 30% ).

Original: When applying CarliniWagnerShell, we ﬁnd that the points that were robust with base model remain robust.
Revised: When applying CarliniWagnerShell, we ﬁnd that the points that were robust with base model remain robust.

Original: Moreover,
Revised: Moreover,

Original: CarliniWagnerShell also successfully defends all the attacks for those 9 vulnerable points where the base model is not robust, thus the attack success rate is reduced to zero (we defend all attacks without introducing new vulnerability).
Revised: CarliniWagnerShell also successfully defends all the attacks for those 9 vulnerable points where the base model is not robust, thus the attack success rate is reduced to zero (we defend all attacks without introducing new vulnerability).

Original: Finally, for all these points (where base model is correct),
Revised: Finally, for all these points (where base model is correct),

Original: CarliniWagnerShell also makes correct predictions except for one of them .
Revised: CarliniWagnerShell also makes correct predictions except for one of them .

Original: Thus the accuracy we retain is 96 .
Revised: Thus the accuracy we retain is 96 .

Original: These results are summarized in 1.
Revised: These results are summarized in 1.

Revised: [Table]

Original: [Table]
Revised: [Table]

Original: The wrong data point.
Revised: The wrong data point.

Original: We now describe the data point that the model shell gives wrong prediction while the base model is correct.
Revised: We now describe the data point that the model shell gives wrong prediction while the base model is correct.

Original: We note that the model gives low conﬁdence ( ∼ 50% ) on the original image (where the correct label is a “cat”), and it has signiﬁcantly higher conﬁdence ( ∼ 75% ) on the perturbed image (classiﬁed as a “frog”).
Revised: We note that the model gives low conﬁdence ( ∼ 50% ) on the original image (where the correct label is a “cat”), and it has signiﬁcantly higher conﬁdence ( ∼ 75% ) on the perturbed image (classiﬁed as a “frog”).

Original: The results are collected in Figure 1, with a frog image for comparison.
Revised: The results are collected in Figure 1, with a frog image for comparison.

Original: In this case, it seems that it is the vagueness of the signals/data that leads to a natural difﬁculty to separate these two images into different manifolds.
Revised: In this case, it seems that it is the vagueness of the signals/data that leads to a natural difﬁculty to separate these two images into different manifolds.

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: We now discuss ramiﬁcations of our postulate, algorithms and encouraging experimental results.
Revised: We now discuss ramiﬁcations of our postulate, algorithms and encouraging experimental results.

Original: Our technical contributions.
Revised: Our technical contributions.

Original: Our ﬁrst technical contribution is the proposal to directly exploit model semantics to defend against adversarial perturbation for neural networks.
Revised: Our ﬁrst technical contribution is the proposal to directly exploit model semantics to defend against adversarial perturbation for neural networks.

Original: Our proposal is fundamentally different from previous proposals, which are based on modifying the training objectives, and hoping that the learning algorithm will “learn” to be robust using the new objectives.
Revised: Our proposal is fundamentally different from previous proposals, which are based on modifying the training objectives, and hoping that the learning algorithm will “learn” to be robust using the new objectives.

Original: [Figure]
Revised: [Figure]

Original: Our second technical contribution is concrete algorithms for the proposed semantics defenses and an empirical evaluation of these algorithms which shows early promise.

Original: Most importantly, we show that when a “good” model, such as the residual network trained by Madry et al., is armed with our proposed embedding procedure, it achieves both good generalization and almost perfect robustness.

Original: Interpretations of our results.

Original: It is clear that we cannot expect our method to work for arbitrary models (as we need assumptions about what a model has learned).

Original: On the other hand, we interpret our results as suggesting that for good models , adversarial perturbation can naturally coexist with good generalization .

Original: In fact, even if one has learned perfect behaviors over natural manifolds, adversarial perturbation may still exist in abundance .

Original: In hindsight this is somewhat not surprising: if we only train a learner using certain universe, then we cannot expect the learner to have good behavior outside of the universe, and it is easy to fool the learner if it is easy to produce points that are outside of the learner’s universe.

Original: In this sense, our work can be viewed as a post-processing reduction to reduce points that are “slightly” out of the universe back to it.

Original: We also note that if one is only interested in robustness, then the manifold assumption is not necessary.

Original: For robustness all we want from a “good model” is that it is never conﬁdently wrong on points that it does not learn.

Original: For example, consider the following “good model”:

Original: If a data point is from the training set, then the model outputs the correct label with conﬁdence 1 , otherwise it outputs a uniform distribution over labels.

Original: In other words, this model learns nothing but ﬁtting the training set.

Original: We note that in this case, the adversarial perturbation problem is only well deﬁned around the training points, and, our defense method still works, which is 1 -nearest neighbor search among the training points.

Original: As a result, the model is still perfectly robust with our method if training points are well separated.

Original: Clearly, this model will never generalize, but is only good in the sense that it satisfy our needs for “robustness.”

Original: As a result of the discussion so far, we feel that a somewhat surprising part of our work is that it does suggest that one can simultaneously achieve generalization and robustness with our method.

Original: This suggests that deep neural networks may have indeed effectively learned some underlying geometric structure, while claiming “not enough conﬁdence” outside the structure learned.

Original: Highly conﬁdent predictions on random noises.

Original: We note that several work shows that neural networks can have highly conﬁdent predictions on random noises (for example, Nguyen et al.

Original: In view of our work this is again somewhat not surprising after all: These points are essentially ones that are far from the universe the learner is asked to learn, and so we do not expect that the learning result can make any sense of those points (and thus it is valid to have “divergent” behaviors there).

Original: In fact, we note that the adversarial perturbation problem is not well deﬁned even near those points.