Original: T HE M ANIFOLD A SSUMPTION AND D EFENSES A GAINST A DVERSARIAL P ERTURBATIONS
Revised: T HE M ANIFOLD A SSUMPTION AND D EFENSES A GAINST A DVERSARIAL P ERTURBATIONS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: In the adversarial-perturbation problem of neural networks, an adversary starts with a neural network model F and a point x that F classiﬁes correctly, and applies a small perturbation to x to produce another point x (cid:48) that F classiﬁes incorrectly .
Revised: In the adversarial-perturbation problem of neural networks, an adversary starts with a neural network model F and a point x that F classiﬁes correctly, and applies a small perturbation to x to produce another point x (cid:48) that F classiﬁes incorrectly .

Original: In this paper, we propose taking into account the inherent conﬁdence information produced by models when studying adversarial perturbations, where a natural measure of “conﬁdence” is (cid:107) F ( x ) (cid:107) ∞ (i.e. how conﬁdent F is about its prediction?).
Revised: In this paper, we propose taking into account the inherent conﬁdence information produced by models when studying adversarial perturbations, where a natural measure of “conﬁdence” is (cid:107) F ( x ) (cid:107) ∞ (i.e. how conﬁdent F is about its prediction?).

Original: Motivated by a thought experiment based on the manifold assumption, we propose a “goodness property” of models which states that conﬁdent regions of a good model should be well separated .
Revised: Motivated by a thought experiment based on the manifold assumption, we propose a “goodness property” of models which states that conﬁdent regions of a good model should be well separated .

Original: We give formalizations of this property and examine existing robust training objectives in view of them.
Revised: We give formalizations of this property and examine existing robust training objectives in view of them.

Original: Interestingly, we ﬁnd that a recent objective by Madry et al.
Revised: Interestingly, we ﬁnd that a recent objective by Madry et al.

Original: encourages training a model that satisﬁes well our formal version of the goodness property, but has a weak control of points that are wrong but with low conﬁdence.
Revised: encourages training a model that satisﬁes well our formal version of the goodness property, but has a weak control of points that are wrong but with low conﬁdence.

Original: However, if Madry et al.
Revised: However, if Madry et al.

Original: ’s model is indeed a good solution to their objective, then good and bad points are now distinguishable and we can try to embed uncertain points back to the closest conﬁdent region to get (hopefully) correct predictions.
Revised: ’s model is indeed a good solution to their objective, then good and bad points are now distinguishable and we can try to embed uncertain points back to the closest conﬁdent region to get (hopefully) correct predictions.

Original: We thus propose embedding objectives and algorithms, and perform an empirical study using this method.
Revised: We thus propose embedding objectives and algorithms, and perform an empirical study using this method.

Original: Our experimental results are encouraging: Madry et al.
Revised: Our experimental results are encouraging: Madry et al.

Original: ’s model wrapped with our embedding procedure achieves almost perfect success rate in defending against attacks that the base model fails on, while retaining good generalization behavior.
Revised: ’s model wrapped with our embedding procedure achieves almost perfect success rate in defending against attacks that the base model fails on, while retaining good generalization behavior.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: In the adversarial perturbation problem of neural networks, an adversary starts with a neural network model F and a point x that F classiﬁes correctly (we assume that F ends with a softmax layer, which is common in the literature), and applies a small perturbation to x to produce another point x (cid:48) that F classiﬁes incorrectly .
Revised: In the adversarial perturbation problem of neural networks, an adversary starts with a neural network model F and a point x that F classiﬁes correctly (we assume that F ends with a softmax layer, which is common in the literature), and applies a small perturbation to x to produce another point x (cid:48) that F classiﬁes incorrectly .

Original: Szegedy et al.
Revised: Szegedy et al.

Original: (2013) ﬁrst noticed the vulnerability of existing (deep) neural networks to adversarial perturbations, which is a somewhat surprising phenomenon given the great generalization capability of these networks.
Revised: (2013) ﬁrst noticed the vulnerability of existing (deep) neural networks to adversarial perturbations, which is a somewhat surprising phenomenon given the great generalization capability of these networks.

Original: Since then, a line of research (see, for example, Goodfellow et al.
Revised: Since then, a line of research (see, for example, Goodfellow et al.

Original: (2014); Papernot et al.
Revised: (2014); Papernot et al.

Original: (2016b)
Revised: (2016b)

Original: ; Miyato et al.
Revised: ; Miyato et al.

Original: ; Madry et al.
Revised: ; Madry et al.

Original: (2017)) has been devoted to harden neural networks against adversarial perturbation.
Revised: (2017)) has been devoted to harden neural networks against adversarial perturbation.

Original: However, while modest progress has been made, until now there is still a large gap in successfully defending against more advanced attacks, such as attack by Carlini & Wagner (2017).
Revised: However, while modest progress has been made, until now there is still a large gap in successfully defending against more advanced attacks, such as attack by Carlini & Wagner (2017).

Original: In this paper, we propose taking into account the inherent conﬁdence information produced by models when studying adversarial perturbations.
Revised: In this paper, we propose taking into account the inherent conﬁdence information produced by models when studying adversarial perturbations.

Original: To this end, a natural measure of conﬁdence is (cid:107) F ( x ) (cid:107) ∞ (i.e., how conﬁdent F is about its prediction?)
Revised: To this end, a natural measure of conﬁdence is (cid:107) F ( x ) (cid:107) ∞ (i.e., how conﬁdent F is about its prediction?)

Original: We motivate this consideration with a thought experiment based on the manifold assumption (Zhu & Goldberg (2009); man) that is commonly made in unsupervised and semi-supervised learning, which states that natural data points lie on (or near to) separate low dimensional manifolds for different classes.
Revised: We motivate this consideration with a thought experiment based on the manifold assumption (Zhu & Goldberg (2009); man) that is commonly made in unsupervised and semi-supervised learning, which states that natural data points lie on (or near to) separate low dimensional manifolds for different classes.

Original: Essentially, if one believes that deep neural networks learn to approximate well these low dimensional manifolds, then an ideal model should have the property that it can conﬁdently distinguish points from natural manifolds as they are well separated due to the assumption.
Revised: Essentially, if one believes that deep neural networks learn to approximate well these low dimensional manifolds, then an ideal model should have the property that it can conﬁdently distinguish points from natural manifolds as they are well separated due to the assumption.

Original: Moreover, since the learner never sees points that are far away from the natural manifolds, an ideal model should not claim conﬁdence there.
Revised: Moreover, since the learner never sees points that are far away from the natural manifolds, an ideal model should not claim conﬁdence there.

Original: Taking this perspective, we propose a goodness property which states that conﬁdent regions of a good model should be well separated .
Revised: Taking this perspective, we propose a goodness property which states that conﬁdent regions of a good model should be well separated .

Original: We give formalizations of this property and examine existing robust training objectives in view of them.
Revised: We give formalizations of this property and examine existing robust training objectives in view of them.

Original: Interestingly, we ﬁnd that a recent objective function by Madry et al.
Revised: Interestingly, we ﬁnd that a recent objective function by Madry et al.

Original: (2017) encourages training a model that satisﬁes well our formal version of the goodness property, in the sense that high-conﬁdence predictions of different classes are well separated.
Revised: (2017) encourages training a model that satisﬁes well our formal version of the goodness property, in the sense that high-conﬁdence predictions of different classes are well separated.

Original: On the other hand, our analysis also indicates that there could be many points with wrong but low-conﬁdence predictions.
Revised: On the other hand, our analysis also indicates that there could be many points with wrong but low-conﬁdence predictions.

Original: Therefore, if Madry et al.
Revised: Therefore, if Madry et al.

Original: ’s model is indeed a good solution to their objective, then we can distinguish between good and bad points with conﬁdence, and try to embed a low-conﬁdence point back to conﬁdent regions to get (hopefully) the correct prediction.
Revised: ’s model is indeed a good solution to their objective, then we can distinguish between good and bad points with conﬁdence, and try to embed a low-conﬁdence point back to conﬁdent regions to get (hopefully) the correct prediction.

Original: We propose two embedding objectives: (1) δ -Most Conﬁdent Neighbor ( MCN δ ), where MCN δ ( F, x ) = arg max z ∈ N ( x,δ ) (cid:107) F ( z ) (cid:107) ∞ for a point x , a radius parameter δ > 0 , and N ( x, δ ) the δ -neighborhood around x .
Revised: We propose two embedding objectives: (1) δ -Most Conﬁdent Neighbor ( MCN δ ), where MCN δ ( F, x ) = arg max z ∈ N ( x,δ ) (cid:107) F ( z ) (cid:107) ∞ for a point x , a radius parameter δ > 0 , and N ( x, δ ) the δ -neighborhood around x .

Original: (2) p -Nearest Conﬁdent Neighbor ( NCN p ), where NCN p ( F, x ) =
Revised: (2) p -Nearest Conﬁdent Neighbor ( NCN p ), where NCN p ( F, x ) =

Original: arg min z (cid:107) z − x (cid:107) subject to (cid:107) F ( z ) (cid:107) ∞ ≥ p , for a point x and a conﬁdence parameter p ∈ (0 , 1) .
Revised: arg min z (cid:107) z − x (cid:107) subject to (cid:107) F ( z ) (cid:107) ∞ ≥ p , for a point x and a conﬁdence parameter p ∈ (0 , 1) .

Original: With these, the end to end predictions become F ( MCN δ ( F, x )) and F ( NCN p ( F, x )) .
Revised: With these, the end to end predictions become F ( MCN δ ( F, x )) and F ( NCN p ( F, x )) .

Original: We note that these objectives are semantic :
Revised: We note that these objectives are semantic :

Original: They fail only when the model has a conﬁdent but wrong prediction in the neighborhood.
Revised: They fail only when the model has a conﬁdent but wrong prediction in the neighborhood.

Original: We perform an empirical study over CIFAR10.
Revised: We perform an empirical study over CIFAR10.

Original: We ﬁrst empirically validate that Madry et al.
Revised: We ﬁrst empirically validate that Madry et al.

Original: ’s model is better, in view of our goodness property, than models trained without a robustness objective.
Revised: ’s model is better, in view of our goodness property, than models trained without a robustness objective.

Original: We then give end-to-end defense results based on our method.
Revised: We then give end-to-end defense results based on our method.

Original: Speciﬁcally, we propose using gradient based optimization, such as Carlini-Wagner attacks Carlini & Wagner (2017) (which, however, are now used for defense) to solve MCN δ or NCN p .
Revised: Speciﬁcally, we propose using gradient based optimization, such as Carlini-Wagner attacks Carlini & Wagner (2017) (which, however, are now used for defense) to solve MCN δ or NCN p .

Original: Our empirical results are encouraging: (1)
Revised: Our empirical results are encouraging: (1)

Original: It achieves almost perfect success rate in defending against attacks that the base model fails on, and (2) It also retains the good generalization behavior of the base model.
Revised: It achieves almost perfect success rate in defending against attacks that the base model fails on, and (2) It also retains the good generalization behavior of the base model.

Original: The rest of the paper is organized as follows: We ﬁrst discuss important prior work in Sectionand some preliminaries in Section 3.
Revised: The rest of the paper is organized as follows: We ﬁrst discuss important prior work in Sectionand some preliminaries in Section 3.

Original: Then Section 4 proposes the goodness property, and examines
Revised: Then Section 4 proposes the goodness property, and examines

Original: Madry et al.
Revised: Madry et al.

Original: ’s robust training objective function in view of the property.
Revised: ’s robust training objective function in view of the property.

Original: We then present embedding objectives and algorithms for handling low-conﬁdence points in Section 5.
Revised: We then present embedding objectives and algorithms for handling low-conﬁdence points in Section 5.

Original: Section 6 performs an empirical study where we validate that Madry et al.
Revised: Section 6 performs an empirical study where we validate that Madry et al.

Original: ’s robust model satisﬁes well our goodness property, and then give defense results for our technique.
Revised: ’s robust model satisﬁes well our goodness property, and then give defense results for our technique.

Original: Section 7 concludes with discussions on implications of our method.
Revised: Section 7 concludes with discussions on implications of our method.

Original:  P RIOR W ORK
Revised:  P RIOR W ORK

Original: Szegedy et al.
Revised: Szegedy et al.

Original: ﬁrst observed the susceptibility of deep neural networks to adversarial perturbations.
Revised: ﬁrst observed the susceptibility of deep neural networks to adversarial perturbations.

Original: Since then, a large body of work have been devoted to studying hardening neural networks for this problem (a subset of work in this direction is Goodfellow et al.
Revised: Since then, a large body of work have been devoted to studying hardening neural networks for this problem (a subset of work in this direction is Goodfellow et al.

Original: Papernot et al.
Revised: Papernot et al.

Original: (2016b)
Revised: (2016b)

Original: ; Miyato et al.
Revised: ; Miyato et al.

Original: Simultaneously, another line of work have been devoted to devise more effective or efﬁcient attacks (a small set of work in this direction is Moosavi-Dezfooli et al.
Revised: Simultaneously, another line of work have been devoted to devise more effective or efﬁcient attacks (a small set of work in this direction is Moosavi-Dezfooli et al.

Original: (2016); Papernot et al.
Revised: (2016); Papernot et al.

Original: (2016a)
Revised: (2016a)

Original: ; Carlini & Wagner (2017)).
Revised: ; Carlini & Wagner (2017)).

Original: Unfortunately, there still seems to be a large gap for the defense methods to defend against more sophisticated attacks, such as CarliniWagner attacks Carlini & Wagner (2017).
Revised: Unfortunately, there still seems to be a large gap for the defense methods to defend against more sophisticated attacks, such as CarliniWagner attacks Carlini & Wagner (2017).

Original: For example, while the recent robust residual network constructed by Madry et al.
Revised: For example, while the recent robust residual network constructed by Madry et al.

Original: (2017) achieves encouraging robustness results on MNIST, on CIFAR the accuracy against a strong adversary, such as attacks by Carlini & Wagner (2017), can be as low as 45 .
Revised: (2017) achieves encouraging robustness results on MNIST, on CIFAR the accuracy against a strong adversary, such as attacks by Carlini & Wagner (2017), can be as low as 45 .

Original: A dominant defense approach is to add to the training objective function an appropriate robustness component to improve robustness of the resulting model.
Revised: A dominant defense approach is to add to the training objective function an appropriate robustness component to improve robustness of the resulting model.

Original: Our method is fundamentally different and tries to complement training-based method by exploiting fundamental structures a model has learned.
Revised: Our method is fundamentally different and tries to complement training-based method by exploiting fundamental structures a model has learned.

Original: Our method has some remote similarities with a recent proposal by Lu et al.
Revised: Our method has some remote similarities with a recent proposal by Lu et al.

Original: (2017), which tries to exploit features deep in the network (that is “semantics” of a model) to detect adversarial examples.
Revised: (2017), which tries to exploit features deep in the network (that is “semantics” of a model) to detect adversarial examples.

Original: However, beyond detecting adversarial attacks, our work takes a step further to compute correct prediction on the adversarial examples.
Revised: However, beyond detecting adversarial attacks, our work takes a step further to compute correct prediction on the adversarial examples.

Original: Our work is inspired by a recent line of research which demonstrates that neural networks are effective devices to represent low dimensional manifold data (for example, Rifai et al.
Revised: Our work is inspired by a recent line of research which demonstrates that neural networks are effective devices to represent low dimensional manifold data (for example, Rifai et al.

Original: (2011); Shaham et al.
Revised: (2011); Shaham et al.

Original: ; Basri & Jacobs (2016)).
Revised: ; Basri & Jacobs (2016)).

Original: Speciﬁcally, in Basri & Jacobs (2016), the authors studied a speciﬁc type of manifolds called monotonic chains.
Revised: Speciﬁcally, in Basri & Jacobs (2016), the authors studied a speciﬁc type of manifolds called monotonic chains.

Original: They showed that neural networks can effectively represent data that both fall on, and near the manifolds (see “ Section 4.
Revised: They showed that neural networks can effectively represent data that both fall on, and near the manifolds (see “ Section 4.

Original: Error Analysis ” of the corresponding paper).
Revised: Error Analysis ” of the corresponding paper).

Original: While their network is not trained but is speciﬁcally constructed to ﬁt the data, we believe that it has provided evidence that deep networks can extract intrinsic geometric structures.
Revised: While their network is not trained but is speciﬁcally constructed to ﬁt the data, we believe that it has provided evidence that deep networks can extract intrinsic geometric structures.

Original: To this end, one may interpret our work as arguing that if manifold assumption indeed holds, and the model approximates natural manifolds well, then it may not be surprising to see the coexistence of good generalization and adversarial perturbations.
Revised: To this end, one may interpret our work as arguing that if manifold assumption indeed holds, and the model approximates natural manifolds well, then it may not be surprising to see the coexistence of good generalization and adversarial perturbations.

Original: In particular, one may expect to see many wrong predictions though with low conﬁdence.
Revised: In particular, one may expect to see many wrong predictions though with low conﬁdence.

Original:  P RELIMINARIES
Revised:  P RELIMINARIES

Original: As in existing work, such as Carlini & Wagner (2017); Papernot et al.
Revised: As in existing work, such as Carlini & Wagner (2017); Papernot et al.

Original: (2016b), we deﬁne F to be a neural network after the softmax layer.
Revised: (2016b), we deﬁne F to be a neural network after the softmax layer.

Original: With this notation, the ﬁnal classiﬁcation is then C F ( x ) =
Revised: With this notation, the ﬁnal classiﬁcation is then C F ( x ) =

Original: arg max i F ( x ) i where F ( x ) i gives the conﬁdence of the network in classifying x for class
Revised: arg max i F ( x ) i where F ( x ) i gives the conﬁdence of the network in classifying x for class

Original: i .
Revised: i .

Original: We use Z ( x ) to denote part of F except the softmax layer.
Revised: We use Z ( x ) to denote part of F except the softmax layer.

Original: That is, Z ( x ) computes the logits to be fed into the softmax function.
Revised: That is, Z ( x ) computes the logits to be fed into the softmax function.

Original: We use C to denote the class of all labels.
Revised: We use C to denote the class of all labels.

Original: We need the following deﬁnitions:
Revised: We need the following deﬁnitions:

Original: Deﬁnition 1 ( p -conﬁdent point and p -conﬁdent region) .
Revised: Deﬁnition 1 ( p -conﬁdent point and p -conﬁdent region) .

Original: Let p ∈
Revised: Let p ∈

Original: [0 , 1] be a parameter, F be a model, and l ∈ C .
Revised: [0 , 1] be a parameter, F be a model, and l ∈ C .

Original: A point x is p -conﬁdent for label l if F ( x ) l ≥ p .
Revised: A point x is p -conﬁdent for label l if F ( x ) l ≥ p .

Original: Further, the p -conﬁdent region of F for label l is M pl = { x | F ( x ) l ≥ p } .
Revised: Further, the p -conﬁdent region of F for label l is M pl = { x | F ( x ) l ≥ p } .

Original: That is M l is the set of points where F has conﬁdence at least p for label l .
Revised: That is M l is the set of points where F has conﬁdence at least p for label l .

Original: Deﬁnition 2.
Revised: Deﬁnition 2.

Original: Let d ( · , · ) be a distance metric between points, and M 1 , M 2 be two sets of points, the distance between M 1 and M 2 , d
Revised: Let d ( · , · ) be a distance metric between points, and M 1 , M 2 be two sets of points, the distance between M 1 and M 2 , d

Original: ( M 1 , M 2 ) is deﬁned to be d ( M 1 , M 2 ) =
Revised: ( M 1 , M 2 ) is deﬁned to be d ( M 1 , M 2 ) =

Original: inf x ∈M 1 ,y ∈M 2 d ( x, y ) .
Revised: inf x ∈M 1 ,y ∈M 2 d ( x, y ) .

Original: For example, a common distance metric used in studying adversarial perturbations is d ( x, y )
Revised: For example, a common distance metric used in studying adversarial perturbations is d ( x, y )

Original: = (cid:107) x − y (cid:107) ∞ where (cid:107) · (cid:107) ∞ is the inﬁnity norm of a vector.
Revised: = (cid:107) x − y (cid:107) ∞ where (cid:107) · (cid:107) ∞ is the inﬁnity norm of a vector.

Original: Therefore d ( M 1 , M 2 ) is the inﬁmum of (cid:107) x − y (cid:107) ∞ for x ∈ M 1 , y ∈ M 2 .
Revised: Therefore d ( M 1 , M 2 ) is the inﬁmum of (cid:107) x − y (cid:107) ∞ for x ∈ M 1 , y ∈ M 2 .

Original: Deﬁnition 3 (Cross entropy) .
Revised: Deﬁnition 3 (Cross entropy) .

Original: Let p, q be two discrete probability distributions, the cross entropy between p, q is H( p, q ) =
Revised: Let p, q be two discrete probability distributions, the cross entropy between p, q is H( p, q ) =

Original: − (cid:80)
Revised: − (cid:80)

Original: i p
Revised: i p

Original: i log q i , where the log is base 2 .
Revised: i log q i , where the log is base 2 .

Original:  G OODNESS P ROPERTY AND M ADRY ET AL .’ S O BJECTIVE
Revised:  G OODNESS P ROPERTY AND M ADRY ET AL .’ S O BJECTIVE

Original: This section develops our main arguments.
Revised: This section develops our main arguments.

Original: In particular:
Revised: In particular:

Original: • We propose a goodness property of models which states that conﬁdent regions of a good model should be well separated.
Revised: • We propose a goodness property of models which states that conﬁdent regions of a good model should be well separated.

Original: • We examine existing robust training objectives and demonstrate that an objective function of Madry et al.
Revised: • We examine existing robust training objectives and demonstrate that an objective function of Madry et al.

Original: (2017) encourages very good separation for p -conﬁdent points with large p , but has a weak control of points that are wrong but with low conﬁdence.
Revised: (2017) encourages very good separation for p -conﬁdent points with large p , but has a weak control of points that are wrong but with low conﬁdence.

Original: Goodness Property .
Revised: Goodness Property .

Original: Manifold assumption in unsupervised and semi-supervised learning states that natural data points lie on (or near to) separate low dimensional manifolds for different classes.
Revised: Manifold assumption in unsupervised and semi-supervised learning states that natural data points lie on (or near to) separate low dimensional manifolds for different classes.

Original: Under this assumption what would an ideal model look like?
Revised: Under this assumption what would an ideal model look like?

Original: Clearly, we would expect that an ideal model can conﬁdently classify points from the manifolds, while not claiming conﬁdence for points that are far away from those manifold.
Revised: Clearly, we would expect that an ideal model can conﬁdently classify points from the manifolds, while not claiming conﬁdence for points that are far away from those manifold.

Original: Therefore, we propose the following goodness property
Revised: Therefore, we propose the following goodness property

Original: Conﬁdent regions of a good model should be well separated.
Revised: Conﬁdent regions of a good model should be well separated.

Original: The following deﬁnition captures well-separatedness in a strict sense,
Revised: The following deﬁnition captures well-separatedness in a strict sense,

Original: Deﬁnition 4 ( ( p, δ ) -separation) .
Revised: Deﬁnition 4 ( ( p, δ ) -separation) .

Original: Let p ∈
Revised: Let p ∈

Original: [0 , 1] , δ ≥ 0 , and d ( · , · ) be a distance metric.
Revised: [0 , 1] , δ ≥ 0 , and d ( · , · ) be a distance metric.

Original: F is said to have ( p, δ ) -separation
Revised: F is said to have ( p, δ ) -separation

Original: if
Revised: if

Original: [Equation]
Revised: [Equation]

Original: where M pl and M pl (cid:48) are p -conﬁdent regions for labels l, l (cid:48) , respectively.
Revised: where M pl and M pl (cid:48) are p -conﬁdent regions for labels l, l (cid:48) , respectively.

Original: It is also natural to consider the following probabilistic version of Deﬁnition 4,
Revised: It is also natural to consider the following probabilistic version of Deﬁnition 4,

Original: Deﬁnition 5 ( ( p, q, δ ) -separation) .
Revised: Deﬁnition 5 ( ( p, q, δ ) -separation) .

Original: Let D be a data generating distribution, p, q ∈ [0 , 1] , δ ≥ 0 , and d ( · , · ) be a distance metric.
Revised: Let D be a data generating distribution, p, q ∈ [0 , 1] , δ ≥ 0 , and d ( · , · ) be a distance metric.

Original: F is said to have ( p, q, δ ) -separation
Revised: F is said to have ( p, q, δ ) -separation

Original: if
Revised: if

Original: [Equation]
Revised: [Equation]

Original: ≤ q
Revised: ≤ q

Original: [Equation]
Revised: [Equation]

Original: where N ( x, δ ) =
Revised: where N ( x, δ ) =

Original: { x (cid:48) | d ( x, x (cid:48) ) ≤ δ } .
Revised: { x (cid:48) | d ( x, x (cid:48) ) ≤ δ } .

Original: Note that these two deﬁnitions have some important differences: (1) Deﬁnition 4 is deﬁned over sets of points while Deﬁnition 5 is deﬁned point-wise.
Revised: Note that these two deﬁnitions have some important differences: (1) Deﬁnition 4 is deﬁned over sets of points while Deﬁnition 5 is deﬁned point-wise.

Original: Further, Deﬁnition 5 allows certain points to be “bad” in the sense that they are not separated from conﬁdent points with wrong predictions.
Revised: Further, Deﬁnition 5 allows certain points to be “bad” in the sense that they are not separated from conﬁdent points with wrong predictions.

Original: Perhaps more importantly, Deﬁnition 5 depends on the data generating distribution, but Deﬁnitionis solely a property of the model.
Revised: Perhaps more importantly, Deﬁnition 5 depends on the data generating distribution, but Deﬁnitionis solely a property of the model.

Original: We are not aware of how to train a model to satisfy Deﬁnition 4.
Revised: We are not aware of how to train a model to satisfy Deﬁnition 4.

Original: However, interestingly, we ﬁnd that existing robust training can indeed give guarantees with respect to Deﬁnition 5.
Revised: However, interestingly, we ﬁnd that existing robust training can indeed give guarantees with respect to Deﬁnition 5.

Original: We next give an analysis of this.
Revised: We next give an analysis of this.

Original: Objective function of Madry et al.
Revised: Objective function of Madry et al.

Original: We now examine objective function used in Madry et al.
Revised: We now examine objective function used in Madry et al.

Original: (2017), and prove that it encourages training a model that satisﬁes Deﬁnition 5.
Revised: (2017), and prove that it encourages training a model that satisﬁes Deﬁnition 5.

Original: Complementing this, we show that their objective has a weak control of points with low-conﬁdence but wrong predictions.
Revised: Complementing this, we show that their objective has a weak control of points with low-conﬁdence but wrong predictions.

Original: To start with, their objective function is of the form minimize ρ
Revised: To start with, their objective function is of the form minimize ρ

Original: [Equation]
Revised: [Equation]

Original: ) , where ρ
Revised: ) , where ρ

Original: [Equation]
Revised: [Equation]

Original: where D is the data generating distribution, S is set of allowed perturbations (for example S = { ∆ | (cid:107) ∆ (cid:107) ∞ ≤ δ } ), and L ( θ, x, y ) is the loss of θ on ( x, y ) .
Revised: where D is the data generating distribution, S is set of allowed perturbations (for example S = { ∆ | (cid:107) ∆ (cid:107) ∞ ≤ δ } ), and L ( θ, x, y ) is the loss of θ on ( x, y ) .

Original: Madry et al.
Revised: Madry et al.

Original: (2017) uses cross entropy loss function:
Revised: (2017) uses cross entropy loss function:

Original: [Equation]
Revised: [Equation]

Original: where F θ is the model instantiated with parameters θ , and 1 y is the indicator vector of label y .
Revised: where F θ is the model instantiated with parameters θ , and 1 y is the indicator vector of label y .

Original: Denote κ ( θ, x, y ) = max ∆ ∈S L ( θ, x + ∆ , y ) , we have the following proposition,
Revised: Denote κ ( θ, x, y ) = max ∆ ∈S L ( θ, x + ∆ , y ) , we have the following proposition,

Original: Proposition 1.
Revised: Proposition 1.

Original: If ρ ( θ ) ≤ ε , then
Revised: If ρ ( θ ) ≤ ε , then

Original: [Equation]
Revised: [Equation]

Original: ( ∃ y (cid:48) (cid:54) =
Revised: ( ∃ y (cid:48) (cid:54) =

Original: y, x (cid:48)
Revised: y, x (cid:48)

Original: ∈ x + S ) F θ ( x (cid:48) )
Revised: ∈ x + S ) F θ ( x (cid:48) )

Original: y (cid:48) ≥ p (cid:3) ≤ − ε log(1 − p )
Revised: y (cid:48) ≥ p (cid:3) ≤ − ε log(1 − p )

Original: That is, the probability of an x (cid:48) ∈ x
Revised: That is, the probability of an x (cid:48) ∈ x

Original: + S that is p -conﬁdent on a different label vanishes as p → 1 .
Revised: + S that is p -conﬁdent on a different label vanishes as p → 1 .

Original: Proof.
Revised: Proof.

Original: Let E be the event {∃ y (cid:48) (cid:54) =
Revised: Let E be the event {∃ y (cid:48) (cid:54) =

Original: y, x (cid:48)
Revised: y, x (cid:48)

Original: ∈ x + S , F θ ( x (cid:48) )
Revised: ∈ x + S , F θ ( x (cid:48) )

Original: y (cid:48) ≥ p } .
Revised: y (cid:48) ≥ p } .

Original: If E happens, then F θ ( x (cid:48) )
Revised: If E happens, then F θ ( x (cid:48) )

Original: y ≤ 1 − p , and κ ( θ, x, y ) ≥ − log(1 − p ) , therefore by Markov’s inequality 1 ,
Revised: y ≤ 1 − p , and κ ( θ, x, y ) ≥ − log(1 − p ) , therefore by Markov’s inequality 1 ,

Original: [Equation]
Revised: [Equation]

Original: The proof is complete.
Revised: The proof is complete.

Original: This immediately generates the following corollary,
Revised: This immediately generates the following corollary,

Original: Corollary 1.
Revised: Corollary 1.

Original: Let S be a region deﬁned as { ∆ | d (∆ , 0 ) ≤ δ } .
Revised: Let S be a region deﬁned as { ∆ | d (∆ , 0 ) ≤ δ } .

Original: If ρ ( θ ) ≤ ε , then the model F θ is ( p, − ε log(1 − p ) , δ ) -separated.
Revised: If ρ ( θ ) ≤ ε , then the model F θ is ( p, − ε log(1 − p ) , δ ) -separated.

Original: This indicates that even if a model is only a somewhat good solution to (1), meaning ρ ( θ ) ≤ ε for somewhat small ε , (note that in reality D is unknown and a learner can only approximate (1)), then p -conﬁdent points will be well separated (in the sense of Deﬁnition 5) as soon as p increases.
Revised: This indicates that even if a model is only a somewhat good solution to (1), meaning ρ ( θ ) ≤ ε for somewhat small ε , (note that in reality D is unknown and a learner can only approximate (1)), then p -conﬁdent points will be well separated (in the sense of Deﬁnition 5) as soon as p increases.

Original: The above proposition considers a situation where we have a conﬁdent but wrong point (with conﬁdence p ).
Revised: The above proposition considers a situation where we have a conﬁdent but wrong point (with conﬁdence p ).

Original: What about points that have wrong but low-conﬁdence predictions?
Revised: What about points that have wrong but low-conﬁdence predictions?

Original: For example, the conﬁdence on the wrong label is only 12 + ν for some tiny ν ?
Revised: For example, the conﬁdence on the wrong label is only 12 + ν for some tiny ν ?

Original: Note that by setting p = 1 / 2 we derive immediately a bound ε (much weaker than − ε/ log(1 − p ) for large p ) on the probability that bad events happen.
Revised: Note that by setting p = 1 / 2 we derive immediately a bound ε (much weaker than − ε/ log(1 − p ) for large p ) on the probability that bad events happen.

Original: Further, without further assumptions, this bound in tight so that in a worst-case sense that there exists a bad model so that bound ε is achieved.
Revised: Further, without further assumptions, this bound in tight so that in a worst-case sense that there exists a bad model so that bound ε is achieved.

Original: We have the following proposition.
Revised: We have the following proposition.

Original: Proposition 2.
Revised: Proposition 2.

Original: Let F θ be a neural network parameterized by θ , and C F θ be the classiﬁcation network.
Revised: Let F θ be a neural network parameterized by θ , and C F θ be the classiﬁcation network.

Original: If ρ ( θ ) ≤ ε , then
Revised: If ρ ( θ ) ≤ ε , then

Original: [Equation]
Revised: [Equation]

Original: Further the bound is tight.
Revised: Further the bound is tight.

Original: Proof.
Revised: Proof.

Original: Let E be the event { ( ∃ y (cid:48) (cid:54) =
Revised: Let E be the event { ( ∃ y (cid:48) (cid:54) =

Original: y, x (cid:48)
Revised: y, x (cid:48)

Original: ∈ x + S ) C F θ ( x (cid:48) )
Revised: ∈ x + S ) C F θ ( x (cid:48) )

Original: =
Revised: =

Original: y (cid:48) } .
Revised: y (cid:48) } .

Original: If E happens then F θ ( x (cid:48) )
Revised: If E happens then F θ ( x (cid:48) )

Original: y ≤(otherwise x (cid:48) will be classiﬁed as y ), and so κ ( θ, x, y ) ≥ − log(1 / 2) = 1 .
Revised: y ≤(otherwise x (cid:48) will be classiﬁed as y ), and so κ ( θ, x, y ) ≥ − log(1 / 2) = 1 .

Original: On the other hand, if E does not happen, then we can lower bound κ ( θ, x, y ) by 0 .
Revised: On the other hand, if E does not happen, then we can lower bound κ ( θ, x, y ) by 0 .

Original: Therefore
Revised: Therefore

Original: [Equation]
Revised: [Equation]

Original: Tightness follows as we can force equality for each of the inequalities.
Revised: Tightness follows as we can force equality for each of the inequalities.

Original: The proof is complete.
Revised: The proof is complete.

Original: Contrasting Proposition 1 and 2, we note the following:
Revised: Contrasting Proposition 1 and 2, we note the following:

Original: • First, let us note that the objective function 1 is deﬁned with respect to the data generating distribution D , which is unknown, and in reality we only have a training set to approximate the objective.
Revised: • First, let us note that the objective function 1 is deﬁned with respect to the data generating distribution D , which is unknown, and in reality we only have a training set to approximate the objective.

Original: Also, cross entropy function is unbounded (so in particular it can happen that ρ ( θ ) ≥ 1 ).
Revised: Also, cross entropy function is unbounded (so in particular it can happen that ρ ( θ ) ≥ 1 ).

Original: Therefore, one may only expect that ρ ( θ ) is somewhat small.
Revised: Therefore, one may only expect that ρ ( θ ) is somewhat small.

Original: • Nevertheless, even with a somewhat large ρ ( θ ) , Proposition 1 indicates that highconﬁdence predictions of different classes will be well separated as soon as conﬁdence p increases.
Revised: • Nevertheless, even with a somewhat large ρ ( θ ) , Proposition 1 indicates that highconﬁdence predictions of different classes will be well separated as soon as conﬁdence p increases.

Original: On the other hand, Proposition 2 then indicates that there may still be many points that have wrong predictions with low conﬁdence .
Revised: On the other hand, Proposition 2 then indicates that there may still be many points that have wrong predictions with low conﬁdence .

Original: Finally, if we take the manifold point of view again, then intuitively it seems harder to control low-conﬁdence points than to separate conﬁdent regions.
Revised: Finally, if we take the manifold point of view again, then intuitively it seems harder to control low-conﬁdence points than to separate conﬁdent regions.

Original: This is because natural manifolds reside in lower dimensions and so there can be a large volume of low-conﬁdence points at the conﬁdence boundary.
Revised: This is because natural manifolds reside in lower dimensions and so there can be a large volume of low-conﬁdence points at the conﬁdence boundary.

Original: This thus indicates that handling low-conﬁdence points may be the core difﬁculty in getting a good solution to (1).
Revised: This thus indicates that handling low-conﬁdence points may be the core difﬁculty in getting a good solution to (1).

Original:  E MBEDDING
Revised:  E MBEDDING

Original: Our analysis from previous section shows that there can be good models with well-separated conﬁdent regions, yet there is essentially no control for uncertain points.
Revised: Our analysis from previous section shows that there can be good models with well-separated conﬁdent regions, yet there is essentially no control for uncertain points.

Original: However, since now uncertain points are distinguishable as they have low conﬁdence, a natural plan to handle these points is to try to “embed them back” to conﬁdent regions.
Revised: However, since now uncertain points are distinguishable as they have low conﬁdence, a natural plan to handle these points is to try to “embed them back” to conﬁdent regions.

Original: The rest of this section presents objective functions and algorithms for this purpose.
Revised: The rest of this section presents objective functions and algorithms for this purpose.

Original: Objective Functions.
Revised: Objective Functions.

Original: We propose two objective functions for embedding.
Revised: We propose two objective functions for embedding.

Original: Deﬁnition 6 ( Most Conﬁdent Neighbor Objective ) .
Revised: Deﬁnition 6 ( Most Conﬁdent Neighbor Objective ) .

Original: Let F be a model, (cid:107) · (cid:107) be a norm, and δ >be a real number.
Revised: Let F be a model, (cid:107) · (cid:107) be a norm, and δ >be a real number.

Original: The δ -Most Conﬁdent Neighbor objective (or simply MCN δ objective) computes:
Revised: The δ -Most Conﬁdent Neighbor objective (or simply MCN δ objective) computes:

Original: [Equation]
Revised: [Equation]

Original: where N ( x , δ ) is the δ -ball around x with respect to (cid:107) · (cid:107) , and (cid:107) · (cid:107) ∞ denotes the L ∞ -norm.
Revised: where N ( x , δ ) is the δ -ball around x with respect to (cid:107) · (cid:107) , and (cid:107) · (cid:107) ∞ denotes the L ∞ -norm.

Original: Deﬁnition 7 ( Nearest Conﬁdent Neighbor Objective ) .
Revised: Deﬁnition 7 ( Nearest Conﬁdent Neighbor Objective ) .

Original: Let F be a model, (cid:107) · (cid:107) be a norm, and p ∈ (0 , 1) be a real number.
Revised: Let F be a model, (cid:107) · (cid:107) be a norm, and p ∈ (0 , 1) be a real number.

Original: The p - Nearest Conﬁdent Neighbor objective (or simply NCN p objective) computes:
Revised: The p - Nearest Conﬁdent Neighbor objective (or simply NCN p objective) computes:

Original: [Equation]
Revised: [Equation]

Original: where (cid:107) · (cid:107) ∞ denotes the L ∞ -norm.
Revised: where (cid:107) · (cid:107) ∞ denotes the L ∞ -norm.

Original: In short, MCN considers a δ -neighborhood of a point and use the most conﬁdent point in the neighborhood for prediction.
Revised: In short, MCN considers a δ -neighborhood of a point and use the most conﬁdent point in the neighborhood for prediction.

Original: On the other hand, NCN considers the nearest neighbor that achieves certain conﬁdence threshold.
Revised: On the other hand, NCN considers the nearest neighbor that achieves certain conﬁdence threshold.

Original: Model Shell.
Revised: Model Shell.

Original: From an algorithmic point of view, our approach essentially tries to wrap around a learned model with a “shell” to handle points that are “slightly” out of the conﬁdent regions it has learned.
Revised: From an algorithmic point of view, our approach essentially tries to wrap around a learned model with a “shell” to handle points that are “slightly” out of the conﬁdent regions it has learned.

Original: There are many possible shells for this purpose.
Revised: There are many possible shells for this purpose.

Original: For example, a valid such shell is weighted majority vote.
Revised: For example, a valid such shell is weighted majority vote.

Original: We thus begin by giving some general deﬁnitions.
Revised: We thus begin by giving some general deﬁnitions.

Original: Deﬁnition 8 ( Model Shell ) .
Revised: Deﬁnition 8 ( Model Shell ) .

Original: Let F be a model.
Revised: Let F be a model.

Original: A model shell G is an algorithm that takes as its input F and a point x ∈ R d to classify, and outputs a softmax layer
Revised: A model shell G is an algorithm that takes as its input F and a point x ∈ R d to classify, and outputs a softmax layer

Original: G ( F, x ) .
Revised: G ( F, x ) .

Original: In this paper we only consider the case where F is a model that is chosen apriori and ﬁxed.
Revised: In this paper we only consider the case where F is a model that is chosen apriori and ﬁxed.

Original: In this case, the resulting model for classiﬁcation is G ( F, · ) , which maps a feature vector to a softmax layer.
Revised: In this case, the resulting model for classiﬁcation is G ( F, · ) , which maps a feature vector to a softmax layer.

Original: In this scenario, we say that F is the base model of the model shell G , and denote G ( F, · ) as G [ F ] .
Revised: In this scenario, we say that F is the base model of the model shell G , and denote G ( F, · ) as G [ F ] .

Original: Model shell captures the intuition that we want to wrap around an existing model with another algorithm that can exploit semantics of the base model.
Revised: Model shell captures the intuition that we want to wrap around an existing model with another algorithm that can exploit semantics of the base model.

Original: In this paper, we only examine model shells with a restricted structure, which we call factor model shells .
Revised: In this paper, we only examine model shells with a restricted structure, which we call factor model shells .

Original: We deﬁne search factor ,
Revised: We deﬁne search factor ,

Original: Deﬁnition 9 ( Search Factor ) .
Revised: Deﬁnition 9 ( Search Factor ) .

Original: A search factor H is an algorithm that takes as its input a model F , and a point x ∈ R d to classify, and output another point x (cid:48) ∈ R d .
Revised: A search factor H is an algorithm that takes as its input a model F , and a point x ∈ R d to classify, and output another point x (cid:48) ∈ R d .

Original: If F is chosen and ﬁxed apriori we say that F is the base model of the search factor H , and denote H ( F, · ) as H [ F ] .
Revised: If F is chosen and ﬁxed apriori we say that F is the base model of the search factor H , and denote H ( F, · ) as H [ F ] .

Original: Deﬁnition 10 ( Factor Model Shell ) .
Revised: Deﬁnition 10 ( Factor Model Shell ) .

Original: A model shell G is said to be a factor model shell if G can be written as a composition of F ◦ H where H is a search factor.
Revised: A model shell G is said to be a factor model shell if G can be written as a composition of F ◦ H where H is a search factor.

Original: In other words, G ( F, x ) works as
Revised: In other words, G ( F, x ) works as

Original: G ( F, x ) ≡ ( F ◦ H [ F ])( x ) =
Revised: G ( F, x ) ≡ ( F ◦ H [ F ])( x ) =

Original: F ( H ( F, x )) .
Revised: F ( H ( F, x )) .

Original: A factor model shell captures our intuitions from MCN or NCN objectives.
Revised: A factor model shell captures our intuitions from MCN or NCN objectives.

Original: That is, we ﬁrst apply a search factor H to ﬁnd another feature point x (cid:48)
Revised: That is, we ﬁrst apply a search factor H to ﬁnd another feature point x (cid:48)

Original: =
Revised: =

Original: H ( F, x ) , and then apply the base model F on x (cid:48) to produce a ﬁnal prediction.
Revised: H ( F, x ) , and then apply the base model F on x (cid:48) to produce a ﬁnal prediction.

Original: As a result, a factor model shell for MCN δ objective should compute the ﬁnal prediction as:
Revised: As a result, a factor model shell for MCN δ objective should compute the ﬁnal prediction as:

Original: [Equation]
Revised: [Equation]

Original: and accordingly for NCN p objective:
Revised: and accordingly for NCN p objective:

Original: [Equation]
Revised: [Equation]

Original: We note the following:
Revised: We note the following:

Original: • Exact computations of (4) and (5) provide semantic defenses.
Revised: • Exact computations of (4) and (5) provide semantic defenses.

Original: That is, the ﬁnal prediction is wrong only if the model has a conﬁdent but wrong prediction in the neighborhood.
Revised: That is, the ﬁnal prediction is wrong only if the model has a conﬁdent but wrong prediction in the neighborhood.

Original: Furthermore, our discussion from Section 4 indicates that for good models, wrong predictions should also have low conﬁdence .
Revised: Furthermore, our discussion from Section 4 indicates that for good models, wrong predictions should also have low conﬁdence .

Original: Therefore, embedding should be effective in ﬁxing low-conﬁdence errors of good models.
Revised: Therefore, embedding should be effective in ﬁxing low-conﬁdence errors of good models.

Original: • We note that the resulting factor model shells, ( F ◦ MCN δ ) and ( F ◦ NCN p ) are nondifferentiable as MCN δ ( F, x ) and NCN p ( F, x ) are non-differentiable with respect to x (they are argmin or argmax over a ball).
Revised: • We note that the resulting factor model shells, ( F ◦ MCN δ ) and ( F ◦ NCN p ) are nondifferentiable as MCN δ ( F, x ) and NCN p ( F, x ) are non-differentiable with respect to x (they are argmin or argmax over a ball).

Original: This renders existing attacks fail to work directly to attack F ◦ MCN δ or F ◦ NCN p , since they all require the assumption that the output softmax layer is differentiable with respect to the input feature vector x 2 As a result, this forces attacks to either attack the base model F and then transfer attacks to the model shell, or has to ﬁnd some differentiable approximations or derivative-free optimization to attack MCN δ and NCN p directly.
Revised: This renders existing attacks fail to work directly to attack F ◦ MCN δ or F ◦ NCN p , since they all require the assumption that the output softmax layer is differentiable with respect to the input feature vector x 2 As a result, this forces attacks to either attack the base model F and then transfer attacks to the model shell, or has to ﬁnd some differentiable approximations or derivative-free optimization to attack MCN δ and NCN p directly.

Original: Algorithms.
Revised: Algorithms.

Original: We now give one concrete algorithms to solve (2) and (3).
Revised: We now give one concrete algorithms to solve (2) and (3).

Original: To start with, we note that the optimization for solving MCN δ (2) is nothing but for each label t ∈ C we try to ﬁnd
Revised: To start with, we note that the optimization for solving MCN δ (2) is nothing but for each label t ∈ C we try to ﬁnd

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where F ( x (cid:48) ) t denote the coordinate of F ( x (cid:48) ) atlabel t .
Revised: at

Original: This objective is differentiable in x (cid:48) as F is differentiable in x (cid:48) .
Revised: This objective is differentiable in x (cid:48) as F is differentiable in x (cid:48) .

Original: We note that such a ﬁrst-order assumption is natural for neural networks because neural networks are typically composition of differentiable functions.
Revised: We note that such a ﬁrst-order assumption is natural for neural networks because neural networks are typically composition of differentiable functions.

Original: With ﬁrst order information, attacks can “relatively accu- rately” follow the gradient direction to modify the image feature so as to increase the conﬁdence of the model in incorrect labels and thus produce adversarial perturbations.
Revised: With ﬁrst order information, attacks can “relatively accu- rately” follow the gradient direction to modify the image feature so as to increase the conﬁdence of the model in incorrect labels and thus produce adversarial perturbations.

Original: and then compute z t ∗ for t ∗ = arg max t F ( z t ) t .
Revised: and then compute z t ∗ for t ∗ = arg max t F ( z t ) t .

Original: Now, however, we can solve (6) using any preferred gradient-based optimization (for example, projected gradient descent Nocedal & Wright (2006)).
Revised: Now, however, we can solve (6) using any preferred gradient-based optimization (for example, projected gradient descent Nocedal & Wright (2006)).

Original: This thus gives the following factor model shell:
Revised: This thus gives the following factor model shell:

Original: Algorithm 1 Solving MCN δ objective
Revised: Algorithm 1 Solving MCN δ objective

Original: We note that solving (6) is similar to a targeted adversarial attack which tries to modify x so as to increase the conﬁdence on label t .
Revised: We note that solving (6) is similar to a targeted adversarial attack which tries to modify x so as to increase the conﬁdence on label t .

Original: Therefore an adversarial attack , such as strong attack proposed in Carlini & Wagner (2017), can be used as O here.
Revised: Therefore an adversarial attack , such as strong attack proposed in Carlini & Wagner (2017), can be used as O here.

Original: Another concern is that gradient-based optimization may only ﬁnd local optimal in the neighborhood.
Revised: Another concern is that gradient-based optimization may only ﬁnd local optimal in the neighborhood.

Original: However, we note that global optimal may not be needed for our method to work, as long as local minima of the correct class are “separated” from local minima of incorrect classes.
Revised: However, we note that global optimal may not be needed for our method to work, as long as local minima of the correct class are “separated” from local minima of incorrect classes.

Original: More speciﬁcally, let l be the correct class, and
Revised: More speciﬁcally, let l be the correct class, and

Original: S l = { z : z ∈ N ( x, δ ) , z is a local maximal of (cid:107) F ( x ) (cid:107) ∞ , (cid:107) F ( z ) (cid:107) ∞
Revised: S l = { z : z ∈ N ( x, δ ) , z is a local maximal of (cid:107) F ( x ) (cid:107) ∞ , (cid:107) F ( z ) (cid:107) ∞

Original: =
Revised: =

Original: F ( z ) l } and
Revised: F ( z ) l } and

Original: S − l = { z : z ∈ N ( x, δ ) , z is a local maximal of (cid:107) F ( x ) (cid:107) ∞ , (cid:107) F ( z ) (cid:107) ∞ (cid:54)
Revised: S − l = { z : z ∈ N ( x, δ ) , z is a local maximal of (cid:107) F ( x ) (cid:107) ∞ , (cid:107) F ( z ) (cid:107) ∞ (cid:54)

Original: = F ( z ) l } then local optimal works as soon as inf z ∈S l (cid:107) F ( z ) (cid:107) ∞ > sup y ∈S − l (cid:107) F ( y ) (cid:107) ∞ .
Revised: = F ( z ) l } then local optimal works as soon as inf z ∈S l (cid:107) F ( z ) (cid:107) ∞ > sup y ∈S − l (cid:107) F ( y ) (cid:107) ∞ .

Original: We can also take a similar path to solve NCN p objective with standard numeric optimization.
Revised: We can also take a similar path to solve NCN p objective with standard numeric optimization.

Original: Note that (cid:107) F ( z ) (cid:107) ∞ ≥ p is equivalent to ∃ t, F ( z ) t ≥ p .
Revised: Note that (cid:107) F ( z ) (cid:107) ∞ ≥ p is equivalent to ∃ t, F ( z ) t ≥ p .

Original: Therefore, we can solve
Revised: Therefore, we can solve

Original: [Equation]
Revised: [Equation]

Original: for every t , and then compute z t ∗ where t ∗ = arg min t (cid:107) z t − x (cid:107) .
Revised: for every t , and then compute z t ∗ where t ∗ = arg min t (cid:107) z t − x (cid:107) .

Original: By a similar reasoning as above, z t ∗ is the solution to problem 3.
Revised: By a similar reasoning as above, z t ∗ is the solution to problem 3.

Original: This thus gives algorithm 2 for NCN p .
Revised: This thus gives algorithm 2 for NCN p .

Original: Algorithm 2 Solving NCN p objective.
Revised: Algorithm 2 Solving NCN p objective.

Original: Input: x a feature vector, p > 0 a real parameter, a base model F , any gradient-based optimization algorithm O to solve the constrained optimization problem deﬁned in (7).
Revised: Input: x a feature vector, p > 0 a real parameter, a base model F , any gradient-based optimization algorithm O to solve the constrained optimization problem deﬁned in (7).

Original: 1: function NCNOracleShell ( x , p, F ) 2: for t ∈ C do 3: z t ← O ( x , F, t ) 4: return z t ∗ where t ∗ =
Revised: 1: function NCNOracleShell ( x , p, F ) 2: for t ∈ C do 3: z t ← O ( x , F, t ) 4: return z t ∗ where t ∗ =

Original: arg min t (cid:107) z t − x (cid:107)
Revised: arg min t (cid:107) z t − x (cid:107)

Original: Solving (7), however, is not trivial.
Revised: Solving (7), however, is not trivial.

Original: A standard method is the augmented Lagrangian method Bertsekas (1996) which turns the problem into solving a series of unconstrained optimization.
Revised: A standard method is the augmented Lagrangian method Bertsekas (1996) which turns the problem into solving a series of unconstrained optimization.

Original: For example, consider the following quadratic penalty function Q ( z , α, t )
Revised: For example, consider the following quadratic penalty function Q ( z , α, t )

Original: = (cid:107)
Revised: = (cid:107)

Original: z − x (cid:107) + α 2 ([ F ( z ) − p ] − ) 2 , where [ y ] − denotes max { y, 0 } .
Revised: z − x (cid:107) + α 2 ([ F ( z ) − p ] − ) 2 , where [ y ] − denotes max { y, 0 } .

Original: Then we can solve the unconstrained optimization min Q ( z , α, t ) for a series of different α , and terminates once we ﬁnd a satisfying solution.
Revised: Then we can solve the unconstrained optimization min Q ( z , α, t ) for a series of different α , and terminates once we ﬁnd a satisfying solution.

Original: The intuition is that (i) solving the unconstrained optimization is easy, (ii) by increasing the coefﬁcient α , we can force the minimizer of the penalty function closer to the feasible region F ( z ) ≥ p , and (iii) for well chosen α , the minimizer of the penalty function is close enough to that of problem 7.
Revised: The intuition is that (i) solving the unconstrained optimization is easy, (ii) by increasing the coefﬁcient α , we can force the minimizer of the penalty function closer to the feasible region F ( z ) ≥ p , and (iii) for well chosen α , the minimizer of the penalty function is close enough to that of problem 7.

Original: Other classic constrained optimization algorithms to solve this problem are interior methods and sequential quadratic programming Nocedal & Wright (2006).
Revised: Other classic constrained optimization algorithms to solve this problem are interior methods and sequential quadratic programming Nocedal & Wright (2006).

Original: We also note a recent work Bienstock & Michalka (2014) which speciﬁcally focuses on optimizing convex objective over non-convex constraints, into which problem 7 falls as well.
Revised: We also note a recent work Bienstock & Michalka (2014) which speciﬁcally focuses on optimizing convex objective over non-convex constraints, into which problem 7 falls as well.

Original:  E MPIRICAL S TUDY
Revised:  E MPIRICAL S TUDY

Original: In this section we perform an empirical study of our approach.
Revised: In this section we perform an empirical study of our approach.

Original: Note that there are three basic components in a defense with our method: (1) the base model we choose, (2) the embedding procedure we choose to wrap around the base model, and (3) the attack we choose to evaluate the defense.
Revised: Note that there are three basic components in a defense with our method: (1) the base model we choose, (2) the embedding procedure we choose to wrap around the base model, and (3) the attack we choose to evaluate the defense.

Original: Therefore we have the following key empirical questions:
Revised: Therefore we have the following key empirical questions:

Original: How well does a base model satisfy our goodness property?
Revised: How well does a base model satisfy our goodness property?

Original: How susceptible is the chosen base model to the chosen attack?
Revised: How susceptible is the chosen base model to the chosen attack?

Original: We want that the chosen attack is strong enough to break even a good base model on many points.
Revised: We want that the chosen attack is strong enough to break even a good base model on many points.

Original: How effective does our approach improve robustness of the base model?
Revised: How effective does our approach improve robustness of the base model?

Original: Ideally, we want to see a signiﬁcant improvement of robustness for a good base model.
Revised: Ideally, we want to see a signiﬁcant improvement of robustness for a good base model.

Original: Will our approach change the generalization behavior of the base model?
Revised: Will our approach change the generalization behavior of the base model?

Original: Since embedding may change predictions, we need to justify if there is little or no change in generalization.
Revised: Since embedding may change predictions, we need to justify if there is little or no change in generalization.

Original: A summary of our ﬁndings are as follows:
Revised: A summary of our ﬁndings are as follows:

Original: • We empirically validate that the model from Madry et al.
Revised: • We empirically validate that the model from Madry et al.

Original: (2017) is signiﬁcantly better than models without robustness training, according to our goodness property.
Revised: (2017) is signiﬁcantly better than models without robustness training, according to our goodness property.

Original: • We thus use Madry et al.
Revised: • We thus use Madry et al.

Original: ’s model as the base model, and attack the model using attacks from Carlini & Wagner (2017) (denoted as CW attack).
Revised: ’s model as the base model, and attack the model using attacks from Carlini & Wagner (2017) (denoted as CW attack).

Original: We use CIFAR10 Krizhevsky (2009) to evaluate the robustness and generalization.
Revised: We use CIFAR10 Krizhevsky (2009) to evaluate the robustness and generalization.

Original: As reported by Madry et al.
Revised: As reported by Madry et al.

Original: , their model is still fairly susceptible to CW attack over CIFAR10, and the accuracy against CW attack can be as low as 45 . 8% , which indicates a signiﬁcant room for improving robustness .
Revised: , their model is still fairly susceptible to CW attack over CIFAR10, and the accuracy against CW attack can be as low as 45 . 8% , which indicates a signiﬁcant room for improving robustness .

Original: • We choose MCNOracleShell (Algorithm 1) and use again CW attack but now as embedding.
Revised: • We choose MCNOracleShell (Algorithm 1) and use again CW attack but now as embedding.

Original: The resulting model shell is called CarliniWagnerShell.
Revised: The resulting model shell is called CarliniWagnerShell.

Original: We ﬁnd that embedding signiﬁcantly improves the robustness of the base model over 5 batches of points we evaluated, reducing the attack success rate from 30% to 1 .
Revised: We ﬁnd that embedding signiﬁcantly improves the robustness of the base model over 5 batches of points we evaluated, reducing the attack success rate from 30% to 1 .

Original: • Finally, embedding incurs almost no effect on generalization.
Revised: • Finally, embedding incurs almost no effect on generalization.

Original: In fact, there are
Revised: In fact, there are

Original: only three such points that the embedding procedure changes the predictions from correct to wrong.
Revised: only three such points that the embedding procedure changes the predictions from correct to wrong.

Original: We note that the model has very low conﬁdence ( ∼ 50% ) on these points, and thus are difﬁcult to defend as predicted by our theoretical analysis.
Revised: We note that the model has very low conﬁdence ( ∼ 50% ) on these points, and thus are difﬁcult to defend as predicted by our theoretical analysis.

Original: Limitations of our experiments.
Revised: Limitations of our experiments.

Original: Due to the complexity of the Madry et al.’s
Revised: Due to the complexity of the Madry et al.’s

Original: residual network model, our embedding procedure takes a long time to compute and we have only tested onpoints which the base model classiﬁes correctly (we do not evaluate robustness for test points that have incorrect predictions in the ﬁrst place).
Revised: residual network model, our embedding procedure takes a long time to compute and we have only tested onpoints which the base model classiﬁes correctly (we do not evaluate robustness for test points that have incorrect predictions in the ﬁrst place).

Original: While our current experiments are limited, they corroborate well our theoretical analysis that embedding should work well with a good model to resolve adversarial perturbation problem.
Revised: While our current experiments are limited, they corroborate well our theoretical analysis that embedding should work well with a good model to resolve adversarial perturbation problem.

Original: We have made our model and code publicly available cod, in the hope of triggering more thoughts and feedback.
Revised: We have made our model and code publicly available cod, in the hope of triggering more thoughts and feedback.

Original: Experimental Setup We use CIFAR10 Krizhevsky (2009) which consists of 60000 color images of objects where 50000 are for training and 10000 are for testing.
Revised: Experimental Setup We use CIFAR10 Krizhevsky (2009) which consists of 60000 color images of objects where 50000 are for training and 10000 are for testing.

Original: Each image is of size 32 × 32 pixels, and there are ten labels { 0 , 1 , . . .
Revised: Each image is of size 32 × 32 pixels, and there are ten labels { 0 , 1 , . . .

Original: , 9 } for classiﬁcation.
Revised: , 9 } for classiﬁcation.

Original: We use Carlini-Wager attacks ( CW attack) for two purposes: (i) as an attack to challenge the defense.
Revised: We use Carlini-Wager attacks ( CW attack) for two purposes: (i) as an attack to challenge the defense.

Original: In this case, we use the original attacks to make attacks as strong as possible.
Revised: In this case, we use the original attacks to make attacks as strong as possible.

Original: We apply non-targeted attacks (namely changing classiﬁcation is the purpose) with L ∞ norm (i.e. try to ﬁnd a perturbation of minimal L ∞ norm).
Revised: We apply non-targeted attacks (namely changing classiﬁcation is the purpose) with L ∞ norm (i.e. try to ﬁnd a perturbation of minimal L ∞ norm).

Original: To set the norm bound δ , we note that Madry et al.
Revised: To set the norm bound δ , we note that Madry et al.

Original: (2017) uses a norm bound δ = 8 . 0 .
Revised: (2017) uses a norm bound δ = 8 . 0 .

Original: Since we normalize images when applying the CW attack, we set δ = 8255 .
Revised: Since we normalize images when applying the CW attack, we set δ = 8255 .

Original: A perturbation is valid if and only if it both changes the classiﬁcation and its L ∞ norm is at most δ .
Revised: A perturbation is valid if and only if it both changes the classiﬁcation and its L ∞ norm is at most δ .

Original: (ii) as an embedding procedure for instantiating
Revised: (ii) as an embedding procedure for instantiating

Original: MCNOracleShell .
Revised: MCNOracleShell .

Original: In our actual CarliniWagnerShell implementation, we slightly tweak parameters of CW attack so that the CW attack used as embedding is weaker but faster than the one used for attacking.
Revised: In our actual CarliniWagnerShell implementation, we slightly tweak parameters of CW attack so that the CW attack used as embedding is weaker but faster than the one used for attacking.

Original: This makes our results stronger because only the defense is weaker.
Revised: This makes our results stronger because only the defense is weaker.

Original: Experimental Methodology
Revised: Experimental Methodology

Original: We perform two experiments.
Revised: We perform two experiments.

Original: First, we compare two models in view of the goodness property: (i)
Revised: First, we compare two models in view of the goodness property: (i)

Original: The robust model trained by Madry et al.
Revised: The robust model trained by Madry et al.

Original: (2017), and (ii) a natural model that has the same model architecture,
Revised: (2017), and (ii) a natural model that has the same model architecture,

Original: but is trained without a robust objective.
Revised: but is trained without a robust objective.

Original: The comparison method is based on Proposition 1.
Revised: The comparison method is based on Proposition 1.

Original: Let P =
Revised: Let P =

Original: { p | p ∈ [0 . 5 , 1] } be a set of probability values.
Revised: { p | p ∈ [0 . 5 , 1] } be a set of probability values.

Original: We ﬁrst sample a batch of points K .
Revised: We ﬁrst sample a batch of points K .

Original: For each p ∈ P , we try to ﬁnd, for each point ( x, y ) in K , a point that is predicted as y (cid:48)
Revised: For each p ∈ P , we try to ﬁnd, for each point ( x, y ) in K , a point that is predicted as y (cid:48)

Original: (cid:54) =
Revised: (cid:54) =

Original: y with conﬁdence ≥ p in its δ -neighborhood.
Revised: y with conﬁdence ≥ p in its δ -neighborhood.

Original: For each p we get a fraction s p of points in K that such conﬁdent attacks can be found.
Revised: For each p we get a fraction s p of points in K that such conﬁdent attacks can be found.

Original: We ﬁnd that the robust model has much smaller s p for larger p and hence it satisﬁes our the goodness property better.
Revised: We ﬁnd that the robust model has much smaller s p for larger p and hence it satisﬁes our the goodness property better.

Original: We then evaluate our end-to-end defense.
Revised: We then evaluate our end-to-end defense.

Original: We go through a random permutation of test data points, and measure the following for a point that the base model predicts correctly: (1) Susceptibility .
Revised: We go through a random permutation of test data points, and measure the following for a point that the base model predicts correctly: (1) Susceptibility .

Original: That is, whether CW attack can successfully attack this point.
Revised: That is, whether CW attack can successfully attack this point.

Original: (2) Robustness .
Revised: (2) Robustness .

Original: There are two cases: (i) the model is already robust to CW attack where we expect CarlinWagnerShell retains the robustness, and (ii) the model is vulnerable to CW attack, where we expect CarliniWagnerShell to harden the model.
Revised: There are two cases: (i) the model is already robust to CW attack where we expect CarlinWagnerShell retains the robustness, and (ii) the model is vulnerable to CW attack, where we expect CarliniWagnerShell to harden the model.

Original: (3) Generalization .
Revised: (3) Generalization .

Original: That is, whether CarliniWagnerShell still predicts correctly on the point.
Revised: That is, whether CarliniWagnerShell still predicts correctly on the point.

Original: Comparing models using the goodness property.
Revised: Comparing models using the goodness property.

Original: From Deﬁnition 5 and Proposition 1, our goal is to statistically estimate Pr ( x,y ) ∼D (cid:2) ( ∃ y (cid:48) (cid:54) =
Revised: From Deﬁnition 5 and Proposition 1, our goal is to statistically estimate Pr ( x,y ) ∼D (cid:2) ( ∃ y (cid:48) (cid:54) =

Original: y, x (cid:48) ∈ N ( x, δ ))
Revised: y, x (cid:48) ∈ N ( x, δ ))

Original: F ( x (cid:48) )
Revised: F ( x (cid:48) )

Original: y (cid:48) ≥ p (cid:3) .
Revised: y (cid:48) ≥ p (cid:3) .

Original: As mentioned above, we compare the robust model in Madry et al.
Revised: As mentioned above, we compare the robust model in Madry et al.

Original: (2017) and its natural variant .
Revised: (2017) and its natural variant .

Original: Let E b denote the bad event ( ∃ y (cid:48) (cid:54) =
Revised: Let E b denote the bad event ( ∃ y (cid:48) (cid:54) =

Original: y, x (cid:48) ∈ N ( x, δ ))
Revised: y, x (cid:48) ∈ N ( x, δ ))

Original: F ( x (cid:48) )
Revised: F ( x (cid:48) )

Original: y (cid:48) ≥ p .
Revised: y (cid:48) ≥ p .

Original: We randomly sample batches of data points from the test set, and compute frequency that E b happens for each batch.
Revised: We randomly sample batches of data points from the test set, and compute frequency that E b happens for each batch.

Original: To do so, we attack samples using a modiﬁed CW attack, which tries to generate an adversarial example within the norm bound where the model classiﬁes incorrectly with high conﬁdence.
Revised: To do so, we attack samples using a modiﬁed CW attack, which tries to generate an adversarial example within the norm bound where the model classiﬁes incorrectly with high conﬁdence.

Original: Table 1 summarizes the results.
Revised: Table 1 summarizes the results.

Original: [Table]
Revised: [Table]

Original: With these statistics we can thus estimate ( p, q, δ ) -separation
Revised: With these statistics we can thus estimate ( p, q, δ ) -separation

Original: (Deﬁnition 5) of the models in comparison.
Revised: (Deﬁnition 5) of the models in comparison.

Original: We have (details are deferred to Appendix A).
Revised: We have (details are deferred to Appendix A).

Original: Proposition 3 (Separation from statistics) .
Revised: Proposition 3 (Separation from statistics) .

Original: The following two hold:
Revised: The following two hold:

Original: • With probability at least .
Revised: • With probability at least .

Original: 9 the robust model in Madry et al.
Revised: 9 the robust model in Madry et al.

Original: (2017) satisﬁes Pr ( x,y ) ∼D (cid:2) ( ∃ y (cid:48) (cid:54) =
Revised: (2017) satisﬁes Pr ( x,y ) ∼D (cid:2) ( ∃ y (cid:48) (cid:54) =

Original: y, x (cid:48) ∈ N ( x, δ ))
Revised: y, x (cid:48) ∈ N ( x, δ ))

Original: F ( x (cid:48) )
Revised: F ( x (cid:48) )

Original: y (cid:48) ≥ .
Revised: y (cid:48) ≥ .

Original: 9 (cid:3) ≤ 14 75 = 0 .
Revised: 9 (cid:3) ≤ 14 75 = 0 .

Original: That is, the robust model has ( 910 , 1475 , 8255 ) -separation.
Revised: That is, the robust model has ( 910 , 1475 , 8255 ) -separation.

Original: • With probability at least .
Revised: • With probability at least .

Original: 9 the natural model satisﬁes Pr ( x,y ) ∼D (cid:2) ( ∃ y (cid:48) (cid:54) =
Revised: 9 the natural model satisﬁes Pr ( x,y ) ∼D (cid:2) ( ∃ y (cid:48) (cid:54) =

Original: y, x (cid:48) ∈ N ( x, δ ))
Revised: y, x (cid:48) ∈ N ( x, δ ))

Original: F ( x (cid:48) )
Revised: F ( x (cid:48) )

Original: y (cid:48) ≥ .
Revised: y (cid:48) ≥ .

Original: 9 (cid:3) ≥ 247 300 = 0 .
Revised: 9 (cid:3) ≥ 247 300 = 0 .

Original: That is, the natural model does not have ( 910 , q, 8255 ) -separation for q < 247300 .
Revised: That is, the natural model does not have ( 910 , q, 8255 ) -separation for q < 247300 .

Original: End-to-end defense results.
Revised: End-to-end defense results.

Original: We have evaluated 5 non-overlapping batches from the test set (each batch consists of 30 points, thus 150 points in total), where the base model classiﬁes test points in these batches correctly (in other words, we skip test points where the base model predicts incorrectly).
Revised: We have evaluated 5 non-overlapping batches from the test set (each batch consists of 30 points, thus 150 points in total), where the base model classiﬁes test points in these batches correctly (in other words, we skip test points where the base model predicts incorrectly).

Original: We ﬁnd that the base model is susceptible to CW attack for 56 of them (thus the attack success rate is 37 .
Revised: We ﬁnd that the base model is susceptible to CW attack for 56 of them (thus the attack success rate is 37 .

Original: We ﬁnd the following after applying CarliniWagnerShell:
Revised: We ﬁnd the following after applying CarliniWagnerShell:

Original: First, the points that were robust with base model remain robust, thus we do not introduce new vulnerabilities.
Revised: First, the points that were robust with base model remain robust, thus we do not introduce new vulnerabilities.

Original: Second, among the 56 vulnerable points of base model, for 53 of them CarliniWagnerShell now is both correct and robust against attacks on which the base model fails.
Revised: Second, among the 56 vulnerable points of base model, for 53 of them CarliniWagnerShell now is both correct and robust against attacks on which the base model fails.

Original: Finally, for the remainingpoints: For only one of them, CarliniWagner shell produces wrong prediction but is robust (makes consistent predictions for both vanilla the perturbed points), and for the remaining two CarliniWagnerShell is both incorrect and not robust.
Revised: Finally, for the remainingpoints: For only one of them, CarliniWagner shell produces wrong prediction but is robust (makes consistent predictions for both vanilla the perturbed points), and for the remaining two CarliniWagnerShell is both incorrect and not robust.

Original: As a summary, the attack success rate reduces from 37 .
Revised: As a summary, the attack success rate reduces from 37 .

Original: 33% to 2 / 150 = 1 .
Revised: 33% to 2 / 150 = 1 .

Original: 33% , and the accuracy retained is 147 / 150 = 98% .
Revised: 33% , and the accuracy retained is 147 / 150 = 98% .

Original: The results are summarized in
Revised: The results are summarized in

Original: Table 2.
Revised: Table 2.

Original: [Table]
Revised: [Table]

Original: Wrong data points, conﬁdence, and the manifold boundary.
Revised: Wrong data points, conﬁdence, and the manifold boundary.

Original: We now describe the three data points where CarliniWagnerShell fails.
Revised: We now describe the three data points where CarliniWagnerShell fails.

Original: We observe that for these three points the model has low conﬁdence, and so by our theory it is expected to be hard to defend.
Revised: We observe that for these three points the model has low conﬁdence, and so by our theory it is expected to be hard to defend.

Original: Taking again a manifold point of view, we suspect that one possibility why these points have low conﬁdence is that “natural manifolds” (which are spaces of points generated by some computational devices ) are not really well separated.
Revised: Taking again a manifold point of view, we suspect that one possibility why these points have low conﬁdence is that “natural manifolds” (which are spaces of points generated by some computational devices ) are not really well separated.

Original: In the following we examine each of these three points.
Revised: In the following we examine each of these three points.

Original: From the ﬁrst batch, we have a single point that CarliniWagnerShell gives a wrong prediction, but does not change its prediction after the image is adversarially perturbed (and thus is robust, but not correct).
Revised: From the ﬁrst batch, we have a single point that CarliniWagnerShell gives a wrong prediction, but does not change its prediction after the image is adversarially perturbed (and thus is robust, but not correct).

Original: The image is presented in Figure 1(a).
Revised: The image is presented in Figure 1(a).

Original: We note that the model gives low conﬁdence ( ∼ 50% ) on the original image.
Revised: We note that the model gives low conﬁdence ( ∼ 50% ) on the original image.

Original: (where the correct label is a “cat”), and it has signiﬁcantly higher conﬁdence ( ∼ 75% ) on the perturbed image (classiﬁed as a “frog”).
Revised: (where the correct label is a “cat”), and it has signiﬁcantly higher conﬁdence ( ∼ 75% ) on the perturbed image (classiﬁed as a “frog”).

Original: In this case, Propositiononly indicates very weak separation, (indeed, ε ) and so it is expected to be hard to defend.
Revised: In this case, Propositiononly indicates very weak separation, (indeed, ε ) and so it is expected to be hard to defend.

Original: Figuredepicts the images, with a similar frog image for comparison.
Revised: Figuredepicts the images, with a similar frog image for comparison.

Original: We note that these points as generated by some cameras have essentially ignored the “size” information of different objects (can we tell from these images that a cat should be bigger than a frog?), and thus may not be well separated.
Revised: We note that these points as generated by some cameras have essentially ignored the “size” information of different objects (can we tell from these images that a cat should be bigger than a frog?), and thus may not be well separated.

Original: From the fourth batch, we have two points that CarlniWagnerShell gives us wrong predictions, and changes its predictions after the images are adversarially perturbed.
Revised: From the fourth batch, we have two points that CarlniWagnerShell gives us wrong predictions, and changes its predictions after the images are adversarially perturbed.

Original: Figure 2(a) and 2(f) depict these images and corresponding results.
Revised: Figure 2(a) and 2(f) depict these images and corresponding results.

Original: The situation is similar to the ﬁrst image above where model predictions have low conﬁdence: the base model gives low conﬁdence on the original image ( ∼ 50% for “automobile” 2(a), and ∼ 60% for “airplane” 2(f)) and adversarially perturbed images ( ∼ 60% for 2(a), and ∼ 50% for 2(f).
Revised: The situation is similar to the ﬁrst image above where model predictions have low conﬁdence: the base model gives low conﬁdence on the original image ( ∼ 50% for “automobile” 2(a), and ∼ 60% for “airplane” 2(f)) and adversarially perturbed images ( ∼ 60% for 2(a), and ∼ 50% for 2(f).

Original: Both predictions are “ship”).
Revised: Both predictions are “ship”).

Original: For each row we attach an image which is a valid “ship” image but resembles “automobile” and “airplane” images in shape.
Revised: For each row we attach an image which is a valid “ship” image but resembles “automobile” and “airplane” images in shape.

Original: Again, we suspect that the underlying natural manifolds are not well separated as one might expect.
Revised: Again, we suspect that the underlying natural manifolds are not well separated as one might expect.

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: We now discuss our method and results.
Revised: We now discuss our method and results.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Our technical contributions.
Revised: Our technical contributions.

Original: Our ﬁrst technical contribution is the proposal to take into account the inherent conﬁdence information produced by models when studying adversarial perturbations.
Revised: Our ﬁrst technical contribution is the proposal to take into account the inherent conﬁdence information produced by models when studying adversarial perturbations.

Original: Our second technical contribution is a formulation of a goodness property of models, and an analysis of existing models in view of the property.
Revised: Our second technical contribution is a formulation of a goodness property of models, and an analysis of existing models in view of the property.

Original: Interestingly, and somewhat surprisingly, we ﬁnd that a recent robust training objective function by Madry et al.
Revised: Interestingly, and somewhat surprisingly, we ﬁnd that a recent robust training objective function by Madry et al.

Original: (2017) encourages good separation of highconﬁdence points, but has essentially no control of low-conﬁdence points.
Revised: (2017) encourages good separation of highconﬁdence points, but has essentially no control of low-conﬁdence points.

Original: Our third contribution is the proposal of embedding to handle low-conﬁdence points.
Revised: Our third contribution is the proposal of embedding to handle low-conﬁdence points.

Original: Our ﬁnal contribution is a ﬁrst empirical study that validates Madry et al.
Revised: Our ﬁnal contribution is a ﬁrst empirical study that validates Madry et al.

Original: ’s model in terms of the goodness property, and further demonstrates that a good model, when wrapped with embedding, simultaneously achieves good generalization and almost perfect robustness.
Revised: ’s model in terms of the goodness property, and further demonstrates that a good model, when wrapped with embedding, simultaneously achieves good generalization and almost perfect robustness.

Original: Interpretations of our results.
Revised: Interpretations of our results.

Original: One interpretation of our results is that adversarial perturbations can naturally coexist with good generalization .
Revised: One interpretation of our results is that adversarial perturbations can naturally coexist with good generalization .

Original: While this is manifested in our analysis of Madry et al.
Revised: While this is manifested in our analysis of Madry et al.

Original: ’s objective function, we think that this phenomenon naturally and generally exists if one takes again a manifold point of view: Since natural manifolds reside in low dimensions, it seems much more challenging to control the conﬁdence boundary, where “adversarial perturbations” exist in abundance, than controlling separation of the learned structures (i.e. conﬁdent regions that approximate the underlying manifolds).
Revised: ’s objective function, we think that this phenomenon naturally and generally exists if one takes again a manifold point of view: Since natural manifolds reside in low dimensions, it seems much more challenging to control the conﬁdence boundary, where “adversarial perturbations” exist in abundance, than controlling separation of the learned structures (i.e. conﬁdent regions that approximate the underlying manifolds).

Original: On the other hand, if all one cares about is robustness (decision does not change in a small neighborhood), then adversarial perturbation problem can be resolved by combining goodness property with embedding.
Revised: On the other hand, if all one cares about is robustness (decision does not change in a small neighborhood), then adversarial perturbation problem can be resolved by combining goodness property with embedding.

Original: For example, consider the following “good model:” If a data point is from the training set, then it outputs the correct label with conﬁdence 1 , otherwise it outputs a uniform distribution over labels.
Revised: For example, consider the following “good model:” If a data point is from the training set, then it outputs the correct label with conﬁdence 1 , otherwise it outputs a uniform distribution over labels.

Original: In other words, this model learns nothing but ﬁtting the training set.
Revised: In other words, this model learns nothing but ﬁtting the training set.

Original: We note that in this case, the adversarial perturbation problem is only well deﬁned around the training points.
Revised: We note that in this case, the adversarial perturbation problem is only well deﬁned around the training points.

Original: Moreover, embedding now becomes 1 -nearest neighbor search among the training points.
Revised: Moreover, embedding now becomes 1 -nearest neighbor search among the training points.

Original: As a result, the model is still perfectly robust with our method if training points are well separated.
Revised: As a result, the model is still perfectly robust with our method if training points are well separated.

Original: Highly conﬁdent predictions on random noises.
Revised: Highly conﬁdent predictions on random noises.

Original: We note that several work shows that neural networks can have highly conﬁdent predictions on random noises (e.g., Nguyen et al. (2015)).
Revised: We note that several work shows that neural networks can have highly conﬁdent predictions on random noises (e.g., Nguyen et al. (2015)).

Original: In view of our work it is somewhat not surprising that neural networks can have such behaviors: These points are essentially ones that are far away from the “universe” the learner is asked to learn, and so if we do not control the training of neural networks to not claim conﬁdence over the structure it has never seen, then it is valid to ﬁt a model that has good behaviors on natural manifolds but also divergent behaviors outside.
Revised: In view of our work it is somewhat not surprising that neural networks can have such behaviors: These points are essentially ones that are far away from the “universe” the learner is asked to learn, and so if we do not control the training of neural networks to not claim conﬁdence over the structure it has never seen, then it is valid to ﬁt a model that has good behaviors on natural manifolds but also divergent behaviors outside.

Original: After all, why is a network supposed to work on points that are far away from the underlying natural manifolds, which is essentially the data generating distribution?
Revised: After all, why is a network supposed to work on points that are far away from the underlying natural manifolds, which is essentially the data generating distribution?

Original: Finally, we note that the adversarial perturbation problem is not well deﬁned even near those points.
Revised: Finally, we note that the adversarial perturbation problem is not well deﬁned even near those points.

Original: A B OUNDING THE PROBABILITY FOR ( p, q, δ ) - SEPARATION
Revised: A B OUNDING THE PROBABILITY FOR ( p, q, δ ) - SEPARATION

Original: This section gives details of our estimation of ( p, q, δ ) -separation from statistics in Table 1.
Revised: This section gives details of our estimation of ( p, q, δ ) -separation from statistics in Table 1.

Original: Note that event E b corresponds to a Bernoulli trial.
Revised: Note that event E b corresponds to a Bernoulli trial.

Original: Let X 1 , . . .
Revised: Let X 1 , . . .

Original: , X t be independent indicator random variables, where
Revised: , X t be independent indicator random variables, where

Original: X i = (cid:26) 1 if E b happens, 0 otherwise , and X = ( (cid:80) ti =1 X i ) /t .
Revised: X i = (cid:26) 1 if E b happens, 0 otherwise , and X = ( (cid:80) ti =1 X i ) /t .

Original: Recall Chebyshev’s inequality:
Revised: Recall Chebyshev’s inequality:

Original: Theorem 1 (Chebyshev’s Inequality) .
Revised: Theorem 1 (Chebyshev’s Inequality) .

Original: For independent random variables X 1 , . . .
Revised: For independent random variables X 1 , . . .

Original: , X t bounded in [0 , 1] , and X = ( (cid:80) ti =1 X i ) /t , we have Pr[ | X − E
Revised: , X t bounded in [0 , 1] , and X = ( (cid:80) ti =1 X i ) /t , we have Pr[ | X − E

Original: [ X ] | ≥ ε ] ≤ Var[ X ] ε 2 .
Revised: [ X ] | ≥ ε ] ≤ Var[ X ] ε 2 .

Original: In our case, E [ X ] =
Revised: In our case, E [ X ] =

Original: E [ X 1 ] = · · · = E [ X t ] and let it be µ , and let the computed frequency be ˆ µ (observed value).
Revised: E [ X 1 ] = · · · = E [ X t ] and let it be µ , and let the computed frequency be ˆ µ (observed value).

Original: Thus Pr[ | ˆ µ − µ | ≥ ε ] ≤ 1 / (4 ε 2 t ) since Var[ X ] = µ (1 − µ ) /t < 1 / 4 t .
Revised: Thus Pr[ | ˆ µ − µ | ≥ ε ] ≤ 1 / (4 ε 2 t ) since Var[ X ] = µ (1 − µ ) /t < 1 / 4 t .

Original: We thus have the following proposition about ( p, q, δ ) -separation.
Revised: We thus have the following proposition about ( p, q, δ ) -separation.

Original: Proposition 4.
Revised: Proposition 4.

Original: Let α, ε ∈ [0 , 1] .
Revised: Let α, ε ∈ [0 , 1] .

Original: For sufﬁciently large t where 14 ε 2 t ≤ 1 − α holds, we have:
Revised: For sufﬁciently large t where 14 ε 2 t ≤ 1 − α holds, we have:

Original: • (Upper bound) With probability at least α , µ is smaller than ˆ µ + ε .
Revised: • (Upper bound) With probability at least α , µ is smaller than ˆ µ + ε .

Original: • (Lower bound)
Revised: • (Lower bound)

Original: With probability at least α , µ is bigger than ˆ µ − ε .
Revised: With probability at least α , µ is bigger than ˆ µ − ε .

Original: For example, we have guarantees for α = .
Revised: For example, we have guarantees for α = .

Original: by putting ε = .
Revised: by putting ε = .

Original: 1 and t ≥ 250 .
Revised: 1 and t ≥ 250 .