Original: T OWARDS D EEP L EARNING M ODELS R ESISTANT TO A DVERSARIAL A TTACKS
Revised: T OWARDS D EEP L EARNING M ODELS R ESISTANT TO A DVERSARIAL A TTACKS

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Recent work has demonstrated that neural networks are vulnerable to adversarial examples, i.e., inputs that are almost indistinguishable from natural data and yet classiﬁed incorrectly by the network.
Revised: Recent work has demonstrated that neural networks are vulnerable to adversarial examples, i.e., inputs that are almost indistinguishable from natural data and yet classiﬁed incorrectly by the network.

Original: To address this problem, we study the adversarial robustness of neural networks through the lens of robust optimization.
Revised: To address this problem, we study the adversarial robustness of neural networks through the lens of robust optimization.

Original: This approach provides us with a broad and unifying view on much prior work on this topic.
Revised: This approach provides us with a broad and unifying view on much prior work on this topic.

Original: Its principled nature also enables us to identify methods for both training and attacking neural networks that are reliable and, in a certain sense, universal.
Revised: Its principled nature also enables us to identify methods for both training and attacking neural networks that are reliable and, in a certain sense, universal.

Original: In particular, they specify a concrete security guarantee that would protect against a well-deﬁned class of adversaries.
Revised: In particular, they specify a concrete security guarantee that would protect against a well-deﬁned class of adversaries.

Original: These methods let us train networks with signiﬁcantly improved resistance to a wide range of adversarial attacks.
Revised: These methods let us train networks with signiﬁcantly improved resistance to a wide range of adversarial attacks.

Original: They also suggest robustness against a ﬁrst-order adversary as a naturaland broad security guarantee.
Revised: They also suggest robustness against a ﬁrst-order adversary as a natural security guarantee.

Original: We believe that robustness against such well-deﬁned classes of adversaries is an important stepping stone towards fully resistant deep learning models.
Revised: We believe that robustness against such well-deﬁned classes of adversaries is an important stepping stone towards fully resistant deep learning models.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Recent breakthroughs in computer vision and speech recognition are bringing trained classiﬁers into the center of security-critical systems.
Revised: Recent breakthroughs in computer vision and speech recognition are bringing trained classiﬁers into the center of security-critical systems.

Original: Important examples include vision for autonomous cars, face recognition, and malware detection.
Revised: Important examples include vision for autonomous cars, face recognition, and malware detection.

Original: These developments make security aspects of machine learning increasingly important.
Revised: These developments make security aspects of machine learning increasingly important.

Original: In particular, resistance to adversarially chosen inputs is becoming a crucial design goal.
Revised: In particular, resistance to adversarially chosen inputs is becoming a crucial design goal.

Original: While trained models tend to be very effective in classifying benign inputs, recent work (Dalvi et al., 2004; Szegedy et al., 2013; Goodfellow et al., 2014; Nguyen et al., 2015;
Revised: While trained models tend to be very effective in classifying benign inputs, recent work (Dalvi et al., 2004; Szegedy et al., 2013; Goodfellow et al., 2014; Nguyen et al., 2015;

Original: Sharif et al., 2016) shows that an adversary is often able to manipulate the input so that the model produces an incorrect output.
Revised: Sharif et al., 2016) shows that an adversary is often able to manipulate the input so that the model produces an incorrect output.

Original: This phenomenon has received particular attention in the context of deep neural networks, and there is now a quickly growing body of work on this topic (Fawzi et al., 2015; Kurakin et al., 2016; Papernot & McDaniel, 2016;
Revised: This phenomenon has received particular attention in the context of deep neural networks, and there is now a quickly growing body of work on this topic (Fawzi et al., 2015; Kurakin et al., 2016; Papernot & McDaniel, 2016;

Original: Rozsa et al., 2016; Torkamani, 2016; Sokolic et al., 2016; Tramèr et al., 2017b).
Revised: Rozsa et al., 2016; Torkamani, 2016; Sokolic et al., 2016; Tramèr et al., 2017b).

Original: Computer vision presents a particularly striking challenge: very small changes to the input image can fool state-of-the-art neural networks with high probability (Szegedy et al., 2013; Goodfellow et al., 2014; Nguyen et al., 2015; Sharif et al., 2016; Moosavi-Dezfooli et al., 2016).
Revised: Computer vision presents a particularly striking challenge: very small changes to the input image can fool state-of-the-art neural networks with high probability (Szegedy et al., 2013; Goodfellow et al., 2014; Nguyen et al., 2015; Sharif et al., 2016; Moosavi-Dezfooli et al., 2016).

Original: This holds even when the benign example was classiﬁed correctly, and the change is imperceptible to a human.
Revised: This holds even when the benign example was classiﬁed correctly, and the change is imperceptible to a human.

Original: Apart from the security implications, this phenomenon also demonstrates that our current models are not learning the underlying concepts in a robust manner.
Revised: Apart from the security implications, this phenomenon also demonstrates that our current models are not learning the underlying concepts in a robust manner.

Original: All these ﬁndings raise a fundamental question:
Revised: All these ﬁndings raise a fundamental question:

Original: How can we learn models robust to adversarial inputs?
Revised: How can we learn models robust to adversarial inputs?

Original: There are now many proposed defense mechanisms for the adversarial setting.
Revised: There are now many proposed defense mechanisms for the adversarial setting.

Original: Examples include defensive distillation (Papernot et al., 2016a; Papernot & McDaniel, 2016), feature squeezing (Xu et al., 2017), and several detection approaches for adversarial inputs (see Carlini & Wagner (2017) for references).
Revised: Examples include defensive distillation (Papernot et al., 2016a; Papernot & McDaniel, 2016), feature squeezing (Xu et al., 2017), and several detection approaches for adversarial inputs (see Carlini & Wagner (2017) for references).

Original: While these works constitute important ﬁrst steps in exploring the realm of possibilities, they do not offer a good understanding of the guarantees they provide.
Revised: While these works constitute important ﬁrst steps in exploring the realm of possibilities, they do not offer a good understanding of the guarantees they provide.

Original: We can never be certain that a particular defense mechanism prevents the existence of some well-deﬁned class of adversarial attacks.
Revised: We can never be certain that a particular defense mechanism prevents the existence of some well-deﬁned class of adversarial attacks.

Original: This makes it difﬁcult to navigate the landscape of adversarial robustness or to fully evaluate the possible security implications.
Revised: This makes it difﬁcult to navigate the landscape of adversarial robustness or to fully evaluate the possible security implications.

Original: Moreover, subsequent work (Carlini & Wagner, 2016a; He et al., 2017) has shown that most of these defenses can be bypassed by stronger, adaptive adversaries.
Revised: Moreover, subsequent work (Carlini & Wagner, 2016a; He et al., 2017) has shown that most of these defenses can be bypassed by stronger, adaptive adversaries.

Original: (Further discussion of related work has been deferred to Section 6.)

Original: In this paper, we study the adversarial robustness of neural networks through the lens of robust optimization.
Revised: In this paper, we study the adversarial robustness of neural networks through the lens of robust optimization.

Original: We use a natural saddle point (min-max) formulation to capture the notion of security against adversarial attacks in a principled manner.
Revised: We use a natural saddle point (min-max) formulation to capture the notion of security against adversarial attacks in a principled manner.

Original: This formulation allows us to be precise about the type of security guarantee we would like to achieve, i.e., the broad class of attacks we want to be resistant to (in contrast to defending only against speciﬁc known attacks).
Revised: This formulation allows us to be precise about the type of security guarantee we would like to achieve, i.e., the broad class of attacks we want to be resistant to (in contrast to defending only against speciﬁc known attacks).

Original: The formulation also enables us to cast both attacks and defenses into a common theoretical framework.
Revised: The formulation also enables us to cast both attacks and defenses into a common theoretical framework.

Original: Most prior work on adversarial examples naturally ﬁts into this framework.
Revised: Most prior work on adversarial examples naturally ﬁts into this framework.

Original: In particular, adversarial training directly corresponds to optimizing this saddle point problem.
Revised: In particular, adversarial training directly corresponds to optimizing this saddle point problem.

Original: Similarly, prior methods for attacking neural networks correspond to speciﬁc algorithms for solving the underlyingconstrained optimization problem.
Revised: Similarly, prior methods for attacking neural networks correspond to speciﬁc algorithms for solving the underlying optimization problem.

Original: Equipped with this perspective, we make the following contributions.
Revised: Equipped with this perspective, we make the following contributions.

Original: We conduct a careful experimental study of the optimization landscape corresponding to this saddle point formulation.
Revised: We conduct a careful experimental study of the optimization landscape corresponding to this saddle point formulation.

Original: Despite the non-convexity and non-concavity of its constituent parts, we ﬁnd that the underlying optimization problem is tractable after all.
Revised: Despite the non-convexity and non-concavity of its constituent parts, we ﬁnd that the underlying optimization problem is tractable after all.

Original: In particular, we provide strong evidence that ﬁrst-order methods can reliably solve this problem and motivate projected gradient descent (PGD) as a universal “ﬁrst-order adversary”, i
Revised: In particular, we provide strong evidence that ﬁrst-order methods can reliably solve this problem and motivate projected gradient descent (PGD) as a universal “ﬁrst-order adversary”, i

Original: e., the strongest attack utilizing the local ﬁrst order information about the network.
Revised: e., the strongest attack utilizing the local ﬁrst order information about the network.

Original: We supplement these insights with ideas from real analysis to further motivate adversarial training against a PGD adversary as a strong and natural defense.
Revised: We supplement these insights with ideas from real analysis to further motivate adversarial training against a PGD adversary as a strong and natural defense.

Original: We explore the impact of network architecture on adversarial robustness and ﬁnd that model capacity plays an important role.
Revised: We explore the impact of network architecture on adversarial robustness and ﬁnd that model capacity plays an important role.

Original: To reliably withstand strong adversarial attacks, networks require a signiﬁcantly larger capacity than for correctly classifying benign examples only.
Revised: To reliably withstand strong adversarial attacks, networks require a signiﬁcantly larger capacity than for correctly classifying benign examples only.

Original: This shows that a robust decision boundary of the saddle point problem can be signiﬁcantly more complicated than a decision boundary that simply separates the benign data points.
Revised: This shows that a robust decision boundary of the saddle point problem can be signiﬁcantly more complicated than a decision boundary that simply separates the benign data points.

Original: Building on the above insights, we train networks on MNIST and CIFAR that are robust to a wide range of adversarial attacks against adversaries bounded by 0 .
Revised: Building on the above insights, we train networks on MNIST and CIFAR10 that are robust to a wide range of adversarial attacks against adversaries bounded by 0 .

Original: 3 and 8 in (cid:96) ∞ norm respectively.
Revised: 3 and 8 in (cid:96) ∞ norm respectively.

Original: Our approach is based on optimizing the aforementioned saddle point formulation and uses our optimal “ﬁrst-order adversary”.
Revised: Our approach is based on optimizing the aforementioned saddle point formulation and uses our optimal “ﬁrst-order adversary”.

Original: Our best MNIST model achieves an accuracy of more than 89% against the strongest adversaries in our test suite.
Revised: Our best MNIST model achieves an accuracy of more than 89% against the strongest adversaries in our test suite.

Original: In particular, our MNIST network is even robust against white box attacks of an iterative adversary.
Revised: In particular, our MNIST network is even robust against white box attacks of an iterative adversary.

Original: Our CIFAR10 model achieves an accuracy of 46% against the same adversary.
Revised: Our CIFAR10 model achieves an accuracy of 46% against the same adversary.

Original: Furthermore, in case of the weaker black box (transfer) attacks, our MNIST and CIFAR10 networks achieve an accuracy of more than 95% and 64%, respectively (a more detailed overview can be found in Tables 1 and 2).
Revised: Furthermore, in case of the weaker black box (transfer) attacks, our MNIST and CIFAR10 networks achieve an accuracy of more than 95% and 64%, respectively (a more detailed overview can be found in Tables 1 and 2).

Original: To the best of our knowledge, we are the ﬁrst to achieve these levels of robustness on MNIST and CIFAR10 against a broad set of attacks.
Revised: To the best of our knowledge, we are the ﬁrst to achieve these levels of robustness on MNIST and CIFAR10 against a broad set of attacks.

Original: Overall, these ﬁndings suggest that secure neural networks are within reach.
Revised: Overall, these ﬁndings suggest that secure neural networks are within reach.

Original: In order to further support this claim, we have invited the community to attempt attacks against our MNIST and CIFAR10 networks in the form of an open challenge.
Revised: In order to further support this claim, we have invited the community to attempt attacks against our MNIST and CIFAR networks in the form of an open challenge 1,2 .

Original: At the time of writing, we received about ﬁfteen submissions to the MNIST challenge and the best submission achieved roughly 93% accuracy in a black box attack.
Revised: At the time of writing, we received about ﬁfteen submissions to the MNIST challenge and the best submission achieved roughly 93% accuracy in a black box attack.

Original: We received no submissions for the CIFAR10 challenge that went beyond the 64% accuracy of our attack.
Revised: We received no submissions for the CIFAR10 challenge that went beyond the 64% accuracy of our attack.

Original: Considering that other proposed defenses were often quickly broken (Carlini & Wagner, 2017), we believe that our robust models are signiﬁcant progress on the defense side.
Revised: Considering that other proposed defenses were often quickly broken (Carlini & Wagner, 2017), we believe that our robust models are signiﬁcant progress on the defense side.

Original: Furthermore, recent work  on veriﬁable adversarial examples showed that our proposed defense reliably increased the robustness to any (cid:96) ∞ -bounded attack.
Revised: Furthermore, recent work (Carlini et al., 2017) on veriﬁable adversarial examples showed that our proposed defense reliably increased the robustness to any (cid:96) ∞ -bounded attack.

Original:  A N O PTIMIZATION V IEW ON A DVERSARIAL R OBUSTNESS
Revised:  A N O PTIMIZATION V IEW ON A DVERSARIAL R OBUSTNESS

Original: Much of our discussion will revolve around an optimization view of adversarial robustness.
Revised: Much of our discussion will revolve around an optimization view of adversarial robustness.

Original: Thisperspective not only captures the phenomena we want to study in a precise manner, but will also inform our investigations.
Revised: This inform our investigations.

Original: To this end, let us consider a standard classiﬁcation task with an underlying data distribution D over pairs of examples x ∈ R d and corresponding labels y ∈ [ k ] .
Revised: To this end, let us consider a standard classiﬁcation task with an underlying data distribution D over pairs of examples x ∈ R d and corresponding labels y ∈ [ k ] .

Original: We also assume that we are given a suitable loss function
Revised: We also assume that we are given a suitable loss function

Original: L ( θ, x, y ) , for instance the cross-entropy loss for a neural network.
Revised: L ( θ, x, y ) , for instance the cross-entropy loss for a neural network.

Original: As usual, θ ∈ R p is the set of model parameters.
Revised: As usual, θ ∈ R p is the set of model parameters.

Original: Our goal then is to ﬁnd model parameters θ that minimize the risk E ( x,y )
Revised: Our goal then is to ﬁnd model parameters θ that minimize the risk E ( x,y )

Original: ∼D [ L ( x, y, θ )] .
Revised: ∼D [ L ( x, y, θ )] .

Original: Empirical risk minimization (ERM) has been tremendously successful as a recipe for ﬁnding classiﬁers with small population risk.
Revised: Empirical risk minimization (ERM) has been tremendously successful as a recipe for ﬁnding classiﬁers with small population risk.

Original: Unfortunately, ERM often does not yield models that are robust to adversarially crafted examples (Goodfellow et al., 2014; Kurakin et al., 2016; Moosavi-Dezfooli et al., 2016;
Revised: Unfortunately, ERM often does not yield models that are robust to adversarially crafted examples (Goodfellow et al., 2014; Kurakin et al., 2016; Moosavi-Dezfooli et al., 2016;

Original: Tramèr et al., 2017b).
Revised: Tramèr et al., 2017b).

Original: Formally, there are efﬁcient algorithms (“adversaries”) that take an example x belonging to class c 1 as input and ﬁnd examples x
Revised: Formally, there are efﬁcient algorithms (“adversaries”) that take an example x belonging to class c 1 as input and ﬁnd examples x

Original: adv such that x adv is very close to x but the model incorrectly classiﬁes x adv as belonging to class c 2 (cid:54) =
Revised: adv such that x adv is very close to x but the model incorrectly classiﬁes x adv as belonging to class c 2 (cid:54) =

Original: c 1 .
Revised: c 1 .

Original: In order to reliably train models that are robust to adversarial attacks, it is necessary to augment the ERM paradigm appropriately.
Revised: In order to reliably train models that are robust to adversarial attacks, it is necessary to augment the ERM paradigm.

Original: Instead of resorting to methods that directly focus on improving the robustness to speciﬁc attacks, our approach is to ﬁrst propose a concrete guarantee that an adversarially robust model should satisfy.
Revised: Instead of resorting to methods that directly focus on improving the robustness to speciﬁc attacks, our approach is to ﬁrst propose a concrete guarantee that an adversarially robust model should satisfy.

Original: We then adapt our training methods towards achieving this guarantee.
Revised: We then adapt our training methods towards achieving this guarantee.

Original: The ﬁrst step towards such a guarantee is to specify an threat model , i.e., a precise deﬁnition of the attacks our models should be resistant to.
Revised: The ﬁrst step towards such a guarantee is to specify an threat model , i.e., a precise deﬁnition of the attacks our models should be resistant to.

Original: For each data point x , we introduce a set of allowed perturbations S ⊆
Revised: For each data point x , we introduce a set of allowed perturbations S ⊆

Original: R d that formalizes the manipulative power of the adversary.
Revised: R d that formalizes the manipulative power of the adversary.

Original: In image classiﬁcation, we choose S so that it captures perceptual similarity between images.
Revised: In image classiﬁcation, we choose S so that it captures perceptual similarity between images.

Original: For instance, the (cid:96) ∞ -ball around x has recently been studied as a natural notion for adversarial perturbations (Goodfellow et al., 2014).
Revised: For instance, the (cid:96) ∞ -ball around x has recently been studied as a natural notion for adversarial perturbations (Goodfellow et al., 2014).

Original: While we focus on robustness against (cid:96) ∞ -bounded attacks in this paper, we remark that more comprehensive notions of perceptual similarity are an important direction for future research.
Revised: While we focus on robustness against (cid:96) ∞ -bounded attacks in this paper, we remark that more comprehensive notions of perceptual similarity are an important direction for future research.

Original: Next, we modify the deﬁnition of population risk E D [ L ] by incorporating the above adversary.
Revised: Next, we modify the deﬁnition of population risk E D [ L ] by incorporating the above adversary.

Original: Instead of computing the loss L directly on samples from the distribution D , we allow the adversary to perturb the input ﬁrst.
Revised: Instead of computing the loss L directly on samples from the distribution D , we allow the adversary to perturb the input ﬁrst.

Original: This gives rise to the following saddle point problem, which is our central object of study:
Revised: This gives rise to the following saddle point problem, which is our central object of study:

Original: [Equation]
Revised: [Equation]

Original: ) , where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: Formulations of this type (and their ﬁnite-sample counterparts) have a long history in robust optimization, going back to Wald (Wald, 1939; 1945; 1992).
Revised: Formulations of this type (and their ﬁnite-sample counterparts) have a long history in robust optimization, going back to Wald (Wald, 1939; 1945; 1992).

Original: It turns out that this formulation is also particularly useful in our context.
Revised: It turns out that this formulation is also particularly useful in our context.

Original: We will refer to the quantity ρ ( θ ) as the adversarial loss of the network with parameters θ .
Revised: We will refer to the quantity ρ ( θ ) as the adversarial loss of the network with parameters θ .

Original: First, this formulation gives us a unifying perspective that encompasses much prior work on adversarial robustness.
Revised: First, this formulation gives us a unifying perspective that encompasses much prior work on adversarial robustness.

Original: Our perspective stems from viewing the saddle point problem as the composition of an inner maximization problem and an outer minimization problem.
Revised: Our perspective stems from viewing the saddle point problem as the composition of an inner maximization problem and an outer minimization problem.

Original: Both of these problems have a natural interpretation in our context.
Revised: Both of these problems have a natural interpretation in our context.

Original: The inner maximization problem aims to ﬁnd an adversarial version of a given data point x that achieves a high loss.
Revised: The inner maximization problem aims to ﬁnd an adversarial version of a given data point x that achieves a high loss.

Original: This is precisely the problem of attacking a given neural network.
Revised: This is precisely the problem of attacking a given neural network.

Original: On the other hand, the goal of the outer minimization problem is to ﬁnd model parameters so that the adversarial loss given by the inner attack problem is minimized.
Revised: On the other hand, the goal of the outer minimization problem is to ﬁnd model parameters so that the adversarial loss given by the inner attack problem is minimized.

Original: This is precisely the problem of training a robust classiﬁer using adversarial training techniques.
Revised: This is precisely the problem of training a robust classiﬁer using adversarial training techniques.

Original: Second, the saddle point problem speciﬁes a clear goal that an ideal robust classiﬁer should achieve, as well as a quantitative measure of its robustness.
Revised: Second, the saddle point problem speciﬁes a clear goal that a robust classiﬁer should achieve, as well as a quantitative measure of its robustness.

Original: In particular, when the parameters θ yield a (nearly) vanishing risk, the corresponding model is perfectly robust to attacks speciﬁed by our threat model.
Revised: In particular, when the parameters θ yield a (nearly) vanishing risk, the corresponding model is perfectly robust to attacks speciﬁed by our threat model.

Original: Our paper investigates the structure of this saddle point problem in the context of deep neural networks.
Revised: Our paper investigates the structure of this saddle point problem in the context of deep neural networks.

Original: This formulation will be the main drive of our investigations that will lead us to training techniques that produce models with high resistance to a wide range of adversarial attacks.
Revised: This formulation will be the main drive of our investigations that will lead us to training techniques that produce models with high resistance to a wide range of adversarial attacks.

Original:  T OWARDS A DVERSARIALLY R OBUST N ETWORKS
Revised:  T OWARDS A DVERSARIALLY R OBUST N ETWORKS

Original: Current work on adversarial examples usually focuses on speciﬁc defensive mechanisms, or on attacks against such defenses.
Revised: Current work on adversarial examples usually focuses on speciﬁc defensive mechanisms, or on attacks against such defenses.

Original: An important feature of formulation (2.1) is that attaining small adversarial loss gives a guarantee that no allowed attack will fool the network.
Revised: An important feature of formulation (2.1) is that attaining small adversarial loss gives a guarantee that no allowed attack will fool the network.

Original: By deﬁnition, no adversarial perturbations are possible because the loss is small for all perturbations allowed by our threat model.
Revised: By deﬁnition, no adversarial perturbations are possible because the loss is small for all perturbations allowed by our threat model.

Original: This perspective allows us to reduce the task of ﬁnding truly robust models to an optimization problem.
Revised: This perspective allows us to reduce the task of ﬁnding truly robust models to an optimization problem.

Original: Hence, we can now focus our attention solely on obtaining a good solution to
Revised: Hence, we can now focus our attention solely on obtaining a good solution to

Original: Problem (2.1).
Revised: Problem (2.1).

Original: Gradients from attacks.
Revised: Gradients from attacks.

Original: Since Stochastic Gradient Descent (SGD) and its variants are by far the most successful algorithms for training neural networks, we also want to apply SGD to Problem (2.1).
Revised: Since Stochastic Gradient Descent (SGD) and its variants are by far the most successful algorithms for training neural networks, we also want to apply SGD to Problem (2.1).

Original: This raises the question how we can compute gradients ∇ θ ρ ( θ ) for the outer minimization problem.
Revised: This raises the question how we can compute gradients ∇ θ ρ ( θ ) for the outer minimization problem.

Original: Since the adversarial loss function ρ ( θ ) corresponds to a maximization problem, we cannot simply apply the usual backpropagation algorithm.
Revised: Since the adversarial loss function ρ ( θ ) corresponds to a maximization problem, we cannot simply apply the usual backpropagation algorithm.

Original: Instead, a natural approach is to compute the gradient at the maximizer of the inner maximization problem.
Revised: Instead, a natural approach is to compute the gradient at the maximizer of the inner maximization problem.

Original: A priori, it is not clear that this is a valid descent direction for the saddle point problem.
Revised: A priori, it is not clear that this is a valid descent direction for the saddle point problem.

Original: However, for the case of continuously differentiable functions,
Revised: However, for the case of continuously differentiable functions,

Original: Danskin’s theorem – a classic theorem in optimization – states that this is indeed true and gradients at maximizers of the inner problem correspond to descent directions for the saddle point problem (see
Revised: Danskin’s theorem – a classic theorem in optimization – states that this is indeed true and gradients at maximizers of the inner problem correspond to descent directions for the saddle point problem (see

Original: Appendix C for details).
Revised: Appendix C for details).

Original: Leveraging this connection, our goal now is to ﬁnd a reliable algorithm for solving the inner maximization problem, i.e., to evaluate ρ ( θ ) .
Revised: Leveraging this connection, our goal now is to ﬁnd a reliable algorithm for solving the inner maximization problem, i.e., to evaluate ρ ( θ ) .

Original: When instantiated for a batch of examples (instead of the expectation over the entire distribution D ), ﬁnding a maximizer δ ∈ S of ρ ( θ ) corresponds exactly to ﬁnding an attack on the neural network.
Revised: When instantiated for a batch of examples (instead of the expectation over the entire distribution D ), ﬁnding a maximizer δ ∈ S of ρ ( θ ) corresponds exactly to ﬁnding an attack on the neural network.

Original: This allows us to employ known attacks as inner maximization algorithms.
Revised: This allows us to employ known attacks as inner maximization algorithms.

Original: Prior work has proposed methods such as the Fast Gradient Sign Method (FGSM) and multiple variations of it (Goodfellow et al., 2014).
Revised: Prior work has proposed methods such as the Fast Gradient Sign Method (FGSM) and multiple variations of it (Goodfellow et al., 2014).

Original: FGSM is an attack for an (cid:96) ∞ -bounded adversary and computes an adversarial example as
Revised: FGSM is an attack for an (cid:96) ∞ -bounded adversary and computes an adversarial example as

Original: [Equation]
Revised: [Equation]

Original: One can interpret this attack as a simple one-step scheme for maximizing the inner part of the saddle point formulation.
Revised: One can interpret this attack as a simple one-step scheme for maximizing the inner part of the saddle point formulation.

Original: A more powerful adversary is the multi-step variant FGSM k , which is essentially projected gradient descent (PGD) on the negative loss function (Kurakin et al., 2016):
Revised: A more powerful adversary is the multi-step variant FGSM k , which is essentially projected gradient descent (PGD) on the negative loss function (Kurakin et al., 2016)

Original: [Equation]
Revised: [Equation]

Original: Loss landscape.
Revised: Loss landscape.

Original: While PGD is a well-motivated approach for the inner maximization problem, it is not clear whether we can actually ﬁnd a good solution in a reasonable amount of time.
Revised: While PGD is a well-motivated approach for the inner maximization problem, it is not clear whether we can actually ﬁnd a good solution in a reasonable amount of time.

Original: The problem is non-concave, so a priori we have no guarantees on the solution quality of PGD.
Revised: The problem is non-concave, so a priori we have no guarantees on the solution quality of PGD.

Original: One of our contributions is demonstrating that, in practice, the inner maximization problem is indeed well-behaved.
Revised: One of our contributions is demonstrating that, in practice, the inner maximization problem is indeed well-behaved.

Original: In particular, we experimentally explore the structure given by the non-concave inner problem and ﬁnd that its loss landscape has a surprisingly tractable structure of local maxima (see Appendix A).
Revised: In particular, we experimentally explore the structure given by the non-concave inner problem and ﬁnd that its loss landscape has a surprisingly tractable structure of local maxima (see Appendix A).

Original: This structure also points towards projected gradient descent as the “ultimate” ﬁrst-order adversary (see Section 5).
Revised: This structure also points towards projected gradient descent as the “ultimate” ﬁrst-order adversary (see Section 5).

Original: Despite the fact that the exact assumptions of Danskin’s theorem do not hold for our problem (the function is not continuously differentiable due to ReLU activations, and we only compute approximate maximizers of the inner problem), our experiments suggest that we can still use these gradients to optimize our problem.
Revised: Despite the fact that the exact assumptions of Danskin’s theorem do not hold for our problem (the function is not continuously differentiable due to ReLU activations, and we only compute approximate maximizers of the inner problem), our experiments suggest that we can still use these gradients to optimize our problem.

Original: By applying SGD using the gradient of the loss at adversarial examples, we can consistently reduce the loss of the saddle point problem during training (e.g., see Figure 1 in
Revised: By applying SGD using the gradient of the loss at adversarial examples, we can consistently reduce the loss of the saddle point problem during training (e.g., see Figure 1 in

Original: Section 4).
Revised: Section 4).

Original: These observations suggest that we reliably optimize the saddle point formulation (2.1) and thus train robust classiﬁers.
Revised: These observations suggest that we reliably optimize the saddle point formulation (2.1) and thus train robust classiﬁers.

Original: Model capacity.
Revised: Model capacity.

Original: Before we proceed to our main experiment results in the next section, we brieﬂy mention another important insight from our robust optimization perspective.
Revised: Before we proceed to our main experiment results in the next section, we brieﬂy mention another important insight from our robust optimization perspective.

Original: Solving the problem from Equation (2.1) successfully is not sufﬁcient to guarantee robust and accurate classiﬁcation.
Revised: Solving the problem from Equation (2.1) successfully is not sufﬁcient to guarantee robust and accurate classiﬁcation.

Original: We also require that the value of the problem (i.e., the ﬁnal loss we achieve against adversarial examples) is small, which then provides guarantees for the performance of our classiﬁer.
Revised: We also require that the value of the problem (i.e., the ﬁnal loss we achieve against adversarial examples) is small, which then provides guarantees for the performance of our classiﬁer.

Original: In particular, achieving a very small value corresponds to a perfect classiﬁer, which is robust to adversarial inputs.
Revised: In particular, achieving a very small value corresponds to a perfect classiﬁer, which is robust to adversarial inputs.

Original: In Appendix B, we show experimentally that network capacity plays a crucial role in enabling robustness.
Revised: In Appendix B, we show experimentally that network capacity plays a crucial role in enabling robustness.

Original: In particular, training a robust classiﬁer requires a signiﬁcantly larger network than only achieving high accuracy on natural examples.
Revised: In particular, training a robust classiﬁer requires a signiﬁcantly larger network than only achieving high accuracy on natural examples.

Original:  E XPERIMENTS : A DVERSARIALLY R OBUST D EEP L EARNING M ODELS ?
Revised:  E XPERIMENTS : A DVERSARIALLY R OBUST D EEP L EARNING M ODELS ?

Original: Following our understanding developed in the previous section, we can now apply our proposed approach to train robust classiﬁers.
Revised: Following our understanding developed in the previous section, we can now apply our proposed approach to train robust classiﬁers.

Original: For both MNIST and CIFAR10, our adversary of choice will be projected gradient descent starting from a random perturbation around the natural example.
Revised: For both MNIST and CIFAR10, our adversary of choice will be projected gradient descent starting from a random perturbation around the natural example.

Original: As our experiments suggest (Appendix A) this algorithm is very efﬁcient at reliably producing examples of (near) maximal loss.
Revised: As our experiments suggest (Appendix A) this algorithm is very efﬁcient at reliably producing examples of (near) maximal loss.

Original: In a sense, it seems to correspond to a “ultimate” f irst order adversary.
Revised: In a sense, it seems to correspond to a “ultimate” f irst order adversary.

Original: Since we are training the model for multiple epochs, we did not see any beneﬁt in restarting PGD multiple times per batch – a new start is chosen each time the same example is encountered.
Revised: Since we are training the model for multiple epochs, we did not see any beneﬁt in restarting PGD multiple times per batch – a new start is chosen each time the same example is encountered.

Original: During the training procedure against the PGD adversary, we observe a steady decrease in the training loss of adversarial examples, illustrated in Figure 1.
Revised: During the training procedure against the PGD adversary, we observe a steady decrease in the training loss of adversarial examples, illustrated in Figure 1.

Original: This behavior indicates that we are consistently decreasing the adversarial loss and indeed successfully solving our original optimization problem.
Revised: This behavior indicates that we are consistently decreasing the adversarial loss and indeed successfully solving our original optimization problem.

Original: [Figure]
Revised: [Figure]

Original: We evaluate the trained models against a range of adversaries.
Revised: We evaluate the trained models against a range of adversaries.

Original: We illustrate our results in Table 1 for
Revised: We illustrate our results in Table 1 for

Original: MNIST and Table 2 for CIFAR10.
Revised: MNIST and Table 2 for CIFAR10.

Original: The adversaries we consider are:
Revised: The adversaries we consider are:

Original: • White-box attacks with PGD for a different number of of iterations and restarts, denoted by source A. • White-box attacks from Carlini & Wagner (2016b).
Revised: • White-box attacks with PGD for a different number of of iterations and restarts, denoted by source A. • White-box attacks from Carlini & Wagner (2016b).

Original: We use their suggested loss function and minimize it using PGD.
Revised: We use their suggested loss function and minimize it using PGD.

Original: This is denoted as CW, where the corresponding attack with a high conﬁdence parameter ( κ = 50 ) is denoted as CW+.
Revised: This is denoted as CW, where the corresponding attack with a high conﬁdence parameter ( κ = 50 ) is denoted as CW+.

Original: • Black-box attacks from an independently trained copy of the network, denoted A’.
Revised: • Black-box attacks from an independently trained copy of the network, denoted A’.

Original: • Black-box attacks from a version of the same network trained only on natural examples, denoted A nat .
Revised: • Black-box attacks from a version of the same network trained only on natural examples, denoted A nat .

Original: • Black-box attacks from a different convolution architecture, denoted B, described in Tramèr et al.
Revised: • Black-box attacks from a different convolution architecture, denoted B, described in Tramèr et al.

Original: (2017a).
Revised: (2017a).

Original: MNIST.
Revised: MNIST.

Original: We run 40 iterations of projected gradient descent as our adversary, with a step size of 0 .
Revised: We run 40 iterations of projected gradient descent as our adversary, with a step size of 0 .

Original: 01 (we choose to take gradient steps in the (cid:96) ∞ norm, i.e. adding the sign of the gradient, since this makes the choice of the step size simpler).
Revised: 01 (we choose to take gradient steps in the (cid:96) ∞ norm, i.e. adding the sign of the gradient, since this makes the choice of the step size simpler).

Original: We train and evaluate against perturbations of size ε = 0 .
Revised: We train and evaluate against perturbations of size ε = 0 .

Original: We use a network consisting of two convolutional layers with 32 and 64 ﬁlters respectively, each followed by 2 × 2 max-pooling, and a fully connected layer of size 1024.
Revised: We use a network consisting of two convolutional layers with 32 and 64 ﬁlters respectively, each followed by 2 × 2 max-pooling, and a fully connected layer of size 1024.

Original: When trained with natural examples, this network reaches 99.2% accuracy on the evaluation set.
Revised: When trained with natural examples, this network reaches 99.2% accuracy on the evaluation set.

Original: However, when evaluating on examples perturbed with FGSM the accuracy drops to 6.4%.
Revised: However, when evaluating on examples perturbed with FGSM the accuracy drops to 6.4%.

Original: Given that the resulting MNIST model is very robust, we investigated the learned parameters in order to understand how they affect adversarial robustness.
Revised: Given that the resulting MNIST model is very robust, we investigated the learned parameters in order to understand how they affect adversarial robustness.

Original: The results of the investigation are presented in Appendix E.
Revised: The results of the investigation are presented in Appendix E.

Original: CIFAR10.
Revised: CIFAR10.

Original: For the CIFAR10 dataset, we use the two architectures described in B (the original
Revised: For the CIFAR10 dataset, we use the two architectures described in B (the original

Original: Resnet and its 10 × wider variant).
Revised: Resnet and its 10 × wider variant).

Original: We trained the network against a PGD adversary with (cid:96) ∞ projected
Revised: We trained the network against a PGD adversary with (cid:96) ∞ projected

Original: [Table]
Revised: [Table]

Original: [Table]

Original: trained network against different adversaries for ε = 8 .

Original: For each model of attack we show the most effective attack in bold.

Original: The source networks considered for the attack are: the network itself (A) (white-box attack), an independtly initialized and trained copy of the network (A’), a copy of the network trained on natural examples (A nat ).

Original: gradient descent again, this time using 7 steps of size 2, and a total ε = 8 .
Revised: gradient descent again, this time using 7 steps of size 2, and a total ε = 8 .

Original: For our hardest adversary we chose 20 steps with the same settings, since other hyperparameter choices didn’t offer a signiﬁcant decrease in accuracy.
Revised: For our hardest adversary we chose 20 steps with the same settings, since other hyperparameter choices didn’t offer a signiﬁcant decrease in accuracy.

Original: The results of our experiments appear in Table 2.
Revised: The results of our experiments appear in Table 2.

Original: The adversarial robustness of our network is signiﬁcant, given the power of iterative adversaries, but still far from satisfactory.
Revised: The adversarial robustness of our network is signiﬁcant, given the power of iterative adversaries, but still far from satisfactory.

Original: We believe that further progress is possible along these lines by understanding how adversarial training works and what techniques can complement it leading to robust models. 
Revised: We believe that further progress is possible along these lines by understanding how adversarial training works and what techniques can complement it leading to robust models. [Table]

Original: Resistance for different values of ε and (cid:96) 2 -bounded attacks.
Revised: Resistance for different values of ε and (cid:96) 2 -bounded attacks.

Original: In order to perform a broader evaluation of the adversarial robustness of our models, we run two kinds of additional experiments.
Revised: In order to perform a broader evaluation of the adversarial robustness of our models, we run two kinds of additional experiments.

Original: On one hand, we investigate the resistance to (cid:96) ∞ -bounded attacks for different values of ε .
Revised: On one hand, we investigate the resistance to (cid:96) ∞ -bounded attacks for different values of ε .

Original: On the other hand, we examine the resistance of our model to attacks that are bounded in (cid:96) 2 as opposed to (cid:96) ∞ norm.
Revised: On the other hand, we examine the resistance of our model to attacks that are bounded in (cid:96) 2 as opposed to (cid:96) ∞ norm.

Original: The results appear in Figure 2.
Revised: The results appear in Figure 2.

Original: We emphasize that the models we are examining[Figure] here correspond to training against (cid:96) ∞ -bounded attacks with the original value of ε = 0 .
Revised: We emphasize that the models we are examining here correspond to training against (cid:96) ∞ -bounded attacks with the original value of ε = 0 .

Original: 3 , for
Revised: 3 , for

Original: MNIST, and ε = 8 for CIFAR10.
Revised: MNIST, and ε = 8 for CIFAR10.

Original: In particular, our MNIST model retains signiﬁcant resistance to (cid:96) 2 -norm-bounded perturbations too – it hasquite good accuracyin this regime even for ε = 4 .5 .
Revised: In particular, our MNIST model retains signiﬁcant resistance to (cid:96) 2 -norm-bounded perturbations too – it has good accuracy even for ε = 4 .

Original: To put this value of ε into perspective, we provide a sample of corresponding adversarial examples in Figure 12 of Appendix F. One can observe that some of the underlying perturbations are large enough that even a human could be confused. 
Revised: 5 .We provide a sample of corresponding adversarial examples in Figure 12 of Appendix F. One can observe that some of the underlying perturbations are large enough that even a human could be confused. Training Accuracy.

Original: F IRST -O RDER A DVERSARIES . 
Revised: It is worth noting our MNIST and (wide) CIFAR10 networks reached 100% adversarial accuracy on the training set. That is we can ﬁt the training set even against a PGD adversary of ε = 0 . 3 and ε = 8 respectively. This shows that the landscape of the underlying optimization problem is tractable and does not present a signiﬁcant barrier to our techniques.

Revised: [Figure]

Revised: Running Time.

Revised: Unfortunately, solving the robust version of the problem instead of the standard one imposes a signiﬁcant computational overhead.

Revised: Standard training requires one forward and one backward pass through the network for each training batch.

Revised: Instead, adversarial training with a k -step PGD adversary, requires additionally k forward and k backward passes through the network to compute the adversarial version of the training batch.

Revised: This implies an increase in running time of a factor of ( k + 1) .

Revised: We hope that future research will propose ways to mitigate this drawback.

Original: Our exploration of the loss landscape (Appendix A) shows that the local maxima found by PGD all have similar loss values, both for normally trained networks and adversarially trained networks.
Revised: Our exploration of the loss landscape (Appendix A) shows that the local maxima found by PGD all have similar loss values, both for normally trained networks and adversarially trained networks.

Original: This concentration phenomenon suggests an intriguing view on the problem in which robustness against the PGD adversary yields robustness against all ﬁrst-order adversaries, i.e., attacks that rely only on ﬁrst-order information.
Revised: This concentration phenomenon suggests an intriguing view on the problem in which robustness against the PGD adversary yields robustness against all ﬁrst-order adversaries, i.e., attacks that rely only on ﬁrst-order information.

Original: As long as the adversary only uses gradients of the loss function with respect to the input, we conjecture that it will not ﬁnd signiﬁcantly better local maxima than PGD.
Revised: As long as the adversary only uses gradients of the loss function with respect to the input, we conjecture that it will not ﬁnd signiﬁcantly better local maxima than PGD.

Original: This hypothesis is validated by the experimental evidence provided in Section 4: if we train a network to be robust against PGD adversaries, it becomes robust against a wide range of other attacks as well.
Revised: This hypothesis is validated by the experimental evidence provided in Section 4: if we train a network to be robust against PGD adversaries, it becomes robust against a wide range of other attacks as well.

Original: Of course, our exploration with PGD does not preclude the existence of some isolated maxima with much larger function value.
Revised: Of course, our exploration with PGD does not preclude the existence of some isolated maxima with much larger function value.

Original: However, our experiments suggest that such better local maxima are hard to ﬁnd with ﬁrst order methods: even a large number of random restarts did not ﬁnd function values with signiﬁcantly different loss values (see Appendix A).
Revised: However, our experiments suggest that such better local maxima are hard to ﬁnd with ﬁrst order methods: even a large number of random restarts did not ﬁnd function values with signiﬁcantly different loss values (see Appendix A).

Original: Incorporating the computational power of the adversary into the threat model should be reminiscent of the notion of polynomially bounded adversary that is a cornerstone of modern cryptography.
Revised: Incorporating the computational power of the adversary into the threat model should be reminiscent of the notion of polynomially bounded adversary that is a cornerstone of modern cryptography.

Original: There, this classic threat model allows the adversary to only solve problems that require at most polynomial computation time.
Revised: There, this classic threat model allows the adversary to only solve problems that require at most polynomial computation time.

Original: Here, we employ an optimization-based view on the power of the adversary as it is more suitable in the context of machine learning.
Revised: Here, we employ an optimization-based view on the power of the adversary as it is more suitable in the context of machine learning.

Original: After all, we have not yet developed a thorough understanding of the computational complexity of many recent machine learning problems.
Revised: After all, we have not yet developed a thorough understanding of the computational complexity of many recent machine learning problems.

Original: However, the vast majority of optimization problems in ML is solved with ﬁrst-order methods, and variants of SGD are the most effective way of training deep learning models in particular.
Revised: However, the vast majority of optimization problems in ML is solved with ﬁrst-order methods, and variants of SGD are the most effective way of training deep learning models in particular.

Original: Hence we believe that the class of attacks relying on ﬁrst-order information is, in some sense, universal for the current practice of deep learning.
Revised: Hence we believe that the class of attacks relying on ﬁrst-order information is, in some sense, universal for the current practice of deep learning.

Original: Put together, these two ideas chart the way towards machine learning models with guaranteed robustness.
Revised: Put together, these two ideas chart the way towards machine learning models with guaranteed robustness.

Original: If we train the network to be robust against PGD adversaries, it will be robust against a wide range of attacks that encompasses all current approaches.
Revised: If we train the network to be robust against PGD adversaries, it will be robust against a wide range of attacks that encompasses all current approaches.

Original: In fact, this robustness guarantee would become even stronger in the context of transfer attacks , i.e., attacks in which the adversary does not have a direct access to the target network.
Revised: In fact, this robustness guarantee would become even stronger in the context of transfer attacks , i.e., attacks in which the adversary does not have a direct access to the target network.

Original: Instead, the adversary only has less speciﬁc information such as the (rough) model architecture and the training data set.
Revised: Instead, the adversary only has less speciﬁc information such as the (rough) model architecture and the training data set.

Original: One can view this threat model as an example of “zero order” attacks, i.e., attacks in which the adversary has no direct access to the classiﬁer and is only able to evaluate it on chosen examples without gradient feedback.
Revised: One can view this threat model as an example of “zero order” attacks, i.e., attacks in which the adversary has no direct access to the classiﬁer and is only able to evaluate it on chosen examples without gradient feedback.

Original: Still, even for the case of zero-order attacks, the gradient of the network can be estimated using a ﬁnite differences method, rendering ﬁrst-order attacks also relevant in this context.
Revised: Still, even for the case of zero-order attacks, the gradient of the network can be estimated using a ﬁnite differences method, rendering ﬁrst-order attacks also relevant in this context.

Original: We discuss transferability in Appendix D. We observe that increasing network capacity and strengthening the adversary we train against (FGSM or PGD training, rather than natural training) improves resistance against transfer attacks.
Revised: We discuss transferability in Appendix D. We observe that increasing network capacity and strengthening the adversary we train against (FGSM or PGD training, rather than natural training) improves resistance against transfer attacks.

Original: Also, as expected, the resistance of our best models to such attacks tends to be signiﬁcantly larger than to the (strongest) ﬁrst order attacks.
Revised: Also, as expected, the resistance of our best models to such attacks tends to be signiﬁcantly larger than to the (strongest) ﬁrst order attacks.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Due to the growing body of work on adversarial examples in the context of deep learning networks (Gu & Rigazio, 2014; Fawzi et al., 2015; Torkamani, 2016; Papernot et al., 2016b; Carlini & Wagner, 2016a; Tramèr et al., 2017b; Goodfellow et al., 2014; Kurakin et al., 2016), we focus only on the most related papers here.
Revised: Due to the growing body of work on adversarial examples in the context of deep learning networks (Gu & Rigazio, 2014; Fawzi et al., 2015; Torkamani, 2016; Papernot et al., 2016b; Carlini & Wagner, 2016a; Tramèr et al., 2017b; Goodfellow et al., 2014; Kurakin et al., 2016), we focus only on the most related papers here.

Original: Before we compare our contributions, we remark that robust optimization has been studied outside deep learning for multiple decades.
Revised: Before we compare our contributions, we remark that robust optimization has been studied outside deep learning for multiple decades.

Original: We refer the reader to Ben-Tal et al.
Revised: We refer the reader to Ben-Tal et al.

Original: (2009) for an overview of this ﬁeld.
Revised: (2009) for an overview of this ﬁeld.

Original: Recent work on adversarial training on ImageNet also observed that the model capacity is important for adversarial training Kurakin et al. However, their work was focused on FGSM attacks, since they report the iterative attacks are too expensive computationally and don’t provide any signiﬁcant beneﬁts. In contrast to that, we discover that for the datasets we considered training against iterative adversaries does result ina model that is robust against such adversaries. In Huang et al.
Revised: To the best of our knowledge, in the context of adversarial examples, an explicit formulation of the min-max optimization ﬁrst appeared in Huang et al.

Original: (2015) and Shaham et al.
Revised: (2015), Shaham et al.

Original: (2015) a version of the min-max optimization problem is also considered for adversarial training.
Revised: (2015), and Lyu et al.

Original: Both of these works, however, consider very weak adversaries/methods for solving the maximization problem, mainly relying on linearizing the loss and performing a single step, similar to FGSM.
Revised: All of these works, however, consider very weak adversaries/methods for solving the maximization problem, mainly relying on linearizing the loss and performing a single step, similar to FGSM.

Original: These adversaries do not capture the full range of possible attacks and thus training only against them leaves the resulting classiﬁer vulnerable to more powerful, iterative attacks.
Revised: These adversaries do not capture the full range of possible attacks and thus training only against them leaves the resulting classiﬁer vulnerable to more powerful, iterative attacks.

Original: Additionally, while the experiments in Shaham et al.(2015) produce promising results, they are only evaluated against FGSM. However, FGSM-only evaluations are not fully reliable. One evidence for that is that Shaham et al.
Revised: Recent work on adversarial training on ImageNet also observed that the model capacity is important for adversarial training Kurakin et al. However, their work was focused on FGSM attacks, since they report the iterative attacks are too expensive computationally and don’t provide any signiﬁcant beneﬁts.

Original: (2015) reports 70% accuracy for ε = 0 . 7 , but any adversary that is allowed to perturb each pixel by more than 0.5 can construct a uniformly gray image, thus fooling any classiﬁer.
Revised: In contrast to that, we discover that for the datasets we considered training against iterative adversaries does result in a model that is robust against such adversaries.

Original: A more recent paper Tramèr et al. (2017b) also explores the transferability phenomenon.
Revised: A more recent paper (Tramèr et al., 2017b) also explores the transferability phenomenon.

Original: This exploration focuses mostly on the region around natural examples where the loss is (close to) linear.
Revised: This exploration focuses mostly on the region around natural examples where the loss is (close to) linear.

Original: When large perturbations are allowed, this region does not give a complete picture of the adversarial landscape.
Revised: When large perturbations are allowed, this region does not give a complete picture of the adversarial landscape.

Original: This is conﬁrmed by our experiments, as well as pointed out by Tramèr et al.
Revised: This is conﬁrmed by our experiments, as well as pointed out by Tramèr et al.

Original: (2017a).
Revised: (2017a).

Original: Another recent paper Tramèr et al. (2017a), considers adversarial training using black-box attacks from similar networks in order to increase the robustness of the network against such adversaries.
Revised: Another recent paper (Tramèr et al., 2017a), considers adversarial training using black-box attacks from similar networks in order to increase the robustness of the network against such adversaries.

Original: However, this is not an effective defense against the white-box setting we consider, since a PGD adversary can reliably produce adversarial examples for such networks.
Revised: However, this is not an effective defense against the white-box setting we consider, since a PGD adversary can reliably produce adversarial examples for such networks.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: Our ﬁndings provide evidence that deep neural networks can be made resistant to adversarial attacks.
Revised: Our ﬁndings provide evidence that deep neural networks can be made resistant to adversarial attacks.

Original: As our theory and experiments indicate, we can design reliable adversarial training methods.
Revised: As our theory and experiments indicate, we can design reliable adversarial training methods.

Original: One of the key insights behind this is the unexpectedly regular structure of the underlying optimization task: even though the relevant problem corresponds to the maximization of a highly non-concave function with many distinct local maxima, their values are highly concentrated.
Revised: One of the key insights behind this is the unexpectedly regular structure of the underlying optimization task: even though the relevant problem corresponds to the maximization of a highly non-concave function with many distinct local maxima, their values are highly concentrated.

Original: Overall, our ﬁndings give us hope that adversarially robust deep learning models may be within current reach.
Revised: Overall, our ﬁndings give us hope that adversarially robust deep learning models may be within current reach.

Original: For the MNIST dataset, our networks are very robust, achieving high accuracy for a wide range of powerful adversaries and large perturbations.
Revised: For the MNIST dataset, our networks are very robust, achieving high accuracy for a wide range of powerful adversaries and large perturbations.

Original: Our experiments on CIFAR10 have not reached the same level of performance yet.
Revised: Our experiments on CIFAR10 have not reached the same level of performance yet.

Original: However, our results already show that our techniques lead to signiﬁcant increase in the robustness of the network.
Revised: However, our results already show that our techniques lead to signiﬁcant increase in the robustness of the network.

Original: We believe that further exploring this direction will lead to adversarially robust networks for this dataset.
Revised: We believe that further exploring this direction will lead to adversarially robust networks for this dataset.

Original: A T HE L ANDSCAPE OF A DVERSARIAL E XAMPLES
Revised: A CKNOWLEDGMENTS

Revised: Aleksander M ˛adry, Aleksandar Makelov and Dimitris Tsipras were supported by the NSF Grant No. 1553428, a Google Research Fellowship, and a Sloan Research Fellowship. Ludwig Schmidt was supported by a Google PhD Fellowship.

Revised: Adrian Vladu was supported by the NSF Grants No.

Revised: 1111109 and No. 1553428.

Revised: We thank Wojciech Matusik for kindly providing us with computing resources to perform this work.

Revised: A T HE L ANDSCAPE OF A DVERSARIAL E XAMPLES

Original: The inner problem of the saddle point formulation (2.1) corresponds to ﬁnding an adversarial example for a given network and data point (subject to our attack model).
Revised: The inner problem of the saddle point formulation (2.1) corresponds to ﬁnding an adversarial example for a given network and data point (subject to our attack model).

Original: As this problem requires us to maximize a highly non-concave function, one would expect it to be intractable.
Revised: As this problem requires us to maximize a highly non-concave function, one would expect it to be intractable.

Original: Indeed, this is the conclusion reached by prior work which then resorted to linearizing the inner maximization problem (Huang et al., 2015; Shaham et al., 2015).
Revised: Indeed, this is the conclusion reached by prior work which then resorted to linearizing the inner maximization problem (Huang et al., 2015; Shaham et al., 2015).

Original: As pointed out above, this linearization approach yields well-known methods such as FGSM.
Revised: As pointed out above, this linearization approach yields well-known methods such as FGSM.

Original: While training against FGSM adversaries has shown some successes, recent work also highlights important shortcomings of this one-step approach (Tramèr et al., 2017a).
Revised: While training against FGSM adversaries has shown some successes, recent work also highlights important shortcomings of this one-step approach (Tramèr et al., 2017a).

Original: To understand the inner problem in more detail, we investigate the landscape of local maxima for multiple models on MNIST and CIFAR10.
Revised: To understand the inner problem in more detail, we investigate the landscape of local maxima for multiple models on MNIST and CIFAR10.

Original: The main tool in our experiments is projected gradient descent (PGD), since it is the standard method for large-scale constrained optimization.
Revised: The main tool in our experiments is projected gradient descent (PGD), since it is the standard method for large-scale constrained optimization.

Original: In order to explore a large part of the loss landscape, we re-start PGD from many points in the (cid:96) ∞ balls around data points from the respective evaluation sets.
Revised: In order to explore a large part of the loss landscape, we re-start PGD from many points in the (cid:96) ∞ balls around data points from the respective evaluation sets.

Original: Surprisingly, our experiments show that the inner problem is tractable after all, at least from the perspective of ﬁrst-order methods.
Revised: Surprisingly, our experiments show that the inner problem is tractable after all, at least from the perspective of ﬁrst-order methods.

Original: While there are many local maxima spread widely apart within x i + S , they tend to have very well-concentrated loss values .
Revised: While there are many local maxima spread widely apart within x i + S , they tend to have very well-concentrated loss values .

Original: This echoes the folklore belief that training neural networks is possible because the loss (as a function of model parameters) typically has many local minima with very similar values.
Revised: This echoes the folklore belief that training neural networks is possible because the loss (as a function of model parameters) typically has many local minima with very similar values.

Original: Speciﬁcally, in our experiments we found the following phenomena:
Revised: Speciﬁcally, in our experiments we found the following phenomena:

Original: [Figure] • We observe that the loss achieved by the adversary increases in a fairly consistent way and plateaus rapidly when performing projected (cid:96) ∞ gradient descent for randomly chosen starting points inside 
Revised: • We observe that the loss achieved by the adversary increases in a fairly consistent way and plateaus rapidly when performing projected (cid:96) ∞ gradient descent for randomly chosen starting points inside x + S (see Figure 3).

Original: + S (see Figure 3).
Revised: [Figure]

Original: • Investigating the concentration of maxima further, we observe that over a large number of random restarts, the loss of the ﬁnal iterate follows a well-concentrated distribution without extreme outliers (see Figure 4; we veriﬁed this concentration based on 10 5 restarts).
Revised: • Investigating the concentration of maxima further, we observe that over a large number of random restarts, the loss of the ﬁnal iterate follows a well-concentrated distribution without extreme outliers (see Figure 4; we veriﬁed this concentration based on 10 5 restarts).

Original: [Figure] • To demonstrate that maxima are noticeably distinct, we also measured the (cid:96) 2 distance and angles between all pairs of them and observed that distances are distributed close to the expected distance between two random points in the (cid:96) ∞ ball, and angles are close to 90 ◦ .
Revised: • To demonstrate that maxima are noticeably distinct, we also measured the (cid:96) 2 distance and angles between all pairs of them and observed that distances are distributed close to the expected distance between two random points in the (cid:96) ∞ ball, and angles are close to 90 ◦ .

Original: Along the line segment between local maxima, the loss is convex, attaining its maximum at the endpoints and is reduced by a constant factor in the middle.
Revised: Along the line segment between local maxima, the loss is convex, attaining its maximum at the endpoints and is reduced by a constant factor in the middle.

Original: Nevertheless, for the entire segment, the loss is considerably higher than that of a random point.
Revised: Nevertheless, for the entire segment, the loss is considerably higher than that of a random point.

Original: • Finally, we observe that the distribution of maxima suggests that the recently developed subspace view of adversarial examples is not fully capturing the richness of attacks (Tramèr et al , 2017b)
Revised: Finally, we observe that the distribution of maxima suggests that the recently developed subspace view of adversarial examples is not fully capturing the richness of attacks (Tramèr et al., 2017b).

Original: In particular we observe adversarial perturbations 
Revised: In particular, we observe adversarial perturbations with negative inner product with the gradient

Original: with
Revised: [Figure]

Original: negative inner product with the gradient of the example, and deteriorating overall correlation with the gradient direction as the scale of perturbation increases.
Revised: of the example, and deteriorating overall correlation with the gradient direction as the scale of perturbation increases.

Original: B N ETWORK C APACITY AND A DVERSARIAL R OBUSTNESS
Revised: B N ETWORK C APACITY AND A DVERSARIAL R OBUSTNESS

Original: For a ﬁxed set S of possible perturbations, the value of the problem (2.1) is entirely dependent on the architecture of the classiﬁer we are learning.
Revised: For a ﬁxed set S of possible perturbations the value of the problem (2.1) is entirely dependent on the architecture of the classiﬁer we are learning.

Original: Consequently, the architectural capacity of the model becomes a major factor affecting its overall performance.
Revised: Consequently, the architectural capacity of the model becomes a major factor affecting its overall performance.

Original: At a high level, classifying examples in a robust way requires a stronger classiﬁer, since the presence of adversarial examples changes the decision boundary of the problem to a more complicated one (see Figure 5 for an illustration).
Revised: At a high level, classifying examples in a robust way requires a stronger classiﬁer, since the presence of adversarial examples changes the decision boundary of the problem to a more complicated one (see Figure 5 for an illustration).

Revised: simple decision boundary not separate

Original: Our experiments verify that capacity is crucial for robustness, as well as for the ability to successfully train against strong adversaries.
Revised: Our experiments verify that capacity is crucial for robustness, as well as for the ability to successfully train against strong adversaries.

Original: For the MNIST dataset, we consider a simple convolutional network and study how its behavior changes against different adversaries as we keep doubling the size of network (i.e. double the number of convolutional ﬁlters and the size of the fully connected layer).
Revised: For the MNIST dataset, we consider a simple convolutional network and study how its behavior changes against different adversaries as we keep doubling the size of network (i.e. double the number of convolutional ﬁlters and the size of the fully connected layer).

Original: The initial network has a convolutional layer with 2 ﬁlters, followed by another convolutional layer with 4 ﬁlters, and a fully connected hidden layer with 64 units.
Revised: The initial network has a convolutional layer with 2 ﬁlters, followed by another convolutional layer with 4 ﬁlters, and a fully connected hidden layer with 64 units.

Original: Convolutional layers are followed by 2 × 2 max-pooling layers and adversarial examples are constructed with ε = 0 .
Revised: Convolutional layers are followed by 2 × 2 max-pooling layers and adversarial examples are constructed with ε = 0 .

Original: The results are in
Revised: The results are in

Original: Figure 6.
Revised: Figure 6.

Original: For the CIFAR10 dataset, we used the Resnet model
Revised: For the CIFAR10 dataset, we used the Resnet model

Original: He et al.
Revised: He et al.

Original: (2016); TFM (2017).
Revised: (2016); TFM (2017).

Original: We performed data augmentation using random crops and ﬂips, as well as per image standarization.
Revised: We performed data augmentation using random crops and ﬂips, as well as per image standarization.

Original: To increase the capacity, we modiﬁed the network incorporating wider layers by a factor of 10.
Revised: To increase the capacity, we modiﬁed the network incorporating wider layers by a factor of 10.

Original: This results in a network with 5 residual units with (16, 160, 320, 640) ﬁlters each.
Revised: This results in a network with 5 residual units with (16, 160, 320, 640) ﬁlters each.

Original: This network can achieve an accuracy of 95.2% when trained with natural examples.
Revised: This network can achieve an accuracy of 95.2% when trained with natural examples.

Original: Adversarial examples were constructed with ε = 8 .
Revised: Adversarial examples were constructed with ε = 8 .

Original: Results on capacity experiments appear in Figure 6.
Revised: Results on capacity experiments appear in Figure 6.

Original: We observe the following phenomena:
Revised: We observe the following phenomena:

Original: Capacity alone helps.
Revised: Capacity alone helps.

Original: We observe that increasing the capacity of the network when training using only natural examples (apart from increasing accuracy on these examples) increases the robustness against one-step perturbations.
Revised: We observe that increasing the capacity of the network when training using only natural examples (apart from increasing accuracy on these examples) increases the robustness against one-step perturbations.

Original: This effect is greater when considering adversarial examples with smaller ε .
Revised: This effect is greater when considering adversarial examples with smaller ε .

Original: FGSM adversaries don’t increase robustness (for large ε ).
Revised: FGSM adversaries don’t increase robustness (for large ε ).

Original: When training the network using adversarial examples generated with the FGSM, we observe that the network overﬁts to these adversarial examples.
Revised: When training the network using adversarial examples generated with the FGSM, we observe that the network overﬁts to these adversarial examples.

Original: This behavior is known as label leaking Kurakin et al.
Revised: This behavior is known as label leaking Kurakin et al.

Original: and stems from the fact that the adversary produces a very restricted set of adversarial examples that the network can overﬁt to.
Revised: and stems from the fact that the adversary produces a very restricted set of adversarial examples that the network can overﬁt to.

Original: These networks have poor performance on natural examples and don’t exhibit any kind of robustness against PGD adversaries.
Revised: These networks have poor performance on natural examples and don’t exhibit any kind of robustness against PGD adversaries.

Original: For the case of smaller ε the loss is ofter linear enough in the (cid:96) ∞ ball around natural examples, that FGSM ﬁnds adversarial examples close to those found by
Revised: For the case of smaller ε the loss is ofter linear enough in the (cid:96) ∞ ball around natural examples, that FGSM ﬁnds adversarial examples close to those found by

Original: PGD thus being a reasonable adversary to train against.
Revised: PGD thus being a reasonable adversary to train against.

Original: Weak models may fail to learn non-trivial classiﬁers.
Revised: Weak models may fail to learn non-trivial classiﬁers.

Original: In the case of small capacity networks, attempting to train against a strong adversary (PGD) prevents the network from learning anything meaningful.
Revised: In the case of small capacity networks, attempting to train against a strong adversary (PGD) prevents the network from learning anything meaningful.

Original: The network converges to always predicting a ﬁxed class, even though it could converge to an accurate classiﬁer through natural training.
Revised: The network converges to always predicting a ﬁxed class, even though it could converge to an accurate classiﬁer through natural training.

Original: The small capacity of the network forces the training procedure to sacriﬁce performance on natural examples in order to provide any kind of robustness against adversarial inputs.
Revised: The small capacity of the network forces the training procedure to sacriﬁce performance on natural examples in order to provide any kind of robustness against adversarial inputs.

Original: The value of the saddle point problem decreases as we increase the capacity.
Revised: The value of the saddle point problem decreases as we increase the capacity.

Original: Fixing an adversary model, and training against it, the value of (2.1) drops as capacity increases, indicating the the model can ﬁt the adversarial examples increasingly well.
Revised: Fixing an adversary model, and training against it, the value of (2.1) drops as capacity increases, indicating the the model can ﬁt the adversarial examples increasingly well.

Original: More capacity and stronger adversaries decrease transferability.
Revised: More capacity and stronger adversaries decrease transferability.

Original: Either increasing the capacity of the network, or using a stronger method for the inner optimization problem reduces the effectiveness of transferred adversarial inputs.
Revised: Either increasing the capacity of the network, or using a stronger method for the inner optimization problem reduces the effectiveness of transferred adversarial inputs.

Original: We validate this experimentally by observing that the correlation between gradients from the source and the transfer network, becomes less signiﬁcant as capacity increases.
Revised: We validate this experimentally by observing that the correlation between gradients from the source and the transfer network, becomes less signiﬁcant as capacity increases.

Original: We describe our experiments in Appendix D.
Revised: We describe our experiments in Appendix D.

Original: MNIST
Revised: Recall that our goal is to minimize the value of the saddle point problem min θ ρ ( θ ) , where [Equation] In practice, we don’t have access to the distribution D so both the gradients and the value of ρ ( θ ) will be computed using sampled input points.

Revised: Therefore we can consider –without loss of generality– the case of a single random example x with label y , in which case the problem becomes

Original: [Figure]
Revised: [Equation]

Revised: where

Revised: [Equation]

Revised: If we assume that the loss L is continuously differentiable in θ , we can compute a descent direction for θ by utilizing the classical theorem of Danskin.

Original: {+MNIST    
+}
Revised: {+MNIST    
+}

Revised: Capacity scale

Original: {+A cc u r ac y    
+}
Revised: {+A cc u r ac y    
+}

Original: {+Capacity scale    
+}
Revised: {+Capacity scale    
+}

Revised: Capacity scale     0 .0 .

Revised: Capacity scale

Revised: A v e r a g e l o ss

Revised: NaturalFGSMPGD

Original: CIFAR
Revised: CIFAR

Original: Simple Wide
Revised: Simple Wide

Original: Natural 92.7% 95.2%
Revised: Natural 92.7% 95.2%

Original: FGSM 27.5% 32.7%
Revised: FGSM 27.5% 32.7%

Original: PGD 0.8% 3.5%
Revised: PGD 0.8% 3.5%

Original: Simple Wide 87.4% 90.3% 90.9% 95.1% 0.0% 0.0%
Revised: Simple Wide 87.4% 90.3% 90.9% 95.1% 0.0% 0.0%

Original: Simple Wide 79.4% 87.3% 51.7% 56.1% 43.7% 45.8%
Revised: Simple Wide 79.4% 87.3% 51.7% 56.1% 43.7% 45.8%

Original: Simple
Revised: Simple

Original: Wide 0.00357 0.
Revised: Wide 0.00357 0.

Original: [Figure]
Revised: [Figure]

Original: C S TATEMENT AND A PPLICATION OF D ANSKIN ’ S T HEOREM

Original: Recall that our goal is to minimize the value of the saddle point problem

Original: [Equation]

Original: where

Original: [Equation]

Original: In practice, we don’t have access to the distribution D so both the gradients and the value of ρ ( θ ) will be computed using sampled input points.

Original: Therefore we can consider –without loss of generality– the case of a single random example x with label y , in which case the problem becomes

Original: [Equation]

Original: where

Original: [Equation]

Original: If we assume that the loss L is continuously differentiable in θ , we can compute a descent direction for θ by utilizing the classical theorem of Danskin.

Original: Theorem C.1 (Danskin) .
Revised: Theorem C.1 (Danskin) .

Original: Let S be nonempty compact topological space and g : R n × S
Revised: Let S be nonempty compact topological space and g : R n × S

Original: → R be such that g ( · , δ ) is differentiable for every δ ∈ S and ∇ θ g ( θ, δ ) is continuous on R n × S .
Revised: → R be such that g ( · , δ ) is differentiable for every δ ∈ S and ∇ θ g ( θ, δ ) is continuous on R n × S .

Original: Also, let δ ∗ ( θ )
Revised: Also, let δ ∗ ( θ )

Original: =
Revised: =

Original: { δ ∈ arg max δ ∈S g ( θ, δ ) } .
Revised: { δ ∈ arg max δ ∈S g ( θ, δ ) } .

Original: Then the corresponding max-function
Revised: Then the corresponding max-function

Original: [Equation]
Revised: [Equation]

Original: is locally Lipschitz continuous, directionally differentiable, and its directional derivatives satisfy
Revised: is locally Lipschitz continuous, directionally differentiable, and its directional derivatives satisfy

Original: [Equation]
Revised: [Equation]

Original: In particular, if for some θ ∈ R n the set δ ∗ ( θ ) =
Revised: In particular, if for some θ ∈ R n the set δ ∗ ( θ ) =

Original: { δ ∗ θ } is a singleton, the the max-function is differentiable at θ and
Revised: { δ ∗ θ } is a singleton, the the max-function is differentiable at θ and

Original: [Equation]
Revised: [Equation]

Original: The intution behind the theorem is that since gradients are local objects, and the function φ ( θ ) is locally the same as g ( θ, δ ∗ θ ) their gradients will be the same.
Revised: The intution behind the theorem is that since gradients are local objects, and the function φ ( θ ) is locally the same as g ( θ, δ ∗ θ ) their gradients will be the same.

Original: The theorem immediately gives us the following corollary, stating the we can indeed compute gradients for the saddle point by computing gradients at the inner optimizers.
Revised: The theorem immediately gives us the following corollary, stating the we can indeed compute gradients for the saddle point by computing gradients at the inner optimizers.

Original: Corollary C.2.
Revised: Corollary C.2.

Original: Let δ be such that δ ∈ S and is a maximizer for max δ L ( θ, x + δ, y ) .
Revised: Let δ be such that δ ∈ S and is a maximizer for max δ L ( θ, x + δ, y ) .

Original: Then, as long as it is nonzero, −∇ θ L ( θ, x + δ, y ) is a descent direction for φ ( θ ) = max δ ∈S L ( θ, x + δ, y ) .
Revised: Then, as long as it is nonzero, −∇ θ L ( θ, x + δ, y ) is a descent direction for φ ( θ ) = max δ ∈S L ( θ, x + δ, y ) .

Original: Proof of Corollary C.2.
Revised: Proof of Corollary C.2.

Original: We apply Theorem C.1 to g ( θ, δ ) :
Revised: We apply Theorem C.1 to g ( θ, δ ) :

Original: = L ( θ, x + δ, y ) and S =
Revised: = L ( θ, x + δ, y ) and S =

Original: B (cid:107)·(cid:107) ( ε ) .
Revised: B (cid:107)·(cid:107) ( ε ) .

Original: We see that the directional derivative in the direction of h = ∇ θ L ( θ, x [Equation] δ [Equation]
Revised: We see that the directional derivative in the direction of h = ∇ θ L ( θ, x + δ, y ) satisﬁes

Original: y ) satisﬁes

Original: [Equation]
Revised: [Equation]

Original: If this gradient is nonzero, then the inequality above is strict.
Revised: If this gradient is nonzero, then the inequality above is strict.

Original: Therefore it gives a descent direction.
Revised: Therefore it gives a descent direction.

Original: A technical issue is that, since we use ReLU and max-pooling units in our neural network architecture, the loss function is not continuously differentiable.
Revised: A technical issue is that, since we use ReLU and max-pooling units in our neural network architecture, the loss function is not continuously differentiable.

Original: Nevertheless, since the set of discontinuities has measure zero, we can assume that this will not be an issue in practice, as we will never encounter the problematic points.
Revised: Nevertheless, since the set of discontinuities has measure zero, we can assume that this will not be an issue in practice, as we will never encounter the problematic points.

Original: Another technical issue is that, due to the not concavity of the inner problem, we are not able to compute global maximizers, since PGD will converge to local maxima.
Revised: Another technical issue is that, due to the not concavity of the inner problem, we are not able to compute global maximizers, since PGD will converge to local maxima.

Original: In such cases, we can consider a subset S (cid:48) of S such that the local maximum is a global maximum in the region S (cid:48) .
Revised: In such cases, we can consider a subset S (cid:48) of S such that the local maximum is a global maximum in the region S (cid:48) .

Original: Applying the theorem for S (cid:48) gives us that the gradient corresponds to a descent direction for the saddle point problem when the adversary is constrained in S (cid:48) .
Revised: Applying the theorem for S (cid:48) gives us that the gradient corresponds to a descent direction for the saddle point problem when the adversary is constrained in S (cid:48) .

Original: Therefore if the inner maximum is a true adversarial example for the network, then SGD using the gradient at that point will decrease the loss value at this particular adversarial examples, thus making progress towards a robust model.
Revised: Therefore if the inner maximum is a true adversarial example for the network, then SGD using the gradient at that point will decrease the loss value at this particular adversarial examples, thus making progress towards a robust model.

Original: These arguments suggest that the conclusions of the theorem are still valid in our saddle point problem, and –as our experiments conﬁrm– we can solve it reliably.
Revised: These arguments suggest that the conclusions of the theorem are still valid in our saddle point problem, and –as our experiments conﬁrm– we can solve it reliably.

Original: D T RANSFERABILITY
Revised: D T RANSFERABILITY

Original: A lot of recent literature on adversarial training discusses the phenomenon of transferability Goodfellow et al.
Revised: A lot of recent literature on adversarial training discusses the phenomenon of transferability Goodfellow et al.

Original: (2014); Kurakin et al.
Revised: (2014); Kurakin et al.

Original: ; Tramèr et al.
Revised: ; Tramèr et al.

Original: (2017b), i.e. adversarial examples transfer between differently trained networks.
Revised: (2017b), i.e. adversarial examples transfer between differently trained networks.

Original: This raises concerns for practical applications, since it suggests that deep networks are extremely vulnerable to attacks, even when there is no direct access to the target network .
Revised: This raises concerns for practical applications, since it suggests that deep networks are extremely vulnerable to attacks, even when there is no direct access to the target network .

Original: This phenomenon is further conﬁrmed by our current experiments.
Revised: This phenomenon is further conﬁrmed by our current experiments.

Original: Moreover, we notice that the extent to which adversarial examples transfer decreases as we increase either network capacity or the power of the adversary used for training the network.
Revised: Moreover, we notice that the extent to which adversarial examples transfer decreases as we increase either network capacity or the power of the adversary used for training the network.

Original: This serves as evidence for the fact that the transferability phenomenon can be alleviated by using high capacity networks in conjunction with strong oracles for the inner optimization problem.
Revised: This serves as evidence for the fact that the transferability phenomenon can be alleviated by using high capacity networks in conjunction with strong oracles for the inner optimization problem.

Original: MNIST.
Revised: MNIST.

Original: In an attempt to understand these phenomena we inspect the loss functions corresponding to the trained models we used for testing transferability.
Revised: In an attempt to understand these phenomena we inspect the loss functions corresponding to the trained models we used for testing transferability.

Original: More precisely, we compute angles between gradients of the loss functions evaluated over a large set of input examples, and plot their distribution.
Revised: More precisely, we compute angles between gradients of the loss functions evaluated over a large set of input examples, and plot their distribution.

Original: Similarly, we plot the value of the loss functions between clean and perturbed examples for both the source and transfer networks.
Revised: Similarly, we plot the value of the loss functions between clean and perturbed examples for both the source and transfer networks.

Original: In Figure 8 we plot our experimental ﬁndings on the MNIST dataset for ε = 0 .
Revised: In Figure 8 we plot our experimental ﬁndings on the MNIST dataset for ε = 0 .

Original: We consider a naturally trained large network (two convolutional layers of sizes 32 and 64 , and a fully connected layer of size 1024 ), which we train twice starting with different initializations.
Revised: We consider a naturally trained large network (two convolutional layers of sizes 32 and 64 , and a fully connected layer of size 1024 ), which we train twice starting with different initializations.

Original: We plot the distribution of angles between gradients for the same test image in the two resulting networks (orange histograms), noting that they are somewhat correlated.
Revised: We plot the distribution of angles between gradients for the same test image in the two resulting networks (orange histograms), noting that they are somewhat correlated.

Original: As opposed to this, we see that pairs of gradients for random pairs of inputs for one architecture are as uncorrelated as they can be (blue histograms), since the distribution of their angles looks Gaussian.
Revised: As opposed to this, we see that pairs of gradients for random pairs of inputs for one architecture are as uncorrelated as they can be (blue histograms), since the distribution of their angles looks Gaussian.

Original: Next, we run the same experiment on a naturally trained very large network (two convolutional layers of sizes 64 and 128 , and a fully connected layer of size 1024 ).
Revised: Next, we run the same experiment on a naturally trained very large network (two convolutional layers of sizes 64 and 128 , and a fully connected layer of size 1024 ).

Original: We notice a mild increase in classiﬁcation accuracy for transferred examples.
Revised: We notice a mild increase in classiﬁcation accuracy for transferred examples.

Original: Finally, we repeat the same set of experiments, after training the large and very large networks against the FGSM adversary.
Revised: Finally, we repeat the same set of experiments, after training the large and very large networks against the FGSM adversary.

Original: We notice that gradients between the two architectures become signiﬁcantly less correlated.
Revised: We notice that gradients between the two architectures become signiﬁcantly less correlated.

Original: Also, the classiﬁcation accuracy for transferred examples increases signiﬁcantly compared to the naturally trained networks.
Revised: Also, the classiﬁcation accuracy for transferred examples increases signiﬁcantly compared to the naturally trained networks.

Original: We further plot how the value of the loss function changes when moving from the natural input towards the adversarially perturbed input (in Figure 8 we show these plots for four images in the
Revised: We further plot how the value of the loss function changes when moving from the natural input towards the adversarially perturbed input (in Figure 8 we show these plots for four images in the

Original: MNIST test dataset), for each pair of networks we considered.
Revised: MNIST test dataset), for each pair of networks we considered.

Original: We observe that, while for the naturally trained networks, when moving towards the perturbed point, the value of the loss function on the transfer architecture tends to start increasing soon after it starts increasing on the source architecture.
Revised: We observe that, while for the naturally trained networks, when moving towards the perturbed point, the value of the loss function on the transfer architecture tends to start increasing soon after it starts increasing on the source architecture.

Original: In contrast, for the stronger models, the loss function on the transfer network tends to start increasing later, and less aggressively.
Revised: In contrast, for the stronger models, the loss function on the transfer network tends to start increasing later, and less aggressively.

Original: CIFAR10.
Revised: CIFAR10.

Original: For the CIFAR10 dataset, we investigate the transferability of the FGSM and PGD adversaries between our simple and wide architectures, each trained on natural, FGSM and PGD examples.
Revised: For the CIFAR10 dataset, we investigate the transferability of the FGSM and PGD adversaries between our simple and wide architectures, each trained on natural, FGSM and PGD examples.

Original: Transfer accuracies for the FGSM adversary and PGD adversary between all pairs of such conﬁgurations (model + training method) with independently random weight initialization are given in tables 3 and 4 respectively.
Revised: Transfer accuracies for the FGSM adversary and PGD adversary between all pairs of such conﬁgurations (model + training method) with independently random weight initialization are given in tables 3 and 4 respectively.

Original: The results exhibit the following trends:
Revised: The results exhibit the following trends:

Original: • Stronger adversaries decrease transferability: In particular, transfer attacks between two PGD-trained models are less successful than transfer attacks between their naturally-trained counterparts.
Revised: • Stronger adversaries decrease transferability: In particular, transfer attacks between two PGD-trained models are less successful than transfer attacks between their naturally-trained counterparts.

Original: Moreover, adding PGD training helps with transferability from all adversarial datasets, except for those with source a PGD-trained model themselves.
Revised: Moreover, adding PGD training helps with transferability from all adversarial datasets, except for those with source a PGD-trained model themselves.

Original: This applies to both FGSM attacks and PGD attacks.
Revised: This applies to both FGSM attacks and PGD attacks.

Original: • Capacity decreases transferability: In particular, transfer attacks between two PGDtrained wide networks are less successful than transfer attacks between their simple PGDtrained counterparts.
Revised: • Capacity decreases transferability: In particular, transfer attacks between two PGDtrained wide networks are less successful than transfer attacks between their simple PGDtrained counterparts.

Original: Moreover, with few close exceptions, changing the architecture from simple to wide (and keeping the training method the same) helps with transferability from all adversarial datasets.
Revised: Moreover, with few close exceptions, changing the architecture from simple to wide (and keeping the training method the same) helps with transferability from all adversarial datasets.

Original: We additionally plotted how the loss of a network behaves in the direction of FGSM and PGD examples obtained from itself and an independently trained copy; results for the simple naturally trained network and the wide PGD trained network are given in Table 7.
Revised: We additionally plotted how the loss of a network behaves in the direction of FGSM and PGD examples obtained from itself and an independently trained copy; results for the simple naturally trained network and the wide PGD trained network are given in Table 7.

Original: As expected, we observe the following phenomena:
Revised: As expected, we observe the following phenomena:

Original: • sometimes, the FGSM adversary manages to increase loss faster near the natural example, but as we move towards the boundary of the (cid:96) ∞ box of radius ε , the PGD attack always achieves higher loss.
Revised: • sometimes, the FGSM adversary manages to increase loss faster near the natural example, but as we move towards the boundary of the (cid:96) ∞ box of radius ε , the PGD attack always achieves higher loss.

Original: • the transferred attacks do worse than their white-box counterparts in terms of increasing the loss; • and yet, the transferred PGD attacks dominate the white-box FGSM attacks for the naturally trained network (and sometimes for the PGD-trained one too).
Revised: • the transferred attacks do worse than their white-box counterparts in terms of increasing the loss; • and yet, the transferred PGD attacks dominate the white-box FGSM attacks for the naturally trained network (and sometimes for the PGD-trained one too).

Original: [Table]
Revised: [Table]

Revised: training)

Revised: Simple (FGSM training)

Revised: Simple (PGD training)

Original: [Table]
Revised: Wide (natural training) Wide (FGSM training) Wide (PGD training) Simple (natural training) 32.9% 74.0% 73.7% 27.6% 71.8% 76.6%

Original: [Table]
Revised: Simple (FGSM training) 64.2% 90.7% 60.9% 61.5% 90.2% 67.3% Simple (PGD training) 77.1% 78.1% 60.2% 77.0% 77.9% 66.3% Wide (natural training) 34.9% 78.7% 80.2% 21.3% 75.8% 80.6% Wide (FGSM training) 64.5% 93.6% 69.1% 53.7% 92.2% 72.8%

Revised: Wide (PGD training) 85.8% 86.6% 73.3% 85.6% 86.2% 67.0%

Original: [Table] 
Revised: [Table] [Table] [Table]

Revised: Simple (PGD training)

Original: [Table]
Revised: [Table]

Original: Clean
Revised: Wide (natural training) 95.2% 32.7% 25.1% 4.1% 3.5% Wide (FGSM training) 90.3% 95.1% 95.0% 0.0% 0.0% Wide (PGD training) 87.3% 56.1% 60.

Original: [Table]
Revised: [Table]

Revised: Source

Original: [Figure]
Revised: [Figure]

Revised: Clean 99.

Revised: [Table]

Revised: [Table]

Revised: FGSM

Original: [Table]
Revised: [Table]

Original: Clean
Revised: PGD [Table] [Table] Large

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: Clean 
Revised: Clean 99. [Table] [Table] FGSM

Revised: [Table]

Revised: [Table]

Revised: PGD

Original: [Table]
Revised: [Table]

Revised: Very

Revised: [Table]

Revised: [Table]

Original: [Figure]
Revised: [Figure]

Revised: Clean 92.

Original: [Table] 
Revised: [Table] [Table] [Figure] [Table]

Revised: PGD

Original:  [Figure] 
Revised: [Table] [Figure] [Table] Large

Original: [Table]
Revised: [Table]

Revised: [Table]

Original: [Figure]
Revised: [Figure]

Revised: Source Transfer

Revised: Clean 96.

Original: [Table]
Revised: [Table]

Revised: [Figure]

Original: Clean
Revised: [Table] [Figure] [Table] [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: PGD [Table] [Table] Very

Original: [Table]
Revised: [Table]

Revised: training

Original: [Figure]
Revised: [Figure]

Original: E MNIST I NSPECTION
Revised: E MNIST I NSPECTION

Original: The robust MNIST model described so far is small enough that we can visually inspect most of its parameters.
Revised: The robust MNIST model described so far is small enough that we can visually inspect most of its parameters.

Original: Doing so will allow us to understand how it is different from a naturally trained variant and what are the general characteristics of a network that is robust against (cid:96) ∞ adversaries.
Revised: Doing so will allow us to understand how it is different from a naturally trained variant and what are the general characteristics of a network that is robust against (cid:96) ∞ adversaries.

Original: We will compare three different networks: a naturally trained model, and two adversarially trained ones.
Revised: We will compare three different networks: a naturally trained model, and two adversarially trained ones.

Original: The latter two models are identical, modulo the random weight initialization, and were used as the public and secret models used for our robustness challenge.
Revised: The latter two models are identical, modulo the random weight initialization, and were used as the public and secret models used for our robustness challenge.

Original: Initially, we examine the ﬁrst convolutional layer of each network.
Revised: Initially, we examine the ﬁrst convolutional layer of each network.

Original: We observe that the robust models only utilize 3 out of the total 32 ﬁlters, and for each of these ﬁlters only one weight is non-zero.
Revised: We observe that the robust models only utilize 3 out of the total 32 ﬁlters, and for each of these ﬁlters only one weight is non-zero.

Original: By doing so, the convolution degrades into a scaling of the original image.
Revised: By doing so, the convolution degrades into a scaling of the original image.

Original: Combined with the bias and the ReLU that follows, this results in a thresholding ﬁlter , or equivalently ReLU ( αx − β ) for some constants α , β .
Revised: Combined with the bias and the ReLU that follows, this results in a thresholding ﬁlter , or equivalently ReLU ( αx − β ) for some constants α , β .

Original: From the perspective of adversarial robustness, thresholding ﬁlters are immune to any perturbations on pixels with value less than β − ε .
Revised: From the perspective of adversarial robustness, thresholding ﬁlters are immune to any perturbations on pixels with value less than β − ε .

Original: We visualize a sample of the ﬁlters in Figure(plots a, c, and e).
Revised: We visualize a sample of the ﬁlters in Figure(plots a, c, and e).

Original: Having observed that the ﬁrst layer of the network essentially maps the original image to three copies thresholded at different values, we examine the second convolutional layer of the classiﬁer.
Revised: Having observed that the ﬁrst layer of the network essentially maps the original image to three copies thresholded at different values, we examine the second convolutional layer of the classiﬁer.

Original: Again, the ﬁlter weights are relatively sparse and have a signiﬁcantly wider value range than the naturally trained version.
Revised: Again, the ﬁlter weights are relatively sparse and have a signiﬁcantly wider value range than the naturally trained version.

Original: Since only three channels coming out of the ﬁrst layer matter, is follows (and is veriﬁed) that the only relevant convolutional ﬁlters are those that interact with these three channels.
Revised: Since only three channels coming out of the ﬁrst layer matter, is follows (and is veriﬁed) that the only relevant convolutional ﬁlters are those that interact with these three channels.

Original: We visualize a sample of the ﬁlters in Figure 9 (plots b, d, and f).
Revised: We visualize a sample of the ﬁlters in Figure 9 (plots b, d, and f).

Original: Finally, we examine the softmax/output layer of the network.
Revised: Finally, we examine the softmax/output layer of the network.

Original: While the weights seem to be roughly similar between all three version of the network, we notice a signiﬁcant difference in the class biases.
Revised: While the weights seem to be roughly similar between all three version of the network, we notice a signiﬁcant difference in the class biases.

Original: The adversarially trained networks heavily utilize class biases (far from uniform), and do so in a way very similar to each other.
Revised: The adversarially trained networks heavily utilize class biases (far from uniform), and do so in a way very similar to each other.

Original: A plausible explanation is that certain classes tend to be very vulnerable to adversarial perturbations, and the network learns to be more conservative in predicting them.
Revised: A plausible explanation is that certain classes tend to be very vulnerable to adversarial perturbations, and the network learns to be more conservative in predicting them.

Original: The plots can be found in Figure 10.
Revised: The plots can be found in Figure 10.

Original: All of the “tricks” described so far seem intuitive to a human and would seem reasonable directions when trying to increase the adversarial robustness of a classiﬁer.
Revised: All of the “tricks” described so far seem intuitive to a human and would seem reasonable directions when trying to increase the adversarial robustness of a classiﬁer.

Original: We emphasize the none of these modiﬁcations were hard-coded in any way and they were all learned solely through adversarial training .
Revised: We emphasize the none of these modiﬁcations were hard-coded in any way and they were all learned solely through adversarial training .

Original: We attempted to manually introduce these modiﬁcations ourselves, aiming to achieve adversarial robustness without adversarial training, but with no success.
Revised: We attempted to manually introduce these modiﬁcations ourselves, aiming to achieve adversarial robustness without adversarial training, but with no success.

Original: A simple PGD adversary could fool the resulting models on all the test set examples.
Revised: A simple PGD adversary could fool the resulting models on all the test set examples.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: F
Revised: F

Original: S UPPLEMENTARY F IGURES
Revised: S UPPLEMENTARY F IGURES

Original: [Figure]

Original: [Figure]

Original: Adversarial

Original: [Figure]

Original: [Figure]

Original: Adversarial

Original: [Figure]
Revised: [Figure]

Original: Adversarial

Original: [Figure]
Revised: [Figure]

Original: Adversarial

Original: [Figure]