Original: SOAR: S ECOND -O RDER A DVERSARIAL R EGULARIZATION
Revised: SOAR: S ECOND -O RDER A DVERSARIAL R EGULARIZATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Adversarial training is a common approach to improving the robustness of deep neural networks against adversarial examples.
Revised: Adversarial training is a common approach to improving the robustness of deep neural networks against adversarial examples.

Original: In this work, we propose a novel regularization approach as an alternative.
Revised: In this work, we propose a novel regularization approach as an alternative.

Original: To derive the regularizer, we formulate the adversarial robustness problem under the robust optimization framework and approximate the loss function using a second-order Taylor series expansion.
Revised: To derive the regularizer, we formulate the adversarial robustness problem under the robust optimization framework and approximate the loss function using a second-order Taylor series expansion.

Original: Our proposed second-order adversarial regularizer (SOAR) is an upper bound based on the Taylor approximation of the inner-max in the robust optimization objective.
Revised: Our proposed second-order adversarial regularizer (SOAR) is an upper bound based on the Taylor approximation of the inner-max in the robust optimization objective.

Original: We empirically show that the proposed method improves the robustness of networks against the (cid:96) ∞ and (cid:96) 2 bounded perturbations on CIFAR-10 and SVHN.
Revised: We empirically show that the proposed method improves the robustness of networks against the (cid:96) ∞ and (cid:96) 2 bounded perturbations on CIFAR-10 and SVHN.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Adversarial training (Szegedy et al., 2013) is the standard approach for improving the robustness of deep neural networks (DNN), or any other model, against adversarial examples.
Revised: Adversarial training (Szegedy et al., 2013) is the standard approach for improving the robustness of deep neural networks (DNN), or any other model, against adversarial examples.

Original: It is a data augmentation method that adds adversarial examples to the training set and updates the network with newly added data points.
Revised: It is a data augmentation method that adds adversarial examples to the training set and updates the network with newly added data points.

Original: Intuitively, this procedure encourages the DNN not to make the same mistakes against an adversary.
Revised: Intuitively, this procedure encourages the DNN not to make the same mistakes against an adversary.

Original: By adding sufﬁciently enough adversarial examples, the network gradually becomes robust to the attack it was trained on.
Revised: By adding sufﬁciently enough adversarial examples, the network gradually becomes robust to the attack it was trained on.

Original: One of the challenges with such a data augmentation approach is the tremendous amount of additional data required for learning a robust model.
Revised: One of the challenges with such a data augmentation approach is the tremendous amount of additional data required for learning a robust model.

Original: Schmidt et al.
Revised: Schmidt et al.

Original: (2018) show that under a Gaussian data model, the sample complexity of robust generalization is √ d times larger than that of standard generalization.
Revised: (2018) show that under a Gaussian data model, the sample complexity of robust generalization is √ d times larger than that of standard generalization.

Original: They further suggest that current datasets (e.g., CIFAR-10) may not be large enough to attain higher adversarial accuracy.
Revised: They further suggest that current datasets (e.g., CIFAR-10) may not be large enough to attain higher adversarial accuracy.

Original: A data augmentation procedure, however, is an indirect way to improve the robustness of a DNN.
Revised: A data augmentation procedure, however, is an indirect way to improve the robustness of a DNN.

Original: Our proposed alternative is to deﬁne a regularizer that penalizes DNN parameters prone to attacks.
Revised: Our proposed alternative is to deﬁne a regularizer that penalizes DNN parameters prone to attacks.

Original: Minimizing the regularized loss function leads to estimators robust to adversarial examples.
Revised: Minimizing the regularized loss function leads to estimators robust to adversarial examples.

Original: Adversarial training and our proposal can both be formulated in terms of robust optimization framework for adversarial robustness (Ben-Tal et al., 2009; Madry et al., 2018; Wong & Kolter, 2018;
Revised: Adversarial training and our proposal can both be formulated in terms of robust optimization framework for adversarial robustness (Ben-Tal et al., 2009; Madry et al., 2018; Wong & Kolter, 2018;

Original: Shaham et al., 2018; Sinha et al., 2018).
Revised: Shaham et al., 2018; Sinha et al., 2018).

Original: In this formulation, one is seeking to improve the worstcase performance of the model, where the performance is measured by a particular loss function (cid:96) .
Revised: In this formulation, one is seeking to improve the worstcase performance of the model, where the performance is measured by a particular loss function (cid:96) .

Original: Adversarial training can be understood as approximating such a worst-case loss by ﬁnding the corresponding worst-case data point, i.e., x + δ with some speciﬁc attack techniques.
Revised: Adversarial training can be understood as approximating such a worst-case loss by ﬁnding the corresponding worst-case data point, i.e., x + δ with some speciﬁc attack techniques.

Original: Our proposed method is more direct.
Revised: Our proposed method is more direct.

Original: It is based on approximating the loss function (cid:96) ( x + δ ) using its second-order
Revised: It is based on approximating the loss function (cid:96) ( x + δ ) using its second-order

Original: Taylor series expansion, i.e.,
Revised: Taylor series expansion, i.e.,

Original: [Equation]
Revised: [Equation]

Original: and then upper bounding the worst-case loss using the expansion terms.
Revised: and then upper bounding the worst-case loss using the expansion terms.

Original: By considering both gradient and Hessian of the loss function with respect to (w.r.t.)
Revised: By considering both gradient and Hessian of the loss function with respect to (w.r.t.)

Original: the input, we can provide a more accurate approximation to the worst-case loss.
Revised: the input, we can provide a more accurate approximation to the worst-case loss.

Original: In our derivations, we consider both (cid:96) 2 and (cid:96) ∞ attacks.
Revised: In our derivations, we consider both (cid:96) 2 and (cid:96) ∞ attacks.

Original: In our derivations, the second-order expansion incorporates both the gradient and Hessian of the loss function with respect to (w.r.t.)
Revised: In our derivations, the second-order expansion incorporates both the gradient and Hessian of the loss function with respect to (w.r.t.)

Original: the input.
Revised: the input.

Original: We call the method Second-Order Adversarial Regularizer (SOAR) (not to be confused with the Soar cognitive architecture Laird 2012).
Revised: We call the method Second-Order Adversarial Regularizer (SOAR) (not to be confused with the Soar cognitive architecture Laird 2012).

Original: In the course of development of SOAR, we make the following contributions:
Revised: In the course of development of SOAR, we make the following contributions:

Original: • We show that an over-parameterized linear regression model can be severely affected by an adversary, even though its population loss is zero.
Revised: • We show that an over-parameterized linear regression model can be severely affected by an adversary, even though its population loss is zero.

Original: We robustify it with a regularizer that
Revised: We robustify it with a regularizer that

Original: exactly mimics the adversarial training.
Revised: exactly mimics the adversarial training.

Original: This suggests that regularization can be used instead of adversarial training (Section 2).
Revised: This suggests that regularization can be used instead of adversarial training (Section 2).

Original: • Inspired by such a possibility, we develop a regularizer which upper bounds the worst-case effect of an adversary under an approximation of the loss.
Revised: • Inspired by such a possibility, we develop a regularizer which upper bounds the worst-case effect of an adversary under an approximation of the loss.

Original: In particular, we derive SOAR, which approximates the inner maximization of the robust optimization formulation based on the second-order Taylor series expansion of the loss function (Section 4).
Revised: In particular, we derive SOAR, which approximates the inner maximization of the robust optimization formulation based on the second-order Taylor series expansion of the loss function (Section 4).

Original: • We study SOAR in the logistic regression setting and reveal challenges with regularization using Hessian w.r.t.
Revised: • We study SOAR in the logistic regression setting and reveal challenges with regularization using Hessian w.r

Original:  the input.
Revised: t. the input.

Original: We develop a simple initialization method to circumvent the issue (Section 4.1).
Revised: We develop a simple initialization method to circumvent the issue (Section 4.1).

Original: • We empirically show that SOAR signiﬁcantly improves the adversarial robustness of the network against (cid:96) ∞ attacks( ε = 8255 ) and (cid:96) 2 attacks( ε = 1 . 74 ) on CIFAR-10 and SVHN.
Revised: • We empirically show that SOAR signiﬁcantly improves the adversarial robustness of the network against (cid:96) ∞ attacks and (cid:96) 2 attacks on CIFAR-10 and SVHN.

Original: Speciﬁcally, we evaluate using a PGD1000 white-box attack, transferred PGD1000 attacks and SimBA (Guo et al., 2019).
Revised: Speciﬁcally, we evaluate using a PGD1000 white-box attack (Madry et al., 2018), transferred PGD1000 attacks, AutoAttack (Croce & Hein, 2020), and SimBA (Guo et al., 2019).

Original: Its adversarial robustness is competitive to SOTA methods, while its negative effect on the standard accuracy (on clean data) is minimum (Section 5).

Original:  L INEAR R EGRESSION WITH AN O VER - PARAMETRIZED M ODEL
Revised:  L INEAR R EGRESSION WITH AN O VER - PARAMETRIZED M ODEL

Original: This section shows that for over-parameterized linear models, gradient descent (GD) ﬁnds a solution that has zero population loss, but is prone to attacks.
Revised: This section shows that for over-parameterized linear models, gradient descent (GD) ﬁnds a solution that has zero population loss, but is prone to attacks.

Original: It also shows that one can avoid this problem with deﬁning an appropriate regularizer.
Revised: It also shows that one can avoid this problem with deﬁning an appropriate regularizer.

Original: Hence, we do not need adversarial training to robustify such a model.
Revised: Hence, we do not need adversarial training to robustify such a model.

Original: This simple illustration motivates the development of our method in next sections.
Revised: This simple illustration motivates the development of our method in next sections.

Original: We only brieﬂy report the main results here, and defer the derivations to Appendix A .
Revised: We only brieﬂy report the main results here, and defer the derivations to Appendix A .

Original: Consider a linear model f w ( x ) = (cid:104) w , x (cid:105) with x, w ∈ R d .
Revised: Consider a linear model f w ( x ) = (cid:104) w , x (cid:105) with x, w ∈ R d .

Original: Suppose that w ∗ =
Revised: Suppose that w ∗ =

Original: . , 0) (cid:62) and the distribution of x ∼ p is such that it is conﬁned on a 1 -dimensional subspace { ( x 1 , 0 , 0 , . .
Revised: . , 0) (cid:62) and the distribution of x ∼ p is such that it is conﬁned on a 1 -dimensional subspace { ( x 1 , 0 , 0 , . .

Original: . , 0) : x 1 ∈ R } .
Revised: . , 0) : x 1 ∈ R } .

Original: This setup can be thought of as using an over-parameterized model that has many irrelevant dimensions with data that is only covering the relevant dimension of the input space.
Revised: This setup can be thought of as using an over-parameterized model that has many irrelevant dimensions with data that is only covering the relevant dimension of the input space.

Original: This is a simpliﬁed model of the situation when the data manifold has a dimension lower than the input space.
Revised: This is a simpliﬁed model of the situation when the data manifold has a dimension lower than the input space.

Original: We consider the squared error pointwise loss l ( x ; w ) = 12 |(cid:104) x , w (cid:105) − (cid:104) x , w ∗ (cid:105)| 2 .
Revised: We consider the squared error pointwise loss l ( x ; w ) = 12 |(cid:104) x , w (cid:105) − (cid:104) x , w ∗ (cid:105)| 2 .

Original: Denote the residual by r ( x ; w ) =
Revised: Denote the residual by r ( x ; w ) =

Original: (cid:104) x , w − w ∗ (cid:105) , and the population loss by L ( w ) =
Revised: (cid:104) x , w − w ∗ (cid:105) , and the population loss by L ( w ) =

Original: E [ l ( X ; w )] .
Revised: E [ l ( X ; w )] .

Original: Suppose that we initialize the weights as w (0)
Revised: Suppose that we initialize the weights as w (0)

Original: =
Revised: =

Original: W ∼ N (0 , σ 2 I d × d ) , and use GD on the population loss, i.e., w ( t + 1) ← w ( t ) − β ∇ w L ( w ) .
Revised: W ∼ N (0 , σ 2 I d × d ) , and use GD on the population loss, i.e., w ( t + 1) ← w ( t ) − β ∇ w L ( w ) .

Original: It is easy to see that the partial derivatives w.r.t.
Revised: It is easy to see that the partial derivatives w.r.t.

Original: w 2 ,...,d are all zero, i.e., no weight adaptation happens.
Revised: w 2 ,...,d are all zero, i.e., no weight adaptation happens.

Original: With a proper choice of learning rate β , we get that the asymptotic solution is ¯ w (cid:44)
Revised: With a proper choice of learning rate β , we get that the asymptotic solution is ¯ w (cid:44)

Original: lim
Revised: lim

Original: r →∞ w ( t ) =
Revised: r →∞ w ( t ) =

Original: ( w ∗ 1 , w 2 (0) , w 3 (0) , . . .
Revised: ( w ∗ 1 , w 2 (0) , w 3 (0) , . . .

Original: , w d (0)) (cid:62) .
Revised: , w d (0)) (cid:62) .

Original: That is, the initial random weights on dimensions 2 , . . .
Revised: That is, the initial random weights on dimensions 2 , . . .

Original: , d do not change.
Revised: , d do not change.

Original: We make two observations.
Revised: We make two observations.

Original: The ﬁrst is that L ( ¯ w ) = 0 , i.e., the population loss is zero.
Revised: The ﬁrst is that L ( ¯ w ) = 0 , i.e., the population loss is zero.

Original: So from the perspective of training under the original loss, we are ﬁnding the optimal solution.
Revised: So from the perspective of training under the original loss, we are ﬁnding the optimal solution.

Original: The second observation is that this model is vulnerable to adversarial examples.
Revised: The second observation is that this model is vulnerable to adversarial examples.

Original: An FGSM-like attack that perturbs x by ∆ x = (0 , ∆ x 2 , ∆ x 3 , . .
Revised: An FGSM-like attack that perturbs x by ∆ x = (0 , ∆ x 2 , ∆ x 3 , . .

Original: . , ∆ x d ) (cid:62) with ∆ x i = ε sign( w i (0))
Revised: . , ∆ x d ) (cid:62) with ∆ x i = ε sign( w i (0))

Original: (for i = 2 , . . .
Revised: (for i = 2 , . . .

Original: , d ) has the population loss of E X,W [ l ( X + ∆ x ); ¯ w )]
Revised: , d ) has the population loss of E X,W [ l ( X + ∆ x ); ¯ w )]

Original: ≈ O ( ε 2 d 2 σ 2 ) under the adversary at the asymptotic solution ¯ w .
Revised: ≈ O ( ε 2 d 2 σ 2 ) under the adversary at the asymptotic solution ¯ w .

Original: When the dimension is large, this loss is quite signiﬁcant.
Revised: When the dimension is large, this loss is quite signiﬁcant.

Original: The culprit is obviously that
Revised: The culprit is obviously that

Original: GD is not forcing the initial weights to go to zero when there is no data from irrelevant and unused dimensions.
Revised: GD is not forcing the initial weights to go to zero when there is no data from irrelevant and unused dimensions.

Original: This simple problem illustrates how the optimizer and an over-parameterized model might interact and lead to a solution that is prone to attacks.
Revised: This simple problem illustrates how the optimizer and an over-parameterized model might interact and lead to a solution that is prone to attacks.

Original: An effective solution is to regularize the loss such that the weights of irrelevant dimensions to go to zero.
Revised: An effective solution is to regularize the loss such that the weights of irrelevant dimensions to go to zero.

Original: Generic regularizers such as ridge and Lasso regression lead to a biased estimate of w ∗ 1 , and thus, one is motivated to deﬁne a regularizer that is specially-designed for improving adversarial robustness.
Revised: Generic regularizers such as ridge and Lasso regression lead to a biased estimate of w ∗ 1 , and thus, one is motivated to deﬁne a regularizer that is specially-designed for improving adversarial robustness.

Original: Bishop (1995) showed the close connection between training with random perturbation and Tikhonov
Revised: Bishop (1995) showed the close connection between training with random perturbation and Tikhonov

Original: Regularization.
Revised: Regularization.

Original: Inspired by this idea, we develop a regularizer that mimics the adversary itself.
Revised: Inspired by this idea, we develop a regularizer that mimics the adversary itself.

Original: For this FGSM-like adversary, the population loss at the perturbed point is
Revised: For this FGSM-like adversary, the population loss at the perturbed point is

Original: [Equation]
Revised: [Equation]

Original: Minimizing L robustiﬁed ( w ) is equivalent to minimizing the model at the point x (cid:48)
Revised: Minimizing L robustiﬁed ( w ) is equivalent to minimizing the model at the point x (cid:48)

Original: = x
Revised: = x

Original: + ∆ x .
Revised: + ∆ x .

Original: The regularizer ε E
Revised: The regularizer ε E

Original: [ r ( X ; w )] (cid:107) w 2: d (cid:107) 1 + ε 2 2 (cid:107) w 2: d (cid:107) 21 incorporates the effect of adversary in exact form.
Revised: [ r ( X ; w )] (cid:107) w 2: d (cid:107) 1 + ε 2 2 (cid:107) w 2: d (cid:107) 21 incorporates the effect of adversary in exact form.

Original: Nonetheless, there are two limitations of this approach.
Revised: Nonetheless, there are two limitations of this approach.

Original: The ﬁrst is that it is designed for a particular choice of attack, an FGSM-like one.
Revised: The ﬁrst is that it is designed for a particular choice of attack, an FGSM-like one.

Original: We would like a regularizer that is robust to a larger class of attacks.
Revised: We would like a regularizer that is robust to a larger class of attacks.

Original: The second is that this regularizer is designed for a linear model and the squared error loss.
Revised: The second is that this regularizer is designed for a linear model and the squared error loss.

Original: How can we design a regularizer for more complicated models, such as DNNs?
Revised: How can we design a regularizer for more complicated models, such as DNNs?

Original: We address these questions by formulating the problem of adversarial robustness within the robust optimization framework (Section 3), and propose an approach to approximately solve it (Section 4).
Revised: We address these questions by formulating the problem of adversarial robustness within the robust optimization framework (Section 3), and propose an approach to approximately solve it (Section 4).

Original:  R OBUST O PTIMIZATION F ORMULATION
Revised:  R OBUST O PTIMIZATION F ORMULATION

Original: Designing an adversarial robust estimator can be formulated as a robust optimization problem (Huang et al., 2015; Madry et al., 2018; Wong & Kolter, 2018; Shaham et al., 2018).
Revised: Designing an adversarial robust estimator can be formulated as a robust optimization problem (Huang et al., 2015; Madry et al., 2018; Wong & Kolter, 2018; Shaham et al., 2018).

Original: To describe it, let us introduce our notations ﬁrst.
Revised: To describe it, let us introduce our notations ﬁrst.

Original: Consider an input space X ⊂ R d , an output space Y , and a parameter (or hypothesis) space W , parameterizing a model f : X × W → Y .
Revised: Consider an input space X ⊂ R d , an output space Y , and a parameter (or hypothesis) space W , parameterizing a model f : X × W → Y .

Original: In the supervised learning scenario, we are given a data distribution D over pairs of examples { ( X i , Y i ) } ni =1 .
Revised: In the supervised learning scenario, we are given a data distribution D over pairs of examples { ( X i , Y i ) } ni =1 .

Original: Given the prediction of f ( x ; w ) and a target value y , the pointwise loss function of the model is denoted by (cid:96) ( x, y ; w ) (cid:44) (cid:96) ( f ( x ; w ) , y ) .
Revised: Given the prediction of f ( x ; w ) and a target value y , the pointwise loss function of the model is denoted by (cid:96) ( x, y ; w ) (cid:44) (cid:96) ( f ( x ; w ) , y ) .

Original: Given the distribution of data, one can deﬁne the population loss as
Revised: Given the distribution of data, one can deﬁne the population loss as

Original: L ( w ) =
Revised: L ( w ) =

Original: E [ (cid:96) ( X, Y ; w )] .
Revised: E [ (cid:96) ( X, Y ; w )] .

Original: The goal of the standard supervised learning problem is to ﬁnd a w ∈ W that minimizes the population loss.
Revised: The goal of the standard supervised learning problem is to ﬁnd a w ∈ W that minimizes the population loss.

Original: A generic approach to do this is through empirical risk minimization (ERM).
Revised: A generic approach to do this is through empirical risk minimization (ERM).

Original: Explicit or implicit regularization is often used to control the complexity of the hypothesis to avoid over- or under-ﬁtting (Hastie et al., 2009).
Revised: Explicit or implicit regularization is often used to control the complexity of the hypothesis to avoid over- or under-ﬁtting (Hastie et al., 2009).

Original: As shown in the previous section, it is possible to ﬁnd a parameter w that minimizes the loss through
Revised: As shown in the previous section, it is possible to ﬁnd a parameter w that minimizes the loss through

Original: ERM, but leads to a model that is vulnerable to adversarial examples.
Revised: ERM, but leads to a model that is vulnerable to adversarial examples.

Original: To incorporate the robustness notion in the model, it requires defenders to reconsider the training objective.
Revised: To incorporate the robustness notion in the model, it requires defenders to reconsider the training objective.

Original: It is also important to formalize and constrain the power of the adversary, so we understand the strength of the attack to which the model is resistant.
Revised: It is also important to formalize and constrain the power of the adversary, so we understand the strength of the attack to which the model is resistant.

Original: This can be speciﬁed by limiting that the adversary can only modify any input x to x + δ with δ ∈ ∆ ⊂ X .
Revised: This can be speciﬁed by limiting that the adversary can only modify any input x to x + δ with δ ∈ ∆ ⊂ X .

Original: Commonly used constraints are ε -balls w.r.t.
Revised: Commonly used constraints are ε -balls w.r.t.

Original: the (cid:96) p -norms, though other constraint sets have been used too (Wong et al., 2019b).
Revised: the (cid:96) p -norms, though other constraint sets have been used too (Wong et al., 2019b).

Original: This goal can be formulated as a robust optimization problem where the objective is to minimize the adversarial population loss given some perturbation constraint ∆ :
Revised: This goal can be formulated as a robust optimization problem where the objective is to minimize the adversarial population loss given some perturbation constraint ∆ :

Original: [Equation]
Revised: [Equation]

Original: We have an interplay between two goals: 1) the inner-max term looks for the worst-case loss around the input, while 2) the outer-min term optimizes the hypothesis by minimizing such a loss.
Revised: We have an interplay between two goals: 1) the inner-max term looks for the worst-case loss around the input, while 2) the outer-min term optimizes the hypothesis by minimizing such a loss.

Original: Note that solving the inner-max problem is often computationally difﬁcult, so one may approximate it with a surrogate loss obtained from a particular attack.
Revised: Note that solving the inner-max problem is often computationally difﬁcult, so one may approximate it with a surrogate loss obtained from a particular attack.

Original: Adversarial training and its variants (Szegedy et al., 2013; Goodfellow et al., 2014; Kurakin et al., 2016;
Revised: Adversarial training and its variants (Szegedy et al., 2013; Goodfellow et al., 2014; Kurakin et al., 2016;

Original: Madry et al., 2018; Wong et al., 2019a) can be intuitively understood as an approximation of this min-max problem via different δ ( x ) .
Revised: Madry et al., 2018; Wong et al., 2019a) can be intuitively understood as an approximation of this min-max problem via different δ ( x ) .

Original: As shown in Section 2, one can design a regularizer that provides the exact value of the loss function at the attacked point for a particular choice of model, loss function, and adversary, cf.
Revised: As shown in Section 2, one can design a regularizer that provides the exact value of the loss function at the attacked point for a particular choice of model, loss function, and adversary, cf.

Original: Such a regularizer relieved us from using a separate inner optimization procedure, as is done in adversarial training.
Revised: Under the robust optimization framework, the regularizer and adversarial training are two realizations of the inner-max objective in (2), but using such a regularizer relieved us from using a separate inner optimization procedure, as is done in adversarial training.

Original: Motivated by that example and the robust optimization framework discussed here, we develop a regularizer that can be understood as an upper-bound on the worst-case value of the loss at an attacked point under a second-order approximation of the loss function.
Revised: Motivated by that example and the robust optimization framework discussed here, we develop a regularizer that can be understood as an upper-bound on the worst-case value of the loss at an attacked point under a second-order approximation of the loss function.

Original:  S ECOND -O RDER A DVERSARIAL R EGULARIZER (SOAR
Revised:  S ECOND -O RDER A DVERSARIAL R EGULARIZER (SOAR)

Original: The main idea of SOAR is to approximate the loss function using the second-order Taylor series expansion around an input x and then solve the inner maximization term of the robust optimization formulation (2) using the approximated form.
Revised: The main idea of SOAR is to approximate the loss function using the second-order Taylor series expansion around an input x and then solve the inner maximization term of the robust optimization formulation (2) using the approximated form.

Original: We show this for both (cid:96) 2 and (cid:96) ∞ attacks; the same idea can be applied to other (cid:96) p norms.
Revised: We show this for both (cid:96) 2 and (cid:96) ∞ attacks; the same idea can be applied to other (cid:96) p norms.

Original: We describe crucial steps of the derivation in this section, and defer details to Appendix B .
Revised: We describe crucial steps of the derivation in this section, and defer details to Appendix B .

Original: Assuming that the loss is twice-differentiable, we can approximate the loss function around input x by the second-order Taylor expansion
Revised: Assuming that the loss is twice-differentiable, we can approximate the loss function around input x by the second-order Taylor expansion

Original: [Equation]
Revised: [Equation]

Original: For brevity, we drop w , y and use ∇ to denote ∇ x .
Revised: For brevity, we drop w , y and use ∇ to denote ∇ x .

Original: Let us focus on the (cid:96) p attacks, where the constraint set in (2) is ∆ =
Revised: Let us focus on the (cid:96) p attacks, where the constraint set in (2) is ∆ =

Original: { δ : (cid:107) δ (cid:107) p ≤ ε } for some ε > 0 and p ≥ 1 .
Revised: { δ : (cid:107) δ (cid:107) p ≤ ε } for some ε > 0 and p ≥ 1 .

Original: We focus on the (cid:96) ∞ attack because of its popularity, but we also derive the formulation for the (cid:96) 2 attacks.
Revised: We focus on the (cid:96) ∞ attack because of its popularity, but we also derive the formulation for the (cid:96) 2 attacks.

Original: As a warm-up, let us solve the inner optimization problem by considering the ﬁrst-order Taylor series expansion.
Revised: As a warm-up, let us solve the inner optimization problem by considering the ﬁrst-order Taylor series expansion.

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: The term ε (cid:107)∇ (cid:96) ( x ) (cid:107) 1 deﬁnes the First-Order Adversarial Regularizer (FOAR).
Revised: The term ε (cid:107)∇ (cid:96) ( x ) (cid:107) 1 deﬁnes the First-Order Adversarial Regularizer (FOAR).

Original: This is similar to the regularizer introduced by Simon-Gabriel et al.
Revised: This is similar to the regularizer introduced by Simon-Gabriel et al.

Original: with the choice of (cid:96) ∞ perturbation set.
Revised: with the choice of (cid:96) ∞ perturbation set.

Original: For a general (cid:96) p -attach with 1 ≤ p ≤ ∞ , we have (cid:107)∇ (cid:96) ( x ) (cid:107) q with q satisfying p − 1 + q − 1 = 1 .
Revised: For a general (cid:96) p -attach with 1 ≤ p ≤ ∞ , we have (cid:107)∇ (cid:96) ( x ) (cid:107) q with q satisfying p − 1 + q − 1 = 1 .

Original: We shall empirically evaluate FOAR-based approach (for the (cid:96) ∞ attack), but our focus is going to be on solving the inner maximization problem based on the second-order Taylor expansion:
Revised: We shall empirically evaluate FOAR-based approach (for the (cid:96) ∞ attack), but our focus is going to be on solving the inner maximization problem based on the second-order Taylor expansion:

Original: [Equation]
Revised: [Equation]

Original: for p = 2 , ∞ .
Revised: for p = 2 , ∞ .

Original: The second-order expansion in (3) can be rewritten as
Revised: The second-order expansion in (3) can be rewritten as

Original: [Equation]
Revised: [Equation]

Original: where δ (cid:48)
Revised: where δ (cid:48)

Original: =
Revised: =

Original: [ δ ; 1] .
Revised: [ δ ; 1] .

Original: This allows us to derive an upper bound on the expansion terms using the characteristics of a single Hessian term H .
Revised: This allows us to derive an upper bound on the expansion terms using the characteristics of a single Hessian term H .

Original: Note that δ (cid:48) is a d + 1 -dimensional vector and H is a ( d + 1) ×
Revised: Note that δ (cid:48) is a d + 1 -dimensional vector and H is a ( d + 1) ×

Original: ( d + 1) matrix.
Revised: ( d + 1) matrix.

Original: We need to ﬁnd an upper bound on δ (cid:48)(cid:62)
Revised: We need to ﬁnd an upper bound on δ (cid:48)(cid:62)

Original: H δ (cid:48) under the attack constraint.
Revised: H δ (cid:48) under the attack constraint.

Original: For the (cid:96) ∞ attack, solving this maximizing problem is not as easy as in (4) since the Boolean quadratic programming problem in formulation (5) is NP-hard.
Revised: For the (cid:96) ∞ attack, solving this maximizing problem is not as easy as in (4) since the Boolean quadratic programming problem in formulation (5) is NP-hard.

Original: But we can relax the constraint set and ﬁnd an upper bound for the maximizer.
Revised: But we can relax the constraint set and ﬁnd an upper bound for the maximizer.

Original: Note that with δ ∈ R d , an (cid:96) ∞ -ball of size ε is enclosed by an (cid:96) 2 -ball of size √ dε with the same centre.
Revised: Note that with δ ∈ R d , an (cid:96) ∞ -ball of size ε is enclosed by an (cid:96) 2 -ball of size √ dε with the same centre.

Original: Therefore, we can upper bound the inner maximization by
Revised: Therefore, we can upper bound the inner maximization by

Original: [Equation]
Revised: [Equation]

Original: which after substituting the second-order Taylor series expansion leads to an (cid:96) 2 -constrained quadratic optimization problem
Revised: which after substituting the second-order Taylor series expansion leads to an (cid:96) 2 -constrained quadratic optimization problem

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: with δ (cid:48)
Revised: with δ (cid:48)

Original: =
Revised: =

Original: [ δ ; 1] as before.
Revised: [ δ ; 1] as before.

Original: The (cid:96) 2 version of SOAR does not require this extra step, and we have ε instead of √ dε in (8).
Revised: The (cid:96) 2 version of SOAR does not require this extra step, and we have ε instead of √ dε in (8).

Original: Before presenting a proposition that suggests a way to solve this maximization problem, we would like to remark on the upper bounding of the ε − (cid:96) ∞ ball with an √ dε − (cid:96) 2 ball.
Revised: Before presenting a proposition that suggests a way to solve this maximization problem, we would like to remark on the upper bounding of the ε − (cid:96) ∞ ball with an √ dε − (cid:96) 2 ball.

Original: If δ is a sparse vector, this upper bound can be loose for large d .
Revised: If δ is a sparse vector, this upper bound can be loose for large d .

Original: But it appears that in practice, this upper bound is not loose after all, as the attack δ is often a dense vector.
Revised: But it appears that in practice, this upper bound is not loose after all, as the attack δ is often a dense vector.

Original: A more detailed discussion is included in Appendix B.2 .
Revised: A more detailed discussion is included in Appendix B.2 .

Original: Proposition 1.
Revised: Proposition 1.

Original: Let (cid:96) :
Revised: Let (cid:96) :

Original: R d → R be a twice-differentiable function.
Revised: R d → R be a twice-differentiable function.

Original: For any ε > 0 , we have
Revised: For any ε > 0 , we have

Original: [Equation]
Revised: [Equation]

Original: where H is deﬁned in (6) and z ∼ N (0 , I ( d +1) ×
Revised: where H is deﬁned in (6) and z ∼ N (0 , I ( d +1) ×

Original: ( d +1) ) .
Revised: ( d +1) ) .

Original: This result upper bounds the maximum of the second-order approximation ˜ (cid:96) 2nd over an (cid:96) ∞ ball with radius ε , and relates it to an expectation of a Hessian-vector product.
Revised: This result upper bounds the maximum of the second-order approximation ˜ (cid:96) 2nd over an (cid:96) ∞ ball with radius ε , and relates it to an expectation of a Hessian-vector product.

Original: Note that there is a simple correspondence between (1) and regularized loss in (9).
Revised: Note that there is a simple correspondence between (1) and regularized loss in (9).

Original: The latter can be understood as an upper bound on the worst-case damage of an adversary under a second-order approximation of the loss.
Revised: The latter can be understood as an upper bound on the worst-case damage of an adversary under a second-order approximation of the loss.

Original: For the (cid:96) 2 attack, the same line of argument leads to ε 2 + 1 instead of dε 2 + 1 .
Revised: For the (cid:96) 2 attack, the same line of argument leads to ε 2 + 1 instead of dε 2 + 1 .

Original: Let us take a closer look at H z .
Revised: Let us take a closer look at H z .

Original: By decomposing z = [ z d , z 1 ] (cid:62) , we get
Revised: By decomposing z = [ z d , z 1 ] (cid:62) , we get

Original: [Equation]
Revised: [Equation]

Original: The term ∇ 2 (cid:96) ( x ) z d can be computed using Finite Difference (FD) approximation.
Revised: The term ∇ 2 (cid:96) ( x ) z d can be computed using Finite Difference (FD) approximation.

Original: Note that
Revised: Note that

Original: E [ (cid:107)
Revised: E [ (cid:107)

Original: z
Revised: z

Original: d (cid:107) 2 ] = √ d for our Normally distributed z .
Revised: d (cid:107) 2 ] = √ d for our Normally distributed z .

Original: To ensure that the approximation direction has the same magnitude, we use the normalized ˜ z
Revised: To ensure that the approximation direction has the same magnitude, we use the normalized ˜ z

Original: d = z d (cid:107) z
Revised: d = z d (cid:107) z

Original: d
Revised: d

Original: [Equation]
Revised: [Equation]

Original: 2 instead, and use the approximation below
Revised: 2 instead, and use the approximation below

Original: [Equation]
Revised: [Equation]

Original: To summarize, SOAR regularizer evaluated at x , with a direction z , and FD step size h > 0 is
Revised: To summarize, SOAR regularizer evaluated at x , with a direction z , and FD step size h > 0 is

Original: [Equation]
Revised: [Equation]

Original: The expectation in (9) can then be approximated by taking multiple samples of z drawn from z ∼ N (0 , I ( d +1) ×
Revised: The expectation in (9) can then be approximated by taking multiple samples of z drawn from z ∼ N (0 , I ( d +1) ×

Original: ( d +1) ) .
Revised: ( d +1) ) .

Original: These samples would be concentrated around its expectation.
Revised: These samples would be concentrated around its expectation.

Original: One can show that P {(cid:107) Hz (cid:107) − E [ (cid:107) Hz (cid:107) ] > t } ≤ 2 exp( − ct 2 (cid:107) H (cid:107) 2 ) , where c is a constant and (cid:107) H (cid:107) 2 is the (cid:96) 2induced norm (see Theorem 6.3.2 of Vershynin 2018).
Revised: One can show that P {(cid:107) Hz (cid:107) − E [ (cid:107) Hz (cid:107) ] > t } ≤ 2 exp( − ct 2 (cid:107) H (cid:107) 2 ) , where c is a constant and (cid:107) H (cid:107) 2 is the (cid:96) 2induced norm (see Theorem 6.3.2 of Vershynin 2018).

Original: In practice, we observed that taking more than one sample of z do not provide signiﬁcant improvement for increasing adversarial robustness.
Revised: In practice, we observed that taking more than one sample of z do not provide signiﬁcant improvement for increasing adversarial robustness, and we include an empirical study on the the effect of sample sizes in Appendix E.4 .

Original: Before we discuss the remaining details, recall that we fully robustify the model with an appropriate regularizer in Section 2.
Revised: Before we discuss the remaining details, recall that we fully robustify the model with an appropriate regularizer in Section 2.

Original: Note the maximizer of the loss based on formulation (2) is exactly the FGSM direction, and (1) shows the population loss with our FGSM-like choice of ∆ x .
Revised: Note the maximizer of the loss based on formulation (2) is exactly the FGSM direction, and (1) shows the population loss with our FGSM-like choice of ∆ x .

Original: To further motivate a second-order approach, note that we can obtain the ﬁrst two terms in (1) with a ﬁrst-order regularizer such as FOAR; and we recover the exact form with a second-order formulation in (5).
Revised: To further motivate a second-order approach, note that we can obtain the ﬁrst two terms in (1) with a ﬁrst-order regularizer such as FOAR; and we recover the exact form with a second-order formulation in (5).

Original: Next, we study SOAR in the simple logistic regression setting, which shows potential failure of the regularizer and reveals why we might observe gradient masking.
Revised: Next, we study SOAR in the simple logistic regression setting, which shows potential failure of the regularizer and reveals why we might observe gradient masking.

Original: Based on that insight, we provide the remaining details of the method afterwards in Section 4.1.
Revised: Based on that insight, we provide the remaining details of the method afterwards in Section 4.1.

Original: 4. A VOIDING G RADIENT M ASKING
Revised: 4. A VOIDING G RADIENT M ASKING

Original: Consider a linear classiﬁer f : R d × R d → R with f ( x ; w )
Revised: Consider a linear classiﬁer f : R d × R d → R with f ( x ; w )

Original: =
Revised: =

Original: φ ( (cid:104) w , x (cid:105) ) , where x, w ∈ R
Revised: φ ( (cid:104) w , x (cid:105) ) , where x, w ∈ R

Original: d
Revised: d

Original: are the input and the weights, and φ ( · ) is the sigmoid function.
Revised: are the input and the weights, and φ ( · ) is the sigmoid function.

Original: Note that the output of f has the interpretation of being a Bernoulli distribution.
Revised: Note that the output of f has the interpretation of being a Bernoulli distribution.

Original: For the cross-entropy loss function (cid:96) ( x, y ; w ) = − [ y log f ( x ; w )+(1 − y ) log(1 − f ( x ; w ))] , the gradient w.r.t.
Revised: For the cross-entropy loss function (cid:96) ( x, y ; w ) = − [ y log f ( x ; w )+(1 − y ) log(1 − f ( x ; w ))] , the gradient w.r.t.

Original: the input x is ∇ (cid:96) ( x ) =
Revised: the input x is ∇ (cid:96) ( x ) =

Original: ( f ( x ; w ) − y ) w and the Hessian w.r.t.
Revised: ( f ( x ; w ) − y ) w and the Hessian w.r.t.

Original: the input x is ∇ 2 (cid:96) ( x ) =
Revised: the input x is ∇ 2 (cid:96) ( x ) =

Original: f
Revised: f

Original: ( x ; w )(1 − f ( x ; w ))
Revised: ( x ; w )(1 − f ( x ; w ))

Original: ww (cid:62) .
Revised: ww (cid:62) .

Original: The second-order Taylor series expansion (3) with the gradient and Hessian evaluated at x is
Revised: The second-order Taylor series expansion (3) with the gradient and Hessian evaluated at x is

Original: [Equation]
Revised: [Equation]

Original: where r = r ( x, y ; w ) =
Revised: where r = r ( x, y ; w ) =

Original: f ( x ; w ) − y is the residual term describing the difference between the predicted probability and the correct label, and u = u ( x ; w ) =
Revised: f ( x ; w ) − y is the residual term describing the difference between the predicted probability and the correct label, and u = u ( x ; w ) =

Original: f ( x ; w )(1 − f ( x ; w )) .
Revised: f ( x ; w )(1 − f ( x ; w )) .

Original: Note that u can be interpreted as how conﬁdent the model is about its predication (correct or incorrect), and is close to 0 whenever the classiﬁer is predicting a value close to 0 or 1 .
Revised: Note that u can be interpreted as how conﬁdent the model is about its predication (correct or incorrect), and is close to 0 whenever the classiﬁer is predicting a value close to 0 or 1 .

Original: With this linear model, the maximization (8) becomes
Revised: With this linear model, the maximization (8) becomes

Original: [Equation]
Revised: [Equation]

Original: The regularization term is encouraging the norm of w to be small, weighted according to the residual r ( x, y ; w ) and the uncertainty u ( x ; w ) .
Revised: The regularization term is encouraging the norm of w to be small, weighted according to the residual r ( x, y ; w ) and the uncertainty u ( x ; w ) .

Revised: Consider a linear interpolation of the cross-entroply loss from x to a perturbed input x (cid:48) .

Revised: Speciﬁcally, we consider (cid:96) ( αx + (1 − α ) x (cid:48) ) for α ∈ [0 , 1] .

Revised: Previous work has empirically shown that the value of the loss behaves logistically as α increases from 0 to 1 (Madry et al., 2018).

Revised: In such a case, since there is very little curvature at x , if we use Hessian exactly at x , it leads to an inaccurate approximation of the value at (cid:96) ( x (cid:48) ) .

Original:  Whenever the classiﬁer is correct with a high conﬁdence, both r and u will be close to zero.
Revised: Consequently, we have a poor approximation of the inner-max, and the derived regularization will not be effective. For the approximation in (12), this issue corresponds to the scenario in which the classiﬁer is very conﬁdent about the clean input at x . Standard training techniques such as minimizing the crossentropy loss optimize the model such that it returns the correct label with a high conﬁdence. Whenever the classiﬁer is correct with a high conﬁdence, both r and u will be close to zero.

Original: As a result, the effect of the regularizer diminishes, i.e., the weights are no longer regularized.
Revised: As a result, the effect of the regularizer diminishes, i.e., the weights are no longer regularized.

Original: In such a case, the Taylor series expansion, computed using the gradient and Hessian evaluated at x , becomes an inaccurate approximation to the loss, and hence its maximizer is not a good solution to the inner maximization problem.
Revised: In such a case, the Taylor series expansion, computed using the gradient and Hessian evaluated at x , becomes an inaccurate approximation to the loss, and hence its maximizer is not a good solution to the inner maximization problem.

Original: Note that this does not mean that one cannot use Taylor series expansion to approximate the loss.
Revised: Note that this does not mean that one cannot use Taylor series expansion to approximate the loss.

Original: In fact, by the mean value theorem, there exists an h (cid:63) ∈ (0 , 1) such that the second-order Taylor expansion is exact: 
Revised: In fact, by the mean value theorem, there exists an h (cid:63) ∈ (0 , 1) such that the second-order Taylor expansion is exact: (cid:96) ( x + δ ) =

Original: [Equation]
Revised: (cid:96) ( x ) + ∇ (cid:96) ( x ) (cid:62) δ + 12 δ

Original:  2 (cid:96) ( x + h (cid:63) δ ) δ .
Revised: (cid:62) ∇ 2 (cid:96) ( x + h (cid:63) δ ) δ .

Original: The issue is that if we compute the Hessian at x (instead of at x + h (cid:63)  ), our approximation might not be very good whenever the  loss function  is close to being ﬂat at x  .
Revised: The issue is that if we compute the Hessian at x (instead of at x + h (cid:63) δ ), our approximation might not be very good whenever the curvature proﬁle of the loss function at x is drastically different from the one at x + h (cid:63) δ .

Original: More importantly, a method relying on the gradient masking can be easily circumvented (Athalye et al., 2018).
Revised: More importantly, a method relying on the gradient masking can be easily circumvented (Athalye et al., 2018).

Original: Our early experimental results had also indicated that gradient masking occurred with
Revised: Our early experimental results had also indicated that gradient masking occurred with

Original: SOAR when the gradient and Hessian were evaluated at x .
Revised: SOAR when the gradient and Hessian were evaluated at x .

Original: In particular, we observe that SOAR with zero-initialization leads to models with nearly 100% conﬁdence on their predictions, leading to an ineffective regularizer.
Revised: In particular, we observe that SOAR with zero-initialization leads to models with nearly 100% conﬁdence on their predictions, leading to an ineffective regularizer.

Original: The result is reported in Table 3 in Appendix D.
Revised: The result is reported in Table 5 in Appendix D.

Original: This suggests a heuristic to improve the quality of SOAR.
Revised: This suggests a heuristic to improve the quality of SOAR.

Original: That is to evaluate the gradient and Hessian, through FD approximation (10) at a less conﬁdent point in the (cid:96) ∞ ball of x .
Revised: That is to evaluate the gradient and Hessian, through FD approximation (10) at a less conﬁdent point in the (cid:96) ∞ ball of x .

Original: We found that evaluating the gradient and Hessian at 1-step PGD adversary successfully circumvent the issue (Line 1-2 in
Revised: We found that evaluating the gradient and Hessian at 1-step PGD adversary successfully circumvent the issue (Line 1-2 in

Original: Algorithm 1).
Revised: Algorithm 1).

Original: We compare other initializations in Table 3 in Appendix D. To ensure the regularization is of the original (cid:96) ∞ ball of ε , we use ε 2 for PGD1 initialization, and then ε 2 in SOAR.
Revised: We compare other initializations in Table 5 in Appendix D. To ensure the regularization is of the original (cid:96) ∞ ball of ε , we use ε 2 for PGD1 initialization, and then ε 2 in SOAR.

Original: Based on this heuristic, the regularized pointwise objective for a data point ( x, y ) is
Revised: Based on this heuristic, the regularized pointwise objective for a data point ( x, y ) is

Original: [Equation]
Revised: [Equation]

Original:  where z ∼ N (0 , I ( d +1) ×
Revised: (13) where z ∼ N (0 , I ( d +1) ×

Original: ( d +1) ) and the point x (cid:48) is initialized at PGD1 adversary.
Revised: ( d +1) ) and the point x (cid:48) is initialized at PGD1 adversary.

Original: Algorithmsummarizes SOAR on a single training data.
Revised: Algorithmsummarizes SOAR on a single training data.

Original: We include the full training procedure in Appendix C .
Revised: We include the full training procedure in Appendix C .

Original: Moreover, we include additional discussions and experiments on gradient masking in Appendix E.9.
Revised: Moreover, we include additional discussions and experiments on gradient masking in Appendix E.11.

Original: 4. R ELATED W ORK
Revised: 4. R ELATED W ORK

Original: Several regularization-based alternatives to adversarial training have been proposed.
Revised: Several regularization-based alternatives to adversarial training have been proposed.

Original: Simon-Gabriel et al.
Revised: Simon-Gabriel et al.

Original: (2019) studied regularization under the ﬁrst-order Taylor approximation.
Revised: (2019) studied regularization under the ﬁrst-order Taylor approximation.

Original: The proposed regularizer for the (cid:96) ∞ perturbation set is the same as FOAR.
Revised: The proposed regularizer for the (cid:96) ∞ perturbation set is the same as FOAR.

Original: Qin et al.
Revised: Qin et al.

Original: (2019) propose local linearity regularization (LLR), where the local linearity measure is deﬁned by the maximum error of the ﬁrstorder Taylor approximation of the loss.
Revised: (2019) propose local linearity regularization (LLR), where the local linearity measure is deﬁned by the maximum error of the ﬁrstorder Taylor approximation of the loss.

Original: LLR minimizes the local linearity mesaure, and minimizes the magnitude of the projection of gradient along the corresponding direction of the local linearity mesaure.
Revised: LLR minimizes the local linearity mesaure, and minimizes the magnitude of the projection of gradient along the corresponding direction of the local linearity mesaure.

Original: It is motivated by the observation of ﬂat loss surfaces during adversarial training.
Revised: It is motivated by the observation of ﬂat loss surfaces during adversarial training.

Original: CURE (Moosavi-Dezfooli et al., 2019) is the closest to our method.
Revised: CURE (Moosavi-Dezfooli et al., 2019) is the closest to our method.

Original: They empirically observed that adversarial training leads to a reduction in the magnitude of eigenvalues of the Hessian w.r.t.
Revised: They empirically observed that adversarial training leads to a reduction in the magnitude of eigenvalues of the Hessian w.r.t.

Original: the input.
Revised: the input.

Original: Thus, they proposed directly minimizing the curvature of the loss function to mimic the effect of adversarial training.
Revised: Thus, they proposed directly minimizing the curvature of the loss function to mimic the effect of adversarial training.

Original: An important advantage of our proposed method is that SOAR is derived from a complete second-order Taylor approximation of the loss, while CURE exclusively focuses on the second-order term for the estimation of the curvature.
Revised: An important advantage of our proposed method is that SOAR is derived from a complete second-order Taylor approximation of the loss, while CURE exclusively focuses on the second-order term for the estimation of the curvature.

Original: Note the ﬁnal optimization objective in SOAR,
Revised: Note the ﬁnal optimization objective in SOAR,

Original: FOAR, LLR and CURE contains derivative w.r.t.
Revised: FOAR, LLR and CURE contains derivative w.r.t.

Original: the input of the DNN, and such a technique was ﬁrst introduced to improve generalization by Drucker & Le Cun (1992) as double backpropagation. 
Revised: the input of the DNN, and such a technique was ﬁrst introduced to improve generalization by Drucker & Le Cun (1992) as double backpropagation. [Table]

Original: Another related line of adversarial regularization methods do not involve approximation to the loss function nor robust optimization.
Revised: Another related line of adversarial regularization methods do not involve approximation to the loss function nor robust optimization.

Original: TRADES (Zhang et al., 2019) introduces a regularization term that penalizes the difference between the output of the model on a training data and its corresponding adversarial example.
Revised: TRADES (Zhang et al., 2019) introduces a regularization term that penalizes the difference between the output of the model on a training data and its corresponding adversarial example.

Original: MART (Wang et al., 2020) reformulated the training objective by explicitly differentiating between the mis-classiﬁed and correctly classiﬁed examples.
Revised: MART (Wang et al., 2020) reformulated the training objective by explicitly differentiating between the mis-classiﬁed and correctly classiﬁed examples.

Original: Ding et al.
Revised: Ding et al.

Original: (2019b) present another regularization approach that leverages adaptive margin maximization (MMA) on correctly classiﬁed example to robustify the model.[Table]
Revised: (2019b) present another regularization approach that leverages adaptive margin maximization (MMA) on correctly classiﬁed example to robustify the model.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we verify the effectiveness of the proposed regularization method against (cid:96) ∞ PGD attacks on CIFAR10.
Revised: In this section, we verify the effectiveness of the proposed regularization method against (cid:96) ∞ PGD attacks on CIFAR10.

Original: Our experiments show that training with SOAR leads to signiﬁcant improvements in adversarial robustness, which is achieved without signiﬁcantly sacriﬁcing standard accuracy.
Revised: Our experiments show that training with SOAR leads to signiﬁcant improvements in adversarial robustness, which is achieved without signiﬁcantly sacriﬁcing standard accuracy.

Original: We focus on (cid:96) ∞ in this section and defer evaluations on (cid:96) 2 in Appendix E.4.
Revised: We focus on (cid:96) ∞ in this section and defer evaluations on (cid:96) 2 in Appendix E.5.

Original: Additionally, we provide a detailed discussion and evaluations on the SVHN dataset in Appendix E.5.
Revised: Additionally, we provide a detailed discussion and evaluations on the SVHN dataset in Appendix E.6.

Original: Finally, we stress that provable robustness as in (Wong & Kolter, 2018) is not the focus of our work and the experiment result only provides an empirical evidence of the robustness.

Original: We train ResNet-

Revised: We train ResNet-

Original: (He et al., 2016) on the CIFAR-10 dataset.
Revised: (He et al., 2016) on the CIFAR-10 dataset.

Original: The baseline methods consist of: (1) Standard: training with no adversarially perturbed data;
Revised: The baseline methods consist of: (1) Standard: training with no adversarially perturbed data;

Original: (2) ADV: training with 10-step PGD adversarial examples; (3) TRADES; (4) MART and (5) MMA.
Revised: (2) ADV: training with 10-step PGD adversarial examples; (3) TRADES; (4) MART and (5) MMA.

Original: Empirical studies in Madry et al.
Revised: Empirical studies in Madry et al.

Original: (2018) and Wang et al.
Revised: (2018) and Wang et al.

Original: (2020) reveal that their approaches beneﬁt from increasing model capacity to achieve higher adversarial robustness, as such, we include WideResNet (Zagoruyko & Komodakis, 2016) for all baseline methods.
Revised: (2020) reveal that their approaches beneﬁt from increasing model capacity to achieve higher adversarial robustness, as such, we include WideResNet (Zagoruyko & Komodakis, 2016) for all baseline methods.

Original: We were not able to reproduce the results of two closely related works, CURE and LLR, which we discuss further in Appendix E.1.
Revised: We were not able to reproduce the results of two closely related works, CURE and LLR, which we discuss further in Appendix E.1.

Original: In Appendix E.11, we compare
Revised: In Appendix E.13, we compare

Original: SOAR and FOAR with different initializations.
Revised: SOAR and FOAR with different initializations.

Original: FOAR achieves the best adversarial robustness using
Revised: FOAR achieves the best adversarial robustness using

Original: PGD1 initialization, so we only present this variation of FOAR in this section.
Revised: PGD1 initialization, so we only present this variation of FOAR in this section.

Original: The optimization procedure is described in detail in Appendix E.2.
Revised: The optimization procedure is described in detail in Appendix E.2.

Original: Note that all  (cid:96) ∞ -PGD adversarial examples used for all training and evaluations are generated with ε = 8 / 255 and a step size of 2 / 255 (pixel values are normalized to [0 , 1] ).
Revised: Note that all methods in this section are trained to defend against (cid:96) ∞ norm attacks with ε = 8 / 255 , as this is a popular choice of ε in the literature. The PGD adversaries discussed in Sections 5.1 and 5.2 are generated with ε = 8 / 255 and a step size of 2 / 255 (pixel values are normalized to [0 , 1] ).

Original: PGD20-50 denotes 20-step PGD attacks withrestarts.
Revised: PGD20-50 denotes 20-step PGD attacks with 50 restarts.

Revised: In Section 5.3, we compare SOAR with baseline methods on (cid:96) ∞

Revised: AutoAttack (Croce & Hein, 2020) adversaries with a varying ε .

Original: Results on ResNet10 are obtained by averaging over 3 independently initialized and trained models, where the standard deviations are reported in Appendix E.8.
Revised: Additionally, results of all methods on ResNet10 are obtained by averaging over 3 independently initialized and trained models, where the standard deviations are reported in Appendix E.10.

Original: We use the provided pretrained
Revised: We use the provided pretrained

Original: WideResNet model provided in the public repository of each method.
Revised: WideResNet model provided in the public repository of each method.

Original: Additionally, we encountered a number of challenges while implementing SOAR which we discuss in details in Appendix E.6.
Revised: [Table]

Original: 5. R OBUSTNESS AGAINST  W HITE -B OX A TTACKS
Revised: 5. R OBUSTNESS AGAINST PGD W HITE -B OX A TTACKS

Original: Before making the comparison between SOAR and the baselines in Table 1, note that FOAR achieves 32 .
Revised: Before making the comparison between SOAR and the baselines in Table [Table] note that FOAR achieves 32 .

Original: 28% against PGD20 attacks.
Revised: 28% against PGD20 attacks.

Original: Despite its uncompetitive performance, this shows that approximating the robust optimization formulation based on Taylor series expansion is a reasonable approach.
Revised: Despite its uncompetitive performance [Table] this shows that approximating the robust optimization formulation based on Taylor series expansion is a reasonable approach.

Original: Furthermore, this justiﬁes our extension to a second-order approximation, as the ﬁrstorder alone is not sufﬁcient.
Revised: Furthermore, this justiﬁes our extension to a second-order approximation, as the ﬁrstorder alone is not sufﬁcient.

Original: Next, we observe that training with SOAR signiﬁcantly improves the adversarial robustness against all types of attacks, leading to higher robustness in all  attacks on the ResNet model.
Revised: Lastly, we observe that training with SOAR signiﬁcantly improves the adversarial robustness against all PGD attacks, leading to higher robustness in all k-step PGD attacks on the ResNet model.

Original: SOAR remains competitive compared to baseline methods trained on high-capacity WideResNet architecture.
Revised: SOAR remains competitive compared to baseline methods trained on high-capacity WideResNet architecture.

Original: We emphasize the importance that improving adversarial robustness should not come at the expense of losing standard accuracy .

Original: Note that the best overall defence method

Original: [Table]

Original: MART on WideResNet achieves only 0 .

Original: 23% higher accuracy against PGD 20 than SOAR on ResNet, while we achieve 4 .

Original: 33% higher accuracy in standard accuracy.

Original: Previous works have usually attributed the drop in standard accuracy to a trade-off between accuracy and robustness (Madry et al., 2018;

Original: Tsipras et al., 2018).

Original: Compared to the baselines, SOAR substantially improves this trade-off while retaining competitive adversarial robustness

Original: [Table]

Original: 5. R OBUSTNESS AGAINST B LACK -B OX A TTACKS
Revised: 5. R OBUSTNESS AGAINST B LACK -B OX A TTACKS

Original: Many defences only reach an illusion of robustness through methods collectively known as gradient masking (Athalye et al., 2018).
Revised: Many defences only reach an illusion of robustness through methods collectively known as gradient masking (Athalye et al., 2018).

Original: These methods often fail against attacks generated from an undefended independently trained model, known as transfer-based black-box attacks.
Revised: These methods often fail against attacks generated from an undefended independently trained model, known as transfer-based black-box attacks.

Original: Recent works (Tramèr et al., 2017; Ilyas et al., 2019) have proposed hypotheses for the success of transfer-based black-box attacks.
Revised: Recent works (Tramèr et al., 2017; Ilyas et al., 2019) have proposed hypotheses for the success of transfer-based black-box attacks.

Original: In our evaluation, the transferred attacks are PGD20 and PGD1000 perturbations generated from two source models: ResNet and WideResNet, which are denoted by the sufﬁx -R and -W respectively.
Revised: In our evaluation, the transferred attacks are PGD20 and PGD1000 perturbations generated from two source models: ResNet and WideResNet, which are denoted by the sufﬁx -R and -W respectively.

Original: The source models are trained separately from the defence models on the unperturbed training set.
Revised: The source models are trained separately from the defence models on the unperturbed training set.

Original: Additionally, Tramer et al.
Revised: Additionally, Tramer et al.

Original: (2020) recommends score-based black-box attacks such as SimBA (Guo et al., 2019).
Revised: (2020) recommends score-based black-box attacks such as SimBA (Guo et al., 2019).

Original: They are more relevant in real-world applications where gradient information is not accessible, and are empirically shown to be more effective than transfer-based attacks.
Revised: They are more relevant in real-world applications where gradient information is not accessible, and are empirically shown to be more effective than transfer-based attacks.

Original: Because they are solely based on the conﬁdence score of the model, score-based attacks are resistant to gradient-masking.
Revised: Because they are solely based on the conﬁdence score of the model, score-based attacks are resistant to gradient-masking.

Original: SimBA works by ﬁrst randomly sampling a vector from a predeﬁned orthonormal basis then either adding or subtracting the random vector from the target image based on model’s conﬁdence score.
Revised: SimBA works by ﬁrst randomly sampling a vector from a predeﬁned orthonormal basis then either adding or subtracting the random vector from the target image based on model’s conﬁdence score.

Original: All black-box attacks are (cid:96) ∞ constrained at ε = 8 / 255 .
Revised: All black-box attacks are (cid:96) ∞ constrained at ε = 8 / 255 .

Original: SOAR achieves the best robustness against all baseline methods trained on ResNet, as shown in
Revised: SOAR achieves the best robustness against all baseline methods trained on ResNet, as shown in

Original: Table 2.
Revised: Table 2.

Original: Compared with the baselines trained on WideResNet, SOAR remains the most robust model against transferred PGD20-W and PGD1000-W, approaching its standard accuracy on unperturbed data.
Revised: Compared with the baselines trained on WideResNet, SOAR remains the most robust model against transferred PGD20-W and PGD1000-W, approaching its standard accuracy on unperturbed data.

Original: Note that all defence methods are substantially more vulnerable to the score-based SimBA attack.
Revised: Note that all defence methods are substantially more vulnerable to the score-based SimBA attack.

Original: SOAR regularized model is the most robust method against SimBA.
Revised: SOAR regularized model is the most robust method against SimBA.

Revised: 5. R OBUSTNESS AGAINST A UTO A TTACK

Revised: During the ICLR rebuttal phase, we evaluated SOAR against Autoattack (Croce & Hein, 2020).

Revised: In this section, we focus on the (cid:96) ∞ -bounded Autoattack, and a similar result with the (cid:96) 2 -bounded attack is included in Appendix E.5.

Revised: We noticed that SOAR has shown greater vulnerabilities against AutoAttack compared to the attacks discussed in Sections 5.1 and 5.2.

Revised: AutoAttack consists of an ensemble of four attacks: two parameter-free versions of the PGD attack (APGD-CE and

Original: C ONCLUSION
Revised: APGD-DLR), a white-box fast adaptive boundary (FAB) attack (Croce & Hein, 2019), a score-based [Table] black- (Andriushchenko et al., 2020).

Revised: Notice major difference is the loss they are based on:

Revised: CE is based on the cross-entropy loss similar to (Madry et al and APGD-DLR is based on the logit difference similar to (Carlini & Wagner, 2017).

Revised: To better understand the source of SOAR’s vulnerability, we tested it against the four attacks individually.

Revised: First, we observed that the result against untargeted APGD-CE is similar to the one shown in Section 5.1.

Revised: This is expected because the attacks are both formulated based on PGD with the cross-entropy loss.

Revised: However, there is a considerable degradation in the accuracy of SOAR against targeted APGD-DLR and targeted FAB.

Revised: At ε = 8 / 255 , SOAR is most vulnerable to targeted

Revised: APGD-DLR with a robust accuracy of only 18 . 25% .

Revised: To further investigate SOAR’s robustness against AutoAttack, we tested with different ε to verify if SOAR can at least improve robustness against (cid:96) ∞ attacks with smaller ε .

Revised: We observed that at ε = 4 / 255 the robustness improvement of

Revised: SOAR becomes more consistent.

Revised: Interestingly, we also noticed that a model with better robustness at ε = 8 / 255 does not guarantee a better robustness at ε = 4 / 255 , as is the case for Square Attack on

Revised: ADV and SOAR.

Revised: Combing the results with the four attacks and with different ε , we provide three hypotheses on the vulnerability of SOAR.

Revised: First, SOAR might overﬁt to a particular type of attack: adversarial examples generated based on the cross-entropy loss.

Revised: APGD-DLR is based on logit difference and FAB is based on ﬁnding minimal pertubation distance, which are both very different from the cross-entropy loss.

Revised: Second, SOAR might still rely on gradient masking, and thus PGD with cross-entropy loss is difﬁcult to ﬁnd adversaries while they still exist.

Revised: This also means that the results with black-box attacks in

Revised: Sections 5.2 are insufﬁcient to eliminate the possibility of gradient masking.

Revised: Third, since SOAR does provide a more consistent robustness improvement at ε = 4 / 255 , this suggests that the techniques discussed in Section 4 (i.e., the relaxation in (7) and the approximation in (9)) did not effectively address the problem raised from the second-order approximation, making SOAR only improvements to ε smaller than desired.

Revised: Finally, we emphasize that this should not rule SOAR as a failed defence.

Revised: Previous work on circumventing defence methods show that a failed mechanism mostly relies on gradient masking and can be completely bypassed ( 0% robust accuracy) with simple attacks that does not rely on gradients (Athalye et al., 2018).

Revised: This is the not the case with SOAR.

Revised: Overall, we think

Revised: SOAR’s vulnerability to AutoAttack is an interesting observation and requires further investigation.

Revised:  C ONCLUSION

Original: This work proposed SOAR, a regularizer that improves the robustness of DNN to adversarial examples.
Revised: This work proposed SOAR, a regularizer that improves the robustness of DNN to adversarial examples.

Original: SOAR was obtained using the second-order Taylor series approximation of the loss function w.r.t.
Revised: SOAR was obtained using the second-order Taylor series approximation of the loss function w.r.t.

Original: the input, and approximately solving the inner maximization of the robust optimization formulation.
Revised: the input, and approximately solving the inner maximization of the robust optimization formulation.

Original: We showed that training with SOAR leads toadversarial robustness, with signiﬁcant improvement in  robustness underwhite-box and black-box (cid:96) ∞ and (cid:96) 2 attacks.
Revised: We showed that training with SOAR leads to signiﬁcant improvement in adversarial robustness under (cid:96) ∞ and (cid:96) 2 attacks.

Original: More importantly, SOAR achieves robustness gain without a signiﬁcant trade-off with standard accuracy.

Original: This is only one step in designing better regularizers to improve the adversarial robustness.
Revised: This is only one step in designing better regularizers to improve the adversarial robustness.

Revised: Several directions deserve further study, with the prominent one being

Revised: SOAR’s vulnerabilities to AutoAttack.

Original: One future direction is to understand the loss surface of DNN better in order to select a good point around which an accurate Taylor approximation can be made.
Revised: Another future direction is to understand the loss surface of DNN better in order to select a good point around which an accurate Taylor approximation can be made.

Original: This is important for designing regularizers that are not affected by gradient masking.
Revised: This is important for designing regularizers that are not affected by gradient masking.

Original: A D ERIVATIONS OF S ECTION 2: L INEAR R EGRESSION WITH AN O VER - PARAMETRIZED M ODEL
Revised: A D ERIVATIONS OF S ECTION 2: L INEAR R EGRESSION WITH AN O VER - PARAMETRIZED M ODEL

Original: We derive the results reported in Section 2 in more detail here.
Revised: We derive the results reported in Section 2 in more detail here.

Original: Recall that we consider a linear model f w ( x ) =
Revised: Recall that we consider a linear model f w ( x ) =

Original: (cid:104) w , x (cid:105) with x, w ∈ R d .
Revised: (cid:104) w , x (cid:105) with x, w ∈ R d .

Original: We suppose that w ∗ =
Revised: We suppose that w ∗ =

Original: . , 0) (cid:62) and the distribution of x ∼ p is such that it is conﬁned on a 1 -dimensional subspace { ( x 1 , 0 , 0 , . .
Revised: . , 0) (cid:62) and the distribution of x ∼ p is such that it is conﬁned on a 1 -dimensional subspace { ( x 1 , 0 , 0 , . .

Original: . , 0) : x 1 ∈ R } .
Revised: . , 0) : x 1 ∈ R } .

Original: So the density of x is p (( x 1 , . . .
Revised: So the density of x is p (( x 1 , . . .

Original: , x d ))
Revised: , x d ))

Original: =
Revised: =

Original: p 1 ( x 1 ) δ ( x 2 ) δ ( x 3 ) . . .
Revised: p 1 ( x 1 ) δ ( x 2 ) δ ( x 3 ) . . .

Original: δ ( x d ) , where δ ( · ) is Dirac’s delta function.
Revised: δ ( x d ) , where δ ( · ) is Dirac’s delta function.

Original: We initialize the weights at the ﬁrst time step as w (0) ∼ N (0 , σ 2 I d × d ) , and use GD to ﬁnd the minimizer of the population loss.
Revised: We initialize the weights at the ﬁrst time step as w (0) ∼ N (0 , σ 2 I d × d ) , and use GD to ﬁnd the minimizer of the population loss.

Original: The partial derivatives of the population loss are
Revised: The partial derivatives of the population loss are

Original: [Equation]
Revised: [Equation]

Original: where µ 1 = E [ X 1 ] .
Revised: where µ 1 = E [ X 1 ] .

Original: Notice that the gradient in dimension j = 1 is non-zero, unless ( w 1 − w ∗ 1 ) µ 1 = 0 .
Revised: Notice that the gradient in dimension j = 1 is non-zero, unless ( w 1 − w ∗ 1 ) µ 1 = 0 .

Original: Assuming that µ 1 (cid:54) = 0 , this implies that the gradient won’t be zero unless w 1 = w ∗ 1 .
Revised: Assuming that µ 1 (cid:54) = 0 , this implies that the gradient won’t be zero unless w 1 = w ∗ 1 .

Original: On the other hand, the gradients in dimensions j = 2 , . . .
Revised: On the other hand, the gradients in dimensions j = 2 , . . .

Original: , d are all zero, so GD does not change the value of w j ( t ) for j = 2 , . . .
Revised: , d are all zero, so GD does not change the value of w j ( t ) for j = 2 , . . .

Original: , d .
Revised: , d .

Original: Therefore, under the proper choice of learning rate β , we get that the asymptotic solution of GD solution is ¯ w (cid:44)
Revised: Therefore, under the proper choice of learning rate β , we get that the asymptotic solution of GD solution is ¯ w (cid:44)

Original: lim
Revised: lim

Original: r →∞ w ( t ) =
Revised: r →∞ w ( t ) =

Original: ( w ∗ 1 , w 2 (0) , w 3 (0) , . . .
Revised: ( w ∗ 1 , w 2 (0) , w 3 (0) , . . .

Original: , w d (0)) (cid:62) .
Revised: , w d (0)) (cid:62) .

Original: It is clear that
Revised: It is clear that

Original: L ( ¯ w ) = 0 , i.e., the population loss is zero, as noted already as our ﬁrst observation in that section.
Revised: L ( ¯ w ) = 0 , i.e., the population loss is zero, as noted already as our ﬁrst observation in that section.

Original: Also note that we can easily attack this model by perturbing x by ∆ x = (0 , ∆ x 2 , ∆ x 3 , . . . , ∆ x d ) (cid:62) .
Revised: Also note that we can easily attack this model by perturbing x by ∆ x = (0 , ∆ x 2 , ∆ x 3 , . . . , ∆ x d ) (cid:62) .

Original: The pointwise loss at x + ∆ x is
Revised: The pointwise loss at x + ∆ x is

Original: [Equation]
Revised: [Equation]

Original: With the choice of ∆ x i = ε sign( w i (0)) (for i = 2 , . . .
Revised: With the choice of ∆ x i = ε sign( w i (0)) (for i = 2 , . . .

Original: , d ) and ∆ x 1 = 0 , an FGSM-like attack (Goodfellow et al., 2014) at the learned weight ¯ w leads to the pointwise loss of
Revised: , d ) and ∆ x 1 = 0 , an FGSM-like attack (Goodfellow et al., 2014) at the learned weight ¯ w leads to the pointwise loss of

Original: [Equation]
Revised: [Equation]

Original: We comment that our choice of ∆ x is not from the same distribution as the training data x .
Revised: We comment that our choice of ∆ x is not from the same distribution as the training data x .

Original: This choice aligns with the hypotheses in Ding et al. (2019a);
Revised: This choice aligns with the hypotheses in Ding et al. (2019a);

Original: Schmidt et al.
Revised: Schmidt et al.

Original: that adversarial examples come from a shifted data distribution; however, techniques such as feature adversaries (Sabour et al., 2015) focus on designing perturbations to be close to input distributions.
Revised: that adversarial examples come from a shifted data distribution; however, techniques such as feature adversaries (Sabour et al., 2015) focus on designing perturbations to be close to input distributions.

Original: We stress that the goal here is to illustrate the loss under this particular attack.
Revised: We stress that the goal here is to illustrate the loss under this particular attack.

Original: In order to get a better sense of this loss, we compute its expected value w.r.t.
Revised: In order to get a better sense of this loss, we compute its expected value w.r.t.

Original: the randomness of weight initialization.
Revised: the randomness of weight initialization.

Original: We have that (including the extra | w 1 (0) | term too)
Revised: We have that (including the extra | w 1 (0) | term too)

Original: E W ∼ N (0 ,σ 2 I d × d ) (cid:104) (cid:107) W (cid:107) 21 (cid:105) =
Revised: E W ∼ N (0 ,σ 2 I d × d ) (cid:104) (cid:107) W (cid:107) 21 (cid:105) =

Original: E   d (cid:88) i,j = | W
Revised: E   d (cid:88) i,j = | W

Original: i ||
Revised: i ||

Original: W j | 
Revised: W j | 

Original:  =
Revised:  [Equation]

Original: d (cid:88)
Revised: d (cid:88)

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: E (cid:2) | W i |
Revised: E (cid:2) | W i |

Original: [Equation]
Revised: [Equation]

Original: (cid:3)
Revised: (cid:3)

Original: [Equation]
Revised: [Equation]

Original: d (cid:88) i,j =1 ,i (cid:54)=
Revised: d (cid:88) i,j =1 ,i (cid:54)

Revised: [Equation]

Original: j
Revised: j

Original: E [ | W i | ] E
Revised: E [ | W i | ] E

Original: [ | W j | ] , where we used the independence of the r.v. W i and W j when i (cid:54) =
Revised: [ | W j | ] , where we used the independence of the r.v. W i and W j when i (cid:54) =

Original: j .
Revised: j .

Original: The expectation E (cid:2) | W i | 2 (cid:3) is the variance σ 2 of W i .
Revised: The expectation E (cid:2) | W i | 2 (cid:3) is the variance σ 2 of W i .

Original: The r.v. | W j | has a folded normal distribution, and its expectation E [ | W j | ] is (cid:113) 2 π σ .
Revised: The r.v. | W j | has a folded normal distribution, and its expectation E [ | W j | ] is (cid:113) 2 π σ .

Original: Thus, we get that
Revised: Thus, we get that

Original: [Equation]
Revised: [Equation]

Original: for d (cid:29) 1 .
Revised: for d (cid:29) 1 .

Original: The expected population loss of the speciﬁed attack ∆ x at the asymptotic solution ¯ w is
Revised: The expected population loss of the speciﬁed attack ∆ x at the asymptotic solution ¯ w is

Original: E X,W [ l ( X + ∆ x ); ¯ w )]
Revised: E X,W [ l ( X + ∆ x ); ¯ w )]

Original: ≈ O ( ε 2 d 2 σ 2 ) .
Revised: ≈ O ( ε 2 d 2 σ 2 ) .

Original: The dependence of this loss on dimension d is signiﬁcant, showing that the learned model is quite vulnerable to attacks.
Revised: The dependence of this loss on dimension d is signiﬁcant, showing that the learned model is quite vulnerable to attacks.

Original: We note that the conclusions would not change much with initial distributions other than the Normal distribution.
Revised: We note that the conclusions would not change much with initial distributions other than the Normal distribution.

Original: An effective solution is to regularize the loss to encourage the weights of irrelevant dimensions going to zero.
Revised: An effective solution is to regularize the loss to encourage the weights of irrelevant dimensions going to zero.

Original: A generic regularizer is to use the (cid:96) 2 -norm of the weights, i.e., formulate the problem as a ridge regression.
Revised: A generic regularizer is to use the (cid:96) 2 -norm of the weights, i.e., formulate the problem as a ridge regression.

Original: In that case, the regularized population loss is
Revised: In that case, the regularized population loss is

Original: [Equation]
Revised: [Equation]

Original: One can see that the solution of ∇ w L ridge ( w ) = 0 is ¯ w 1 ( λ )
Revised: One can see that the solution of ∇ w L ridge ( w ) = 0 is ¯ w 1 ( λ )

Original: = µ 1 µ 1 + λ w ∗ 1 and ¯ w j ( λ ) = 

Revised: = µ 1 µ 1 + λ w ∗ 1 and ¯ w j ( λ ) =

Original: for j (cid:54) = 1 .
Revised: for j (cid:54) = 1 .

Original: [Equation]
Revised: [Equation]

Original: The use of this generic regularizer seems reasonable in this example, as it enforces the weights for dimensions 2 to d to become zero.
Revised: The use of this generic regularizer seems reasonable in this example, as it enforces the weights for dimensions 2 to d to become zero.

Original: Its only drawback is that it leads to a biased estimate of w ∗ 1 .
Revised: Its only drawback is that it leads to a biased estimate of w ∗ 1 .

Original: The bias, however, can be made small with a small choice for λ .
Revised: The bias, however, can be made small with a small choice for λ .

Original: We can obtain a similar conclusion for the (cid:96) 1 regularizer (Lasso).
Revised: We can obtain a similar conclusion for the (cid:96) 1 regularizer (Lasso).

Original: As such, one has to deﬁne a regularizer that is specially-designed for improving adversarial robustness.
Revised: As such, one has to deﬁne a regularizer that is specially-designed for improving adversarial robustness.

Original: Bishop (1995) showed the strong connection between training with random perturbation and Tikhonov Regularization.
Revised: Bishop (1995) showed the strong connection between training with random perturbation and Tikhonov Regularization.

Original: Inspired by this idea, we develop a regularizer that mimics the adversary itself.
Revised: Inspired by this idea, we develop a regularizer that mimics the adversary itself.

Original: Let us assume that a particular adversary attacks the model by adding ∆ x = (0 , ε sign( w 2 (0)) , . .
Revised: Let us assume that a particular adversary attacks the model by adding ∆ x = (0 , ε sign( w 2 (0)) , . .

Original: . , ε sign( w d (0))
Revised: . , ε sign( w d (0))

Original: (cid:62) .
Revised: (cid:62) .

Original: The population loss at the perturbed point is
Revised: The population loss at the perturbed point is

Original: L
Revised: L

Original: [Equation]
Revised: [Equation]

Original: (cid:44)
Revised: (cid:44)

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where (cid:107) w 2: d (cid:107) 1 = (cid:80) dj
Revised: where (cid:107) w 2: d (cid:107) 1 = (cid:80) dj

Original: =2 | w j | .
Revised: =2 | w j | .

Original: 1 This is the same objective as (1) reported in Section 2.
Revised: 1 This is the same objective as (1) reported in Section 2.

Original: Note that minimizing L robustiﬁed ( w ) is equivalent to minimizing the model at the point x (cid:48)
Revised: Note that minimizing L robustiﬁed ( w ) is equivalent to minimizing the model at the point x (cid:48)

Original: = x
Revised: = x

Original: + ∆ x .
Revised: + ∆ x .

Original: The regularizer ε E
Revised: The regularizer ε E

Original: [ r ( X ; w )] (cid:107) w 2: d (cid:107) 1 + ε 2 2 (cid:107) w 2: d (cid:107) 21 incorporates the effect of adversary in exact form.
Revised: [ r ( X ; w )] (cid:107) w 2: d (cid:107) 1 + ε 2 2 (cid:107) w 2: d (cid:107) 21 incorporates the effect of adversary in exact form.

Original: This motivated the possibility of designing a regularized tailored to prevent attacks.
Revised: This motivated the possibility of designing a regularized tailored to prevent attacks.

Original: A. D ERIVATION OF THE POPULATION LOSS UNDER ITS FIRST AND SECOND ORDER APPRXOIMATION
Revised: A. D ERIVATION OF THE POPULATION LOSS UNDER ITS FIRST AND SECOND ORDER APPRXOIMATION

Original: First, we show that the FGSM direction is the maximizer of the loss when the perturbation is (cid:96) ∞ constrained.
Revised: First, we show that the FGSM direction is the maximizer of the loss when the perturbation is (cid:96) ∞ constrained.

Original: Based on the pointwise loss at x + ∆ x , we have
Revised: Based on the pointwise loss at x + ∆ x , we have

Original: [Equation]
Revised: [Equation]

Original: We use the Cauchy-Schwarz inequality to obtain
Revised: We use the Cauchy-Schwarz inequality to obtain

Original: [Equation]
Revised: [Equation]

Original: which leads to
Revised: which leads to

Original: [Equation]
Revised: [Equation]

Original: Next, we show that the ﬁrst-order approximation of E [ l ( X + ∆ x ; w )] obtains the ﬁrst two terms in (1).
Revised: Next, we show that the ﬁrst-order approximation of E [ l ( X + ∆ x ; w )] obtains the ﬁrst two terms in (1).

Original: Note the gradient of the loss w.r.t.
Revised: Note the gradient of the loss w.r.t.

Original: the input is
Revised: the input is

Original: [Equation]
Revised: [Equation]

Original: and the Hessian w.r.t.
Revised: and the Hessian w.r.t.

Original: the input is
Revised: the input is

Original: [Equation]
Revised: [Equation]

Original: The ﬁrst-order Taylor series approximation is
Revised: The ﬁrst-order Taylor series approximation is

Original: [Equation]
Revised: [Equation]

Original: Note that w ∗(cid:62) ∆ x = 0 because of our particular choice of ∆ x and w ∗ .
Revised: Note that w ∗(cid:62) ∆ x = 0 because of our particular choice of ∆ x and w ∗ .

Original: Here we obtain the ﬁrst two terms in (1).
Revised: Here we obtain the ﬁrst two terms in (1).

Original: The second-order Taylor series approximation is
Revised: The second-order Taylor series approximation is

Original: [Equation]
Revised: [Equation]

Original: which recovers the exact form in (1).
Revised: which recovers the exact form in (1).

Original: This completes the motivation of using second-order Taylor series approximation with our warm-up toy example.
Revised: This completes the motivation of using second-order Taylor series approximation with our warm-up toy example.

Original: B D ERIVATIONS OF S ECTION 4: S ECOND -O RDER A DVERSARIAL R EGULARIZER (SOAR) B. R ELAXATION
Revised: B D ERIVATIONS OF S ECTION 4: S ECOND -O RDER A DVERSARIAL R EGULARIZER (SOAR) B. R ELAXATION

Original: Note the Boolean quadratic programming (BQP) problem in formulation (5) is NP-hard (Beasley, 1998; Lima & Grossmann, 2017).
Revised: Note the Boolean quadratic programming (BQP) problem in formulation (5) is NP-hard (Beasley, 1998; Lima & Grossmann, 2017).

Original: Even though there exist semi-deﬁnite programming (SDP) relaxations, such approaches require the exact Hessian w.r.t.
Revised: Even though there exist semi-deﬁnite programming (SDP) relaxations, such approaches require the exact Hessian w.r.t.

Original: the input, which is computationally expensive to obtain for high-dimensional inputs.
Revised: the input, which is computationally expensive to obtain for high-dimensional inputs.

Original: And even if we could compute the exact Hessian, SDP itself is a computationally expensive approach, and not suitable to be within the inner loop of a DNN training.
Revised: And even if we could compute the exact Hessian, SDP itself is a computationally expensive approach, and not suitable to be within the inner loop of a DNN training.

Original: As such, we relax the (cid:96) ∞ constraint to an (cid:96) 2 constraint, which as we see, leads to a computationally efﬁcient solution.
Revised: As such, we relax the (cid:96) ∞ constraint to an (cid:96) 2 constraint, which as we see, leads to a computationally efﬁcient solution.

Original: B. T HE TIGHTNESS OF THE BOUND IN E Q (7)
Revised: B. T HE TIGHTNESS OF THE BOUND IN E Q (7)

Original: Inpractice, the upper bound may not be loose after all.
Revised: In the ICLR rebuttal phase, the reviewer pointed out that, from the perspective of the volume ratio between the two (cid:96) p balls, replacing (cid:107) δ (cid:107) ∞ ≤ ε with (cid:107) δ (cid:107) 2 ≤ √ dε can be problematic since the volume of { δ : (cid:107) δ (cid:107) ∞

Original: To explain this, note that the claim that an ε − (cid:96) ∞ ball isenclosed within an √ dε − (cid:96) 2 ball follows from
Revised: ≤ ε } is 2 d ε d whereas the volume of { δ : (cid:107) δ (cid:107) 2 ≤ (cid:112) ( d ) ε } is π d [Equation] d [Equation]

Revised: ε

Original: [Equation]
Revised: [Equation]

Original: This bound is loose when δ is sparse.
Revised: Their ratio goes to 0 as the dimension increases.

Original: It is tight, however, when δ isdense with components that are mostly ± (cid:107) δ (cid:107) ∞ . When all components are ofthis form, the inequality becomes an equality.
Revised: The implication is that the search space for the (cid:96) ∞ maximizer is inﬁnitesimal compared to the one for the (cid:96) 2 maximizer, leading to a loose upper-bound. As a preliminary study on the tightness of the bound, we evaluated the two slides of (7) by approximating the maximum using PGD attacks.

Original: This seems to be the type of δ that an (cid:96)  ∞ -based attack often generates.
Revised: In particular, we approximate max || δ || ∞ ≤ (cid:15) (cid:96) ( x + δ ) using (cid:96) ( x + δ ∞ ) where δ ∞ is generated using 20-iteration (cid:96) ∞ -PGD with (cid:15) = 8255 .

Original: For example, we empirically observed that for the (cid:96) ∞
Revised: Similarly, we approximate max || δ || 2 ≤√ d(cid:15)

Original: PGD20 attack ( ε = 8255 ) on a standard trained model on CIFAR10 testset, the average (cid:96)  2-norm of the perturbations is 1.37, which should be compared to √ dε = 1 . 74 .
Revised: (cid:96) ( x + δ ) using (cid:96) ( x + δ 2 ) where δ 2 is generated using 100-iteration (cid:96) 2 -PGD with (cid:15) = 1 . The reason for this particular conﬁguration of attack parameter is to match the ones used during our previous evaluations. [Table] From this preliminary study, we observe that there is indeed a gap between the approximated

Revised: LHS and RHS of (7), and thus, we think it is a valuable future research direction to explore other possibilities that allow us to use a second-order approximation to study the worst-case loss subject to an constrained perturbation.

Original: B. U NIFIED O BJECTIVE
Revised: B. U NIFIED O BJECTIVE

Original: We could maximize each term inside (8) separately and upper bound the max by
Revised: We could maximize each term inside (8) separately and upper bound the max by

Original: [Equation]
Revised: [Equation]

Original: where σ max ( ∇ 2 (cid:96) ( x )) is the largest singular value of the Hessian matrix, ∇ 2 (cid:96) ( x ) .
Revised: where σ max ( ∇ 2 (cid:96) ( x )) is the largest singular value of the Hessian matrix, ∇ 2 (cid:96) ( x ) .

Original: Even though the norm of the gradient and the singular value of the Hessian have an intuitive appeal, separately optimizing these terms might lead to a looser upper bound than necessary.
Revised: Even though the norm of the gradient and the singular value of the Hessian have an intuitive appeal, separately optimizing these terms might lead to a looser upper bound than necessary.

Original: The reason is that the maximizer of the ﬁrst two terms are argmax (cid:12)(cid:12) ∇ [Equation] x [Equation]
Revised: The reason is that the maximizer of the ﬁrst two terms are argmax (cid:12)(cid:12) ∇ (cid:96) ( x ) (cid:62) δ (cid:12)(cid:12) =

Original: x

Original: [Equation] x ) [Equation] 2 and the direction corresponding to the largest singular value of ∇ 2 (cid:96) ( x ) .
Revised: ∇ (cid:96) ( x ) (cid:107)∇ (cid:96) ( x ) (cid:107) 2 and the direction corresponding to the largest singular value of ∇ 2 (cid:96) ( x ) .

Original: In general, these two directions are not aligned.
Revised: In general, these two directions are not aligned.

Original: B. P ROOF OF P ROPOSITION
Revised: B. P ROOF OF P ROPOSITION

Original: Proof.
Revised: Proof.

Original: By the inclusion of the (cid:96) ∞ -ball of radius ε within the (cid:96) 2 -ball of radius √ dε and the deﬁnition of H in (6), we have
Revised: By the inclusion of the (cid:96) ∞ -ball of radius ε within the (cid:96) 2 -ball of radius √ dε and the deﬁnition of H in (6) [Equation] we have

Original: [Equation]
Revised: [Equation]

Original: It remains to upper bound max (cid:107) δ
Revised: It remains to upper bound max (cid:107) δ

Original: [Equation]
Revised: [Equation]

Original: δ
Revised: (cid:107) 2 ≤ ε

Original: [Equation]
Revised: [Equation]

Original:  H δ [Equation] with ε [Equation]
Revised: δ (cid:48)(cid:62) H δ (cid:48) with ε (cid:48)

Original: =
Revised: =

Original: [Equation]

Original:  dε 2 + 1 .
Revised: √ dε 2 + 1 .

Original: We use the Cauchy-Schwarz inequality to obtain
Revised: We use the Cauchy-Schwarz inequality to obtain

Original: [Equation]
Revised: [Equation]

Original: where the last equality is obtained using properties of the (cid:96) 2 -induced matrix norm (this is the spectral norm).
Revised: where the last equality is obtained using properties of the (cid:96) 2 -induced matrix norm (this is the spectral norm).

Original: Since computing (cid:107) H (cid:107) 2 would again require the exact input Hessian, and we would like to avoid it, we further upper bound the spectral norm by the Frobenius norm as
Revised: Since computing (cid:107) H (cid:107) 2 would again require the exact input Hessian, and we would like to avoid it, we further upper bound the spectral norm by the Frobenius norm as

Original: [Equation]
Revised: [Equation]

Original: The Frobenius norm itself satisﬁes
Revised: The Frobenius norm itself satisﬁes

Original: [Equation]
Revised: [Equation]

Original:  where z ∼ N (0 , I ( d +1) ×
Revised: ) where z ∼ N (0 , I ( d +1) ×

Original: ( d +1) ) .
Revised: ( d +1) ) .

Original: Therefore, we can estimate (cid:107) H (cid:107) F by sampling random vectors z and compute the sample average of (cid:107) H z (cid:107) 2 .
Revised: Therefore, we can estimate (cid:107) H (cid:107) F by sampling random vectors z and compute the sample average of (cid:107) H z (cid:107) 2 .

Original: C SOAR A LGROITHM : A C OMPLETE I LLUSTRATION
Revised: C SOAR A LGROITHM : A C OMPLETE I LLUSTRATION

Original: In Algorithm 1, we present the inner-loop operation of SOAR using a single data point.
Revised: In Algorithm 1, we present the inner-loop operation of SOAR using a single data point.

Original: Here we summarize the full training procedure with SOAR in Algorithm 2.
Revised: Here we summarize the full training procedure with SOAR in Algorithm 2.

Original: Note that it is presented as if the optimizer is SGD, but we may use other optimizers as well.
Revised: Note that it is presented as if the optimizer is SGD, but we may use other optimizers as well.

Original: D P OTENTIAL C AUSES OF G RADIENT M ASKING
Revised: D P OTENTIAL C AUSES OF G RADIENT M ASKING

Original: [Table]
Revised: [Table]

Original: We summarize the average value of the highest probability output for test set data initialized with zero, random and PGD1 perturbations in Table 3.
Revised: We summarize the average value of the highest probability output for test set data initialized with zero, random and PGD1 perturbations in Table 5.

Original: We notice that training with SOAR using zero or random initialization leads to models with nearly 100% conﬁdence on their predictions.
Revised: We notice that training with SOAR using zero or random initialization leads to models with nearly 100% conﬁdence on their predictions.

Original: This is aligned with the analysis of SOAR for a linear classiﬁer (Section 4.1)
Revised: This is aligned with the analysis of SOAR for a linear classiﬁer (Section 4.1)

Original: [Table]
Revised: [Table]

Original: which shows that the regularizer becomes ineffective as the model outputs high conﬁdence predictions.
Revised: which shows that the regularizer becomes ineffective as the model outputs high conﬁdence predictions.

Original: Indeed, results in Table 5 show that those models are vulnerable under black-box attacks.
Revised: Indeed, results in Table 7 show that those models are vulnerable under black-box attacks.

Original: Results in Table 3 suggest that highly conﬁdent predictions could be an indication for gradient masking.
Revised: Results in Table 5 suggest that highly conﬁdent predictions could be an indication for gradient masking.

Original: We demonstrate this using the gradient-based PGD attack.
Revised: We demonstrate this using the gradient-based PGD attack.

Original: Recall that we generate PGD attacks by ﬁrst initializing the clean data x n with a randomly chosen η within the (cid:96) ∞ ball of size ε , followed by gradient ascent at x n + η .
Revised: Recall that we generate PGD attacks by ﬁrst initializing the clean data x n with a randomly chosen η within the (cid:96) ∞ ball of size ε , followed by gradient ascent at x n + η .

Original: Suppose that the model makes predictions with 100% conﬁdence on any given input.
Revised: Suppose that the model makes predictions with 100% conﬁdence on any given input.

Original: This leads to a piece-wise loss surface that is either zero (correct predictions) or inﬁnity (incorrect predictions).
Revised: This leads to a piece-wise loss surface that is either zero (correct predictions) or inﬁnity (incorrect predictions).

Original: The gradient of this loss function is either zero or undeﬁned, and thus making gradient ascent ineffective.
Revised: The gradient of this loss function is either zero or undeﬁned, and thus making gradient ascent ineffective.

Original: Therefore, white-box gradient-based attacks are unable to ﬁnd adversarial examples.
Revised: Therefore, white-box gradient-based attacks are unable to ﬁnd adversarial examples.

Original: E S UPPLEMENTARY E XPERIMENTS E. D ISCUSSION ON THE R EPRODUCIBILITY OF CURE AND LLR
Revised: E S UPPLEMENTARY E XPERIMENTS E. D ISCUSSION ON THE R EPRODUCIBILITY OF CURE AND LLR

Original: We were not able to reproduce results of two closely related works, CURE (Moosavi-Dezfooli et al., 2019) and LLR (Qin et al., 2019).
Revised: We were not able to reproduce results of two closely related works, CURE (Moosavi-Dezfooli et al., 2019) and LLR (Qin et al., 2019).

Original: For CURE, we found the open-source implementation 2 , but were not able to reproduce their reported results using their implmentation.
Revised: For CURE, we found the open-source implementation 2 , but were not able to reproduce their reported results using their implmentation.

Original: We were not able to reproduce the results of CURE with our own implementation either.
Revised: We were not able to reproduce the results of CURE with our own implementation either.

Original: For LLR, Yang et al.
Revised: For LLR, Yang et al.

Original: (2020) were not able to reproduce the results, they also provided an open-source implementation 3 .
Revised: (2020) were not able to reproduce the results, they also provided an open-source implementation 3 .

Original: Regardless, we compare SOAR to the reported result by CURE and LLR in Table 4:
Revised: Regardless, we compare SOAR to the reported result by CURE and LLR in Table 6:

Original: [Table]
Revised: [Table]

Original: E. T RAINING AND E VALUATION SETUP
Revised: E. T RAINING AND E VALUATION SETUP

Original: CIFAR-10 :
Revised: CIFAR-10 :

Original: Training data is augmented with random crops and horizontal ﬂips.
Revised: Training data is augmented with random crops and horizontal ﬂips.

Original: ResNet : We used an open-source ResNet-10 implementation 4 .
Revised: ResNet : We used an open-source ResNet-10 implementation 4 .

Original: More speciﬁcally, we initialize the model with ResNet(BasicBlock, [1,1,1,1]) .
Revised: More speciﬁcally, we initialize the model with ResNet(BasicBlock, [1,1,1,1]) .

Original: Note that we remove the BatchNorm layers in the ResNet-10 architecture, and we discuss this further in Appendix E.6 .
Revised: Note that we remove the BatchNorm layers in the ResNet-10 architecture, and we discuss this further in Appendix E.7 .

Original: WideResNet : We used the implementation 5 of WideResNet-34-10 model found in public repository maintained by the authors of TRADES (Zhang et al., 2019).
Revised: WideResNet : We used the implementation 5 of WideResNet-34-10 model found in public repository maintained by the authors of TRADES (Zhang et al., 2019).

Original: Standard training on ResNet and WideResNet : Both are trained for a total of 200 epochs, with an initial learning rate of 0.1.
Revised: Standard training on ResNet and WideResNet : Both are trained for a total of 200 epochs, with an initial learning rate of 0.1.

Original: The learning rate decays by an order of magnitude at epoch 100 and 150.
Revised: The learning rate decays by an order of magnitude at epoch 100 and 150.

Original: We used a minibatch size of 128 for testing and training.
Revised: We used a minibatch size of 128 for testing and training.

Original: We used SGD optimizer with momentum of 0.9 and a weight decay of 2e-4.
Revised: We used SGD optimizer with momentum of 0.9 and a weight decay of 2e-4.

Original: Adversarial training with PGD10 examples on ResNet : The optimization setting is the same as the one used for standard training.
Revised: Adversarial training with PGD10 examples on ResNet : The optimization setting is the same as the one used for standard training.

Original: Additionally, to ensure that the ﬁnal model has the highest adversarial robustness, we save the model at the end of every epoch, and the ﬁnal evaluation is based on the one with the highest PGD20 accuracy.
Revised: Additionally, to ensure that the ﬁnal model has the highest adversarial robustness, we save the model at the end of every epoch, and the ﬁnal evaluation is based on the one with the highest PGD20 accuracy.

Original: SOAR on ResNet : SOAR refers to continuing the training of the Standard model on ResNet.
Revised: SOAR on ResNet : SOAR refers to continuing the training of the Standard model on ResNet.

Original: It is trained for a total of 200 epochs with an initial learning rate of 0.004 and decay by an order of magnitude at epoch 100.
Revised: It is trained for a total of 200 epochs with an initial learning rate of 0.004 and decay by an order of magnitude at epoch 100.

Original: We used SGD optimizer with momentum of 0.9 and a weight decay of 2e-4.
Revised: We used SGD optimizer with momentum of 0.9 and a weight decay of 2e-4.

Original: We use a FD step-size h = 0 .
Revised: We use a FD step-size h = 0 .

Original: 01 for the regularizer.
Revised: 01 for the regularizer.

Original: Additionally, we apply a clipping of 10 on the regularizer, and we discuss this clipping operation in Appendix E.6 .
Revised: Additionally, we apply a clipping of 10 on the regularizer, and we discuss this clipping operation in Appendix E.7 .

Original: MART and TRADES on ResNet : We used the same optimization setup as the ones in their respective public repository 6 .
Revised: MART and TRADES on ResNet : We used the same optimization setup as the ones in their respective public repository 6 .

Original: We brieﬂy summarize it here.
Revised: We brieﬂy summarize it here.

Original: The model is trained for a total ofepochs, with an initial learning rate of 0.1.
Revised: The model is trained for a total ofepochs, with an initial learning rate of 0.1.

Original: The learning rate decays by an order of magnitude at epoch 75, 90, 100.
Revised: The learning rate decays by an order of magnitude at epoch 75, 90, 100.

Original: We used SGD optimizer with momentum of 0.9 and a weight decay of 2e-4.
Revised: We used SGD optimizer with momentum of 0.9 and a weight decay of 2e-4.

Original: We performed a hyperparameter sweep on the strength of the regularization term β and selected one that resulted in the best performance against PGD20 attacks.
Revised: We performed a hyperparameter sweep on the strength of the regularization term β and selected one that resulted in the best performance against PGD20 attacks.

Original: A complete result is reported in
Revised: A complete result is reported in

Original: Appendix E.10 .
Revised: Appendix E.12 .

Original: MMA on ResNet : We used the same optimization setup as the one in its public repository 7 .
Revised: MMA on ResNet : We used the same optimization setup as the one in its public repository 7 .

Original: We brieﬂy summarize it here.
Revised: We brieﬂy summarize it here.

Original: The model is trained for a total of 50000 iterations, with an initial learning rate of 0.3.
Revised: The model is trained for a total of 50000 iterations, with an initial learning rate of 0.3.

Original: The learning rate changes to 0.09 at the 20000 iteration, 0.03 at the 30000 iteration and lastly 0.009 at the 40000 iteration.
Revised: The learning rate changes to 0.09 at the 20000 iteration, 0.03 at the 30000 iteration and lastly 0.009 at the 40000 iteration.

Original: We used SGD optimizer with momentum of 0.9 and a weight decay of 2e-4.
Revised: We used SGD optimizer with momentum of 0.9 and a weight decay of 2e-4.

Original: We performed a hyperparameter sweep on the margin term and selected the one that resulted in the best performance against PGD20 attacks.
Revised: We performed a hyperparameter sweep on the margin term and selected the one that resulted in the best performance against PGD20 attacks.

Original: A complete result is reported in Appendix E.10 .
Revised: A complete result is reported in Appendix E.12 .

Original: ADV, TRADES, MART and MMA on WideResNet : We use the pretrained checkpoint provided in their respective repositories.
Revised: ADV, TRADES, MART and MMA on WideResNet : We use the pretrained checkpoint provided in their respective repositories.

Original: Note that we use the pretrained checkpoint for PGD10 adversarially trained WideResNet in Madry’s CIFAR10 Challenge 8 .
Revised: Note that we use the pretrained checkpoint for PGD10 adversarially trained WideResNet in Madry’s CIFAR10 Challenge 8 .

Original: Evaluations : For FGSM and PGD attacks, we use the implementation in AdverTorch (Ding et al., 2019c).
Revised: Evaluations : For FGSM and PGD attacks, we use the implementation in AdverTorch (Ding et al., 2019c).

Original: For SimBA (Guo et al., 2019), we use the authors’ open-source implementation 9 .
Revised: For SimBA (Guo et al., 2019), we use the authors’ open-source implementation 9 .

Original: E. A DVERSARIAL R OBUSTNESS OF THE M ODEL T RAINED USING SOAR WITH D IFFERENT I NITIALIZATIONS
Revised: E. A DVERSARIAL R OBUSTNESS OF THE M ODEL T RAINED USING SOAR WITH D IFFERENT I NITIALIZATIONS

Original: [Table]
Revised: [Table]

Original: We report the adversarial robustness of the model trained using SOAR with different initialization techniques in Table 5.
Revised: We report the adversarial robustness of the model trained using SOAR with different initialization techniques in Table 7.

Original: The second column shows the accuracy against white-box PGD 20 adversaries.
Revised: The second column shows the accuracy against white-box PGD 20 adversaries.

Original: The third column shows the accuracy against black-box PGD 20 adversaries transferred from an independently initialized and standard-trained ResNet-10 model.
Revised: The third column shows the accuracy against black-box PGD 20 adversaries transferred from an independently initialized and standard-trained ResNet-10 model.

Original: Note that despite the high adversarial accuracy against white-box PGD attacks, models trained using
Revised: Note that despite the high adversarial accuracy against white-box PGD attacks, models trained using

Original: SOAR with zero and random initialization perform poorly against transferred attacks.
Revised: SOAR with zero and random initialization perform poorly against transferred attacks.

Original: This suggests the presence of gradient masking when using SOAR with zero and random initializations.
Revised: This suggests the presence of gradient masking when using SOAR with zero and random initializations.

Original: Evidently,
Revised: Evidently,

Original: SOAR with PGD 1 initialization alleviates the gradient masking problem.
Revised: SOAR with PGD 1 initialization alleviates the gradient masking problem.

Revised: E. C OMPARING THE VALUES OF THE SOAR REGULARIZED LOSS COMPUTED USING DIFFERENT NUMBERS OF RANDOMLY SAMPLED z

Revised: [Table]

Original: E. R OBUSTNESS UNDER (cid:96) 2 ATTACKS ON CIFAR-{+SOAR ( x, y, n ) = (cid:96) ( x (cid:48) , y ) + 1 n (cid:80) ni =
+}
Revised: Suppose we slightly modify Eq (13) by (cid:96) {+SOAR ( x, y, n ) = (cid:96) ( x (cid:48) , y ) + 1 n (cid:80) ni =
+}

Revised: R ( x (cid:48) ; z ( i ) , h ) to incorporate the effect of using multiple randomly sampled vectors z ( i ) in computing the SOAR regularized loss.

Revised: Notice that the current implementation is equivalent to using n = 1 .

Revised: We observed the model at two checkpoints, at the beginning and the end of SOAR regularization, the value of the regularized loss remains unchanged as we increase n from 1 to 100.

Revised: E. R OBUSTNESS UNDER (cid:96) 2 ATTACKS ON CIFAR-

Original: We evaluate SOAR and two of the baseline methods, ADV and TRADES, against (cid:96) 2 white-box and black-box attacks on CIFAR-10 in Table 6.
Revised: We evaluate SOAR and two of the baseline methods, ADV and TRADES, against (cid:96) 2 white-box and black-box attacks on CIFAR-10 in Table 9.

Original: No (cid:96) 2 results were reported by MART
Revised: No (cid:96) 2 results were reported by MART

Original: and we are not able to reproduce the (cid:96) 2 results using the implementation by MMA, thus those two methods are not included in our evaluation.
Revised: and we are not able to reproduce the (cid:96) 2 results using the implementation by MMA, thus those two methods are not included in our evaluation.

Original: In Section 4, we show that the (cid:96) ∞ formulation of SOAR with (cid:107) δ (cid:107) ∞ = ε is equivalent to the (cid:96)formulation of SOAR with (cid:107) δ (cid:107) 2 = ε √ d .
Revised: In Section 4, we show that the (cid:96) ∞ formulation of SOAR with (cid:107) δ (cid:107) ∞ = ε is equivalent to the (cid:96)formulation of SOAR with (cid:107) δ (cid:107) 2 = ε √ d .

Original: In other words, models trained with SOAR to be robust against (cid:96) ∞ attacks with ε = 8255 should also obtain improved robustness against (cid:96) 2 attacks with ε = 8255 √ 32 ∗ 32 ∗ 3 = 1 . 74 .
Revised: In other words, models trained with SOAR to be robust against (cid:96) ∞ attacks with ε = 8255 should also obtain improved robustness against (cid:96) 2 attacks with ε = 8255 √ 32 ∗ 32 ∗ 3 = 1 . 74 .

Original: In our evaluation, all (cid:96) 2 adversaries used during ADV and TRADES are generated with 10-step PGD ( ε = 1 . 74 ) and a step size of 0 . 44 .
Revised: In our evaluation, all (cid:96) 2 adversaries used during ADV and TRADES are generated with 10-step PGD ( ε = 1 . 74 ) and a step size of 0 . 44 .

Original: Note that the goal here is to show the improved robustness of SOAR against (cid:96) 2 attacks other than being SOTA, thus the optimization procedures are the same as the ones used in the (cid:96) ∞ evaluation.
Revised: Note that the goal here is to show the improved robustness of SOAR against (cid:96) 2 attacks other than being SOTA, thus the optimization procedures are the same as the ones used in the (cid:96) ∞ evaluation.

Original: We observe that training with SOAR improves the robustness of the model against (cid:96) 2 attacks.
Revised: We observe that training with SOAR improves the robustness of the model against (cid:96) 2 attacks.

Original: Instead of a ﬁxed (cid:96) 2 norm, we demonstrate the improved robustness using an increasing range of ε .
Revised: Instead of a ﬁxed (cid:96) 2 norm, we demonstrate the improved robustness using an increasing range of ε .

Original: For all attacks, we use 100 iterations of PGD and a step size of 2 . 5 ε 100 .
Revised: For all attacks, we use 100 iterations of PGD and a step size of 2 . 5 ε 100 .

Original: In Table 6, we ﬁnd that training with
Revised: In Table 9, we ﬁnd that training with

Original: SOAR leads to a signiﬁcant increase in robustness against white-box and black-box (cid:96) 2 adversaries.
Revised: SOAR leads to a signiﬁcant increase in robustness against white-box and black-box (cid:96) 2 adversaries.

Original: As ε increases, SOAR model remain robust against white-box (cid:96) 2 attacks ( ε = 1 ), while other methods falls off.
Revised: As ε increases, SOAR model remain robust against white-box (cid:96) 2 attacks ( ε = 1 ), while other methods falls off.

Original: The last column of Table 6 shows the robustness against transferred (cid:96) 2 attacks ( ε = 1 . 74 ).
Revised: The last column of Table 9 shows the robustness against transferred (cid:96) 2 attacks ( ε = 1 . 74 ).

Original: The source model is a ResNet10 network trained separately from the defence models on the unperturbed training set.
Revised: The source model is a ResNet10 network trained separately from the defence models on the unperturbed training set.

Original: We observe that SOAR achieves the second highest robustness compared to baseline methods against transferred (cid:96) 2 attacks.
Revised: We observe that SOAR achieves the second highest robustness compared to baseline methods against transferred (cid:96) 2 attacks.

Original: This result empirically veriﬁes our previous claim that (cid:96) 2 and (cid:96) ∞ formulation of SOAR only differs by a factor of √ d .
Revised: This result empirically veriﬁes our previous claim that (cid:96) 2 and (cid:96) ∞ formulation of SOAR only differs by a factor of √ d .

Original: Moreover, it aligns with ﬁndings by Simon-Gabriel et al.
Revised: Moreover, it aligns with ﬁndings by Simon-Gabriel et al.

Original: (2019), that empirically showed adversarial robustness through regularization gains robustness against more than one norm-ball attack at the same time.
Revised: (2019), that empirically showed adversarial robustness through regularization gains robustness against more than one norm-ball attack at the same time [Table] [Table]

Original: [Table]
Revised: [Table]

Original: E. A DDITIONAL EVALUATION ON SVHN DATASET
Revised: E. A DDITIONAL EVALUATION ON SVHN DATASET

Original: We use the same ResNet-10 architecture as the one for CIFAR-10 evaluation.
Revised: We use the same ResNet-10 architecture as the one for CIFAR-10 evaluation.

Original: Training data is augmented with random crops and horizontal ﬂips.
Revised: Training data is augmented with random crops and horizontal ﬂips.

Original: For Standard training, we use the same optimization procedure as the one used for CIFAR-10.
Revised: For Standard training, we use the same optimization procedure as the one used for CIFAR-10.

Original: For SOAR and TRADES, we use the exact same hyperparameter for the regularizer.
Revised: For SOAR and TRADES, we use the exact same hyperparameter for the regularizer.

Original: For SOAR, we use early-stopping at epoch 130 to prevent catastrophic over-ﬁtting.
Revised: For SOAR, we use early-stopping at epoch 130 to prevent catastrophic over-ﬁtting.

Original: Besides, the optimization schedule is identical for SOAR and TRADES as the ones used for CIFAR-10.
Revised: Besides, the optimization schedule is identical for SOAR and TRADES as the ones used for CIFAR-10.

Original: We emphasize again that the goal of evaluting using SVHN is to demonstrate the improved robustness with SOAR on a different dataset, thus we did not perform an additional hyper-parameter sweep.
Revised: We emphasize again that the goal of evaluting using SVHN is to demonstrate the improved robustness with SOAR on a different dataset, thus we did not perform an additional hyper-parameter sweep.

Original: The optimization procedures are the same as the ones used in the CIFAR-10 evaluation. 
Revised: The optimization procedures are the same as the ones used in the CIFAR-10 evaluation. [Table]

Original:  For PGD10 adversarial training, we observe that ResNet-10 is not able to learn anything meaningful.
Revised: [Table] For PGD10 adversarial training, we observe that ResNet-10 is not able to learn anything meaningful.

Original: Speciﬁcally, when trained with PGD10 examples, ResNet-10 does not perform better than a randomlyinitialized network in both standard and adversarial accuracy.
Revised: Speciﬁcally, when trained with PGD10 examples, ResNet-10 does not perform better than a randomlyinitialized network in both standard and adversarial accuracy.

Original: Cai et al.
Revised: Cai et al.

Original: (2018) made a similar observation on ResNet-50, where training accuracy is not improving over a long period of adversarial training with PGD10.
Revised: (2018) made a similar observation on ResNet-50, where training accuracy is not improving over a long period of adversarial training with PGD10.

Original: They further investigated models with different capacities and found that even
Revised: They further investigated models with different capacities and found that even

Original: ResNet-50 might not be sufﬁciently deep for PGD10 adversarial training on SVHN.
Revised: ResNet-50 might not be sufﬁciently deep for PGD10 adversarial training on SVHN.

Original: Wang & Zhang (2019) reported PGD10 adversarial training result on SVHN with WideResNet, which we include in
Revised: Wang & Zhang (2019) reported PGD10 adversarial training result on SVHN with WideResNet, which we include in

Original: Table 7.
Revised: Table 11.

Original: [Table]

Original: [Table] For MART [Table] we were not able to translate their CIFAR-10 results on SVHN.
Revised: For MART, we were not able to translate their CIFAR-10 results on SVHN.

Original: We performed the same hyperparameter sweep as the one in Table [Table] as well as different optimization settings [Table]
Revised: We performed the same hyperparameter sweep as the one in Table 18, as well as different optimization settings, but none resulted in a meaningful model.

Original: but none resulted in a meaningful model.

Original: It is likely that the potential cause is the small capacity of ResNet-10.
Revised: It is likely that the potential cause is the small capacity of ResNet-10.

Original: For MMA, the implementation included in its public repository is very speciﬁc to the CIFARdataset, so we did not include it in the comparison.
Revised: For MMA, the implementation included in its public repository is very speciﬁc to the CIFARdataset, so we did not include it in the comparison.

Original: Overall [Table] we observe a similar performance on SVHN vs. on CIFAR-10.
Revised: Overall, we observe a similar performance on SVHN vs. on CIFAR-10.

Original: Compared to the result in Table 1, we observe a slight increase in standard accuracy and robust accuracy for both SOAR and TRADES.
Revised: Compared to the result in Table 1, we observe a slight increase in standard accuracy and robust accuracy for both SOAR and TRADES.

Original: In particular, the standard accuracy increases by 8 .
Revised: In particular, the standard accuracy increases by 8 .

Original: 87% and 3 .
Revised: 87% and 3 .

Original: 28% , and the PGD accuracy increases by 3 .
Revised: 28% , and the PGD accuracy increases by 3 .

Original: 52% and 2 .
Revised: 52% and 2 .

Original: 93% for TRADES and SOAR respectively.
Revised: 93% for TRADES and SOAR respectively.

Original: More notably, we observe on SVHN that SOAR regularized model gains robustness without signiﬁcantly sacriﬁcing its standard accuracy.
Revised: More notably, we observe on SVHN that SOAR regularized model gains robustness without signiﬁcantly sacriﬁcing its standard accuracy.

Original: Table 8 compares the performance of SOAR to TRADES on SimBa and on transferred (cid:96) ∞ attacks.
Revised: Table 12 compares the performance of SOAR to TRADES on SimBa and on transferred (cid:96) ∞ attacks.

Original: The evaluation setting for transferred attacks is identical to the one used for CIFAR-10, where we use an undefended independently trained ResNet-10 as the source model.
Revised: The evaluation setting for transferred attacks is identical to the one used for CIFAR-10, where we use an undefended independently trained ResNet-10 as the source model.

Original: Despite a smaller gap on the accuracy against transferred attacks, we see that SOAR regularized model yields a signiﬁcant higher accuracy against the stronger SimBA attacks.
Revised: Despite a smaller gap on the accuracy against transferred attacks, we see that SOAR regularized model yields a signiﬁcant higher accuracy against the stronger SimBA attacks.

Original: Note that we did not perform any extensive hyperparameter sweep on SVHN, and we simply took what worked on CIFAR-10.
Revised: Note that we did not perform any extensive hyperparameter sweep on SVHN, and we simply took what worked on CIFAR-10.

Original: We stress that the goal is to demonstrate the effectiveness of SOAR, and its performance relative to other baseline methods.
Revised: We stress that the goal is to demonstrate the effectiveness of SOAR, and its performance relative to other baseline methods.

Original: Next, we evaluate SOAR and TRADES under (cid:96) 2 bounded white-box and black-box attacks.
Revised: Next, we evaluate SOAR and TRADES under (cid:96) 2 bounded white-box and black-box attacks.

Original: All (cid:96)
Revised: All (cid:96)

Original: PGD adversaries are generated using the same method as the one in the evaluation for CIFAR-10.
Revised: PGD adversaries are generated using the same method as the one in the evaluation for CIFAR-10.

Original: Also, we do not include ADV due to the same result discussed above.
Revised: Also, we do not include ADV due to the same result discussed above.

Original: Our results show that training with SOAR signiﬁcantly improves the robustness against (cid:96) 2 PGD white-box attacks compared to
Revised: Our results show that training with SOAR signiﬁcantly improves the robustness against (cid:96) 2 PGD white-box attacks compared to

Original: TRADES.
Revised: TRADES.

Original: For transferred attacks, TRADES and SOAR performs similarly.
Revised: For transferred attacks, TRADES and SOAR performs similarly.

Original: [Table]

Original: E. C HALLENGES
Revised: E. C HALLENGES

Original: Batch Normalization : We observe that networks with BatchNorm layers do not beneﬁt from SOAR in adversarial robustness.
Revised: Batch Normalization : We observe that networks with BatchNorm layers do not beneﬁt from SOAR in adversarial robustness.

Original: Speciﬁcally, we performed an extensive hyper-parameter search for SOAR on networks with BatchNorm layers, and we were not able to achieve meaningful improvement  in adversarial robustness.
Revised: Speciﬁcally, we performed an extensive hyper-parameter search for SOAR on networks with BatchNorm layers, and we were not able to achieve meaningful improvement [Table] in adversarial robustness.

Original: A related work by Galloway et al.
Revised: A related work by Galloway et al.

Original: (2019) focuses on the connection between BatchNorm and adversarial robustness.
Revised: (2019) focuses on the connection between BatchNorm and adversarial robustness.

Original: In particular, their results show that on VGG-based architecture (Simonyan & Zisserman, 2014), there is a signiﬁcant gap in adversarial robustness between networks with and without BatchNorm layers under standard training.
Revised: In particular, their results show that on VGG-based architecture (Simonyan & Zisserman, 2014), there is a signiﬁcant gap in adversarial robustness between networks with and without BatchNorm layers under standard training.

Original: Needless to say, the interaction between SOAR and BatchNorm requires further investigations, and we consider this as an important future direction.
Revised: Needless to say, the interaction between SOAR and BatchNorm requires further investigations, and we consider this as an important future direction.

Original: As such, we use a small-capacity ResNet (ResNet-10) in our experiment, and modiﬁed it by removing its BatchNorm layers.
Revised: As such, we use a small-capacity ResNet (ResNet-10) in our experiment, and modiﬁed it by removing its BatchNorm layers.

Revised: Speciﬁcally, we removed BatchNorm layers from all models used in the baseline experiments with ResNet.

Original: Note that BatchNorm layers makes the training process less sensitive to hyperparameters (Ioffe & Szegedy, 2015), and removing them makes it difﬁcult to train a very deep network such as WideResNet.
Revised: Note that BatchNorm layers makes the training process less sensitive to hyperparameters (Ioffe & Szegedy, 2015), and removing them makes it difﬁcult to train a very deep network such as WideResNet.

Original: As such, we did not performSOAR on
Revised: As such, we did not perform

Original:  WideResNet.
Revised: SOAR on WideResNet.

Original: Starting from pretrained model : We notice that it is difﬁcult to train with SOAR on a newlyinitialized model.
Revised: Starting from pretrained model : We notice that it is difﬁcult to train with SOAR on a newlyinitialized model.

Original: Note that it is a common technique to perform ﬁne-tuning on a pretrained model for a speciﬁc task.
Revised: Note that it is a common technique to perform ﬁne-tuning on a pretrained model for a speciﬁc task.

Original: In CURE, regularization is performed after a model is ﬁrst trained with a cross-entropy loss to reach a high accuracy on clean data.
Revised: In CURE, regularization is performed after a model is ﬁrst trained with a cross-entropy loss to reach a high accuracy on clean data.

Original: They call the process adversarial ﬁne-tuning .
Revised: They call the process adversarial ﬁne-tuning .

Original: Cai et al.
Revised: Cai et al.

Original: ; Sitawarin et al.
Revised: ; Sitawarin et al.

Original: (2020) study the connection between curriculum learning (Bengio et al., 2009) and training using adversarial examples with increasing difﬁculties.
Revised: (2020) study the connection between curriculum learning (Bengio et al., 2009) and training using adversarial examples with increasing difﬁculties.

Original: Our idea is similar.
Revised: Our idea is similar.

Original: The model is ﬁrst optimized for an easier task (standard training), and then regularized for a related, but more difﬁcult task (improving adversarial robustness).
Revised: The model is ﬁrst optimized for an easier task (standard training), and then regularized for a related, but more difﬁcult task (improving adversarial robustness).

Original: Since the model has been trained to minimize its standard loss, the loss gradient can be very small compared to the regularizer gradient, and thus we apply a clipping of 10 on the regularizer.
Revised: Since the model has been trained to minimize its standard loss, the loss gradient can be very small compared to the regularizer gradient, and thus we apply a clipping of 10 on the regularizer.

Original: Catastrophic Over-ﬁtting : We observe that when the model achieves a high adversarial accuracy and continues training for a long period of time, both the standard and adversarial accuracy drop signiﬁcantly.
Revised: Catastrophic Overﬁtting : We observe that when the model achieves a high adversarial accuracy and continues training for a long period of time, both the standard and adversarial accuracy drop signiﬁcantly.

Original: A similar phenomenon was observed in (Cai et al., 2018;
Revised: A similar phenomenon was observed in (Cai et al., 2018;

Original: Wong et al., 2019a), which they refer to as catastrophic forgetting and catastrophic over-ﬁtting respectively.
Revised: Wong et al., 2019a), which they refer to as catastrophic forgetting and catastrophic over-ﬁtting respectively.

Original: Wong et al.
Revised: Wong et al.

Original: (2019a) use early-stopping as a simple solution.
Revised: (2019a) use early-stopping as a simple solution.

Original: We observe that with a large learning rate, the model reaches a high adversarial accuracy faster and catastrophic over-ﬁtting happens sooner.
Revised: We observe that with a large learning rate, the model reaches a high adversarial accuracy faster and catastrophic over-ﬁtting happens sooner.

Original: As such, our solution is to ﬁx the number of epochs to 200 and then carefully sweep over various learning rates to make sure that catastrophic over-ﬁtting do not happen.
Revised: As such, our solution is to ﬁx the number of epochs to 200 and then carefully sweep over various learning rates to make sure that catastrophic over-ﬁtting do not happen.

Original: Discussion on Computation Cost : We emphasize that our primary goal is to propose regularization as an alternative approach to improving adversarial robustness.
Revised: Discussion on Computation Complexity : We emphasize that our primary goal is to propose regularization as an alternative approach to improving adversarial robustness.

Original: We discussed techniques towards an efﬁcient implementation, however, there is still potential for a faster implementation.
Revised: We discussed techniques towards an efﬁcient implementation, however, there is still potential for a faster implementation.

Original: In our current implementation, a single epoch with WideResNet takes: 19 mins on PGD10 adversarial training, 26. mins on SOAR, 29 mins on MART, and 39.6 mins on TRADES.
Revised: In our current implementation, a single epoch with WideResNet takes: 19 mins on PGD10 adversarial training, 26.5 mins on SOAR, 29 mins on MART, and 39.6 mins on TRADES.

Original: We observe that despite being a faster method than MART and TRADES, SOAR is still quite slow compared to PGD10 adversarial training.
Revised: We observe that despite being a faster method than MART and TRADES, SOAR is still quite slow compared to PGD10 adversarial training.

Original: Finally, compared to no initialization or random initialization, the overhead for the proposed
Revised: We characterize the computation complexity as a function of the number of forward and backward passes required for a single mini-batch. Standard training: 1 forward pass and 1 backward pass; Adversarial training with k-step PGD: k+1 forward passes and k+1 backward passes; FOAR: 1 forward pass and 2 backward passes; SOAR: 3 forward passes and 4 backward passes E. D IFFERENTIABILITY OF R E LU AND ITS EFFECT ON SOAR

Original: PGD1 initialization requires one additional backward/forward pass.
Revised: The SOAR regularizer is derived based on the second-order Taylor approximation of the loss which requires the loss to be twice-differentiable. Although ReLU is not differentiable at 0, the probability of its input being at exactly 0 is very small. That is also why we can train ReLU networks through backpropagation.

Original:  E. P OTENTIAL ROBUSTNESS GAIN WITH INCREASING CAPACITIES
Revised: This is true for the Hessian too. In addition, notice that from a computation viewpoint, we never need to compute the exact Hessian as we approximate it through ﬁrst-order approximation. E. P OTENTIAL ROBUSTNESS GAIN WITH INCREASING CAPACITIES

Original: Empirical studies in Madry et al.
Revised: Empirical studies in Madry et al.

Original: (2018) and Wang et al.
Revised: (2018) and Wang et al.

Original: (2020) reveal that their approaches beneﬁt from increasing model capacity to achieve higher adversarial robustness.
Revised: (2020) reveal that their approaches beneﬁt from increasing model capacity to achieve higher adversarial robustness.

Original: We have a similar observation with SOAR.
Revised: We have a similar observation with SOAR.

Original: [Table]
Revised: [Table]

Original: Table 10 compares the performance of SOAR against (cid:96) ∞ bounded white-box attacks on networks with different capacities.
Revised: Table 14 compares the performance of SOAR against (cid:96) ∞ bounded white-box attacks on networks with different capacities.

Original: CNN6(CNN8) refers to a simple 6-layer(8-layer) convolutional network, and ResNet-10 is the network we use in Section 5.
Revised: CNN6(CNN8) refers to a simple 6-layer(8-layer) convolutional network, and ResNet-10 is the network we use in Section 5.

Original: Evidently, as network capacity increases, we observe improvements in both standard accuracy and adversarial accuracy.
Revised: Evidently, as network capacity increases, we observe improvements in both standard accuracy and adversarial accuracy.

Original: As such, we expect a similar gain in performance with larger capacity networks such as WideResNet.
Revised: As such, we expect a similar gain in performance with larger capacity networks such as WideResNet.

Original: E. E XPERIMENT RESULTS ON R ES N ET 10 IN T ABLE 1 AND T ABLE 2 WITH STANDARD DEVIAITIONS
Revised: E. E XPERIMENT RESULTS ON R ES N ET 10 IN T ABLE 1 AND T ABLE 2 WITH STANDARD DEVIAITIONS

Original: All results on ResNet10 are obtained by averaging over 3 independently initialized and trained models.
Revised: All results on ResNet10 are obtained by averaging over 3 independently initialized and trained models.

Original: Here, we report the standard deviation of the results in Table 1 and Table 2.
Revised: Here [Table] we report the standard deviation of the results in Table 1 and Table 2.

Original: Notice we omit results on PGD100 and PGD200 due to space constraint.
Revised: Notice we omit results on PGD100 and PGD200 due to space constraint.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: E. A DDITIONAL E XPERIMENTS ON G RADIENT M ASKING
Revised: E. A DDITIONAL E XPERIMENTS ON G RADIENT M ASKING

Original: To verify that SOAR improves robustness of the model without gradient masking, we include the following experiments to empirically support our claim.
Revised: To verify that SOAR improves robustness of the model without gradient masking, we include the following experiments to empirically support our claim.

Original: First, from the result in Appendix E.3, we conclude that SOAR with zero initilaization results in gradient masking.
Revised: First, from the result in Appendix E.3, we conclude that SOAR with zero initilaization results in gradient masking.

Original: This is shown by the high accuracy ( 89 .
Revised: This is shown by the high accuracy ( 89 .

Original: 24% , close to standard accuracy) under white-box PGD attacks and low accuracy ( 2 . 86% ) under black-box transferred attacks.
Revised: 24% , close to standard accuracy) under white-box PGD attacks and low accuracy ( 2 . 86% ) under black-box transferred attacks.

Original: Next, prior work has veriﬁed that adversarial training with PGD20 adversaries (ADV) results model without gradient maskingAthalye et al.
Revised: Next, prior work has veriﬁed that adversarial training with PGD20 adversaries (ADV) results model without gradient maskingAthalye et al.

Original: Therefore, let us use models trained using ADV and
Revised: Therefore, let us use models trained using ADV and

Original: SOAR(zero-init) as examples of models with/without gradient masking respectively.
Revised: SOAR(zero-init) as examples of models with/without gradient masking respectively.

Original: In the (cid:96) ∞ attack setting, PGD uses the sign of the gradient sign( ∇ x (cid:96) ( x )) to generate perturbations.
Revised: In the (cid:96) ∞ attack setting, PGD uses the sign of the gradient sign( ∇ x (cid:96) ( x )) to generate perturbations.

Original: As such, one way to verify the strength of gradient is to measure the average number of none-zero elements in the gradient.
Revised: As such, one way to verify the strength of gradient is to measure the average number of none-zero elements in the gradient.

Original: A model with gradient masking is expected to have much less non-zero elements than one without.
Revised: A model with gradient masking is expected to have much less non-zero elements than one without.

Original: In our experiment, the average non-zero element in gradient is 3072 for
Revised: In our experiment, the average non-zero element in gradient is 3072 for

Original: ADV trained (no GM), 3069 for SOAR (PGD1-init) and 1043 for SOAR (zero-init, has GM).
Revised: ADV trained (no GM), 3069 for SOAR (PGD1-init) and 1043 for SOAR (zero-init, has GM).

Original: We observe that SOAR with PGD1-init has a similar number of non-zero gradient elements compared to ADV, meaning PGD adversary can use sign of those non-zero gradient elements to generate meaningful perturbations.
Revised: We observe that SOAR with PGD1-init has a similar number of non-zero gradient elements compared to ADV, meaning PGD adversary can use sign of those non-zero gradient elements to generate meaningful perturbations.

Original: In Section 5, the 20-iteration (cid:96) ∞
Revised: In Section 5, the 20-iteration (cid:96) ∞

Original: PGD adversaries are generated with a step-size of 2255 and ε = 8255 .
Revised: PGD adversaries are generated with a step-size of 2255 and ε = 8255 .

Original: Suppose we use ε = 1 instead of ε = 8255 and other parameters remain the same, that is, we allow the maximum (cid:96) ∞ perturbation to reach the input range ( [0 , 1] ) and generate PGD20 attacks.
Revised: Suppose we use ε = 1 instead of ε = 8255 and other parameters remain the same, that is, we allow the maximum (cid:96) ∞ perturbation to reach the input range ( [0 , 1] ) and generate PGD20 attacks.

Original: We observe such attacks result in near black-and-white images on SOAR with PGD-1 init; it has a 0% accuracy against such PGD20 attacks, similar to the 3 . 3% on ADV trained model.
Revised: We observe such attacks result in near black-and-white images on SOAR with PGD-1 init; it has a 0% accuracy against such PGD20 attacks, similar to the 3 . 3% on ADV trained model.

Original: On the other hand, the robust accuracy for SOAR (zero-init) is 9 .
Revised: On the other hand, the robust accuracy for SOAR (zero-init) is 9 .

Original: E. H YPERPARAMETER SWEEP FOR TRADES, MART AND MMA ON R ES N ET
Revised: E. H YPERPARAMETER SWEEP FOR TRADES, MART AND MMA ON R ES N ET

Original: The following results show the hyperparameter sweep on TRADES, MART and MMA respectively.
Revised: The following results show the hyperparameter sweep on TRADES, MART and MMA respectively.

Original: We include the one with the highest PGD20 accuracy in Section 5.
Revised: We include the one with the highest PGD20 accuracy in Section 5.

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: A DVERSARIAL R OBUSTNESS OF THE M ODEL T RAINED USING FOAR WITH D IFFERENT I NITIALIZATIONS
Revised: A DVERSARIAL R OBUSTNESS OF THE M ODEL T RAINED USING FOAR WITH D IFFERENT I NITIALIZATIONS

Original: FOAR achieves the best adversarial robustness using PGD1 initialization, so we only present this variation of FOAR in Section 5.
Revised: FOAR achieves the best adversarial robustness using PGD1 initialization, so we only present this variation of FOAR in Section 5.

Original: [Table]
Revised: [Table]