Original: W HY ADVERSARIAL TRAINING CAN HURT ROBUST ACCURACY
Revised: W HY ADVERSARIAL TRAINING CAN HURT ROBUST ACCURACY

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Machine learning classiﬁers with high test accuracy often perform poorly under adversarial perturbations.
Revised: Machine learning classiﬁers with high test accuracy often perform poorly under adversarial perturbations.

Original: It is commonly believed that adversarial training alleviates this issue.
Revised: It is commonly believed that adversarial training alleviates this issue.

Original: In this paper, we demonstrate that, surprisingly, the opposite can be true for a natural class of perceptible perturbations — even though adversarial training helps when enough data is available, it may in fact hurt robust generalization in the small sample size regime.
Revised: In this paper, we demonstrate that, surprisingly, the opposite can be true for a natural class of perceptible perturbations — even though adversarial training helps when enough data is available, it may in fact hurt robust generalization in the small sample size regime.

Original: We ﬁrst prove this phenomenon for a high-dimensional linear classiﬁcation setting with noiseless observations.
Revised: We ﬁrst prove this phenomenon for a high-dimensional linear classiﬁcation setting with noiseless observations.

Original: Using intuitive insights from the proof, we could ﬁnd perturbations on standard image datasets for which this behavior persists.
Revised: Using intuitive insights from the proof, we could ﬁnd perturbations on standard image datasets for which this behavior persists.

Original: Speciﬁcally, it occurs for perceptible perturbations that effectively reduce class information such as object occlusions or corruptions
Revised: Speciﬁcally, it occurs for perceptible perturbations that effectively reduce class information such as object occlusions or corruptions.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Today’s best-performing classiﬁers are vulnerable to adversarial attacks Goodfellow et al.
Revised: Today’s best-performing classiﬁers are vulnerable to adversarial attacks Goodfellow et al.

Original: ; Szegedy et al.
Revised: ; Szegedy et al.

Original: and exhibit high robust error : for many inputs, their predictions change under adversarial perturbations, even though the true class stays the same.
Revised: and exhibit high robust error : for many inputs, their predictions change under adversarial perturbations, even though the true class stays the same.

Original: Such contentpreserving (Gilmer et al., 2018), consistent (Raghunathan et al., 2020)
Revised: Such contentpreserving (Gilmer et al., 2018), consistent (Raghunathan et al., 2020)

Original: attacks can be either perceptible or imperceptible.
Revised: attacks can be either perceptible or imperceptible.

Original: For image datasets, most work to date studies imperceptible attacks that are based on perturbations with limited strength or attack budget .
Revised: For image datasets, most work to date studies imperceptible attacks that are based on perturbations with limited strength or attack budget .

Original: These include bounded ` p -norm perturbations (Goodfellow et al., 2015; Madry et al., 2018; Moosavi-Dezfooli et al., 2016), small transformations using image processing techniques (Ghiasi et al., 2019; Zhao et al., 2020;
Revised: These include bounded ` p -norm perturbations (Goodfellow et al., 2015; Madry et al., 2018; Moosavi-Dezfooli et al., 2016), small transformations using image processing techniques (Ghiasi et al., 2019; Zhao et al., 2020;

Original: Laidlaw et al., 2021;
Revised: Laidlaw et al., 2021;

Original: Luo et al., 2018) or nearby samples on the data manifold (Lin et al., 2020; Zhou et al., 2020).
Revised: Luo et al., 2018) or nearby samples on the data manifold (Lin et al., 2020; Zhou et al., 2020).

Original: Even though they do not visibly change the image by deﬁnition, imperceptible attacks can often successfully fool a learned classiﬁer.
Revised: Even though they do not visibly change the image by deﬁnition, imperceptible attacks can often successfully fool a learned classiﬁer.

Original: On the other hand, perturbations that naturally occur and are physically realizable are commonly perceptible.
Revised: On the other hand, perturbations that naturally occur and are physically realizable are commonly perceptible.

Original: Some perceptible perturbations speciﬁcally target the object to be recognized: these include occlusions (e.g. stickers placed on trafﬁc signs (Eykholt et al., 2018) or masks of different sizes that cover important features of human faces (Wu et al., 2020)) or corruptions that are caused by the image capturing process (animals that move faster than the shutter speed or objects that are not well-lit, see Figure 2).
Revised: Some perceptible perturbations speciﬁcally target the object to be recognized: these include occlusions (e.g. stickers placed on trafﬁc signs (Eykholt et al., 2018) or masks of different sizes that cover important features of human faces (Wu et al., 2020)) or corruptions that are caused by the image capturing process (animals that move faster than the shutter speed or objects that are not well-lit, see Figure 2).

Original: Others transform the whole image and are not conﬁned to the object itself, such as rotations, translations or corruptions Engstrom et al.
Revised: Others transform the whole image and are not conﬁned to the object itself, such as rotations, translations or corruptions Engstrom et al.

Original: ; Kang et al.
Revised: ; Kang et al.

Original: In this paper, we refer to such perceptible attacks as directed attacks .
Revised: In this paper, we refer to such perceptible attacks as directed attacks .

Original: In contrast to other attacks, they effectively reduce useful class information in the input for any model, without necessarily changing the true label — we say that they are directed and consistent , more formally deﬁned in Section 2.
Revised: In contrast to other attacks, they effectively reduce useful class information in the input for any model, without necessarily changing the true label — we say that they are directed and consistent , more formally deﬁned in Section 2.

Original: For example, a stop sign with a small sticker could partially cover the text without losing its semantic meaning.
Revised: For example, a stop sign with a small sticker could partially cover the text without losing its semantic meaning.

Original:  Similarly, a ﬂying bird captured with a long exposure time can induce motion blur in the ﬁnal image without becoming unrecognizable to the observer.[Figure]
Revised: [Figure] Similarly, a ﬂying bird captured with a long exposure time can induce motion blur in the ﬁnal image without becoming unrecognizable to the observer.

Original: In the literature so far, it is widely acknowledged that adversarial training with the same perturbation type and budget as during test time often achieves signiﬁcantly lower robust error than standard training (Madry et al., 2018; Zhang et al., 2019; Bai et al., 2021).
Revised: In the literature so far, it is widely acknowledged that adversarial training with the same perturbation type and budget as during test time often achieves signiﬁcantly lower robust error than standard training (Madry et al., 2018; Zhang et al., 2019; Bai et al., 2021).

Original: In contrast, we show that adversarial training not only increases standard error (Zhang et al., 2019;
Revised: In contrast, we show that adversarial training not only increases standard error (Zhang et al., 2019;

Original: Tsipras et al., 2019; Stutz et al., 2019; Raghunathan et al., 2020), but surprisingly, in the low sample regime, adversarial training may even increase the robust error compared to standard training!
Revised: Tsipras et al., 2019; Stutz et al., 2019; Raghunathan et al ), but surprisingly, in the low sample regime, adversarial training may even increase the robust error compared to standard training!

Original: Figure 1 illustrates the main message of our paper on the Waterbirds dataset: Although adversarial training with directed attacks outperforms standard training when enough training samples are available, it is inferior when the sample size is small (but still large enough to obtain a small standard test error).
Revised: Figure 1 illustrates the main message of our paper on the Waterbirds dataset: Although adversarial training with directed attacks outperforms standard training when enough training samples are available, it is inferior when the sample size is small (but still large enough to obtain a small standard test error).

Original: Our contributions are as follows:
Revised: Our contributions are as follows:

Original: • We prove that, almost surely, adversarially training a linear classiﬁer on separable data yields a monotonically increasing robust error as the perturbation budget grows.
Revised: • We prove that, almost surely, adversarially training a linear classiﬁer on separable data yields a monotonically increasing robust error as the perturbation budget grows.

Original: We further establish high-probability non-asymptotic lower bounds on the robust error gap between adversarial and standard training.
Revised: We further establish high-probability non-asymptotic lower bounds on the robust error gap between adversarial and standard training.

Original: • Our proof provides intuition for why this lower bound on the gap is particularly large for directed attacks in the low sample regime.
Revised: • Our proof provides intuition for why this lower bound on the gap is particularly large for directed attacks in the low sample regime.

Original: • We observe empirically for different directed attacks on real-world image datasets that this behavior persists: adversarial training for directed attacks hurts robust accuracy when the sample size is small.
Revised: • We observe empirically for different directed attacks on real-world image datasets that this behavior persists: adversarial training for directed attacks hurts robust accuracy when the sample size is small

Original:  R OBUST CLASSIFICATION
Revised:  R OBUST CLASSIFICATION

Original: We ﬁrst introduce our robust classiﬁcation setting more formally by deﬁning the notions of adversarial robustness, directed attacks and adversarial training used throughout the paper.
Revised: We ﬁrst introduce our robust classiﬁcation setting more formally by deﬁning the notions of adversarial robustness, directed attacks and adversarial training used throughout the paper.

Original: Adversarially robust classiﬁers For inputs x 2 R d , we consider multi-class classiﬁers associated with parameterized functions f ✓ : R d !
Revised: Adversarially robust classiﬁers For inputs x 2 R d , we consider multi-class classiﬁers associated with parameterized functions f ✓ : R d !

Original: R K
Revised: R K

Original: if K > 2 and f ✓ : R d !
Revised: if K > 2 and f ✓ : R d !

Original: R if K = 2 , where K is the number of labels.
Revised: R if K = 2 , where K is the number of labels.

Original: For example, f ✓ ( x ) could be a linear model (as in Section 3) or a neural network (as in Section 4).
Revised: For example, f ✓ ( x ) could be a linear model (as in Section 3) or a neural network (as in Section 4).

Original: The output label predictions are obtained by h ( f ✓ ( x ))
Revised: The output label predictions are obtained by h ( f ✓ ( x ))

Original: =
Revised: =

Original: sign ( f ✓ ( x )) for K =and h ( f ✓ ( x ))
Revised: sign ( f ✓ ( x )) for K =and h ( f ✓ ( x ))

Original: =
Revised: =

Original: arg max k 2 { 1 ,..,K } f ✓ ( x ) k for K > 2 .
Revised: arg max k 2 { 1 ,..,K } f ✓ ( x ) k for K > 2 .

Original: In order to convince practitioners to use machine learning models in the wild, it is key to demonstrate that they exhibit robustness.
Revised: In order to convince practitioners to use machine learning models in the wild, it is key to demonstrate that they exhibit robustness.

Original: One kind of robustness is that they do not change prediction when the input is subject to consistent perturbations, which are small class-preserving perturbations.
Revised: One kind of robustness is that they do not change prediction when the input is subject to consistent perturbations, which are small class-preserving perturbations.

Original: Mathematically speaking, for the underlying joint data distribution P , the model should have a small ✏ te -robust error , deﬁned as
Revised: Mathematically speaking, for the underlying joint data distribution P , the model should have a small ✏ te -robust error , deﬁned as

Original: [Equation]
Revised: [Equation]

Original: where ` is 0 if the class determined by h ( f ✓ ( x )) is equal to y and 1 otherwise.
Revised: where ` is 0 if the class determined by h ( f ✓ ( x )) is equal to y and 1 otherwise.

Original: Further, T ( x ; ✏ te ) indicates a perturbation set around x of a certain transformation type with size ✏ test .
Revised: Further, T ( x ; ✏ te ) indicates a perturbation set around x of a certain transformation type with size ✏ test .

Original: Note that[Equation] the (standard) error E ( x,y )
Revised: Note that the (standard) error E ( x,y )

Original: ⇠ P ` ( f ✓ ( x ) , y ) of a classiﬁer corresponds to Err ( ✓ ; 0) – the robust error evaluated at ✏ te = 0 .
Revised: ⇠ P ` ( f ✓ ( x ) , y ) of a classiﬁer corresponds to Err ( ✓ ; 0) – the robust error evaluated at ✏ te = 0 .

Original: Directed attacks The inner maximization in Equation 1 is often called the adversarial attack of the input x for the model f ✓ and the corresponding solution is referred to as the adversarial example .
Revised: Directed attacks The inner maximization in Equation 1 is often called the adversarial attack of the input x for the model f ✓ and the corresponding solution is referred to as the adversarial example .

Original: In this paper, we consider directed attacks that effectively reduce the information about the true classes, with image-based examples depicted in Figure 2.
Revised: In this paper, we consider directed attacks that effectively reduce the information about the true classes, with image-based examples depicted in Figure 2.

Original: For linear classiﬁcation, we analyze directed attacks in the form of additive perturbations that are constrained to the direction of the optimal decision boundary (see details in Section 3.1).
Revised: For linear classiﬁcation, we analyze directed attacks in the form of additive perturbations that are constrained to the direction of the optimal decision boundary (see details in Section 3.1).

Original: In particular, note that the set of directed perturbations is restricted to directions attacking the Bayes optimal classiﬁer.
Revised: In particular, note that the set of directed perturbations is restricted to directions attacking the Bayes optimal classiﬁer.

Original: Adversarial training A common approach to obtain classiﬁers with a good robust accuracy is to minimize the training objective L ✏ tr with a surrogate robust classiﬁcation loss
Revised: training A common approach to obtain classiﬁers with a good robust accuracy is to minimize the training objective L ✏ tr with a surrogate robust classiﬁcation loss

Original: L
Revised: L

Original: [Equation]
Revised: [Equation]

Original: also called adversarial training .
Revised: also called adversarial training .

Original: In practice, we often use the cross entropy loss L ( z )
Revised: In practice, we often use the cross entropy loss L ( z )

Original: = log(

Revised: = log(

Original: + e  z ) and minimize the robust objective by using ﬁrst order optimization methods such as (stochastic) gradient descent.
Revised: + e  z ) and minimize the robust objective by using ﬁrst order optimization methods such as (stochastic) gradient descent.

Original: SGD is also the algorithm that we focus on in both the theoretical and experimental sections.
Revised: SGD is also the algorithm that we focus on in both the theoretical and experimental sections.

Original: When the desired type of robustness is known in advance, it is standard practice to use the same perturbation set for training as for testing, i.e. T ( x ; ✏ tr ) =
Revised: When the desired type of robustness is known in advance, it is standard practice to use the same perturbation set for training as for testing, i.e. T ( x ; ✏ tr ) =

Original: T ( x ; ✏ te ) .
Revised: T ( x ; ✏ te ) .

Original: For example, Madry et al.
Revised: For example, Madry et al.

Original: (2018) show that the robust error sharply increases for ✏ tr < ✏ te .
Revised: (2018) show that the robust error sharply increases for ✏ tr < ✏ te .

Original: In this paper, we demonstrate that for directed attacks in the small sample size regime, in fact, the opposite is true.
Revised: In this paper, we demonstrate that for directed attacks in the small sample size regime, in fact, the opposite is true.

Original:  T HEORETICAL RESULTS
Revised:  T HEORETICAL RESULTS

Original: In this section, we prove for linear functions f ✓ ( x ) =
Revised: In this section, we prove for linear functions f ✓ ( x ) =

Original: ✓ > x that in the case of directed attacks, robust generalization deteriorates with increasing ✏ tr .
Revised: ✓ > x that in the case of directed attacks, robust generalization deteriorates with increasing ✏ tr .

Original: The proof, albeit in a simple setting, provides explanations for why adversarial training fails in the high-dimensional regime for such attacks.
Revised: The proof, albeit in a simple setting, provides explanations for why adversarial training fails in the high-dimensional regime for such attacks.

Original: 3. S ETTING
Revised: 3. S ETTING

Original: We now introduce the precise linear setting used in our theoretical results.
Revised: We now introduce the precise linear setting used in our theoretical results.

Original: Data model We assume that the ground truth and hypothesis class are given by linear functions f ✓ ( x ) = ✓ > x and the sample size n is lower than the ambient dimension d minus one.
Revised: Data model We assume that the ground truth and hypothesis class are given by linear functions f ✓ ( x ) = ✓ > x and the sample size n is lower than the ambient dimension d minus one.

Original: The generative distribution P r is similar to (Tsipras et al., 2019; Nagarajan & Kolter, 2019):
Revised: The generative distribution P r is similar to (Tsipras et al., 2019; Nagarajan & Kolter, 2019):

Original: The label y 2 { +1 ,  1 } is drawn with equal probability and the covariate vector is sampled as x =
Revised: The label y 2 { +1 ,  1 } is drawn with equal probability and the covariate vector is sampled as x =

Original: [ y r 2 , ˜ x ] with the random vector ˜ x 2 R d  1 drawn from a standard normal distribution, i.e. ˜ x ⇠ N (0 ,  2 I d  1 ) .
Revised: [ y r 2 , ˜ x ] with the random vector ˜ x 2 R d  1 drawn from a standard normal distribution, i.e. ˜ x ⇠ N (0 ,  2 I d  1 ) .

Original: We would like to learn a classiﬁer that has low robust error by using a dataset D =
Revised: We would like to learn a classiﬁer that has low robust error by using a dataset D =

Original: ( x i , y i ) ni
Revised: ( x i , y i ) ni

Original: =1 with n i.i.d.
Revised: =1 with n i.i.d.

Original: samples from P r .
Revised: samples from P r .

Original: Intuitively, the separation distance r reﬂects the signal strength of the data distribution.
Revised: Intuitively, the separation distance r reﬂects the signal strength of the data distribution.

Original: Notice that the distribution P r is noiseless: for a given input x , the label y = sign( x [1] ) is deterministic.
Revised: Notice that the distribution P r is noiseless: for a given input x , the label y = sign( x [1] ) is deterministic.

Original: Further, the Bayes optimal linear classiﬁer (also referred to as the ground truth ) is parameterized by the ﬁrst standard coordinate vector, ✓ ?
Revised: Further, the Bayes optimal linear classiﬁer (also referred to as the ground truth ) is parameterized by the ﬁrst standard coordinate vector, ✓ ?

Original: =
Revised: =

Original: e 1 .
Revised: e 1 .

Original: By deﬁnition, the ground truth is robust against all perturbations that do not change the sign in the ﬁrst coordinate of the sample, i.e. consistent perturbations, and hence so is the optimal robust classiﬁer.
Revised: By deﬁnition, the ground truth is robust against all perturbations that do not change the sign in the ﬁrst coordinate of the sample, i.e. consistent perturbations, and hence so is the optimal robust classiﬁer.

Original: Directed attacks In this paper, we focus on consistent directed attacks that by deﬁnition efﬁciently concentrate their attack budget to reduce the class information.
Revised: Directed attacks In this paper, we focus on consistent directed attacks that by deﬁnition efﬁciently concentrate their attack budget to reduce the class information.

Original: For our linear setting this information lies in the ﬁrst entry.
Revised: For our linear setting this information lies in the ﬁrst entry.

Original: Hence, we can model such attacks by additive perturbations in the ﬁrst dimension
Revised: Hence, we can model such attacks by additive perturbations in the ﬁrst dimension

Original: [Equation]
Revised: [Equation]

Revised: [Figure]

Original: Note that this attack is always in the direction of the signal dimension, i.e. the Bayes optimal classiﬁer or equivalently the ground truth.
Revised: Note that this attack is always in the direction of the signal dimension, i.e. the Bayes optimal classiﬁer or equivalently the ground truth.

Original: Furthermore, when ✏ < r 2 , it is a consistent directed attack.
Revised: Furthermore, when ✏ < r 2 , it is a consistent directed attack.

Original: Observe how this is different from ` p -attacks — an ` p attack, depending on the model, may add a perturbation that only has a very small component in the signal direction.[Figure]
Revised: Observe how this is different from ` p -attacks — an ` p attack, depending on the model, may add a perturbation that only has a very small component in the signal direction.

Original: Robust max- ` 2 -margin classiﬁer We study a classiﬁer that is the solution of running gradient descent on the adversarial logistic loss.
Revised: Robust max- ` 2 -margin classiﬁer We study a classiﬁer that is the solution of running gradient descent on the adversarial logistic loss.

Original: A long line of work (Soudry et al., 2018; Ji & Telgarsky, 2019; Chizat & Bach, 2020; Nacson et al., 2019; Liu et al., 2020) studies the implicit bias of (S)GD on the (standard) logistic loss and separable data.
Revised: A long line of work (Soudry et al., 2018; Ji & Telgarsky, 2019; Chizat & Bach, 2020; Nacson et al., 2019; Liu et al., 2020) studies the implicit bias of (S)GD on the (standard) logistic loss and separable data.

Original: In particular, they show directional convergence to the max-margin solution.
Revised: In particular, they show directional convergence to the max-margin solution.

Original: For the adversarial logistic loss and linear models in particular, (S)GD converges to the robust max- ` 2 -margin solution (Li et al., 2020),
Revised: For the adversarial logistic loss and linear models in particular, (S)GD converges to the robust max- ` 2 -margin solution (Li et al., 2020),

Original: [Equation]
Revised: [Equation]

Original: Even though our result is proven for the max- ` 2 -margin classiﬁer, it can easily be extended to other interpolators.
Revised: Even though our result is proven for the max- ` 2 -margin classiﬁer, it can easily be extended to other interpolators.

Original: 3. M AIN RESULTS
Revised: 3. M AIN RESULTS

Original: We are now ready to characterize the ✏ te -robust error as a function of ✏ tr , the separation r , the dimension d and sample size n of the data.
Revised: We are now ready to characterize the ✏ te -robust error as a function of ✏ tr , the separation r , the dimension d and sample size n of the data.

Original: In the theorem statement we use the following quantities
Revised: In the theorem statement we use the following quantities

Original: [Equation]
Revised: [Equation]

Original: that arise from concentration bounds for the singular values of the random data matrix.
Revised: that arise from concentration bounds for the singular values of the random data matrix.

Original: Further, let ˜ ✏ := r 2  ' max p 2 and denote by  the cumulative distribution function of a standard normal.
Revised: Further, let ˜ ✏ := r 2  ' max p 2 and denote by  the cumulative distribution function of a standard normal.

Original: Theorem 3.1.
Revised: Theorem 3.1.

Original: Assume d  1 > n .
Revised: Assume d  1 > n .

Original: For test samples from P r , perturbation set type T as in Equationand any 0  ✏ te < r 2 , the following holds for the ✏ te -robust error of the classiﬁer (Equation 1) resulting from ✏ tr -adversarial training:
Revised: For test samples from P r , perturbation set type T as in Equationand any 0  ✏ te < r 2 , the following holds for the ✏ te -robust error of the classiﬁer (Equation 1) resulting from ✏ tr -adversarial training:

Original: The ✏ te -robust error of the ✏ tr -robust max-margin estimator reads
Revised: The ✏ te -robust error of the ✏ tr -robust max-margin estimator reads

Original: [Equation]
Revised: [Equation]

Original: for a random quantity ˜ ' > 0 depending on  , r, ✏ te and is hence strictly increasing in the adversarial training budget ✏ tr . 2.
Revised: for a random quantity ˜ ' > 0 depending on  , r, ✏ te and is hence strictly increasing in the adversarial training budget ✏ tr . 2.

Original: With probability at least 1   , we further have ' min  ˜ '  ' max and the following lower bound on the robust error increase by adversarially training with size ✏ tr
Revised: With probability at least 1   , we further have ' min  ˜ '  ' max and the following lower bound on the robust error increase by adversarially training with size ✏ tr

Original: [Equation]
Revised: [Equation]

Revised: [Figure]

Original: The proof can be found in Appendix A and primarily relies on estimation of singular values of high-dimensional matrices.
Revised: The proof can be found in Appendix A and primarily relies on estimation of singular values of high-dimensional matrices.

Original: Note that the theorem holds for any 0  ✏ te < r 2 and hence also directly[Figure] applies to the standard error by setting ✏ te = 0 .
Revised: Note that the theorem holds for any 0  ✏ te < r 2 and hence also directly applies to the standard error by setting ✏ te = 0 .

Original: In Figure 3, we empirically conﬁrm the statements of
Revised: In Figure 3, we empirically conﬁrm the statements of

Original: Theorem 3.1 by performing multiple experiments on synthetic datasets as described in Subsection 3.1 with different choices of d/n and ✏ tr .
Revised: Theorem 3.1 by performing multiple experiments on synthetic datasets as described in Subsection 3.1 with different choices of d/n and ✏ tr .

Original: In the ﬁrst statement, we prove that for small sample-size ( n < d  1 ) noiseless data, almost surely, the robust error increases monotonically with adversarial training budget ✏ tr > 0 .
Revised: In the ﬁrst statement, we prove that for small sample-size ( n < d  1 ) noiseless data, almost surely, the robust error increases monotonically with adversarial training budget ✏ tr > 0 .

Original: In Figure 3a, we plot the robust error gap between standard and adversarial logistic regression as a function of the adversarial training budget ✏ tr for 5 runs.
Revised: In Figure 3a, we plot the robust error gap between standard and adversarial logistic regression as a function of the adversarial training budget ✏ tr for 5 runs.

Original: The second statement establishes a simpliﬁed lower bound on the robust error increase for adversarial training (for a ﬁxed ✏ tr = ✏ te ) compared to standard training.
Revised: The second statement establishes a simpliﬁed lower bound on the robust error increase for adversarial training (for a ﬁxed ✏ tr = ✏ te ) compared to standard training.

Original: In Figures 3a and 3c, we show how the lower bound closely predicts the robust error gap in our synthetic experiments.
Revised: In Figures 3a and 3c, we show how the lower bound closely predicts the robust error gap in our synthetic experiments.

Original: Furthermore, by the dependence of ' min on the overparameterization ratio d/n , the lower bound on the robust error gap is ampliﬁed for large d/n .
Revised: Furthermore, by the dependence of ' min on the overparameterization ratio d/n , the lower bound on the robust error gap is ampliﬁed for large d/n .

Original: Indeed, Figure 3c shows how the error gap increases with d/n both theoretically and experimentally.
Revised: Indeed, Figure 3c shows how the error gap increases with d/n both theoretically and experimentally.

Original: However, when d/n increases above a certain threshold, the gap decreases again, as standard training fails to learn the signal and yields a high error (see Figure 3b).
Revised: However, when d/n increases above a certain threshold, the gap decreases again, as standard training fails to learn the signal and yields a high error (see Figure 3b).

Original: 3. P ROOF INTUITION
Revised: 3. P ROOF INTUITION

Original: The reason that adversarial training hurts robust generalization is based on an extreme robust vs. standard error trade-off.
Revised: The reason that adversarial training hurts robust generalization is based on an extreme robust vs. standard error trade-off.

Original: We now provide intuition for the effect of directed attacks and the low sample regime on the ✏ tr -robust max- ` 2 -margin solution by decomposing the robust error Err ( ✓ ; ✏ te ) .
Revised: We now provide intuition for the effect of directed attacks and the low sample regime on the ✏ tr -robust max- ` 2 -margin solution by decomposing the robust error Err ( ✓ ; ✏ te ) .

Original: Notice that ✏ te -robust error Err ( ✓ ; ✏ te ) can be written as the probability of the union of two events: the event that the classiﬁer based on ✓ is wrong and the event that the classiﬁer is susceptible to attacks:Err ( ✓ ; ✏ te ) [Equation] x,y ⇠ P
Revised: Notice that ✏ te -robust error Err ( ✓ ; ✏ te ) can be written as the probability of the union of two events: the event that the classiﬁer based on ✓ is wrong and the event that the classiﬁer is susceptible to attacks:

Original: [Equation]
Revised: [Equation]

Original: where Susc ( ✓ ; ✏ te ) is the expectation of the maximization term in Equation 7.
Revised: where Susc ( ✓ ; ✏ te ) is the expectation of the maximization term in Equation 7.

Original: Susc ( ✓ ; ✏ te ) represents the ✏ te - attack-susceptibility of a classiﬁer induced by ✓ and Err ( ✓ ; 0) its standard error.
Revised: Susc ( ✓ ; ✏ te ) represents the ✏ te - attack-susceptibility of a classiﬁer induced by ✓ and Err ( ✓ ; 0) its standard error.

Original: In our linear setting, we can lower bound Equation 7 by Err ( ✓ ; 0) + 12 Susc ( ✓ ; ✏ te ) .
Revised: In our linear setting, we can lower bound Equation 7 by Err ( ✓ ; 0) + 12 Susc ( ✓ ; ✏ te ) .

Original: Hence, Equation 7 suggests that the robust error can only be small if both the standard error and susceptibility are small.
Revised: Hence, Equation 7 suggests that the robust error can only be small if both the standard error and susceptibility are small.

Original: In Figure 4b, we plot the decomposition of the robust error in standard error and susceptibility for adversarial logistic regression with increasing ✏ tr .
Revised: In Figure 4b, we plot the decomposition of the robust error in standard error and susceptibility for adversarial logistic regression with increasing ✏ tr .

Original: We observe that increasing ✏ tr increases the standard error too drastically compared to the decrease in susceptibility, leading to a drop in robust accuracy.
Revised: We observe that increasing ✏ tr increases the standard error too drastically compared to the decrease in susceptibility, leading to a drop in robust accuracy.

Original: For completeness, in Appendix B, we provide upper and lower bounds for the susceptibility score.
Revised: For completeness, in Appendix B, we provide upper and lower bounds for the susceptibility score.

Original: We now explain why, in the small-sample size regime, adversarial training with directed attacks 3 may increase standard error to the extent that it dominates the decrease in susceptibility.
Revised: We now explain why, in the small-sample size regime, adversarial training with directed attacks 3 may increase standard error to the extent that it dominates the decrease in susceptibility.

Original: A key observation is that the robust max- ` 2 -margin solution of a dataset D
Revised: A key observation is that the robust max- ` 2 -margin solution of a dataset D

Original: = { ( x i , y i ) }
Revised: = { ( x i , y i ) }

Original: ni =1 maximizes the minimum margin that reads min i 2 [ n ] y i ✓ > ( x i  y i ✏ tr | ✓ [1] | e 1 ) , where ✓ [ i ] refers to the i -th entry of vector ✓ .
Revised: ni =1 maximizes the minimum margin that reads min i 2 [ n ] y i ✓ > ( x i  y i ✏ tr | ✓ [1] | e 1 ) , where ✓ [ i ] refers to the i -th entry of vector ✓ .

Original: Therefore, it simply corresponds to the max- ` 2 -margin solution of the dataset shifted towards the decision boundary D ✏ tr = { ( x i  y i ✏ tr | b ✓ ✏ tr [1] | e 1 , y i ) } ni =1 .
Revised: Therefore, it simply corresponds to the max- ` 2 -margin solution of the dataset shifted towards the decision boundary D ✏ tr = { ( x i  y i ✏ tr | b ✓ ✏ tr [1] | e 1 , y i ) } n i =1 .

Original: Using this fact, we obtain a closed-form expression of the (normalized) max-margin solution 4 as a function of ✏ tr that reads
Revised: Using this fact, we obtain a closed-form expression of the (normalized) max-margin solution 4 as a function of ✏ tr that reads

Original: [Equation]
Revised: [Equation]

Original: where k ˜ ✓ k 2 = 1 and ˜  > 0 is a random quantity associated with the max- ` 2 -margin solution of the d  1 dimensional Gaussian inputs orthogonal to the signal direction (see Lemma A.1 in Section A).
Revised: where k ˜ ✓ k 2 = 1 and ˜  > 0 is a random quantity associated with the max- ` 2 -margin solution of the d  1 dimensional Gaussian inputs orthogonal to the signal direction (see Lemma A.1 in Section A).

Original: In high dimensions, with high probability any two Gaussian random vectors are far apart – in our distributional setting, this corresponds to the vectors being far apart in the non-signal directions.
Revised: In high dimensions, with high probability any two Gaussian random vectors are far apart – in our distributional setting, this corresponds to the vectors being far apart in the non-signal directions.

Original: In
Revised: In

Original: Figure 4c, we illustrate the phenomenon using a 2D cartoon, where the few samples in the dataset are all far apart in the non-signal direction.
Revised: Figure 4c, we illustrate the phenomenon using a 2D cartoon, where the few samples in the dataset are all far apart in the non-signal direction.

Original: We see how shifting the dataset closer to the true decision boundary, may result in a max-margin solution (yellow) that aligns much worse with the ground truth (gray), compared to the estimator learned from the original points (blue).
Revised: We see how shifting the dataset closer to the true decision boundary, may result in a max-margin solution (yellow) that aligns much worse with the ground truth (gray), compared to the estimator learned from the original points (blue).

Original: Even though the new (robust max-margin) classiﬁer (yellow) is less susceptible to attacks in the signal dimension, it also uses the signal dimension less.
Revised: Even though the new (robust max-margin) classiﬁer (yellow) is less susceptible to attacks in the signal dimension, it also uses the signal dimension less.

Original: Mathematically, this is reﬂected in the expression of the max-margin solution in Equation 8: We see that the ﬁrst (signal) dimension is used less as ✏ tr increases.
Revised: Mathematically, this is reﬂected in the expression of the max-margin solution in Equation 8: We see that the ﬁrst (signal) dimension is used less as ✏ tr increases.

Original: 3. G ENERALITY OF THE RESULTS
Revised: 3. G ENERALITY OF THE RESULTS

Original: In this section we discuss how the theorem might generalize to other perturbation sets, models and training procedures.
Revised: In this section we discuss how Theorem 3.1 might generalize to other perturbation sets and models.

Original: Signal direction is known The type of additive perturbations used in Theorem 3.1, deﬁned in
Revised: Signal direction is known The type of additive perturbations used in Theorem 3.1, deﬁned in

Original: Equation 3, is explicitly constrained to the direction of the true signal.
Revised: Equation 3, is explicitly constrained to the direction of the true signal.

Original: This choice is reminiscent of corruptions where every possible perturbation in the set is directly targeted at the object to be recognized, such as motion blur of moving objects.
Revised: This choice is reminiscent of corruptions where every possible perturbation in the set is directly targeted at the object to be recognized, such as motion blur of moving objects.

Original: Such corruptions are also studied in the context of domain generalization and adaptation (Schneider et al., 2020).
Revised: Such corruptions are also studied in the context of domain generalization and adaptation (Schneider et al., 2020).

Original: Directed attacks in general, however, may also consist of perturbation sets that are only strongly biased towards the true signal direction.
Revised: Directed attacks in general, however, may also consist of perturbation sets that are only strongly biased towards the true signal direction.

Original: They may ﬁnd the true signal direction only when the inner maximization is exact.
Revised: They may ﬁnd the true signal direction only when the inner maximization is exact.

Original: The following corollary extends Theorem 3.1 to small ` 1 -perturbations
Revised: The following corollary extends Theorem 3.1 to small ` 1 -perturbations

Original: [Equation]
Revised: [Equation]

Original: (9) for 0 < ✏ < r 2 that reﬂect such attacks.
Revised: for 0 < ✏ < r 2 that reﬂect such attacks.

Original: We state the corollary here and give the proof in Appendix A.
Revised: We state the corollary here and give the proof in Appendix A.

Original: Corollary 3.2.
Revised: Corollary 3.2.

Original: Theorem 3.1 also holds for 4 with perturbation sets deﬁned in 9.
Revised: Theorem 3.1 also holds for 4 with perturbation sets deﬁned in 9.

Original: The proof uses the fact that the inner maximization effectively results in a sparse perturbation equivalent to the attack resulting from the perturbation set  3.
Revised: The proof uses the fact that the inner maximization effectively results in a sparse perturbation equivalent to the attack resulting from the perturbation set deﬁned in Equation 3.

Original: Other models Motivated by the implicit bias results of (stochastic) gradient descent on the logistic loss, Theorem 3.1 is proven for the max- ` 2 -margin solution.
Revised: Other models Motivated by the implicit bias results of (stochastic) gradient descent on the logistic loss, Theorem 3.1 is proven for the max- ` 2 -margin solution.

Original: We would like to conjecture that for the data distribution in Section 3, adversarial training can hurt robust generalization also for other models with zero training error ( interpolators in short).
Revised: We would like to conjecture that for the data distribution in Section 3, adversarial training can hurt robust generalization also for other models with zero training error ( interpolators in short).

Original: For example, Adaboost is a widely used algorithm that converges to the max- ` 1 -margin classiﬁer (Telgarsky, 2013).
Revised: For example, Adaboost is a widely used algorithm that converges to the max- ` 1 -margin classiﬁer (Telgarsky, 2013).

Original: One might argue that for a sparse ground truth, the max- ` 1 -margin classiﬁer should (at least in the noiseless case) have the right inductive bias to alleviate large bias in high dimensions.
Revised: One might argue that for a sparse ground truth, the max- ` 1 -margin classiﬁer should (at least in the noiseless case) have the right inductive bias to alleviate large bias in high dimensions.

Original: Hence, in many cases the (sparse) max- ` 1 -margin solution might align with the ground truth for a given dataset.
Revised: Hence, in many cases the (sparse) max- ` 1 -margin solution might align with the ground truth for a given dataset.

Original: However, we conjecture that even in this case, the robust max- ` 1 -margin solution would be misled to choose a wrong sparse solution.
Revised: However, we conjecture that even in this case, the robust max- ` 1 -margin solution would be misled to choose a wrong sparse solution.

Original: This can be seen with the help of the cartoon illustration in Figure 4c.
Revised: This can be seen with the help of the cartoon illustration in Figure 4c.

Original:  R EAL - WORLD EXPERIMENTS
Revised:  R EAL - WORLD EXPERIMENTS

Original: In this section, we demonstrate that the proof intuition of the linear case may generalize to more complex models.
Revised: In this section, we demonstrate that the proof intuition of the linear case may generalize to more complex models.

Original: Speciﬁcally, the insights from Section 3 helped us to identify realistic directed attacks on standard image datasets for which adversarial training hurts robust accuracy in the low sample regime.
Revised: Speciﬁcally, the insights from Section 3 helped us to identify realistic directed attacks on standard image datasets for which adversarial training hurts robust accuracy in the low sample regime.

Original: In what follows, we present experimental results for corruption attacks on the
Revised: In what follows, we present experimental results for corruption attacks on the

Original: Waterbirds dataset.
Revised: Waterbirds dataset.

Original: Due to space constraints, results on the mask attacks on CIFAR-10 can be found inApp.
Revised: Due to space constraints, results on the mask attacks on CIFAR-10 can be found in

Original:  E.
Revised: Appendix E.

Original: The corresponding experimental details and more results on other additional image datasets (such as the hand gestures dataset) can be found in Appendices D, E and F. 
Revised: The corresponding experimental details and more results on other additional image datasets (such as the hand gestures dataset) can be found in Appendices D, E and F. [Figure]

Original: 4. D ATASETS AND MODELS
Revised: 4. D ATASETS AND MODELS

Original: We consider three datasets: the Waterbirds dataset, CIFAR-10 and a hand gesture datasets.
Revised: We consider three datasets: the Waterbirds dataset, CIFAR-10 and a hand gesture datasets.

Original: Due to space constraints, we describe CIFAR-10 and the hand gesture dataset in Appendix E and F. Apart[Figure] from CIFAR-10 and the hand gesture dataset, we build a new version of the Waterbirds dataset, consisting of images of water- and landbirds of size 256 ⇥ 256 and labels that distinguish the two types of birds.
Revised: Due to space constraints, we describe CIFAR-10 and the hand gesture dataset in Appendix E and F. Apart from CIFAR-10 and the hand gesture dataset, we build a new version of the Waterbirds dataset, consisting of images of water- and landbirds of size 256 ⇥ 256 and labels that distinguish the two types of birds.

Original: Using code provided by Sagawa et al.
Revised: Using code provided by Sagawa et al.

Original: (2020), we construct the dataset as follows:
Revised: (2020), we construct the dataset as follows:

Original: First, we sample equally many water- and landbirds from the CUB-200 dataset (Welinder et al., 2010).
Revised: First, we sample equally many water- and landbirds from the CUB-200 dataset (Welinder et al., 2010).

Original: Then, we segment the birds and paste them onto a background image that is randomly sampled (without replacement) from the Places-256 dataset (Zhou et al., 2017).
Revised: Then, we segment the birds and paste them onto a background image that is randomly sampled (without replacement) from the Places-256 dataset (Zhou et al., 2017).

Original: Also, following the choice of
Revised: Also, following the choice of

Original: Sagawa et al.
Revised: Sagawa et al.

Original: (2020), we use as models a ResNet50 and a ResNet18 that were both pretrained on
Revised: (2020), we use as models a ResNet50 and a ResNet18 that were both pretrained on

Original: ImageNet and achieve near perfect standard accuracy.
Revised: ImageNet and achieve near perfect standard accuracy.

Original: In Appendix D, we complement the results of this section by reporting the results of similar experiments with different architectures.
Revised: In Appendix D, we complement the results of this section by reporting the results of similar experiments with different architectures.

Original: 4. I MPLEMENTATION OF THE DIRECTED ATTACKS
Revised: 4. I MPLEMENTATION OF THE DIRECTED ATTACKS

Original: In this section, we consider two attacks on the Waterbirds dataset: motion blur and adversarial illumination as depicted in Figure 2.
Revised: In this section, we consider two attacks on the Waterbirds dataset: motion blur and adversarial illumination as depicted in Figure 2.

Original: In Appendix E, we also discuss the mask attack, which should mimic occlusions of objects in images that are physically realizable (Eykholt et al., 2018; Wu et al., 2020).
Revised: In Appendix E, we also discuss the mask attack, which should mimic occlusions of objects in images that are physically realizable (Eykholt et al., 2018; Wu et al., 2020).

Original: On the other hand, motion blur may arise naturally when photographing fast moving objects with a slow shutter speed.
Revised: On the other hand, motion blur may arise naturally when photographing fast moving objects with a slow shutter speed.

Original: Lastly, adversarial illumination may result from adversarial lighting conditions.
Revised: Lastly, adversarial illumination may result from adversarial lighting conditions.

Original: Next, we describe the motion blur and adversarial illumination attacks in more detail.
Revised: Next, we describe the motion blur and adversarial illumination attacks in more detail.

Original: Motion blur For the Waterbirds dataset we can implement motion blur attacks on the object (the bird) speciﬁcally, a natural corruption that could occur if birds move at speeds that are faster than the shutter speed.
Revised: Motion blur For the Waterbirds dataset we can implement motion blur attacks on the object (the bird) speciﬁcally, a natural corruption that could occur if birds move at speeds that are faster than the shutter speed.

Original: The aim is to be robust against all motion blur severity levels up to M max = 15 .
Revised: The aim is to be robust against all motion blur severity levels up to M max = 15 .

Original: To simulate motion blur, we apply a motion blur ﬁlter with a kernel of size M on the segmented bird before we paste it onto the background image.
Revised: To simulate motion blur, we apply a motion blur ﬁlter with a kernel of size M on the segmented bird before we paste it onto the background image.

Original: We can change the severity level of the motion blur by increasing the kernel size of the ﬁlter.
Revised: We can change the severity level of the motion blur by increasing the kernel size of the ﬁlter.

Original: See Appendix D for concrete expressions of the motion blur kernel.
Revised: See Appendix D for concrete expressions of the motion blur kernel.

Original: Intuitively the worst attack should be the most severe blur, rendering a search over a range of severity superﬂuous.
Revised: Intuitively the worst attack should be the most severe blur, rendering a search over a range of severity superﬂuous.

Original: However, similar to rotations, this is not necessarily true in practice since the training loss on neural networks is generally nonconvex.
Revised: However, similar to rotations, this is not necessarily true in practice since the training loss on neural networks is generally nonconvex.

Original: Therefore, for an exact evaluation of the robust error at test time, we perform a full grid search over all kernel sizes in [1 , 2 , . . .
Revised: Therefore, for an exact evaluation of the robust error at test time, we perform a full grid search over all kernel sizes in [1 , 2 , . . .

Original: , M max ] .
Revised: , M max ] .

Original: We refer to Figure 2d and Section D for an illustration of our motion blur attack.
Revised: We refer to Figure 2d and Section D for an illustration of our motion blur attack.

Original: During training time, we perform an approximate search over kernels with sizes 2 i for i = 1 , . . .
Revised: During training time, we perform an approximate search over kernels with sizes 2 i for i = 1 , . . .

Original: , M max / 2 .
Revised: , M max / 2 .

Original: Adversarial illumination As a second attack on the Waterbirds dataset, we consider adversarial illumination.
Revised: Adversarial illumination As a second attack on the Waterbirds dataset, we consider adversarial illumination.

Original: The adversary can darken or brighten the bird without corrupting the background of the image.
Revised: The adversary can darken or brighten the bird without corrupting the background of the image.

Original: The attack aims to model images where the object at interest is hidden in shadows or placed against bright light.
Revised: The attack aims to model images where the object at interest is hidden in shadows or placed against bright light.

Original: To compute the adversarial illumination attack, we modify the brightness of the segmented bird by adding a constant a 2 [  ✏ te , ✏ te ] to all pixel values, before pasting the bird onto the background image.
Revised: To compute the adversarial illumination attack, we modify the brightness of the segmented bird by adding a constant a 2 [  ✏ te , ✏ te ] to all pixel values, before pasting the bird onto the background image.

Original: With an analogous argument as for the adversarial search for motion blur, the  exact evaluation requires an actual search over the interval [  ✏ te , ✏ te ] .
Revised: With an analogous argument as for the adversarial search for motion blur, the [Figure] exact evaluation requires an actual search over the interval [  ✏ te , ✏ te ] .

Original: We ﬁnd the most adversarial lighting level, i.e. the value of a , by equidistantly partitioning the interval [  ✏ te , ✏ te ] in K steps and performing a full list-search over all steps.
Revised: We ﬁnd the most adversarial lighting level, i.e. the value of a , by equidistantly partitioning the interval [  ✏ te , ✏ te ] in K steps and performing a full list-search over all steps.

Original: See Figure 2c and Appendix D for an illustration of the adversarial illumination attack.
Revised: See Figure 2c and Appendix D for an illustration of the adversarial illumination attack.

Original: We choose K = 65 , 33 during test and training time respectively.
Revised: We choose K = 65 , 33 during test and training time respectively.

Original: [Figure]

Original: Adversarial training For all datasets and attacks, we run SGD until convergence on the robust cross-entropy loss 2.
Revised: Adversarial training For all datasets and attacks, we run SGD until convergence on the robust cross-entropy loss 2.

Original: In each iteration, we search for an adversarial example as described above and update the weights using a gradient with respect to the resulting perturbed example (Goodfellow et al., 2015; Madry et al., 2018).
Revised: In each iteration, we search for an adversarial example as described above and update the weights using a gradient with respect to the resulting perturbed example (Goodfellow et al., 2015; Madry et al., 2018).

Original: For every experiment, we choose the learning rate and weight decay parameters that minimize the robust error on a hold-out dataset.
Revised: For every experiment, we choose the learning rate and weight decay parameters that minimize the robust error on a hold-out dataset.

Original: 4. A DVERSARIAL TRAINING CAN HURT ROBUST GENERALIZATION
Revised: 4. A DVERSARIAL TRAINING CAN HURT ROBUST GENERALIZATION

Original: We now present our experimental results on the Waterbirds dataset for both motion blur and adversarial illumination attacks.
Revised: We now present our experimental results on the Waterbirds dataset for both motion blur and adversarial illumination attacks.

Original: First of all, Figure 5a and 6a show that the phenomenon characterized in the linear setting by Theorem 3.1 also occurs for directed attacks on the Waterbirds dataset: as we increase the adversarial training budget ✏ tr starting from zero (standard training), the robust error monotonically increases.
Revised: First of all, Figure 5a and 6a show that the phenomenon characterized in the linear setting by Theorem 3.1 also occurs for directed attacks on the Waterbirds dataset: as we increase the adversarial training budget ✏ tr starting from zero (standard training), the robust error monotonically increases.

Original: Furthermore, to gain intuition as described in Section 3.3, we also plot the robust error decomposition (Equation 7) consisting of the standard error and susceptibility in Figure 5b and 6b.
Revised: Furthermore, to gain intuition as described in Section 3.3, we also plot the robust error decomposition (Equation 7) consisting of the standard error and susceptibility in Figure 5b and 6b.

Original: Recall that we measure susceptibility as the fraction of data points in the test set for which the classiﬁer predicts a different class under an adversarial attack.
Revised: Recall that we measure susceptibility as the fraction of data points in the test set for which the classiﬁer predicts a different class under an adversarial attack.

Original: As in our linear example, we observe an increase in robust error despite a slight drop in susceptibility, because of the more severe increase in standard error.
Revised: As in our linear example, we observe an increase in robust error despite a slight drop in susceptibility, because of the more severe increase in standard error.

Original: Moreover, Figures 1 and 6c show that analogous to our linear example, this phenomenon is speciﬁc to the low sample regime: for large sample size adversarial training outperforms standard training as expected.
Revised: Moreover, Figures 1 and 6c show that analogous to our linear example, this phenomenon is speciﬁc to the low sample regime: for large sample size adversarial training outperforms standard training as expected.

Original: Note again that even the smallest sample size is large enough to yield a standard test error < 10% for standard training.
Revised: Note again that even the smallest sample size is large enough to yield a standard test error < 10% for standard training.

Original: Similar experiments for CIFAR-10 can be found in Appendix E.
Revised: Similar experiments for CIFAR-10 can be found in Appendix E.

Original: Finally, we empirically conﬁrm in Appendix D.8 that our phenomenon is speciﬁc to directed attacks: for undirected attacks such as bounded ` 1  and ` 2  ball perturbations, adversarial training helps robust generalization also in the low sample size regime.
Revised: Finally, we empirically conﬁrm in Appendix D.8 that our phenomenon is speciﬁc to directed attacks: for undirected attacks such as bounded ` 1  and ` 2  ball perturbations, adversarial training helps robust generalization also in the low sample size regime.

Original: 4. D ISCUSSION
Revised: 4. D ISCUSSION

Original: In this section, we discuss how different algorithmic choices, motivated by related work, might affect how adversarial training hurts robust generalization.
Revised: We now discuss how different algorithmic choices, motivated by related work, might affect how adversarial training hurts robust generalization.

Original: Catastrophic overﬁtting Often the worst-case perturbation during adversarial training is found using an approximate algorithm such as SGD.
Revised: Catastrophic overﬁtting Often the worst-case perturbation during adversarial training is found using an approximate algorithm such as SGD.

Original: It is common belief that using the strongest attack (in the motion blur case, full grid search) during training also results in better robust generalization.
Revised: It is common belief that using the strongest attack (in the motion blur case, full grid search) during training also results in better robust generalization.

Original: In particular, the literature on catastrophic overﬁtting shows that weaker attacks during training lead to bad performance on stronger attacks during testing (Wong et al., 2020; Andriushchenko &
Revised: In particular, the literature on catastrophic overﬁtting shows that weaker attacks during training lead to bad performance on stronger attacks during testing (Wong et al., 2020; Andriushchenko &

Original: Flammarion, 2020; Li et al., 2021).
Revised: Flammarion, 2020; Li et al., 2021).

Original: Our results suggests the opposite in the low sample size regime for directed attacks: the weaker the attack during training, the better adversarial training performs.
Revised: Our results suggests the opposite in the low sample size regime for directed attacks: the weaker the attack during training, the better adversarial training performs.

Original: Robust overﬁtting Recent work observes empirically (Rice et al., 2020) and theoretically (Sanyal et al., 2020;
Revised: Robust overﬁtting Recent work observes empirically (Rice et al., 2020) and theoretically (Sanyal et al., 2020;

Original: Donhauser et al., 2021), that perfectly minimizing the adversarial loss during training might in fact be suboptimal for robust generalization; that is, classical regularization techniques might lead to higher robust accuracy.
Revised: Donhauser et al., 2021), that perfectly minimizing the adversarial loss during training might in fact be suboptimal for robust generalization; that is, classical regularization techniques might lead to higher robust accuracy.

Original: This phenomenon is often referred to as robust overﬁtting.
Revised: This phenomenon is often referred to as robust overﬁtting.

Original: May the phenomenon be mitigated using standard regularization techniques?
Revised: May the phenomenon be mitigated using standard regularization techniques?

Original: In Appendix D we shed light on this question and show that adversarial training hurts robust generalization even when standard regularization methods such as early stopping are used.
Revised: In Appendix D we shed light on this question and show that adversarial training hurts robust generalization even when standard regularization methods such as early stopping are used.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: We now discuss how our results relate to phenomena that have been studied in the literature before.

Original: Robust and non-robust useful features In the words of Ilyas et al.
Revised: Robust and non-robust useful features In the words of Ilyas et al.

Original: and Springer et al.
Revised: and Springer et al.

Original: (2021) we can describe the intuition behind ”our phenomenon” as follows: for directed attacks, all robust features become less useful, but adversarial training uses robust features more.
Revised: (2021) we can describe the intuition behind ”our phenomenon” as follows: for directed attacks, all robust features become less useful, but adversarial training uses robust features more.

Original: In the small sample-size regime, n < d  1 in particular, robust learning assigns too much weight on the robust (possibly non-useful) features that then dominate the non-robust (but useful)features.
Revised: In the small sample-size regime, n < d  1 in particular, robust learning assigns too much weight on the robust (possibly non-useful) features that then dominate the non-robust (but useful)features.

Original: Even though they deﬁne these concepts, they don’t make our statement, but show that adversarial training reduces the reliance on non-robust but possibly useful features.
Revised: Even though they deﬁne these concepts, they don’t make our statement, but show that adversarial training reduces the reliance on non-robust but possibly useful features.

Original: Small sample size and robustness A direct consequence of Theorem 3.1 is that in order to achieve the same robust error as standard training, adversarial training requires more samples.
Revised: Small sample size and robustness A direct consequence of Theorem 3.1 is that in order to achieve the same robust error as standard training, adversarial training requires more samples.

Original: This statement might remind the reader of sample complexity results for robust generalization in Schmidt et al.
Revised: This statement might remind the reader of sample complexity results for robust generalization in Schmidt et al.

Original: ; Yin et al.
Revised: ; Yin et al.

Original: ; Khim & Loh (2018).
Revised: ; Khim & Loh (2018).

Original: While those results compare sample complexity bounds for standard vs. robust error, our theorem statement compares two algorithms, standard vs. adversarial training, with respect to the robust error.
Revised: While those results compare sample complexity bounds for standard vs. robust error, our theorem statement compares two algorithms, standard vs. adversarial training, with respect to the robust error.

Original: Trade-off between standard and robust error Many papers observed that even though adversarial training decreases robust error compared to standard training, it may lead to an increase in standard test error Madry et al.
Revised: Trade-off between standard and robust error Many papers observed that even though adversarial training decreases robust error compared to standard training, it may lead to an increase in standard test error Madry et al.

Original: ; Zhang et al.
Revised: ; Zhang et al.

Original: For example, Tsipras et al.
Revised: For example, Tsipras et al.

Original: ; Zhang et al.
Revised: ; Zhang et al.

Original: ; Javanmard et al.
Revised: ; Javanmard et al.

Original: ; Dobriban et al.
Revised: ; Dobriban et al.

Original: ; Chen et al.
Revised: ; Chen et al.

Original: (2020) study settings where the Bayes optimal robust classiﬁer is not equal to the Bayes optimal (standard) classiﬁer (i.e. the perturbations are inconsistent or the dataset is non-separable).
Revised: (2020) study settings where the Bayes optimal robust classiﬁer is not equal to the Bayes optimal (standard) classiﬁer (i.e. the perturbations are inconsistent or the dataset is non-separable).

Original: Raghunathan et al.
Revised: Raghunathan et al.

Original: (2020) study consistent perturbations, as in our paper, and prove that for small sample size, ﬁtting adversarial examples can increase standard error even in the absence of noise.
Revised: (2020) study consistent perturbations, as in our paper, and prove that for small sample size, ﬁtting adversarial examples can increase standard error even in the absence of noise.

Revised: Empirically, Dong et al.

Revised: Mendonc¸a et al.

Revised: (2022) show that for ` p -attacks low-quality data might be the main cause of the trade-off.

Original: While aforementioned works focus on the decrease in standard error, we prove that for directed attacks, in the small sample regime adversarial training may in fact increase robust error .
Revised: While aforementioned works focus on the decrease in standard error, we prove that for directed attacks, in the small sample regime adversarial training may in fact increase robust error .

Original: Mitigation of the trade-off
Revised: Mitigation of the trade-off

Original: A long line of work has proposed procedures to mitigate the trade-off between robust and standard accuracy.
Revised: A long line of work has proposed procedures to mitigate the trade-off between robust and standard accuracy.

Original: For example Alayrac et al.
Revised: For example Alayrac et al.

Original: ; Carmon et al.
Revised: ; Carmon et al.

Original: ; Zhai et al.
Revised: ; Zhai et al.

Original: ; Raghunathan et al.
Revised: ; Raghunathan et al.

Original: (2020) study robust self training, which leverages a large set of unlabelled data, while Lee et al.
Revised: (2020) study robust self training, which leverages a large set of unlabelled data, while Lee et al.

Original: ; Lamb et al.
Revised: ; Lamb et al.

Original: ; Xu et al.
Revised: ; Xu et al.

Original: (2020) use data augmentation by interpolation.
Revised: (2020) use data augmentation by interpolation.

Original: Ding et al.
Revised: Ding et al.

Original: Balaji et al.
Revised: Balaji et al.

Original: ; Cheng et al.
Revised: ; Cheng et al.

Original: (2020) on the other hand propose to use adaptive perturbation budgets ✏ tr that vary across inputs.
Revised: (2020) on the other hand propose to use adaptive perturbation budgets ✏ tr that vary across inputs.

Original: The intuition behind our theoretical analysis suggests that the standard mitigation procedures for imperceptible perturbations may not work for perceptible directed attacks, because all relevant features are non-robust.
Revised: The intuition behind our theoretical analysis suggests that the standard mitigation procedures for imperceptible perturbations may not work for perceptible directed attacks, because all relevant features are non-robust.

Original: We leave a thorough empirical study as interesting future work.
Revised: We leave a thorough empirical study as interesting future work.

Original:  S UMMARY AND FUTURE WORK
Revised:  S UMMARY AND FUTURE WORK

Original: This paper aims to caution the practitioner against blindly following current widespread practices to increase the robust performance of machine learning models.
Revised: This paper aims to caution the practitioner against blindly following current widespread practices to increase the robust performance of machine learning models.

Original: Speciﬁcally, adversarial training is currently recognized to be one of the most effective defense mechanisms for ` p -perturbations, signiﬁcantly outperforming robust performance of standard training.
Revised: Speciﬁcally, adversarial training is currently recognized to be one of the most effective defense mechanisms for ` p -perturbations, signiﬁcantly outperforming robust performance of standard training.

Original: However, we prove that in the low sample size regime this common wisdom is not applicable for consistent directed attacks, which efﬁciently focus their attack budget to target the ground truth class information.
Revised: However, we prove that in the low sample size regime this common wisdom is not applicable for consistent directed attacks, which efﬁciently focus their attack budget to target the ground truth class information.

Original: In terms of follow-up work on directed attacks in the low sample regime, there are some concrete questions that would be interesting to explore.
Revised: In terms of follow-up work on directed attacks in the low sample regime, there are some concrete questions that would be interesting to explore.

Original: For example, as discussed in Section 5, it would be useful to test whether some methods to mitigate the standard accuracy vs. robustness trade-off would also relieve the perils of adversarial training for directed attacks.
Revised: For example, as discussed in Section 5, it would be useful to test whether some methods to mitigate the standard accuracy vs. robustness trade-off would also relieve the perils of adversarial training for directed attacks.

Original: Further, we hypothesize that when few samples are available, one should avoid training with attacks that may heavily reduce class information, independently of the attacks at test time.
Revised: Further, we hypothesize that when few samples are available, one should avoid training with attacks that may heavily reduce class information, independently of the attacks at test time.

Original: If this hypothesis were conﬁrmed, it would break with yet another general rule that the best defense perturbation type should always match the attack during evaluation.
Revised: If this hypothesis were conﬁrmed, it would break with yet another general rule that the best defense perturbation type should always match the attack during evaluation.