Original: O N THE I MPACT OF H ARD A DVERSARIAL I NSTANCES ON O VERFITTING IN A DVERSARIAL T RAINING
Revised: O N THE I MPACT OF H ARD A DVERSARIAL I NSTANCES ON O VERFITTING IN A DVERSARIAL T RAINING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Adversarial training is a popular method to robustify models against adversarial attacks.
Revised: Adversarial training is a popular method to robustify models against adversarial attacks.

Original: However, it exhibits much more severe overﬁtting than training on clean inputs.
Revised: However, it exhibits much more severe overﬁtting than training on clean inputs.

Original: In this work, we investigate this phenomenon from the perspective of training instances, i e., training input-target pairs.
Revised: In this work, we investigate this phenomenon from the perspective of training instances, i e., training input-target pairs.

Original: We provide a quantitativeand model-agnostic metric measuring the difﬁculty of an instance in the training set and analyze the model’s behavior on instances of different difﬁculty levels.
Revised: We provide a quantitative metric measuring the difﬁculty of an instance in the training set and analyze the model’s behavior on instances of different difﬁculty levels.

Original: This lets us show that the decay in generalization performance of adversarial training is a result of the model’s attempt to ﬁt hard adversarial instances.
Revised: This lets us show that the decay in generalization performance of adversarial training is a result of the model’s attempt to ﬁt hard adversarial instances.

Original: We theoretically verify our observations for both linear and general nonlinear models, proving that models trained on hard instances have worse generalization performance than ones trained on easy instances, and that this generalization gap is larger in adversarial training.
Revised: We theoretically verify our observations for both linear and general nonlinear models, proving that models trained on hard instances have worse generalization performance than ones trained on easy instances, and that this generalization gap is larger in adversarial training.

Original: Finally, we investigate solutions to mitigate adversarial overﬁtting in several scenarios, including when relying on fast adversarial training and in the context of ﬁne-tuning a pretrained model with additional data.
Revised: Finally, we investigate solutions to mitigate adversarial overﬁtting in several scenarios, including when relying on fast adversarial training and in the context of ﬁne-tuning a pretrained model with additional data.

Original: Our results demonstrate that using training data adaptively improves the model’s robustness
Revised: Our results demonstrate that using training data adaptively improves the model’s robustness.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: The existence of adversarial examples (Szegedy et al., 2014) causes serious safety concerns when deploying modern deep learning models.
Revised: The existence of adversarial examples (Szegedy et al., 2014) causes serious safety concerns when deploying modern deep learning models.

Original: For example, for classiﬁcation tasks, imperceptible perturbations of the input instance can fool state-of-the-art classiﬁers.
Revised: For example, for classiﬁcation tasks, imperceptible perturbations of the input instance can fool state-of-the-art classiﬁers.

Original: Many strategies to obtain models that are robust against adversarial attacks have been proposed (Buckman et al., 2018; Dhillon et al., 2018;
Revised: Many strategies to obtain models that are robust against adversarial attacks have been proposed (Buckman et al., 2018; Dhillon et al., 2018;

Original: Ma et al., 2018; Pang et al., 2020; 2019; Samangouei et al., 2018; Xiao et al., 2020), but most of them have been found to be ineffective in the presence of adaptive attacks (Athalye et al., 2018; Croce &
Revised: Ma et al., 2018; Pang et al., 2020; 2019; Samangouei et al., 2018; Xiao et al., 2020), but most of them have been found to be ineffective in the presence of adaptive attacks (Athalye et al., 2018; Croce &

Original: Hein, 2020b; Tramer et al., 2020).
Revised: Hein, 2020b; Tramer et al., 2020).

Original: Ultimately, this leaves adversarial training (Madry et al., 2018) and its variants (Alayrac et al., 2019; Carmon et al., 2019; Gowal et al., 2020; Hendrycks et al., 2019;
Revised: Ultimately, this leaves adversarial training (Madry et al., 2018) and its variants (Alayrac et al., 2019; Carmon et al., 2019; Gowal et al., 2020; Hendrycks et al., 2019;

Original: Kumari et al., 2019; Wu et al., 2020; Zhang et al., 2019a)
Revised: Kumari et al., 2019; Wu et al., 2020; Zhang et al., 2019a)

Original: as the most effective and popular approach to construct robust models.
Revised: as the most effective and popular approach to construct robust models.

Original: Unfortunately, adversarial training yields much worse performance on the test data than vanilla training.
Revised: Unfortunately, adversarial training yields much worse performance on the test data than vanilla training.

Original: In particular, it strongly suffers from overﬁtting (Rice et al., 2020), with the model’s performance decaying signiﬁcantly on the test set in the later phase of adversarial training.
Revised: In particular, it strongly suffers from overﬁtting (Rice et al., 2020), with the model’s performance decaying signiﬁcantly on the test set in the later phase of adversarial training.

Original: While this can be mitigated by early stopping (Rice et al., 2020) or model smoothing (Chen et al., 2021b), the reason behind the overﬁtting of adversarial training remains poorly understood.
Revised: While this can be mitigated by early stopping (Rice et al., 2020) or model smoothing (Chen et al., 2021b), the reason behind the overﬁtting of adversarial training remains poorly understood.

Original: In this paper, we study this phenomenon from the perspective of training instances, i.e., training input-target pairs.
Revised: In this paper, we study this phenomenon from the perspective of training instances, i.e., training input-target pairs.

Original: We introduce a quantitative and model-agnostic metric to deﬁne the difﬁculty of the instances and analyze the model’s behavior, such as its loss and intermediate activations, on training instances of different difﬁculty levels.
Revised: We introduce a quantitative and model-agnostic metric to deﬁne the difﬁculty of the instances and analyze the model’s behavior, such as its loss and intermediate activations, on training instances of different difﬁculty levels.

Original: This lets us discover that the model’s generalization performance decays signiﬁcantly when it ﬁts the hard adversarial instances in the later training phase.
Revised: This lets us discover that the model’s generalization performance decays signiﬁcantly when it ﬁts the hard adversarial instances in the later training phase.

Original: To more rigorously study this phenomenon, we then perform theoretical analyses on both linear and nonlinear models.
Revised: To more rigorously study this phenomenon, we then perform theoretical analyses on both linear and nonlinear models.

Original: For linear models, we study logistic regression on a Gaussian mixture model, in which we can calculate the analytical expression of the model parameters upon convergence and thus the robust test accuracy.
Revised: For linear models, we study logistic regression on a Gaussian mixture model, in which we can calculate the analytical expression of the model parameters upon convergence and thus the robust test accuracy.

Original: Our theorem demonstrates that adversarial training on harder instances leads to larger generalization gaps.
Revised: Our theorem demonstrates that adversarial training on harder instances leads to larger generalization gaps.

Original: We further prove that the gap in robust test accuracy between models trained by hard instances and ones trained by easy instances increases with the size of the adversarial budget.
Revised: We further prove that the gap in robust test accuracy between models trained by hard instances and ones trained by easy instances increases with the size of the adversarial budget.

Original: In the case of nonlinear models, we derive the lower bound of the model’s Lipschitz constant when the model is well ﬁt to the adversarial training examples.
Revised: In the case of nonlinear models, we derive the lower bound of the model’s Lipschitz constant when the model is well ﬁt to the adversarial training examples.

Original: This bound increases with the difﬁculty level of the training instances and the size of the adversarial budget.
Revised: This bound increases with the difﬁculty level of the training instances and the size of the adversarial budget.

Original: Since a larger Lipschitz constant indicates a higher adversarial vulnerability (Ruan et al., 2018; Weng et al., 2018a;b), our theoretical analysis conﬁrms our empirical observations.
Revised: Since a larger Lipschitz constant indicates a higher adversarial vulnerability (Ruan et al., 2018; Weng et al., 2018a;b), our theoretical analysis conﬁrms our empirical observations.

Original: Our ﬁndings can be broadly applied to obtain robust models.
Revised: Our ﬁndings can be broadly applied to obtain robust models.

Original: To evidence this, in addition to the standard adversarial training settings of (Madry et al., 2018), we study the following two scenarios: fast adversarial training and adversarial ﬁne-tuning with additional training data.
Revised: To evidence this, in addition to the standard adversarial training settings of (Madry et al., 2018), we study the following two scenarios: fast adversarial training and adversarial ﬁne-tuning with additional training data.

Original: Our proposed method assigns adaptive targets or adaptive weights on training instances to avoid ﬁtting hard inputtarget pairs.
Revised: Our proposed method assigns adaptive targets or adaptive weights on training instances to avoid ﬁtting hard inputtarget pairs.

Original: We show it can mitigate adversarial overﬁtting and improve models’ performance.
Revised: We show it can mitigate adversarial overﬁtting and improve models’ performance.

Original: For fast adversarial training, our results are better than other accelerated adversarial training methods available on RobustBench (Croce et al., 2020).
Revised: For fast adversarial training, our results are better than other accelerated adversarial training methods available on RobustBench (Croce et al., 2020).

Original: For adversarial ﬁne-tuning with additional training data, we show improved performance over the methods in (Alayrac et al., 2019; Carmon et al., 2019).
Revised: For adversarial ﬁne-tuning with additional training data, we show improved performance over the methods in (Alayrac et al., 2019; Carmon et al., 2019).

Original: Contributions.
Revised: Contributions.

Original: In summary, our contributions are as follows: 1) Based on a quantitativeand modelagnostic metric of instance difﬁculty, we show that ﬁtting hard adversarial instances leads to degraded generalization performance in adversarial training.
Revised: In summary, our contributions are as follows: 1) Based on a quantitative metric of instance difﬁculty, we show that ﬁtting hard adversarial instances leads to degraded generalization performance in adversarial training.

Original: We conduct a rigorous theoretical analysis on both linear and nonlinear models.
Revised: We conduct a rigorous theoretical analysis on both linear and nonlinear models.

Original: For linear models, we show analytically that models trained on harder instances have larger robust test error than the ones trained on easy instances; the gap increases with the size of the adversarial budget.
Revised: For linear models, we show analytically that models trained on harder instances have larger robust test error than the ones trained on easy instances; the gap increases with the size of the adversarial budget.

Original: For nonlinear models, we derive a lower bound of the model’s Lipschitz constant.
Revised: For nonlinear models, we derive a lower bound of the model’s Lipschitz constant.

Original: The lower bound increases with the difﬁculty of the training instances and the size of the adversarial budget, indicating both factors make adversarial overﬁtting more severe.
Revised: The lower bound increases with the difﬁculty of the training instances and the size of the adversarial budget, indicating both factors make adversarial overﬁtting more severe.

Original: We show that the adaptive use of the easy and hard training instances can improve the performance in fast adversarial training and adversarial ﬁne-tuning with additional training data.
Revised: We show that the adaptive use of the easy and hard training instances can improve the performance in fast adversarial training and adversarial ﬁne-tuning with additional training data.

Original: Notation and terminology.
Revised: Notation and terminology.

Original: In this paper, x and x (cid:48) are the clean input and its adversarial counterpart.
Revised: In this paper, x and x (cid:48) are the clean input and its adversarial counterpart.

Original: We use f w to represent a model parameterized by w and omit the subscript w unless ambiguous.
Revised: We use f w to represent a model parameterized by w and omit the subscript w unless ambiguous.

Original: o = f w ( x ) and o (cid:48)
Revised: o = f w ( x ) and o (cid:48)

Original: =
Revised: =

Original: f w ( x (cid:48) ) are the model’s output of the clean input and the adversarial input.
Revised: f w ( x (cid:48) ) are the model’s output of the clean input and the adversarial input.

Original: L w ( x , y ) and L w ( x (cid:48) , y ) represent the loss of the clean and adversarial instances, receptively, in which we sometimes omit w and y for notation simplicity.
Revised: L w ( x , y ) and L w ( x (cid:48) , y ) represent the loss of the clean and adversarial instances, receptively, in which we sometimes omit w and y for notation simplicity.

Original: We use (cid:107) w (cid:107) and (cid:107) X (cid:107) to represent the l 2 norm of the vector w and the largest singular value of the matrix X , respectively.
Revised: We use (cid:107) w (cid:107) and (cid:107) X (cid:107) to represent the l 2 norm of the vector w and the largest singular value of the matrix X , respectively.

Original: sign is an elementwise function which returns +1 for positive elements, − 1 for negative elements and 0 for 0 .
Revised: sign is an elementwise function which returns +1 for positive elements, − 1 for negative elements and 0 for 0 .

Original: 1 y is the one-hot vector with only the y -th dimension being 1 .
Revised: 1 y is the one-hot vector with only the y -th dimension being 1 .

Original: The term adversarial budget refers to the allowable perturbations applied to the input instance.
Revised: The term adversarial budget refers to the allowable perturbations applied to the input instance.

Original: It is characterized by l p norm and the size (cid:15) as a set S ( p ) ( (cid:15) ) =
Revised: It is characterized by l p norm and the size (cid:15) as a set S ( p ) ( (cid:15) ) =

Original: { ∆ |(cid:107) ∆ (cid:107) p ≤ (cid:15) } , with (cid:15) deﬁning the budget size.
Revised: { ∆ |(cid:107) ∆ (cid:107) p ≤ (cid:15) } , with (cid:15) deﬁning the budget size.

Original: Therefore, given the training set D , the robust learning problem can be formulated as the min-max optimization min w E x ∼D max ∆ ∈S ( p ) ( (cid:15) )
Revised: Therefore, given the training set D , the robust learning problem can be formulated as the min-max optimization min w E x ∼D max ∆ ∈S ( p ) ( (cid:15) )

Original: L w ( x + ∆) .
Revised: L w ( x + ∆) .

Original: A notation table is provided in Appendix A.
Revised: A notation table is provided in Appendix A.

Original: In this paper, vanilla training refers to training on the clean inputs, and vanilla adversarial training to the adversarial training method in (Madry et al., 2018).
Revised: In this paper, vanilla training refers to training on the clean inputs, and vanilla adversarial training to the adversarial training method in (Madry et al., 2018).

Original: RN18 and WRN34 are the 18-layer ResNet (He et al., 2016) and the 34-layer WideResNet (Zagoruyko & Komodakis, 2016) used in (Madry et al., 2018) and (Wong et al., 2020), respectively.
Revised: RN18 and WRN34 are the 18-layer ResNet (He et al., 2016) and the 34-layer WideResNet (Zagoruyko & Komodakis, 2016) used in (Madry et al., 2018) and (Wong et al., 2020), respectively.

Original: To avoid confusion with the general term overﬁtting , which denotes the gap between the training and test accuracy, we employ the term adversarial overﬁtting to indicate the phenomenon that robust accuracy on the test set decreases signiﬁcantly in the later phase of vanilla adversarial training.
Revised: To avoid confusion with the general term overﬁtting , which denotes the gap between the training and test accuracy, we employ the term adversarial overﬁtting to indicate the phenomenon that robust accuracy on the test set decreases signiﬁcantly in the later phase of vanilla adversarial training.

Original: This phenomenon was pointed out in (Rice et al., 2020) and does not occur in vanilla training.
Revised: This phenomenon was pointed out in (Rice et al., 2020) and does not occur in vanilla training.

Original: Our code is submitted on GoogleDrive anonymously 1 .
Revised: Our code is submitted on GoogleDrive anonymously 1 .

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: We concentrate on white-box attacks, where the attacker has access to the model parameters.
Revised: We concentrate on white-box attacks, where the attacker has access to the model parameters.

Original: Such attacks are usually based on ﬁrst-order information and stronger than black-box attacks (Andriushchenko et al., 2020; Dong et al., 2018).
Revised: Such attacks are usually based on ﬁrst-order information and stronger than black-box attacks (Andriushchenko et al., 2020; Dong et al., 2018).

Original: For example, the fast gradient sign method (FGSM) (Goodfellow et al., 2014) perturbs the input based on its gradient’s sign,
Revised: For example, the fast gradient sign method (FGSM) (Goodfellow et al., 2014) perturbs the input based on its gradient’s sign,

Original: i.e., ∆ =
Revised: i.e., ∆ =

Original: (cid:15) sign ( (cid:79) x
Revised: (cid:15) sign ( (cid:79) x

Original: L w ( x )) .
Revised: L w ( x )) .

Original: The iterative fast gradient sign method (IFGSM) (Kurakin et al., 2016) iteratively runs FGSM using a smaller step size and projects the perturbation back to the adversarial budget after each iteration.
Revised: The iterative fast gradient sign method (IFGSM) (Kurakin et al., 2016) iteratively runs FGSM using a smaller step size and projects the perturbation back to the adversarial budget after each iteration.

Original: On top of IFGSM, projected gradient descent (PGD) (Madry et al., 2018) use random initial perturbations and restarts to boost the strength of the attack.
Revised: On top of IFGSM, projected gradient descent (PGD) (Madry et al., 2018) use random initial perturbations and restarts to boost the strength of the attack.

Original: Many methods have been proposed to defend a model against adversarial attacks (Buckman et al., 2018; Dhillon et al., 2018; Ma et al., 2018; Pang et al., 2020; 2019; Samangouei et al., 2018; Xiao et al., 2020).
Revised: Many methods have been proposed to defend a model against adversarial attacks (Buckman et al., 2018; Dhillon et al., 2018; Ma et al., 2018; Pang et al., 2020; 2019; Samangouei et al., 2018; Xiao et al., 2020).

Original: However, most of them were shown to utilize obfuscated gradients (Athalye et al., 2018;
Revised: However, most of them were shown to utilize obfuscated gradients (Athalye et al., 2018;

Original: Croce & Hein, 2020b; Tramer et al., 2020), that is, training the model to tackle some speciﬁc types of attacks instead of achieving true robustness.
Revised: Croce & Hein, 2020b; Tramer et al., 2020), that is, training the model to tackle some speciﬁc types of attacks instead of achieving true robustness.

Original: This makes these falsely robust models vulnerable to stronger adaptive attacks.
Revised: This makes these falsely robust models vulnerable to stronger adaptive attacks.

Original: By contrast, several works have designed training algorithms to obtain provably robust models (Cohen et al., 2019; Gowal et al., 2019; Raghunathan et al., 2018;
Revised: By contrast, several works have designed training algorithms to obtain provably robust models (Cohen et al., 2019; Gowal et al., 2019; Raghunathan et al., 2018;

Original: Salman et al., 2019; Wong & Kolter, 2018).
Revised: Salman et al., 2019; Wong & Kolter, 2018).

Original: Unfortunately, these methods either do not generalize to modern network architectures or have a prohibitively large computational complexity.
Revised: Unfortunately, these methods either do not generalize to modern network architectures or have a prohibitively large computational complexity.

Original: As a consequence, adversarial training (Madry et al., 2018) and its variants (Alayrac et al., 2019; Carmon et al., 2019;
Revised: As a consequence, adversarial training (Madry et al., 2018) and its variants (Alayrac et al., 2019; Carmon et al., 2019;

Original: Hendrycks et al., 2019; Kumari et al., 2019; Wu et al., 2020; Zhang et al., 2019a) have become the de facto approach to obtain robust models in practice.
Revised: Hendrycks et al., 2019; Kumari et al., 2019; Wu et al., 2020; Zhang et al., 2019a) have become the de facto approach to obtain robust models in practice.

Original: In essence, these methods generate adversarial examples, usually using PGD, and use them to optimize the model parameters.
Revised: In essence, these methods generate adversarial examples, usually using PGD, and use them to optimize the model parameters.

Original: While effective, adversarial training is more challenging than vanilla training.
Revised: While effective, adversarial training is more challenging than vanilla training.

Original: It was shown to require larger models (Xie & Yuille, 2020) and to exhibit a poor convergence behavior (Liu et al., 2020).
Revised: It was shown to require larger models (Xie & Yuille, 2020) and to exhibit a poor convergence behavior (Liu et al., 2020).

Original: Furthermore, as observed in (Rice et al., 2020), it suffers from adversarial overﬁtting : the robust accuracy on the test set signiﬁcantly decreases in the late adversarial training phase.
Revised: Furthermore, as observed in (Rice et al., 2020), it suffers from adversarial overﬁtting : the robust accuracy on the test set signiﬁcantly decreases in the late adversarial training phase.

Original: (Rice et al., 2020) thus proposed to perform early stopping based on a separate validation set to improve the generalization performance in adversarial training.
Revised: (Rice et al., 2020) thus proposed to perform early stopping based on a separate validation set to improve the generalization performance in adversarial training.

Original: Furthermore, (Chen et al., 2021b) introduced logit smoothing and weight smoothing strategies to reduce adversarial overﬁtting.
Revised: Furthermore, (Chen et al., 2021b) introduced logit smoothing and weight smoothing strategies to reduce adversarial overﬁtting.

Original: In parallel to this, several techniques to improve the model’s robust test accuracy were proposed (Wang et al., 2020; Wu et al., 2020; Zhang et al., 2021), but without solving the adversarial overﬁtting issue.
Revised: In parallel to this, several techniques to improve the model’s robust test accuracy were proposed (Wang et al., 2020; Wu et al., 2020; Zhang et al., 2021), but without solving the adversarial overﬁtting issue.

Original: By contrast, other works (Balaji et al., 2019; Huang et al., 2020) were empirically shown to mitigate adversarial overﬁtting but without providing any explanations as to how this phenomenon was addressed.
Revised: By contrast, other works (Balaji et al., 2019; Huang et al., 2020) were empirically shown to mitigate adversarial overﬁtting but without providing any explanations as to how this phenomenon was addressed.

Original: In this paper, we study the causes of adversarial overﬁtting from both an empirical and a theoretical point of view.
Revised: In this paper, we study the causes of adversarial overﬁtting from both an empirical and a theoretical point of view.

Original: We also identify the reasons why prior attempts (Balaji et al., 2019; Chen et al., 2021a; Huang et al., 2020) successfully mitigate it.
Revised: We also identify the reasons why prior attempts (Balaji et al., 2019; Chen et al., 2021a; Huang et al., 2020) successfully mitigate it.

Original:  A M ETRIC FOR I NSTANCE D IFFICULTY
Revised:  A M ETRIC FOR I NSTANCE D IFFICULTY

Original: Parametric models are trained to minimize a loss objective based on several input-target pairs called training set, and are then evaluated on a held-out set called test set.
Revised: Parametric models are trained to minimize a loss objective based on several input-target pairs called training set, and are then evaluated on a held-out set called test set.

Original: By comparing the loss value of each instance, we can understand which ones, in either the training or the test set, are more difﬁcult for the model to ﬁt.
Revised: By comparing the loss value of each instance, we can understand which ones, in either the training or the test set, are more difﬁcult for the model to ﬁt.

Original: In this section, we introduce a metric for instance difﬁculty, which mainly depends on the data and on the perturbations applied to the instances.
Revised: In this section, we introduce a metric for instance difﬁculty, which mainly depends on the data and on the perturbations applied to the instances.

Original: Let L ( x ) denote the average loss of x ’s corresponding perturbed input across all training epochs.
Revised: Let L ( x ) denote the average loss of x ’s corresponding perturbed input across all training epochs.

Original: We deﬁne the difﬁculty of an instance x within a ﬁnite set D as
Revised: We deﬁne the difﬁculty of an instance x within a ﬁnite set D as

Original: [Equation]
Revised: [Equation]

Original: where (cid:101) x ∼ U
Revised: where (cid:101) x ∼ U

Original: ( D ) indicatesthat (cid:101) x is uniformly sampled from the ﬁnite set D .
Revised: ( D ) indicates (cid:101) x is uniformly sampled from the ﬁnite set D .

Original: d ( x ) is a bounded function, close to 0 for the hardest instances, andclose to 1 for the easiest ones.
Revised: d ( x ) is a bounded function, close to 0 for the hardest instances, and 1 for the easiest ones.

Original: We discuss the  properties of d ( x ) in Appendix D.1, and show that it mainly depends on the data and the perturbation applied to it, but not the model architecture or the training duration.
Revised: We discuss the motivation and properties of d ( x ) in Appendix D.1, and show that it mainly depends on the data and the perturbation applied, the model architecture or the training duration can hardly affect the difﬁculty function.

Original: That is, d ( x ) canbe utilized to represent thenormalized difﬁculty of x within a set under a speciﬁc type of perturbation.
Revised: That is, d ( x ) can represent the difﬁculty of x within a set under a speciﬁc type of perturbation.

Original:  I NSTANCE D IFFICULTY AND A DVERSARIAL O VERFITTING
Revised:  I NSTANCE D IFFICULTY AND A DVERSARIAL O VERFITTING

Original: The model-agnostic difﬁculty metric of Section 3 allows us to select training instances based on their difﬁculty.
Revised: The model-agnostic difﬁculty metric of Section 3 allows us to select training instances based on their difﬁculty.

Original: In Figures 14 and 15 of Appendix D.2, we show some samples of the easiest and the hardest instances of each class in CIFAR10 (Krizhevsky et al., 2009) and SVHN (Netzer et al., 2011), respectively.
Revised: In Figures 20 and 21 of Appendix D.2, we show some samples of the easiest and the hardest instances of each class in CIFAR10 (Krizhevsky et al., 2009) and SVHN (Netzer et al., 2011), respectively.

Original: In both cases, the easiest instances are visually highly similar, whereas the hardest ones are much more diverse, some of them being ambiguous or even incorrectly labeled.
Revised: In both cases, the easiest instances are visually highly similar, whereas the hardest ones are much more diverse, some of them being ambiguous or even incorrectly labeled.

Original: Below, we study how easy and hard instances impact the performance of adversarial training, with a focus on the adversarial overﬁtting phenomenon.
Revised: Below, we study how easy and hard instances impact the performance of adversarial training, with a focus on the adversarial overﬁtting phenomenon.

Original: The detailed experimental settings are deferred to Appendix C.1.
Revised: The detailed experimental settings are deferred to Appendix C.1.

Original: 4. U SING A S UBSET OF T RAINING D ATA
Revised: 4. U SING A S UBSET OF T RAINING D ATA

Original: We start by training RN18 models for 200 epochs using either the 10000 easiest, random or hardest instances of the CIFAR10 training set via either vanilla training, FGSM adversarial training or PGD adversarial training.
Revised: We start by training RN18 models for 200 epochs using either the 10000 easiest, random or hardest instances of the CIFAR10 training set via either vanilla training, FGSM adversarial training or PGD adversarial training.

Original: The adversarial budget is based on the l ∞ norm and (cid:15) = 8 / 255 .
Revised: The adversarial budget is based on the l ∞ norm and (cid:15) = 8 / 255 .

Original: Note that the instance’s difﬁculty is deﬁned under the corresponding perturbation type, and we enforce these subsets to be class-balanced.
Revised: Note that the instance’s difﬁculty is deﬁned under the corresponding perturbation type, and we enforce these subsets to be class-balanced.

Original: For example, the easiest 10000 instances consist of the easiestinstances in each class.
Revised: For example, the easiest 10000 instances consist of the easiestinstances in each class.

Original: We provide the learning curves under different settings in Figure 1.
Revised: We provide the learning curves under different settings in Figure 1.

Original: [Figure]
Revised: [Figure]

Original: For PGD adversarial training, in Figure 1(a), while we observe adversarial overﬁtting when using the random instances, as in (Rice et al., 2020), no such phenomenon occurs when using the easiest instances: the performance on the test set does not degrade during training.
Revised: For PGD adversarial training, in Figure 1(a), while we observe adversarial overﬁtting when using the random instances, as in (Rice et al., 2020), no such phenomenon occurs when using the easiest instances: the performance on the test set does not degrade during training.

Original: However, PGD adversarial training fails and suffers more severe overﬁtting when using the hardest instances.
Revised: However, PGD adversarial training fails and suffers more severe overﬁtting when using the hardest instances.

Original: Note that, in
Revised: Note that, in

Original: Figure 3 (Appendix D.3), we show that this failure is not due to improper optimization.
Revised: Figure 5 (Appendix D.3), we show that this failure is not due to improper optimization.

Original: In contrast to PGD, FGSM adversarial training and vanilla training (Figure 1(b), 1(c)), through which the model does not achieve true robustness (Madry et al., 2018), do not suffer from severe adversarial overﬁtting.
Revised: In contrast to PGD, FGSM adversarial training and vanilla training (Figure 1(b), 1(c)), through which the model does not achieve true robustness (Madry et al., 2018), do not suffer from severe adversarial overﬁtting.

Original: In these cases, the models trained with the hardest instances also achieve non-trivial test accuracy.
Revised: In these cases, the models trained with the hardest instances also achieve non-trivial test accuracy.

Original: Furthermore, the gaps in robust test accuracy between the models trained by easy instances and by hard ones are much smaller.
Revised: Furthermore, the gaps in robust test accuracy between the models trained by easy instances and by hard ones are much smaller.

Original: In Appendix D.3, we perform additional and comprehensive experiments, evidencing that our conclusions hold for different datasets and values of (cid:15) , and for an adversarial budget based on the l 2 norm.
Revised: In Appendix D.3, we perform additional and comprehensive experiments, evidencing that our conclusions hold for different datasets and values of (cid:15) , and for an adversarial budget based on the l 2 norm.

Original: We show that more severe adversarial overﬁtting happens when the size of the adversarial budget (cid:15) increases.
Revised: We show that more severe adversarial overﬁtting happens when the size of the adversarial budget (cid:15) increases.

Original: Furthermore, we experiment with training models using increasingly many training instances, start with the easiest ones.
Revised: Furthermore, we experiment with training models using increasingly many training instances, start with the easiest ones.

Original: Our results in Figure 7 show that the models can beneﬁt from using more data, but only using early stopping as done in (Rice et al., 2020).
Revised: Our results in Figure 10 show that the models can beneﬁt from using more data, but only using early stopping as done in (Rice et al., 2020).

Original: This indicates that the hard instances can still beneﬁt adversarial training, but need to be utilized in an adaptive manner.
Revised: This indicates that the hard instances can still beneﬁt adversarial training, but need to be utilized in an adaptive manner.

Original: 4. H ARD I NSTANCES L EAD TO O VERFITTING
Revised: 4. H ARD I NSTANCES L EAD TO O VERFITTING

Original: Let us now turn to the more standard setting where we train the model with the entire training set.
Revised: Let us now turn to the more standard setting where we train the model with the entire training set.

Original: To nonetheless analyze the inﬂuence of instance difﬁculty in this scenario, we divide the training set
Revised: To nonetheless analyze the inﬂuence of instance difﬁculty in this scenario, we divide the training set

Original: D into 10 non-overlapping groups {G i } 9 i =0 , with G i = { x ∈ D| 0 .
Revised: D into 10 non-overlapping groups {G i } 9 i =0 , with G i = { x ∈ D| 0 .

Original: 1 × i ≤ d ( x ) < 0 .
Revised: 1 × i ≤ d ( x ) < 0 .

Original: 1 × ( i + 1) } .
Revised: 1 × ( i + 1) } .

Original: That is, G 0 is the hardest group, whereas G 9 is the easiest one.
Revised: That is, G 0 is the hardest group, whereas G 9 is the easiest one.

Original: We then train an RN18 model on the entire CIFAR10 training set using PGD adversarial training and monitor the training behavior of the different groups.
Revised: We then train an RN18 model on the entire CIFAR10 training set using PGD adversarial training and monitor the training behavior of the different groups.

Original: In particular, in Figure 2(a), we plot the average loss of the instances in the groups
Revised: In particular, in Figure 2(a), we plot the average loss of the instances in the groups

Original: G 0 , G 3 , G 6 and G 9 .
Revised: G 0 , G 3 , G 6 and G 9 .

Original: The resulting curves show that, in the early training stages, the model ﬁrst ﬁts the easy instances, as evidenced by the average loss of group G 9 decreasing much faster than that of the other groups.
Revised: The resulting curves show that, in the early training stages, the model ﬁrst ﬁts the easy instances, as evidenced by the average loss of group G 9 decreasing much faster than that of the other groups.

Original: By contrast, in the late training phase, the model tries to ﬁt the more difﬁcult instances, with the average loss of groups G 0 and G 3 decreasing much faster than that of the other groups.
Revised: By contrast, in the late training phase, the model tries to ﬁt the more difﬁcult instances, with the average loss of groups G 0 and G 3 decreasing much faster than that of the other groups.

Original: In this period, however, the robust test error (solid grey line) increases, which indicates that adversarial overﬁtting arises from the model’s attempt to ﬁt the hard adversarial instances.
Revised: In this period, however, the robust test error (solid grey line) increases, which indicates that adversarial overﬁtting arises from the model’s attempt to ﬁt the hard adversarial instances.

Original: In addition to average losses, inspired by (Ilyas et al., 2019), which showed that the penultimate layer’s activations of a robust model correspond to its robust features that cannot be misaligned by adversarial attacks, we monitor the group-wise average magnitudes of the penultimate layer’s activations.
Revised: In addition to average losses, inspired by (Ilyas et al., 2019), which showed that the penultimate layer’s activations of a robust model correspond to its robust features that cannot be misaligned by adversarial attacks, we monitor the group-wise average magnitudes of the penultimate layer’s activations.

Original: As shown in Figure 2(b), the model ﬁrst focuses on extracting robust features for the easy instances, as evidenced by the comparatively large activations of the instances in G 9 .
Revised: As shown in Figure 2(b), the model ﬁrst focuses on extracting robust features for the easy instances, as evidenced by the comparatively large activations of the instances in G 9 .

Original: In the late phase of training, the norm of the activations of the hard instances increases signiﬁcantly, bridging the gap between easy and hard instances.
Revised: In the late phase of training, the norm of the activations of the hard instances increases signiﬁcantly, bridging the gap between easy and hard instances.

Original: This further indicates that the model focuses more on the hard instances in the later training phase, at which point it starts overﬁtting.
Revised: This further indicates that the model focuses more on the hard instances in the later training phase, at which point it starts overﬁtting.

Original: [Figure]
Revised: [Figure]

Original:  T HEORETICAL A NALYSIS
Revised:  T HEORETICAL A NALYSIS

Original: We now study the relationship between adversarial overﬁtting and instance difﬁculty from a theoretical viewpoint.
Revised: We now study the relationship between adversarial overﬁtting and instance difﬁculty from a theoretical viewpoint.

Original: We start with the linear model, where the loss function has an analytical expression
Revised: We start with the linear model, where the loss function has an analytical expression

Original: [Figure]
Revised: [Figure]

Original: and then generalize our analysis to the nonlinear cases.
Revised: and then generalize our analysis to the nonlinear cases.

Original: We use { x i , y i } ni =1 to represent the training data, and ( X , y ) as its matrix form.
Revised: We use { x i , y i } ni =1 to represent the training data, and ( X , y ) as its matrix form.

Original: { x (cid:48)
Revised: { x (cid:48)

Original: i , y i } ni =1 and ( X (cid:48) , y ) are their adversarial counterparts.
Revised: i , y i } ni =1 and ( X (cid:48) , y ) are their adversarial counterparts.

Original: Here, x i ∈ R m , y i ∈ {− 1 , +1 } , X ∈ R n × m and y ∈ {− 1 , +1 } n .
Revised: Here, x i ∈ R m , y i ∈ {− 1 , +1 } , X ∈ R n × m and y ∈ {− 1 , +1 } n .

Original: The notation used for our theoretical analysis is summarized in Table 3 of Appendix A.
Revised: The notation used for our theoretical analysis is summarized in Table 3 of Appendix A.

Original: 5. L OGISTIC R EGRESSION
Revised: 5. L OGISTIC R EGRESSION

Original: We study the logistic regression model under an l 2 norm based adversarial budget.
Revised: We study the logistic regression model under an l 2 norm based adversarial budget.

Original: In this case, the model is parameterized by w ∈ R m and outputs sign ( w T x (cid:48) i ) given the adversarial example x (cid:48)
Revised: In this case, the model is parameterized by w ∈ R m and outputs sign ( w T x (cid:48) i ) given the adversarial example x (cid:48)

Original: i of the input x i .
Revised: i of the input x i .

Original: The loss function for this instance is 1+ e yi w T x (cid:48)
Revised: The loss function for this instance is 1+ e yi w T x (cid:48)

Original: i .
Revised: i .

Original: We assume over-parameterization, which means n < m .
Revised: We assume over-parameterization, which means n < m .

Original: The following theorem shows that, under mild assumptions, the parameters of the adversarially trained logistic regression model converge to the l 2 max-margin direction of the training data.
Revised: The following theorem shows that, under mild assumptions, the parameters of the adversarially trained logistic regression model converge to the l 2 max-margin direction of the training data.

Original: Theorem 1.
Revised: Theorem 1.

Original: For a dataset { x i , y i } ni =1 that is linearly separable under the adversarial budget S (2) ( (cid:15) ) , any initial point w 0 and step size α ≤ 2 (cid:107) X (cid:107) − 2 , the gradient descent w u
Revised: For a dataset { x i , y i } ni =1 that is linearly separable under the adversarial budget S (2) ( (cid:15) ) , any initial point w 0 and step size α ≤ 2 (cid:107) X (cid:107) − 2 , the gradient descent w u

Original: +1 =
Revised: +1 =

Original: w u − α (cid:79) w L w u ( X (cid:48) ) converges asymptotically to the l 2 max-margin vector of the training data.
Revised: w u − α (cid:79) w L w u ( X (cid:48) ) converges asymptotically to the l 2 max-margin vector of the training data.

Original: That is, lim 
Revised: That is, lim u →∞

Original: [Equation]
Revised: [Equation]

Revised: w

Revised: [Equation]

Revised: , where (cid:98) w = arg min w

Original: [Equation]
Revised: [Equation]

Original: The proof is deferred to Appendix B.1.
Revised: The proof is deferred to Appendix B.1.

Original: Theorem 1 extends the conclusion in (Soudry et al., 2018), which only studies the non-adversarial case.
Revised: Theorem 1 extends the conclusion in (Soudry et al., 2018), which only studies the non-adversarial case.

Original: It also indicates that the optimal parameters are only determined by the support vectors of the training data, which are the ones with the smallest margin.
Revised: It also indicates that the optimal parameters are only determined by the support vectors of the training data, which are the ones with the smallest margin.

Original: According to the loss function, the smallest margin means the largest loss values and thus the hardest training instances based on our deﬁnition in Section 3.
Revised: According to the loss function, the smallest margin means the largest loss values and thus the hardest training instances based on our deﬁnition in Section 3.

Original: To further study how the training instances’ difﬁculty inﬂuences the model’s generalization performance, we assume that the data points are drawn from a K -mode Gaussian mixture model (GMM).
Revised: To further study how the training instances’ difﬁculty inﬂuences the model’s generalization performance, we assume that the data points are drawn from a K -mode Gaussian mixture model (GMM).

Original: Speciﬁcally, the k -th component has a probability p k of being sampled and is formulated as follows:
Revised: Speciﬁcally, the k -th component has a probability p k of being sampled and is formulated as follows:

Original: [Equation]
Revised: [Equation]

Original: Here, η ∈ R m is the unit vector indicating the mean of the positive instances, and r k ∈ R + controls the average distance between the positive and negative instances.
Revised: Here, η ∈ R m is the unit vector indicating the mean of the positive instances, and r k ∈ R + controls the average distance between the positive and negative instances.

Original: The mean values of all modes in this GMM are colinear, so r k indicates the difﬁculty of instances sampled from the k -th component.
Revised: The mean values of all modes in this GMM are colinear, so r k indicates the difﬁculty of instances sampled from the k -th component.

Original: Without the loss of generality, we assume r 1 < r 2 < ... <
Revised: Without the loss of generality, we assume r 1 < r 2 < ... <

Original: r K − 1 < r K .
Revised: r K − 1 < r K .

Original: As in Section 4, we consider models trained with the subsets of the training data, e.g., n instances from the l -th component.
Revised: As in Section 4, we consider models trained with the subsets of the training data, e.g., n instances from the l -th component.

Original: l =then indicates training on the hardest examples, while l = K means using the easiest.
Revised: l =then indicates training on the hardest examples, while l = K means using the easiest.

Original: In matrix form, we have X =
Revised: In matrix form, we have X =

Original: r l yη T + Q for the instances sampled from the l -th component, where the rows of noise matrix Q are sampled from N ( 0 , I ) .
Revised: r l yη T + Q for the instances sampled from the l -th component, where the rows of noise matrix Q are sampled from N ( 0 , I ) .

Original: Although the max-margin direction in Equation (2), where the parameters converge based on Theorem 1, does not have an analytical expression, the results in (Wang & Thrampoulidis, 2020) indicate that, in the over-parameterization regime and when the training data is sampled from a GMM, the max-margin direction is the min-norm interpolation of the data with high probability.
Revised: Although the max-margin direction in Equation (2), where the parameters converge based on Theorem 1, does not have an analytical expression, the results in (Wang & Thrampoulidis, 2020) indicate that, in the over-parameterization regime and when the training data is sampled from a GMM, the max-margin direction is the min-norm interpolation of the data with high probability.

Original: Since the latter has an analytical form given by X T ( XX T ) − 1 y , we can then calculate the exact generalization performance of the trained model as stated in the following theorem.
Revised: Since the latter has an analytical form given by X T ( XX T ) − 1 y , we can then calculate the exact generalization performance of the trained model as stated in the following theorem.

Original: Theorem 2.
Revised: Theorem 2.

Original: If a logistic regression model is adversarially trained on n separable training instances sampled from the l -th component of the GMM model described in (3).
Revised: If a logistic regression model is adversarially trained on n separable training instances sampled from the l -th component of the GMM model described in (3).

Original: If m n log n is sufﬁciently large 2 , then with probability 1 − O ( 1 n ) , the expected adversarial test error R under the adversarial budget
Revised: If m n log n is sufﬁciently large 2 , then with probability 1 − O ( 1 n ) , the expected adversarial test error R under the adversarial budget

Original: S (2) ( (cid:15) ) , which is a function of r l and (cid:15) , on the whole GMM model described in (3) is given by
Revised: S (2) ( (cid:15) ) , which is a function of r l and (cid:15) , on the whole GMM model described in (3) is given by

Original: [Equation]
Revised: [Equation]

Original: C 1 , C 2 are independent of (cid:15) and r l .
Revised: C 1 , C 2 are independent of (cid:15) and r l .

Original: The function Φ is deﬁned as Φ( x ) = P ( Z > x ) , Z ∼ N (0 , 1) .
Revised: The function Φ is deﬁned as Φ( x ) = P ( Z > x ) , Z ∼ N (0 , 1) .

Original: We defer the proof of Theorem 2 to Appendix B.2, in which we calculate the exact expression of
Revised: We defer the proof of Theorem 2 to Appendix B.2, in which we calculate the exact expression of

Original: R ( r l , (cid:15) ) , C 1 , C 2 , and show that C 1 , C 2 are positive numbers almost surely.
Revised: R ( r l , (cid:15) ) , C 1 , C 2 , and show that C 1 , C 2 are positive numbers almost surely.

Original: Since C 1 and C 2 are independent of r l , and Φ( x ) is a monotonically decreasing function, we conclude that the robust test error R ( r l , (cid:15) ) becomes smaller when r l increases.
Revised: Since C 1 and C 2 are independent of r l , and Φ( x ) is a monotonically decreasing function, we conclude that the robust test error R ( r l , (cid:15) ) becomes smaller when r l increases.

Original: That is, when the training instances become easier, the corresponding generalization error under the adversarial attack becomes smaller.
Revised: That is, when the training instances become easier, the corresponding generalization error under the adversarial attack becomes smaller.

Original: Theorem 2 holds for all (cid:15) only if the training data is separable under the corresponding adversarial budget.
Revised: Theorem 2 holds for all (cid:15) only if the training data is separable under the corresponding adversarial budget.

Original: The following corollary shows that the difference in the robust test error between models trained with easy instances and the ones with hard ones increases when (cid:15) becomes larger.
Revised: The following corollary shows that the difference in the robust test error between models trained with easy instances and the ones with hard ones increases when (cid:15) becomes larger.

Original: Corollary 1.
Revised: Corollary 1.

Original: Under the conditions of Theorem 2 and the deﬁnition of R in Equation (4), if (cid:15) 1 < (cid:15) 2 , then we have ∀ 0 ≤ i < j ≤ K, R ( r i , (cid:15) 1 ) − R ( r j , (cid:15) 1 ) <
Revised: Under the conditions of Theorem 2 and the deﬁnition of R in Equation (4), if (cid:15) 1 < (cid:15) 2 , then we have ∀ 0 ≤ i < j ≤ K, R ( r i , (cid:15) 1 ) − R ( r j , (cid:15) 1 ) <

Original: R ( r i , (cid:15) 2 ) − R ( r j , (cid:15) 2 ) .
Revised: R ( r i , (cid:15) 2 ) − R ( r j , (cid:15) 2 ) .

Original: The proof is in Appendix B.3.
Revised: The proof is in Appendix B.3.

Original: This indicates that, compared with training on clean inputs, i.e., (cid:15) = 0 , the generalization performance of adversarial training with (cid:15) > 0 is more sensitive to the difﬁculty of the training instances.
Revised: This indicates that, compared with training on clean inputs, i.e., (cid:15) = 0 , the generalization performance of adversarial training with (cid:15) > 0 is more sensitive to the difﬁculty of the training instances.

Original: This is consistent with our empirical observations in Figure 1.
Revised: This is consistent with our empirical observations in Figure 1.

Original: 5. G ENERAL N ONLINEAR M ODELS
Revised: 5. G ENERAL N ONLINEAR M ODELS

Original: In this section, we study the binary classiﬁcation problem using a general nonlinear model.
Revised: In this section, we study the binary classiﬁcation problem using a general nonlinear model.

Original: We consider a model with b parameters, i.e., w ∈ R b .
Revised: We consider a model with b parameters, i.e., w ∈ R b .

Original: Without loss of generality, we assume the output of the function f w to lie in [ − 1 , +1] .
Revised: Without loss of generality, we assume the output of the function f w to lie in [ − 1 , +1] .

Original: Furthermore, we assume isoperimetry of the data distribution:
Revised: Furthermore, we assume isoperimetry of the data distribution:

Original: Assumption 1.
Revised: Assumption 1.

Original: The data distribution µ is a composition of K c -isoperimetric distributions on
Revised: The data distribution µ is a composition of K c -isoperimetric distributions on

Original: R m , each of which has a positive conditional variance.
Revised: R m , each of which has a positive conditional variance.

Original: That is, µ = (cid:80) Kk =

Revised: That is, µ = (cid:80) Kk =

Original: α k µ k , where α k >and (cid:80) Kk
Revised: α k µ k , where α k >and (cid:80) Kk

Original: =

Revised: =

Original: α k = 1 .
Revised: α k = 1 .

Original: We deﬁne σ 2 k = E µ k
Revised: We deﬁne σ 2 k = E µ k

Original: [ V ar [ y | x ]] , and without loss of generality assume that σ 1 ≥ σ 2 ≥ ... ≥ σ K > 0 .
Revised: [ V ar [ y | x ]] , and without loss of generality assume that σ 1 ≥ σ 2 ≥ ... ≥ σ K > 0 .

Original: Furthermore, given any L -Lipschitz function f w , i.e., ∀ x 1 , x 2 , (cid:107) f w ( x 1 ) − f w ( x 2 )
Revised: Furthermore, given any L -Lipschitz function f w , i.e., ∀ x 1 , x 2 , (cid:107) f w ( x 1 ) − f w ( x 2 )

Original: [Equation]
Revised: [Equation]

Original: ≤ L (cid:107) x
Revised: ≤ L (cid:107) x

Original: 1 − x 2 (cid:107) , we have
Revised: 1 − x 2 (cid:107) , we have

Original: [Equation]
Revised: [Equation]

Original: This is a benign assumption; the data distribution is a mixture of K components and each of them contains samples from a sub-Gaussian distribution.
Revised: This is a benign assumption; the data distribution is a mixture of K components and each of them contains samples from a sub-Gaussian distribution.

Original: These components correspond to training instances of different difﬁculty levels measured by the conditional variance.
Revised: These components correspond to training instances of different difﬁculty levels measured by the conditional variance.

Original: We then study the property of the model f w under adversarial attacks.
Revised: We then study the property of the model f w under adversarial attacks.

Original: Deﬁnition 1.
Revised: Deﬁnition 1.

Original: Given the dataset { x i , y i } ni =1 , the model f w , the adversarial budget S ( p ) ( (cid:15) ) and a positive constant C , we deﬁne the function h ( C, (cid:15) ) as
Revised: Given the dataset { x i , y i } ni =1 , the model f w , the adversarial budget S ( p ) ( (cid:15) ) and a positive constant C , we deﬁne the function h ( C, (cid:15) ) as

Original: [Equation]
Revised: [Equation]

Original: where h i, w ( (cid:15) )
Revised: where h i, w ( (cid:15) )

Original: = max ζ, s.t.
Revised: = max ζ, s.t.

Original: [ f w ( x i
Revised: [ f w ( x i

Original: [Equation]
Revised: [Equation]

Original: − ζ
Revised: − ζ

Original: [Equation]
Revised: [Equation]

Original: x
Revised: x

Original: [Equation]
Revised: [Equation]

Original: Here, x (cid:48)
Revised: Here, x (cid:48)

Original: i is the adversarial example of x .
Revised: i is the adversarial example of x .

Original: We omit the superscript ( p ) for notation simplicity.
Revised: We omit the superscript ( p ) for notation simplicity.

Original: By deﬁnition, h i, w ( (cid:15) ) ≥ 0 depicts the bandwidth ζ of the model’s output range in the domain of the adversarial budget on a training instance.
Revised: By deﬁnition, h i, w ( (cid:15) ) ≥ 0 depicts the bandwidth ζ of the model’s output range in the domain of the adversarial budget on a training instance.

Original: h ( C, (cid:15) ) is the minimum bandwidth among the models whose mean squared error on the adversarial training set is smaller than C .
Revised: h ( C, (cid:15) ) is the minimum bandwidth among the models whose mean squared error on the adversarial training set is smaller than C .

Original: Based on the deﬁnitions of T and h i, w , and for a ﬁxed value of C , we have ∀ (cid:15) 1 < (cid:15) 2 , h i, w ( (cid:15) 1 ) ≤ h i, w ( (cid:15) 2 ) and
Revised: Based on the deﬁnitions of T and h i, w , and for a ﬁxed value of C , we have ∀ (cid:15) 1 < (cid:15) 2 , h i, w ( (cid:15) 1 ) ≤ h i, w ( (cid:15) 2 ) and

Original: T ( C, (cid:15) 2 ) ⊂ T ( C, (cid:15) 1 ) .
Revised: T ( C, (cid:15) 2 ) ⊂ T ( C, (cid:15) 1 ) .

Original: As a result, ∀ (cid:15) 1 < (cid:15) 2 , h ( C, (cid:15) 1 ) ≤ h ( C, (cid:15) 2 ) .
Revised: As a result, ∀ (cid:15) 1 < (cid:15) 2 , h ( C, (cid:15) 1 ) ≤ h ( C, (cid:15) 2 ) .

Original: In addition, since ∀ C 1 < C 2 ,
Revised: In addition, since ∀ C 1 < C 2 ,

Original: T ( C 1 , (cid:15) )
Revised: T ( C 1 , (cid:15) )

Original: ⊂ T ( C 2 , (cid:15) ) for a ﬁxed value of (cid:15) , we have ∀ C 1 < C 2 , h
Revised: ⊂ T ( C 2 , (cid:15) ) for a ﬁxed value of (cid:15) , we have ∀ C 1 < C 2 , h

Original: ( C 1 , (cid:15) )
Revised: ( C 1 , (cid:15) )

Original: ≥ h ( C 2 , (cid:15) ) .
Revised: ≥ h ( C 2 , (cid:15) ) .

Original: That is to say, h ( C, (cid:15) ) is a monotonically non-decreasing function on (cid:15) and a monotonically non-increasing function on C .
Revised: That is to say, h ( C, (cid:15) ) is a monotonically non-decreasing function on (cid:15) and a monotonically non-increasing function on C .

Original: In practice, when f w represents a deep neural network, h ( C, (cid:15) ) increases with (cid:15) almost surely, because the attack algorithm usually generates adversarial examples at the boundary of the adversarial budget.
Revised: In practice, when f w represents a deep neural network, h ( C, (cid:15) ) increases with (cid:15) almost surely, because the attack algorithm usually generates adversarial examples at the boundary of the adversarial budget.

Original: We then state our main theorem below.
Revised: We then state our main theorem below.

Original: Theorem 3.
Revised: Theorem 3.

Original: Given n training pairs { x i , y i } ni =1 sampled from the l -th component µ l of the distribution in Assumption 1, the parametric model f w , the adversarial budget S ( p ) ( (cid:15) ) and the corresponding function h deﬁned in Deﬁnition 1, we assume that the model f w is in the function space F = { f w , w ∈ W} with W ⊂ R b having a ﬁnite diameter diam ( W ) ≤ W and, ∀ w 1 , w 2 ∈ W , (cid:107) f
Revised: Given n training pairs { x i , y i } ni =1 sampled from the l -th component µ l of the distribution in Assumption 1, the parametric model f w , the adversarial budget S ( p ) ( (cid:15) ) and the corresponding function h deﬁned in Deﬁnition 1, we assume that the model f w is in the function space F = { f w , w ∈ W} with W ⊂ R b having a ﬁnite diameter diam ( W ) ≤ W and, ∀ w 1 , w 2 ∈ W , (cid:107) f

Original: w 1 − f w 2 (cid:107) ∞ ≤ J (cid:107) w 1 − w 2 (cid:107) ∞ .
Revised: w 1 − f w 2 (cid:107) ∞ ≤ J (cid:107) w 1 − w 2 (cid:107) ∞ .

Original: We train the model f
Revised: We train the model f

Original: w adversarially using these n data points.
Revised: w adversarially using these n data points.

Original: Let x (cid:48) be the adversarial example of the data point x and δ ∈ (0 , 1) .
Revised: Let x (cid:48) be the adversarial example of the data point x and δ ∈ (0 , 1) .

Original: If we have 1 n (cid:80) ni =1 ( f w ( x (cid:48) i ) − y i ) 2 = C and γ :
Revised: If we have 1 n (cid:80) ni =1 ( f w ( x (cid:48) i ) − y i ) 2 = C and γ :

Original: = σ 2 l + h 2 ( C, (cid:15) ) − C ≥ 0 , then with probability at least 1 − δ , the Lipschitz constant of f w is lower bounded as
Revised: = σ 2 l + h 2 ( C, (cid:15) ) − C ≥ 0 , then with probability at least 1 − δ , the Lipschitz constant of f w is lower bounded as

Original: [Equation]
Revised: [Equation]

Original: where Lip ( f w ) is the Lipschitz constant of f w : ∀ x 1 , x 2 , (cid:107) f w ( x 1 ) − f w ( x 2 ) (cid:107) ≤ Lip ( f w ) (cid:107) x 1 − x 2 (cid:107) .
Revised: where Lip ( f w ) is the Lipschitz constant of f w : ∀ x 1 , x 2 , (cid:107) f w ( x 1 ) − f w ( x 2 ) (cid:107) ≤ Lip ( f w ) (cid:107) x 1 − x 2 (cid:107) .

Original: The proof is deferred to Appendix B.4.
Revised: The proof is deferred to Appendix B.4.

Original: Theorem 3 extends the results in (Bubeck & Sellke, 2021) to the case of adversarial training.
Revised: Theorem 3 extends the results in (Bubeck & Sellke, 2021) to the case of adversarial training.

Original: Note that modern deep neural network models typically have millions of parameters, so b (cid:29) max { c, m, n } .
Revised: Note that modern deep neural network models typically have millions of parameters, so b (cid:29) max { c, m, n } .

Original: In this case, we can approximate the lower bound (7) by Lip ( f w ) (cid:38) γ
Revised: In this case, we can approximate the lower bound (7) by Lip ( f w ) (cid:38) γ

Original: [Equation]
Revised: [Equation]

Original: 7 (cid:113)
Revised: 7 (cid:113)

Original: [Equation]
Revised: [Equation]

Original: 1 ) , and the right hand side increases with γ .
Revised: 1 ) , and the right hand side increases with γ .

Original: Since γ :
Revised: Since γ :

Original: = σ 2 l + h 2 ( C, (cid:15) ) − C , the lower bound increases with both σ l and (cid:15) but decreases as C increases.
Revised: = σ 2 l + h 2 ( C, (cid:15) ) − C , the lower bound increases with both σ l and (cid:15) but decreases as C increases.

Original: The Lipschitz constant is widely used to bound a model’s adversarial vulnerability (Ruan et al., 2018;
Revised: The Lipschitz constant is widely used to bound a model’s adversarial vulnerability (Ruan et al., 2018;

Original: Weng et al., 2018a;b): larger Lipschitz constants indicate higher adversarial vulnerability.
Revised: Weng et al., 2018a;b): larger Lipschitz constants indicate higher adversarial vulnerability.

Original: Recall that γ needs to be non-negative, so C is upper bounded, which means that the model is well ﬁt to the adversarial training set and the adversarial vulnerability is approximately the generalization gap.
Revised: Recall that γ needs to be non-negative, so C is upper bounded, which means that the model is well ﬁt to the adversarial training set and the adversarial vulnerability is approximately the generalization gap.

Original: Based on this, the adversarial vulnerability of a model increases with the size of the adversarial budget and the difﬁculty level of the training instances; it also increases as the training mean squared error decreases.
Revised: Based on this, the adversarial vulnerability of a model increases with the size of the adversarial budget and the difﬁculty level of the training instances; it also increases as the training mean squared error decreases.

Original: That is, under the same adversarial budget, the adversarial vulnerability increases with the instances’ difﬁculty, measured by σ l in our theorem; using the same training instances, the adversarial vulnerability increases with the adversarial budget measured by (cid:15) .
Revised: That is, under the same adversarial budget, the adversarial vulnerability increases with the instances’ difﬁculty, measured by σ l in our theorem; using the same training instances, the adversarial vulnerability increases with the adversarial budget measured by (cid:15) .

Original: In addition, as adversarial training progresses, the mean squared error C on the adversarial training instances becomes smaller, which makes the lower bound of the Lipschitz constant larger.
Revised: In addition, as adversarial training progresses, the mean squared error C on the adversarial training instances becomes smaller, which makes the lower bound of the Lipschitz constant larger.

Original: This indicates that adversarial vulnerability becomes larger in the later phase of adversarial training.
Revised: This indicates that adversarial vulnerability becomes larger in the later phase of adversarial training.

Original: We provide empirical evidence to conﬁrm the conclusions of Theorem 3 in Appendix D.4.
Revised: We provide empirical evidence to conﬁrm the conclusions of Theorem 3 in Appendix D.4.

Original: Since calculating the Lipschitz constant of a deep neural network is NP-hard (Scaman & Virmaux, 2018), exactly calculating the Lipschitz constant (Jordan & Dimakis, 2020) can only be achieved for simple multi-layer perceptron (MLP) models, not for modern deep networks.
Revised: Since calculating the Lipschitz constant of a deep neural network is NP-hard (Scaman & Virmaux, 2018), exactly calculating the Lipschitz constant (Jordan & Dimakis, 2020) can only be achieved for simple multi-layer perceptron (MLP) models, not for modern deep networks.

Original: Instead, we estimate the upper bound of the Lipschitz constant numerically, as in (Scaman & Virmaux, 2018).
Revised: Instead, we estimate the upper bound of the Lipschitz constant numerically, as in (Scaman & Virmaux, 2018).

Original:  M ITIGATING A DVERSARIAL O VERFITTING
Revised:  M ITIGATING A DVERSARIAL O VERFITTING

Original: Existing methods mitigating adversarial overﬁtting study the standard adversarial training scenario:
Revised: Existing methods mitigating adversarial overﬁtting study the standard adversarial training scenario:

Original: PGD adversarial training from scratch.
Revised: PGD adversarial training from scratch.

Original: In Appendix D.5, we show that they use either adaptive inputs (Balaji et al., 2019) or adaptive targets (Huang et al., 2020; Chen et al., 2021b) to implicitly avoid ﬁtting hard input-target pairs, thus providing an explanation for their success.
Revised: In Appendix D.5, we show that they use either adaptive inputs (Balaji et al., 2019) or adaptive targets (Huang et al., 2020; Chen et al., 2021b) to implicitly avoid ﬁtting hard input-target pairs, thus providing an explanation for their success.

Original: Below, we study two settings other than standard adversarial training: fast adversarial training and adversarial ﬁne-tuning with additional data, to validate our ﬁndings.
Revised: Below, we study two settings other than standard adversarial training: fast adversarial training and adversarial ﬁne-tuning with additional data, to validate our ﬁndings.

Original: We show that methods inspired by our ﬁndings can avoid adversarial overﬁtting and improve the performance.
Revised: We show that methods inspired by our ﬁndings can avoid adversarial overﬁtting and improve the performance.

Original: The detailed experimental settings for this section are in Appendix C.2.
Revised: The detailed experimental settings for this section are in Appendix C.2.

Original: 6. F AST A DVERSARIAL T RAINING
Revised: 6. F AST A DVERSARIAL T RAINING

Original: Adversarial training in (Madry et al., 2018) introduces a signiﬁcant computational overhead.
Revised: Adversarial training in (Madry et al., 2018) introduces a signiﬁcant computational overhead.

Original: Thus it is desirable to accelerate this method.
Revised: Thus it is desirable to accelerate this method.

Original: Our experiments in this section are based on adversarial training with transferable adversarial examples (ATTA in (Zheng et al., 2020)), which stores the adversarial perturbation for each training instance as an initial point for the next epoch.
Revised: Our experiments in this section are based on adversarial training with transferable adversarial examples (ATTA in (Zheng et al., 2020)), which stores the adversarial perturbation for each training instance as an initial point for the next epoch.

Original: We show that adaptively utilizing the easy and hard training instances not only mitigates adversarial overﬁtting, but also signiﬁcantly improves the performance of the ﬁnal model.
Revised: We show that adaptively utilizing the easy and hard training instances not only mitigates adversarial overﬁtting, but also signiﬁcantly improves the performance of the ﬁnal model.

Original: First, we introduce a reweighting scheme to assign lower weights to hard instances when calculating the loss objective.
Revised: First, we introduce a reweighting scheme to assign lower weights to hard instances when calculating the loss objective.

Original: Speciﬁcally, each training instance is assigned a weight equal to the adversarial output probability of the true label.
Revised: Speciﬁcally, each training instance is assigned a weight equal to the adversarial output probability of the true label.

Original: Then this weight is normalized to ensure that the weights in a mini-batch sum to 1 .
Revised: Then this weight is normalized to ensure that the weights in a mini-batch sum to 1 .

Original: Note that our reweighting scheme is based on the adversarial output instead of the clean output, because the adversarial output probability will also be used to calculate the loss objective.
Revised: Note that our reweighting scheme is based on the adversarial output instead of the clean output, because the adversarial output probability will also be used to calculate the loss objective.

Original: As a result, the computational overhead of the reweighting scheme is negligible.
Revised: As a result, the computational overhead of the reweighting scheme is negligible.

Original: In addition to reweighting, we also use adaptive targets to improve the performance.
Revised: In addition to reweighting, we also use adaptive targets to improve the performance.

Original: For each training instance ( x , y ) , we maintain an adaptive moving average target (cid:101) t .
Revised: For each training instance ( x , y ) , we maintain an adaptive moving average target (cid:101) t .

Original: (cid:101) t is updated in an exponential average manner in each epoch (cid:101) t ← ρ (cid:101) t + (1 − ρ ) o (cid:48) where ρ is the momentum factor.
Revised: (cid:101) t is updated in an exponential average manner in each epoch (cid:101) t ← ρ (cid:101) t + (1 − ρ ) o (cid:48) where ρ is the momentum factor.

Original: This is similar to the target in (Huang et al., 2020), but, similarly to the reweighting scheme, we use the adversarial output o (cid:48) instead of the clean output o to avoid an increase in computational complexity.
Revised: This is similar to the target in (Huang et al., 2020), but, similarly to the reweighting scheme, we use the adversarial output o (cid:48) instead of the clean output o to avoid an increase in computational complexity.

Original: The ﬁnal adaptive target we use is t = β 1 y + (1 − β ) (cid:101) t and thus the loss objective is L w ( x (cid:48) , t ) .
Revised: The ﬁnal adaptive target we use is t = β 1 y + (1 − β ) (cid:101) t and thus the loss objective is L w ( x (cid:48) , t ) .

Original: The factor β controls how “adaptive” our target is: β = 0 yields a fully adaptive moving average target (cid:101) t and β = 1 yields a one-hot target 1 y .
Revised: The factor β controls how “adaptive” our target is: β = 0 yields a fully adaptive moving average target (cid:101) t and β = 1 yields a one-hot target 1 y .

Original: We provide the pseudocode as Algorithm 1 in Appendix C.2.
Revised: We provide the pseudocode as Algorithm 1 in Appendix C.2.

Original: We run experiments on CIFAR10 using WRN34 models under the l ∞ adversarial budget of size (cid:15) = 8 / 255 , the standard setting where most fast adversarial training algorithms are benchmarked (Croce et al., 2020).
Revised: We run experiments on CIFAR10 using WRN34 models under the l ∞ adversarial budget of size (cid:15) = 8 / 255 , the standard setting where most fast adversarial training algorithms are benchmarked (Croce et al., 2020).

Original: We evaluate the model’s robust accuracy on the test set by AutoAttack (Croce & Hein, 2020b), the popular and reliable attack for evaluation.
Revised: We evaluate the model’s robust accuracy on the test set by AutoAttack (Croce & Hein, 2020b), the popular and reliable attack for evaluation.

Original: The results are provided in Table 2, where the results of the baseline methods are taken from RobustBench (Croce et al., 2020).
Revised: The results are provided in Table 2, where the results of the baseline methods are taken from RobustBench (Croce et al., 2020).

Original: We also report the number of epochs and the number of forward and backward passes in a mini-batch update of each method.
Revised: We also report the number of epochs and the number of forward and backward passes in a mini-batch update of each method.

Original: The product of these two values indicates the training complexity.
Revised: The product of these two values indicates the training complexity.

Original: We can clearly see that both reweighting and adaptive targets improve the performance on top of ATTA (Zheng et al., 2020).
Revised: We can clearly see that both reweighting and adaptive targets improve the performance on top of ATTA (Zheng et al., 2020).

Original: Note that our method based on adaptive targets achieve the best performance while needing only 1 / 4 of the training time of (Chen et al., 2021a), the strongest baseline.
Revised: Note that our method based on adaptive targets achieve the best performance while needing only 1 / 4 of the training time of (Chen et al., 2021a), the strongest baseline.

Original: (Wong et al., 2020) is the only baseline consuming less training time than ours, but its performance is much worse than ours; it suffers from catastrophic overﬁtting when using a WRN model.
Revised: (Wong et al., 2020) is the only baseline consuming less training time than ours, but its performance is much worse than ours; it suffers from catastrophic overﬁtting when using a WRN model.

Original: In Appendix D.7, we provide the learning curves of our methods under different settings and show that both reweighting and adaptive targets mitigate adversarial overﬁtting.
Revised: In Appendix D.6, we provide the learning curves of our methods under different settings and show that both reweighting and adaptive targets mitigate adversarial overﬁtting.

Original: We also conduct an ablation study on the value of β and ﬁnd that a decrease in β decreases the generalization gap.
Revised: We also conduct an ablation study on the value of β and ﬁnd that a decrease in β decreases the generalization gap.

Original: This indicates that the more adaptive the targets, the smaller the generalization gap.
Revised: This indicates that the more adaptive the targets, the smaller the generalization gap.

Original: 6. A DVERSARIAL F INETUNING WITH A DDITIONAL D ATA
Revised: 6. A DVERSARIAL F INETUNING WITH A DDITIONAL D ATA

Original: We observe that adversarial overﬁtting occurs in the small learning rate regime in Section 4.
Revised: We observe that adversarial overﬁtting occurs in the small learning rate regime in Section 4.

Original: To further study this, we propose to ﬁne-tune an adversarially pretrained model using additional training data, because we also use small learning rate to ﬁne-tune a model.
Revised: To further study this, we propose to ﬁne-tune an adversarially pretrained model using additional training data, because we also use small learning rate to ﬁne-tune a model.

Original: While additional training data was
Revised: While additional training data was

Original: [Table] [Table]
Revised: [Table] [Table]

Original: shown to be beneﬁcial in (Alayrac et al., 2019; Carmon et al., 2019), we demonstrate that letting the model adaptively ﬁt the easy and hard instances can further improve the performance.
Revised: shown to be beneﬁcial in (Alayrac et al., 2019; Carmon et al., 2019), we demonstrate that letting the model adaptively ﬁt the easy and hard instances can further improve the performance.

Original: We conduct experiments on both CIFAR10 and SVHN, using WRN34 and RN18 models, respectively.
Revised: We conduct experiments on both CIFAR10 and SVHN, using WRN34 and RN18 models, respectively.

Original: The experimental settings are the same as (Carmon et al., 2019) except the learning rate.
Revised: The experimental settings are the same as (Carmon et al., 2019) except the learning rate.

Original: We tune the learning rate and ﬁnd that ﬁxing it to 10 − 3 is the best choice.
Revised: We tune the learning rate and ﬁnd that ﬁxing it to 10 − 3 is the best choice.

Original: The model is ﬁne-tuned for eitherepoch or 5 epochs, which means that each additional training instance is used either 5 times or only once.
Revised: The model is ﬁne-tuned for eitherepoch or 5 epochs, which means that each additional training instance is used either 5 times or only once.

Original: This is because we observed the performance of vanilla adversarial training to start decaying after 5 epochs.
Revised: This is because we observed the performance of vanilla adversarial training to start decaying after 5 epochs.

Original: As such, methods requiring many epochs such as (Balaji et al., 2019) and (Huang et al., 2020) are not applicable here.
Revised: As such, methods requiring many epochs such as (Balaji et al., 2019) and (Huang et al., 2020) are not applicable here.

Original: Our ﬁrst technique, reweighting, is the same as in Section 6.1.
Revised: Our ﬁrst technique, reweighting, is the same as in Section 6.1.

Original: In addition to reweighting, we can also add a KL regularization term measuring the KL divergence between the output probability of the clean instance and of the adversarial instance.
Revised: In addition to reweighting, we can also add a KL regularization term measuring the KL divergence between the output probability of the clean instance and of the adversarial instance.

Original: The KL term encourages the adversarial output to be close to the clean one.
Revised: The KL term encourages the adversarial output to be close to the clean one.

Original: In other words, the clean output probability serves as the adaptive target.
Revised: In other words, the clean output probability serves as the adaptive target.

Original: For hard instances, the clean and adversarial inputs are usually both misclassiﬁed.
Revised: For hard instances, the clean and adversarial inputs are usually both misclassiﬁed.

Original: Therefore, the clean outputs of these instances constitute simpler targets compared with the ground-truth labels.
Revised: Therefore, the clean outputs of these instances constitute simpler targets compared with the ground-truth labels.

Original: Ultimately, the loss objective of a mini-batch { x i } Bi
Revised: Ultimately, the loss objective of a mini-batch { x i } Bi

Original: =1 used for ﬁne-tuning is expressed as
Revised: =1 used for ﬁne-tuning is expressed as

Original: L FT ( { x i } Bi =1 ) =
Revised: L FT ( { x i } Bi =1 ) =

Original: (cid:80) Bi
Revised: (cid:80) Bi

Original: =1 w i
Revised: =1 w i

Original: [ L w ( x (cid:48) i ) + λKL ( o i || o (cid:48) i )] where w i is the adaptive weight when we use re-weighting, or 1 /B otherwise.
Revised: [ L w ( x (cid:48) i ) + λKL ( o i || o (cid:48) i )] where w i is the adaptive weight when we use re-weighting, or 1 /B otherwise.

Original: λ is 6 when using the regularization term and 0 otherwise.
Revised: λ is 6 when using the regularization term and 0 otherwise.

Original: We use both reweighting and KL regularization to ﬁne-tune the model.
Revised: We use both reweighting and KL regularization to ﬁne-tune the model.

Original: Our results are shown in
Revised: Our results are shown in

Original: Table 2, where the robust test accuracy is also evaluated by AutoAttack.
Revised: Table 2, where the robust test accuracy is also evaluated by AutoAttack.

Original: It is clear that our methods can improve the performance under all settings.
Revised: It is clear that our methods can improve the performance under all settings.

Original: We also conduct an ablation study in Appendix D.6.
Revised: We also conduct an ablation study in Appendix D.7.

Original: All these results show that avoiding ﬁtting hard adversarial examples helps to improve the generalization performance in adversarial ﬁne-tuning with additional training data.
Revised: All these results show that avoiding ﬁtting hard adversarial examples helps to improve the generalization performance in adversarial ﬁne-tuning with additional training data.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We have investigated adversarial overﬁtting from the perspective of the easy and hard training instances.
Revised: We have investigated adversarial overﬁtting from the perspective of the easy and hard training instances.

Original: By introducing a quantitative, model-agnostic and normalized metric to measure the instance difﬁculty, we have shown that a model’s generalization performance under adversarial attacks degrades during the later phase of training as the model ﬁts the hard adversarial instances.
Revised: By introducing a quantitative, model-agnostic and normalized metric to measure the instance difﬁculty, we have shown that a model’s generalization performance under adversarial attacks degrades during the later phase of training as the model ﬁts the hard adversarial instances.

Original: We have conducted theoretical analyses on both linear and nonlinear models.
Revised: We have conducted theoretical analyses on both linear and nonlinear models.

Original: On an over-parameterized logistic regression model, we have shown that training on harder adversarial instances leads to poorer generalization performance.
Revised: On an over-parameterized logistic regression model, we have shown that training on harder adversarial instances leads to poorer generalization performance.

Original: On general nonlinear models, we have proven that the lower bound of a well-trained model’s Lipschitz constant increases with the difﬁculty of the training instances.
Revised: On general nonlinear models, we have proven that the lower bound of a well-trained model’s Lipschitz constant increases with the difﬁculty of the training instances.

Original: Finally, our experiments on fast adversarial training and adversarial ﬁne-tuning with additional data have demonstrated that adaptively using the training instances mitigates adversarial overﬁtting and improves the model’s robustness.
Revised: Finally, our experiments on fast adversarial training and adversarial ﬁne-tuning with additional data have demonstrated that adaptively using the training instances mitigates adversarial overﬁtting and improves the model’s robustness.

Original: E THICS S TATEMENT
Revised: E THICS S TATEMENT

Original: Adversarial attacks may cause unexpected failure of modern deep learning models, especially in some safety-critical applications such as self-driving and medical imaging.
Revised: Adversarial attacks may cause unexpected failure of modern deep learning models, especially in some safety-critical applications such as self-driving and medical imaging.

Original: Our work studies the defense methods and focuses on how to improve training robust models to prevent such failure from happening.
Revised: Our work studies the defense methods and focuses on how to improve training robust models to prevent such failure from happening.

Original: We are not aware of any ethics issues about our work, as we are using publicly available data and models.
Revised: We are not aware of any ethics issues about our work, as we are using publicly available data and models.

Original: R EPRODUCIBILITY S TATEMENT
Revised: R EPRODUCIBILITY S TATEMENT

Original: The experimental settings of our empirical study are demonstrated in detail in Appendix C. Especially, we also provide the pseudocode of our algorithm there.
Revised: The experimental settings of our empirical study are demonstrated in detail in Appendix C. Especially, we also provide the pseudocode of our algorithm there.

Original: Our code is submitted anonymously on
Revised: Our code is submitted anonymously on

Original: GoogleDrive 3 .
Revised: GoogleDrive 3 .

Original: It will be made publicly available when our work is published.
Revised: It will be made publicly available when our work is published.

Original: A N OTATION
Revised: A N OTATION

Original: b
Revised: b

Original: Section 5.
Revised: Section 5.

Original: The number of parameters in a general nonlinear model.
Revised: The number of parameters in a general nonlinear model.

Original: c
Revised: c

Original: Assumption 1, Section 5.
Revised: Assumption 1, Section 5.

Original: The coefﬁcient in isoperimetry.
Revised: The coefﬁcient in isoperimetry.

Original: C
Revised: C

Original: Section 5.
Revised: Section 5.

Original: The mean squared error on the adversarial training set.
Revised: The mean squared error on the adversarial training set.

Original: d
Revised: d

Original: Equation 1, Section
Revised: Equation 1, Section

Original: The function introduced by the proposed difﬁculty metric.
Revised: The function introduced by the proposed difﬁculty metric.

Original: D
Revised: D

Original: Section
Revised: Section

Original: The data set.
Revised: The data set.

Original: f w
Revised: f w

Original: Section
Revised: Section

Original: The model parameterized by w .
Revised: The model parameterized by w .

Original: F
Revised: F

Original: Theorem 3, Section 5.
Revised: Theorem 3, Section 5.

Original: The function space of the model.
Revised: The function space of the model.

Original: G
Revised: G

Original: Section 4.
Revised: Section 4.

Original: Groups of the training set divided by instance difﬁculty.
Revised: Groups of the training set divided by instance difﬁculty.

Original: h
Revised: h

Original: Deﬁnition 1, Section 5.
Revised: Deﬁnition 1, Section 5.

Original: The bandwidth of the model’s output range.
Revised: The bandwidth of the model’s output range.

Original: J
Revised: J

Original: Theorem 3, Section 5.
Revised: Theorem 3, Section 5.

Original: The Lipschitz constant of f w w.r.t w .
Revised: The Lipschitz constant of f w w.r.t w .

Original: K
Revised: K

Original: Section
Revised: Section

Original: The number of components in the data distribution.
Revised: The number of components in the data distribution.

Original: l
Revised: l

Original: Section
Revised: Section

Original: The component index where the training data is sampled.
Revised: The component index where the training data is sampled.

Original: L
Revised: L

Original: Assumption 1, Section 5.
Revised: Assumption 1, Section 5.

Original: The Lipschitz constant of f w w.r.t the input.
Revised: The Lipschitz constant of f w w.r.t the input.

Original: L
Revised: L

Original: Section
Revised: Section

Original: The loss function.
Revised: The loss function.

Original: m
Revised: m

Original: Section
Revised: Section

Original: Dimension of the input data.
Revised: Dimension of the input data.

Original: n
Revised: n

Original: Section
Revised: Section

Original: The number of training instances.
Revised: The number of training instances.

Original: o , o (cid:48)
Revised: o , o (cid:48)

Original: Section
Revised: Section

Original: Model’s output of the clean and the adversarial input.
Revised: Model’s output of the clean and the adversarial input.

Original: p
Revised: p

Original: Section
Revised: Section

Original: Shape of the adversarial budget.
Revised: Shape of the adversarial budget.

Original: r
Revised: r

Original: Equation 3, Section 5.
Revised: Equation 3, Section 5.

Original: The coefﬁcient in the GMM model.
Revised: The coefﬁcient in the GMM model.

Original: R
Revised: R

Original: Theorem 2, Section 5.
Revised: Theorem 2, Section 5.

Original: The robust test error.
Revised: The robust test error.

Original: t , (cid:101) t
Revised: t , (cid:101) t

Original: Section 6.
Revised: Section 6.

Original: The adaptive target and the moving average target.
Revised: The adaptive target and the moving average target.

Original: w
Revised: w

Original: Section
Revised: Section

Original: Model parameters.
Revised: Model parameters.

Original: W
Revised: W

Original: Theorem 3, Section 5.
Revised: Theorem 3, Section 5.

Original: The diameter upper bound of the parameter space.
Revised: The diameter upper bound of the parameter space.

Original: W
Revised: W

Original: Theorem 3, Section 5.
Revised: Theorem 3, Section 5.

Original: The space of model parameters.
Revised: The space of model parameters.

Original: x , x (cid:48) , X
Revised: x , x (cid:48) , X

Original: Section 1 & Section
Revised: Section 1 & Section

Original: Clean input, adversarial input and its matrix form.
Revised: Clean input, adversarial input and its matrix form.

Original: y , y
Revised: y , y

Original: Section 1 & Section
Revised: Section 1 & Section

Original: Label and its vector form.
Revised: Label and its vector form.

Original: α
Revised: α

Original: Algorithm
Revised: Algorithm

Original: The step size of the adversarial attacks.
Revised: The step size of the adversarial attacks.

Original: β
Revised: β

Original: Section 6.
Revised: Section 6.

Original: The coefﬁcient controlling how adaptive the target is.
Revised: The coefﬁcient controlling how adaptive the target is.

Original: γ
Revised: γ

Original: Theorem 3, Section 5.
Revised: Theorem 3, Section 5.

Original: The non-negative variable introduced in Theorem 3.
Revised: The non-negative variable introduced in Theorem 3.

Original: δ
Revised: δ

Original: Theorem 3, Section 5.
Revised: Theorem 3, Section 5.

Original: The probability introduced in Theorem 3.
Revised: The probability introduced in Theorem 3.

Original: (cid:15)
Revised: (cid:15)

Original: Section
Revised: Section

Original: The size of the adversarial budget.
Revised: The size of the adversarial budget.

Original: η
Revised: η

Original: Equation 3, Section 5.
Revised: Equation 3, Section 5.

Original: The direction of the mean of each GMM’s component.
Revised: The direction of the mean of each GMM’s component.

Original: ρ
Revised: ρ

Original: Section 6.
Revised: Section 6.

Original: The momentum calculating the moving average target.
Revised: The momentum calculating the moving average target.

Original: µ l , µ l
Revised: µ l , µ l

Original: Assumption 1, Section 5.
Revised: Assumption 1, Section 5.

Original: Data distribution and its l -th component.
Revised: Data distribution and its l -th component.

Original: σ
Revised: σ

Original: Assumption 1, Section 5.
Revised: Assumption 1, Section 5.

Original: The conditional variance of the data distribution.
Revised: The conditional variance of the data distribution.

Original: B P ROOFS IN T HEORETICAL A NALYSIS B. P ROOF OF T HEOREM
Revised: B P ROOFS IN T HEORETICAL A NALYSIS B. P ROOF OF T HEOREM

Original: Similar to (Soudry et al., 2018), we can assume all instances are positive without the loss of generality, this is because we can always redeﬁne y i
Revised: Similar to (Soudry et al., 2018), we can assume all instances are positive without the loss of generality, this is because we can always redeﬁne y i

Original: x i as the input.
Revised: x i as the input.

Original: In this regard, the loss to optimize in a logistic regression model under the adversarial budget S (2) ( (cid:15) ) is:
Revised: In this regard, the loss to optimize in a logistic regression model under the adversarial budget S (2) ( (cid:15) ) is:

Original: [Equation]
Revised: [Equation]

Original: Here l ( · ) is the logistic function: l ( x ) = 1 1+ e − x .
Revised: Here l ( · ) is the logistic function: l ( x ) = 1 1+ e − x .

Original: We use X ∈ R n × m to represent the training set as said in Section 5, then the loss function L ( w ) is (cid:107) X (cid:107) 2 -smooth, where (cid:107) X
Revised: We use X ∈ R n × m to represent the training set as said in Section 5, then the loss function L ( w ) is (cid:107) X (cid:107) 2 -smooth, where (cid:107) X

Original: (cid:107) 2 is the maximal singular value of X .
Revised: (cid:107) 2 is the maximal singular value of X .

Original: Since function L w is convex on w , so gradient descent of step size smaller than 2 (cid:107) X (cid:107) −will asymptotically converge to the global inﬁmum of the function L w on w .
Revised: Since function L w is convex on w , so gradient descent of step size smaller than 2 (cid:107) X (cid:107) −will asymptotically converge to the global inﬁmum of the function L w on w .

Original: Before proving Theorem 1, we ﬁrst introduce the following lemma:
Revised: Before proving Theorem 1, we ﬁrst introduce the following lemma:

Original: Lemma 1.
Revised: Lemma 1.

Original: Consider the max-margin vector (cid:98) w of the vanilla case deﬁned in Equation (2), we then introduce the max margin vector (cid:99) w (cid:48) deﬁned under the adversarial attack of budget S (2) ( (cid:15) ) as follows:
Revised: Consider the max-margin vector (cid:98) w of the vanilla case deﬁned in Equation (2), we then introduce the max margin vector (cid:99) w (cid:48) deﬁned under the adversarial attack of budget S (2) ( (cid:15) ) as follows:

Original: [Equation]
Revised: [Equation]

Original: Then we have (cid:99) w (cid:48) is collinear with (cid:98) w , i.e.,
Revised: Then we have (cid:99) w (cid:48) is collinear with (cid:98) w , i.e.,

Original: [Equation]
Revised: [Equation]

Original: w
Revised: w

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: We show that (cid:98) w =
Revised: We show that (cid:98) w =

Original: [Equation]
Revised: [Equation]

Original: w
Revised: w

Original: [Equation]
Revised: [Equation]

Original: and prove it by contraction.
Revised: and prove it by contraction.

Original: Let’s assume ∃ v , s.t.
Revised: Let’s assume ∃ v , s.t.

Original: (cid:107) v (cid:107) <
Revised: (cid:107) v (cid:107) <

Original: [Equation]
Revised: [Equation]

Original: w
Revised: w

Original: [Equation]
Revised: [Equation]

Original: 1+ (cid:15) (cid:107) (cid:99) w (cid:48) (cid:107) and ∀ i ∈ { 1 , 2 , ..., n } , v T x i ≥ 1 , then we can consider v (cid:48)
Revised: 1+ (cid:15) (cid:107) (cid:99) w (cid:48) (cid:107) and ∀ i ∈ { 1 , 2 , ..., n } , v T x i ≥ 1 , then we can consider v (cid:48)

Original: = (1 + (cid:107) (cid:99) w (cid:48) (cid:107) ) v .
Revised: = (1 + (cid:107) (cid:99) w (cid:48) (cid:107) ) v .

Original: The l 2 norm of v (cid:48) is smaller than that of (cid:99) w (cid:48) , and we have ∀ i ∈ { 1 , 2 , ..., n } , v (cid:48)
Revised: The l 2 norm of v (cid:48) is smaller than that of (cid:99) w (cid:48) , and we have ∀ i ∈ { 1 , 2 , ..., n } , v (cid:48)

Original: [Equation]
Revised: [Equation]

Original: Inequality 10 shows we can construct a vector v (cid:48) whose l 2 norm is smaller than (cid:99) w (cid:48) and satisfying the condition (9), this contracts with the optimality of (cid:99) w (cid:48) .
Revised: Inequality 10 shows we can construct a vector v (cid:48) whose l 2 norm is smaller than (cid:99) w (cid:48) and satisfying the condition (9), this contracts with the optimality of (cid:99) w (cid:48) .

Original: Therefore, there is no solution of condition (2) whose norm is smaller than
Revised: Therefore, there is no solution of condition (2) whose norm is smaller than

Original: [Equation]
Revised: [Equation]

Original: On the other hand,
Revised: On the other hand,

Original: [Equation]
Revised: [Equation]

Original: w (cid:48) satisﬁes the condition (2) and its l 2 norm is
Revised: w (cid:48) satisﬁes the condition (2) and its l 2 norm is

Original: [Equation]
Revised: [Equation]

Original: As a result, we have (cid:98) w =
Revised: As a result, we have (cid:98) w =

Original: [Equation]
Revised: [Equation]

Original: w
Revised: w

Original: [Equation]
Revised: [Equation]

Original: That means (cid:98) w and (cid:99) w (cid:48) are collinear.
Revised: That means (cid:98) w and (cid:99) w (cid:48) are collinear.

Original: With Lemma 1, Theorem 1 is more straightforward, whose proof is shown below.
Revised: With Lemma 1, Theorem 1 is more straightforward, whose proof is shown below.

Original: Regarding the convergence analysis of the logistic regression model in non-adversarial cases, we encourage the readers to ﬁnd more details in (Ji & Telgarsky, 2019; Soudry et al., 2018).
Revised: Regarding the convergence analysis of the logistic regression model in non-adversarial cases, we encourage the readers to ﬁnd more details in (Ji & Telgarsky, 2019; Soudry et al., 2018).

Original: Proof.
Revised: Proof.

Original: Theorem 1 in (Ji & Telgarsky, 2019) and Theorem 3 in (Soudry et al., 2018) proves the convergence of the direction of the logistic regression parameters in different cases.
Revised: Theorem 1 in (Ji & Telgarsky, 2019) and Theorem 3 in (Soudry et al., 2018) proves the convergence of the direction of the logistic regression parameters in different cases.

Original: In this regard, we can let w ∞ =
Revised: In this regard, we can let w ∞ =

Original: lim u →∞ w ( u ) (cid:107) w ( u ) (cid:107) .
Revised: lim u →∞ w ( u ) (cid:107) w ( u ) (cid:107) .

Original: That is to say, for sufﬁciently large u , the direction of the parameter w ( u ) can be considered ﬁxed.
Revised: That is to say, for sufﬁciently large u , the direction of the parameter w ( u ) can be considered ﬁxed.

Original: As a result, the adversarial perturbations of each data instance x i is ﬁxed, i.e., (cid:15) w ∞ .
Revised: As a result, the adversarial perturbations of each data instance x i is ﬁxed, i.e., (cid:15) w ∞ .

Original: We can then apply the conclusion of Theorem 3 in (Soudry et al., 2018) here, the only difference is the data points are { x i − (cid:15) w ∞ } ni =1 .
Revised: We can then apply the conclusion of Theorem 3 in (Soudry et al., 2018) here, the only difference is the data points are { x i − (cid:15) w ∞ } ni =1 .

Original: Therefore, the parameter w ( u ) will converge to the lmax margin of the dataset { x i − (cid:15) w ∞ } ni =1 .
Revised: Therefore, the parameter w ( u ) will converge to the lmax margin of the dataset { x i − (cid:15) w ∞ } ni =1 .

Original: When t → ∞ , we have w ( u ) T ( x i − (cid:15) w ∞ ) =
Revised: When t → ∞ , we have w ( u ) T ( x i − (cid:15) w ∞ ) =

Original: w ( u ) T x i − (cid:15) (cid:107) w ( u ) (cid:107) .
Revised: w ( u ) T x i − (cid:15) (cid:107) w ( u ) (cid:107) .

Original: This is exactly the adversarial max margin condition in (9).
Revised: This is exactly the adversarial max margin condition in (9).

Original: Based on
Revised: Based on

Original: Lemma 1, we have lim u →∞
Revised: Lemma 1, we have lim u →∞

Original: [Equation]
Revised: [Equation]

Original: ( u
Revised: ( u

Original: [Equation]
Revised: [Equation]

Original: B. P ROOF OF T HEOREM
Revised: B. P ROOF OF T HEOREM

Original: Given the parameter w of the logistic regression model, we can ﬁrst calculate the robust error for the k -th component of the GMM model deﬁned in (3).
Revised: Given the parameter w of the logistic regression model, we can ﬁrst calculate the robust error for the k -th component of the GMM model deﬁned in (3).

Original: Lemma 2.
Revised: Lemma 2.

Original: The 0-1 classiﬁcation error of a linear classiﬁer w under the adversarial attack of the budget S (2) ( (cid:15) ) for the k -th component of the GMM model deﬁned in (3) is:
Revised: The 0-1 classiﬁcation error of a linear classiﬁer w under the adversarial attack of the budget S (2) ( (cid:15) ) for the k -th component of the GMM model deﬁned in (3) is:

Original: [Equation]
Revised: [Equation]

Original: (11) where Φ( x ) =
Revised: (11) where Φ( x ) =

Original: P ( Z > x ) , Z ∼ N (0 , 1) .
Revised: P ( Z > x ) , Z ∼ N (0 , 1) .

Original: Proof.
Revised: Proof.

Original: For a random drawn data instance ( x , y ) , the adversarial perturbation is − y(cid:15) w (cid:107) w (cid:107) .
Revised: For a random drawn data instance ( x , y ) , the adversarial perturbation is − y(cid:15) w (cid:107) w (cid:107) .

Original: Let’s decompose x as r k y η
Revised: Let’s decompose x as r k y η

Original: + z , where z ∼ N (0 , I ) .
Revised: + z , where z ∼ N (0 , I ) .

Original: Then, we have
Revised: Then, we have

Original: [Equation]
Revised: [Equation]

Original: Since z ∼ N (0 , I ) , we have − y w T z ∼ N (0 , ( − y w T ) T ( − y w T ))
Revised: Since z ∼ N (0 , I ) , we have − y w T z ∼ N (0 , ( − y w T ) T ( − y w T ))

Original: =
Revised: =

Original: N (0 , w T w ) .
Revised: N (0 , w T w ) .

Original: Furthermore − y w T z (cid:107) w (cid:107) ∼ N (0 , 1) , and we can further simplify (cid:98) R k ( (cid:15) ) as follows:
Revised: Furthermore − y w T z (cid:107) w (cid:107) ∼ N (0 , 1) , and we can further simplify (cid:98) R k ( (cid:15) ) as follows:

Original: [Equation]
Revised: [Equation]

Original: With Lemma 2, we can straightforwardly calculate the robust error for all components of the GMM model deﬁned in (3): (cid:98)
Revised: With Lemma 2, we can straightforwardly calculate the robust error for all components of the GMM model deﬁned in (3): (cid:98)

Original: [Equation]
Revised: [Equation]

Original: On the other hand, Theorem 1 indicates the parameter w will converge to the l 2 max margin.
Revised: On the other hand, Theorem 1 indicates the parameter w will converge to the l 2 max margin.

Original: However, for arbitrary training set, we do not have the closed form of w , which is a barrier for the further analysis.
Revised: However, for arbitrary training set, we do not have the closed form of w , which is a barrier for the further analysis.

Original: Nevertheless, results from (Wang & Thrampoulidis, 2020) indicates in the over-parameterization regime, the parameter w will converge to min-norm interpolation of the data with high probability.
Revised: Nevertheless, results from (Wang & Thrampoulidis, 2020) indicates in the over-parameterization regime, the parameter w will converge to min-norm interpolation of the data with high probability.

Original: Lemma 3.
Revised: Lemma 3.

Original: (Directly from Theorem 1 in (Wang & Thrampoulidis, 2020))
Revised: (Directly from Theorem 1 in (Wang & Thrampoulidis, 2020))

Original: Assume n training instances drawn from the l -th mode of the described distribution in (3) and each of them is a m -dimensional vector.
Revised: Assume n training instances drawn from the l -th mode of the described distribution in (3) and each of them is a m -dimensional vector.

Original: If m n log n is sufﬁciently large 4 , then the l 2 max margin vector in Equation (2) will be the same as the solution of the min-norm interpolation described below with probability at least (1 − O ( 1 n )) .
Revised: If m n log n is sufﬁciently large 4 , then the l 2 max margin vector in Equation (2) will be the same as the solution of the min-norm interpolation described below with probability at least (1 − O ( 1 n )) .

Original: [Equation]
Revised: [Equation]

Original: Since the min-norm interpolation has a closed solution ¯ w = X T ( XX T ) − 1 y , Lemma 3 will greatly facilitate the calculation of R ( w ) in Theorem 2.
Revised: Since the min-norm interpolation has a closed solution ¯ w = X T ( XX T ) − 1 y , Lemma 3 will greatly facilitate the calculation of R ( w ) in Theorem 2.

Original: To simplify the notation, we ﬁrst deﬁne the following variables.
Revised: To simplify the notation, we ﬁrst deﬁne the following variables.

Original: [Equation]
Revised: [Equation]

Original: The proof of Theorem 2 is then presented below.
Revised: The proof of Theorem 2 is then presented below.

Original: Proof.
Revised: Proof.

Original: Based on (14), the key is to simplify the term w T η (cid:107) w (cid:107) , let’s denote it by A , then we have:
Revised: Based on (14), the key is to simplify the term w T η (cid:107) w (cid:107) , let’s denote it by A , then we have:

Original: [Equation]
Revised: [Equation]

Original: The key challenge here is to calculate the term ( XX T ) − 

Revised: The key challenge here is to calculate the term ( XX T ) −

Original: where X =
Revised: where X =

Original: r l yη T + Q .
Revised: r l yη T + Q .

Original: Here we utilize
Revised: Here we utilize

Original: Lemma 3 of (Wang & Thrampoulidis, 2020) and Woodbury identity (Horn & Johnson, 2012), we have:
Revised: Lemma 3 of (Wang & Thrampoulidis, 2020) and Woodbury identity (Horn & Johnson, 2012), we have:

Original: [Equation]
Revised: [Equation]

Original: Here, s , t , v , U and d are deﬁned in Equation (16).
Revised: Here, s , t , v , U and d are deﬁned in Equation (16).

Original: The scalar divisor comes from the matrix inverse calculation.
Revised: The scalar divisor comes from the matrix inverse calculation.

Original: Base of Equation (18), we can then calculate y T ( XX T ) − 1 y and y T ( XX T ) − 1 X η .
Revised: Base of Equation (18), we can then calculate y T ( XX T ) − 1 y and y T ( XX T ) − 1 X η .

Original: [Equation]
Revised: [Equation]

Original: Plug Equation (19) and (20) into (17), we have:
Revised: Plug Equation (19) and (20) into (17), we have:

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Plug (21) into (14), we then obtain the robust error on all components of the GMM deﬁned in (3):
Revised: Plug (21) into (14), we then obtain the robust error on all components of the GMM deﬁned in (3):

Original: [Equation]
Revised: [Equation]

Original: r
Revised: r

Original: [Equation]
Revised: [Equation]

Original: (cid:15)
Revised: (cid:15)

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: We study the sign of C 1 and C 2 .
Revised: We study the sign of C 1 and C 2 .

Original: Consider U =
Revised: Consider U =

Original: QQ T is a positive semideﬁnite matrix, so s =
Revised: QQ T is a positive semideﬁnite matrix, so s =

Original: y U − 1 y T ≥ 0 .
Revised: y U − 1 y T ≥ 0 .

Original: In addition, we have (cid:107) η (cid:107) 2 − t =
Revised: In addition, we have (cid:107) η (cid:107) 2 − t =

Original: η T (cid:0) I − ( QQ T ) − 1 (cid:1) η .
Revised: η T (cid:0) I − ( QQ T ) − 1 (cid:1) η .

Original: Since I − ( QQ T ) − 

Revised: Since I − ( QQ T ) −

Original: = ( I − ( QQ T ) − 1 ) T ( I − ( QQ T ) − 1 ) is a positive semideﬁnite matrix, we can obtain I − ( QQ T ) − 1 is also a positive semideﬁnite matrix.
Revised: = ( I − ( QQ T ) − 1 ) T ( I − ( QQ T ) − 1 ) is a positive semideﬁnite matrix, we can obtain I − ( QQ T ) − 1 is also a positive semideﬁnite matrix.

Original: As a result, C 1 and C 2 are both non-negative.
Revised: As a result, C 1 and C 2 are both non-negative.

Original: B. P ROOF OF C OROLLARY
Revised: B. P ROOF OF C OROLLARY

Original: To prove Corollary 1, we ﬁrst prove the following lemma:
Revised: To prove Corollary 1, we ﬁrst prove the following lemma:

Original: Lemma 4.
Revised: Lemma 4.

Original: Under the condition of Theorem 2 and R in Equation (4), ∂ R ( r l ,(cid:15) )
Revised: Under the condition of Theorem 2 and R in Equation (4), ∂ R ( r l ,(cid:15) )

Original: ∂r l is negative and monotonically decreases with (cid:15) .
Revised: ∂r l is negative and monotonically decreases with (cid:15) .

Original: Proof.
Revised: Proof.

Original: Based on Equation (22), we have:
Revised: Based on Equation (22), we have:

Original: [Equation]
Revised: [Equation]

Original: Since the training data is separable, we have ∀ k, r k w T η − (cid:15) (cid:107) w (cid:107) > 0 , which is equivalent to the following:
Revised: Since the training data is separable, we have ∀ k, r k w T η − (cid:15) (cid:107) w (cid:107) > 0 , which is equivalent to the following:

Original: [Equation]
Revised: [Equation]

Original: First, p k is a positive number by deﬁnition.
Revised: First, p k is a positive number by deﬁnition.

Original: Consider function Φ( x ) monotonically decrease with x and is convex when x > 0 , so ∀ k, Φ (cid:48)
Revised: Consider function Φ( x ) monotonically decrease with x and is convex when x > 0 , so ∀ k, Φ (cid:48)

Original: ( r k g ( r l ) − (cid:15) ) is negative and decreases with (cid:15) .
Revised: ( r k g ( r l ) − (cid:15) ) is negative and decreases with (cid:15) .

Original: In addition, g ( r l ) increases with r l and is independent on (cid:15) , so ∂g ( r l ) ∂r l can be considered as a positive constant.
Revised: In addition, g ( r l ) increases with r l and is independent on (cid:15) , so ∂g ( r l ) ∂r l can be considered as a positive constant.

Original: Therefore, ∂ R ( r l ,(cid:15) )
Revised: Therefore, ∂ R ( r l ,(cid:15) )

Original: ∂r l is negative and monotonically decreases with (cid:15) .
Revised: ∂r l is negative and monotonically decreases with (cid:15) .

Original: Now, we are ready to prove Corollary 1:
Revised: Now, we are ready to prove Corollary 1:

Original: Proof.
Revised: Proof.

Original: We subtract the left hand side from the right hand side in the inequality of Corollary 1:
Revised: We subtract the left hand side from the right hand side in the inequality of Corollary 1:

Original: [Equation]
Revised: [Equation]

Original: The last inequality is based on r j > r i , (cid:15) 2 > (cid:15) 1 , Lemma 4, which indicates (cid:104) ∂ R
Revised: The last inequality is based on r j > r i , (cid:15) 2 > (cid:15) 1 , Lemma 4, which indicates (cid:104) ∂ R

Original: [Equation]
Revised: [Equation]

Original: r
Revised: r

Original: [Equation]
Revised: [Equation]

Original: 1 ) ∂r l
Revised: 1 ) ∂r l

Original: [Equation]
Revised: [Equation]

Original: is always positive.
Revised: is always positive.

Original: We reorganize (25) and obtain R ( r i , (cid:15) 1 ) − R ( r j , (cid:15) 1 ) <
Revised: We reorganize (25) and obtain R ( r i , (cid:15) 1 ) − R ( r j , (cid:15) 1 ) <

Original: R ( r i , (cid:15) 2 ) − R ( r j , (cid:15) 2 ) .
Revised: R ( r i , (cid:15) 2 ) − R ( r j , (cid:15) 2 ) .

Original: B. P ROOF OF T HEOREM
Revised: B. P ROOF OF T HEOREM

Original: We start with the following lemma.
Revised: We start with the following lemma.

Original: Lemma 5.
Revised: Lemma 5.

Original: Given the assumptions of Theorem 3, we deﬁne g ( x ) =
Revised: Given the assumptions of Theorem 3, we deﬁne g ( x ) =

Original: E ( y | x ) , z ( x ) =
Revised: E ( y | x ) , z ( x ) =

Original: y − g ( x ) and consider γ = σ 2 l + h 2 ( C, (cid:15) ) − C , then the following inequality holds.
Revised: y − g ( x ) and consider γ = σ 2 l + h 2 ( C, (cid:15) ) − C , then the following inequality holds.

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Given the deﬁnition of h ( C, (cid:15) ) , we have:
Revised: Given the deﬁnition of h ( C, (cid:15) ) , we have:

Original: [Equation]
Revised: [Equation]

Original: For the ﬁrst inequality, x (cid:48) i is the adversarial example which tries to maximize the loss objective, y i ∈ {− 1 , +1 } and the range of f w is [ − 1 , +1] , so (cid:104)
Revised: For the ﬁrst inequality, x (cid:48) i is the adversarial example which tries to maximize the loss objective, y i ∈ {− 1 , +1 } and the range of f w is [ − 1 , +1] , so (cid:104)

Original: y
Revised: y

Original: i − f w ( x i ) , f w ( x i ) − f w ( x (cid:48) i ) (cid:105) ≥ 0 .
Revised: i − f w ( x i ) , f w ( x i ) − f w ( x (cid:48) i ) (cid:105) ≥ 0 .

Original: The second inequality is based on the deﬁnition of h 2 ( C, (cid:15) ) in Deﬁnition 1.
Revised: The second inequality is based on the deﬁnition of h 2 ( C, (cid:15) ) in Deﬁnition 1.

Original: As a result, we can simplify the left hand side of (26) as follows:
Revised: As a result, we can simplify the left hand side of (26) as follows:

Original: [Equation]
Revised: [Equation]

Original: We consider the sequence { z ( x i ) } ni =1 , it is i.i.d with E µ l
Revised: We consider the sequence { z ( x i ) } ni =1 , it is i.i.d with E µ l

Original: ( z ( x ) 2 )
Revised: ( z ( x ) 2 )

Original: =
Revised: =

Original: E µ l [ V ar ( y | x )]
Revised: E µ l [ V ar ( y | x )]

Original: = σ 2 l .
Revised: = σ 2 l .

Original: Since the range of the prediction is [ − 1 , +1] , so z 2 ( x ) ∈ [0 , 4] .
Revised: Since the range of the prediction is [ − 1 , +1] , so z 2 ( x ) ∈ [0 , 4] .

Original: Then, we have the following inequality by
Revised: Then, we have the following inequality by

Original: Hoeffding’s inequality (Hoeffding, 1994).
Revised: Hoeffding’s inequality (Hoeffding, 1994).

Original: [Equation]
Revised: [Equation]

Original: Similarly, we consider the sequence { z ( x i ) g ( x i ) } ni =1 , the following inequality holds based on the
Revised: Similarly, we consider the sequence { z ( x i ) g ( x i ) } ni =1 , the following inequality holds based on the

Original: Hoeffding’s inequality and the fact E ( z ( x ) g ( x
Revised: Hoeffding’s inequality and the fact E ( z ( x ) g ( x

Original: [Equation]
Revised: [Equation]

Original: = 0 , z
Revised: = 0 , z

Original: [Equation]
Revised: [Equation]

Original: Now we study the right hand side of (28):
Revised: Now we study the right hand side of (28):

Original: [Equation]
Revised: [Equation]

Original: Consider the following reasoning:
Revised: Consider the following reasoning:

Original: [Equation]
Revised: [Equation]

Original: As a result
Revised: As a result

Original: [Equation]
Revised: [Equation]

Original: we have:
Revised: we have:

Original: [Equation]
Revised: [Equation]

Original: The ﬁrst inequality is based on the reasoning of (32).
Revised: The ﬁrst inequality is based on the reasoning of (32).

Original: The second inequality is based on (29) and (30).
Revised: The second inequality is based on (29) and (30).

Original: Based on the inequality (28) and (33), we conclude the proof
Revised: Based on the inequality (28) and (33), we conclude the proof

Original: [Equation]
Revised: [Equation]

Original: To further simplify the right hand side of (26), P ( ∃ f w ∈ F : 1 n (cid:80) ni =1 z ( x i ) f w ( x i ) ≥ 12 (1 − 3 a ) γ ) needs to be bounded, and this is solved by the following lemma.
Revised: To further simplify the right hand side of (26), P ( ∃ f w ∈ F : 1 n (cid:80) ni =1 z ( x i ) f w ( x i ) ≥ 12 (1 − 3 a ) γ ) needs to be bounded, and this is solved by the following lemma.

Original: Lemma 6.
Revised: Lemma 6.

Original: Given the assumptions of Theorem 3 and the deﬁnition of g ( x ) , z ( x ) in Lemma 5, then the following inequality holds.
Revised: Given the assumptions of Theorem 3 and the deﬁnition of g ( x ) , z ( x ) in Lemma 5, then the following inequality holds.

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: We recall that the data points { x i , y i } ni =1 are sampled from the distribution µ l , which is c -isoperimetric.
Revised: We recall that the data points { x i , y i } ni =1 are sampled from the distribution µ l , which is c -isoperimetric.

Original: For any L -Lipschitz function f , we have:
Revised: For any L -Lipschitz function f , we have:

Original: [Equation]
Revised: [Equation]

Original: Since z ( x ) =
Revised: Since z ( x ) =

Original: y − g ( x ) ∈ [ − 2 , +2] , we can then bound P [ z ( x )( f w
Revised: y − g ( x ) ∈ [ − 2 , +2] , we can then bound P [ z ( x )( f w

Original: [Equation]
Revised: [Equation]

Original: x
Revised: x

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: Here we utilize the proposition in (Vershynin, 2018; Van Handel, 2014) 5 , which claims if { X i } ni = are independent variables and all C -subgaussian, then 1 √ n (cid:80) ni
Revised: Here we utilize the proposition in (Vershynin, 2018; Van Handel, 2014) 5 , which claims if { X i } ni = are independent variables and all C -subgaussian, then 1 √ n (cid:80) ni

Original: =
Revised: =

Original: 1 X
Revised: 1 X

Original: i is 18 C -subgaussian.
Revised: i is 18 C -subgaussian.

Original: Therefore, we have:
Revised: Therefore, we have:

Original: [Equation]
Revised: [Equation]

Original: Let t = a 1 γ √ n , then we have:
Revised: Let t = a 1 γ √ n , then we have:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: In addition, we can bound P [ 1 n (cid:80) ni
Revised: In addition, we can bound P [ 1 n (cid:80) ni

Original: =1 z ( x i ) E µ l ( f w ) ≥ a 2 γ ] by:
Revised: =1 z ( x i ) E µ l ( f w ) ≥ a 2 γ ] by:

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: The ﬁrst inequality is based on the fact E µ l ( f w ) ∈
Revised: The ﬁrst inequality is based on the fact E µ l ( f w ) ∈

Original: [ − 1 , +1] ; the second inequality is based on
Revised: [ − 1 , +1] ; the second inequality is based on

Original: Hoeffding’s inequality.
Revised: Hoeffding’s inequality.

Original: Now, we are ready to bound the probability P ( ∃ f w ∈ F : 1 n (cid:80) ni
Revised: Now, we are ready to bound the probability P ( ∃ f w ∈ F : 1 n (cid:80) ni

Original: =1 z ( x i )
Revised: =1 z ( x i )

Original: f w ( x i ) ≥ 12 (1 − 3 a ) γ ) .
Revised: f w ( x i ) ≥ 12 (1 − 3 a ) γ ) .

Original: [Equation]
Revised: [Equation]

Original: ≤
Revised: ≤

Original: [Equation]
Revised: [Equation]

Original: |F| e − nm 144 cL
Revised: |F| e − nm 144 cL

Original: [Equation]
Revised: [Equation]

Original: The ﬁrst inequality is based on the fact a 1 + a 2 = 12 (1 − 3 a ) ; the second inequality is based on the
Revised: The ﬁrst inequality is based on the fact a 1 + a 2 = 12 (1 − 3 a ) ; the second inequality is based on the

Original: Boole’s inequality (Boole, 1847), inequality (38) and (39).
Revised: Boole’s inequality (Boole, 1847), inequality (38) and (39).

Original: To simplify the constant notation, we let a = 18 , a 1 = 316 and a 2 = 18 .
Revised: To simplify the constant notation, we let a = 18 , a 1 = 316 and a 2 = 18 .

Original: We plug this into the inequality (26) and (34), then:
Revised: We plug this into the inequality (26) and (34), then:

Original: [Equation]
Revised: [Equation]

Original: Now we turn to the proof of Theorem 3.
Revised: Now we turn to the proof of Theorem 3.

Original: Proof.
Revised: Proof.

Original: We let F L = { f w | w ∈ W , Lip ( f w ) ≤ L } , F γ = { f w | w ∈ W , w = γ 4 J (cid:12) z , z ∈ Z b } and
Revised: We let F L = { f w | w ∈ W , Lip ( f w ) ≤ L } , F γ = { f w | w ∈ W , w = γ 4 J (cid:12) z , z ∈ Z b } and

Original: F γ,L = F γ ∩ F L .
Revised: F γ,L = F γ ∩ F L .

Original: Correspondingly, we let W L = { w | w ∈ W , Lip ( f w ) ≤ L } , W γ = { w | w ∈
Revised: Correspondingly, we let W L = { w | w ∈ W , Lip ( f w ) ≤ L } , W γ = { w | w ∈

Original: W , w = γ 4 J (cid:12) z , z ∈ Z b } and W γ,L = W γ ∩ W L .
Revised: W , w = γ 4 J (cid:12) z , z ∈ Z b } and W γ,L = W γ ∩ W L .

Original: Because the diameter of W is W , we have |F γ,L | ≤ |F γ | ≤ (cid:16) 4 WJγ (cid:17) b .
Revised: Because the diameter of W is W , we have |F γ,L | ≤ |F γ | ≤ (cid:16) 4 WJγ (cid:17) b .

Original: Here, (cid:12) means the element-wise multiplication.
Revised: Here, (cid:12) means the element-wise multiplication.

Original: Note that the inequality (41) is valid for any values of C as long as it satisﬁes γ ≥ 0 .
Revised: Note that the inequality (41) is valid for any values of C as long as it satisﬁes γ ≥ 0 .

Original: Based on this, we apply the substitution
Revised: Based on this, we apply the substitution

Original: [Equation]
Revised: [Equation]

Original: , then:
Revised: , then:

Original: [Equation]
Revised: [Equation]

Original: Based on the deﬁnition of W γ,L , we can conclude that ∀ w 1 ∈ W L , ∃
Revised: Based on the deﬁnition of W γ,L , we can conclude that ∀ w 1 ∈ W L , ∃

Original: [Equation]
Revised: [Equation]

Original: (cid:107) w 1 − w 2 (cid:107) ∞ ≤ γ 8 J .
Revised: (cid:107) w 1 − w 2 (cid:107) ∞ ≤ γ 8 J .

Original: Therefore, ∀ f w 1 ∈ F L , ∃ f w 2 ∈ F γ,L s.t.
Revised: Therefore, ∀ f w 1 ∈ F L , ∃ f w 2 ∈ F γ,L s.t.

Original: (cid:107) f
Revised: (cid:107) f

Original: w 1 − f w 2 (cid:107) ∞ ≤ γ 8 .
Revised: w 1 − f w 2 (cid:107) ∞ ≤ γ 8 .

Original: Let choose such f w 2 ∈ F γ,L given an arbitrary f w 1 ∈ F L , then:
Revised: Let choose such f w 2 ∈ F γ,L given an arbitrary f w 1 ∈ F L , then:

Original: [Equation]
Revised: [Equation]

Original: The ﬁrst inequality in (43) is based on Hölder’s inequality; the second inequality is based on y ∈ {− 1 , +1 } and the range of ∀ f w ∈ F is [ − 1 , +1] .
Revised: The ﬁrst inequality in (43) is based on Hölder’s inequality; the second inequality is based on y ∈ {− 1 , +1 } and the range of ∀ f w ∈ F is [ − 1 , +1] .

Original: We combine (41) with (43), then:
Revised: We combine (41) with (43), then:

Original: [Equation]
Revised: [Equation]

Original: Note that F L is the set of functions in F whose Lipschitz constant is no larger than L .
Revised: Note that F L is the set of functions in F whose Lipschitz constant is no larger than L .

Original: We set the right hand side of (44) to be δ and then get L = γ 2 7 (cid:113)
Revised: We set the right hand side of (44) to be δ and then get L = γ 2 7 (cid:113)

Original: [Equation]
Revised: [Equation]

Original: This concludes the proof.
Revised: This concludes the proof.

Original: C E XPERIMENTAL S ETTINGS C. G ENERAL S ETTINGS
Revised: C E XPERIMENTAL S ETTINGS C. G ENERAL S ETTINGS

Original: The ResNet-18 (RN18) architecture is same as the one in (Wong et al., 2020); the WideResNet- (WRN34) architecture is same as the one in (Madry et al., 2018).
Revised: The ResNet-18 (RN18) architecture is same as the one in (Wong et al., 2020); the WideResNet- (WRN34) architecture is same as the one in (Madry et al., 2018).

Original: Unless speciﬁed, the l ∞ adversarial budget used for CIFAR10 dataset (Krizhevsky et al., 2009) 6 is 8 / 255 and for SVHN dataset (Netzer et al., 2011) 7 is 0 .
Revised: Unless speciﬁed, the l ∞ adversarial budget used for CIFAR10 dataset (Krizhevsky et al., 2009) 6 is 8 / 255 and for SVHN dataset (Netzer et al., 2011) 7 is 0 .

Original: In PGD adversarial training, the step size is 2 / 255 for CIFAR10 and 0 .for
Revised: In PGD adversarial training, the step size is 2 / 255 for CIFAR10 and 0 .for

Original: SVHN; PGD is run for 10 iterations for both datasets.
Revised: SVHN; PGD is run for 10 iterations for both datasets.

Original: For adversarial attacks using a different adversarial budget, the step size is always 1 / 4 of the adversarial budget’s size, and we always run it for 10 iterations.
Revised: For adversarial attacks using a different adversarial budget, the step size is always 1 / 4 of the adversarial budget’s size, and we always run it for 10 iterations.

Original: To comprehensively and reliably evaluate the robustness of the model, we use
Revised: To comprehensively and reliably evaluate the robustness of the model, we use

Original: AutoAttack (Croce & Hein, 2020b), which is an ensemble of 4 different attacks: AutoPGD on cross entropy, AutoPGD on difference of logits ratio, fast adaptive boundary (FAB) attack (Croce & Hein, 2020a) and square attack (Andriushchenko et al., 2020).
Revised: AutoAttack (Croce & Hein, 2020b), which is an ensemble of 4 different attacks: AutoPGD on cross entropy, AutoPGD on difference of logits ratio, fast adaptive boundary (FAB) attack (Croce & Hein, 2020a) and square attack (Andriushchenko et al., 2020).

Original: Unless speciﬁed, we use stochastic gradient descent (SGD) with a momentum to optimize the model parameters, we also use weight decay whose factor is 0 . 0005 .
Revised: Unless speciﬁed, we use stochastic gradient descent (SGD) with a momentum to optimize the model parameters, we also use weight decay whose factor is 0 . 0005 .

Original: Unless speciﬁed, the momentum factor is 0 .
Revised: Unless speciﬁed, the momentum factor is 0 .

Original: 9 , the learning rate starts with 0 .
Revised: 9 , the learning rate starts with 0 .

Original: and is divided by 10 in the 1 / 2 and 3 / 4 of the whole training duration.
Revised: and is divided by 10 in the 1 / 2 and 3 / 4 of the whole training duration.

Original: The size of the mini-batch is always 128 .
Revised: The size of the mini-batch is always 128 .

Original: We run the experiments on a machine with 4 NVIDIA TITAN XP GPUs.
Revised: We run the experiments on a machine with 4 NVIDIA TITAN XP GPUs.

Original: It takes about 6 hours to adversarially train a RN18 model for 200 epochs, and a whole day to adversarially train a WRN model for 200 epochs.
Revised: It takes about 6 hours to adversarially train a RN18 model for 200 epochs, and a whole day to adversarially train a WRN model for 200 epochs.

Original: C. S ETTINGS OF E XPERIMENTS IN S ECTION
Revised: C. S ETTINGS OF E XPERIMENTS IN S ECTION

Original: Algorithm 1: One epoch of the accelerated adversarial training we use in Section 6.1.
Revised: Algorithm 1: One epoch of the accelerated adversarial training we use in Section 6.1.

Original: Sample a mini batch {
Revised: Sample a mini batch {

Original: Bi
Revised: Bi

Original: ∼ D do if then else end if
Revised: ∼ D do if then else end if

Original: Calculate the loss end for
Revised: Calculate the loss end for

Original: Fast Adversarial Training
Revised: Fast Adversarial Training

Original: Our experiment on fast adversarial training is on CIFAR10 and (cid:15)
Revised: Our experiment on fast adversarial training is on CIFAR10 and (cid:15)

Original: = 8 / 255 .
Revised: = 8 / 255 .

Original: The pseudocode of our method is demonstrated as Algorithm 1.
Revised: The pseudocode of our method is demonstrated as Algorithm 1.

Original: We use ATTA (Zheng et al., 2020) to initialize the perturbation of each training instance.
Revised: We use ATTA (Zheng et al., 2020) to initialize the perturbation of each training instance.

Original: The step size α of the perturbation update is 4 / 255 , same as (Zheng et al., 2020).
Revised: The step size α of the perturbation update is 4 / 255 , same as (Zheng et al., 2020).

Original: The average coefﬁcient ρ and β is 0 .
Revised: The average coefﬁcient ρ and β is 0 .

Original: 9 and 0 .
Revised: 9 and 0 .

Original: 1 unless explicitly stated.
Revised: 1 unless explicitly stated.

Original: Our learning rate scheduler also follows (Zheng et al., 2020): we train the model for 38 epochs, the learning rate is 0 .
Revised: Our learning rate scheduler also follows (Zheng et al., 2020): we train the model for 38 epochs, the learning rate is 0 .

Original: 1 on the ﬁrst 30 epochs, it decays to 0 .
Revised: 1 on the ﬁrst 30 epochs, it decays to 0 .

Original: 01 in the next 6 epochs and further decays to 0 .
Revised: 01 in the next 6 epochs and further decays to 0 .

Original: 001 in the last 2 epochs.
Revised: 001 in the last 2 epochs.

Original: When we use adaptive targets, the ﬁrst 5 epochs are the warmup period in which we use ﬁxed targets.
Revised: When we use adaptive targets, the ﬁrst 5 epochs are the warmup period in which we use ﬁxed targets.

Original: Since the goal here is to accelerate adversarial training, we do not use a validation set to do model selection as in (Rice et al., 2020).
Revised: Since the goal here is to accelerate adversarial training, we do not use a validation set to do model selection as in (Rice et al., 2020).

Original: We use the standard data augmentation on CIFAR10: random crop and random horizontal ﬂip.
Revised: We use the standard data augmentation on CIFAR10: random crop and random horizontal ﬂip.

Original: Adversarial Finetuning with Additional Data For CIFAR10, we use 500000 images from
Revised: Adversarial Finetuning with Additional Data For CIFAR10, we use 500000 images from

Original: Million Tiny Images dataset (Torralba et al., 2008) with pseudo labels in (Carmon et al., 2019) 8 .
Revised: Million Tiny Images dataset (Torralba et al., 2008) with pseudo labels in (Carmon et al., 2019) 8 .

Original: For
Revised: For

Original: SVHN, we use the extra held-out set provided by SVHN itself, which contains 531131 somewhat less difﬁcult samples.
Revised: SVHN, we use the extra held-out set provided by SVHN itself, which contains 531131 somewhat less difﬁcult samples.

Original: When we construct a mini-batch, half of its instances are sampled from the original training set and the other half are sampled from the additional data.
Revised: When we construct a mini-batch, half of its instances are sampled from the original training set and the other half are sampled from the additional data.

Original: The learning rate in the ﬁne-tuning phase is always 10 − 3 and is always ﬁxed.
Revised: The learning rate in the ﬁne-tuning phase is always 10 − 3 and is always ﬁxed.

Original: Since we only ﬁne-tune the model for only 1 or 5 epochs, we do not use a validation set for model selection.
Revised: Since we only ﬁne-tune the model for only 1 or 5 epochs, we do not use a validation set for model selection.

Original: D A DDITIONAL E XPERIMENTS AND D ISCUSSION D. P ROPERTIES OF THE D IFFICULTY M ETRIC
Revised: D A DDITIONAL E XPERIMENTS AND D ISCUSSION D. P ROPERTIES OF THE D IFFICULTY M ETRIC

Original: To study the factors affecting the difﬁculty function deﬁned in (1), let us denote by d 1 , d 2 the difﬁculty functions obtained under two different training settings, such as different network architectures or adversarial attack strategies.
Revised: To study the factors affecting the difﬁculty function deﬁned in (1), let us denote by d 1 , d 2 the difﬁculty functions obtained under two different training settings, such as different network architectures or adversarial attack strategies.

Original: We then deﬁne the difﬁculty distance ( D-distance ) between two such functions in the following equation.
Revised: We then deﬁne the difﬁculty distance ( D-distance ) between two such functions in the following equation.

Original: In this regard, the expected D-distance between two random difﬁculty functions is 0 . 375 .
Revised: In this regard, the expected D-distance between two random difﬁculty functions is 0 . 375 .

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: We then study the properties of the difﬁculty metric of Equation (1) by performing experiments on the CIFAR10 and CIFAR10-C (Hendrycks & Dietterich, 2019) dataset, varying factors of interest.
Revised: We then study the properties of the difﬁculty metric of Equation (1) by performing experiments on the CIFAR10 and CIFAR10-C (Hendrycks & Dietterich, 2019) dataset, varying factors of interest.

Original: In particular, we ﬁrst study the inﬂuence of the network by using either a RN18 model, trained for either 100 or 200 epochs (RN18-100 or RN18-200), or a WRN34 model trained for 200 epochs (WRN34).
Revised: In particular, we ﬁrst study the inﬂuence of the network by using either a RN18 model, trained for either 100 or 200 epochs (RN18-100 or RN18-200), or a WRN34 model trained for 200 epochs (WRN34).

Original: To generate adversarial attacks, we make use of PGD with an adversarial budget based on the l ∞ norm with (cid:15)
Revised: To generate adversarial attacks, we make use of PGD with an adversarial budget based on the l ∞ norm with (cid:15)

Original: = 8 / 255 .
Revised: = 8 / 255 .

Original: This corresponds to the settings used in other works (Hendrycks & Dietterich, 2019; Madry et al., 2018).
Revised: This corresponds to the settings used in other works (Hendrycks & Dietterich, 2019; Madry et al., 2018).

Original: The other hyper-parameters follow the general settings in Appendix C
Revised: The other hyper-parameters follow the general settings in Appendix C

Original: In the left part of Table 4, we report the D-distance for all pairs of settings.
Revised: In the left part of Table 4, we report the D-distance for all pairs of settings.

Original: Each result is averaged over 4 runs, and the variances are all below 0 .
Revised: Each result is averaged over 4 runs, and the variances are all below 0 .

Original: The D-distances in all scenarios are all very small and close to 0 , indicating the architecture and the training duration have little inﬂuence on instance difﬁculty based on our deﬁnition.
Revised: The D-distances in all scenarios are all very small and close to 0 , indicating the architecture and the training duration have little inﬂuence on instance difﬁculty based on our deﬁnition.

Original: [Table]
Revised: [Table]

Original: We then perform experiments by varying the attack strategy using a RN18 network.
Revised: We then perform experiments by varying the attack strategy using a RN18 network.

Original: As shown by the D-distances reported in the right portion of Table 4, the discrepancy between values obtained with clean, FGSM-perturbed and PGD-perturbed inputs is much larger, thus indicating that our difﬁculty function correctly reﬂects the inﬂuence of an attack on an instance.
Revised: As shown by the D-distances reported in the right portion of Table 4, the discrepancy between values obtained with clean, FGSM-perturbed and PGD-perturbed inputs is much larger, thus indicating that our difﬁculty function correctly reﬂects the inﬂuence of an attack on an instance.

Original: In addition, Tabledemonstrates the D-distance between the difﬁculty functions based on clean instances, FGSMperturbed instance, PGD-perturbed instances and different common corruptions from CIFAR10C (Hendrycks & Dietterich, 2019) 9 .
Revised: In addition, Tabledemonstrates the D-distance between the difﬁculty functions based on clean instances, FGSMperturbed instance, PGD-perturbed instances and different common corruptions from CIFAR10C (Hendrycks & Dietterich, 2019) 9 .

Original: Note that (Hendrycks & Dietterich, 2019) only provides corrupted instances on the test set, so we train models on the clean training set and test model on corrupted test set in these cases.
Revised: Note that (Hendrycks & Dietterich, 2019) only provides corrupted instances on the test set, so we train models on the clean training set and test model on corrupted test set in these cases.

Original: We use RN18 architecture and train it for 100 epochs in all cases, results are reported on the test set.
Revised: We use RN18 architecture and train it for 100 epochs in all cases, results are reported on the test set.

Original: Compared with the results in the left half of Table 4, the D-distance is much larger here.
Revised: Compared with the results in the left half of Table 4, the D-distance is much larger here.

Original: This indicates the difﬁculty function depends on the perturbation type applied to the input, including the common corruptions.
Revised: This indicates the difﬁculty function depends on the perturbation type applied to the input, including the common corruptions.

Original: The results in Table 4 and 5 demonstrate that our difﬁculty metric mainly depends on the data and on the perturbation type; not the model architecture or the training duration.
Revised: The results in Table 4 and 5 demonstrate that our difﬁculty metric mainly depends on the data and on the perturbation type; not the model architecture or the training duration.

Original: [Table]
Revised: [Table]

Original: [Table] 
Revised: [Table] [Table] [Table] [Table]

Original: [Table] [Table] 
Revised: [Table] [Table] [Table]

Original: [Table] [Table] 
Revised: [Table] [Table] [Table]

Original: [Table] 
Revised: [Table] [Table] [Table] [Figure]

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: D i ff i c u l t y M e a s u r e d b y A v e r a g e 0E rr o r 0. 1.0 Difficulty Measured by Average Loss 0.

Original: [Table]
Revised: D i ff i c

Original: [Table]
Revised: u l t y M e a s u r e d b y A v e r a g e 0E rr o r In the deﬁnition of our difﬁculty metric in Equation (1), the difﬁculty of one instance is based on its average loss values during the training procedure. It is intuitive, because the values of the loss objective represents the cost that model needs to ﬁt the corresponding data point.

Original: [Table]
Revised: The bigger this cost is, the more difﬁculty this instance will be. To make the metric stable and prevent the metric from being sensitive to the stochasticity in the training dynamics, we use the average value of the loss objective for each instance to deﬁne its difﬁculty. In addition to the average loss objectives, we can also use the average 0-1 error to deﬁne the difﬁculty function. In Figure 3, we plot the relationship between the difﬁculty metric based on the average loss values and the one based on the average 0-1 error for instances in the CIFAR10 training set when we train

Original: [Table]
Revised: RN18-100 model and WRN34 model. We can see a strong correlation between them for both models. The correlation of the difﬁculty measured by two metrics for the same instance is 0 .

Original: [Table]
Revised: 9466 in the RN18-100 case and 0 . 9545 in the WRN34 case. The high correlation indicates we can use either metric to measure the difﬁculty. Since the loss objective values are continuous and ﬁner-grained, we choose it as the basis of the difﬁculty function we use in this paper.

Original: D. E XAMPLES OF E ASY AND H ARD I NSTANCES OF CIFAR10 AND SVHN
Revised: D. E XAMPLES OF E ASY AND H ARD I NSTANCES OF CIFAR10 AND SVHN

Original: Figure 14 and 15 demonstrate the easiest and the hardest examples of each class from CIFAR10 and
Revised: Figure 20 and 21 demonstrate the easiest and the hardest examples of each class from CIFAR10 and

Original: SVHN, respectively.
Revised: SVHN, respectively.

Original: The difﬁculty of these instances is calculated based on PGD attacks.
Revised: The difﬁculty of these instances is calculated based on PGD attacks.

Original: We can see most easy examples are visually consistent, while most hard examples are ambiguous or even incorrectly labeled.
Revised: We can see most easy examples are visually consistent, while most hard examples are ambiguous or even incorrectly labeled.

Original: D. T RAINING ON A S UBSET
Revised: D. T RAINING ON A S UBSET

Original: Different Optimization Method on Hard Instances
Revised: Different Optimization Method on Hard Instances

Original: We ﬁnd the failure of PGD adversarial training on the hardest 10000 instances of CIFAR10 training set does not arise from the optimizers.
Revised: We ﬁnd the failure of PGD adversarial training on the hardest 10000 instances of CIFAR10 training set does not arise from the optimizers.

Original: In
Revised: In

Original: Figure 3, we use SGD with different initial learning rates (“SGD, lr=1e-2” and “SGD, lr=1e-3”) and the adaptive optimizer such as Adam (Kingma & Ba, 2014) (“Adam, lr=1e-4”).
Revised: Figure 5, we use SGD with different initial learning rates (“SGD, lr=1e-2” and “SGD, lr=1e-3”) and the adaptive optimizer such as Adam (Kingma & Ba, 2014) (“Adam, lr=1e-4”).

Original: The learning rate of
Revised: The learning rate of

Original: SGD optimizer decrease to its 1 / 10 in the 100th and 150th epoch, while the learning rate of Adam optimizer is ﬁxed during training.
Revised: SGD optimizer decrease to its 1 / 10 in the 100th and 150th epoch, while the learning rate of Adam optimizer is ﬁxed during training.

Original: Although optimizers like Adam can make the model ﬁt the training subset better, none of these methods can make the robust test accuracy signiﬁcantly above the chance of random guesses, i.e., 10% .
Revised: Although optimizers like Adam can make the model ﬁt the training subset better, none of these methods can make the robust test accuracy signiﬁcantly above the chance of random guesses, i.e., 10% .

Revised: Longer Training Duration In Figures 7 and 8, we conduct adversarial training for a longer duration until the loss on the hard training instances converges.

Revised: Speciﬁcally, the model is trained forepochs, with a learning rate is initialized to 0 .

Revised: 1 and divided by 10 after every 100 epochs.

Original: Results on SVHN dataset Figure 4 demonstrates the learning curves of PGD adversarial trainingbased ona subset of the easiest, the random and  the hardest instances of SVHN dataset.
Revised: In Figure 7, we adversarially train a RN18 model on the hardest 10000 training instances. Our conclusions from Section 4.1 still hold: Adversarial training on the hard instances leads to much more severe overﬁtting, greatly widening the generalization gap. In Figure 8, we adversarially train a RN18 model on the whole training set and calculate the average loss on the groups G 0 , G 3 , G 6 and G 9 .

Revised: Similarly to training for 200 epochs, the model ﬁrst ﬁts the easy training instances and then the hard ones.

Revised: This can be seen by the fact that the average loss of G 9 decreases much faster in the beginning and quickly saturates.

Revised: In other words, the harder the group, the later we see a signiﬁcant decrease in its average loss value.

Revised: This observation is also consistent with our ﬁndings in Section 4.2.

Revised: Results on SVHN dataset Figure 6 demonstrates the learning curves of PGD adversarial training based on a subset of the easiest, the random and the hardest instances of SVHN dataset.

Original: We let the size of each subset be 20000 , because the training set of SVHN is larger than that of CIFAR10.
Revised: We let the size of each subset be 20000 , because the training set of SVHN is larger than that of CIFAR10.

Original: The model architecture is RN18 in these cases.
Revised: The model architecture is RN18 in these cases.

Original: We have the same observations here: training on the hardest subset yields trivial performance, training on the random subset has signiﬁcant generalization decay in the late phase of training while training on the easiest subset does not.
Revised: We have the same observations here: training on the hardest subset yields trivial performance, training on the random subset has signiﬁcant generalization decay in the late phase of training while training on the easiest subset does not.

Original: Different Values of (cid:15) and l 2 -based Adversarial Budget Figure 5 and Figure 6 demonstrate the learning curves of RN18 models under different adversarial budgets on CIFAR10, in both l ∞ and lcases.
Revised: Different Values of (cid:15) and l 2 -based Adversarial Budget Figure 4 and Figure 9 demonstrate the learning curves of RN18 models under different adversarial budgets on CIFAR10, in both l ∞ and lcases.

Original: In l ∞ cases, the adversarial budgets are 2 / 255 , 4 / 255 and 6 / 255 ; in l 2 cases, the adversarial budgets are 0 . 5 , 0 .
Revised: In l ∞ cases, the adversarial budgets are 2 / 255 , 4 / 255 and 6 / 255 ; in l 2 cases, the adversarial budgets are 0 . 5 , 0 .

Original: 75 and 1 .
Revised: 75 and 1 .

Original: With the increase in the size of the adversarial budget, we can see a clear transition from the vanilla training: more and more severe generalization decay when training on the random or the hardest 
Revised: With the increase in the size of the adversarial budget, we can see a clear transition from the vanilla training: more and more severe generalization decay when training on the random or the hardest subset.

Original: subset Training with Different Amount of Data In Figure 7, we compare the learning curves of PGD adversarial training on increasing more training data, with the easiest ones coming ﬁrst.
Revised: Training with Different Amount of Data In Figure 10, we compare the learning curves of PGD adversarial training on increasing more training data, with the easiest ones coming ﬁrst.

Original: If we do model selection on a validation set as in (Rice et al., 2020), the selected models are still better on both CIFAR10 and SVHN when they are trained with more data, although the ﬁnal models in these cases are not necessarily better.
Revised: If we do model selection on a validation set as in (Rice et al., 2020), the selected models are still better on both CIFAR10 and SVHN when they are trained with more data, although the ﬁnal models in these cases are not necessarily better.

Original: The results indicate the hard instances are still useful to improve the model’s performance, but we need to utilize them in a different way.
Revised: The results indicate the hard instances are still useful to improve the model’s performance, but we need to utilize them in a different way.

Original: [Figure]
Revised: [Figure]

Original: [Figure][Figure] [Figure]
Revised: [Figure]

Original: We conduct numerical analysis to validate Theorem 2 of Section 5.2. Calculating the exact Lipschitz constant of a neural network models is NP-hard, so we utilize the method introduced in (Scaman &
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Lipschitz in l

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Lipschitz in l [Figure] [Table]
Revised: D. L IPSCHITZ A NALYSIS Training Set

Original: [Table]
Revised: [Table]

Original: = 2 /(cid:15) = 4 /(cid:15) = 8 /(cid:15)
Revised: Lipschitz

Original: [Table]
Revised: [Table]

Original: (cid:15)

Original: [Table]
Revised: [Table]

Original: (cid:15)

Original: [Table]
Revised: [Table]

Original: Easy10K

Original: [Table]
Revised: [Table]

Original: [Table]

Original: [Table]
Revised: [Table]

Original: 91 Random10K [Table] [Table]
Revised: We conduct numerical analysis to validate Theorem 2 of Section 5.2.

Original: [Table] [Table] [Table] 34 Hard10K 72.
Revised: Calculating the exact Lipschitz constant of a neural network models is NP-hard, so we utilize the method introduced in (Scaman &

Original: Virmaux, 2018) to derive the upper bound of Lipschitz constants instead.
Revised: Virmaux, 2018) to derive the upper bound of Lipschitz constants instead.

Original: We use CIFAR10 as the dataset and RN18 as the network architecture.
Revised: We use CIFAR10 as the dataset and RN18 as the network architecture.

Original: Table 6 demonstrates the upper bound of the Lipschitz constant of models trained in different settings.
Revised: Table 6 demonstrates the upper bound of the Lipschitz constant of models trained in different settings.

Original: In the l ∞ cases, we set (cid:15) to be 2 / 255 , 4 / 255 and 8 / 255 ; in the l 2 cases, we set (cid:15) to be 0 . 5 , 0 .
Revised: In the l ∞ cases, we set (cid:15) to be 2 / 255 , 4 / 255 and 8 / 255 ; in the l 2 cases, we set (cid:15) to be 0 . 5 , 0 .

Original: 75 and 1 .
Revised: 75 and 1 .

Original: Due to the stochasticity introduced by power method, we run the algorithm in (Scaman & Virmaux, 2018) for 20 times and report the average, we ﬁnd the variance is negligible.
Revised: Due to the stochasticity introduced by power method, we run the algorithm in (Scaman & Virmaux, 2018) for 20 times and report the average, we ﬁnd the variance is negligible.

Original: Based on the results in
Revised: Based on the results in

Original: Table 6, it is clear that models adversarially trained by the hard training instances have much larger
Revised: Table 6, it is clear that models adversarially trained by the hard training instances have much larger

Original: Lipschitz constant than the ones by the easy training instances in all cases.
Revised: Lipschitz constant than the ones by the easy training instances in all cases.

Original: Figure 8 demonstrates the curves of the Lipschitz upper bound when the model is adversarially trained by the easiest, the random and the hardest 10000 instances.
Revised: Figure 11 demonstrates the curves of the Lipschitz upper bound when the model is adversarially trained by the easiest, the random and the hardest 10000 instances.

Original: The adversarial budget is l ∞ norm based and (cid:15)
Revised: The adversarial budget is l ∞ norm based and (cid:15)

Original: = 8 / 255 .
Revised: = 8 / 255 .

Original: We can clearly see that as the training goes, the Lipschitz upper bound increases in all cases, which is consistent with our analysis in Section 5.2.
Revised: We can clearly see that as the training goes, the Lipschitz upper bound increases in all cases, which is consistent with our analysis in Section 5.2.

Original: In addition, compared with training on easy instances, the Lipschitz upper bound of the models adversarially trained on hard instances increases much faster.[Figure]
Revised: In addition, compared with training on easy instances, the Lipschitz upper bound of the models adversarially trained on hard instances increases much faster.

Original: D. R EVISITING E XISTING M ETHODS M ITIGATING A DVERSARIAL O VERFITTING
Revised: D. R EVISITING E XISTING M ETHODS M ITIGATING A DVERSARIAL O VERFITTING

Original: Existing methods mitigating adversarial overﬁtting can be generally divided into two categories: one is to use adaptive inputs, such as (Balaji et al., 2019); the other is to use adaptive targets, such as (Chen et al., 2021b;
Revised: Existing methods mitigating adversarial overﬁtting can be generally divided into two categories: one is to use adaptive inputs, such as (Balaji et al., 2019); the other is to use adaptive targets, such as (Chen et al., 2021b;

Original: Huang et al., 2020).
Revised: Huang et al., 2020).

Original: Both categories aim to prevent the model from ﬁtting hard input-target pairs.
Revised: Both categories aim to prevent the model from ﬁtting hard input-target pairs.

Original: In this section, we pick one example from each category for investigation.
Revised: In this section, we pick one example from each category for investigation.

Revised: We provide the learning curves of the methods we study in Figure 12.

Revised: We use the same hyper-parameters as in these methods’ original paper, except for the training duration and learning rate scheduler, which follow our settings.

Revised: These methods clearly mitigate adversarial overﬁtting: The robust test error does not increase much in the late phase of training, and the generalization gap is much smaller

Revised: that that of

Original:  Instance-Adaptive Training Using an instance-adaptive adversarial budget has been shown to mitigate adversarial overﬁtting and yield a better trade-off between the clean and robust accuracy (Balaji et al., 2019).
Revised: PGD adversarial training. [Figure] [Figure] Instance-Adaptive Training Using an instance-adaptive adversarial budget has been shown to mitigate adversarial overﬁtting and yield a better trade-off between the clean and robust accuracy (Balaji et al., 2019).

Original: In instance-adaptive adversarial training (IAT), each training instance x i maintains its own adversarial budget’s size (cid:15) i during training.
Revised: In instance-adaptive adversarial training (IAT), each training instance x i maintains its own adversarial budget’s size (cid:15) i during training.

Original: In each epoch, (cid:15) i increases to (cid:15) i + (cid:15) ∆ if the instance is robust under this enlarged adversarial budget.
Revised: In each epoch, (cid:15) i increases to (cid:15) i + (cid:15) ∆ if the instance is robust under this enlarged adversarial budget.

Original: By contrast, (cid:15) i decreases to (cid:15) i − (cid:15)
Revised: By contrast, (cid:15) i decreases to (cid:15) i − (cid:15)

Original: ∆ if the instance is not robust under the original adversarial budget.
Revised: ∆ if the instance is not robust under the original adversarial budget.

Original: Here, (cid:15) ∆ is the step size of the adjustment.
Revised: Here, (cid:15) ∆ is the step size of the adjustment.

Original: We use the same settings as in (Balaji et al., 2019) except that we use the same number of training epochs and learning rate scheduling as the one in other experiments for fair comparison.
Revised: We use the same settings as in (Balaji et al., 2019) except that we use the same number of training epochs and learning rate scheduling as the one in other experiments for fair comparison.

Original: Specially, we set the value of (cid:15) and (cid:15) ∆ to be 8 / 255 and 1 . 9 / 255 , respectively, same as in (Balaji et al., 2019).
Revised: Specially, we set the value of (cid:15) and (cid:15) ∆ to be 8 / 255 and 1 . 9 / 255 , respectively, same as in (Balaji et al., 2019).

Original: The ﬁrst 5 epochs are warmup, when we use vanilla adversarial training (Madry et al., 2018).
Revised: The ﬁrst 5 epochs are warmup, when we use vanilla adversarial training (Madry et al., 2018).

Original: Figure 9 demonstrate the relationship between the instancewise adversarial budget (cid:15) i and the corresponding instance’s difﬁculty d ( x i ) .
Revised: Figure 13 demonstrate the relationship between the instancewise adversarial budget (cid:15) i and the corresponding instance’s difﬁculty d ( x i ) .

Original: It is obvious that they are highly correlated: the correlation is 0 . 844 .
Revised: It is obvious that they are highly correlated: the correlation is 0 . 844 .

Original: Therefore, instance-adaptive training adaptively uses smaller adversarial budgets for hard training instances, which prevents the model from ﬁtting hard input-target pairs.[Figure]
Revised: Therefore, instance-adaptive training adaptively uses smaller adversarial budgets for hard training instances, which prevents the model from ﬁtting hard input-target pairs.

Original: The model architecture is RN18 and the value of (cid:15) is

Original: [Figure] Self-Adaptive Training Self-adaptive training (SAT) (Huang et al., 2020) solves the adversarial overﬁtting issue by adapting the target.
Revised: Self-Adaptive Training Self-adaptive training (SAT) (Huang et al., 2020) solves the adversarial overﬁtting issue by adapting the target.

Original: By contrast with common practice consisting of using a ﬁxed target, usually the ground-truth, SAT adapts the target of each instance to the model’s output.
Revised: By contrast with common practice consisting of using a ﬁxed target, usually the ground-truth, SAT adapts the target of each instance to the model’s output.

Original: Speciﬁcally, after a warm-up period, the target t i for an instance x i is initialized as a one-hot vector by its ground-truth label y i and updated in an iterative manner after each epoch as t i ← ρ t i + (1 − ρ )
Revised: Speciﬁcally, after a warm-up period, the target t i for an instance x i is initialized as a one-hot vector by its ground-truth label y i and updated in an iterative manner after each epoch as t i ← ρ t i + (1 − ρ )

Original: o i .
Revised: o i .

Original: Here, ρ is a predeﬁned momentum factor and o i is the output probability of the current model on the corresponding clean instance.
Revised: Here, ρ is a predeﬁned momentum factor and o i is the output probability of the current model on the corresponding clean instance.

Original: SAT uses the loss of TRADES (Zhang et al., 2019b) but replaces the ground-truth label y with the adaptive target t i : L SAT ( x i ) =
Revised: SAT uses the loss of TRADES (Zhang et al., 2019b) but replaces the ground-truth label y with the adaptive target t i : L SAT ( x i ) =

Original: L ( x i , t i ) + λ max ∆ i ∈S ( (cid:15) )
Revised: L ( x i , t i ) + λ max ∆ i ∈S ( (cid:15) )

Original: KL ( o i || o (cid:48) i ) , where KL refers to the Kullback–Leibler divergence and λ is the weight for the regularizer.
Revised: KL ( o i || o (cid:48) i ) , where KL refers to the Kullback–Leibler divergence and λ is the weight for the regularizer.

Original: Furthermore, SAT uses a weighted average to calculate the loss of a mini-batch; the weight assigned to each instance x i is proportional to the maximum element of its target t i but normalized to ensure that all instances’ weights sum up to 1 .
Revised: Furthermore, SAT uses a weighted average to calculate the loss of a mini-batch; the weight assigned to each instance x i is proportional to the maximum element of its target t i but [Figure] [Figure]

Revised: normalized to ensure that all instances’ weights sum up to 1 .

Original: By weighted averaging, the instances with conﬁdent predictions are strengthened, whereas the ambiguous instances are downplayed.
Revised: By weighted averaging, the instances with conﬁdent predictions are strengthened, whereas the ambiguous instances are downplayed.

Original: Similarly, we use the same settings as in (Huang et al., 2020) except we use the same number of training epochs and learning rate scheduling: we train the model for 200 epochs and the ﬁrstepochs are the warmup period.
Revised: Similarly, we use the same settings as in (Huang et al., 2020) except we use the same number of training epochs and learning rate scheduling: we train the model for 200 epochs and the ﬁrstepochs are the warmup period.

Original: Figure 10 demonstrates the average weight assigned to instances belonging to the group G 0 , G 3 , Gand G 9 .
Revised: Figure 14 demonstrates the average weight assigned to instances belonging to the group G 0 , G 3 , Gand G 9 .

Original: It is clear that the hard instances are assigned much lower weights than the easy instances.
Revised: It is clear that the hard instances are assigned much lower weights than the easy instances.

Original: For example, the weight assigned to G 0 , the easiest 10% training instances, is close to 1 , while the weight assigned to G 9 , the hardest 10% training instances, is only around 0 .
Revised: For example, the weight assigned to G 0 , the easiest 10% training instances, is close to 1 , while the weight assigned to G 9 , the hardest 10% training instances, is only around 0 .

Original: Furthermore, Figure 11 shows the accuracy of the group G 0 , G 3 , G 6 and G 9 during training using the original ground-truth label 1 y and the adaptive target t , respectively.
Revised: Furthermore, Figure 15 shows the accuracy of the group G 0 , G 3 , G 6 and G 9 during training using the original ground-truth label 1 y and the adaptive target t , respectively.

Original: For the adaptive target, one adversarial instance x (cid:48) is considered to be correctly classiﬁed if and only if arg max i f w ( x (cid:48) )
Revised: For the adaptive target, one adversarial instance x (cid:48) is considered to be correctly classiﬁed if and only if arg max i f w ( x (cid:48) )

Original: i = arg max
Revised: i = arg max

Original: i t i .
Revised: i t i .

Original: For easy instances, 1 y is mostly close to t , so accuracy in both cases is high and the gap between them is small.
Revised: For easy instances, 1 y is mostly close to t , so accuracy in both cases is high and the gap between them is small.

Original: For hard instances, 1 y is usually not consistent with t , while accuracy under the adaptive target t is much higher than the group-truth label y .
Revised: For hard instances, 1 y is usually not consistent with t , while accuracy under the adaptive target t is much higher than the group-truth label y .

Original: This indicates self-adaptive training makes adaptive target easier to ﬁt for the originally hard instances.
Revised: This indicates self-adaptive training makes adaptive target easier to ﬁt for the originally hard instances.

Original: [Figure]
Revised: [Figure]

Original: D. E XTRA R ESULTS AND D ISCUSSION ON A DVERSARIAL F INETUNING

Original: We conduct ablation study and the results are demonstrated in Table 7.

Original: It is clear that both reweighting and the KL regularization term beneﬁt the performance of the ﬁnetuned model.

Original: Duration Method

Original: AutoAttack Duration Method

Original: AutoAttack

Original: WRN34 on CIFAR10, (cid:15) = 8 /

Original: RN18 on SVHN, (cid:15) = 0 .

Original: No Fine Tuning 52.

Original: No Fine Tuning

Original: [Table]

Original: D. E XTRA R ESULTS AND D ISCUSSION ON F AST A DVERSARIAL T RAINING
Revised: D. E XTRA R ESULTS AND D ISCUSSION ON F AST A DVERSARIAL T RAINING

Original: We also conduct ablation study in the context of fast adversarial training.
Revised: We also conduct ablation study in the context of fast adversarial training.

Original: In Figure 12, we change the value of β in our algorithm (pseudocode in Algorithm 1) and plot the learning curves.
Revised: In Figure 16, we change the value of β in our algorithm (pseudocode in Algorithm 1) and plot the learning curves.

Original: Lower the value of β is, more weights assigned to the adaptive part of the target: β = 0 means we directly utilize the moving average target as the ﬁnal target, β = 1 means we use the one-hot groundtruth label.
Revised: Lower the value of β is, more weights assigned to the adaptive part of the target: β = 0 means we directly utilize the moving average target as the ﬁnal target, β = 1 means we use the one-hot groundtruth [Figure] [Figure] [Figure]

Original:  Figure 12 clearly shows us the generalization gap decreases with the decrease in β .
Revised: [Figure] label. Figure 16 clearly shows us the generalization gap decreases with the decrease in β .

Original: That is to say, the adaptive target can indeed improve the generalization performance.
Revised: That is to say, the adaptive target can indeed improve the generalization performance.

Original: Figure 13 compare the learning curves of ATTA (Zheng et al., 2020) with and without reweighting.
Revised: Figure 17 compare the learning curves of ATTA (Zheng et al., 2020) with and without reweighting.

Original: The ﬁrst 5 epochs are the warmup period.
Revised: The ﬁrst 5 epochs are the warmup period.

Original: The results conﬁrm that the reweighting scheme can prevent adversarial overﬁtting and decrease the generalization gap.
Revised: The results conﬁrm that the reweighting scheme can prevent adversarial overﬁtting and decrease the generalization gap.

Revised: To conﬁrm that the algorithm we use is consistent with our theoretical and empirical analysis in

Revised: Section 4 and 5, we study the relationship between the instance difﬁculty and the weight assigned to them when using reweighting, as well as the soft target when using adaptive targets.

Original: [Figure]
Revised: Since the evaluation of model robustness is based on the PGD attack, the difﬁculty value here is based on the PGD perturbation. In Figure 18, we demonstrate the relationship between the difﬁculty value and the average assigned weight for each instance when using reweighting. We calculate the correlation between these two values on the training set, it is 0 .

Revised: This indicates we indeed assign smaller weights for hard training instances and assign bigger weights for easy training instances.

Original: [Figure]
Revised: In Figure 19, we show the relationship between the difﬁculty value and the average value of the true label’s probability in the soft target when we use the adaptive targets. Similarly, we calculate the correlation between these two values on the training set, it is 0 . 9604 . This indicates the adaptive target is similar to the ground-truth one-hot target for the easy training instances, while the adaptive target is very different from the ground-truth one-hot target for the hard training instances. This means, adaptive targets prevent the model from ﬁtting hard training instances while encourage the model to ﬁt the easy training instances.

Revised: D. E XTRA R ESULTS AND D ISCUSSION ON A DVERSARIAL F INETUNING

Revised: We conduct ablation study and the results are demonstrated in Table 7.

Revised: It is clear that both reweighting and the KL regularization term beneﬁt the performance of the ﬁnetuned model

Original: [Table]
Revised: [Table]

Revised: [Table]

Original: 0.995 truck truck, 0.994 truck truck, 0.993 truck truck, 0.993 truck truck, 0.992 truck truck, 0.992 truck truck, 0.992 truck truck, 0.992 truck truck, 0.991 truck truck, 0.991 truck (s) Truck, Easy. truck, 0.001 plane truck, 0.002 deer truck, 0.002 bird truck, 0.003 deer truck, 0.003 frog truck, 0.003 deer truck, 0.004 cat truck, 0.005 ship truck, 0.006 ship truck, 0.010 bird truck, 0.011 car truck, 0.011 ship truck, 0.011 deer truck, 0.011 cat truck, 0.013 ship truck, 0.013 plane truck, 0.015 plane truck, 0.016 horse (t)
Revised: 0.995 truck truck, 0.994 truck truck, 0.993 truck truck, 0.993 truck truck, 0.992 truck truck, 0.992 truck truck, 0.992 truck truck, 0.992 truck truck, 0.991 truck truck, 0.991 truck (s) Truck, Easy. truck, 0.001 plane truck, 0.002 deer truck, 0.002 bird truck, 0.003 deer truck, 0.003 frog truck, 0.003 deer truck, 0.004 cat truck, 0.005 ship truck, 0.006 ship truck, 0.010 bird truck, 0.011 car truck, 0.011 ship truck, 0.011 deer truck, 0.011 cat truck, 0.013 ship truck, 0.013 plane truck, 0.015 plane truck, 0.016 horse (t)

Original: [Figure]
Revised: [Figure]

Original: , Hard.
Revised: , Hard.

Original: [Table]
Revised: [Table]

Original: 0.002 1 5, 0.0025, 0.002 1 5, 0.003 1 5, 0.0035, 0.004 1 5, 0.004 1 5, 0.0045, 0.004 2 5, 0.004 6 5, 0.005(l) 5, Hard 6, 0.999 6 6, 0.997 6 6, 0.9976, 0.994 6 6, 0.990 6 6, 0.9906, 0.989 6 6, 0.988 6 6, 0.9886, 0.987 6 6, 0.987 6 6, 0.9866, 0.983 6 6, 0.983 6 6, 0.9816, 0.980 6 6, 0.979 6 6, 0.979(m) 6, Easy 6, 0.003 1 6, 0.003 5 6, 0.0036, 0.003 1 6, 0.005 7 6, 0.0056, 0.005 3 6, 0.006 7 6, 0.0066, 0.006 5 6, 0.007 1 6, 0.0076, 0.008 2 6, 0.009 4 6, 0.0096, 0.009 3 6, 0.009 0 6, 0.009(n) 6, Hard 7, 0.998 7 7, 0.998 7 7, 0.9977, 0.996 7 7, 0.996 7 7, 0.9947, 0.994 7 7, 0.994 7 7, 0.9927, 0.992 7 7, 0.992 7 7, 0.9917, 0.991 7 7, 0.990 7 7, 0.9897, 0.988 7 7, 0.988 7 7, 0.988(o) 7, Easy 7, 0.000 0 7, 0.001 4 7, 0.0027, 0.002 2 7, 0.002 1 7, 0.0037, 0.003 9 7, 0.004 1 7, 0.0047, 0.004 1 7, 0.005 1 7, 0.0057, 0.005 2 7, 0.006 1 7, 0.0067, 0.006 1 7, 0.006 1 7, 0.007(p) 7, Hard 8, 1.000 8 8, 0.999 8 8, 0.9988, 0.994 8 8, 0.992 8 8, 0.9928, 0.992 8 8, 0.988 8 8, 0.9878, 0.987 8 8, 0.984 8 8, 0.9818, 0.981 8 8, 0.980 8 8, 0.9788, 0.977 8 8, 0.976 8 8, 0.974(q) 8, Easy 8, 0.001 4 8, 0.001 1 8, 0.0018, 0.001 4 8, 0.001 3 8, 0.0038, 0.004 1 8, 0.004 5 8, 0.0048, 0.005 1 8, 0.005 0 8, 0.0058, 0.006 2 8, 0.006 1 8, 0.0068, 0.007 4 8, 0.007 1 8, 0.007(r) 8, Hard 9, 0.999 9 9, 0.997 9 9, 0.9969, 0.996 9 9, 0.995 9 9, 0.9939, 0.991 9 9, 0.990 9 9, 0.9889, 0.988 9 9, 0.988 9 9, 0.9889, 0.985 9 9, 0.984 9 9, 0.9839, 0.983 9 9, 0.981 9 9, 0.978(s) 9, Easy 9, 0.000 5 9, 0.001 8 9, 0.0019, 0.003 1 9, 0.003 4 9, 0.0039, 0.003 6 9, 0.004 1 9, 0.0049, 0.005 0 9, 0.005 2 9, 0.0069, 0.006 1 9, 0.007 6 9, 0.0089, 0.009 2 9, 0.010 2 9, 0.011(t) 9, Hard
Revised: 0.002 1 5, 0.0025, 0.002 1 5, 0.003 1 5, 0.0035, 0.004 1 5, 0.004 1 5, 0.0045, 0.004 2 5, 0.004 6 5, 0.005(l) 5, Hard 6, 0.999 6 6, 0.997 6 6, 0.9976, 0.994 6 6, 0.990 6 6, 0.9906, 0.989 6 6, 0.988 6 6, 0.9886, 0.987 6 6, 0.987 6 6, 0.9866, 0.983 6 6, 0.983 6 6, 0.9816, 0.980 6 6, 0.979 6 6, 0.979(m) 6, Easy 6, 0.003 1 6, 0.003 5 6, 0.0036, 0.003 1 6, 0.005 7 6, 0.0056, 0.005 3 6, 0.006 7 6, 0.0066, 0.006 5 6, 0.007 1 6, 0.0076, 0.008 2 6, 0.009 4 6, 0.0096, 0.009 3 6, 0.009 0 6, 0.009(n) 6, Hard 7, 0.998 7 7, 0.998 7 7, 0.9977, 0.996 7 7, 0.996 7 7, 0.9947, 0.994 7 7, 0.994 7 7, 0.9927, 0.992 7 7, 0.992 7 7, 0.9917, 0.991 7 7, 0.990 7 7, 0.9897, 0.988 7 7, 0.988 7 7, 0.988(o) 7, Easy 7, 0.000 0 7, 0.001 4 7, 0.0027, 0.002 2 7, 0.002 1 7, 0.0037, 0.003 9 7, 0.004 1 7, 0.0047, 0.004 1 7, 0.005 1 7, 0.0057, 0.005 2 7, 0.006 1 7, 0.0067, 0.006 1 7, 0.006 1 7, 0.007(p) 7, Hard 8, 1.000 8 8, 0.999 8 8, 0.9988, 0.994 8 8, 0.992 8 8, 0.9928, 0.992 8 8, 0.988 8 8, 0.9878, 0.987 8 8, 0.984 8 8, 0.9818, 0.981 8 8, 0.980 8 8, 0.9788, 0.977 8 8, 0.976 8 8, 0.974(q) 8, Easy 8, 0.001 4 8, 0.001 1 8, 0.0018, 0.001 4 8, 0.001 3 8, 0.0038, 0.004 1 8, 0.004 5 8, 0.0048, 0.005 1 8, 0.005 0 8, 0.0058, 0.006 2 8, 0.006 1 8, 0.0068, 0.007 4 8, 0.007 1 8, 0.007(r) 8, Hard 9, 0.999 9 9, 0.997 9 9, 0.9969, 0.996 9 9, 0.995 9 9, 0.9939, 0.991 9 9, 0.990 9 9, 0.9889, 0.988 9 9, 0.988 9 9, 0.9889, 0.985 9 9, 0.984 9 9, 0.9839, 0.983 9 9, 0.981 9 9, 0.978(s) 9, Easy 9, 0.000 5 9, 0.001 8 9, 0.0019, 0.003 1 9, 0.003 4 9, 0.0039, 0.003 6 9, 0.004 1 9, 0.0049, 0.005 0 9, 0.005 2 9, 0.0069, 0.006 1 9, 0.007 6 9, 0.0089, 0.009 2 9, 0.010 2 9, 0.011(t) 9, Hard