Original: Label Noise in Adversarial Training: A Novel Perspective to Study Robust Overfitting
Revised: Label Noise in Adversarial Training: A Novel Perspective to Study Robust Overﬁtting

Original: Abstract
Revised: Abstract

Original: We show that label noise exists in adversarial training.
Revised: We show that label noise exists in adversarial training.

Original: Such label noise is due to the mismatch between the true label distribution of adversarial examples and the label inherited from clean examples – the true label distribution is distorted by the adversarial perturbation, but is neglected by the common practice that inherits labels from clean examples.
Revised: Such label noise is due to the mismatch between the true label distribution of adversarial examples and the label inherited from clean examples – the true label distribution is distorted by the adversarial perturbation, but is neglected by the common practice that inherits labels from clean examples.

Original: Recognizing label noise sheds insights on the prevalence of robust overfitting in adversarial training, and explains its intriguing dependence on perturbation radius and data quality.
Revised: Recognizing label noise sheds insights on the prevalence of robust overﬁtting in adversarial training, and explains its intriguing dependence on perturbation radius and data quality.

Original: Also, our label noise perspective aligns well with our observations of the epoch-wise double descent in adversarial training.
Revised: Also, our label noise perspective aligns well with our observations of the epoch-wise double descent in adversarial training.

Original: Guided by our analyses, we proposed a method to automatically calibrate the label to address the label noise and robust overfitting.
Revised: Guided by our analyses, we proposed a method to automatically calibrate the label to address the label noise and robust overﬁtting.

Original: Our method achieves consistent performance improvements across various models and datasets without introducing new hyper-parameters or additional tuning.
Revised: Our method achieves consistent performance improvements across various models and datasets without introducing new hyper-parameters or additional tuning.

Original:  Introduction
Revised:  Introduction

Original: Adversarial training (Goodfellow et al., 2015; Huang et al., 2015; Kurakin et al., 2017; Madry et al., 2018) is known as one of the most effective ways (Athalye et al., 2018;
Revised: Adversarial training (Goodfellow et al., 2015; Huang et al., 2015; Kurakin et al., 2017; Madry et al., 2018) is known as one of the most effective ways (Athalye et al., 2018;

Original: Uesato et al., 2018) to enhance the adversarial robustness of deep neural networks (Szegedy et al., 2014; Goodfellow et al., 2015).
Revised: Uesato et al., 2018) to enhance the adversarial robustness of deep neural networks (Szegedy et al., 2014; Goodfellow et al., 2015).

Original: It augments training data with adversarial perturbations to prepare the model for adversarial attacks.
Revised: It augments training data with adversarial perturbations to prepare the model for adversarial attacks.

Original: Despite various efforts to generate more effective adversarial training examples (Ding et al., 2020;
Revised: Despite various efforts to generate more effective adversarial training examples (Ding et al., 2020;

Original: Zhang et al., 2020), the labels assigned to them attracts little attention.
Revised: Zhang et al., 2020), the labels assigned to them attracts little attention.

Original: As the common practice, the assigned labels of adversarial training examples are simply inherited from their clean counterparts.
Revised: As the common practice, the assigned labels of adversarial training examples are simply inherited from their clean counterparts.

Original: In this paper we argue that the existing labeling practice of the adversarial training examples introduces label noise implicitly, since adversarial perturbation can distort the data semantics (Tsipras et al., 2019; Ilyas et al., 2019).
Revised: In this paper we argue that the existing labeling practice of the adversarial training examples introduces label noise implicitly since adversarial perturbation can distort the data semantics (Tsipras et al., 2019; Ilyas et al., 2019).

Original: For example, as illustrated in Figure 1, even with a slight distortion of the data semantics (e.g., more ambiguous), the label distribution of the adversarially perturbed data may not match the label distribution of the clean counterparts.
Revised: For example, as illustrated in Figure 1, even with a slight distortion of the data semantics (e.g., more ambiguous), the label distribution of the adversarially perturbed data may not match the label distribution of the clean counterparts.

Original: Such distribution shift is neglected when assigning labels to adversarial examples, which are directly copied from the clean counterparts.
Revised: Such distribution shift is neglected when assigning labels to adversarial examples, which are directly copied from the clean counterparts.

Original: We observe that distribution mismatch caused by adversarial perturbation along with improper labeling practice will cause label noise in adversarial training .
Revised: We observe that distribution mismatch caused by adversarial perturbation along with improper labeling practice will cause label noise in adversarial training .

Original: It is a mysterious and prominent phenomenon that the robust test error would start to increase after conducting adversarial training for a certain number of epochs (Rice et al., 2020), and our label noise perspective provides an adequate explanation for this phenomenon.
Revised: It is a mysterious and prominent phenomenon that the robust test error would start to increase after conducting adversarial training for a certain number of epochs (Rice et al., 2020), and our label noise perspective provides an adequate explanation for this phenomenon.

Original: Specifically, from a classic bias-variance view of model generalization, label noise that implicitly exists in adversarial training
Revised: Speciﬁcally, from a classic bias-variance view of model generalization, label noise that implicitly exists in adversarial training

Original: [Table]
Revised: [Table]

Original: True
Revised: True

Original: [Table]
Revised: [Table]

Original: /sB6/
Revised: /sB6/

Original: wEAyZJB</latexit>
Revised: wEAyZJB</latexit>

Original: P (˜ Y 0 | x 0 ) <latexit sha1_base64="AhqicuZyvRZ3dmjd8VBTevD9huM=">AAAB+XicbVDLSsNAFJ34rPUVdaebYJHWTUlEUHcFNy5ctGAf0oYymUzaoZNJmLkpltgv8BfcuFDErZ/gH7jzL/wEp4+Fth64cDjnXu69x4s5U2DbX8bC4tLyympmLbu+sbm1be7s1lSUSEKrJOKRbHhYUc4ErQIDThuxpDj0OK17vcuRX+9TqVgkbmAQUzfEHcECRjBoqW2a5UILGPdpejvM39/lj9tmzi7aY1jzxJmSXMmsfH9c7z+U2+Zny49IElIBhGOlmo4dg5tiCYxwOsy2EkVjTHq4Q5uaChxS5abjy4fWkVZ8K4ikLgHWWP09keJQqUHo6c4QQ1fNeiPxP6+ZQHDupkzECVBBJouChFsQWaMYLJ9JSoAPNMFEMn2rRbpYYgI6rKwOwZl9eZ7UTorOafGiotOw0QQZdIAOUQE56AyV0BUqoyoiqI8e0TN6MVLjyXg13iatC8Z0Zg/9gfH+A+rBlig=</latexit> can increase the modelvariance
Revised: P (˜ Y 0 | x 0 ) <latexit sha1_base64="AhqicuZyvRZ3dmjd8VBTevD9huM=">AAAB+XicbVDLSsNAFJ34rPUVdaebYJHWTUlEUHcFNy5ctGAf0oYymUzaoZNJmLkpltgv8BfcuFDErZ/gH7jzL/wEp4+Fth64cDjnXu69x4s5U2DbX8bC4tLyympmLbu+sbm1be7s1lSUSEKrJOKRbHhYUc4ErQIDThuxpDj0OK17vcuRX+9TqVgkbmAQUzfEHcECRjBoqW2a5UILGPdpejvM39/lj9tmzi7aY1jzxJmSXMmsfH9c7z+U2+Zny49IElIBhGOlmo4dg5tiCYxwOsy2EkVjTHq4Q5uaChxS5abjy4fWkVZ8K4ikLgHWWP09keJQqUHo6c4QQ1fNeiPxP6+ZQHDupkzECVBBJouChFsQWaMYLJ9JSoAPNMFEMn2rRbpYYgI6rKwOwZl9eZ7UTorOafGiotOw0QQZdIAOUQE56AyV0BUqoyoiqI8e0TN6MVLjyXg13iatC8Z0Zg/9gfH+A+rBlig=</latexit> can increase the model

Original: (Yang et al. ) andmake the muchmore compared to  noise in  explain the intriguing dependence ofrobust overfitting on the  (Dong et al., 2021b) and data quality (Dong et al., 2021a) presented in the literature 
Revised: (Yang et al., 2020) and much compared to analyses noise in training also explain the intriguing dependence of on the radius (Dong et al., 2021b) and data quality (Dong et al., 2021a) presented in the literature viewed as an early part of the epochwise double descent.

Original: [Figure]
Revised: We employ PGD training (Madry et al., 2018) on CIFAR10 (Krizhevsky, 2009) with Wide ResNet (WRN) (Zagoruyko & Komodakis, 2016) and a ﬁxed learning rate. WRN- 28 - k refers to WRN with depth 28 and widen factor k .

Original: Providing the label noise in adversarial training, one can further expect the existence of double descent based on the modern generalization theory of deep neural networks.
Revised: Providing the label noise in adversarial training, one can further expect the existence of double descent based on the modern generalization theory of deep neural networks.

Original: Epoch-wise double descent refers to the phenomenon that the test error will first decrease and then increase as predicted by the classic bias-variance tradeoff, but it will decrease again as the training continues.
Revised: Epoch-wise double descent refers to the phenomenon that the test error will ﬁrst decrease and then increase as predicted by the classic bias-variance tradeoff, but it will decrease again as the training continues.

Original: Such phenomenon is only reported in standard training of deep neural networks, often requiring significant label noise in the training set (Nakkiran et al., 2020).
Revised: Such phenomenon is only reported in standard training of deep neural networks, often requiring signiﬁcant label noise in the training set (Nakkiran et al., 2020).

Original: As the label noise intrinsically exists in adversarial training, such epoch-wise double descent phenomenon also emerges when the training goes longer.
Revised: As the label noise intrinsically exists in adversarial training, such epoch-wise double descent phenomenon also emerges when the training goes longer.

Original: Indeed, as shown in Figure 2, for a relatively large model such as WRN-28-5, on top of the existing robust overfitting phenomenon, the robust test error will eventually decrease again after 1 , 000 epochs .
Revised: Indeed, as shown in Figure 2, for a relatively large model such as WRN-28-5, on top of the existing robust overﬁtting phenomenon, the robust test error will eventually decrease again after 1 , 000 epochs .

Original: Following Nakkiran et al.
Revised: Following Nakkiran et al.

Original: (2020), we further experiment different model sizes.
Revised: (2020), we further experiment different model sizes.

Original: One can find that a medium-sized model will follow a classic U-curve, which means only overfitting is observed; and the robust test error for a small model will monotonically decrease.
Revised: One can ﬁnd that a medium-sized model will follow a classic U-curve, which means only overﬁtting is observed; and the robust test error for a small model will monotonically decrease.

Original: These are well aligned with the observations in standard training regime.
Revised: These are well aligned with the observations in standard training regime.

Original: This again consolidates our understanding of label noise in adversarial training.
Revised: This again consolidates our understanding of label noise in adversarial training.

Original: In light of our analyses, we design a theoretically-grounded method to mitigate the label noise in adversarial training automatically.
Revised: In light of our analyses, we design a theoretically-grounded method to mitigate the label noise in adversarial training automatically.

Original: The key idea is to resort to an alternative labeling of the adversarial examples.
Revised: The key idea is to resort to an alternative labeling of the adversarial examples.

Original: We show that the predictive label distribution of an adversarially trained probabilistic classifier can approximate the true label distribution with high probability.
Revised: We show that the predictive label distribution of an adversarially trained probabilistic classiﬁer can approximate the true label distribution with high probability.

Original: Thus it can be utilized as a better labeling of the adversarial examples and provably reduce the label noise.
Revised: Thus it can be utilized as a better labeling of the adversarial examples and provably reduce the label noise.

Original: We also show that with proper temperature scaling and interpolation, such predictive label distribution can further reduce the label noise.
Revised: We also show that with proper temperature scaling and interpolation, such predictive label distribution can further reduce the label noise.

Original: This echoes the recent empirical practice of incorporating knowledge distillation (Hinton et al., 2015) into adversarial training (Chen et al., 2021).
Revised: This echoes the recent empirical practice of incorporating knowledge distillation (Hinton et al., 2015) into adversarial training (Chen et al., 2021).

Original: While previous works heuristically select fixed scaling and interpolation parameters for knowledge distillation, we show that it is possible to fully unleash the potential of knowledge distillation by automatically determining the set of parameters that maximally reduces the label noise, with a strategy similar to confidence calibration (Guo et al., 2017).
Revised: While previous works heuristically select ﬁxed scaling and interpolation parameters for knowledge distillation, we show that it is possible to fully unleash the potential of knowledge distillation by automatically determining the set of parameters that maximally reduces the label noise, with a strategy similar to conﬁdence calibration (Guo et al., 2017).

Original: Such strategy can further mitigate robust overfitting to a minimal amount without additional human tuning effort.
Revised: Such strategy can further mitigate robust overﬁtting to a minimal amount without additional human tuning effort.

Original: Extensive experiments on different datasets, training methods, neural architectures and robustness evaluation metrics verify the effectiveness of our method.
Revised: Extensive experiments on different datasets, training methods, neural architectures and robustness evaluation metrics verify the effectiveness of our method.

Original: In summary, our findings and contributions are: 1) we show that the labeling of adversarial examples in adversarial training practice introduces label noise implicitly; 2) we show that robust overfitting can be adequately explained by such label noise, and it is the early part of an epoch-wise double descent; 3) 2e show an alternative labeling of the adversarial examples can be established to provably reduce the label noise and mitigate the robust overfitting.
Revised: In summary, our ﬁndings and contributions are: 1) we show that the labeling of adversarial examples in adversarial training practice introduces label noise implicitly; 2) we show that robust overﬁtting can be adequately explained by such label noise, and it is the early part of an epoch-wise double descent; 3) 2e show an alternative labeling of the adversarial examples can be established to provably reduce the label noise and mitigate the robust overﬁtting.

Original:  Related Work
Revised:  Related Work

Original: Robust overfitting and double descent in adversarial training.
Revised: Robust overﬁtting and double descent in adversarial training.

Original: Double descent refers to the phenomenon that overfitting by increasing model complexity will eventually improve test set performance (Neyshabur et al., 2017; Belkin et al., 2019).
Revised: Double descent refers to the phenomenon that overﬁtting by increasing model complexity will eventually improve test set performance (Neyshabur et al., 2017; Belkin et al., 2019).

Original: This appears to conflict with the robust overfitting phenomenon in adversarial training, where increasing model complexity by training longer will impair test set performance constantly after a certain point during training.
Revised: This appears to conﬂict with the robust overﬁtting phenomenon in adversarial training, where increasing model complexity by training longer will impair test set performance constantly after a certain point during training.

Original: It is thus believed in the literature that robust overfitting and epoch-wise double descent are separate phenomena (Rice et al., 2020).
Revised: It is thus believed in the literature that robust overﬁtting and epoch-wise double descent are separate phenomena (Rice et al., 2020).

Original: In this work we show this is not the complete picture by conducting adversarial training for exponentially more epochs than the typical practice.
Revised: In this work we show this is not the complete picture by conducting adversarial training for exponentially more epochs than the typical practice.

Original: A recent work also considers a different notion of double descent that is defined with respect to the perturbation size (Yu et al., 2021).
Revised: A recent work also considers a different notion of double descent that is deﬁned with respect to the perturbation size (Yu et al., 2021).

Original: Such double descent might be more related to the robustnessaccuracy trade-off problem (Papernot et al., 2016;
Revised: Such double descent might be more related to the robustnessaccuracy trade-off problem (Papernot et al., 2016;

Original: Su et al., 2018; Tsipras et al., 2019; Zhang et al., 2019), rather than the classic understanding of double descent based on model complexity.
Revised: Su et al., 2018; Tsipras et al., 2019; Zhang et al., 2019), rather than the classic understanding of double descent based on model complexity.

Original: Mitigate robust overfitting.
Revised: Mitigate robust overﬁtting.

Original: Robust overfitting hinders the practical deployment of adversarial training methods as the final performance is often sub-optimal.
Revised: Robust overﬁtting hinders the practical deployment of adversarial training methods as the ﬁnal performance is often sub-optimal.

Original: Various regularization methods including classic approaches such as ℓ 1 and ℓ 2 regularization and modern approaches such as cutout (Devries & Taylor, 2017) and mixup (Zhang et al., 2018) have been attempted to tackle robust overfitting, whereas they are shown to perform no better than simply early stopping the training on a validation set
Revised: Various regularization methods including classic approaches such as ` 1 and ` 2 regularization and modern approaches such as cutout (Devries & Taylor, 2017) and mixup (Zhang et al., 2018) have been attempted to tackle robust overﬁtting, whereas they are shown to perform no better than simply early stopping the training on a validation set

Original: (Rice et al., 2020).
Revised: (Rice et al., 2020).

Original: However, early stopping raises additional concern as the best checkpoint of the robust test accuracy and that of the standard accuracy often do not coincide (Chen et al., 2021), thus inevitably sacrificing the performance on either criterion.
Revised: However, early stopping raises additional concern as the best checkpoint of the robust test accuracy and that of the standard accuracy often do not coincide (Chen et al., 2021), thus inevitably sacriﬁcing the performance on either criterion.

Original: Various regularization methods specifically designed for adversarial training are thus proposed to outperform early stopping, including regularization the flatness of the weight loss landscape (Wu et al., 2020; Stutz et al., 2021), introducing low-curvature activation functions (Singla et al., 2021), data-driven augmentations that adds high-quality additional data into the training (Rebuffi et al., 2021) and adopting stochastic weight averaging (Izmailov et al., 2018) and knowledge distillation (Hinton et al., 2015)
Revised: Various regularization methods speciﬁcally designed for adversarial training are thus proposed to outperform early stopping, including regularization the ﬂatness of the weight loss landscape (Wu et al., 2020; Stutz et al., 2021), introducing low-curvature activation functions (Singla et al., 2021), data-driven augmentations that adds high-quality additional data into the training (Rebufﬁ et al., 2021) and adopting stochastic weight averaging (Izmailov et al., 2018) and knowledge distillation (Hinton et al., 2015)

Original: (Chen et al., 2021).
Revised: (Chen et al., 2021).

Original: These methods are likely to suppress the label noise in adversarial training, with the self-distillation framework (i.e. the teacher shares the same architecture as the student model) introduced by (Chen et al., 2021) as a particular example since introducing teacher’s outputs as supervision is almost equivalent to the alternative labeling inspired by our understanding of the origin of label noise in adversarial training.
Revised: These methods are likely to suppress the label noise in adversarial training, with the self-distillation framework (i.e. the teacher shares the same architecture as the student model) introduced by (Chen et al., 2021) as a particular example since introducing teacher’s outputs as supervision is almost equivalent to the alternative labeling inspired by our understanding of the origin of label noise in adversarial training.

Original:  Preliminaries
Revised:  Preliminaries

Original: A statistic model of label noise (Frénay & Verleysen, 2014).
Revised: A statistic model of label noise (Frénay & Verleysen, 2014).

Original: Let X ⊂ R d define the input space equipped with a norm ∥ · ∥ : X → R + and Y=
Revised: Let X ⇢ d deﬁne the input space equipped with a norm k · k : X ! + and Y

Original:  [ K ] := { 1 , 2 , . .
Revised: = [ K ] := { 1 , 2 , . .

Original:  , K } define the label space.
Revised: . , K } deﬁne the label space.

Original: We introduce four random variables to describe noisy labeling process.
Revised: We introduce four random variables to describe noisy labeling process.

Original: Let X ∈ X denote the input,
Revised: Let X 2 X denote the input,

Original: Y ∈ Y denote the true label of the input, ˜ Y ∈ Y denote the assigned label of an input provided by an annotator, and finally E denote the occurrence of a label error by this annotator.
Revised: Y 2 Y denote the true label of the input, ˜ Y 2 Y denote the assigned label of an input provided by an annotator, and ﬁnally E denote the occurrence of a label error by this annotator.

Original: E is a binary random variable with value 1 indicating that the assigned label is different from the true label for a given input, i.e., E = 1 ( ˜ Y ̸ = Y ) .
Revised: E is a binary random variable with value 1 indicating that the assigned label is different from the true label for a given input, i.e., E = 1 ( ˜ Y 6 = Y ) .

Original: We study the case where the label error depends on both the input
Revised: We study the case where the label error depends on both the input

Original: X and the true label Y .
Revised: X and the true label Y .

Original: For a classification problem, a training set consists of a set of examples that are sampled as D = { ( x i , ˜ y i ) } i ∈ [ N ] .
Revised: For a classiﬁcation problem, a training set consists of a set of examples that are sampled as D = { ( x i , ˜ y i ) } i 2 [ N ] .

Original: Definition 3.1 (Label noise) .
Revised: Deﬁnition 3.1 (Label noise) .

Original: We define label noise p e in a training set D as the empirical measure of the label error, namely p e( D ) = 1 /N (cid:80) i ∈
Revised: We deﬁne label noise p e in a training set D as the empirical measure of the label error, namely p e

Original: [ N ] 1 (˜ y i
Revised: [Equation]

Original: ̸

Original: = y i ) . Assumption 3.1.
Revised: Assumption 3.1.

Original: We assume the annotation of a clean dataset involves no label error, namely
Revised: We assume the annotation of a clean dataset involves no label error, namely

Original: P ( E = 1 | Y =
Revised: P ( E = 1 | Y =

Original: y, x ) = 0 .
Revised: y, x ) = 0 .

Original: This directly implies P ( ˜ Y | x ) =
Revised: This directly implies P ( ˜ Y | x ) =

Original: P ( Y | x ) (see proof in the Appendix).
Revised: P ( Y | x ) (see proof in the Appendix).

Original: Definition 3.2 (Data quality) .
Revised: Deﬁnition 3.2 (Data quality) .

Original: Given a training set D , we define its data quality as q ( D ) =E ( x,y ) ∈D P
Revised: Given a training set D , we deﬁne its data quality as q ( D ) =

Original: [Equation] Y = y | x )
Revised: E ( x,y ) 2 D P ( Y = y | x )

Original: Adversarially augmented training set.
Revised: Adversarially augmented training set.

Original: Let f : X → Y be a probabilistic classifier and f ( · ) j be its predictive probability at class j .
Revised: Let f : X ! Y be a probabilistic classiﬁer and f ( · ) j be its predictive probability at class j .

Original: The adversarial example of x generated by f is obtained by solving the maximization problem x ′ = arg max z ∈B ε ( x ) ℓ ( f ( z ) , y ) .
Revised: The adversarial example of x generated by f is obtained by solving the maximization problem x 0 = arg max z 2 B " ( x ) ` ( f ( z ) , y ) .

Original: Here ℓ can be a typical loss function such as cross-entropy.
Revised: Here ` can be a typical loss function such as cross-entropy.

Original: And B ε ( x ) denotes the norm ball centered at x with radius ε , i.e.,
Revised: And B " ( x ) denotes the norm ball centered at x with radius " , i.e.,

Original: B ε ( x ) ={ z ∈ X : ∥
Revised: B " ( x ) =

Original:  z − x ∥ ≤ ε } .
Revised: { z 2 X : k z  x k  " } .

Original: Following previous notations, we denote Y ′ as a random variable representing the true label of x ′ and ˜ Y ′ as a random variable representing the assigned label of x ′ .
Revised: Following previous notations, we denote Y 0 as a random variable representing the true label of xand ˜ Y 0 as a random variable representing the assigned label of x 0 .

Original: We refer D ′{+
+}
Revised: We refer D {+
+}

Original: = { ( x ′ , ˜ y ′ ) } as the adversarially augmented training set.
Revised: = { ( x 0 , ˜ y 0 ) } as the adversarially augmented training set.

Original: Adversarial training.
Revised: training.

Original: Adversarial training can be viewed as a data augmentation technique that trains the parametric classifier f θ on the adversarially augmented training set (Tsipras et al., 2019), namely
Revised: Adversarial training can be viewed as a data augmentation technique that trains the parametric classiﬁer f ✓ on the adversarially augmented training set (Tsipras et al., 2019), namely

Original: [Equation]
Revised: [Equation]

Original:  Label noise implicitly exists in adversarial training
Revised:  Label noise implicitly exists in adversarial training

Original: In this section, we demonstrate the implicit existence of label noise in the adversarially augmented training set.
Revised: In this section, we demonstrate the implicit existence of label noise in the adversarially augmented training set.

Original: We first consider a simple case where the adversarial perturbation is generated based on an ideal classifier that predicts the true label distribution.
Revised: We ﬁrst consider a simple case where the adversarial perturbation is generated based on an ideal classiﬁer that predicts the true label distribution.

Original: Under such a case we prove that the label noise in the adversarially augmented training set is lower-bounded.
Revised: Under such a case we prove that the label noise in the adversarially augmented training set is lower-bounded.

Original: We then show that in realistic cases an adversarially trained classifier can approximate the true label distribution with high probability.
Revised: We then show that in realistic cases an adversarially trained classiﬁer can approximate the true label distribution with high probability.

Original: Therefore, additional error terms will be required to lower bound the label noise.
Revised: Therefore, additional error terms will be required to lower bound the label noise.

Original: All proofs for the remainder of this paper are provided in the appendix.
Revised: All proofs for the remainder of this paper are provided in the appendix.

Original: 4.1 When adversarial perturbation is generated by the true probabilistic classifier
Revised: 4.1 When adversarial perturbation is generated by the true probabilistic classiﬁer

Original: We first consider an ideal case where the adversarial perturbation is generated by the true probabilistic classifier f ( x ) :
Revised: We ﬁrst consider an ideal case where the adversarial perturbation is generated by the true probabilistic classiﬁer f ( x ) :

Original: = P ( Y | x ) , namely the classifier producing the true label distribution on any input x .
Revised: = P ( Y | x ) , namely the classiﬁer producing the true label distribution on any input x .

Original: The true label distribution is distorted by adversarial perturbation.
Revised: The true label distribution is distorted by adversarial perturbation.

Original: We quantify the mismatch between two probability distributions using the total variation (TV) distance .
Revised: We quantify the mismatch between two probability distributions using the total variation (TV) distance .

Original: Definition 4.1 (TV distance) .
Revised: Deﬁnition 4.1 (TV distance) .

Original: Let A be a collection of the subsets of the label sample space Y .
Revised: Let A be a collection of the subsets of the label sample space Y .

Original: The
Revised: The

Original: TV distance between two probability distributions P ( Y ) and P ( Y ′ ) can be defined as ∥ P ( Y ) − P ( Y ′ ) ∥ TV = sup J ∈A | P ( Y ∈
Revised: TV distance between two probability distributions P ( Y ) and P ( Y 0 ) can be deﬁned as k P ( Y )

Original: [Equation]
Revised: [Equation]

Original: ) − P ( Y ′ ∈ [Equation] ) | . We now show that adversarial perturbation generated by the true probabilistic classifier can induce a mismatch between the true label distributions of clean inputs and their adversarial examples.
Revised: We now show that adversarial perturbation generated by the true probabilistic classiﬁer can induce a mismatch between the true label distributions of clean inputs and their adversarial examples.

Original: For simplicity we consider adversarial perturbation based on FGSM and cross-entropy loss, namely x ′ = 
Revised: For simplicity we consider adversarial perturbation based on FGSM and cross-entropy loss, namely x 0 = x  " kr f ( x )

Original: x − ε ∥∇ f ( x ) y ∥ − 1 ∇ f ( x ) y .
Revised: y k  1 r f ( x ) y .

Original: The distribution mismatch induced by such adversarial perturbation can be lower bounded.
Revised: The distribution mismatch induced by such adversarial perturbation can be lower bounded.

Original: Lemma 4.1.
Revised: Lemma 4.1.

Original: Assume f
Revised: Assume f

Original: ( x ) y is L -locally Lipschitz around x with bounded Hessian.
Revised: ( x ) y is L -locally Lipschitz around x with bounded Hessian.

Original: Let σ m = {+inf z 2 B " ( x )  min ( r 2 f ( z ) y ) > 
+}
Revised: Let  m = {+inf z 2 B " ( x )  min ( r 2 f ( z ) y ) > 
+}

Original: inf
Revised: and

Original: z ∈B ε ( x ) σ min ( ∇ 2 f ( z ) y ) > 0 and σ M = sup z ∈B ε ( x ) σ max ( ∇ 2 f ( z ) y ) > 0 .
Revised:  M = sup z 2 B " ( x )  max ( r 2 f ( z ) y ) > 0 .

Original: Here σ min and σ max denote the minimum and maximum eigenvalues of the Hessian, respectively.
Revised: Here  min and  max denote the minimum and maximum eigenvalues of the Hessian, respectively.

Original: We then have
Revised: We then have

Original: [Equation]
Revised: [Equation]

Original: One can find that the right-hand side is positive as long as the upper bound of the Hessian norm is not too large, which is reasonable as previous works have shown that small hessian norm is critical to both standard (Keskar et al., 2017) and robust generalization (Moosavi-Dezfooli et al., 2019).
Revised: One can ﬁnd that the right-hand side is positive as long as the upper bound of the Hessian norm is not too large, which is reasonable as previous works have shown that small hessian norm is critical to both standard (Keskar et al., 2017) and robust generalization (Moosavi-Dezfooli et al., 2019).

Original: Assigned label distribution is unchanged.
Revised: Assigned label distribution is unchanged.

Original: Despite the fact that the true label distribution is distorted by adversarial perturbation, we note that the assigned label distribution of adversarial examples is still the same as their clean counterparts.
Revised: Despite the fact that the true label distribution is distorted by adversarial perturbation, we note that the assigned label distribution of adversarial examples is still the same as their clean counterparts.

Original: Remark 4.1.
Revised: Remark 4.1.

Original: In adversarial training, it is the common practice that directly copies the label of a clean input to its adversarial counterpart, namely ˜ y ′{+
+} = 
Revised: In adversarial training, it is the common practice that directly copies the label of a clean input to its adversarial counterpart, namely ˜ y {+
+} = ˜

Original: ˜ y and P ( ˜ Y ′{+
+}
Revised: y and P ( ˜ Y {+
+}

Original: | x ′ ) =
Revised: | x 0 ) =

Original: P ( ˜ Y | x ) .[Figure]
Revised: P ( ˜ Y | x ) .

Original:  augmented training set (e. 16 / 255 ) can also produce prominent overfitting.
Revised: versarially augmented training set (e.g. " = 16 / 255 ) can also produce prominent overﬁtting.

Original: In contrast, on the original training set without adversarial perturbation applied ( ε = 0 ), no overfitting is observed.[Figure] will not produce overfitting. Here we select extremely large perturbation radius to reduce the test error to be comparable to the
Revised: In contrast, on the original training set without adversarial perturbation applied ( " = 0 ), no overﬁtting is observed.

Original: Distribution mismatchindicates labelnoise.
Revised: Distribution mismatch label

Original: We show that a mismatch between the true label distribution and the assigned label distribution in a training set will always indicate the existence of label noise.
Revised: We show that a mismatch between the true label distribution and the assigned label distribution in a training set will always indicate the existence of label noise.

Original: Lemma 4.2.
Revised: Lemma 4.2.

Original: Given a training set D = { ( x i , ˜ y i ) } i ∈ [ N ] , the label noise is lower-bounded by the mismatch between the true label distribution and the assigned label distribution.
Revised: Given a training set D = { ( x i , ˜ y i ) } i 2 [ N ] , the label noise is lower-bounded by the mismatch between the true label distribution and the assigned label distribution.

Original: Specifically, with probability 1 − δ , we have
Revised: Speciﬁcally, with probability 1   , we have

Original: [Equation]
Revised: [Equation]

Original: Label noise implicitly exists in adversarial training.
Revised: Label noise implicitly exists in adversarial training.

Original: In the adversarially augmented training set
Revised: In the adversarially augmented training set

Original: D ′ , such distribution mismatch exists exactly.
Revised: D 0 , such distribution mismatch exists exactly.

Original: By Remark 4.1 we have P ( ˜ Y ′ | x ′ ) 
Revised: By Remark 4.1 we have P ( ˜ Y 0 | x 0 ) =

Original: = P ( ˜ Y | x ) and by property of the clean dataset (Assumption 3.1) we have P ( ˜ Y | x ) =
Revised: P ( ˜ Y | x ) and by property of the clean dataset (Assumption 3.1) we have P ( ˜ Y | x ) =

Original: P ( Y | x ) , which together means
Revised: P ( Y | x ) , which together means

Original: P ( ˜ Y ′ | x ′ ) =
Revised: P ( ˜ Y 0 | x 0 ) =

Original: P ( Y | x ) .
Revised: P ( Y | x ) .

Original: However, Lemma 4.1 shows that P ( Y ′ | x ′ ) {+
+}
Revised: However, Lemma 4.1 shows that P ( Y 0 | x 0 ) {+
+}

Original: ̸

Original: = P ( Y | x ) , which implies thatP ( ˜ Y ′ | x ′ )
Revised: = P ( Y | x ) , which implies that

Original: ̸ = P ( Y ′ | x ′ ) .
Revised: P ( ˜ Y 0 | x 0 ) 6 = P ( Y 0 | x 0 ) .

Original: This indicates that label noise exists in the adversarially augmented training set.
Revised: This indicates that label noise exists in the adversarially augmented training set.

Original: We now have the following theorem, which is our main result.
Revised: We now have the following theorem, which is our main result.

Original: Theorem 4.3.
Revised: Theorem 4.3.

Original: Assume f
Revised: Assume f

Original: ( x ) y is L -locally Lipschitz around x with Hessian bounded below.
Revised: ( x ) y is L -locally Lipschitz around x with Hessian bounded below.

Original: Instantiate the same notations as in Lemma 4.1.
Revised: Instantiate the same notations as in Lemma 4.1.

Original: With probability 1 − δ , we have
Revised: With probability 1   , we have

Original: [Equation]
Revised: [Equation]

Original: The above results suggest that as long as a training set is augmented by adversarial perturbation, but with assigned labels unchanged, label noise emerges.
Revised: The above results suggest that as long as a training set is augmented by adversarial perturbation, but with assigned labels unchanged, label noise emerges.

Original: We demonstrate this by showing that standard training on a fixed adversarially augmented training set can also produce overfitting.
Revised: We demonstrate this by showing that standard training on a ﬁxed adversarially augmented training set can also produce overﬁtting.

Original: Specifically, for each example in a clean training set we apply adversarial perturbation generated by a adversarially trained classifier.
Revised: Speciﬁcally, for each example in a clean training set we apply adversarial perturbation generated by a adversarially trained classiﬁer.

Original: We then fix such an augmented training set and conduct standard training on it.
Revised: We then ﬁx such an augmented training set and conduct standard training on it.

Original: We experiment on CIFAR-10 with WRN-28-5.
Revised: We experiment on CIFAR-10 with WRN-28-5.

Original: A training subset of size 5k is randomly sampled to speed up the training.
Revised: A training subset of size 5k is randomly sampled to speed up the training.

Original: More details about the experiment settings can be found in the appendix.
Revised: More details about the experiment settings can be found in the appendix.

Original: Figureshows that prominent overfitting (as well as epoch-wise double descent) can be observed when the perturbation radius is relatively large.
Revised: Figureshows that prominent overﬁtting (as well as epoch-wise double descent) can be observed when the perturbation radius is relatively large.

Original: On the other hand, if a training set is augmented by perturbation that will not distort the true label distribution, there will not be label noise.
Revised: On the other hand, if a training set is augmented by perturbation that will not distort the true label distribution, there will not be label noise.

Original: We demonstrate this by showing that standard training on a training set augmented with Gaussian noise will not induce overfitting.
Revised: We demonstrate this by showing that standard training on a training set augmented with Gaussian noise will not induce overﬁtting.

Original: As shown in Figure 4, even with a extremely large radius of Gaussian perturbation, no overfitting is observed.
Revised: As shown in Figure 4, even with a extremely large radius of Gaussian perturbation, no overﬁtting is observed.

Original: This also demonstrates that input perturbation not necessarily leads to overfitting.
Revised: This also demonstrates that input perturbation not necessarily leads to overﬁtting.

Original: Intuitive interpretation of label noise in adversarial training.
Revised: Intuitive interpretation of label noise in adversarial training.

Original: We introduce a simple example to help understand the emergence of label noise in adversarial training.
Revised: We introduce a simple example to help understand the emergence of label noise in adversarial training.

Original: Example 4.1 (Label noise due to a symmetric distribution shift) .
Revised: Example 4.1 (Label noise due to a symmetric distribution shift) .

Original: Let D =
Revised: Let D =

Original: ( x i , y i ) } i ∈ [ N ] be a clean labeled training subset where all inputs x i = x are identical and have a one-hot true label distribution, i.e., P ( Y | x ) = 1 y .
Revised: ( x i , y i ) } i 2 [ N ] be a clean labeled training subset where all inputs x i = x are identical and have a one-hot true label distribution, i.e., P ( Y | x ) = 1 y .

Original: We now construct an adversarially augmented training subset D ′ ={+
+}
Revised: We now construct an adversarially augmented training subset D {+
+}

Original:  { ( x ′ i , ˜ y ′ i ) } i ∈ [ N ] , where ˜y ′
Revised: = { ( x 0 i , ˜ y 0 i ) } i 2 [ N ] , where ˜

Original: ={+y 
+}
Revised: {+y 
+}

Original:  y and x ′ is generated based on adversarial perturbation that distorts the true label distribution symmetrically.
Revised: = y and x 0 is generated based on adversarial perturbation that distorts the true label distribution symmetrically.

Original: Specifically,
Revised: Speciﬁcally,

Original: [Equation]
Revised: [Equation]

Original: Then by Lemma 4.2 we have p e ( D ′ ) ≳ η .
Revised: Then by Lemma 4.2 we have p e ( D 0 ) & ⌘ .

Original: One can find that there is indeed η faction of noisy labels in D ′ .
Revised: One can ﬁnd that there is indeed ⌘ faction of noisy labels in D 0 .

Original: This is because if we sample the labels of x ′ based on its true label distribution, we expect 1 − η faction of x ′ are labeled as y , while η fraction of x ′ are labeled to be other classes.
Revised: This is because if we sample the labels of x 0 based on its true label distribution, we expect 1  ⌘ faction of x 0 are labeled as y , while ⌘ fraction of x 0 are labeled to be other classes.

Original: However, in D ′ , all x ′ are assigned with label y , which means η fraction of x ′ are labeled incorrectly.
Revised: However, in D 0 , all x 0 are assigned with label y , which means ⌘ fraction of x 0 are labeled incorrectly.

Original: In realistic datasets we can consider inputs with similar features for such reasoning.
Revised: In realistic datasets we can consider inputs with similar features for such reasoning.

Original: The above example also shows that label noise in adversarial training may be stronger than one’s impression.
Revised: The above example also shows that label noise in adversarial training may be stronger than one’s impression.

Original: Even a slight distortion of the true label distribution, e.g. η = 0 .
Revised: Even a slight distortion of the true label distribution, e.g. ⌘ = 0 .

Original: 1 , will be equivalent to at least 10% noisy label in the training set.
Revised: 1 , will be equivalent to at least 10% noisy label in the training set.

Original: This is because the true label distribution of every training input is distorted, resulting in significant noise in the population.
Revised: This is because the true label distribution of every training input is distorted, resulting in signiﬁcant noise in the population.

Original: Dependence of label noise in adversarial training.
Revised: Dependence of label noise in adversarial training.

Original: Theorem 4.3 shows that the label noise in adversarial training is proportional to (1) the perturbation radius (2) the data quality.
Revised: Theorem 4.3 shows that the label noise in adversarial training is proportional to (1) the perturbation radius (2) the data quality.

Original: Considering label noise can be an important source of variance in the generalization of deep neural networks (Nakkiran et al., 2020; Yang et al., 2020), such dependence of label noise explains the intriguing observations in the literature that robust overfitting (or epoch-wise double descent) in adversarial training will vanish with small perturbation
Revised: Considering label noise can be an important source of variance in the generalization of deep neural networks (Nakkiran et al., 2020; Yang et al., 2020), such dependence of label noise explains the intriguing observations in the literature that robust overﬁtting (or epoch-wise double descent) in adversarial training will vanish with small perturbation

Original: radii (Dong et al., 2021b) or high-quality data (Dong et al., 2021a).
Revised: radii (Dong et al., 2021b) or high-quality data (Dong et al., 2021a).

Original: We conduct more controlled experiments to verify this correlation empirically, as shown in Figure 5,[Figure]
Revised: We conduct more controlled experiments to verify this correlation empirically, as shown in Figure 5,

Original: 4.2 Adversarial perturbation generated by a realistic classifier
Revised: 4.2 Adversarial perturbation generated by a realistic classiﬁer

Original: We now consider a realistic case where the adversarial perturbation is generated by a probabilistic classifier f θ .
Revised: We now consider a realistic case where the adversarial perturbation is generated by a probabilistic classiﬁer f ✓ .

Original: Approximation of the true label distribution.
Revised: Approximation of the true label distribution.

Original: We show that after sufficient adversarial training, the predictive label distribution of f θ can approximate the true label distribution with high probability.
Revised: We show that after sufﬁcient adversarial training, the predictive label distribution of f ✓ can approximate the true label distribution with high probability.

Original: Lemma 4.4.
Revised: Lemma 4.4.

Original: Denote S =
Revised: Denote S =

Original: { x : ( x, y ) ∈ D} as the collection of all training inputs.
Revised: { x : ( x, y ) 2 D} as the collection of all training inputs.

Original: Let ρ ≥ 1 and
Revised: Let ⇢  1 and

Original: C be an ρε -external covering of S with covering number N ρε .
Revised: C be an ⇢" -external covering of S with covering number N ⇢" .

Original: Let f θ be a probabilistic classifier that minimizes the adversarial empirical risk (1).
Revised: Let f ✓ be a probabilistic classiﬁer that minimizes the adversarial empirical risk (1).

Original: Assume f θ is L θ -locally Lipschitz continuous in a norm ball of radius ρε around x ∈ C .
Revised: Assume f ✓ is L ✓ -locally Lipschitz continuous in a norm ball of radius ⇢" around x 2 C .

Original: Let κ ≥ 1 and ˆ S be a subset of S with cardinality at least (1 [-− 
 /κ-] +1 / ( κN ρε ))
Revised: Let   1 and ˆ S be a subset of S with cardinality at least (1 [-− 
 /κ-] 1 /  +1 / (  N ⇢" ))

Original: N .
Revised: N .

Original: Let N ε ( ˆ S ) denote the neighborhood of the set ˆ S , i.e. N ε
Revised: Let N " ( ˆ S ) denote the neighborhood of the set ˆ S , i.e. N " ( ˆ S ) =

Original: ( ˆ S) = (cid:83) x ∈ ˆ S B ε ( x ) .
Revised: S x 2 ˆ S B " ( x ) .

Original: Then for any x ∈ N ε ( ˆ S ) , with probability at least 1 − δ ,
Revised: Then for any x 2 N " ( ˆ S ) , with probability at least 1   ,

Original: [Equation]
Revised: [Equation]

Original: Label noise in adversarial training with a realistic classifier.
Revised: Label noise in adversarial training with a realistic classiﬁer.

Original: Adversarial perturbation generated by a realistic classifier f θ will distort its predictive label distribution by gradient ascent.
Revised: Adversarial perturbation generated by a realistic classiﬁer f ✓ will distort its predictive label distribution by gradient ascent.

Original: Subsequently, the true label distribution will also be distorted with high probability since the predictive label distribution of a realistic classifier f θ can approximate the true label distribution.
Revised: Subsequently, the true label distribution will also be distorted with high probability since the predictive label distribution of a realistic classiﬁer f ✓ can approximate the true label distribution.

Original: Specifically, by the triangle inequality we have
Revised: Speciﬁcally, by the triangle inequality we have

Original: [Equation]
Revised: [Equation]

Original: where the last two terms are the approximation error of true label distribution on both clean and adversarial examples, which are guaranteed to be small.
Revised: where the last two terms are the approximation error of true label distribution on both clean and adversarial examples, which are guaranteed to be small.

Original: To conclude, we have the following result.
Revised: To conclude, we have the following result.

Original: Theorem 4.5.
Revised: Theorem 4.5.

Original: Instantiate the notations of Lemma 4.4.
Revised: Instantiate the notations of Lemma 4.4.

Original: For any x ∈ N ε ( ˆ S ) , with probability at least 1 − 3 δ , we have
Revised: For any x 2 N " ( ˆ S ) , with probability at least 3  , we have

Original: [Equation]2 −
Revised: [Equation]

Original: K (cid:19) L θ + L (cid:19)(cid:21) − ε 2 σ M 4 − ξ (cid:114)2 N log δ , (7) where ξ = 1 + (cid:112) 4 κN ρε K .
Revised: N log [Equation] (7) where ⇠ = 1 + p 4  N ⇢" K .

Original:  Mitigate Label Noise in Adversarial Training
Revised:  Mitigate Label Noise in Adversarial Training

Original: Since the label noise is incurred by the mismatch between the true label distribution and assigned label distribution of adversarial examples in the training set, we wish to find an alternative label (distribution) for the adversarial example to reduce such distribution mismatch.
Revised: Since the label noise is incurred by the mismatch between the true label distribution and assigned label distribution of adversarial examples in the training set, we wish to ﬁnd an alternative label (distribution) for the adversarial example to reduce such distribution mismatch.

Original: We’ve already shown that the predictive label distribution of a classifier trained by conventional adversarial training, which we denote as model probability in the following discussion, can in fact approximate the true label distribution.
Revised: We’ve already shown that the predictive label distribution of a classiﬁer trained by conventional adversarial training, which we denote as model probability in the following discussion, can in fact approximate the true label distribution.

Original: Here we show that it is possible to further improve the predictive label distribution and reduce the label noise by calibration.
Revised: Here we show that it is possible to further improve the predictive label distribution and reduce the label noise by calibration.

Original: 5.1 Rectify model probability to reduce distribution mismatch
Revised: 5.1 Rectify model probability to reduce distribution mismatch

Original: We show that it is possible to reduce the distribution mismatch by temperature scaling (Hinton et al., 2015; Guo et al., 2017) enabled in the softmax function.
Revised: We show that it is possible to reduce the distribution mismatch by temperature scaling (Hinton et al., 2015; Guo et al., 2017) enabled in the softmax function.

Original: Theorem 5.1 (Temperature scaling can reduce the distribution mismatch) .
Revised: Theorem 5.1 (Temperature scaling can reduce the distribution mismatch) .

Original: Let f θ ( x ; T ) denote the predictive probability of a probabilistic classifier scaled by temperature T , namely f θ ( x ; T ) 
Revised: Let f ✓ ( x ; T ) denote the predictive probability of a probabilistic classiﬁer scaled by temperature T , namely f ✓ ( x ; T ) j = exp( z j /T ) / (

Original: j = exp( z j /T ) / ( (cid:80) j exp( z j /T )) , where z is the logits of the classifier from x .
Revised: P j exp( z j /T )) , where z is the logits of the classiﬁer from x .

Original: Let x ′ be an adversarial example correctly classified by a classifier f θ , i.e. arg max j f θ ( x ′ )
Revised: Let x 0 be an adversarial example correctly classiﬁed by a classiﬁer f ✓ , i.e. arg max j f ✓ ( x 0 )

Original: j = y ′ , then there exists T , such that
Revised: j = y 0 , then there exists T , such that

Original: [Equation]
Revised: [Equation]

Original: Another way to further reduce the distribution mismatch is to interpolate between the model probability and the one-hot assigned label.
Revised: Another way to further reduce the distribution mismatch is to interpolate between the model probability and the one-hot assigned label.

Original: We show that the interpolation works specifically for incorrectly classified examples and thus can be viewed as a complement to temperature scaling.
Revised: We show that the interpolation works speciﬁcally for incorrectly classiﬁed examples and thus can be viewed as a complement to temperature scaling.

Original: Theorem 5.2 (Interpolation can further reduce the distribution mismatch) .
Revised: Theorem 5.2 (Interpolation can further reduce the distribution mismatch) .

Original: Let x ′ be an adversarial example incorrectly classified by a classifier f θ , i.e. arg max j f θ ( x ′ ; T ) j ̸ = y ′ .
Revised: Let x 0 be an adversarial example incorrectly classiﬁed by a classiﬁer f ✓ , i.e. arg max j f ✓ ( x 0 ; T ) j 6 = y 0 .

Original: Assume max j P ( Y ′{+
+} =
Revised: Assume max j P ( Y {+
+} =

Original: j | x ′ ) ≥ 1 / 2 , then there exists an interpolation ratio λ , such that
Revised: j | x 0 )  1 / 2 , then there exists an interpolation ratio  , such that

Original: [Equation]
Revised: [Equation]

Original: where f θ ( x ′ ; T, λ ) =
Revised: where f ✓ ( x 0 ; T,  ) =

Original: λ · f θ ( x ′ ; T ) + (1 − λ ) · P ( ˜ Y ′ | x ′ ) .
Revised:  · f ✓ ( x 0 ; T ) + (1   ) · P ( ˜ Y 0 | x 0 ) .

Original: As a summarization, to reduce the distribution mismatch, we propose to use f θ ( x ′ ; T, λ ) as the assigned label of the adversarial example in adversarial training, which we refer as the rectified model probability .
Revised: As a summarization, to reduce the distribution mismatch, we propose to use f ✓ ( x 0 ; T,  ) as the assigned label of the adversarial example in adversarial training, which we refer as the rectiﬁed model probability .

Original: In Appendix E, we show that the optimal hyper-parameters (i.e. T and λ ) of almost all training examples concentrate on the same set of values by studying on a synthetic dataset with known true label distribution.
Revised: In Appendix E, we show that the optimal hyper-parameters (i.e. T and  ) of almost all training examples concentrate on the same set of values by studying on a synthetic dataset with known true label distribution.

Original: Therefore it is possible to find an universal set of hyper-parameters that reduce the distribution mismatch for all adversarial examples.
Revised: Therefore it is possible to ﬁnd an universal set of hyper-parameters that reduce the distribution mismatch for all adversarial examples.

Original: 5.2 Determine the optimal temperature and interpolation ratio
Revised: 5.2 Determine the optimal temperature and interpolation ratio

Original: The set of temperature and interpolation ratio in the rectified model probability that maximally reduces the distribution mismatch is not straightforward to find as the true label distribution of the adversarial example is unknown in reality.
Revised: The set of temperature and interpolation ratio in the rectiﬁed model probability that maximally reduces the distribution mismatch is not straightforward to ﬁnd as the true label distribution of the adversarial example is unknown in reality.

Original: Fortunately, given a sufficiently large validation dataset as a whole, it is possible to measure the overall distribution mismatch in a frequentist’s view without knowing the true label distribution of every single example.
Revised: Fortunately, given a sufﬁciently large validation dataset as a whole, it is possible to measure the overall distribution mismatch in a frequentist’s view without knowing the true label distribution of every single example.

Original: A popular metric adopted here is the negative log-likelihood (NLL) loss, which is known as a proper scoring rule (Gneiting & Raftery, 2007) and is also employed in the confidence calibration of deep networks (Guo et al., 2017).
Revised: A popular metric adopted here is the negative log-likelihood (NLL) loss, which is known as a proper scoring rule (Gneiting & Raftery, 2007) and is also employed in the conﬁdence calibration of deep networks (Guo et al., 2017).

Original: By
Revised: By

Original: Gibbs’s inequality it is easy to show that the NLL loss will only be minimized when the assigned label distribution matches the true label distribution (Hastie et al., 2001), namely
Revised: Gibbs’s inequality it is easy to show that the NLL loss will only be minimized when the assigned label distribution matches the true label distribution (Hastie et al., 2001), namely

Original: [Equation]
Revised: [Equation]

Original: Therefore, we propose to find the optimal T and λ as[Equation] 5.3 Rectified model probability mitigates robust overfitting
Revised: Therefore, we propose to ﬁnd the optimal T and  as

Original: [Equation][Figure]
Revised: [Equation]

Original: N LL ( V a l ) 1010 1 T

Original: R o b u s t

Original: T e s t A cc ( % ) Best Checkpoint Last Checkpoint 0.5 0.6 0.7 0.8 0.9 1.0 46. R o b u s t
Revised: 5.3 Rectiﬁed model probability mitigates robust overﬁtting

Original: T e s t A cc ( % ) Best Checkpoint Last Checkpoint We now work on a realistic dataset (CIFAR-10) to demonstrate the rectified model probability can effectively mitigate the robust overfitting, or equivalently the epoch-wise double descent in adversarial training.
Revised: We now work on a realistic dataset (CIFAR-10) to demonstrate the rectiﬁed model probability can effectively mitigate the robust overﬁtting, or equivalently the epoch-wise double descent in adversarial training.

Original: The outer minimization of adversarial training (Equation (1)) now becomes
Revised: The outer minimization of adversarial training (Equation (1)) now becomes

Original: [Equation]
Revised: [Equation]

Original: where ˆ θ denotes the parameters of a classifier adversarially trained beforehand.
Revised: where ˆ ✓ denotes the parameters of a classiﬁer adversarially trained beforehand.

Original: The details of the experimental setting are available in the
Revised: The details of the experimental setting are available in the

Original: Appendix.
Revised: Appendix.

Original: As shown in Figure 6, adversarial training on rectified model probability can mitigate the robust overfitting when the temperature T and interpolation ratio λ are optimal.
Revised: As shown in Figure 6, adversarial training on rectiﬁed model probability can mitigate the robust overﬁtting when the temperature T and interpolation ratio  are optimal.

Original: Such optimal hyperparameters perfectly aligns with the ones automatically determined by Equation (9).
Revised: Such optimal hyperparameters perfectly aligns with the ones automatically determined by Equation (9).

Original:  Experiments
Revised:  Experiments

Original: Experiment setup.
Revised: Experiment setup.

Original: We conduct experiments on three datasets including CIFAR-10, CIFAR100 (Krizhevsky, 2009) and Tiny-ImageNet (Le & Yang, 2015).
Revised: We conduct experiments on three datasets including CIFAR-10, CIFAR100 (Krizhevsky, 2009) and Tiny-ImageNet (Le & Yang, 2015).

Original: We conduct PGD training on preactivation ResNet-18 (He et al., 2016) with 10 iterations and perturbation radius 8 / 255 by default.
Revised: We conduct PGD training on preactivation ResNet-18 (He et al., 2016) with 10 iterations and perturbation radius 8 / 255 by default.

Original: We evaluate robustness against ℓ ∞ norm-bounded adversarial attack with perturbation radius 8 / 255 , and employ AutoAttack (Croce & Hein, 2020) for reliable evaluation.
Revised: We evaluate robustness against ` 1 norm-bounded adversarial attack with perturbation radius 8 / 255 , and employ AutoAttack (Croce & Hein, 2020) for reliable evaluation.

Original: Appendix D.2 includes results on additional model architectures (e.g., VGG (Simonyan & Zisserman, 2015), WRN), adversarial training[Figure] Tiny [Figure]
Revised: Appendix D.2 includes results on additional model architectures (e.g., VGG (Simonyan & Zisserman, 2015), WRN), adversarial training

Original: ImageNet

Original: [Figure] [Table]
Revised: [Table]

Original: CIFARAT - [Table] methods (e.g., TRADES (Zhang et al., 2019), FGSM (Goodfellow et al., 2015)), and evaluation metrics (e.g., PGD-1000 (PGD attack with 1000 iterations), Square Attack (Andriushchenko et al., 2020), RayS (Chen & Gu, 2020)).
Revised: methods (e.g., TRADES (Zhang et al., 2019), FGSM (Goodfellow et al., 2015)), and evaluation metrics (e.g., PGD-1000 (PGD attack with 1000 iterations), Square Attack (Andriushchenko et al., 2020), RayS (Chen & Gu, 2020)).

Original: More setup details can be found in Appendix G.
Revised: More setup details can be found in Appendix G.

Original: Results & Discussions.
Revised: Results & Discussions.

Original: Our method is essentially the baseline adversarial training with a robusttrained self-teacher, equipped with an algorithm automatically deciding the optimal hyper-parameters, which we now denote as KD-AT-Auto.
Revised: Our method is essentially the baseline adversarial training with a robusttrained self-teacher, equipped with an algorithm automatically deciding the optimal hyper-parameters, which we now denote as KD-AT-Auto.

Original: We compare KD-AT-Auto with two baselines: regular adversarial training (AT), and adversarial training combined with self-distillation (KD-AT) with fixed temperature T = 2 and interpolation ratio λ = 0 .
Revised: We compare KD-AT-Auto with two baselines: regular adversarial training (AT), and adversarial training combined with self-distillation (KD-AT) with ﬁxed temperature T = 2 and interpolation ratio  = 0 .

Original: 5 as suggested by Chen et al.
Revised: 5 as suggested by Chen et al.

Original: As shown in Figure 7, our method can effectively mitigate robust overfitting for all datasets, with both standard accuracy (SA) and robust accuracy (RA) constantly increasing throughout training.
Revised: As shown in Figure 7, our method can effectively mitigate robust overﬁtting for all datasets, with both standard accuracy (SA) and robust accuracy (RA) constantly increasing throughout training.

Original: In Table 1, we measure the difference between the RA at the best checkpoint (Best) and at the last checkpoint (Last) to clearly show the overfitting gap.
Revised: In Table 1, we measure the difference between the RA at the best checkpoint (Best) and at the last checkpoint (Last) to clearly show the overﬁtting gap.

Original: Our method can reduce the overfitting gap to less than 0 .
Revised: Our method can reduce the overﬁtting gap to less than 0 .

Original: 5% for all datasets.
Revised: 5% for all datasets.

Original: One may note that self-distillation with fixed hyper-parameters is in fact inferior in terms of reducing robust overfitting, while its effectiveness can be significantly improved with the optimal hyper-parameters automatically determined by our method, which further verifies our understanding of robust overfitting.
Revised: One may note that self-distillation with ﬁxed hyper-parameters is in fact inferior in terms of reducing robust overﬁtting, while its effectiveness can be signiﬁcantly improved with the optimal hyper-parameters automatically determined by our method, which further veriﬁes our understanding of robust overﬁtting.

Original: Compared with self-distillation with fixed hyper-parameters, our method can also boost both RA and SA at the best checkpoint for all datasets.
Revised: Compared with self-distillation with ﬁxed hyper-parameters, our method can also boost both RA and SA at the best checkpoint for all datasets.

Original: Our method can further be combined with orthogonal techniques such as Stochastic Weight Averaging (SWA) (Izmailov et al., 2018) and additional standard teachers as mentioned in previous work (Chen et al., 2021) to achieve better performance.
Revised: Our method can further be combined with orthogonal techniques such as Stochastic Weight Averaging (SWA) (Izmailov et al., 2018) and additional standard teachers as mentioned in previous work (Chen et al., 2021) to achieve better performance.

Original: More results and discussion can be found in Appendix D.3.
Revised: More results and discussion can be found in Appendix D.3.

Original:  Conclusion and Discussions
Revised:  Conclusion and Discussions

Original: In this paper, we show that label noise exists implicitly in adversarial training due to the mismatch between the true label distribution and the assigned label distribution of adversarial examples.
Revised: In this paper, we show that label noise exists implicitly in adversarial training due to the mismatch between the true label distribution and the assigned label distribution of adversarial examples.

Original: Such label noise can explain the dominant overfitting phenomenon.
Revised: Such label noise can explain the dominant overﬁtting phenomenon.

Original: Based on a label noise perspective, we also extend the understanding of robust overfitting and show that it is the early part of an epoch-wise double descent in adversarial training.
Revised: Based on a label noise perspective, we also extend the understanding of robust overﬁtting and show that it is the early part of an epoch-wise double descent in adversarial training.

Original: Finally, we propose an alternative labeling of adversarial examples by rectifying model probability, which can effectively mitigate robust overfitting without any manual hyper-parameter tuning.
Revised: Finally, we propose an alternative labeling of adversarial examples by rectifying model probability, which can effectively mitigate robust overﬁtting without any manual hyper-parameter tuning.

Original: The label noise implicitly exists in adversarial training may have other important effects on adversarially robust learning.
Revised: The label noise implicitly exists in adversarial training may have other important effects on adversarially robust learning.

Original: This can potentially consolidate the theoretical grounding of robust learning.
Revised: This can potentially consolidate the theoretical grounding of robust learning.

Original: For instance, since label noise induces model variance, from a model-wise view, one may need to increase model capacity to reduce the variance.This may partially explain why robust generalization requires significantly larger model than standard generalization. Checklist The checklist follows the references.
Revised: For instance, since label noise induces model variance, from a model-wise view, one may need to increase model capacity to reduce the variance.

Original: Please read the checklist guidelines carefully for information on how to answer these questions. For each question, change the default [TODO] to [Yes] , [No] or [N/A] . You are strongly encouraged to include a justification to your answer , either by referencing the appropriate section of your paper or providing a brief inline description. For example:
Revised: This may partially explain why robust generalization requires signiﬁcantly larger model than standard generalization.

Original: • Did you include the license to the code and datasets? [Yes] See Section ?? . • Did you include the license to the code and datasets? [No] The code and the data are proprietary.
Revised: Acknowledgments

Original: • Did you include the license to the code and datasets? [N/A] Please do not modify the questions and only use the provided macros for your answers. Note that the
Revised: We sincerely thank Like Hui for the discussions on the theory of double descent and conﬁdence calibration.

Original: Checklist section does not count towards the page limit.

Original: In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope? [Yes] (b) Did you describe the limitations of your work? [Yes] See Section B (c) Did you
Revised: We thank anonymous reviewers for their valuable comments and suggestions.

Original: discuss any potential negative societal impacts of your work?

Original: [Yes] See

Original: Section B

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes] 2. If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results? [Yes] (b) Did you include complete proofs of all theoretical results?
Revised: We also thank the audience at Neurips 2022 main conference for their thoughtful questions and feedback.

Original: [Yes] 3.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [Yes] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [Yes] 4.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] (b) Did you mention the license of the assets?

Original: [N/A] (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [N/A] (d) Did you discuss whether

Original: and how consent was obtained from people whose data you’re using/curating?

Original: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?

Original: [N/A] 5.

Original: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A] A Proofs A.1 Preliminaries
Revised: Our work is sponsored in part by NSF Convergence Accelerator under award OIA-2040727, NIH

Original: Proof in Assumption 3.1.

Original: Here we prove that if there no label error in the clean dataset, then

Original: P ( ˜ Y | x ) =

Original: P ( Y | x ) .

Original: Proof.

Original: First, we note that

Original: [Equation]

Original: Since P ( E = 1 | Y =

Original: [-j, x ) = 
-]
Revised: [-j, x ) = 
-]

Original: we have,

Original: [Equation]

Original: Therefore we have for all j ′ ,

Original: [Equation]

Original: Total Variation distance for discrete probability distributions.

Original: For two discrete probability distributions P ( Y ) and P ( Y ′ ) where Y, Y ′ ∈ Y , the total variation distance between them can be equally defined as

Original: [Equation]

Original: A.2 Proofs in Section 4.

Original: Proof of Lemma 4.1.

Original: Proof.

Original: For simplicity, we consider the adversarial perturbation generated by FGSM.

Original: Other adversarial perturbation can be viewed as a Taylor series of such perturbation.

Original: [Equation]

Original: First, we bound the distribution mismatch by gradient norm.

Original: [Equation]

Original: TV distance

Original: [Equation]

Original: Bounded Hessian

Original: [Equation]

Original: Now if ∥ · ∥ = ∥ · ∥ 2 , we have

Original: [Equation]

Original: If ∥ · ∥ = ∥ · ∥ ∞ , we can utilize the fact that ∥ · ∥ ∞ ≤ ∥ · ∥ 2 ≤ √ d ∥ · ∥ ∞ , thus

Original: [Equation]

Original: [Equation]

Original: Second, we bound the gradient norm by the L -local Lipschitzness assumption.

Original: Let x ∗ be a closest input that achieves the local maximum on the predicted probability at y , namely x ∗ = arg min z ∈ X,f ( z ) y

Original: =1 ∥ x − z ∥ .

Original: Because x ∗ is the local maximum and f is continuously differentiable, ∇ f ( x ∗ )

Original: y = 0 , thus

Original: [Equation]

Original: Therefore we have

Original: [Equation]

Original: Plug this into Equation (12) or Equation (13) we then obtain the desired result

Original: [Equation]

Original: Proof of Lemma 4.2.

Original: Proof.

Original: First, we show that the expectation of the label error is lower bounded by the mismatch between the true label distribution and the assigned label distribution.

Original: [Equation]

Original: Second, given a sampled training set D = { ( x i , ˜ y i ) } i ∈

Original: [ N ] , the empirical measure of label error E should converge to its expectation almost surely, namely

Original: [Equation] Using standard concentration inequality such as Hoeffding’s inequality we have, with probability 1 − δ , [Equation] This implies
Revised: Bridge2AI Center Program under award 1U54HG012510- as well as generous gifts from Google,

Original: [Equation]

Original: Since P ( E = 1) =

Original: E x P ( E = 1 | x ) , we have, with probability 1 − δ ,

Original: [Equation]

Original: which means p e ( D ) > 0 as long as N is large.

Original: Proof of Theorem 4.3.

Original: Proof.

Original: First, by the fact that P ( ˜ Y ′ | x ′ )

Original: =

Original: P ( ˜ Y | x ) and P ( ˜ Y | x ) =

Original: P ( Y | x ) we have P ( ˜ Y ′

Original: | x ′ ) =

Original: P ( Y | x ) .

Original: Therefore, apply Lemma 4.2 to an adversarially augmented training set we have with probability 1 − δ ,

Original: [Equation]

Original: Further, apply Lemma 4.1 and the definition of data quality, we have with probability 1 − δ ,

Original: [Equation]

Original: A.3 Proofs in Section 4. Proof of Lemma 4.4. Proof. Let D =
Revised: Adobe and Teradata.

Original: ( x, y ) be the adversarially augmented training set.

Original: Let S

Original: = { x : ( x, y ) ∈ D} be the collection ofall training inputs. First we note that the set of all training inputs can be grouped into several subsets such that the inputs in each subset possess similar true label distribution. More formally, Let C = { ¯ x j } N ρε j =1 be an ρε -external covering of S with minimum cardinality, namely
Revised: Any opinions, ﬁndings, and conclusions or recommendations expressed herein are those of the authors and should not be interpreted as necessarily representing the views, either expressed or implied, of the U

Original: S ⊆ (cid:83) x ∈C { x ′ | ∥ x ′ − x ∥ ≤ ρε } , where we refer ¯ x j as the covering center, and N ρε is the covering number.

Original: Let {S j } N ρε j =1 be any disjoint partition of S such that S j ⊆ { x ′ |∥ x ′ − ¯ x j ∥ ≤ ρε } .

Original: We show that

Original: S j attains a property that the true label distribution of any input x in this subset will not be too far from the sample mean of one-hot labels ¯y j =

Original: |S j | − 1 (cid:80) x ∈S j 1 y in this subset.

Original: Specifically, let p ( x ) =

Original: P ( Y | x ) .

Original: We have with probability 1 − δ ,

Original: [Equation]

Original: To prove this property we first present two lemmas.

Original: Lemma A.1 (Lipschitz constraint of the true label distribution) .

Original: Let S j be a subset constructed above and ¯y j

Original: = |S j | − 1 (cid:80) x ∈S j 1 y .

Original: Then for any x ∈ S j we have,

Original: [Equation]

Original: Proof.

Original: First, since x ∈ S j , we have ∥ x − ¯ x j ∥ 1 ≤ ρε , which implies ∥ p ( x ) − p (¯ x j ) ∥ 1 ≤ Lρε by the locally Lipschitz continuity of p .

Original: Then for any x, x ′ ∈ S j , we will have ∥ p ( x ) − p ( x ′ )

Original: ∥ ≤ 2 Lρε by the triangle inequality.

Original: Let N

Original: S = |S j | . Therefore, [Equation]
Revised: S. Government.

Original: Further, the linearity of the expectation implies

Original: [Equation]

Original: Therefore ∥ p ( x ) − E [ ¯y j ] ∥ ≤ 2 Lρε .

Original: Lemma A.2 (Concentration inequality of the sample mean) .

Original: Let S be a set of x with cardinality N .

Original: Let ¯y =

Original: N − 1 (cid:80) x ∈S 1 y be the sample mean.

Original: Then for any p -norm ∥ · ∥ and any ε > 0 , we have with probability

Original: [Equation]

Original: Proof.

Original: Note that ¯y obeys a multinomial distribution, i.e. ¯y ∼ N − 1 multinomial

Original: N, E

Original: [ ¯y ]) .

Original: This lemma is thus the classic result on the concentration properties of multinomial distribution based on ℓ 1 norm Weissman et al.

Original: ; Qian et al.

Original: One can see that Lemma A.1 bounds the difference between true label distribution of individual inputs and the mean true label distribution, while Lemma A.2 bounds the difference between the sample mean and the mean true label distribution.

Original: Therefore the difference between the true label distribution and the sample mean is also bounded, since by the triangle inequality we have with probability 1 − δ ,

Original: [Equation]

Original: We now show that given the locally Lipschitz constraint established in each disjoint partition we constructed above, the prediction given by the empirical risk minimizer will be close to the sample mean.

Original: As an example, we focus on the negative log-likelihood loss, namely ℓ ( f θ ( x ) , y )

Original: = − 1 y · log f θ ( x ) .

Original: Other loss functions that are subject to the proper scoring rule can be investigated in a similar manner.

Original: First, we regroup the sum in the empirical risk based on the partition constructed above, namely

Original: [Equation]

Original: where ˆ R ( f θ , S j ) = −|S

Original: [Equation]

Original: | −

Original: [Equation]

Original: S

Original: [Equation]

Original: |

Original: [Equation]

Original: log f θ ( x i ) is the empirical risk in each partition.

Original: Since we are only concerned with the existence of a desired minimizer of the empirical risk, we can view f θ as able to achieve any labeling of the training inputs that suffices the local Lipschitz constraint.

Original: Thus the empirical risk minimization is equivalent to the minimization of the empirical risk in each partition.

Original: The problem can thus be defined as, for each j = 1 , · · · , N ρε ,

Original: [Equation]

Original: where the constraint is imposed by the locally-Lipschitz continuity of f θ .

Original: By the following lemma, we show that the minimizer of such problem is achieved only if f θ (¯ x j ) is close to the sample mean.

Original: Lemma A.3.

Original: Let ¯ y =

Original: |S j | − 1 (cid:80) x ∈S j 1 y .

Original: The minimum of the problem (22) is achieved only if

Original: [Equation]

Original: Proof.

Original: We note that since the loss function we choose is strongly convex, to minimize the empirical risk, the prediction of any input x must be as close to the one-hot labeling as possible.

Original: Therefore the problem (22) can be formulated into a vector minimization where we can employ

Original: Karush–Kuhn–Tucker (KKT) theorem to find the necessary conditions of the minimizer.

Original: Let p i :

Original: =

Original: f θ ( x i ) and ˜ ε =

Original: Lρε for simplicity.

Original: We rephrase the problem (22) as

Original: [Equation]

Original: Case I.

Original: We first discuss the case when p k + ˜ ε < 1 for all k .

Original: First, we observe that for any p , the minimum of the above problem is achieved only if p y i i

Original: = p y i + ˜ ε .

Original: Because by contradiction, if p y i i < p

Original: y i + ˜ ε , we will have − log p

Original: y i i > − log( p y i + ˜ ε ) , and p y i + ˜ ε belongs to the feasible set, which means p y

Original: i i does not attain the minimum.

Original: The above problem can then be rephrased as

Original: [Equation]

Original: where we have neglected the condition associated with p k ̸

Original: =

Original: y i

Original: i , since they do not contribute to the objective, they can be chosen arbitrarily as long as the constraints are sufficed, and clearly the constraints are underdetermined.

Original: Let N k

Original: = (cid:80) i 1( y i = k ) , we have (cid:80) i log( p y i + ˜ ε ) =

Original: (cid:80) k N k log( p k + ˜ ε ) .

Original: Therefore the above problem is equivalent to

Original: [Equation]

Original: where ¯y ≡ [ N 1 /N, · · · , N k /N ]

Original: T is equal to the sample mean N − 1 (cid:80)

Original: i 1 y i .

Original: To solve the strongly convex minimization problem (25) it is easy to employ KKT conditions to show that

Original: [Equation]

Original: Case II.

Original: We now discuss the case when ˆp is the minimizer of (23) and there exists k ′ such that ˆp k ′ + ˜ ε ≥ 1 .

Original: And ˆp ̸ = p , where p = ¯y (1 + K ˜ ε ) − ˜ ε is the form of the minimizer in the previous case.

Original: Considering a non-trivial case p ∗ k ′ < 1 − ˜ ε .

Original: Otherwise the true label distribution is already close to the one-hot labeling, which is the minimizer of the empirical risk.

Original: Therefore by (cid:80) k ̸ = k ′ p k > ˜ ε we have the condition

Original: [Equation]

Original: [Equation]

Original: Now considering the minimization objective R ( p ) =

Original: − N − 1 (cid:80)

Original: i 1 y i · log p i .

Original: For all i with y i

Original: =

Original: k ′ , we must have p

Original: y i

Original: i = 1 , otherwise the optimal cannot be attained by contradiction.

Original: Then the minimization problem can be rephrased as

Original: [Equation]

Original: where the first constraint is imposed by ˆp k ′ ≥ 1 − ˜

Original: [Equation]

Original: Employ KKT conditions similarly we can have ˆp k = ¯y k /λ − ˜ ε where λ is a constant.

Original: By checking the constraint we can derive λ ≥ (cid:80) k ¯y k / ( K ˜ ε ) .

Original: However, the minimization objective

Original: [Equation]

Original: requires λ to be minimized.

Original: Therefore λ = (cid:80)

Original: k ̸ = k ′ ¯y k / ( K ˜ ε ) , which implies

Original: [Equation]

Original: [Equation]

Original: Now since ˆp = arg min p R ( p ) and ˆp ̸ = p , we must have R ( ˆp ) <

Original: R ( p ) .

Original: This means −

Original: [Equation]

Original: [Equation]

Original: which is reduced to (cid:88)

Original: [Equation]

Original: [Equation]

Original: But this is contradict to our assumption.

Original: We are now be able to bound the difference between the predictions of the training inputs produced by the empirical risk minimizer and the sample mean in each S j .

Original: To see that we have for each x ∈ S j .

Original: [Equation]

Original: By Equation (15) we then have for any x ∈ S j , with probability 1 − δ ,

Original: [Equation]

Original: which means the difference between the predictions and the true label distribution is also bounded.

Original: Step III:

Original: Show the disjoint partition is non-trivial.

Original: In (32), we have managed to bound the difference between the predictions yielded by an empirical risk minimizer and the true label distribution based on the cardinality of the subset |S j | , namely the number of inputs in j -partition.

Original: However |S j | is critical to the bound here as if |S j | = 1 , then (32) becomes a trivial bound.

Original: Here we show |S j | is non-negligible based on simple combinatorics.

Original: Lemma A.4.

Original: Let {S j } N ρε j =1 be a disjoint partition of the entire training set S .

Original: Denote S ( x ) as the partition that includes x . Let N ( x ) =

Original: |S ( x ) | and N = |S| .

Original: Then for any κ ≥ 1 ,

Original: [Equation]

Original: Proof.

Original: We note that the problem is to show the minimum number of x such that N ( x ) ≥ N/ ( κN ρε ) .

Original: This is equivalent to find the maximum number of x such that N ( x ) ≤ N/ ( κN ρε ) .

Original: Since we only have

Original: N ρε subsets, the maximum can be attained only if for N ρε − 1 subsets S , |S| = N/ ( κN ρε ) .

Original: Otherwise, if for any one of these subsets |S| < N/ ( κN ρε ) , then it is always feasible to let |S| = N/ ( κN ρε ) and the maximum increases.

Original: Similarly, if the number of such subsets is less than N ρε − 1 , then it is always feasible to let another subset subject to |S| = N/ ( κN ρε ) and the maximum increases.

Original: We can then conclude that at most N ( N ρε − 1) / ( κN ρε ) inputs can have the property N ( x ) ≤ N/ ( κN ρε ) .

Original: The above lemma basically implies when partitioning N inputs into N ρε subsets, a large fraction of the inputs will be assigned to a subset with cardinality at least N/ ( κN ρε ) .

Original: Here N ρε is the covering number and is bounded above based on the property of the covering in the Euclidean space. Apply Lemma A.4 to (32), and use the fact that ∥ · ∥ TV = ∥ · ∥ 1 / 2 for category distributions, we then arrive at Lemma 4.4. Proof of Theorem 4.5.
Revised: The U.S. Government is authorized to reproduce and distribute reprints for government purposes not withstanding any copyright annotation hereon.

Original: Proof.

Original: First, we show that adversarial perturbation generated by a realistic classifier can change its predictive distribution.

Original: Considering adversarial perturbation based on FGSM and cross-entropy loss, namely x ′ = x − ε ∥∇ f θ ( x ) y ∥

Original: − 1 ∇ f θ ( x ) y , we can obtain a result similar to Lemma 4.1.

Original: Lemma A.5.

Original: Assume f θ ( x ) y is L θ -locally

Original: Lipschitz around x with bounded Hessian.

Original: Let σ m =

Original: inf

Original: z ∈B ε ( x ) σ min ( ∇ 2 f θ ( z ) y ) > 0 and σ M = sup z ∈B ε ( x ) σ max ( ∇ 2 f θ ( z ) y ) > 0 .

Original: Here σ min and σ max denote the minimum and maximum eigenvalues of the Hessian, respectively.

Original: We then have

Original: [Equation]

Original: Second, We prove that the true label distribution will be distorted by the adversarial perturbation generated by a realistic classifier.

Original: This is guaranteed if the predictive distribution of a realistic classifier can approximate the true label distribution.

Original: Specifically, by utilizing Lemma A.5 and

Original: Lemma 4.4, we have with probability 1 − 2 δ ,

Original: [Equation]

Original: Finally, we show that such distribution mismatch induces label noise in the adversarially augmented training set.

Original: Similar to the proof for the true classifier, by the common labeling practice of adversarial examples we have P ( ˜ Y ′

Original: | x ′ )

Original: =

Original: P ( ˜ Y | x ) =

Original: P ( Y | x ) .

Original: By utilizing Lemma 4.2 1 we then have with probability 1 − 3 δ ,

Original: [Equation]

Original: where ξ = 1 + (cid:112) 4 κN ρε K .

Original: A.4 Proofs in

Original: Proof of Theorem 5.1.

Original: Proof.

Original: Let j ∗ =

Original: argmax P ( Y ′ =

Original: j | x ′ ) and thus P ( Y ′ =

Original: j ∗ | x ′ ) ∈

Original: [1 /c, 1] .

Original: Let g ( T ) :

Original: = f ( x ′ ; θ, T ) j ∗ , which is a continuous function defined on [0 , ∞ ] .

Original: The condition j ∗ = arg max j f ( x ′ ; θ, T ) j ensures that g ( T ) ∈

Original: [1 /c, 1] , where c is the number of classes.

Original: By the intermediate value theorem, there exists T ∗ , such that g ( T ∗ ) =

Original: P ( Y ′ =

Original: j ∗ | x ′ ) .

Original: Let T

Original: =

Original: T ∗ , we have

Original: [Equation]

Original: where the inequality holds by the triangle inequality.

Original: Meanwhile, we have

Original: [Equation]

Original: Therefore, it can seen that for T

Original: =

Original: T ∗ ,

Original: [Equation]

Original: Proof of Theorem 5.2.

Original: Lemma A.6.

Original: Let x ′ be an example incorrectly classified by a classifier f

Original: in terms of the true label distribution P ( Y ′ =

Original: j | x ′ ) , namely

Original: [Equation]

Original: where j ∗ = arg max j P

Original: ( Y ′ =

Original: j | x ′ ) .

Original: Assume P

Original: ( Y ′ =

Original: j ∗ | x ′ ) ≥ 1 / 2 , then f ( x ′ ; θ, T

Original: [Equation]

Original: j ∗ ≤ P

Original: Y

Original: [Equation]

Original: = j ∗ | x ′ ) .

Original: Proof.

Original: We prove it by contradiction.

Original: Assume f ( x ′ ; θ, T ) j ∗ > P ( Y ′ =

Original: j ∗ | x ′ ) , we have f ( x ′ ; θ, T ) j ∗ > P ( Y ′ =

Original: j ∗ | x ′ ) ≥ 1 / 2 .

Original: Therefore,

Original: [Equation]

Original: =

Original: [Equation]

Original: ∗ f ( x ′ ; θ, T ) j = 1 − f ( x ′ ; θ, T ) j ∗ < 1 / 2 , ∀ j ̸ =

Original: j ∗ , which means f ( x ′ ; θ, T )

Original: j < f ( x ′ ; θ, T ) j ∗ , ∀ j ̸

Original: = j ∗ .

Original: This leads to j ∗ = arg max j f ( x ′ ; θ, T ) j , which contradicts our condition.

Original: Now we prove Theorem 5.

Original: Proof.

Original: First let P ( Y ′ | x ′ )

Original: =

Original: P ( y | x ) ≈ 1 ( y ) .

Original: Let j ∗

Original: = arg max j P

Original: ( Y ′ =

Original: j | x ′ ) .

Original: By Lemma A.6 we have f ( x ′ ; θ, T ) j ∗ ≤ P ( Y ′∗ =

Original: j ∗ | x ′ ) ≤ 1 .

Original: Then there exists λ ∗ > 0 , such that λ ∗ · f ( x ′ ; θ, T ) j ∗ + (1 − λ ∗ )

Original: =

Original: P ( Y ′ =

Original: j ∗ | x ′ ) by the intermediate value theorem.

Original: Let λ = λ ∗ , we have

Original: [Equation]

Original: | f ( x ′ ; θ, T ) j ∗ − P

Original: ( Y ′ =

Original: j ∗ | x ′ ) | ≤ (cid:88) j,j ̸

Original: = j ∗ | λ · f ( x ′ ; θ, T ) j − f ( x ′ ; θ, T ) j | − | f ( x ′ ; θ, T ) j ∗ − P

Original: ( Y ′ =

Original: j ∗ | x ′ ) | = (cid:88) j,j ̸ = j ∗ [ f ( x ′ ; θ, T ) j − λ · f ( x ′ ; θ, T ) j ] − [ P ( Y ′ =

Original: j ∗ | x ′ )

Original: − f ( x ′ ; θ, T ) j ∗ ] =

Original: (cid:88) j,j ̸

Original: = j ∗ [ f ( x ′ ; θ, T ) j − λ · f ( x ′ ; θ, T ) j ] − [ λ · f ( x ′ ; θ, T ) j ∗ + (1 − λ ) − f ( x ′ ; θ, T ) j ∗ ] = (cid:88)

Original: j f

Original: [Equation]

Original: = 0 .

Original: B Limitations

Original: We note that alternative labeling of adversarial examples proposed in this paper is based on the fact that the predictive distribution of a classifier trained with empirical risk minimization can approximate the true label distribution of training examples.

Original: However, such approximation may not be accurate especially if the classifier is not carefully regularized during training.

Original: Post-training confidence calibration techniques such as temperature scaling and interpolation can only improve the approximation in terms of the entire training set, but cannot improve it in a sample-wise manner.

Original: How to learn the true label distribution of adversarial training examples during adversarial training more accurately remains an open problem.

Original: Also, such alternative labeling also requires to train another independent classifier beforehand, which induces additional training cost.

Original: C More empirical analyses C.1 Epoch-wise double descent is ubiquitous in adversarial training

Original: In this section, we conduct extensive experiments with different model architectures, and learning rate schedulers to verify the connection between robust overfitting and epoch-wise double descent.

Original: The default experiment settings are listed in Appendix G.2 in detail.

Original: Model

Original: capacity

Original: We modulate the capacity of the deep model by varying the widening factor of the Wide ResNet.

Original: To extend the lower limit of the capacity, we allow the widening factor to be less than 1 .

Original: In such case, the number of channels in each residual block is scaled similarly but rounded, and the number of channels in the first convolutional layer will be reduced accordingly to ensure the width monotonically increasing through the forward propagation.

Original: Model architecture.

Original: We also experiment on model architectures other than Wide ResNet, including pre-activation ResNet-18 (He et al., 2016) and VGG-11 (Simonyan & Zisserman, 2015).

Original: We select

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: these configurations to ensure comparable model capacities 2 .

Original: As shown in Figure 8, different model architectures may produce slightly different double descent curves.

Original: The second descent of VGGin particular will be delayed due to its inferior performance compared to residual architectures.

Original: Learning rate scheduler.

Original: A specific learning rate scheduler may shape the robust overfitting differently as suggested by Rice et al.

Original: We consider the following learning rate schedulers in our experiments.

Original: • Piecewise decay : The initial learning rate rate is set as 0 .

Original: and is decayed by a factor of 10 at the 100 th and 500 th epochs within a total of 1000 epochs.

Original: • Cyclic : This scheduler was initially proposed by Smith (2017) and has been popular in adversarial training.

Original: We set the maximum learning rate to be 0 . 2 , and the learning rate will linearly increase from 0 to 0 .

Original: 2 for the initial 400 epochs and decrease to 0 for the later 600 epochs.

Original: • Cosine : This scheduler was initially proposed by Loshchilov & Hutter (2017).

Original: The learning rate starts at 0 .

Original: 1 and gradually decrease to 0 following a cosine function for a total of 1000 epochs.

Original: Experiments

Original: on various learning rate schedulers show the second descent can be widely observed except the piecewise

Original: decay, where the appearance of

Original: second

Original: descent might be delayed due to

Original: extremely small learning rate in the late stage of training.

Original: D More experiment results D.1 Training longer

Original: As shown in Figure 9, We show that our method can maintain the robust test accuracy with more training epochs.

Original: Here, we follow the settings in Figure 7 except we train for additional epochs up to 400 epochs for each dataset.

Original: D.2 Adversarial training methods, neural architectures and evaluation metrics

Original: In this section we conduct extensive experiments with different neural architectures, adversarial training methods and robustness evaluation metrics to verify the effectiveness of our method.

Original: [Figure]

Original: Tiny

Original: [Figure]

Original: ImageNet

Original: [Figure]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: KD

Original: [Table]

Original: Auto

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: WRN-28AT - -

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: KD

Original: [Table]

Original: AT

Original: [Table]

Original: KD

Original: [Table]

Original: AT-

Original: [Table]

Original: [Table]

Original: [Table]

Original: D.3 Combined with additional orthogonal techniques

Original: We note that motivated from our theoretical analyses, our proposed method (KD-AT-Auto) is essentially the baseline knowledge distillation for adversarial training (KD-AT) with a robustly trained self-teacher, equipped with an algorithm that automatically finds its optimal hyperparameters (i

Original: [Table]

Original: e. the temperature T and the interpolation ratio λ ).

Original: Stochastic Weight Averaging (SWA) and additional standard teachers (KD-Std) employed in (Chen et al., 2021) are orthogonal contributions.

Original: KD-AT-Auto can certainly be combined with SWA and KD-Std to achieve better performance.

Original: As shown in Table

Original: [Table]

Original: on CIFAR-

Original: [Table]

Original: KD-AT + KD-Std + SWA

Original: (Chen et al., 2021) can already reduce the overfitting gap (difference between the best and last robust accuracy) to almost 0 .

Original: It is thus hard to see any further reduction by combining our method.

Original: To this end, we introduce an extra dataset

Original: SVHN (Netzer et al., 2011).

Original: As shown in Table 5, on SVHN, KD-AT + KD-Std + SWA still produces a high overfitting gap (also see Appendix A1.3 in (Chen et al., 2021)), whereas by combining with our algorithm to automatically find the optimal hyper-parameters (KD-AT-Auto + KD-Std + SWA), the overfitting gap can be further reduced to almost 0 .

Original: This demonstrates the effectiveness and wide applicability of our principle-guided method on mitigating robust overfitting.

Original: [Table]

Original: [-Here, the interpolation ratio of the standard teacher is fixed as 0 . 
-]
Revised: [-Here, the interpolation ratio of the standard teacher is fixed as 0 . 
-]

Original: and the SWA starts at the first learning rate decay for all experiments.

Original: We employ PGD-AT (Madry et al., 2018) as the base adversarial training method and conduct experiments with a pre-activation ResNet-18.

Original: The robust accuracy is evaluated with AutoAttack.

Original: Other experiment details are in line with Appendix G.1.

Original: Furthermore, we note that (Chen et al., 2021) shows SWA and KD-Std are essential components to mitigate robust overfitting on top of KD-AT, while we show that KD-AT itself can mitigate robust overfitting by proper parameter tuning.

Original: We are thus able to separate these components and allow a more flexible selection of hyperparameters in diverse training scenarios without fear of overfitting.

Original: In particular, although (Chen et al., 2021) suggests SWA starting at the first learning rate decay (exactly when the overfitting starts) mitigates robust overfitting, the effectiveness of SWA on mitigating overfitting may strongly depend on its hyper-parameter selection including s 0 , i.e., the starting epoch and τ , i.e., the decay rate 3 , which is also mentioned in recent work (Rebuffi et al., 2021).

Original: We also did some additional experiments on CIFAR-10 following the SWA setting in (Rebuffi et al., 2021) to demonstrate the wide applicability of our method.

Original: As shown by Table 6, when changing the hyperparameters of SWA, KD-AT + KD-Std + SWA cannot consistently mitigate robust overfitting, while KD-AT-Auto + KD-Std + SWA can maintain an overfitting gap close to 0 and achieve better robustness as well.

Original: [Table]

Original: E Study on a synthetic dataset with known true label distribution

Original: [Figure]

Original: Synthetic Dataset.

Original: Since the true label distribution is typically unknown for adversarial examples in real-world datasets, we simulate the mechanism of implicit label noise in adversarial training from a feature learning perspective.

Original: Specifically, we adapt mixup (Zhang et al., 2018) for data augmentation on

Original: CIFAR-10.

Original: For every example x in the training set, we randomly select another example x ′ in a different class and linearly interpolate them by a ratio ρ , namely x := ρx + (1 − ρ ) x ′ , which essentially perturbs x with features from other classes.

Original: Therefore, the true label distribution is arguably y ∼ ρ · 1 ( y )

Original: +(1 − ρ ) · 1 ( y ′ ) .

Original: Unlike mixup, we intentionally set the assigned label as ˆ y ∼ 1 ( y ) , thus deliberately create a mismatch between the true label distribution and the assigned label distribution.

Original: We refer this strategy as mixup augmentation and only perform it once before the training.

Original: In this way, the true label distribution of every example in the synthetic dataset is fixed.

Original: Concentration of optimal temperature and interpolation ratio of individual examples.

Original: In

Original: Section 5.1 we have shown that in terms of individual examples, the rectified model probability can provably reduce the distribution mismatch between the assigned label distribution and true label distribution of the adversarial example.

Original: However, since the true label distribution is unknown in realistic scenarios, it is not possible to directly follow Theorems 5.1 and 5.2 and calculate the optimal set of hyper-parameters for each example in the training set.

Original: The best we can do is to employ a validation set and determine a universal set of hyper-parameters based on the NLL loss, which expects all training examples to share similar optimal temperatures and interpolation ratios.

Original: Here, based on the synthetic dataset where a true label distribution is known, we empirically verify this assumption is reasonable.

Original: [Figure]

Original: In Figure 11 left, we solve the optimal temperature for each correctly classified training example based on Theorem 5.1 with the interpolation ratio fixed as 1 . 0 .

Original: One can find that the individual optimal temperatures mostly concentrate between 0 . 5 and 1 . 5 .

Original: In Figure 11 right, we solve the optimal interpolation ratio for each incorrectly classified training example based on Theorem 5.

Original: with the temperature fixed as 1 . 0 .

Original: One can find that the individual optimal interpolation ratio mostly concentrate between 0 . 5 and 0 .

Original: F Method details F.1 Determine the optimal hyper-parameters

Original: One may note that Equation (9) cannot be directly optimized since the traditional adversarial label is only defined on the example in the training set and cannot be simply generalized to the validation set.

Original: A reasonable solution is using the nearest neighbour classifier to find the closest traditional adversarial label for every example in the validation set.

Original: However, to speed up the optimization, we propose to employ the classifier overfitted by the traditional adversarial labels on the training set as an surrogate, which works well in practice.

Original: Specially, we employ a model overfitted on the training set to generate approximate traditional adversarial label of the adversarial example in the validation set.

Original: Such overfitted model is typically the model at the final checkpoint when conducting regular adversarial training for sufficient epochs.

Original: Mathematically, our final method to determine the optimal temperature and interpolation ratio in rectified model probability can be described as

Original: [Equation]

Original: [Equation]

Original: where f θ s ( x ′ ; T ) denotes the temperature-scaled predictive probability of a surrogate model on x ′ .

Original: Here the validation set is constructed by applying adversarial perturbation generated by f θ to the clean validation set.

Original: For adversarial perturbation we utilize PGD attack with 10 iterations, the perturbation radius as 8 / 255 and the step size as 2 / 255 .

Original: Note that Such process incurs almost no additional computation as we simply obtain the logits of a surrogate classifier.

Original: G Experimental details G.1 Settings for main experiment results

Original: Dataset.

Original: We include experiment results on CIFAR-10, CIFAR-100, Tiny-ImageNet and SVHN.

Original: Training setting.

Original: We employ SGD as the optimizer.

Original: The batch size is fixed to 128.

Original: The momentum and weight decay are set to 0 .

Original: 9 and 0 . 0005 respectively.

Original: Other settings are listed as follows

Original: • CIFAR-10/CIFAR-100: we conduct the adversarial training for 160 epochs, with the learning rate starting at 0 . 1 and reduced by a factor of 10 at the 80 and 120 epochs.

Original: • Tiny-ImageNet: we conduct the adversarial training for 80 epochs, with the learning rate starting at 0 . 1 and reduced by a factor of 10 at the 40 and 60 epochs.

Original: • SVHN: we conduct the adversarial training for 80 epochs, with the learning rate starting at 0 .

Original: 01 (as suggested by (Chen et al., 2021)) and reduced by a factor of 10 at the 40 and 60 epochs.

Original: Adversary setting.

Original: We conduct adversarial training with ℓ ∞ norm-bounded perturbations.

Original: We employ adversarial training methods including PGD-AT, TRADES and FGSM.

Original: We set the perturbation radius to be 8 / 255 .

Original: For PGD-AT and TRADES, the step size is 2 / 255 and the number of attack iterations is 10 .

Original: Robustness evaluation.

Original: We consider the robustness against ℓ ∞ norm-bounded adversarial attack with perturbation radius 8 / 255 .

Original: We employ AutoAttack for reliable evaluation.

Original: We also include the evaluation results again PGD-1000, Square Attack and RayS.

Original: Neural architectures.

Original: We include experiments results on pre-activation ResNet-18, WRN-28-5,

Original: WRN-34-10 and VGG-19.

Original: Hardware.

Original: We conduct experiments on NVIDIA Quadro RTX A6000.

Original: G.

Original: Settings for analyzing double descent in adversarial training

Original: Dataset.

Original: We conduct experiments on the CIFAR-10 dataset, without additional data.

Original: Training setting

Original: We conduct the adversarial training for 1000 epochs unless otherwise noted.

Original: By default we use SGD as the optimizer with a fixed learning rate 0 . 1 .

Original: When we experiment on a subset (see below)

Original: we use the Adam optimizer to improve training stability, where the learning rate is fixed as 0 . 0001 .

Original: The batch size will be fixed to 128 , and the momentum will be set as 0 .

Original: 9 wherever necessary.

Original: No regularization such as weight decay is used.

Original: These settings are mostly aligned with the empirical analyse of double descent under standard training (Nakkiran et al., 2020).

Original: Sample size.

Original: To reduce the computation load demanded by an exponential number of training epochs, we reduce the size of the training set by randomly sampled a subset of size 5000 from the original training set without replacement.

Original: We adopt this setting for extensive experiments for analyzing the dependence of epoch-wise double descent on the perturbation radius and data quality (i.e. Figure 5)..

Original: Adversary

Original: setting.

Original: We conduct adversarial training with ℓ ∞ norm-bounded perturbations.

Original: We employ standard PGD training with the perturbation radius set to 8 / 255 unless otherwise noted.

Original: The number of attack iterations is fixed as 10 , and the perturbation step size is fixed as 2 / 255 .

Original: Robustness evaluation.

Original: We consider the robustness against ℓ ∞ norm-bounded adversarial attack with perturbation radius 8 / 255 .

Original: We use PGD attack with 10 attack iterations and step size set to 2 / 255 .

Original: Neural architecture.

Original: By default we experiment on Wide ResNet (Zagoruyko & Komodakis, 2016) with depth 28 and widening factor 5 (WRN-28-5) to speed up training.

Original: Hardware.

Original: We conduct experiments on NVIDIA Quadro RTX A6000.

Original: G.3 Estimation of the data quality

Original: In this section we elaborate on the calculation of data quality for analyzing the dependence on label noise in adversarial training.

Original: We use the predicative probabilities of classifiers trained on CIFAR-10 to score its training data.

Original: Similar strategy is employed in previous works to select high-quality unlabeled data to improve adversarial robustness (Uesato et al., 2019; Carmon et al., 2019; Gowal et al., 2020).

Original: Slightly deviating from these works focusing on out-of-distribution data, we use adversarially trained instead of regularly trained models to measure the quality of in-distribution data, since under standard training almost all training examples will be overfitted and gain overwhelmingly high confidence.

Original: Specifically, we adversarially train a pre-activation ResNet-18 with PGD and select the model at the best checkpoint in terms of the robustness.

Original: The quality of an example is estimated by the model probability corresponding to the true label without adversarial perturbation and random data augmentation (flipping and clipping).

Original: We repeat this process 10 times with random initialization to obtain a relatively accurate estimation.

Original: G.4 Settings for standard training on fixed augmented training sets

Original: G

Original: 1 General settings for both adversarial augmentation and Gaussian augmentation

Original: Dataset.

Original: We conduct experiments on the CIFAR-10 dataset, without additional data.

Original: Training setting.

Original: We conduct the standard training for 1000 epochs.

Original: We use Adam as the optimizer with a fixed learning rate 0 . 0001 to improve training stability with a small training set (see below).

Original: The batch size will be fixed to 128 , and the momentum will be set as 0 .

Original: 9 wherever necessary.

Original: No regularization such as weight decay is used.

Original: Sample size.

Original: To reduce the computation load demanded by an exponential number of training epochs, we reduce the size of the training set by randomly sampled a subset of size 5000 from the original training set without replacement.

Original: Neural architecture.

Original: By default we experiment on Wide ResNet (Zagoruyko & Komodakis, 2016) with depth 28 and widening factor 5 (WRN-28-5).

Original: Hardware.

Original: We conduct experiments on NVIDIA Quadro RTX A6000.

Original: G.4.2 Construction of the training set

Original: Adversarial augmentation.

Original: We first obtain a robust model by conduct PGD training with preactivation ResNet-18 on CIFAR-10.

Original: We use early stopping to obtain the most robust model on a validation set.

Original: The specific settings are aligned with Section G.1.

Original: Using this model, we then generate adversarial examples with PGD attack on the 5000 examples randomly sampled from CIFAR-10 training set.

Original: The number of attack iterations is fixed as 10 and the step size is fixed as 2 / 255 .

Original: The adversarial examples along with their original labels are then grouped into a training set for adversarial augmentation experiments.

Original: Gaussian augmentation.

Original: We apply Gaussian noise to the 5000 examples randomly sampled from

Original: CIFAR-10 training set.

Original: The perturbed examples along with their original labels are then grouped into a training set for Gaussian augmentation experiments.