Original: U NDERSTANDING AND M ITIGATING R OBUST O VERFITTING THROUGH THE L ENS OF F EATURE D YNAMICS
Revised: U NDERSTANDING AND M ITIGATING R OBUST O VERFITTING THROUGH THE L ENS OF F EATURE D YNAMICS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Adversarial Training (AT) has become arguably the state-of-the-art algorithm for extracting robust features.
Revised: Adversarial Training (AT) has become arguably the state-of-the-art algorithm for extracting robust features.

Original: However, researchers recently notice that AT suffers from severe robust overfitting problems, particularly after the learning rate (LR) decay, while the existing static view of feature robustness fails to explain this phenomenon.
Revised: However, researchers recently notice that AT suffers from severe robust overfitting problems, particularly after the learning rate (LR) decay, while the existing static view of feature robustness fails to explain this phenomenon.

Original: In this paper, we propose a new dynamic feature robustness framework which takes the dynamic interplay between the model trainer and the attacker into consideration.
Revised: In this paper, we propose a new dynamic feature robustness framework which takes the dynamic interplay between the model trainer and the attacker into consideration.

Original: By tracing temporal and dataset-specific feature robustness, we develop a new understanding of robust overfitting from the dynamics of non-robust features, and empirically verify it on real-world datasets.
Revised: By tracing temporal and dataset-specific feature robustness, we develop a new understanding of robust overfitting from the dynamics of non-robust features, and empirically verify it on real-world datasets.

Original: Built upon this understanding, we explore three techniques to restore the balance between the model trainer and the attacker, and show that they could effectively alleviate robust overfitting and attain state-of-the-art robustness on benchmark datasets.
Revised: Built upon this understanding, we explore three techniques to restore the balance between the model trainer and the attacker, and show that they could effectively alleviate robust overfitting and attain state-of-the-art robustness on benchmark datasets.

Original: Notably, different from previous studies, our interpretation highlights the necessity of considering the min-max nature of AT for robust overfitting.
Revised: Notably, different from previous studies, our interpretation highlights the necessity of considering the min-max nature of AT for robust overfitting.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Overfitting seems to have become a history in the deep learning era.
Revised: Overfitting seems to have become a history in the deep learning era.

Original: Contrary to the traditional belief in statistical learning theory that large hypothesis class will lead to overfitting, Zhang et al.
Revised: Contrary to the traditional belief in statistical learning theory that large hypothesis class will lead to overfitting, Zhang et al.

Original: (2019) note that DNNs have good generalization ability on test data even if they are capable of memorizing random training labels.
Revised: (2019) note that DNNs have good generalization ability on test data even if they are capable of memorizing random training labels.

Original: Nowadays, large-scale training often does not require early stopping, and longer training simply brings better generalization (Hoffer et al., 2017).
Revised: Nowadays, large-scale training often does not require early stopping, and longer training simply brings better generalization (Hoffer et al., 2017).

Original: However, researchers recently notice that in Adversarial Training (AT), overfitting is still a severe issue on both small and large scale data and models (Rice et al., 2020).
Revised: However, researchers recently notice that in Adversarial Training (AT), overfitting is still a severe issue on both small and large scale data and models (Rice et al., 2020).

Original: AT is arguably the most effective defense method (Athalye et al., 2018) against adversarially crafted perturbations to images (Szegedy et al., 2014).
Revised: AT is arguably the most effective defense method (Athalye et al., 2018) against adversarially crafted perturbations to images (Szegedy et al., 2014).

Original: Specifically, given training data D train and model f θ , AT can be formulated as a min-max optimization problem (Madry et al., 2018; Goodfellow et al., 2015):
Revised: Specifically, given training data D train and model f θ , AT can be formulated as a min-max optimization problem (Madry et al., 2018; Goodfellow et al., 2015):

Original: [Equation]
Revised: [Equation]

Original: where ℓ CE denotes the cross entropy (CE) loss, E p (¯ x ) = { x | ∥ x − ¯ x ∥ p ≤ ε } denotes the ℓ p -norm ball with radius ε .
Revised: where ℓ CE denotes the cross entropy (CE) loss, E p (¯ x ) = { x | ∥ x − ¯ x ∥ p ≤ ε } denotes the ℓ p -norm ball with radius ε .

Original: However, in practice, researchers notice this min-max training scheme suffers severely from the robust overfitting (RO) problem: after a particular point ( e.g., learning rate decay), its training robustness will keep increasing (Figure 1a, red line) while its test robustness will begin to dramatically decrease (Figure 1b, red line).
Revised: However, in practice, researchers notice this min-max training scheme suffers severely from the robust overfitting (RO) problem: after a particular point ( e.g., learning rate decay), its training robustness will keep increasing (Figure 1a, red line) while its test robustness will begin to dramatically decrease (Figure 1b, red line).

Original: This abnormal behavior of AT has attracted many interests recently.
Revised: This abnormal behavior of AT has attracted many interests recently.

Original: Previous work correctly points out that during AT, the robust loss landscape becomes much sharper, and RO can be largely surpassed by enforcing a smoother landscape with additional regularizations (Stutz et al., 2021;
Revised: Previous work correctly points out that during AT, the robust loss landscape becomes much sharper, and RO can be largely surpassed by enforcing a smoother landscape with additional regularizations (Stutz et al., 2021;

Original: Chen et al., 2021; Wu et al., 2020).
Revised: Chen et al., 2021; Wu et al., 2020).

Original: However, this phenomenological perspective does not explain the rise of either the sharp landscape or RO.
Revised: However, this phenomenological perspective does not explain the rise of either the sharp landscape or RO.

Original: Some researchers try to explain RO though the phenomenons of standard training (ST), such as random memorization (Dong et al., 2022) and double descent Dong et al.
Revised: Some researchers try to explain RO though the phenomenons of standard training (ST), such as random memorization (Dong et al., 2022) and double descent Dong et al.

Original: , but they fail to characterize the uniqueness of AT from ST (why AT has the overfitting issue while ST does not).
Revised: , but they fail to characterize the uniqueness of AT from ST (why AT has the overfitting issue while ST does not).

Original: In this paper, we seek to establish a generic explanation for robust overfitting during AT.
Revised: In this paper, we seek to establish a generic explanation for robust overfitting during AT.

Original: Different from previous attempts to relate the behaviors of AT to existing theories of ST, we believe that the reasons of RO should lie exactly in the differences between AT and ST.
Revised: Different from previous attempts to relate the behaviors of AT to existing theories of ST, we believe that the reasons of RO should lie exactly in the differences between AT and ST.

Original: In particular, we emphasize a critical difference: RO usually happens afterthe learning rate (LR) decay in AT, while in ST, LR decay does not lead to (severe) overfitting, so our paper focus on the change of learning behaviors before and after the LR decay.
Revised: In particular, we emphasize a critical difference: RO usually happens after learning rate (LR) decay in AT, while in ST, LR decay does not lead to (severe) overfitting, so our paper focus on the change of learning behaviors before and after the LR decay.

Original: We notice that feature robustness actually changes during this process: a non-robust feature could become robust afterLR decays, which is contrary to the static feature
Revised: We notice that feature robustness actually changes during this process: a non-robust feature could become robust after

Revised: LR decays

Original: [Figure]
Revised: which is

Revised: contrary

Original:  robustness framework developed by Ilyas et al.
Revised: to the static feature robustness framework developed by Ilyas et al.

Original: This motivates us to design a dynamic feature  robustness framework that takes the learning process into consideration, and discussion the dynamics of feature robustness.
Revised: This motivates us to design a dynamic feature [Figure] robustness framework that takes the learning process into , and discussion the dynamics of feature robustness.

Original: Moreover, this dynamic perspective also suggests a new understanding of robust overfitting.
Revised: Moreover, this dynamic perspective also suggests a new understanding of robust overfitting.

Original: Specifically, we hypothesize that due to the strong local fitting ability endowed by smaller LR, the model learns false mapping of non-robust features contained in adversarial examples, which opens shortcuts for the test-time attacker and induces large test robust error.
Revised: Specifically, we hypothesize that due to the strong local fitting ability endowed by smaller LR, the model learns false mapping of non-robust features contained in adversarial examples, which opens shortcuts for the test-time attacker and induces large test robust error.

Original: Accordingly, we design a series of experiments that empirically verify this understanding.
Revised: Accordingly, we design a series of experiments that empirically verify this understanding.

Original: Based on this dynamic view of feature robustness, we investigate three new strategies to avoid fitting non-robust features: stronger model regularization, smaller LR decay, and stronger attacker.
Revised: Based on this dynamic view of feature robustness, we investigate three new strategies to avoid fitting non-robust features: stronger model regularization, smaller LR decay, and stronger attacker.

Original: We show that all three strategies help mitigate robust overfitting to a large extent.
Revised: We show that all three strategies help mitigate robust overfitting to a large extent.

Original: Based on these insights, we propose Bootstrapped Adversarial Training (BoAT) that has neglectable (if any) degree of robust overfitting without using additional data augmentations, even if we train for 500 epochs.
Revised: Based on these insights, we propose Bootstrapped Adversarial Training (BoAT) that has neglectable (if any) degree of robust overfitting without using additional data augmentations, even if we train for 500 epochs.

Original: Meanwhile,
Revised: Meanwhile,

Original: BoAT attains state-of-the-art robustness (both best-epoch and final-epoch) on benchmark datasets including CIFAR-10, CIFAR-100, and Tiny-ImageNet.
Revised: BoAT attains state-of-the-art robustness (both best-epoch and final-epoch) on benchmark datasets including CIFAR-10, CIFAR-100, and Tiny-ImageNet.

Original: This suggests that with appropriate design of training strategies, AT could also enjoy similar “train longer, generalize better” property like ST.
Revised: This suggests that with appropriate design of training strategies, AT could also enjoy similar “train longer, generalize better” property like ST.

Original: To summarize, our main contributions are:
Revised: To summarize, our main contributions are:

Original: • We point out that the existing static robust feature framework fails to explain robust overfitting, particularly after the LR decay.
Revised: • We point out that the existing static robust feature framework fails to explain robust overfitting, particularly after the LR decay.

Original: Therefore, we propose a new dynamic robust feature framework to account for this change by taking the interplay among the model trainer and the attacker into consideration.
Revised: Therefore, we propose a new dynamic robust feature framework to account for this change by taking the interplay among the model trainer and the attacker into consideration.

Original: • Based on our dynamic framework, we analyze the change of feature robustness during LR decay.
Revised: • Based on our dynamic framework, we analyze the change of feature robustness during LR decay.

Original: We further propose a new understanding of robust overfitting through the change of non-robust features, and empirically verify it on three nontrivial implications.
Revised: We further propose a new understanding of robust overfitting through the change of non-robust features, and empirically verify it on three nontrivial implications.

Original: • From our dynamic perspective, we propose three effective approaches that mitigate robust overfitting by re-striking a balance between the model trainer and the attacker.
Revised: • From our dynamic perspective, we propose three effective approaches that mitigate robust overfitting by re-striking a balance between the model trainer and the attacker.

Original: Experiments show that our proposed BoAT largely alleviates robust overfitting and attains state-of-theart robustness.
Revised: Experiments show that our proposed BoAT largely alleviates robust overfitting and attains state-of-theart robustness.

Original:  A N EW D YNAMIC F RAMEWORK FOR R OBUST AND N ON - ROBUST F EATURES
Revised:  A N EW D YNAMIC F RAMEWORK FOR R OBUST AND N ON - ROBUST F EATURES

Original: In this section, we explain how traditional static framework of feature robustness analysis fails to explain the robust overfitting problem in AT.
Revised: In this section, we explain how traditional static framework of feature robustness analysis fails to explain the robust overfitting problem in AT.

Original: Motivated by this fact, we establish a dynamic view of feature robustness that centers on the adversarial learning process.
Revised: Motivated by this fact, we establish a dynamic view of feature robustness that centers on the adversarial learning process.

Original: 2. L IMITATIONS OF T RADITIONAL S TATIC F EATURE R OBUSTNESS F RAMEWORK
Revised: 2. L IMITATIONS OF T RADITIONAL S TATIC F EATURE R OBUSTNESS F RAMEWORK

Original: The arguably most prevailing understanding of adversarial training is the feature robustness framework proposed by Ilyas et al.
Revised: The arguably most prevailing understanding of adversarial training is the feature robustness framework proposed by Ilyas et al.

Original: (Ilyas et al., 2019).
Revised: (Ilyas et al., 2019).

Original: They regard natural images as a composition of both robust and non-robust features that can both generalize ( i.e., useful), and the use of non-robust features in standard training results in its adversarial vulnerability.
Revised: They regard natural images as a composition of both robust and non-robust features that can both generalize ( i.e., useful), and the use of non-robust features in standard training results in its adversarial vulnerability.

Original: Adversarial training is to prune the non-robust features and the learned robust model only makes use of robust features.
Revised: Adversarial training is to prune the non-robust features and the learned robust model only makes use of robust features.

Original: Generally, for a distribution P and model class F , we can define the robustness of a feature f ∈ F : X → Y as a real value, and decide whether it is a (non-)robust feature based on a given threshold γ ∈ [0 , 1] : 
Revised: Generally, for a distribution P and model class F , we can define the robustness of a feature f ∈ F : X → Y as a real value, and decide whether it is a (non-)robust feature based on a given threshold γ ∈ [0 , 1] : [Equation] Ilyas [Equation]

Revised: al. ’s definition)

Original: [Equation]
Revised: [Equation]

Original: Limitations.
Revised: Limitations.

Original: However, we notice that this feature robustness definition fails to explain robust overfitting.
Revised: However, we notice that this feature robustness definition fails to explain robust overfitting.

Original: Because the robustness of a feature f only depends on the attacker A and the distribution
Revised: Because the robustness of a feature f only depends on the attacker A and the distribution

Original: P that are both static across training, the robustness of f should also be unchanged under different decays.
Revised: P that are both static across training, the robustness of f should also be unchanged under different decays.

Original: The core issue here is that they only consider the attacker A , while adversarial training is essentially a dynamic min-max game between two players, the attacker A (inner-loop) and the model trainer T (the outer-loop), both of which are typically based on first-order optimization algorithms.
Revised: The core issue here is that they only consider the attacker A , while adversarial training is essentially a dynamic min-max game between two players, the attacker A (inner-loop) and the model trainer T (the outer-loop), both of which are typically based on first-order optimization algorithms.

Original: As a result, feature robustness will dynamically vary depending on the relative strength between A and T .
Revised: As a result, feature robustness will dynamically vary depending on the relative strength between A and T .

Original: Another failure of Ilyas et al.
Revised: Another failure of Ilyas et al.

Original: (2019) is that they only consider the population-level robustness over the distribution P (Eq. 2) and regard robustness as an intrinsic data property.
Revised: (2019) is that they only consider the population-level robustness over the distribution P (Eq. 2) and regard robustness as an intrinsic data property.

Original: However, the large robust generalization gap in Figure 1a and 1b shows that a model can behave quite differently even on the same distribution.
Revised: However, the large robust generalization gap in Figure 1a and 1b shows that a model can behave quite differently even on the same distribution.

Original: Therefore, we should distinguish the robustness on the training and test dataset.
Revised: Therefore, we should distinguish the robustness on the training and test dataset.

Original: To summarize, in order to explain the robust overfitting phenomenon, we need to go beyond Ilyas et al.
Revised: To summarize, in order to explain the robust overfitting phenomenon, we need to go beyond Ilyas et al.

Original: ’s
Revised: ’s

Original: original framework, and incorporate the adversarial game between A and T on a specific dataset
Revised: original framework, and incorporate the adversarial game between A and T on a specific dataset

Original: D into the characterization of robust features, as we will do in the next part.
Revised: D into the characterization of robust features, as we will do in the next part.

Original: 2. T HE P ROPOSED D YNAMIC F EATURE R OBUSTNESS F RAMEWORK
Revised: 2. T HE P ROPOSED D YNAMIC F EATURE R OBUSTNESS F RAMEWORK

Original: Different from Ilyas et al.
Revised: Different from Ilyas et al.

Original: (2019), we define a feature g as a filter of the input g : X → S , where S denotes the feature space.
Revised: (2019), we define a feature g as a filter of the input g : X → X .

Original: Canonical concepts of features (such as, shape, texture, SIFT features) naturally fall into this definition.
Revised: Canonical data features (such as, shape, texture) naturally fall into this definition.

Original: More importantly, our definition of feature is classifier-independent, which enables us to explicitly model the attacker A and the trainer T as two independent factors that affect feature robustness.
Revised: More importantly, our definition of feature is classifier-independent, which enables us to explicitly model the attacker A and the trainer T as two independent factors that affect feature robustness.

Original: Specifically, we define the robustness of feature g as a function of the attack region ∆ , the model trainer T , and the dataset D : 
Revised: Specifically, we define the robustness of feature g as a function of the attack region ∆ , the model trainer T , and the dataset D : [Equation] our

Original: [Equation]
Revised: [Equation]

Original: where F T ⊂ F denotes the attainable subset of models under the trainer T , and F T g further denotes the subset of F T that utilzes feature g .
Revised: where F T ⊂ F denotes the attainable subset of models under the trainer T (a specific training configuration), and F T g further denotes the subset of F T that utilzes feature g .

Original: 1 Many choices of the trainer T affect have a large impact on F t , such as the optimizer, learning rate and its schedule, weight decay, etc.
Revised: 1 Many choices of the trainer T affect have a large impact on F t , such as the optimizer, learning rate and its schedule, weight decay, etc.

Original: In particular, as shown in Figure 1b, the change of learning rate has a huge impact on model robustness.
Revised: In particular, as shown in Figure 1b, the change of learning rate has a huge impact on model robustness.

Original: Theoretically, Li et al.
Revised: Theoretically, Li et al.

Original: (2019) prove that different learning rates indeed have a strong regularization effect on NN training.
Revised: (2019) prove that different learning rates indeed have a strong regularization effect on NN training.

Original: Therefore, in this work, we mainly study the effect of LR decay on the change of feature robustness.
Revised: Therefore, in this work, we mainly study the effect of LR decay on the change of feature robustness.

Original: Besides the introduction of dataset D , a key difference from Ilyas et al.
Revised: Besides the introduction of dataset D , a key difference from Ilyas et al.

Original: (2019) is that in our framework, the robustness of a feature is not decided as a priori , but dependent on a dynamic game between
Revised: (2019) is that in our framework, the robustness of a feature is not decided as a priori , but dependent on a dynamic game between

Original: A and T .
Revised: A and T .

Original: Specifically, a feature g is robust if the trainer T can find a model f
Revised: Specifically, a feature g is robust if the trainer T can find a model f

Original: g that is resistant against the attacker A , otherwise, g is non-robust and discarded by the trainer.
Revised: g that is resistant against the attacker A , otherwise, g is non-robust and discarded by the trainer.

Original: More importantly, this situation could change over time.
Revised: More importantly, this situation could change over time.

Original: When we adopt a smaller LR after decay, the trainer poses a different regularization effect on the model class F T .
Revised: When we adopt a smaller LR after decay, the trainer poses a different regularization effect on the model class F T .

Original: As pointed out by Li et al.
Revised: As pointed out by Li et al.

Original: , small
Revised: , small

Original: LR often favors easy-to-generalize and hard-to-fit patterns.
Revised: LR often favors easy-to-generalize and hard-to-fit patterns.

Original: In the context of adversarial training, these features are often non-robust features (before LR decay).
Revised: In the context of adversarial training, these features are often non-robust features (before LR decay).

Original: Indeed, non-robust features enjoy good generalization (Ilyas et al., 2019), but they are hard to fit under the attacker.
Revised: Indeed, non-robust features enjoy good generalization (Ilyas et al., 2019), but they are hard to fit under the attacker.

Original: However, with a smaller LR, the trainer becomes more capable of memorizing these features by drawing more complex decision boundaries in a small local region.
Revised: However, with a smaller LR, the trainer becomes more capable of memorizing these features by drawing more complex decision boundaries in a small local region.

Original: Therefore, due to the change of relative strength between A and T , these non-robust features could become robust after LR decay.
Revised: Therefore, due to the change of relative strength between A and T , these non-robust features could become robust after LR decay.

Original: To verify the change of robustness, following a similar procedure as Ilyas et al.
Revised: To verify the change of robustness, following a similar procedure as Ilyas et al.

Original: (2019) (details in
Revised: (2019) (details in

Original: Appendix B.2), we extract non-robust features from models before LR decay, and examine their robustness after LR decay .
Revised: Appendix B.3) , we extract non-robust features from models before LR decay, and examine their robustness after LR decay .

Original: We also incorporate a control group that adopts the same training configuration except for the LR decay.
Revised: We also incorporate a control group that adopts the same training configuration except for the LR decay.

Original: Results are shown in Figure 1c.
Revised: Results are shown in Figure 1c.

Original: Compared with the control group with constant LR (blue line), we can see that LR decay (red line) introduces a significant increase in feature robustness on the training set ( 48% → 77% ).
Revised: Compared with the control group with constant LR (blue line), we can see that LR decay (red line) introduces a significant increase in feature robustness on the training set ( 48% → 77% ).

Original: Therefore, indeed there are many non-robust features that become robust on the training set due to LR decay, as hypothesized above.
Revised: Therefore, indeed there are many non-robust features that become robust on the training set due to LR decay, as hypothesized above.

Original: [Figure]

Original: However, we also notice that on the test set, robustness only changes slightly ( 25% → 30% ), and the non-robust features largely remain non-robust.
Revised: However, we also notice that on the test set, robustness only changes slightly ( 25% → 30% ), and the non-robust features largely remain non-robust.

Original: This indicates that the robustness obtained by these non-robust features after decay are not generalizable from the training set D train to the test set D test .
Revised: This indicates that the robustness obtained by these non-robust features after decay are not generalizable from the training set D train to the test set D test .

Original: The discrepancy in feature robustness explains why the rise of training robustness does not consistently  lead to higher test robustness, leading to a large robust generalization gap.
Revised: The discrepancy in feature robustness shows that the rise of training robustness does not consistently [Figure] adversarial examples lead to higher test robustness, leading to a large robust generalization gap.

Original: Nevertheless, we still do not fully understand why test robustness even degrades shortly after LR decay (Figure 1b).
Revised: Nevertheless, we still do not fully understand why test robustness even degrades shortly after LR decay (Figure 1b).

Original: Below, we show that it is also induced by fitting these non-robust features with smaller LR.
Revised: Below, we show that it is also induced by fitting these non-robust features with smaller LR.

Original: 2. R OBUST O VERFITTING FROM THE L ENS OF D YNAMIC F EATURE R OBUSTNESS
Revised: 2. R OBUST O VERFITTING FROM THE L ENS OF D YNAMIC F EATURE R OBUSTNESS

Original: In this section, we study how the change of feature robustness during LR decay induces the robust overfitting phenomenon.
Revised: In this section, we study how the change of feature robustness during LR decay induces the robust overfitting phenomenon.

Original: As discussed above, a significant change during LR decay is that many non-robust features become robust on the training set, while remaining non-robust on the test set.
Revised: As discussed above, a significant change during LR decay is that many non-robust features become robust on the training set, while remaining non-robust on the test set.

Original: To begin with, we design a controlled experiment showing that fitting these non-robust features on the training set is strongly correlated with the degradation of robustness on the test set.
Revised: To begin with, we design a controlled experiment showing that fitting these non-robust features on the training set is strongly correlated with the degradation of robustness on the test set.

Original: Memorizing More Non-robust Features Brings More Harm to Test Robustness.
Revised: Memorizing More Non-robust Features Brings More Harm to Test Robustness.

Original: To study the effect of non-robust features alone, we deliberately add non-robust features (generated with different perturbation budgets ε ) to the clean CIFAR-10 dataset, where a larger ε indicates stronger non-robust features in the synthetic dataset (details in Appendix B.2).
Revised: To study the effect of non-robust features alone, we deliberately add non-robust features (generated with different perturbation budgets ε ) to the clean CIFAR-10 dataset, where a larger ε indicates stronger non-robust features in the synthetic dataset (details in Appendix B.3).

Original: As shown in Figure 1d, with LR decays, stronger non-robust features induce severe test robustness degradation, while it is much less severe without LR decay.
Revised: As shown in Figure 1d, with LR decays, stronger non-robust features induce severe test robustness degradation, while it is much less severe without LR decay.

Original: This shows that the memorization of non-robust features induced by LR decays indeed hurts test robustness a lot.
Revised: This shows that the memorization of non-robust features induced by LR decays indeed hurts test robustness a lot.

Original: But how does this robust overfitting phenomenon happen?
Revised: But how does this robust overfitting phenomenon happen?

Original: A Hypothetical Explanation for Robust Overfitting.
Revised: A Hypothetical Explanation for Robust Overfitting.

Original: From the perspective of dynamic feature robustness, the essential role of LR decay is to break the original balance between the attacker A and the trainer T .
Revised: From the perspective of dynamic feature robustness, the essential role of LR decay is to break the original balance between the attacker A and the trainer T .

Original: The decay tips the scale towards the trainer who seizes more and more adversarial examples (containing non-robust features) and push up the training robustness.
Revised: The decay tips the scale towards the trainer who seizes more and more adversarial examples (containing non-robust features) and push up the training robustness.

Original: However, it is not at no cost.
Revised: However, it is not at no cost.

Original: In the analysis below, we show that it also sets a trap for itself in the test-time attack.
Revised: In the analysis below, we show that it also sets a trap for itself in the test-time attack.

Original: 1) Model Learns False Mapping of Non-robust Training Features after Decay.
Revised: 1) Model Learns False Mapping of Non-robust Training Features after Decay.

Original: Consider an adversarial example x from class A that is misclassified to class B ̸
Revised: Consider an adversarial example x from class A that is misclassified to class B ̸

Original: =
Revised: =

Original: A before decay.
Revised: A before decay.

Original: According to
Revised: According to

Original: Ilyas et al.
Revised: Ilyas et al.

Original: (2019), the non-robust feature g of x belongs to class B .
Revised: (2019), the non-robust feature g of x belongs to class B .

Original: After LR decay, the trainer T becomes more capable of memorization with a smaller LR, and we assume that it successfully learns to map this adversarial example x to its correct label A .
Revised: After LR decay, the trainer T becomes more capable of memorization with a smaller LR, and we assume that it successfully learns to map this adversarial example x to its correct label A .

Original: Accordingly, the non-robust feature g from class B in x is also (falsely) mapped to class A . 2)
Revised: Accordingly, the non-robust feature g from class B in x is also (falsely) mapped to class A . 2)

Original: False Non-robust Mapping Opens Shortcuts for Test-time Adversarial Attack.
Revised: False Non-robust Mapping Opens Shortcuts for Test-time Adversarial Attack.

Original: Considering a test sample ˆ x from class B , before LR decay, the attacker A has to add non-robust features from a different class , say A , in order to misclassify it.
Revised: Considering a test sample ˆ x from class B , before LR decay, the attacker A has to add non-robust features from a different class , say A , in order to misclassify it.

Original: However, after decay, because of the existence of false non-robust mapping, the attacker A can simply add a non-robust feature from the same class , denoted as g , such that it is misclassified to a wrong class A .
Revised: However, after decay, because of the existence of false non-robust mapping, the attacker A can simply add a non-robust feature from the same class , denoted as g , such that it is misclassified to a wrong class A .

Original: In other words, the injected non-robust features do not even have to come from a different class, which makes it much easier for the attacker to find g when starting from ˆ x (compared to non-robust features from other classes).
Revised: In other words, the injected non-robust features do not even have to come from a different class, which makes it much easier for the attacker to find g when starting from ˆ x (compared to non-robust features from other classes).

Original: Therefore, the attacker will have a larger attack success rate with the help of these false non-robust mappings, as they create easy-to-find shortcuts for adversarial attack.
Revised: Therefore, the attacker will have a larger attack success rate with the help of these false non-robust mappings, as they create easy-to-find shortcuts for adversarial attack.

Original: Summary: when the balance breaks, the trainer has an (overly) strong fitting ability such that it learns false mapping of non-robust features.
Revised: Summary: when the balance breaks, the trainer has an (overly) strong fitting ability such that it learns false mapping of non-robust features.

Original: At test time, these non-robust features induce shortcuts for adversarial attack, which leads to robustness degradation.
Revised: At test time, these non-robust features induce shortcuts for adversarial attack, which leads to robustness degradation.

Revised: From a causal perspective, these non-robust features are spurious features (Zhang et al., 2022; Sagawa et al., 2020).

Revised: Therefore, when they are falsely memorized, it would also induce significant performance degradation.

Original: Empirical Verification.
Revised: Empirical Verification.

Original: On real-world data like images, it is hard to know what the data distribution or the non-robust features actually are.
Revised: On real-world data like images, it is hard to know what the data distribution or the non-robust features actually are.

Original: Instead, we validate this theory by verifying some nontrivial yet testable implications of it.
Revised: Instead, we validate this theory by verifying some nontrivial yet testable implications of it.

Original: We refer the readers to Appendix B.2 for details of these experiments. 
Revised: We refer the readers to Appendix B.3 for details of these experiments. [Figure]

Original: 1) Lack of Target-class Features in Adversarial Examples.
Revised: 1) Lack of Target-class Features in Adversarial Examples.

Original: In our example above, the added nonrobust feature g comes from the same class B as ˆ x , so the resulting adversarial example ˆ x ′ actually has little features of class A , even if it is (mis)classified to class A .
Revised: In our example above, the added nonrobust feature g comes from the same class B as ˆ x , so the resulting adversarial example ˆ x ′ actually has little features of class A , even if it is (mis)classified to class A .

Original: Figure 2a shows that adversarial examples generated by models after LR decay indeed have less and less target-class features after the decay.
Revised: Figure 2a shows that adversarial examples generated by models after LR decay indeed have less and less target-class features after the decay.

Original: A Strong Bilateral Class Correlation.
Revised: A Strong Bilateral Class Correlation.

Original: Our theory suggests that if overfitting happens, the training misclassification from class A → B (before decay) will induce an increase in test misclassification from class B → A (after decay) as A → B false non-robust mappings create
Revised: Our theory suggests that if overfitting happens, the training misclassification from class A → B (before decay) will induce an increase in test misclassification from class B → A (after decay) as A → B false non-robust mappings create

Original: B → A shortcuts.
Revised: B → A shortcuts.

Original: We verify this in Figure 2b, where we observe that this bilateral correlation indeed consistently increases after LR decay, while remaining very low (nearly no correlation) without decay.
Revised: We verify this in Figure 2b, where we observe that this bilateral correlation indeed consistently increases after LR decay, while remaining very low (nearly no correlation) without decay.

Original: 3) Symmetrized Confusion Matrix.
Revised: 3) Symmetrized Confusion Matrix.

Original: As a result of the bilateral correlation, interestingly, we can deduce that the A → B and B → A misclassification will have more similar error rates, which means that the confusion matrix of test robustness will become more symmetric.
Revised: As a result of the bilateral correlation, interestingly, we can deduce that the A → B and B → A misclassification will have more similar error rates, which means that the confusion matrix of test robustness will become more symmetric.

Original: Indeed, as shown in Figure 2c, the confusion matrix (denoted as C ) becomes much more symmetric (smaller distance between A and A ⊤ ) after LR decay, particularly when compared to constant learning rate.
Revised: Indeed, as shown in Figure 2c, the confusion matrix (denoted as C ) becomes much more symmetric (smaller distance between A and A ⊤ ) after LR decay, particularly when compared to constant learning rate.

Original: The three experiments above help justify our understanding of robust overfitting from the perspective of dynamic feature robustness.
Revised: The three experiments above help justify our understanding of robust overfitting from the perspective of dynamic feature robustness.

Original: In the next section, we further show that this theory also sheds new lights on the design of adversarial training algorithms for alleviating robust overfitting.
Revised: In the next section, we further show that this theory also sheds new lights on the design of adversarial training algorithms for alleviating robust overfitting.

Original:  M ITIGATING R OBUST O VERFITTING FROM A D YNAMIC P ERSPECTIVE
Revised:  M ITIGATING R OBUST O VERFITTING FROM A D YNAMIC P ERSPECTIVE

Original: In Section 2, we have introduced our dynamic feature robustness and explain robust overfitting from the shortcut effects of memorizing non-robust features.
Revised: In Section 2, we have introduced our dynamic feature robustness and explain robust overfitting from the shortcut effects of memorizing non-robust features.

Original: In this section, we further investigate how to alleviate robust overfitting.
Revised: In this section, we further investigate how to alleviate robust overfitting.

Original: From the perspective of our theory, the main goal is to prevent the model trainer T from fitting non-robust features too quickly and too adequately.
Revised: From the perspective of our theory, the main goal is to prevent the model trainer T from fitting non-robust features too quickly and too adequately.

Original: As AT is a dynamic game between the trainer T and the attacker A , we can either weaken the trainer’s fitting ability (Sections 3.1 & 3.2), or strengthen the attacker’s attacking ability (Section 3.3), as we show below.
Revised: As AT is a dynamic game between the trainer T and the attacker A , we can either weaken the trainer’s fitting ability (Sections 3.1 & 3.2), or strengthen the attacker’s attacking ability (Section 3.3), as we show below.

Original: 3. M ODEL R EGULARIZATION WITH B OOTSTRAPPED F LATNESS
Revised: 3. M ODEL R EGULARIZATION WITH B OOTSTRAPPED F LATNESS

Original: In order to suppress robust overfitting of the model trainer T after LR decays, we need to put further regularization on its local fitting ability, e.g., by enforcing better landscape flatness.
Revised: In order to suppress robust overfitting of the model trainer T after LR decays, we need to put further regularization on its local fitting ability, e.g., by enforcing better landscape flatness.

Original: In this way, we could prevent the learner from drawing complex decision boundaries and overfit to non-robust features.
Revised: In this way, we could prevent the learner from drawing complex decision boundaries and overfit to non-robust features.

Original: Indeed, Stutz et al.
Revised: Indeed, Stutz et al.

Original: (2021) find that models with less robust overfitting often enjoy better landscape flatness.
Revised: (2021) find that models with less robust overfitting often enjoy better landscape flatness.

Original: Among existing works, there are mainly two approaches for better flatness.
Revised: Among existing works, there are mainly two approaches for better flatness.

Original: One is
Revised: One is

Original: AWP (Wu et al., 2020) by adversarially perturbing weights, and one is weight average (WA) (Izmailov et al., 2018) with equal or moving average of model parameters.
Revised: AWP (Wu et al., 2020) by adversarially perturbing weights, and one is weight average (WA) (Izmailov et al., 2018) with equal or moving average of model parameters.

Original: Compared with AWP that introduces extra training steps and computation cost, WA is rather simple and efficient, as model average brings neglectable training overhead.
Revised: Compared with AWP that introduces extra training steps and computation cost, WA is rather simple and efficient, as model average brings neglectable training overhead.

Original: Therefore, we focus on WA in this work.
Revised: Therefore, we focus on WA in this work.

Original: Along the online model f θ that is adversarially trained, we also maintain a WA model f φ that is an exponential moving average of the online parameters φ ← γ · φ + (1 − γ ) · θ , where γ ∈ [0 , 1] is the decay rate.
Revised: Along the online model f θ that is adversarially trained, we also maintain a WA model f φ that is an exponential moving average of the online parameters φ ← γ · φ + (1 − γ ) · θ , where γ ∈ [0 , 1] is the decay rate.

Original: Although WA can produce a flatter landscape (Izmailov et al., 2018), we notice that it is not enough to prevent robust overfitting.
Revised: Although WA can produce a flatter landscape (Izmailov et al., 2018), we notice that it is not enough to prevent robust overfitting.

Original: As shown in Figure 3a, the WA model still overfits shortly after the online model overfits.
Revised: As shown in Figure 3a, the WA model still overfits shortly after the online model overfits.

Original: Rebuffi et al.
Revised: Rebuffi et al.

Original: (2021) further combine WA with CutMix (Yun et al., 2019), but it still cannot alleviate robust overfitting under longer training (see Figure 4a).
Revised: (2021) further combine WA with CutMix (Yun et al., 2019), but it still cannot alleviate robust overfitting under longer training (see Figure 4a).

Original: Here, we consider an alternative solution with a new regularization loss.
Revised: Here, we consider an alternative solution with a new regularization loss.

Original: We notice that when the online model f θ deteriorates, the WA model, although flatter, will eventually overfit as it is only a moving average of f θ .
Revised: We notice that when the online model f θ deteriorates, the WA model, although flatter, will eventually overfit as it is only a moving average of f θ .

Original: Therefore, we also need to improve the flatness of the online model as well.
Revised: Therefore, we also need to improve the flatness of the online model as well.

Original: How to achieve this?
Revised: How to achieve this?

Original: Inspired by the latent bootstrap mechanism in self-supervised learning (Grill et al., 2020), we realize that the WA model f φ is a flatter version of the online model f θ itself, and thus we can bootstrap its own flatness by aligning the online model with the WA model (as a target).
Revised: Inspired by the latent bootstrap mechanism in self-supervised learning (Grill et al., 2020), we realize that the WA model f φ is a flatter version of the online model f θ itself, and thus we can bootstrap its own flatness by aligning the online model with the WA model (as a target).

Original: Specifically, given an adversarial example x generated by PGD attack as in AT, we devise a new regularization term that aligns their predictions using KL divergence,
Revised: Specifically, given an adversarial example x generated by PGD attack as in AT, we devise a new regularization term that aligns their predictions using KL divergence,

Original: [Equation]
Revised: [Equation]

Original: where λ is a coefficient balancing the CE loss (for better training robustness) and the regularization term (for better landscape flatness).
Revised: where λ is a coefficient balancing the CE loss (for better training robustness) and the regularization term (for better landscape flatness).

Original: We name it as Bootstrapped Adversarial Training (BoAT) loss.
Revised: We name it as Bootstrapped Adversarial Training (BoAT) loss.

Original: [Figure]

Original: Because this loss is designed to alleviate robust overfitting after decay, we only apply WA and BoAT loss after decay to save computation.
Revised: Because this loss is designed to alleviate robust overfitting after decay, we only apply WA and BoAT loss after decay to save computation.

Original: We also detach the gradients from the WA model as it only serves a flatter target.
Revised: We also detach the gradients from the WA model as it only serves a flatter target.

Original: Compared with CE loss, the KL regularization provides more fine-grained supervision from a flatter model that helps avoid overfitting to training labels.
Revised: Compared with CE loss, the KL regularization provides more fine-grained supervision from a flatter model that helps avoid overfitting to training labels.

Original: As a result, the online model trained with the BoAT loss will have better flatness than the vanilla AT loss.
Revised: As a result, the online model trained with the BoAT loss will have better flatness than the vanilla AT loss.

Original: A flatter online model will further improve the flatness of the WA model, which, in return, also benefits the flatness of the online model.
Revised: A flatter online model will further improve the flatness of the WA model, which, in return, also benefits the flatness of the online model.

Original: In this way, we could improve the flatness of both online and WA models in a bootstrapped fashion, which, consequently, alleviates the robust overfitting problems of both models.
Revised: In this way, we could improve the flatness of both online and WA models in a bootstrapped fashion, which, consequently, alleviates the robust overfitting problems of both models.

Original: To verify the intuition above, we compare the AT loss and the BoAT loss under the same training configurations.
Revised: To verify the intuition above, we compare the AT loss and the BoAT loss under the same training configurations.

Original: From Figure 3a, we can see that contrary to vanilla AT that quickly degrades after decay, the online and WA robustness of the BoAT loss continues to gradually increase after decay.
Revised: From Figure 3a, we can see that contrary to vanilla AT that quickly degrades after decay, the online and WA robustness of the BoAT loss continues to gradually increase after decay.

Original: In particular, with AT loss, a typical behavior is that the online model will dramatically jump to high robustness after decay with a stronger fitting ability, but because it overfits to non-robust features, its robustness also dramatically degrades shortly afterwards.
Revised: In particular, with AT loss, a typical behavior is that the online model will dramatically jump to high robustness after decay with a stronger fitting ability, but because it overfits to non-robust features, its robustness also dramatically degrades shortly afterwards.

Original: Instead, with the bootstrap regularization,
Revised: Instead, with the bootstrap regularization,

Original: BoAT’s online model has a slow but steady improvement of robustness, which also contributes to better robustness of the WA model.
Revised: BoAT’s online model has a slow but steady improvement of robustness, which also contributes to better robustness of the WA model.

Original: At last, BoAT’s WA model achieves better robustness than
Revised: At last, BoAT’s WA model achieves better robustness than

Original: AT’s best WA model, showing that alleviating robust overfitting is indeed helpful for attaining better robustness.
Revised: AT’s best WA model, showing that alleviating robust overfitting is indeed helpful for attaining better robustness.

Original: When further comparing their flatness at the last checkpoints (Figure 3b), we notice that both BoAT’s online and WA models enjoy better flatness than AT’s online and WA models, showing that our bootstrap mechanism effectively improves the flatness of both online and WA models.
Revised: When further comparing their flatness at the last checkpoints (Figure 3b), we notice that both BoAT’s online and WA models enjoy better flatness than AT’s online and WA models, showing that our bootstrap mechanism effectively improves the flatness of both online and WA models.

Original: 3. S MALLER L EARNING R ATE D ECAY
Revised: 3. S MALLER L EARNING R ATE D ECAY

Original: The discussion above also suggests that a large LR decay can be harmful for robustness, as it induces severer robust overfitting.
Revised: The discussion above also suggests that a large LR decay can be harmful for robustness, as it induces severer robust overfitting.

Original: A canonical recipe of AT (Madry et al., 2018) is to adopt a piecewise linear
Revised: A canonical recipe of AT (Madry et al., 2018) is to adopt a piecewise linear

Original: LR schedule with a large decay ( e.g., LR divided by d = 10 ) at a certain epoch.
Revised: LR schedule with a large decay ( e.g., LR divided by d = 10 ) at a certain epoch.

Original: As shown in Figure 3a, the online robustness quickly jumps up by nearly 10%, and Rice et al.
Revised: As shown in Figure 3a, the online robustness quickly jumps up by nearly 10%, and Rice et al.

Original: (2020) propose to early stop the training before it deteriorates.
Revised: (2020) propose to early stop the training before it deteriorates.

Original: However, our discussion on the BoAT loss shows that a longer training without overfitting might bring further improvement over this jump-and-early-stop strategy.
Revised: However, our discussion on the BoAT loss shows that a longer training without overfitting might bring further improvement over this jump-and-early-stop strategy.

Original: Apart from the regularization loss (Section 3.1), here we investigate the effect of different decay factors d on robust overfitting.
Revised: Apart from the regularization loss (Section 3.1), here we investigate the effect of different decay factors d on robust overfitting.

Original: In our dynamic robustness framework, the LR decay plays an important role, as it breaks the balance between the trainer and the attacker by endowing the trainer with stronger local fitting ability (Section 2.2).
Revised: In our dynamic robustness framework, the LR decay plays an important role, as it breaks the balance between the trainer and the attacker by endowing the trainer with stronger local fitting ability (Section 2.2).

Original: However, we also find when it is relatively too strong, the trainer could memorize many non-robust features that are poisonous for its test robustness (Section 2.3).
Revised: However, we also find when it is relatively too strong, the trainer could memorize many non-robust features that are poisonous for its test robustness (Section 2.3).

Original: Therefore, a smaller decay rate with milder change of relative strength could help prevent robust overfitting.
Revised: Therefore, a smaller decay rate with milder change of relative strength could help prevent robust overfitting.

Original: To verify this, we compare six decay factors d : 1 , 2 , 3 , 4 , 6 , 10 , where d = 10 corresponds to canonical decay and d = 1 corresponds to constant LR (no decay).
Revised: To verify this, we compare six decay factors d : 1 , 2 , 3 , 4 , 6 , 10 , where d = 10 corresponds to canonical decay and d = 1 corresponds to constant LR (no decay).

Original: As shown in Figure 3c, the online model using smaller decay indeed has less robust overfitting, though their best robustness is also lower than d = 10 .
Revised: As shown in Figure 3c, the online model using smaller decay indeed has less robust overfitting, though their best robustness is also lower than d = 10 .

Original: However, interestingly, when combined with WA, the flatter WA model can benefit a lot from smaller decay and improve both best-epoch and final-epoch robustness.
Revised: However, interestingly, when combined with WA, the flatter WA model can benefit a lot from smaller decay and improve both best-epoch and final-epoch robustness.

Original: Specifically,
Revised: Specifically,

Original: WA with a very small decay rate d = 2 attains better best-epoch robustness than d = 10 with only slight overfitting.
Revised: WA with a very small decay rate d = 2 attains better best-epoch robustness than d = 10 with only slight overfitting.

Original: Intuitively, because the WA model is a moving average of many online models, for better robustness and flatness, it would prefer a group of similar and growing (at least less overfitting) online models (under smaller decay) instead of a group of quickly overfitting models (under large decay).
Revised: Intuitively, because the WA model is a moving average of many online models, for better robustness and flatness, it would prefer a group of similar and growing (at least less overfitting) online models (under smaller decay) instead of a group of quickly overfitting models (under large decay).

Original: This investigation shows that with flatter landscape like WA, a smaller decay is indeed beneficial for both attaining better robustness and alleviating robust overfitting.
Revised: This investigation shows that with flatter landscape like WA, a smaller decay is indeed beneficial for both attaining better robustness and alleviating robust overfitting.

Revised: 3. C OUNTERING S TRONGER M ODEL T RAINER WITH S TRONGER T RAINING A TTACKER

Revised: In the discussions above, we have shown that regularizing the trainer’s local fitting ability with the BoAT loss or smaller learning rate decay can both help prevent it from overfitting to non-

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: perturbation norm ε ∞ = 8 / 255 based on the PreActResNet-18 architecture.

Original: [Table]
Revised: [Table]

Original: PGD{+PGD-AT 55.52 57.35 29.58 23.02 24.53 20.21 45.54 48.73 22.00 18.08 17.30 14. TRADES 55.
+}
Revised: {+PGD-AT 55.52 57.35 29.58 23.02 24.53 20.21 45.54 48.73 22.00 18.08 17.30 14. TRADES 55.
+}

Original: [Table] [Table] 
Revised: [Table] [Table] [Table]

Original: best final best final best final best final best final best final
Revised: [Table] [Table] [Table] [Table]

Original: PGD-AT 55.52 57.35 29.58 23.02 24.53 20.21 45.54 48.73 22.00 18.08 17.30 14.
Revised: [Table]

Original: WA 57.96 58.81 30.90 24.04 25.95 21.02 48.76 50.00 24.66 19.82 19.76 15.
Revised: WA 57.96 58.81 30.90 24.04 25.95 21.02 48.76 50.00 24.66 19.82 19.76 15.

Original: KD+SWA 57.23 57.66 30.06 30.02 26.04 25.99 50.59 50.83 24.83 24.79 19.78 19.
Revised: KD+SWA 57.23 57.66 30.06 30.02 26.04 25.99 50.59 50.83 24.83 24.79 19.78 19.

Original: PGD-AT+TE 56.52 57.30 31.23 29.25 26.04 25.13 46.84 50.51 22.89 19.40 18.16 15.
Revised: PGD-AT+TE 56.52 57.30 31.23 29.25 26.04 25.13 46.84 50.51 22.89 19.40 18.16 15.

Original: AWP 53.92 54.81 30.67 30.43 25.26 25.07 43.31 43.30 23.60 23.26 18.08 17.{+43.
+}
Revised: AWP 53.92 54.81 30.67 30.43 25.26 25.07 {+43.
+}

Revised: [Table]

Original: BoAT 56.44 56.51 32.70 32.38 27.59 27.52 47.79 48.20 25.21 24.52 20.41 19.
Revised: BoAT 56.44 56.51 32.70 32.38 27.59 27.52 47.79 48.20 25.21 24.52 20.41 19.

Original: WA+CutMix 56.84 56.96 32.35 32.34 26.39 26.35 46.58 46.87 24.97 24.87 18.97 18.
Revised: WA+CutMix 56.84 56.96 32.35 32.34 26.39 26.35 46.58 46.87 24.97 24.87 18.97 18.

Original: BoAT+CutMix 55.74 55.88 33.03 33.02 27.21 27.20 46.02 46.15 25.04 25.02 19.51 19.
Revised: BoAT+CutMix 55.74 55.88 33.03 33.02 27.21 27.20 46.02 46.15 25.04 25.02 19.51 19.

Original: In the discussions above, we have shown that regularizing the trainer’s local fitting ability with the BoAT loss or smaller learning rate decay can both help prevent it from overfitting to nonrobust features.
Revised: robust features.

Original: As noted in our dynamic framework, AT is a dynamic game between the trainer
Revised: As noted in our dynamic framework, AT is a dynamic game between the trainer

Original: T and the attacker A .
Revised: T and the attacker A .

Original: Another natural approach is to strengthen the attacker’s attacking ability to counter the stronger trainer after decay, which would also help strike a new balance between two players.
Revised: Another natural approach is to strengthen the attacker’s attacking ability to counter the stronger trainer after decay, which would also help strike a new balance between two players.

Original: To examine this, we apply four different training perturbation budgets after decay, ε = 8 / 255 , 10 / 255 , 12 / 255 , 16 / 255 (with fixed step size α = 2 / 255 and increasing PGD steps k = 10 · ε/ (8 / 255) ), where ε = 8 / 255 is the default budget used before decay and in test-time attack.
Revised: To examine this, we apply four different training perturbation budgets after decay, ε = 8 / 255 , 10 / 255 , 12 / 255 , 16 / 255 (with fixed step size α = 2 / 255 and increasing PGD steps k = 10 · ε/ (8 / 255) ), where ε = 8 / 255 is the default budget used before decay and in test-time attack.

Original: As can be seen from Figure 3d, models trained with stronger attacker (larger ε ) indeed has much less overfitting, which helps verify our understanding of robust overfitting through the dynamic game perspective.
Revised: As can be seen from Figure 3d, models trained with stronger attacker (larger ε ) indeed has much less overfitting, which helps verify our understanding of robust overfitting through the dynamic game perspective.

Original: Besides, a too strong training attacker also prevents the trainer from learning useful robust features (under ε = 8 / 255 ).
Revised: Besides, a too strong training attacker also prevents the trainer from learning useful robust features (under ε = 16 / 255 ).

Original: Empirically, we observe that selecting a moderately larger attack strength, ε = 12 / 255 , attains the optimal best-epoch robustness (see also Appendix B.3).
Revised: Empirically, we observe that selecting a moderately larger attack strength, ε = 12 / 255 , attains the optimal best-epoch robustness (see also Appendix B.4).

Original: This shows that we need to re-design the training attacker according to the change of the model trainer.
Revised: This shows that we need to re-design the training attacker according to the change of the model trainer.

Original: 3. O VERALL A PPROACH
Revised: 3. O VERALL A PPROACH

Original: Above, we have introduced three effective techniques to mitigate robust overfitting from different perspectives: model regularization, learning rate decay, and attacker strength.
Revised: Above, we have introduced three effective techniques to mitigate robust overfitting from different perspectives: model regularization, learning rate decay, and attacker strength.

Original: At last, we combine the BoAT loss and smaller learning rate as our final approach.
Revised: At last, we combine the BoAT loss and smaller learning rate as our final approach.

Original: We do not include stronger attacker because1) the former two methods combined are already enough to almost alleviate robust overfitting, and 2) stronger attacker often induces a large decrease in natural  accuracy, although it could further bring a slight increase of robustness.
Revised: We do not include stronger attacker because stronger attacker often induces a large decrease in natural accuracy. This is because stronger attacks will make all features less robust, which will decrease the number of usable features in the classifier and hurt natural accuracy, although it could further bring a slight increase of robustness.

Original: We include a study on this trade-off in Appendix B.3.
Revised: We include a study on this trade-off in Appendix B.4.

Original:  E XPERIMENTS ON B ENCHMARK D ATASETS
Revised:  E XPERIMENTS ON B ENCHMARK D ATASETS

Original: In this section, we evaluate the effectiveness of the purposed BoAT on several benchmark datasets using different network architectures.
Revised: In this section, we evaluate the effectiveness of the proposed BoAT on several benchmark datasets using different network architectures.

Original: [Figure]
Revised: [Figure]

Original: Setup.
Revised: Setup.

Original: We consider the classification tasks on CIFAR-10, CIFAR-100 (Krizhevsky et al., 2009), and
Revised: We consider the classification tasks on CIFAR-10, CIFAR-100 (Krizhevsky et al., 2009), and

Original: Tiny-ImageNet (Deng et al., 2009) with the PreActResNet-18 (He et al., 2016) and WideResNet34-10 (Zagoruyko & Komodakis, 2016) architectures.
Revised: Tiny-ImageNet (Deng et al., 2009) with the PreActResNet-18 (He et al., 2016) and WideResNet34-10 (Zagoruyko & Komodakis, 2016) architectures.

Original: Besides the vanilla PGD-AT (Madry et al., 2018), we include the following baseline methods for alleviating robust overfitting: WA (Izmailov et al., 2018), KD+SWA (Chen et al., 2021), PGD-AT+TE (Dong et al., 2022), AWP (Wu et al., 2020) and WA+CutMix (Rebuffi et al., 2021).
Revised: Besides the vanilla PGD-AT (Madry et al., 2018), we include the following baseline methods for alleviating robust overfitting: WA (Izmailov et al., 2018), KD+SWA (Chen et al., 2021), PGD-AT+TE (Dong et al., 2022), AWP (Wu et al., 2020) and WA+CutMix (Rebuffi et al., 2021).

Original: During evaluation, we use PGD-20 (Madry et al., 2018) and
Revised: During evaluation, we use PGD-20 (Madry et al., 2018) and

Original: AutoAttack (AA) (Croce & Hein, 2020) for the adversarial attack methods.
Revised: AutoAttack (AA) (Croce & Hein, 2020) for the adversarial attack methods.

Original: We train each model for 200 epochs and record the best-epoch (selected based on PGD attack) and final-epoch robust test accuracy.
Revised: We train each model for 200 epochs and record the best-epoch (selected based on PGD attack) and final-epoch robust test accuracy.

Original: Please refer to Appendix B.4 forfurther details and Appendix C for additional results.
Revised: Please refer to Appendix B.5 and B.6 for details and Appendix C for additional results.

Original: 4. R ESULTS ON B ENCHMARK D ATASETS
Revised: 4. R ESULTS ON B ENCHMARK D ATASETS

Original: Performance across Different Networks.
Revised: Performance across Different Networks.

Original: In Table 1, we evaluate BoAT against previous AT variants on two popular backbone networks: PreActResNet-18 and WideResNet-34-10.
Revised: In Table 1, we evaluate BoAT against previous AT variants on two popular backbone networks: PreActResNet-18 and WideResNet-34-10.

Original: We can see that without strong data augmentations like CutMix, BoAT attains state-of-the-art robustness on both models and outperforms previous methods by at least 1% against AutoAttack (AA), arguably the strongest white-box attacker.
Revised: We can see that without strong data augmentations like CutMix, BoAT attains state-of-the-art robustness on both models and outperforms previous methods by at least 1% against AutoAttack (AA), arguably the strongest white-box attacker.

Original: Meanwhile, BoAT almost alleviates robust overfitting on PreActResNet18 with only 0.03% gap (51.39% best and 51.36% final), and the gap is also small (0.30%) on
Revised: Meanwhile, BoAT almost alleviates robust overfitting on PreActResNet18 with only 0.03% gap (51.39% best and 51.36% final), and the gap is also small (0.30%) on

Original: WideResNet.
Revised: WideResNet.

Original: Because of this, remarkably, the final-epoch robustness of BoAT even consistently outperforms the best-epoch robustness of previous approaches using early stopping.
Revised: Because of this, remarkably, the final-epoch robustness of BoAT even consistently outperforms the best-epoch robustness of previous approaches using early stopping.

Original: Therefore, with
Revised: Therefore, with

Original: BoAT, we might not need per-epoch validation and early stopping techniques any more, which helps save training computation cost, simplify the training recipe, and improve the applicability of AT.
Revised: BoAT, we might not need per-epoch validation and early stopping techniques any more, which helps save training computation cost, simplify the training recipe, and improve the applicability of AT.

Original: Strong Augmentations.
Revised: Strong Augmentations.

Original: WideResNet is more prone to overfit as it has larger capacity, and existing
Revised: WideResNet is more prone to overfit as it has larger capacity, and existing

Original: WA+CutMix strategy (Rebuffi et al., 2021) still overfits by more than 1% (55.16% best and 54.10% last).
Revised: WA+CutMix strategy (Rebuffi et al., 2021) still overfits by more than 1% (55.16% best and 54.10% last).

Original: In comparison, BoAT+Cutmix achieves 56.13% best robustness (chosen according to PGD) and 56.23% final robustness, showing that this combination indeed alleviates robust overfitting on
Revised: In comparison, BoAT+Cutmix achieves 56.13% best robustness (chosen according to PGD) and 56.23% final robustness, showing that this combination indeed alleviates robust overfitting on

Original: WideResNet and attain state-of-the-art robustness.
Revised: WideResNet and attain state-of-the-art robustness.

Original: In comparison, CutMix brings little improvement on PreActResNet-18, mainly because this model has lower capacity and still underfits under strong augmentations, and it requires longer training to reach higher performance.
Revised: In comparison, CutMix brings little improvement on PreActResNet-18, mainly because this model has lower capacity and still underfits under strong augmentations, and it requires longer training to reach higher performance.

Original: Even though,
Revised: Even though,

Original: BoAT+CutMix still outperforms WA+CutMix by 0.5% AA robustness by the end of 200 epochs.
Revised: BoAT+CutMix still outperforms WA+CutMix by 0.5% AA robustness by the end of 200 epochs.

Original: Performance across Different Datasets.
Revised: Performance across Different Datasets.

Original: We also conduct experiments on two additional benchmark datasets, CIFAR-100 and Tiny-ImageNet, that are more complex than CIFAR-10.
Revised: We also conduct experiments on two additional benchmark datasets, CIFAR-100 and Tiny-ImageNet, that are more complex than CIFAR-10.

Original: As shown in Table 2, our BoAT and BoAT+Cutmix still achieves the highest best and last robustness, demonstrating its scalability to larger scale datasets.
Revised: As shown in Table 2, our BoAT and BoAT+Cutmix still achieve the highest best and last robustness, demonstrating its scalability to larger scale datasets.

Original: Similarly, CutMix does not help much as it often leads to underfitting with stronger regularization effects within a relatively short training length.
Revised: Similarly, CutMix does not help much as it often leads to underfitting with stronger regularization effects within a relatively short training length.

Original: 4. E MPIRICAL U NDERSTANDINGS
Revised: 4. E MPIRICAL U NDERSTANDINGS

Original: BoAT Can Benefit from Longer Training.
Revised: BoAT Can Benefit from Longer Training.

Original: Table 1 suggests that when CutMix is applied to a
Revised: Table 1 suggests that when CutMix is applied to a

Original: PreActResNet-18 model, it demands longer training to fully unleash the beast.
Revised: PreActResNet-18 model, it demands longer training to fully unleash the beast.

Original: This motivates us to examine the effect of longer training, e.g., 500 epochs.
Revised: This motivates us to examine the effect of longer training, e.g., 500 epochs.

Original: As shown in Figure 4a, longer training with
Revised: As shown in Figure 4a, longer training with

Original: WA+CutMix brings little improvement on final robustness (49.87% (200 epochs) v.s. 49.90% ( epochs) under AA).
Revised: WA+CutMix brings little improvement on final robustness (49.87% (200 epochs) v.s. 49.90% ( epochs) under AA).

Original: Instead, BoAT+CutMix achieves higher best robustness and the robustness keeps improving along training ( > 1% improvement under AA).
Revised: Instead, BoAT+CutMix achieves higher best robustness and the robustness keeps improving along training ( > 1% improvement under AA).

Original: In particular, the additional 300 epochs improve the final AA robustness of BoAT+CutMix from 50.39% to 51.28%.
Revised: In particular, the additional 300 epochs improve the final AA robustness of BoAT+CutMix from 50.39% to 51.28%.

Original: This provides strong evidence on the effectiveness of BoAT against robust overfitting.
Revised: This provides strong evidence on the effectiveness of BoAT against robust overfitting.

Original: It also shows that combining BoAT with strong data augmentations can benefit from long training as in standard training.
Revised: It also shows that combining BoAT with strong data augmentations can benefit from long training as in standard training.

Original: Verification on Robust Overfitting Experiments.
Revised: Verification on Robust Overfitting Experiments.

Original: Beyond its success, we empirically study how
Revised: Beyond its success, we empirically study how

Original: BoAT manages to suppress robust overfitting through a lens of feature robustness.
Revised: BoAT manages to suppress robust overfitting through a lens of feature robustness.

Original: To this end, following the experiments in Section 2.3, we compare PGD-AT and BoAT in terms of 1) the amount of target-class features of test adversarial examples and 2) the strength of bilateral class correlation.
Revised: To this end, following the experiments in Section 2.3, we compare PGD-AT and BoAT in terms of 1) the amount of target-class features of test adversarial examples and 2) the strength of bilateral class correlation.

Original: First, we can see that test adversarial examples still have to contain much target-class features to be misclassified (Figure 4b) and there is no long a strong bilateral class correlation (Figure 4c).
Revised: First, we can see that test adversarial examples still have to contain much target-class features to be misclassified (Figure 4b) and there is no longer a strong bilateral class correlation (Figure 4c).

Original: This confirms that BoAT indeed successfully prevent the model from learning false mapping of non-robust training features which creates shortcuts for test-time attack.
Revised: This confirms that BoAT indeed successfully prevent the model from learning false mapping of non-robust training features which creates shortcuts for test-time attack.

Original:  R EVISITING E XISTING A PPROACHES TO A LLEVIATE R OBUST O VERFITTING
Revised:  R EVISITING E XISTING A PPROACHES TO A LLEVIATE R OBUST O VERFITTING

Original: Remind that our definition of feature robustness (Eq. 3) is composed of three factors: the dataset D , the model trainer T , and attacker A .
Revised: Remind that our definition of feature robustness (Eq. 3) is composed of three factors: the dataset D , the model trainer T , and attacker A .

Original: Based on our understanding of robust overfitting (RO), we can categorize existing AT methods that help alleviate RO into the following three classes.
Revised: Based on our understanding of robust overfitting (RO), we can categorize existing AT methods that help alleviate RO into the following three classes.

Original: Data Regularization.
Revised: Data Regularization.

Original: As RO is induced by non-robust features in training inputs , one way is to corrupt non-robust features with various input transformations.
Revised: As RO is induced by non-robust features in training inputs , one way is to corrupt non-robust features with various input transformations.

Original: As can be seen from Ilyas et al.
Revised: As can be seen from Ilyas et al.

Original: (2019), non-robust features often correspond to local and high-frequency patterns in images.
Revised: (2019), non-robust features often correspond to local and high-frequency patterns in images.

Original: Therefore, stronger augmentations like CutMix will help mitigate RO, as analyzed by Rice et al.
Revised: Therefore, stronger augmentations like CutMix will help mitigate RO, as analyzed by Rice et al.

Original: Recently, Li et al.
Revised: Recently, Li et al.

Original: (2022) show that random or even fixed masks can eliminate the catastrophic overfitting of FGSM-AT.
Revised: (2022) show that random or even fixed masks can eliminate the catastrophic overfitting of FGSM-AT.

Original: As the FGSM attacker is much weaker than PGD, FGSM-AT would be more prone to overfit, and we reckon that the masks help corrupt non-robust features in original data.
Revised: As the FGSM attacker is much weaker than PGD, FGSM-AT would be more prone to overfit, and we reckon that the masks help corrupt non-robust features in original data.

Original: A complementary approach is to correct the supervision of the adversarial examples accordingly to avoid learning false label mapping (Section 2.3), e.g., the soft labels adopted in Dong et al.
Revised: A complementary approach is to correct the supervision of the adversarial examples accordingly to avoid learning false label mapping (Section 2.3), e.g., the soft labels adopted in Dong et al.

Original: (2021), and the distillation-alike methods in KD+SWA (Chen et al., 2021) and
Revised: (2021), and the distillation-alike methods in KD+SWA (Chen et al., 2021) and

Original: TE (Dong et al., 2022).
Revised: TE (Dong et al., 2022).

Original: Training Regularization.
Revised: Training Regularization.

Original: A necessary condition of RO is that the model trainer is stronger than the attacker, which breaks their original balance.
Revised: A necessary condition of RO is that the model trainer is stronger than the attacker, which breaks their original balance.

Original: Therefore, we can prevent RO by regularizing the trainer to be relatively weaker on fitting non-robust features.
Revised: Therefore, we can prevent RO by regularizing the trainer to be relatively weaker on fitting non-robust features.

Original: One approach is to directly promote model flatness, e.g., AWP (Wu et al., 2020), EMA Rebuffi et al.
Revised: One approach is to directly promote model flatness, e.g., AWP (Wu et al., 2020), EMA Rebuffi et al.

Original: (2021), SWA Chen et al.
Revised: (2021), SWA Chen et al.

Original: (2021), and our BoAT loss (Section 3.1). 
Revised: (2021), and our BoAT loss (Section 3.1). Yu et al.

Revised: (2022) propose to regularize samples of small loss values (typically using non-robust features).

Original: Another approach is to adjust the learning rate schedule to avoid dramatic change of robustness that could be irreparable, as studied in Section 3.2.
Revised: Another approach is to adjust the learning rate schedule to avoid dramatic change of robustness that could be irreparable, as studied in Section 3.2.

Original: Previously, Rice et al.
Revised: Previously, Rice et al.

Original: and Wang & Wang (2022) also study different types of learning schedules, while we focus on the effect of LR decay factors on robust overfitting.
Revised: and Wang & Wang (2022) also study different types of learning schedules, while we focus on the effect of LR decay factors on robust overfitting.

Original: In particular, we firstly show that piecewise decay with a small decay factor can be helpful for both better robustness and less RO.
Revised: In particular, we firstly show that piecewise decay with a small decay factor can be helpful for both better robustness and less RO.

Original: Stronger Training Attacker.
Revised: Stronger Training Attacker.

Original: Another way to restore the balance is to adopt a stronger training attacker after LR decay , as we have explored in Section 3.3.
Revised: Another way to restore the balance is to adopt a stronger training attacker after LR decay , as we have explored in Section 3.3.

Original: Previously, from a curriculum training perspective, Cai et al.
Revised: Previously, from a curriculum training perspective, Cai et al.

Original: (2018), Qin et al.
Revised: (2018), Qin et al.

Original: (2019), and Wang et al.
Revised: (2019), and Wang et al.

Original: (2019) propose to start training from a weaker attacker ( e.g., ε = 0 ), and gradually lift its strength to the standard level ( e.g., ε = 8 / 255 ).
Revised: (2019) propose to start training from a weaker attacker ( e.g., ε = 0 ), and gradually lift its strength to the standard level ( e.g., ε = 8 / 255 ).

Original: These strategies are designed as a warmup stage at the earlier training stages.
Revised: These strategies are designed as a warmup stage at the earlier training stages.

Original: Pang et al.
Revised: Pang et al.

Original: (2020) point out that these warmup methods are hardly useful when measured by AA robustness.
Revised: (2020) point out that these warmup methods are hardly useful when measured by AA robustness.

Original: Instead, our method is designed from a dynamic game perspective, and we propose to adopt a stronger training attacker ( ε > 8 / 255 ) after the LR decay in order to counter robust overfitting.
Revised: Instead, our method is designed from a dynamic game perspective, and we propose to adopt a stronger training attacker ( ε > 8 / 255 ) after the LR decay in order to counter robust overfitting.

Original: Besides, our method shows clear benefits on improving best AA robustness as well as mitigating robust overfitting.
Revised: Besides, our method shows clear benefits on improving best AA robustness as well as mitigating robust overfitting.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we investigate the underlying cause of robust overfitting through the lens of dynamics feature robustness.
Revised: In this paper, we investigate the underlying cause of robust overfitting through the lens of dynamics feature robustness.

Original: To achieve this, we have proposed a new dynamic definition of feature robustness by taking the attacker A , the model M , and the dataset D into consideration.
Revised: To achieve this, we have proposed a new dynamic definition of feature robustness by taking the attacker A , the model M , and the dataset D into consideration.

Original: Using our new framework, we analyze the change of robust and non-robust features during the learning rate decay, and attribute robust overfitting to the falsely memorized non-robust features.
Revised: Using our new framework, we analyze the change of robust and non-robust features during the learning rate decay, and attribute robust overfitting to the falsely memorized non-robust features.

Original: Based on this understanding, we have proposed three new approaches to restore the balance between the model trainer and the attacker.
Revised: Based on this understanding, we have proposed three new approaches to restore the balance between the model trainer and the attacker.

Original: Experiments on benchmark datasets show that our proposed Bootstrapped Adversarial
Revised: Experiments on benchmark datasets show that our proposed Bootstrapped Adversarial

Original: Training (BoAT) largely resolves the overfitting issue, even under very long training.
Revised: Training (BoAT) largely resolves the overfitting issue, even under very long training.

Original: Meanwhile, it also achieves state-of-the-art robustness even without early stopping, showing that adversarial training can also benefit from longer training as long as the overfitting issue is resolved.
Revised: Meanwhile, it also achieves state-of-the-art robustness even without early stopping, showing that adversarial training can also benefit from longer training as long as the overfitting issue is resolved.

Original: A A DDITIONAL E XPERIMENT : A DVERSARIAL E XAMPLES ( IN A DVERSARIAL T RAINING ) C ONTAIN N ON - ROBUST F EATURES
Revised: A A DDITIONAL E XPERIMENT : A DVERSARIAL E XAMPLES ( IN A DVERSARIAL T RAINING ) C ONTAIN N ON - ROBUST F EATURES

Original: At the end of Section 2.2, we craft adversarial examples on model A and evaluate the misclassified ones on another model B with their correct labels to indicate the amount of features that have become robust.
Revised: At the end of Section 2.2, we craft adversarial examples on model A and evaluate the misclassified ones on another model B with their correct labels to indicate the amount of features that have become robust.

Original: We argue that this is reasonable because adversarial examples used for AT consist of robust features from the original label y and non-robust features from the misclassified label ˆ y , so B must have robustified those features (non-robust on A ) to correctly classify them.
Revised: We argue that this is reasonable because adversarial examples used for AT consist of robust features from the original label y and non-robust features from the misclassified label ˆ y , so B must have robustified those features (non-robust on A ) to correctly classify them.

Original: In this section, we further provide a rigorous discussion on this through an additional experiment.
Revised: In this section, we further provide a rigorous discussion on this through an additional experiment.

Original: Experiment Design.
Revised: Experiment Design.

Original: Recall that Ilyas et al.
Revised: Recall that Ilyas et al.

Original: (2019) leverage targeted PGD attack (Madry et al., 2018) to craft adversarial examples on a standard (non-robust) classifier and relabel them with their target labels to build a non-robust dataset.
Revised: (2019) leverage targeted PGD attack (Madry et al., 2018) to craft adversarial examples on a standard (non-robust) classifier and relabel them with their target labels to build a non-robust dataset.

Original: Finding standard training on it yields good accuracy on clean test data, they prove that those adversarial examples contain non-robust features corresponding to the target labels (Section 3.2 of their paper).
Revised: Finding standard training on it yields good accuracy on clean test data, they prove that those adversarial examples contain non-robust features corresponding to the target labels (Section 3.2 of their paper).

Original: However, different from their settings where non-robust features lie statically in the examples, we lay emphasis on mining non-robust features from adversarial examples generated on-the-fly in AT.
Revised: However, different from their settings where non-robust features lie statically in the examples, we lay emphasis on mining non-robust features from adversarial examples generated on-the-fly in AT.

Original: To this end, we first craft adversarial examples on the AT checkpoint at some epoch using untargeted
Revised: To this end, we first craft adversarial examples on the AT checkpoint at some epoch using untargeted

Original: PGD attack (Madry et al., 2018) to restore the real setting of AT, then relabel the misclassified ones ˆ x i ( e.g., a dog) with their misclassified labels ˆ y
Revised: PGD attack (Madry et al., 2018) to restore the real setting of AT, then relabel the misclassified ones ˆ x i ( e.g., a dog) with their misclassified labels ˆ y

Original: i ̸
Revised: i ̸

Original: = y i
Revised: = y i

Original: ( e.g., cat) to build a non-robust dataset { (ˆ x i , ˆ y i ) } .
Revised: ( e.g., cat) to build a non-robust dataset { (ˆ x i , ˆ y i ) } .

Original: Also, in order to capture non-robust features at exactly the training epoch these adversarial examples are used, we continue to perform standard training directly from the checkpoint on the non-robust dataset, and finally evaluate natural accuracy.
Revised: Also, in order to capture non-robust features at exactly the training epoch these adversarial examples are used, we continue to perform standard training directly from the checkpoint on the non-robust dataset, and finally evaluate natural accuracy.

Original: See Figure 5 for an illustration of our experiment.
Revised: See Figure 5 for an illustration of our experiment.

Original: In details, we select the AT models at epoch 60 (before LR decay) and epoch 1,000 (after LR decay) to conduct the above experiment.
Revised: In details, we select the AT models at epoch 60 (before LR decay) and epoch 1,000 (after LR decay) to conduct the above experiment.

Original: We use untargeted PGD-20 with perturbation norm ε ∞ =
Revised: We use untargeted PGD-20 with perturbation norm ε ∞ =

Original: 16 /to craft adversarial examples on those checkpoints from the training data of CIFAR-10 (Krizhevsky et al., 2009).
Revised: 16 /to craft adversarial examples on those checkpoints from the training data of CIFAR-10 (Krizhevsky et al., 2009).

Original: The attack we adopt is a little bit stronger than the attack of PGD-10 with ε ∞ = 8 /that is commonly used to generate adversarial examples in baseline adversarial training, because the training robust accuracy rises to as high as 94 .
Revised: The attack we adopt is a little bit stronger than the attack of PGD-10 with ε ∞ = 8 /that is commonly used to generate adversarial examples in baseline adversarial training, because the training robust accuracy rises to as high as 94 .

Original: 67 % at epoch 1,000 (Figure 1a) and gives less than 2,700 misclassified examples, which is significantly insufficient for further training.
Revised: 67 % at epoch 1,000 (Figure 1a) and gives less than 2,700 misclassified examples, which is significantly insufficient for further training.

Original: Using the stronger attack, we obtain a success rate of 78.38% on the checkpoint at epoch 60 and a success rate of 34.86% on the checkpoint at epoch 1,000.
Revised: Using the stronger attack, we obtain a success rate of 78.38% on the checkpoint at epoch 60 and a success rate of 34.86% on the checkpoint at epoch 1,000.

Original: Since the attack success rates are different, we randomly select a same number of misclassified adversarial examples for standard training for a fair comparison.
Revised: Since the attack success rates are different, we randomly select a same number of misclassified adversarial examples for standard training for a fair comparison.

Original: The learning rate is initially set to 0.1 and decays to 0.01 after 20 epochs for anotherepochs of fine-tuning.
Revised: The learning rate is initially set to 0.1 and decays to 0.01 after 20 epochs for anotherepochs of fine-tuning.

Original: Given that the original checkpoints already have nontrivial natural accuracy, we also add two control groups that train with random labels instead of ˆ y to exclude the influence that may brought by the original accuracy.
Revised: Given that the original checkpoints already have nontrivial natural accuracy, we also add two control groups that train with random labels instead of ˆ y to exclude the influence that may brought by the original accuracy.

Original: Results.
Revised: Results.

Original: As shown in Figure 6, we find that standard training on the non-robust dataset { (ˆ x i , ˆ y i ) } successfully converges to fairly good accuracy ( > 55% ) on natural test images, i.e., predicting cats as cats, no matter from which AT checkpoints (either epoch 60 or epoch 1,000) the adversarial examples are crafted.
Revised: As shown in Figure 6, we find that standard training on the non-robust dataset { (ˆ x i , ˆ y i ) } successfully converges to fairly good accuracy ( > 55% ) on natural test images, i.e., predicting cats as cats, no matter from which AT checkpoints (either epoch 60 or epoch 1,000) the adversarial examples are crafted.

Original: Also, we can see that the nontrivial natural accuracy has nothing to do with the original accuracy of the AT checkpoints, as the accuracy plummets to around 10% (random guessing) as soon as the standard training starts with random labels.
Revised: Also, we can see that the nontrivial natural accuracy has nothing to do with the original accuracy of the AT checkpoints, as the accuracy plummets to around 10% (random guessing) as soon as the standard training starts with random labels.

Original: This proves that adversarial examples in adversarial training do contain non-robust features w.r.t.
Revised: This proves that adversarial examples in adversarial training do contain non-robust features w.r.t.

Original: the classes to which they are misclassified , which strongly corroborates the validity of the experiments in Section 2.2.
Revised: the classes to which they are misclassified , which strongly corroborates the validity of the experiments in Section 2.2.

Revised: > 55% ).

Revised: The failure of training with random labels proves that the good accuracy has nothing to do with the original natural accuracy of the AT checkpoint.

Original: A Discussion on the Influence Brought by Robust Features During Standard Training.
Revised: A Discussion on the Influence Brought by Robust Features During Standard Training.

Original: Since untargeted PGD attack cannot be assigned with a target label, we cannot guarantee that the misclassified labels ˆ y to be uniformly distributed regardless of the original labels (especially for real-world datasets).
Revised: Since untargeted PGD attack cannot be assigned with a target label, we cannot guarantee that the misclassified labels ˆ y to be uniformly distributed regardless of the original labels (especially for real-world datasets).

Original: This implies that the non-robust features are not completely decoupled from robust features, i.e., training on { (ˆ x i , ˆ y i ) } may take advantage of ˆ x i ’s robust features from y i through the correlation between y i and ˆ y i .
Revised: This implies that the non-robust features are not completely decoupled from robust features, i.e., training on { (ˆ x i , ˆ y i ) } may take advantage of ˆ x i ’s robust features from y i through the correlation between y i and ˆ y i .

Original: However, we argue that robust features from y i mingling with non-robust features from ˆ y i only increases the difficulty of obtaining a good natural accuracy.
Revised: However, we argue that robust features from y i mingling with non-robust features from ˆ y i only increases the difficulty of obtaining a good natural accuracy.

Original: This is because learning through the shortcut will only wrongly map the robust features from y i to ˆ y i that never equals to y i , but during the evaluation, each clean test example x i will always have robust and non-robust features from y i , and such wrong mapping will induce x i to be misclassified to some ˆ y i due to the label of its robust features.
Revised: This is because learning through the shortcut will only wrongly map the robust features from y i to ˆ y i that never equals to y i , but during the evaluation, each clean test example x i will always have robust and non-robust features from y i , and such wrong mapping will induce x i to be misclassified to some ˆ y i due to the label of its robust features.

Original: As a result, despite the negative influence brought by robust features, we still achieve good natural accuracy at last, which even solidifies our conclusion. 
Revised: As a result, despite the negative influence brought by robust features, we still achieve good natural accuracy at last, which even solidifies our conclusion. B E XPERIMENT D ETAILS

Revised: In this section, we provide more experiment details that are omitted before due to the page limit.

Original: B E XPERIMENT D ETAILS B. B ASELINE A DVERSARIAL T RAINING
Revised: B. B ASELINE A DVERSARIAL T RAINING

Original: In this paper, we mainly consider classification task on CIFAR-10 (Krizhevsky et al., 2009).
Revised: In this paper, we mainly consider classification task on CIFAR-10 (Krizhevsky et al., 2009).

Original: The dataset contains 60,000 32 × 32 RGB images from 10 classes.
Revised: The dataset contains 60,000 32 × 32 RGB images from 10 classes.

Original: For each class, there are 5,000 images for training and 1,000 images for evaluation.
Revised: For each class, there are 5,000 images for training and 1,000 images for evaluation.

Original: For baseline adversarial training, We use PreActResNet-18 (He et al., 2016) model as the classifier.
Revised: For baseline adversarial training, We use PreActResNet-18 (He et al., 2016) model as the classifier.

Original: We use PGD-10 attack (Madry et al., 2018) with step size α = 2 / 255 and perturbation norm ε ∞ = 8 / 255 to craft adversarial examples on-the-fly.
Revised: We use PGD-10 attack (Madry et al., 2018) with step size α = 2 / 255 and perturbation norm ε ∞ = 8 / 255 to craft adversarial examples on-the-fly.

Original: We use SGD optimizer with momentum 0 .
Revised: We use SGD optimizer with momentum 0 .

Original: 9 , weight decay 5 × 10 − 4 and batch size 128 to train the model for as many as 1,000 epochs.
Revised: 9 , weight decay 5 × 10 − 4 and batch size 128 to train the model for as many as 1,000 epochs.

Original: The learning rate (LR) is initially set to be 0.1 and decays to 0.01 at epoch 100 and further decays to 0.
Revised: The learning rate (LR) is initially set to be 0.1 and decays to 0.01 at epoch 100 and further decays to 0.

Original: at epoch 150.
Revised: at epoch 150.

Original: For the version without LR decay used for comparison in our paper, we simply keep the LR to be 0.1 during the whole training process.
Revised: For the version without LR decay used for comparison in our paper, we simply keep the LR to be 0.1 during the whole training process.

Original: Each model included in this paper is trained on a single NVIDIA GeForce RTX 3090 GPU.
Revised: Each model included in this paper is trained on a single NVIDIA GeForce RTX 3090 GPU.

Original: For
Revised: For

Original: PGD-AT, it takes about 3d 14h to finish 1,000 epochs of training.
Revised: PGD-AT, it takes about 3d 14h to finish 1,000 epochs of training.

Revised: B. L EARNING R ATE D ECAY I NDUCES R OBUST O VERFITTING

Revised: In this paper, one of our most essential observations is that RO often happens after LR decays.

Revised: We further verify that this is the general case instead of a phenomenon that happens to occur only when the LR first decays at epoch 100 by allowing it to decay at different times during the training.

Revised: Figure 7 demonstrates that whenever the LR decays, the test robust accuracy immediately increases by more than 5% and then slowly drops, implying that RO is indeed caused by LR decay.

Original: B. D YNAMIC F EATURE R OBUSTNESS
Revised: B. D YNAMIC F EATURE R OBUSTNESS

Original: Features of the Training Data Become Robust After LR Decay.
Revised: Features of the Training Data Become Robust After LR Decay.

Original: We craft adversarial examples on a checkpoint before LR decay and evaluate the misclassified ones on a checkpoint after LR decay with their correct labels to evaluate the amount of features that have become robust (see Appendix
Revised: We craft adversarial examples on a checkpoint before LR decay and evaluate the misclassified ones on a checkpoint after LR decay with their correct labels to evaluate the amount of features that have become robust (see Appendix

Original: A for detailed discussions) at the end of Section 2.2.
Revised: A for detailed discussions) at the end of Section 2.2.

Original: Following Appendix A, we adopt PGDattack with perturbation norm ε ∞ = 16 / 255 to craft adversarial examples which is stronger than[Figure] the common attack setting we use in PGD-AT.
Revised: Following Appendix A, we adopt PGDattack with perturbation norm ε ∞ = 16 / 255 to craft adversarial examples which is stronger than the common attack setting we use in PGD-AT.

Original: We note that test adversarial examples crafted by a stronger attack indicates stronger extraction of their non-robust features, so they are more indicative of feature robustification when still correctly classified.
Revised: We note that test adversarial examples crafted by a stronger attack indicates stronger extraction of their non-robust features, so they are more indicative of feature robustification when still correctly classified.

Original: Memorizing Non-robust Features Harms Test Robustness.
Revised: Memorizing Non-robust Features Harms Test Robustness.

Original: At the beginning of Section 2.3, we create synthetic datasets to demonstrate that memorizing the non-robust training features indeed harms test-time model robustness.
Revised: At the beginning of Section 2.3, we create synthetic datasets to demonstrate that memorizing the non-robust training features indeed harms test-time model robustness.

Original: To instill non-robust features into the training dataset, we minimize the adversarial loss w.r.t.
Revised: To instill non-robust features into the training dataset, we minimize the adversarial loss w.r.t.

Original: the training data in a way that just like PGD attack, with the only difference that we minimize the adversarial loss instead of maximizing it.
Revised: the training data in a way that just like PGD attack, with the only difference that we minimize the adversarial loss instead of maximizing it.

Original: Since we only use a very small perturbation norm ε ∞ ≤ 4 / 255 , the added features are bound to be non-robust.
Revised: Since we only use a very small perturbation norm ε ∞ ≤ 4 / 255 , the added features are bound to be non-robust.

Original: For a fair comparison, we also perturb the training set with random uniform noise of the same perturbation norm to exclude the influence brought by (slight) data distribution shifts.
Revised: For a fair comparison, we also perturb the training set with random uniform noise of the same perturbation norm to exclude the influence brought by (slight) data distribution shifts.

Original: We continue training from the 100-th baseline
Revised: We continue training from the 100-th baseline

Original: AT checkpoint (before LR decay) on each synthetic dataset for 10 epochs, and then evaluate model robustness with clean test data.
Revised: AT checkpoint (before LR decay) on each synthetic dataset for 10 epochs, and then evaluate model robustness with clean test data.

Original: Verification: Lack of Target-class Features.
Revised: Verification: Lack of Target-class Features.

Original: This is to say that when RO happens, we expect that test adversarial examples become less informative of the classes to which they are misclassified according to our theory.
Revised: This is to say that when RO happens, we expect that test adversarial examples become less informative of the classes to which they are misclassified according to our theory.

Original: To verify this, we first craft adversarial examples on a checkpoint T after
Revised: To verify this, we first craft adversarial examples on a checkpoint T after

Original: RO begins, then evaluate the misclassified ones with their misclassified labels ˆ y on the checkpoint saved at epoch 60.
Revised: RO begins, then evaluate the misclassified ones with their misclassified labels ˆ y on the checkpoint saved at epoch 60.

Original: As a result, the accuracy reflects how much features from ˆ y the examples have to contain to be misclassified on T .
Revised: As a result, the accuracy reflects how much features from ˆ y the examples have to contain to be misclassified on T .

Original: All adversarial examples evaluated in the experiments in Section 2.3 are crafted using PGD-10 attack with perturbation norm ε ∞ = 8 / 255 .
Revised: All adversarial examples evaluated in the experiments in Section 2.3 are crafted using PGD-10 attack with perturbation norm ε ∞ = 8 / 255 .

Original: Verification: Bilateral Class Correlation.
Revised: Verification: Bilateral Class Correlation.

Original: To quantitatively analyse the correlation strength of bilateral misclassification described in Section 2.3, empirical verification 2, we first summarize all y i
Revised: To quantitatively analyse the correlation strength of bilateral misclassification described in Section 2.3, empirical verification 2, we first summarize all y i

Original: → y j misclassification rates into two confusion matrices P train , P test ∈ R C × C for the training and test data, respectively.
Revised: → y j misclassification rates into two confusion matrices P train , P test ∈ R C × C for the training and test data, respectively.

Original: Because we are mainly interested in the effect of the LR decay, we focus on the relative change on the test data, i.e., ∆ P test =
Revised: Because we are mainly interested in the effect of the LR decay, we focus on the relative change on the test data, i.e., ∆ P test =

Original: P testnew − P testold .
Revised: P testnew − P testold .

Original: According to our analysis above, for each class pair ( i, j ) , there should be a strong correlation between the training misclassification of i → j before LR decays, i.e., ( P trainold ) ij , and the increase of test misclassification of j → i , i.e., ∆ P test ji .
Revised: According to our analysis above, for each class pair ( i, j ) , there should be a strong correlation between the training misclassification of i → j before LR decays, i.e., ( P trainold ) ij , and the increase of test misclassification of j → i , i.e., ∆ P test ji .

Original: To examine their relationship, we plot the two variables (( P trainold ) ij , ∆ P test ji ) and compute their Pearson correlation coefficient ρ . 
Revised: To examine their relationship, we plot the two variables (( P trainold ) ij , ∆ P test ji ) and compute their Pearson correlation coefficient ρ . [Figure]

Original:  Symmetrized Confusion Matrix.
Revised: Verification: Symmetrized Confusion Matrix.

Original: In Section 2.3, we mention the growing symmetry of the test-time confusion matrix after LR decays as an evidence of the strengthening A → B and B → A correlation.
Revised: In Section 2.3, we mention the growing symmetry of the test-time confusion matrix after LR decays as an evidence of the strengthening A → B and B → A correlation.

Original: Here we present more confusion matrices during the first 200 epochs of the training in Figure 7, and it is very clear that the confusion matrices soon become symmetric after LR decays at epochwhen robust overfitting starts.
Revised: Here we present more confusion matrices during the first 200 epochs of the training in Figure 8, and it is very clear that the confusion matrices soon become symmetric after LR decays at epoch 100 when robust overfitting starts.

Original: For a deeper comprehension of this phenomenon, we first visualize the confusion matrices of the training and the test data at an epoch before robust overfitting starts in Figure 8.
Revised: For a deeper comprehension of this phenomenon, we first visualize the confusion matrices of the training and the test data at an epoch before robust overfitting starts in Figure 9.

Original: They exhibit nearly a same pattern of attacking preference among classes ( e.g., A → B ) due to the bias rooted in the dataset, e.g., class 6 is intrinsically vulnerable in this case.
Revised: They exhibit nearly a same pattern of attacking preference among classes ( e.g., A → B ) due to the bias rooted in the dataset, e.g., class 6 is intrinsically vulnerable in this case.

Original: For the test data, this intrinsic bias wouldn’t be wiped out through learning due to the non-generalizability of robustified training data features demonstrated in Section 2.2 ( i.e., A → B still holds); and for the training data, this biased feature memorization will open shortcuts for test-time adversarial attack as discussed in Section 2.3 ( i.e., B
Revised: For the test data, this intrinsic bias wouldn’t be wiped out through learning due to the non-generalizability of robustified training data features demonstrated in Section 2.2 ( i.e., A → B still holds); and for the training data, this biased feature memorization will open shortcuts for test-time adversarial attack as discussed in Section 2.3 ( i.e., B

Original: → A begins).
Revised: → A begins).

Original: Combining both A → B and B → A , we arrive at the symmetry of test-time confusion matrix. 
Revised: Combining both A → B and B → A , we arrive at the symmetry of test-time confusion matrix. Changing LR Decay Schedule.

Revised: Our discussion above is based on the piecewise LR decay schedule, in which the sudden decay of LR most obviously reflects our understandings.

Revised: Besides, we also explore using other decay schedules, including Cosine/Linear LR decay, to check whether different decay schedules will affect the observations and claims we made in this paper.

Revised: For each schedule, we train the model for 200 epochs following the settings in Rice et al.

Revised: As demonstrated in

Original: B. T HE E FFECT OF S TRONGER T RAINING A TTACKER
Revised: Figure 10a, we arrive at the same finding as Rice et al. (2020) that with Cosine/Linear LR decay schedule, the training still suffers from severe RO after epoch 130. Then, we rerun the three empirical experiments in Section 2.3 and find that under both two LR decay schedules 1) the test adversarial examples indeed contain less and less target-class information as the training goes (Figure 10b), 2) the bilateral class correlation becomes increasingly strong (Figure 10c) and 3) the confusion matrix [Figure]

Revised: indeed becomes symmetric (Figure 11).

Revised: The results are almost the same as the results achieved when we adopt piecewise LR decay schedule indicating that our understandings in the cause of RO is fundamental and regardless of whatever LR decay schedule is used.

Revised: B. T HE E FFECT OF S TRONGER T RAINING A TTACKER

Original: Stronger Attack for Stronger Defense.
Revised: Stronger Attack for Stronger Defense.

Original: In Section 3.3, we point out that using a stronger attacker in
Revised: In Section 3.3, we point out that using a stronger attacker in

Original: AT is able to mitigate RO to some extent by neutralizing the learning power when it is overly strong.
Revised: AT is able to mitigate RO to some extent by neutralizing the learning power when it is overly strong.

Original: To achieve the results reported in Figure 3d 
Revised: To achieve the results reported in Figure 3d we craft adversarial examples on-the-fly with more PGD iteration steps when ε is larger (see Table 3) and further evaluate the best and last robustness of the WA models against PGD-20 and AA.

Original: [Table]
Revised: [Table]

Original: we craft adversarial examples on-the-fly with more PGD iteration steps when ε is larger (see Table 3), and further evaluate the best and last robustness of the WA models
Revised: ε = 12 / 255 , PGD- [Table] ε = 14 / 255 , PGD-

Original: [Table]
Revised: [Table]

Original: Although RO is only partially mitigated and natural accuracy decreases when a stronger attacker is applied
Revised: ε = 16 / 255 , PGD-

Original: [Table]
Revised: [Table]

Original:  it is surprising to find from Table 3 that an attacker of appropriate strength may significantly boost the best robustness.
Revised: Although RO is only partially mitigated and natural accuracy decreases when a stronger attacker is applied, it is surprising to find from Table 3 that an attacker of appropriate strength may significantly boost the best robustness.

Original: This suggests using a stronger attack could potentially be an interesting new path to stronger adversarial defense, and we leave it for future work.
Revised: This suggests using a stronger attack could potentially be an interesting new path to stronger adversarial defense, and we leave it for future work.

Original: Using Stronger Training Adversarial Attack in BoAT.
Revised: Using Stronger Training Adversarial Attack in BoAT.

Original: Above we show BoAT is almost capable of eliminating RO.
Revised: Above we show BoAT is almost capable of eliminating RO.

Original: However, since we also show leveraging stronger adversarial attack in AT may help mitigate RO while boosting model robustness above, it is natural to further combine BoAT with this training scheme.
Revised: However, since we also show leveraging stronger adversarial attack in AT may help mitigate RO while boosting model robustness above, it is natural to further combine BoAT with this training scheme.

Original: According to Figure 9, though a slightly stronger attack ( e.g., ε = 9 / 255 ) may marginally improves the best and last robust accuracy, it heavily degrades natural accuracy, particularly when much stronger attack is used.
Revised: According to Figure 12, though a slightly stronger attack ( e.g., ε = 9 / 255 ) may marginally improves the best and last robust accuracy, it heavily degrades natural accuracy, particularly when much stronger attack is used.

Original: We deem that this is because it breaks the balance that BoAT has achieved: with relatively weak learning ability of the trainer T , a stronger attacker A naturally dominates the adversarial game and results in an underfitting state that harms both robust and natural accuracy.
Revised: We deem that this is because it breaks the balance that BoAT has achieved: with relatively weak learning ability of the trainer T , a stronger attacker A naturally dominates the adversarial game and results in an underfitting state that harms both robust and natural accuracy.

Original: B. M ITIGATING R OBUST O VERFITTING WITH B O AT
Revised: B. M ITIGATING R OBUST O VERFITTING WITH B O AT

Original: Datasets.
Revised: Datasets.

Original: Beside CIFAR-10, we also include CIFAR-100 (Krizhevsky et al., 2009) and TinyImageNet (Deng et al., 2009) for evaluation of the effectiveness of BoAT.
Revised: Beside CIFAR-10, we also include CIFAR-100 (Krizhevsky et al., 2009) and TinyImageNet (Deng et al., 2009) for evaluation of the effectiveness of BoAT.

Original: CIFAR-100 shares the same training and test images with CIFAR-10, but it classifies them into 100 categories, i.e., training images and 100 test images for each class.
Revised: CIFAR-100 shares the same training and test images with CIFAR-10, but it classifies them into 100 categories, i.e.,training images and 100 test images for each class.

Original: Tiny-ImageNet is a subset of ImageNet (Deng et al., 2009) which contains labeled 64 × 64 RGB images from 200 classes.
Revised: Tiny-ImageNet is a subset of ImageNet (Deng et al., 2009) which contains labeled 64 × 64 RGB images from 200 classes.

Original: For each class, it includes 500 and 50 images for training and evaluation respectively.
Revised: For each class, it includes 500 and 50 images for training and evaluation respectively.

Original: Training Strategy.
Revised: Training Strategy.

Original: For CIFAR-10 and CIFAR-100, we follow exactly the same training strategy as introduced in B.1.
Revised: For CIFAR-10 and CIFAR-100, we follow exactly the same training strategy as introduced in B.1.

Original: For Tiny-ImageNet, we follow the learning schedule of Chen et al.
Revised: For Tiny-ImageNet, we follow the learning schedule of Chen et al.

Original: (2021), in which the model is trained for a total of 100 epochs and the LR decays twice (by 0.1) at epoch 50 and 80.
Revised: (2021), in which the model is trained for a total of 100 epochs and the LR decays twice (by 0.1) at epoch 50 and 80.

Original: Choices of Hyperparameters.
Revised: Choices of Hyperparameters.

Original: For KD+SWA (Chen et al., 2021), PGD-AT+TE (Dong et al., 2022),
Revised: For KD+SWA (Chen et al., 2021), PGD-AT+TE (Dong et al., 2022),

Original: AWP (Wu et al., 2020) and WA+CutMix (Rebuffi et al., 2021), we strictly follow their original settings of hyperparameters.
Revised: AWP (Wu et al., 2020) and WA+CutMix (Rebuffi et al., 2021), we strictly follow their original settings of hyperparameters.

Original: SWA/WA as well as the BoAT regularization purposed in Section 3.
Revised: SWA/WA as well as the BoAT regularization purposed in Section 3.

Original: starts at epoch 105 (5 epochs later than the first LR decay where robust overfitting often begins), and following (Rebuffi et al., 2021)
Revised: starts at epoch 105 (5 epochs later than the first LR decay where robust overfitting often begins), and following (Rebuffi et al., 2021)

Original: we choose the EMA decay rate of WA to be γ = 0 . 999 .
Revised: we choose the EMA decay rate of WA to be γ = 0 . 999 .

Original: Please refer to Table 4 for our choices of the decay factor d and regularization strength λ .
Revised: Please refer to Table 4 for our choices of the decay factor d and regularization strength λ .

Original: We notice that since
Revised: We notice that since

Original: CutMix improves the difficulty of learning, the model demands a relatively larger decay factor to better fit the augmented data.
Revised: CutMix improves the difficulty of learning, the model demands a relatively larger decay factor to better fit the augmented data.

Original: For Tiny-ImageNet, we also apply a larger λ after the second LR decay to better maintain the flatness of adversarial loss landscape and control robust overfitting.
Revised: For Tiny-ImageNet, we also apply a larger λ after the second LR decay to better maintain the flatness of adversarial loss landscape and control robust overfitting.

Original: [Table]
Revised: [Table]

Original: BoAT d = 1 . 5 , 
Revised: d = 1 . 5 , λ = 1 .d

Original: λ
Revised: =

Original: = 1 .d = 4 . 0 , λ 1 = 2 . 0 , λ 2 = 10 .
Revised: 4 . 0 , λ 1 = 2 . 0 , λ 2 = 10 .

Original: [Table]

Original: BoAT+Cutmix d = 4 . 0 , λ
Revised: BoAT+Cutmix d = 4 . 0 , λ

Original: = 2 .d
Revised: = 2 .d

Original: = 6 . 0 , λ = 1 .
Revised: = 6 . 0 , λ = 1 .

Original: WideResNet-34BoAT d = 1 . 3 , λ = 0 .-
Revised: WideResNet- [Table] BoAT d = 1 . 3 , λ = 0 .-

Original: BoAT+Cutmix d = 4 . 0 , λ = 2 .-
Revised: BoAT+Cutmix d = 4 . 0 , λ = 2 .-

Revised: B. T RAINING R OBUST A CCURACY

Revised: Figure 13 shows the robust accuracy change on the training data during the training.

Revised: Compared with vanilla PGD-AT that yields training robust accuracy over 80% at epoch 200, BoAT manages to suppress it to only 65%.

Revised: It successfully prevents the model from learning the non-robust features w.r.t.

Revised: the training data too fast and too well, and therefore significantly reduces the robust generalization gap (from ∼ 35% to ∼ 9% ) and mitigates robust overfitting.

Original: C A BLATION S TUDY ON B O AT
Revised: C M ORE E XPERIMENTS ON B O AT

Revised: In this section, we conduct extensional experiments on the proposed BoAT method to further demonstrate its effectiveness, efficiency and flexibility.

Revised: C. A BLATION S TUDY

Original: Effectiveness of BoAT Loss.
Revised: Effectiveness of BoAT Loss.

Original: In Section 3.1, Figure 3a has demonstrated the effectiveness of BoAT loss.
Revised: In Section 3.1, Figure 3a has demonstrated the effectiveness of BoAT loss.

Original: Table 5 further report the detailed robustness against PGD-20 and AA, showing that the purposed regularization not only boosts the best robustness by a large margin (0.67% higher against AA) but also significantly suppresses RO (1.52% v.s. 6.10% against AA).
Revised: Table 5 further report the detailed robustness against PGD-20 and AA, showing that the purposed regularization not only boosts the best robustness by a large margin (0.67% higher against AA) but also significantly suppresses RO (1.52% v.s. 6.10% against AA).

Original: To achieve the reported robustness, we first use λ 1 = 10 . 

Revised: To achieve the reported robustness, we first use λ 1 = 10 .

Original: after the first LR decay and then apply λ 2 = 60 . 

Revised: after the first LR decay and then apply λ 2 = 60 .

Original: after the second LR decay to better maintain the flatness of adversarial loss landscape and control robust overfitting. 
Revised: after the second LR decay to better maintain the flatness of adversarial loss landscape and control robust overfitting. [Table]

Original: Larger Decay Factor Goes with Stronger BoAT Regularization.
Revised: Larger Decay Factor Goes with Stronger BoAT Regularization.

Original: Below we show that when a relatively large decay factor d is applied, i.e., the model has overly strong fitting ability that results in robust overfitting, a large regularization coefficient λ should be choosen for better performance.
Revised: Below we show that when a relatively large decay factor d is applied, i.e., the model has overly strong fitting ability that results in robust overfitting, a large regularization coefficient λ should be choosen for better performance.

Original: Table 6 reveals this relationship between d and λ .
Revised: Table 6 reveals this relationship between d and λ .

Original: When d = 1 . 3 , even a λ as small as 1.0 will harm both the best and last robustness as well as natural accuracy, as d = 1 . 3 is already too small a decay factor that makes the model suffering from underfitting and naturally requiring no more flatness [Table]
Revised: When d = 1 . 3 , even a λ as small as 1.0 will harm both the best and last robustness as well as natural accuracy, as d = 1 . 3 is already too small a decay factor that makes the model suffering from underfitting and naturally requiring no more flatness regularization.

Original: regularization. On the other side, when d is relatively large, even a strong regularization of λ = 4 .is not adequate to fully suppress robust overfitting.
Revised: On the other side, when d is relatively large, even a strong regularization of λ = 4 .is not adequate to fully suppress robust overfitting.

Original: Besides, comparing the situation of λ > 0 and λ = 0 for a fixed d ( ≥ 1 . 5) , we emphasize that the purposed BoAT loss again exhibits its apparent effectiveness in simultaneously boosting the best robustness and mitigating robust overfitting. 
Revised: Besides, comparing the situation of λ > 0 and λ = 0 for a fixed d ( ≥ 1 . 5) , we emphasize that the purposed BoAT loss again exhibits its apparent effectiveness in simultaneously boosting the best robustness and mitigating robust overfitting. [Table] C [Table]

Revised: I MPROVING N ATURAL A CCURACY WITH K NOWLEDGE D ISTILLATION

Original: [Table]
Revised: [Table]

Revised: to adopt knowledge distillation (KD) (Hinton et al., 2015) to mitigate RO and it is worth mentioning that their method achieves relatively good natural accuracy according to

Revised: Table 1 and 2.

Revised: Since our BoAT method is orthogonal to KD, we propose to combine our techniques with KD to further improve natural accuracy.

Revised: Specifically, we simplify their method by using only a non-robust standard classifier as a teacher (ST teacher) instead of using both a ST teacher and a AT teacher, because i) a large sum of computational cost for training the AT teacher will be saved, ii) our main goal is to improve natural accuracy so the ST teacher matters more and

Revised: iii) BoAT already use the WA model as a very good teacher.

Revised: This gives the final loss function as

Revised: [Equation]

Revised: where f ST indicates the ST teacher and λ ST is a trade-off parameter.

Revised: [Table]

Revised: Table 7 compares the performance of BoAT+KD when different λ ST is applied, and clearly a large λ ST results in improvement in natural accuracy and decreases robustness (may due to the theoretically principled trade-off between natural accuracy and robustness (Zhang et al., 2019)).

Revised: However, it is still noteworthy that when λ ST = 0 . 4 , a notable increase in natural accuracy ( ∼ 1 .

Revised: 70% ) is achieved at the cost of only a small slide of ∼ 0 .

Revised: 25% in robustness against AutoAttack.

Revised: Moreover, we emphasize that RO is almost completely eliminated regardless of the trade-off, which is the main concern of this paper and demonstrates the superiority of our method against previous ones.

Revised: C. C HANGING L EARNING R ATE D ECAY S CHEDULE

Revised: In the previous experiments we only investigate the piecewise LR decay schedule.

Revised: However, a natural idea would be using mild LR decay schedules, e.g., Cosine and Linear decay schedule, instead of suddenly decaying it by a factor of d at some epoch in the piecewise decay schedule.

Revised: Therefore, we experiment with Cosine and Linear decay schedule and summarise the results in Table 8.

Revised: To be specific, the LR still decays to 0.01 at epoch 150 and 0.001 at epoch 200, though the two decay stages (from epoch 100 to 150 and 150 to 200) are designed to be gradual following the Cosine/Linear schedule.

Revised: We also gradually increase the strength of BoAT regularization from zero as the LR gradually decreases, following the ”larger decay factor goes with stronger BoAT regularization” principle that we introduced in Appendix C.1.

Revised: [Table]

Revised: It can be concluded from Table 8 that simply changing the LR decay schedule indeed improves the best robust accuracy from 49.92% to 50.79% and 50.83% against AutoAttack respectively, but it provides no help at all in mitigating RO as the last robust accuracy is still around 44% against

Revised: AutoAttack.

Revised: We also note that in this situation, the application of BoAT loss not only significantly mitigates RO but also further improves the best model robustness, which also proves its effectiveness.

Revised: C. U SING V ALIDATION S ET

Revised: Following Wu et al.

Revised: Dong et al.

Revised: (2022) and Yu et al.

Revised: (2022), we report the test accuracy on the best checkpoint that achieves the highest robust test accuracy.

Revised: Another way to evaluate model robustness is selecting the model that achieves the highest robustness on validation set as the best checkpoint.

Revised: Therefore, following Rice et al.

Revised: , we hold out 1,000 examples from the CIFARtraining set for validation purposes and rerun the experiments using PGD-AT and BoAT.

Revised: - architecture.

Revised: [Table]

Revised: Using Validation Set PGD-AT

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: BoAT

Revised: [Table]

Revised: Highest on Test Data

Revised: [Table]

Revised: From Table 9, we can see that the selection criterion seems to make little difference.

Revised: In our BoAT, changing test to validation only causes very slight drop in natural (0.1%) and AA (0.2%) accuracy.

Revised: Therefore, our validation-based best-epoch robustness (51.13% against AutoAttack) is still significantly better than all the other test-based methods in Table 1.

Revised: C. R ESULTS ON M ORE N ETWORK A RCHITECTURES

Revised: In previous experiments we compare methods based on PreActResNet-18 and WideResNet-34architecture, and here we also adopt VGG-16 (Simonyan & Zisserman, 2015) and MobileNetV (Sandler et al., 2018) architecture.

Revised: The significant improvement against baseline PGD-AT in both best and final robust accuracy and in mitigating RO reported in Table 10 further demonstrates that our method is applicable to a wide range of network architectures.

Revised: C. T RAINING E FFICIENCY

Revised: We also test and report the training time (per epoch) of several methods evaluated in this paper.

Revised: For a fair comparison, all the compared methods are integrated into a universal training framework and each test runs on a single NVIDIA GeForce RTX 3090 GPU.

Revised: From Table 11, we can see that BoAT requires nearly no extra computational cost compared with vanilla PGD-AT (136.2s v.s. 131.6s per epoch), implying that it is an efficient training method in practical.

Revised: We also remark that KD+SWA, one of the most competitive methods that aims to address

Revised: [Table]

Revised: [Table]

Revised: the RO issue

Revised: [Table]

Revised: is not really computationally efficient as it requires to pretrain a robust classifier and a non-robust one as AT teacher and ST teacher respectively.

Revised: C. C OMPARISON WITH MLCAT

Revised: Recently, Yu et al.

Revised: (2022) propose their understandings in robust overfitting and propose a method

Revised: MLCAT to alleviate RO.

Revised: Here we compare our work with theirs in details.

Revised: Methodologies.

Revised: Yu et al.

Revised: (2022) analyze robust overfitting by investigating the roles of easy and hard samples and propose to regularize samples of small loss values (typically using non-robust features).

Revised: Nevertheless, their discussions cannot fully explain how robust overfitting rises from easy samples.

Revised: In comparison

Revised: [Table]

Revised: we provide a generic explanation for the rise of robust overfitting from a feature dynamics perspective, and we verify this understanding through extensive experiments.

Revised: Our proposed

Revised: BoAT is also designed from a bootstrapping perspective, which is different from theirs.

Revised: At last, our understandings can also help understand how Yu et al.

Revised: (2022)’s method works: by regularizing small-loss data, they enhance robust features and diminish non-robust features, which helps alleviate robust overfitting.

Revised: [Table]

Revised: Performance and Efficiency.

Revised: Table 12 shows that 1) our BoAT method is consistently superior (¿1% robust improvements against AutoAttack) to MLCAT across different architectures and different backbone networks, and

Revised: 2) BoAT has weaker robust overfitting than MLCAT.

Revised: Table 11 further shows that 3) BoAT is much more training efficient by requiring only 1/3 training time of MLCAT.

Revised: Therefore,

Revised: BoAT is both more effective, more efficient, and has less overfitting than MLCAT.

Revised: D V ISUALIZING A DVERSARIAL E XAMPLES

Revised: In Figure 14, We visualize some adversarial examples generated in the experiments (following the settings introduced in Appendix B.3) in Section 2.2 and 2.3.