Paper: On the Onset of Robust Overfitting in Adversarial Training (ICLR-2024)

Reviewer comment (Writing):
There are a lot of undefined words: e.g., non-effective features and learning state. Since these words are frequently used in the analysis for developing the proposed method, readers could not understand why the proposed method is effective to mitigate overfitting. Non-effective features and learning state should be defined by using equations and empirically evaluated or theoretically evaluated in the existence of adversarial training.

Author response:
**A1:** The non-effective features denote the robust features in the training set that lack generalization. The term "learning state gap" between the training and test sets has been revised to "robustness gap" in our updated manuscript. This change has been made consistently throughout the paper. Additionally, we employ quantitative metrics such as adversarial loss and robust accuracy to measure the model's robustness, providing evidential support for the existence of a robustness gap between the training and test datasets, as illustrated in Figure 1(b). Regarding the generation of non-effective features, our analysis is as follows: during the adversarial training process, the model's adversarial robustness on the training set exceeds that on the test set, as evidenced by the adversarial loss and robust accuracy observed in Figure 1(b). This results in a robustness gap between the training and test data. Considering that adversarial perturbations are generated on-the-fly and adaptively adjusted based on the model's robustness. The robustness gap between the training and test data leads to distinct adversarial perturbation on the robust features in these datasets. The varying degree of adversarial perturbation on the robust features in the training and test data amplify the distribution differences between these two datasets, thereby degrading the generalization of robust features on the training set and facilitating the generation of non-effective features. The increasing non-effective features further exacerbates the robustness gap between training and test data, forming a vicious cycle. These analyses are detailed in our updated manuscript presented in Section 1, Section 3, and Figure 2. We appreciate the valuable suggestion to refine our analysis, and we believe this adjustment enhances the clarity and rigour of our research.

Reviewer comment (Experiments):
Most parts of claim do not have objective evidence. For scientific articles, most of claims should be supported by the evidence. For example, the following states do not supported by the experimental or theoretical results:   
_"In the initial stages of adversarial training, due to the similarity in the model’s learning states between the training and test datasets, the boundary between the robust and non-robust features doesn’t significantly differ between the training and test sets."_  
_"However, the improvement in the model’s learning state on the test dataset is relatively limited, far from matching the model’s learning state on the training dataset"_  
_"As a result, the boundary between the robust and non-robust features becomes progressively more distinct between the training and test sets."_  
_"adversarial data with small loss indicates that the model’s learning state on these data is excellent, maintaining a substantial gap compared to the learning state on the test set."_  
I suggest you to provide more empirical results or theoretical results that support your claims and the effectiveness of the proposed method.

Author response:
**A1:** The non-effective features denote the robust features in the training set that lack generalization. The term "learning state gap" between the training and test sets has been revised to "robustness gap" in our updated manuscript. This change has been made consistently throughout the paper. Additionally, we employ quantitative metrics such as adversarial loss and robust accuracy to measure the model's robustness, providing evidential support for the existence of a robustness gap between the training and test datasets, as illustrated in Figure 1(b). Regarding the generation of non-effective features, our analysis is as follows: during the adversarial training process, the model's adversarial robustness on the training set exceeds that on the test set, as evidenced by the adversarial loss and robust accuracy observed in Figure 1(b). This results in a robustness gap between the training and test data. Considering that adversarial perturbations are generated on-the-fly and adaptively adjusted based on the model's robustness. The robustness gap between the training and test data leads to distinct adversarial perturbation on the robust features in these datasets. The varying degree of adversarial perturbation on the robust features in the training and test data amplify the distribution differences between these two datasets, thereby degrading the generalization of robust features on the training set and facilitating the generation of non-effective features. The increasing non-effective features further exacerbates the robustness gap between training and test data, forming a vicious cycle. These analyses are detailed in our updated manuscript presented in Section 1, Section 3, and Figure 2. We appreciate the valuable suggestion to refine our analysis, and we believe this adjustment enhances the clarity and rigour of our research.

Reviewer comment (Reproducibility):
The proposed method is not clearly written, and its explanation does not have equations or pseudo codes. Readers cannot reproduce the results. I could not understand how to control the attack strength in the proposed method and how to use data augmentation. What value of the loss do you call small loss for small-loss adversarial data? What is the specified threshold for the proposed method? Regarding changing adversarial budgets in adversarial training, [a] might be related work, which schedules adversarial budgets for considering loss landscapes.  

[a] Liu, Chen, et al. "On the loss landscape of adversarial training: Identifying challenges and how to overcome them." Advances in Neural Information Processing Systems 33 (2020): 21476-21487.

Author response:
**A2:** In the revised manuscript, we have incorporated the pseudocodes of the proposed methods in Algorithm 1 and Algorithm 2. We sincerely appreciate the valuable suggestion to refine our method, and we believe this adjustment enhances the logical flow of ideas.  

**A3:** In the revised manuscript, we have included detailed experimental hyperparameters in Table 3. We sincerely appreciate the valuable suggestion to refine our experiment, and we believe this adjustment enhances the clarity and rigor of our research.  

**A4:** Thank you for pointing out this related work (Liu et al., PAS), and we have cited it in the revised manuscript. Similar to our attack strength approach, PAS also adjusts the adversarial budgets. However, it is important to note that there are some key differences:
- **Different research problems.** PAS investigates the optimization challenges in adversarial training, specifically tackling the increased curvature and gradient scattering caused by the large adversarial budget. Instead, our focus is on the phenomenon of robust overfitting within adversarial training, specifically analyzing the onset of robust overfitting.
- **Different perspectives.** The main focus/contribution of our work lies in understanding the generation of non-effective features. The attack strength method is designed based on our understanding to specifically eliminate non-effective features. Instead, the design of PAS is motivated directly by the influence of the perturbation budget on loss landscape. Therefore, the two works clearly have different motivations and design principles.
- **Different solutions.** The two are also different in the strategies for adjusting perturbation budgets. PAS employs a periodic budget scheduling scheme where the perturbation budget dynamically changes along training and is applied to all samples. Instead, ours is much simpler and easy to use: the modified perturbation budget is constant and is applied solely to the small-loss adversarial samples.

Thus, according to these key differences, our method is quite different from PAS and provides new perspectives and solutions for understanding and alleviating robust overfitting.

Reviewer comment (Experiments):
Chapter 3 lacks experimental results to support analysis. For example, the study mentions the gap between training and test learning state several times, which would be better accompanied by a figure illustrating the difference of robust overfitting between training and test data.

Author response:
The term "learning state gap" between the training and test sets has been revised to "robustness gap" in our updated manuscript. This change has been made consistently throughout the paper. Additionally, we employ quantitative metrics such as adversarial loss and robust accuracy to measure the model's robustness, providing evidential support for the existence of a robustness gap between the training and test datasets, as illustrated in Figure 1(b). Regarding the generation of non-effective features, our analysis is as follows: during the adversarial training process, the model's adversarial robustness on the training set exceeds that on the test set, as evidenced by the adversarial loss and robust accuracy observed in Figure 1(b). This results in a robustness gap between the training and test data. Considering that adversarial perturbations are generated on-the-fly and adaptively adjusted based on the model's robustness. The robustness gap between the training and test data leads to distinct adversarial perturbation on the robust features in these datasets. The varying degree of adversarial perturbation on the robust features in the training and test data amplify the distribution differences between these two datasets, thereby degrading the generalization of robust features on the training set and facilitating the generation of non-effective features. The increasing non-effective features further exacerbates the robustness gap between training and test data, forming a vicious cycle. These analyses are detailed in our updated manuscript presented in Section 1, Section 3, and Figure 2. We appreciate the valuable suggestion to refine our analysis, and we believe this adjustment enhances the clarity and rigour of our research.

Reviewer comment (Presentation):
The writing of analysis and method is too lengthy, whereas the experiment part is too condensed. These sections have to be reorganized to alleviate the reading burden.

Author response:
In the revised manuscript, we have relocated the analysis of existing techniques for mitigating robust overfitting to the appendix to alleviate the reading burden. Additionally, we've incorporated the pseudocodes of the proposed methods in Algorithm 1 and Algorithm 2 to enhance the logical flow of ideas. Thank you for bringing this to our attention.

Reviewer comment (Experiments):
Experiments are solely conducted on CIFAR10 and CIFAR100. Larger datasets like ImageNet should be explored.

Author response:
Due to constraints on computational resources, training adversarially robust models on a large-scale dataset like ImageNet is currently beyond our capacity. However, to the best of my knowledge, the phenomenon of robust overfitting is primarily investigated on standard benchmark datasets such as CIFAR10.

Reviewer comment (Experiments):
Lack comparisons with existing mitigations of robust overfitting. The authors has included various previous works in the related works/revisiting section, which is good. However, there is no empirical comparison.

Author response:
Indeed, we have conducted experiments on two advanced methods for suppressing robust overfitting, namely AWP and MLCAT. It’s noteworthy that both AWP and MLCAT have already effectively mitigated robust overfitting. The proposed approaches still contribute to a complementary improvement in adversarial robustness. The enhanced adversarial robustness on AWP and MLCAT by our proposed methods further highlights the significance of understanding the underlying mechanisms of robust overfitting.

Reviewer comment (Novelty):
the main conclusion that overfitting is because of learning non-effective features is too trivial. It may be true for any type of overfitting but specifically suitable for adversarial training.

Author response:
It's worth noting that the main focus/contribution of our work lies in understanding the onset of robust overfitting through a detailed analysis of the generation of non-effective features, rather than drawing the conclusion that non-effective features cause robust overfitting. Regarding the generation of non-effective features, our analysis is as follows: during the adversarial training process, the model's adversarial robustness on the training set exceeds that on the test set, as evidenced by the adversarial loss and robust accuracy observed in Figure 1(b). This results in a robustness gap between the training and test data. Considering that adversarial perturbations are generated on-the-fly and adaptively adjusted based on the model's robustness. The robustness gap between the training and test data leads to distinct adversarial perturbation on the robust features in these datasets. The varying degree of adversarial perturbation on the robust features in the training and test data amplify the distribution differences between these two datasets, thereby degrading the generalization of robust features on the training set and facilitating the generation of non-effective features. The increasing non-effective features further exacerbates the robustness gap between training and test data, forming a vicious cycle. These analyses are detailed in our updated manuscript presented in Section 1, Section 3, and Figure 2. We appreciate the valuable suggestion to refine our analysis, and we believe this adjustment enhances the clarity and rigour of our research.

Reviewer comment (Theory):
as I said, it is good to design interesting experiments to find something. But still it is better to also include theoretical discussions, especially on the specific properties for adversarial training.

Author response:
In Section 3, we humbly offer the discussions on the generation of non-effective features during the adversarial training process: during adversarial training, adversarial perturbations are generated on-the-fly and adaptively adjusted based on the model’s robustness. In the initial stages of adversarial training, due to the similarity in the model's robustness between the training and test datasets, the generated adversarial perturbations on the robust features are relatively close in these datasets. Thus, most robust features in the training set can still exhibit good generalization, thereby enhancing the model's adversarial robustness on the test set. However, as training progresses, the model's robustness in the training dataset increases significantly faster than its robustness on the test dataset, as evidenced in Figure 1(b) by the adversarial loss and robustness observed on both datasets. This leads to a widening robustness gap between the training and test datasets. Consequently, the generated adversarial perturbation on the robust features becomes progressively more distinct in these datasets, which degrades the generalization of robust features on the training set and facilitates the generation of non-effective features, causing the model to learn an increasing number of non-effective features. As non-effective features proliferate, the robustness gap between the training and test sets continues to grow, forming a vicious cycle, as illustrated in Figure 2. Once the model's optimization is governed by these non-effective features, the model's adversarial robustness on the test dataset will continue to decline. This, in turn, gives rise to the phenomenon of robust overfitting.

Reviewer comment (Novelty):
the methods derived from the main conclusion is not interesting. Data augmentation is almost the most natural way to suppress overfitting. Attack strength adjustment is also common for adversarial training. For example. PGD-based on AT can be regarded as adaptive attack adjustment.

Author response:
We regret that the proposed methods did not capture your interest. In fact, our proposed methods are specifically crafted to validate the underlying mechanisms of robust overfitting, as outlined in Algorithm 1 and Algorithm 2. It is worth noting that, while data augmentation is almost the most natural way to suppress overfitting, applying the same data augmentation techniques in adversarial training, following the standard practice, proves ineffective in mitigating robust overfitting. Similarly, although attack strength adjustment is common for adversarial training, employing the same attack strength adjustment in accordance with conventional manner does not alleviate robust overfitting. In summary, while our methods may seem relatively simple and common, they are purposefully designed to validate our analysis of robust overfitting. Furthermore, they consistently enhance adversarial robustness across a wide range of experimental settings, demonstrating their efficacy.

Reviewer comment (Experiments):
It is OK if the authors choose to evaluate the proposed method numerically. However, the experiments should be enhanced largely. The performance should be compared not only to vanilla adversarial training but also other robust overfitting suppression method. Notice that the training time should be reported.

Author response:
Indeed, we have conducted experiments on two advanced methods for suppressing robust overfitting, namely AWP and MLCAT. It’s noteworthy that both AWP and MLCAT have already effectively mitigated robust overfitting. The proposed approaches still contribute to a complementary improvement in adversarial robustness. The enhanced adversarial robustness on AWP and MLCAT by our proposed methods further highlights the significance of understanding the underlying mechanisms of robust overfitting.

Reviewer comment (Theory):
The central discussions of the paper are vague and mainly intuitive in nature, with no concrete equations or an algorithm for the proposed methods based on attack strength and data augmentation. Furthermore, there is no analysis to support the claims about robust overfitting.

Author response:
**Q2:** No algorithm for the proposed methods.\
**A2:** In the revised manuscript, we have incorporated the pseudocodes of the proposed methods in Algorithm 1 and Algorithm 2. We sincerely appreciate the valuable suggestion to refine our method, and we believe this adjustment enhances the clarity of our research.

**Q3:** No analysis to support the claims about robust overfitting.\
**A3:** The term "learning state gap" between the training and test sets has been revised to "robustness gap" in our updated manuscript. This change has been made consistently throughout the paper. Additionally, we employ quantitative metrics such as adversarial loss and robust accuracy to measure the model's robustness, providing evidential support for the existence of a robustness gap between the training and test datasets, as illustrated in Figure 1(b). Regarding the process of how the robustness gap induces robust overfitting, our analysis is as follows: during the adversarial training process, the model's adversarial robustness on the training set exceeds that on the test set, as evidenced by the adversarial loss and robust accuracy observed in Figure 1(b). This results in a robustness gap between the training and test data. Considering that adversarial perturbations are generated on-the-fly and adaptively adjusted based on the model's robustness. The robustness gap between the training and test data leads to distinct adversarial perturbation on the robust features in these datasets. The varying degree of adversarial perturbation on the robust features in the training and test data amplify the distribution differences between these two datasets, thereby degrading the generalization of robust features on the training set and facilitating the generation of non-effective features. The increasing non-effective features further exacerbates the robustness gap between training and test data, forming a vicious cycle. These analyses are detailed in our updated manuscript presented in Section 1, Section 3, and Figure 2. We appreciate the valuable suggestion to refine our analysis, and we believe this adjustment enhances the clarity and rigour of our research.

Reviewer comment (Writing):
In Eqn (4), please clarify that the max is over the perturbation $\delta_i \in \Delta$, and that $x^\prime_i = x_i + \delta_i$.

Author response:
**Q5:** The error in Eqn(4).\
**A5:** We have rectified the error in Eqn (4) in the revised manuscript. Thank you for bringing it to our attention.

Reviewer comment (Experiments):
For the factor ablation experiments in Section 3.1 and Figure 1, clarify whether the results are averaged over a few trials to account for randomness.

Author response:
**Q1:** Factor ablation experiments over a few trials.\
**A1:** The results of factor ablation experiments in Figure 1 show the outcomes of a single run. However, it is worth noting that we have actually presented the results of these factor ablation experiments across different datasets, network architectures, and adversarial training variants, as illustrated in Figure 3. This broader presentation suggests that the observed phenomenon is a general finding in adversarial training.

Reviewer comment (Writing):
Provide a concrete definition of "effective features" and the idea of "similarity of a model's learning states between the training and test data."

Author response:
**Q4:** The definition of "effective features".\
**A4:** The term "effective features" has been deprecated in our revised manuscript. We now explicitly denote them as robust features in the training dataset with generalization. For instance, "In the initial stages of adversarial training, due to the similarity in the model's robustness between the training and test datasets, the generated adversarial perturbation on the robust features is relatively close in these datasets. Thus, most robust features in the training set can still exhibit good generalization, thereby enhancing the model's adversarial robustness on the test set." We appreciate the feedback and have made the necessary adjustments to improve clarity.

Reviewer comment (Theory):
The theoretical analysis relies on intuitive reasoning. Formalizing the notions of robust/non-robust features and quantifying the learning state gap could strengthen the conceptual framing.

Author response:
The term "learning state gap" between the training and test sets has been revised to "robustness gap" in our updated manuscript. This change has been made consistently throughout the paper. Additionally, we employ quantitative metrics such as adversarial loss and robust accuracy to measure the model's robustness, providing evidential support for the existence of a robustness gap between the training and test datasets, as illustrated in Figure 1(b). Regarding the process of how the robustness gap induces robust overfitting, our analysis is as follows: during the adversarial training process, the model's adversarial robustness on the training set exceeds that on the test set, as evidenced by the adversarial loss and robust accuracy observed in Figure 1(b). This results in a robustness gap between the training and test data. Considering that adversarial perturbations are generated on-the-fly and adaptively adjusted based on the model's robustness. The robustness gap between the training and test data leads to distinct adversarial perturbation on the robust features in these datasets. The varying degree of adversarial perturbation on the robust features in the training and test data amplify the distribution differences between these two datasets, thereby degrading the generalization of robust features on the training set and facilitating the generation of non-effective features. The increasing non-effective features further exacerbates the robustness gap between training and test data, forming a vicious cycle. These analyses are detailed in our updated manuscript presented in Section 1, Section 3, and Figure 2. We appreciate the valuable suggestion to refine our analysis, and we believe this adjustment enhances the clarity and rigour of our research.

Reviewer comment (Theory):
The analysis relies on the notion of a "gap" between training and test learning states. Is there a principled way to quantify this gap? Are there any theoretical bounds on the gap size that induces robust overfitting?

Author response:
The term "learning state gap" between the training and test sets has been revised to "robustness gap" in our updated manuscript. This change has been made consistently throughout the paper. Additionally, we employ quantitative metrics such as adversarial loss and robust accuracy to measure the model's robustness, providing evidential support for the existence of a robustness gap between the training and test datasets, as illustrated in Figure 1(b). Regarding the process of how the robustness gap induces robust overfitting, our analysis is as follows: during the adversarial training process, the model's adversarial robustness on the training set exceeds that on the test set, as evidenced by the adversarial loss and robust accuracy observed in Figure 1(b). This results in a robustness gap between the training and test data. Considering that adversarial perturbations are generated on-the-fly and adaptively adjusted based on the model's robustness. The robustness gap between the training and test data leads to distinct adversarial perturbation on the robust features in these datasets. The varying degree of adversarial perturbation on the robust features in the training and test data amplify the distribution differences between these two datasets, thereby degrading the generalization of robust features on the training set and facilitating the generation of non-effective features. The increasing non-effective features further exacerbates the robustness gap between training and test data, forming a vicious cycle. These analyses are detailed in our updated manuscript presented in Section 1, Section 3, and Figure 2. We appreciate the valuable suggestion to refine our analysis, and we believe this adjustment enhances the clarity and rigour of our research.

Reviewer comment (Experiments):
Have you experimented with more advanced data augmentation techniques like MixUp or CutMix? Could these further help with state alignment and reducing non-effective features?

Author response:
Indeed, we conducted experiments with four different data augmentation techniques, as shown in Figure 1(d) and Figure 6, all of which demonstrated similar effects on robust overfitting. This consistency arises from the fact that data augmentation techniques are applied with a targeted transformation objective to modify the proportion of small-loss adversarial data, as outlined in Algorithm 2. Consequently, they inherently achieve the effect of robustness alignment, and the specific data augmentation techniques employed do not significantly impact their effectiveness.