Paper: Why Clean Generalization and Robust Overfitting Both Happen in Adversarial Training (ICLR-2024)

Reviewer comment (Theory):
The assumption of the patch data is somewhat weird to me. The realistic images are much more complicated and cannot be simplified as a patch structual. Specifically, the authors assume $\alpha=d^{0.249}$ and $\sigma^{-0.509}$. I am puzzled as to why they chose specific values of $0.249$ and $-0.509$? Does the real data distribution satisfy this assumption?

Author response:
We thank the reviewer of the insightful feedback. We address the reviewer’s concerns as below, and we are happy to clarify any follow-up questions/confusions.

**Question**  The assumption of the patch data is somewhat weird to me. The realistic images are much more complicated and cannot be simplified as a patch structual. Specifically, the authors assume $\alpha=d^{0.249}$ and $\sigma^{-0.509}$. I am puzzled as to why they chose specific values of 0.249 and -0.509 ? Does the real data distribution satisfy this assumption?

**Answer** Indeed, under our theoretical framework, the meaningful signal patch has the form $X[signal(X)] = \alpha y w^{\*}$ and the noise patch has the form $X[j] \sim N(0,(I_d-w^{\*}w^{\*T})\sigma^2)$, so the norm of meaningful signal patch is \alpha and the norm of noise patch w.h.p. is $\Theta(\sigma\sqrt{d-1})$, which enables meaningful signal is stronger than noise when we set $\alpha = d^{0.249}polylog(d)$, $\sigma = d^{-0.509}$. And if the data cannot satisfy this assumption, in other word, if the signal-to-noise ratio is small, then there even exists no clean generalization, which has been theoretically shown in the previous work [1][2][3] under the similar patch-structured setting. In this paper, we aim to provide a theoretical understanding of CGRO phenomenon in adversarial training, thereby the assumption that meaningful signal is stronger than random noise is reasonable and necessary. We will add some discussion to clarify this point about the signal and noise in the revision of paper. Thanks for the reviewer’s suggestion.

[1] Towards understanding how momentum improves generalization in deep learning.


[2] Benign Overfitting in Two-layer Convolutional Neural Networks.


[3] Benign Overfitting in Two-layer ReLU Convolutional Neural Networks.

Reviewer comment (Theory):
For the patch dataset, why is $\alpha$ assumed to be proportional to $d^{0.249}$? Where is the number $0.249$ from? Similarly, where is the number $-0.509$ from in the assumption about $\sigma$?

Author response:
Indeed, under our theoretical framework, the meaningful signal patch has the form $X[signal(X)] = \alpha y w^{\*}$ and the noise patch has the form $X[j] \sim N(0,(I_d-w^{\*}w^{\*T})\sigma^2)$, so the norm of meaningful signal patch is \alpha and the norm of noise patch w.h.p. is $\Theta(\sigma\sqrt{d-1})$, which enables meaningful signal is stronger than noise when we set $\alpha = d^{0.249}polylog(d)$, $\sigma = d^{-0.509}$. And if the data cannot satisfy this assumption, in other word, if the signal-to-noise ratio is small, then there even exists no clean generalization, which has been theoretically shown in the previous work [1][2][3] under the similar patch-structured setting. In this paper, we aim to provide a theoretical understanding of CGRO phenomenon in adversarial training, thereby the assumption that meaningful signal is stronger than random noise is reasonable and necessary. We will add some discussion to clarify this point about the signal and noise in the revision of paper. Thanks for the reviewer’s suggestion.

Reviewer comment (Theory):
The sentence above Section 4 reads, "Intuitively, when the model $f_W$ has achieve (typo here, should be achieved) mid-high clean test accuracy, the decision boundary of W will have a significant correlation with the separating plane of $g$, which thus makes adversarial examples generated by $g$ transferable." Is there any justifications or source for this statement?

Author response:
Theoretically, traditional formulation of adversarial training algorithms is a highly non-convex and non-concave min-max optimization, under which, in general, analyzing global convergence is NP-hard problem [4]. To overcome this hardness barrier, we leverage geometry-inspired attack as adversarial training method, under which we propose a non-trivial three-stage analysis technique to decouple the complicated feature learning process as follows.

 **Phase I**: At the beginning, the signal component of lottery tickets winner $max_{r\in [m]}u_{r}^{(t)}$ increases quadratically (Lemma 4.2). At this point, the model starts to learn partial true feature.

 **Phase II**: Once the maximum signal component $max_{r\in [m]}u_{r}^{(t)}$ attains the order $\tilde{\Omega}(\alpha^{-1})$, the growth of signal component nearly stop updating since that the increment of signal component is now mostly dominated by the noise component (Lemma 4.3).

 **Phase III**: After that, by the quadratic increment of noise component (Lemma 4.4), the total noise $V_{i}^{(t)}$ eventually attains the order $\Omega(1)$, which implies the model memorizes the spurious feature (data-wise noise) in final.

 **On the empirical side**, it is verified that it has a comparable performance with classical adversarial training methods such as FGSM and PGD attacks (e.g., [5][6]). Understand the feature learning process of FGSM and PGD attacks is an existing future direction.



[1] Towards understanding how momentum improves generalization in deep learning.  
[2] Benign Overfitting in Two-layer Convolutional Neural Networks.  
[3] Benign Overfitting in Two-layer ReLU Convolutional Neural Networks.  


[4] Some NP-complete problems in quadratic and nonlinear programming.  

[5] Deepfool: a simple and accurate method to fool deep neural networks.  

[6] Geometry-inspired top-k adversarial perturbations.

Reviewer comment (Writing):
The first sentence of Section 4.1 reads, "which is widely applied in theoretical works explore what and how neural networks learn in different tasks." Check grammar.

Author response:
this sentence should be “which is widely applied in theoretical works that study what and how neural networks learn in different tasks”.

Reviewer comment (Theory):
Major Concern 1: Robust Optimization overlooked. The reviewer finds it hard to agree with the motivation of this paper, especially for the "clean generalization" part. Training a model adversarially but without attacks in the test phase corresponds to traditional robust optimization. In robust optimization, uncertainty sets are built around noisy observed features, and the test phase risk is with respect to the true data. Earlier papers have already established the clean generalization result — optimizing the worst-case during training does not produce solutions significantly away from the nominal solutions, and robust optimization can even improve out-of-sample performance. The reviewer would have preferred further discussion on this connection.

Author response:
while some works (Tsipras et al., 2018; Zhang et al., 2019; Hassani & Javanmard, 2022) point out that achieving robustness may hurt clean test accuracy, in most of the cases, it is observed that drop of robust test accuracy is much higher than drop of clean test accuracy in adversarial training (Madry et al., 2017; Schmidt et al., 2018; Raghunathan et al., 2019) (see in Figure 1, where clean test accuracy is more than 80% but robust test accuracy only attains nearly 50%). Namely, a weak version of benign overfitting (Zhang et al., 2017), which means that overparameterized deep neural networks can both fit random data powerfully and generalize well for unseen clean data, remains after adversarial training. Therefore, it is natural to ask the following question: What is the underlying mechanism that results in both Clean Generalization and Robust Overfitting (CGRO) during adversarial training? In this paper, We provide a theoretical understanding why clean generalization and robust overfitting both happen in adversarial training.

Reviewer comment (Writing):
Major Concern 2: The paper is not written well, and includes several typos and errors. The reviewer will list some examples in the Questions section, but in their view the paper is not ready yet, and they had to read many times before understanding most of the claims.

Author response:
Thank the reviewer for pointing out the typos in this paper, we will fix these in the revision of our paper.

Reviewer comment (Writing):
Major Concern 3: The setting is very specific, and conclusions are general. The paper introduces assumptions, parameterizations, architectures, and dataset structures without discussing why. Definitions are ad-hoc, and there is an implicit assumption that the reader has deep understanding of adversarial robustness, feature learning theory, and CNNs. There is no introduction to key concepts like signals, patches, and spurious feature learning, making it hard for readers unfamiliar with these areas to follow. The main message is about robust overfitting and clean generalization, but the image-data-related work presented is just a special case; a clearer and more inclusive language is recommended.

Author response:
Theoretically, traditional formulation of adversarial training algorithms is a highly non-convex and non-concave min-max optimization, under which, in general, analyzing global convergence is NP-hard problem [1]. To overcome this hardness barrier, we leverage geometry-inspired attack as adversarial training method, under which we propose a non-trivial three-stage analysis technique to decouple the complicated feature learning process as follows.

 **Phase I**: At the beginning, the signal component of lottery tickets winner $max_{r\in [m]}u_{r}^{(t)}$ increases quadratically (Lemma 4.2). At this point, the model starts to learn partial true feature.

 **Phase II**: Once the maximum signal component $max_{r\in [m]}u_{r}^{(t)}$ attains the order $\tilde{\Omega}(\alpha^{-1})$, the growth of signal component nearly stop updating since that the increment of signal component is now mostly dominated by the noise component (Lemma 4.3).

 **Phase III**: After that, by the quadratic increment of noise component (Lemma 4.4), the total noise $V_{i}^{(t)}$ eventually attains the order $\Omega(1)$, which implies the model memorizes the spurious feature (data-wise noise) in final.

 On the empirical side, it is verified that it has a comparable performance with classical adversarial training methods such as FGSM and PGD attacks (e.g., [2][3]). Understand the feature learning process of FGSM and PGD attacks is an existing future direction.

In all, we present a theoretical understanding of clean generalization and robust overfitting (CGRO) phenomenon in adversarial training. Our main contribution is that, under our theoretical framework, we prove that neural network trained by adversarial training partially learns the true feature but memorizes the random noise in training data, which leads to CGRO phenomenon. In all, we believe that our work provides some theoretical insights into existing adversarial training methods. An important future work is to generalize our analysis of feature learning process to deep CNN models with other adversarial-example generative algorithms, such as FGSM and PGD attacks. 

[1] Some NP-complete problems in quadratic and nonlinear programming.

[2] Deepfool: a simple and accurate method to fool deep neural networks.

[3] Geometry-inspired top-k adversarial perturbations.

Reviewer comment (Theory):
The theoretical results focus on the cubic activation function. Do the results hold for other polynomial activation functions?

Author response:
Different from some previous work analyzing the behavior of linear classifier in adversarial training, we improve the expressive power of model by using non-linear cubic activation that can characterize the data structure, in order that we can analyze learning process more precisely. Moreover, under our theoretical framework, the results on feature learning process still hold for more general activation function $z^{q}$, where q is arbitrary odd number at least three to balance the positive-labeled data and negative-labeled data.

Reviewer comment (Theory):
Parameterization 3.1 requires the adversarial training data to be created for the optimal linear model with \( w^* \) at every training iteration. Do the theoretical results remain true if the adversarial training data are optimized for the model at the current iteration \( g(x)=\langle w^{(t)} , x\rangle \)?

Author response:
Theoretically, traditional formulation of adversarial training algorithms is a highly non-convex and non-concave min-max optimization, under which, in general, analyzing global convergence is NP-hard problem [1]. To overcome this hardness barrier, we leverage geometry-inspired attack as adversarial training method, under which we propose a non-trivial three-stage analysis technique to decouple the complicated feature learning process as follows.

 **Phase I**: At the beginning, the signal component of lottery tickets winner $max_{r\in [m]}u_{r}^{(t)}$ increases quadratically (Lemma 4.2). At this point, the model starts to learn partial true feature.

 **Phase II**: Once the maximum signal component $max_{r\in [m]}u_{r}^{(t)}$ attains the order $\tilde{\Omega}(\alpha^{-1})$, the growth of signal component nearly stop updating since that the increment of signal component is now mostly dominated by the noise component (Lemma 4.3).

 **Phase III**: After that, by the quadratic increment of noise component (Lemma 4.4), the total noise $V_{i}^{(t)}$ eventually attains the order $\Omega(1)$, which implies the model memorizes the spurious feature (data-wise noise) in final.

 **On the empirical side**, it is verified that it has a comparable performance with classical adversarial training methods such as FGSM and PGD attacks (e.g., [2][3]). Understand the feature learning process of FGSM and PGD attacks is an existing future direction.

[1] Some NP-complete problems in quadratic and nonlinear programming.

[2] Deepfool: a simple and accurate method to fool deep neural networks.

[3] Geometry-inspired top-k adversarial perturbations.