From: ICLR.cc/2023/Conference/Program_Chairs
title: Paper Decision
decision: Reject
metareview:_summary,_strengths_and_weaknesses: Based on the collected information from all reviewers and my personal judgment, I can make the recommendation on this paper, **rejection**. Here are the comments that I summarized, which include my opinion and evidence.

**Research Problem**

The authors study the catastrophic overfitting problem, that classifier decision boundaries are highly distorted, and robust accuracy against iterative-step adversarial attacks suddenly drops from peak to nearly 0% in a few epochs. Specifically, the authors noticed some adversarial examples generated on the network with distorted decision boundaries exhibit anomalous behavior, which co-occurs with catastrophic overfitting. In light of this, the authors aim to prevent catastrophic overfitting by suppressing generated abnormal adversarial examples and designing a regularizer. 

**Motivation**

Personally, I like that the motivation is derived from some empirical findings. 

**Philosophy**

In general, the whole story makes sense to me. This paper starts from the conjecture that catastrophic overfitting might result from abnormal adversarial examples. Based on that, the authors design a regularizer to suppress the generation of abnormal adversarial examples.

**Presentation**

The authors need to improve the presentation greatly. Several reviewers have many clarification questions. Moreover, the authors can define notations for Eqs. (8)&(9) and use them in Eq. (10). No legends in Figure 3 (left and middle). The yellow line in the left figure means the number of AAER or the number of AAE? 

**Technique**

The novelty of this paper is the proposed Abnormal Adversarial Examples Regularization. 

**Experiments**

(1) The experimental results are extensive. 

(2) To increase the readability, Table 1 can be separated into two tables 

(3) Since the novelty of this paper lies in the regularizer term, the authors are suggested to compare it with other regularizers for tackling catastrophic overfitting. More details can be found in Reviewer gJHh’s response.

(4) Several reviewers have concerns about the unclipped perturbation. I suggest the authors add extra experiments by employing the proposed regularizer on other min-max optimization procedures to demonstrate its effectiveness. 

(5) The authors need to demonstrate the proposed regularizer can effectively reduce the number of abnormal adversarial examples. More results are expected beyond Figure 3. 
justification_for_why_not_higher_score: Although all the reviewers and I do not have a firm reason to reject this paper, the presentation must be improved (see my above comments) and the experiments can be more richer by comparing other other regularizers for tackling catastrophic overfitting. 

In our zoom discussion, we temporarily put this paper as borderline. In my batch, there is another paper which is better than this one. So I currently give the rejection recommendation and would like SAC to make the final decision according to the acceptance rate.
justification_for_why_not_lower_score: N/A
summary_of_AC-reviewer_meeting: We had the zoom discussion on Paper 656 with three reviewers. The fourth one cannot attend, but sent her/his comments in advance. I am very appreciative of the reviewers' effort to make this happen, especially their different time zones. Reviewer gJHh also demonstrated his reproduced results. The discussion was informative. Everyone agrees that this paper is borderline, which has a clear motivation, philosophy and technical contribution, and also pointed out several extra points (presentation, additional competition methods) to make this paper stronger. In conclusion, we temporarily put this paper as borderline and will accept or reject upon the comparisons of other high-quality papers and the acceptance rate.

From: ICLR.cc/2023/Conference/Paper656/Authors
title: Response to Reviewer gJHh 
comment: We sincerely thank you for the comprehensive comments on our paper and please find the answers to your questions below.

**Q1:** Unfair comparison

**A1:** N-FGSM has made two improvements based on RS-FGSM: (1) use $(-k\cdot\epsilon,k\cdot\epsilon)$ for random initialization to obtain larger analytical upper bound to avoid CO, (2) use the unclipped perturbation setting which does not use $\max (\min (\delta, \epsilon),-\epsilon)$ in training to improve robustness. Therefore, we will also analyze the difference between AAER and N-FGSM from CO and robustness. 

We first want to explain that AAER does not rely on $(-k\cdot\epsilon,k\cdot\epsilon)$ for random initialization to avoid CO, at the same time N-FGSM cannot guarantee to defend against CO in some extreme settings which can break the upper bound (e.g., for CIFAR-100 with 32/255 noise magnitude). Therefore, our AAER effect eliminates CO independent of the N-FGSM. 

Then we will analyze the impact of the unclipped perturbation setting on the robustness, given that the unclipped perturbation setting can effectively improve the robustness of N-FGSM and AAER. For a fair comparison, we also add the unclipped perturbation setting for the other methods, which is the same setting as $AAER_{RUC}$. The results of Align Grad, TRADES and PGD-10 on CIFAR10 are shown in the table below, where the natural and robust accuracy is formatted as x/**y**. For PGD, the unclipped perturbation setting only improves the robustness with large noise magnitude, but reduces the robustness with small noise magnitudes. For the regularization method TRADES, the unclipped perturbation setting reduces robust accuracy for all noise magnitudes. For the regularization method Grad Align, the unclipped perturbation setting has a small improvement of robust accuracy (from 47.77 to 48.00 with 8/255 noise magnitude), but will lead to training failure with the large perturbation (we have tried to roughly tune the hyperparameters). Therefore, combining regularization methods with an unclipped perturbation setting does not always seem to lead to a significant increase in robustness. 

|Method|8/255|12/255|16/255|32/255|Method|8/255|12/255|16/255|32/255|
|:-----|:----:|:----:|:----:|:----:|:-----|:----:|:----:|:----:|:----:|
|$AAER_{RC}$|84.33/**46.42**|76.07/**32.94**|65.88/**24.67**|26.14/**14.70**|$AAER_{RUC}$|77.41/**51.51**|66.24/**38.66**|55.32/**30.31**|20.57/**12.72**|
|$Grad\ Align$|82.10/**47.77**|74.17/**34.87**|60.37/**27.90**|25.23/**11.53**|$Grad\ Align$ with unclipped perturbation|78.70/**48.00**|68.50/**36.60** |10.30/**10.30**|10.30/**10.30**|
|$TRADES$|78.98/**50.19**|73.62/**32.92**|65.09/**25.67**|48.84/**5.50**|$TRADES$ with unclipped perturbation|65.72/**43.80**|54.12/**25.92**|52.94/**11.74**|55.65/**2.56**|
|$PGD-10(2/255)$|80.55/**50.67**|72.37/**38.60**|67.20/**29.34**|59.35/**5.80**|$PGD-10(2/255)$ with unclipped perturbation|52.87/**43.19**|52.34/**37.85**|52.26/**32.06**|50.28/**10.7**|
|$PGD-10(\epsilon/10)$|84.74/**46.06**|78.31/**34.13**|71.19/**26.07**|40.51/**15.16**|$PGD-10(\epsilon/10)$ with unclipped perturbation|80.12/**49.83**|70.61/**39.05**|61.18/**31.72**|28.91/**16.04**|

We also noticed that the reviewer mentioned that N-FGSM + ALP can achieve comparable results to $AAER_{RUC}$. Therefore, we also try to add TRADES ($\beta$ =6.0) and ALP (logit pairing weight=0.5) loss function to N-FGSM. However, we observed that after adding the regularization term, the robustness of N-FGSM decreases with all noise magnitude, and CO will occur with 32/255 noise magnitude. Therefore, it seems that combining N-FGSM with regularization term does not always improve robustness. The difference between our experimental results and the reviewer mentioned may be due to different specific settings. If the reviewer is willing to provide a specific implementation, we are also very willing to discuss and do more experiments.

|Method|8/255|12/255|16/255|32/255|
|:-----|:----:|:----:|:----:|:----:|
|N-FGSM|80.48/**47.91**|71.30/**36.23**|62.96/**27.14**|32.44/**10.58**|
|N-FGSM + TRADES|81.38/**43.40**|78.65/**23.57**|87.51/**9.04**|65.84/**0.94**|
|N-FGSM + ALP|84.60/**46.00**|83.01/**30.20**|67.02/**23.49**|83.31/**0.01**|

Based on the above experiments, it seems that only the AAER method can achieve a significant improvement in robustness with the unclipped perturbation setting. We speculate that the reason AAER can achieve better robustness is AAER can hinder the generation of abnormal adversarial examples, which makes all generated perturbations (normal adversarial examples) beneficial to robustness. Under the premise of ensuring that almost the generated perturbations are normal adversarial examples, the unclipped setting allows perturbations to retain more features, which allows the network to learn for better robustness.

From: ICLR.cc/2023/Conference/Paper656/Authors
title: Response to Reviewer gJHh  
comment: **Q2:** Lack of learning curves

**A2:** Since revision submission is closed and markdown does not allow image submission, we will show the learning curves of AAER on CIFAR10 with 16/255 noise magnitude in the table below, where the FGSM and PGD-7-1 attack accuracy is formatted as x/**y**. Since our experiment uses linearly increasing $\alpha$, which will lead to less robustness at the beginning of training. However, our method keeps the robustness curve on an upward trend, and achieves high robustness at the end of the training.


|Epoch|0|1|2|3|4|
| :-----| :----: | :----: | :----:| :----:| :----:|
|$AAER_{RC}$ with linearly increasing $\alpha$|2.04/**0.66**|3.05/**0.52**|3.73/**0.72**|7.27/**2.36**|9.64/**3.49**|
|Epoch|5|6|7|8|9|
|$AAER_{RC}$ with linearly increasing $\alpha$|10.92/**4.90**|14.41/**7.30**|15.29/**8.37**|16.96/**10.55**|19.30/**12.91**|
|Epoch|10|11|12|13|14|
|$AAER_{RC}$ with linearly increasing $\alpha$|22.49/**15.27**|23.95/**15.61**|24.56/**17.08**|24.20/**17.92**|22.87/**16.63**|
|Epoch|15|16|17|18|19|
|$AAER_{RC}$ with linearly increasing $\alpha$|24.79/**18.09**|27.18/**20.82**|25.29/**18.98**|28.75/**22.40**|27.85/**22.05**|
|Epoch|20|21|22|23|24|
|$AAER_{RC}$ with linearly increasing $\alpha$|28.01/**21.60**|27.45/**22.49**|30.09/**24.75**|29.98/**24.34**|27.16/**23.76**|
|Epoch|25|26|27|28|29|
|$AAER_{RC}$ with linearly increasing $\alpha$|32.06/**26.65**|32.45/**27.15**|31.14/**26.38**|34.91/**29.08**|35.88/**29.47**|

**Q3:** Presentation could be improved

**A3:** We will proofread the manuscript and ensure the description clearer and avoid misinterpretations in the revision.

From: ICLR.cc/2023/Conference/Paper656/Reviewer_gJHh
title: Thanks for your reply. 
comment: I have read the rebuttal and as well as the other reviews and responses. The authors have clarified some of my previous comments. However, I still have some major concerns.

First, the target of this paper is to mitigate the catastrophic overfitting (CO) under FGSM-based adversarial training. Therefore, the authors introduce a new regularizer, called AAER. In related work, the authors highlight many regularization techniques, but they only compare their AAER with Grad Align. According to the results in Table 1, $\mathrm{AAER}_{\mathrm{RC}}$ is not always better than Grad Align. $\mathrm{AAER}_{\mathrm{RUC}}$ who uses the unclipped perturbation achieves the best results. I think $\mathrm{AAER}_{\mathrm{RUC}}$ is a combination method (N-FGSM + AAER), in which the performance gain is thankful to the N-FGSM. 

I have checked the paper for N-FGSM. And I find that N-FGSM can solve the CO problem well. The authors use a similar technique (like unclipped perturbation) to further enhance the robustness, where the AAER serves as a regularizer that can be integrated into the N-FGSM based training.  It is well-known that there are many regularization techniques for enhancing robustness. So, it’s better to compare the proposed AAER with other typical regularizers, like ALP [1] and TRADES [2], under the setting of N-FGSM. This is because N-FGSM is a simple but effective method for mitigating CO. I quickly confirmed that N-FGSM + ALP can achieve comparable results to $\mathrm{AAER}_{\mathrm{RUC}}$ without any CO phenomenon. If AAER is not the main contribution to the performance gain, I think the AAER is simply a method for eliminating CO. And the authors seem to be overstating their contribution.

For a fair comparison, I suggest that the authors add further experiments on the regularization techniques as mentioned in the related work. If the authors consider the unclipped perturbations, one should also consider combing N-FGSM with a typical robust regularizer. 

Second, I prompt the authors to draw the learning curves that I highlighted in the review comments, but the authors are unwilling to show these results. I think that learning curves can directly reflect progress during training. I do not think that Figure 3 directly shows trends in training losses and test losses. 

Third, the presentation needs to be further improvement. 

Overall, the manuscript needs significant improvement, although both the motivation and the preliminary observations are OK. 

[1] Adversarial logit paring, arxiv 2018.
[2] Theoretically Principled Trade-off between Robustness and Accuracy, ICML 2019.

From: ICLR.cc/2023/Conference/Paper656/Authors
title: Response to Reviewer DBF4
comment: We sincerely thank you for the comprehensive comments on our paper and please find the answers to your questions below.

**Q1:** Explanation on Eq.9.

**A1:** We use the logits variation of normal adversarial examples as the standard, which should be fixed and unchanged (This is like the effect of using .detach, but it seems that .detach has no effect in our case). But, due to technical limitations we cannot keep the second term and Eq.9 indeed allow just increase the logit variation of normal adversarial examples until the objective hits zero, so a clearer statement should be "to avoid the network only focusing on increasing the normal ones logits variation, we use the max function to limit the minimum value to 0".

**Q2:** The relationship between CO and abnormal adversarial examples.

**A2:**  Abnormal adversarial examples are an effect of the distorted decision boundary. At the same time, CO is also an effect of the highly distorted decision boundary. So, abnormal adversarial examples and CO are a correlation rather than causation. In order to clarify the relationship between them, we added their relationship in the ablation study and clearly show that they are not causal.

**Q3:** confused sentence

**A3:** Your understanding of this sentence is correct, we have significantly clarified this sentence. The clearer statement should be "Since the logits variation is a representation of the change magnitude, which is not related to the label, there is no clear target value for the optimization standard".

**Q4:** related work

**A4:** We do cite N-FGSM and Align Grad as de Jorge et al. (2022) and Andriushchenko & Flammarion (2020), respectively. Athalye et al., 2018 showed that AT is the most effective method for defending against adversarial attacks, instead of proposing the min-max formulation (we now cite Madry to avoid this unclear description).

**Q5:** formatting aspects

**A5:** We have enlarged the text in all figures and fixed typos.

From: ICLR.cc/2023/Conference/Paper656/Reviewer_DBF4
title: Thanks for your reply
comment: I have read the authors' rebuttal as well as the other reviews and responses. I have raised my score to 6 as the authors have answered most of my questions, but still have a few remaining concerns:

- Like reviewer 5TBo, I am still a bit confused about the objective (9). In particular, the authors state that "In order to clarify that the network optimization objective is to reduce the logits variation of abnormal adversarial samples instead of increasing the normal ones logits variation, we use the max function to limit the minimum value to 0," but I do not see how this claim follows---can't the network just increase the logit variation of normal adversarial examples until the objective hits zero? I hope the authors can either remove or justify this claim
  - In particular, my guess would be that it's not the clipping but rather the cross-entropy loss term that stops the classifier from increasing the logit variation of normal adversarial examples
  - I think a more reasonable algorithm would be to keep the second term but prevent gradients from going through it (e.g., with ``.detach`` in PyTorch)---that way the network has no choice but to reduce the variation of abnormal examples to match the normal ones. Did the authors try this?

- In general, the writing can still be made much more precise. One example is in my point above, but another example is "we design a novel method ... prevents CO by suppressing generated abnormal adversarial examples"---based on the provided answer to Q1 this doesn't seem quite true, since the abnormal adversarial examples are not the cause of CO, but rather an effect? The authors should go through the paper to make this fact clear as there are many sentences (such as the one I referenced) that make it seem like AAEs are a cause of CO.

- I think I now understand this sentence, but it is very confusing: "unlike the prediction confidence variation, the logits variation does not have a clear evaluation standard as an optimization objective that only has different magnitudes." Do the authors mean to say that unlike the prediction confidence where smaller is clearly better, there is no clear target value for what the logit variation "should be" and so we need to use the normal adversarial examples as a baseline? If so, the sentence can be significantly clarified.

- [Minor] The related work in general is quite poorly written, both in terms of discussing the breadth of related work (e.g., N-FGSM is not mentioned despite being the closest in performance) as well as properly describing the contributions of the related works discussed (e.g., I believe the min-max formulation in Eq. (1) is actually due to Madry et al?)---I would recommend the authors look at paper [2] cited in their rebuttal as a reference for a related work section.

- [Minor] Various formatting aspects could be improved. Two examples below, but the paper would benefit from a proofread.
  - The text in almost all the figures is too small to be visible
  - BaseLines -> Baselines

From: ICLR.cc/2023/Conference/Paper656/Authors
title: Response to Reviewer 5TBo
comment: We sincerely thank you for the comprehensive comments on our paper and please find the answers to your questions below.

For Q1, we also found abnormal adversarial examples in PGD-10 AT, and the number of abnormal adversarial examples was 3504 at the end of training (30 epoch). Therefore, abnormal adversarial examples are not the unique phenomenon of single-step AT. At the same time, PGD-10 AT will not cause CO, because the number of abnormal adversarial examples is relatively small and not enough to cause CO. It is worth noting that at some extreme settings PGD-2 will generate a large number of abnormal adversarial examples and CO will occur.

For Q2, this is not an arbitrary choice. We use the logits variation of normal adversarial samples as the standard and want to limit the abnormal ones. Therefore, we hope that the difference between the logits variation of abnormal and normal adversarial samples is 0, which represents logits variation of these two types of samples is the same. Therefore, if the logits variation of abnormal adversarial samples is always greater than normal ones, we can only use subtraction without the max function. But the network will increase the logits variation of normal adversarial examples to produce a very large negative loss value, which is not the optimization objective we expect. Therefore, we use the max function to clarify that the network optimization objective is to reduce the logits variation of abnormal adversarial examples rather than increase the logits variation of normal adversarial examples.

For Q3, unclipped perturbation means that we do not use $\delta=\max (\min (\delta, \epsilon),-\epsilon)$ in training. However, we want to emphasize that this leads to more reasonable perturbation, and since the perturbation in single-step AT has a smaller step size and is only generated once, this perturbation does not go out of bounds by much.

For Q4, we did not combine our method with Align Grad due to computational resource constraints, but good robustness can be achieved with our method alone.

From: ICLR.cc/2023/Conference/Paper656/Reviewer_5TBo
title: The response does not add clarity to my questions
comment: I thank the authors for preparing the responses. However, I am disappointed to find that the responses did not address my concerns, and nor does the submission to be updated to have better clarity.

For Q1, I've made it very clear how to make a more convincing and motivating example, but the authors seem to ignore my questions and suggestions.

For Q2, it is still unclear why using 0 as a threshold is a good choice for confining logit variation. This seems to be a very arbitrary choice (e.g., one can instead use 1) without any physical meanings.

For Q3, so what is unclipped perturbation exactly? I was expecting a clear and direct mathematical definition, instead of pointing me to another paper (and not answering my question about whether the perturbation goes out of range - from the response I think the answer is yes)

For Q4, since no results are added, I don't see any new insights.

From: ICLR.cc/2023/Conference/Paper656/Authors
title: Response to all Reviewers
comment: We greatly appreciate your time and effort in reviewing our manuscript. Just a reminder that we have uploaded the rebuttal revision and explained your concerns. Let us know if you have any other concerns, we look forward to your feedback.

From: ICLR.cc/2023/Conference/Paper656/Authors
title: Response to Reviewer gJHh
comment: We sincerely thank you for the comprehensive comments on our paper and please find the answers to your questions below.
**Q4:** Lack of experiments.
**A4:** We have submitted the results on the datasets SVHN and Tiny-imagenet in the rebuttal revision Appendix E, and our method can still effectively improve robust accuracy and prevent CO.

From: ICLR.cc/2023/Conference/Paper656/Authors
title: Response to Reviewer 5TBo
comment: We greatly appreciate your comments on our manuscript. Just a reminder, we have uploaded the rebuttal revision and explained your concerns in detail. If you have any other questions or concerns, please let us know, looking forward to your feedback.

From: ICLR.cc/2023/Conference/Paper656/Authors
title: Response to Reviewer gJHh
comment: We greatly appreciate your comments on our manuscript. Just a reminder, we have uploaded the rebuttal revision and explained your concerns in detail. If you have any other questions or concerns, please let us know, looking forward to your feedback.

From: ICLR.cc/2023/Conference/Paper656/Authors
title: Response to Reviewer qRtw
comment: We greatly appreciate your comments on our manuscript. Just a reminder, we have uploaded the rebuttal revision and explained your concerns in detail. If you have any other questions or concerns, please let us know, looking forward to your feedback.

From: ICLR.cc/2023/Conference/Paper656/Authors
title: Response to Reviewer DBF4
comment: We greatly appreciate your comments on our manuscript. Just a reminder, we have uploaded the rebuttal revision and explained your concerns in detail. If you have any other questions or concerns, please let us know, looking forward to your feedback.

From: ICLR.cc/2023/Conference/Paper656/Authors
title: Response to Reviewer 5TBo
comment: We sincerely thank you for the comprehensive comments on our paper and please find the answers to your questions below.

**Q1:** The abnormal adversarial example on standard AT.

**A1:** Abnormal adversarial examples are not the unique phenomenon of single-step AT, but due to the stronger inner maximization process of iterative-step (PGD-10) AT, the number and outputs variation of abnormal adversarial examples are not enough to cause CO. In fact, CO does not only occur in single-step AT, [1] and [2] reported that CO also occurs in PGD-2 with some extreme settings. Iterative gradient ascent will still generate abnormal adversarial examples, which better shows the importance and universality of our perspective. In the future, we hope to improve iterative-step AT from the perspective of abnormal adversarial examples.


**Q2:** The role of second term and $max{⋅,0}$ in eq. (7) (Eq.9 in the rebuttal revision).

**A2:** Unlike the prediction confidence (which is a new optimization objective in rebuttal revision Eq.8) that should not reduce during the inner maximization process. The logits variation does not have a clear evaluation standard as an optimization objective that only has different magnitude. Therefore, we use the logits variation of normal adversarial examples as the standard, and explicitly constrain the logits variation of abnormal ones (usually larger). In order to clarify that the network optimization objective is to reduce the logits variation of abnormal adversarial samples instead of increasing the normal ones logits variation, we use the max function to limit the minimum value. In other words, removing the max function will cause training to fail due to the sharp increase in the logits variation of normal adversarial examples. We also tried removing the second term in Eq.7, we show the results for $AAER_{RC}$ without the second term in the table below. We can observe that the robust accuracy of the model decreases due to setting an incorrect optimization objective (the objective of logits variation is 0).

|AAER without second term in Eq.7 |Natural|PGD|AAER with second term Eq.7 |Natural|PGD|
| :-----| :----: | :----: | :-----| :----: | :----: |
|8/255|83.78|46.29 |8/255|84.33|46.42 |
|12/255|73.49 |31.65|12/255|76.07 |32.94|
|16/255|63.58 |22.93|16/255|65.88 |24.67|
|32/255|23.75|12.08|32/255|26.14|14.70|

**Q3:** Unclipped perturbation.

**A3:** This technique was proposed by N-FGSM, who claimed that clipping is performed after taking a gradient ascent step, which may make adversarial samples no longer effective [2]. Since the total step size of single-step AT is smaller than iterative-step AT, the perturbation expected to exceed the perturbation ball (not the pixel values) will not be too much.

**Q4:** Combination of AAER and existing methods.

**A4:** We have tried to combine our method with N-FGSM and ATTA which can still prevent CO. But we did not try to combine our method with Grad Align, because this combination will introduce additional hyperparameters and may conduct a lot of experiments. For example, we not only need to balance the hyperparameters of AAER and Grad Align, but also need to balance the hyperparameters of two gradient ascent (grad align is based on two gradient ascent and AAER is based on one gradient ascent).

[1] Andriushchenko, M., & Flammarion, N. (2020). Understanding and improving fast adversarial training. Advances in Neural Information Processing Systems, 33, 16048-16059.

[2] de Jorge, P., Bibi, A., Volpi, R., Sanyal, A., Torr, P. H., Rogez, G., & Dokania, P. K. (2022). Make Some Noise: Reliable and Efficient Single-Step Adversarial Training. arXiv preprint arXiv:2202.01181.

From: ICLR.cc/2023/Conference/Paper656/Authors
title: Response to Reviewer gJHh
comment: We sincerely thank you for the comprehensive comments on our paper and please find the answers to your questions below.

**Q1:** In Figure 3 the logits variation of normal adversarial examples also increases.

**A1:** Since the network directly optimizing by abnormal adversarial examples will further exacerbate the model boundary distortion, training samples generated on highly distorted models, even for normal adversarial examples, the logits variation will be very large. To prove this point, we can observe Figure 3(c), we find that the logits variation of normal and abnormal adversarial examples are significantly different (16 times) after an epoch of CO occurrence (due to the limitation of computing resources, we cannot observe the logits variation after iteration of CO). Besides, even after the network directly optimizing by abnormal adversarial examples, the logits variation of abnormal adversarial examples is twice as large as that of normal ones, which is significantly different (almost the same) compared with before CO.

**Q2:** The description of Eq.7 (Eq.9 in the rebuttal revision) second term is unclear.

**A2:** We have fixed this inaccurate description in the rebuttal revision, the Eq.7 second term should only contain normal adversarial examples.

**Q3:** The role of $\frac{n}{m}$ in Eq.8 (Eq.10 in the rebuttal revision).

**A3:** Since M indicates that batch size can be regarded as a hyperparameter, the loss value is positively related to N, and a larger N means that the network is farther from the optimization objective.

**Q4:** Lack of experiments.

**A4:** We have submitted the results on the benchmark dataset SVHN in the rebuttal revision Appendix E. At the same time, we will try to submit results on Tiny-ImageNet before November 18. However, due to the lack of experimental results on Tiny-ImageNet in the single-step AT research community and the limitation of computing resources, we may not be able to evaluate all baselines on Tiny-ImageNet.

**Q5:** Lack of curve of optimization objective.

**A5:** To verify the effectiveness of our method, we show the AAER optimization objectives (number and outputs variation of abnormal adversarial examples) change during training in the rebuttal revision ablation study Figure 3 yellow lines. Since the loss curve is directly composed of the optimization objectives, the change in the loss curve can be reflected by this.

**Q6:** Unclipped perturbation.

**A6:** This technique was proposed by N-FGSM, who claimed that clipping is performed after taking a gradient ascent step, which may make adversarial samples no longer effective [1]. Since the total step size of single-step AT is smaller than iterative-step AT, the perturbation expected to exceed the perturbation ball will not be too much. At the same time, this technology is not commonly used mainly due to the short publication time (first submitted on 2 Feb 2022).

**Q7:** The role of $\beta$.

**A7:** This is the hyperparameter to control the strength of the previous initialization. We will provide detailed information including the choice of hyperparameter $\beta$ in the rebuttal revision Appendix A.

[1] de Jorge, P., Bibi, A., Volpi, R., Sanyal, A., Torr, P. H., Rogez, G., & Dokania, P. K. (2022). Make Some Noise: Reliable and Efficient Single-Step Adversarial Training. arXiv preprint arXiv:2202.01181.

From: ICLR.cc/2023/Conference/Paper656/Authors
title: Response to Reviewer qRtw
comment: We sincerely thank you for the comprehensive comments on our paper and please find the answers to your questions below.

**Q1:** The role of $\eta$ in Eq.5.

**A1:** $\eta$ is the initialization before gradient ascent, and the random initialization $\eta$ is within the perturbation ball in this work.
Initialization before gradient ascent is widely used in single and iterative-step AT (e.g., PGD and RS-FGSM), which can bring input diversity for better approximate solutions.

**Q2:** The role of $\eta_{t}$ and $\delta_{t}$ in algorithm 1.

**A2:** The $\eta$ in Line 7 to line 11 of algorithm 1 (Appendix A in the rebuttal revision) is a different type of initialization -- previous initialization, ATTA [1] found that perturbations are highly transferable between models from adjacent epochs. Therefore, the previous initialization $\eta$ is not obtained by calculation, but saves the $eta_{t}$ and $\delta_{t}$ of the adjacent epoch to construct the $\eta$ in this epoch. We will provide detailed information including the choice of hyperparameter $\beta$ in the rebuttal revision Appendix A.

**Q3:** The role of linearly increasing $\alpha$.

**A3:** We employ linearly increasing $\alpha $ (t/T) to stabilize the optimization objective and avoid model training to crash in the early stages with extremely noise magnitude. It is worth noting that increasing $\alpha$ loses a part of the perturbation information corresponding to a certain degree of robust accuracy decrease, and our method can achieve better robustness even with increasing $\alpha$, which further proves that our method is effective in improving robustness. For comparison, we show the results for $AAER_{RC}$ without linearly increasing $\alpha$ in the table below, and we can observe that we indeed get better robust accuracy except that the model training to crash with 32/255 noise magnitude. It is worth noting that this kind of model training to crash with extreme noise settings is widespread in regularization methods because the model is very fragile in the early stages of training but the regularization is very strong. However, unlike Align [2] manual reduction of learning rate with 32/255 noise magnitude, the linearly increasing $\alpha$ can prevent training failure while avoiding manual tuning of hyperparameters in different settings. We also noticed that N-FGSM [3] also use a linearly increasing $\alpha$ on the SVHN dataset. In general, unlike iterative-step AT has a strict step size setting, the step size setting of single-step AT is very diverse (most methods have different $\alpha$), and mainly focuses on training time, robust accuracy and whether CO occurs.
|AAER without linearly increasing $\alpha$ |Natural|PGD|AAER with linearly increasing $\alpha$ |Natural|PGD|
| :-----| :----: | :----: | :-----| :----: | :----: |
|8/255|83.51|46.74 |8/255|84.33|46.42 |
|12/255|73.13 |33.39|12/255|76.07 |32.94|
|16/255|61.29 |25.23|16/255|65.88 |24.67|
|32/255|**10.00**|**10.00**|32/255|26.14|14.70|

**Q4:** Experiment details and robustness changes.

**A4:** We mentioned this setting in Experiment Setting (4.1): Attack Methods and Learning Rate Schedule that we use the cyclical learning rate schedule with 30 epochs that reach its maximum learning rate (0.2 in our experiments) when half of the epochs (15) are passed on Cifar-10/100. We did not verify our method robustness change during training, all reported data (including baselines) are the last checkpoint, this is because, unlike robust overfitting in iterative-step AT, the overfitting of single-step AT is shown as CO (last checkpoint robust accuracy is 0). To verify the effectiveness of our method, we show the AAER optimization objectives (number and outputs variation of abnormal adversarial examples) change during training in the rebuttal revision ablation study Figure 3 yellow lines. We think that the changes in the optimization objective can better reflect the effect of our method than robustness changes.

**Q5:** Lack of baselines evaluation on AutoAttack.

**A5:** We have supplemented the robust accuracy of all baselines against AutoAttack in the rebuttal revision Appendix C, and our method can still achieve boost adversarial robustness.

[1] Zheng, H., Zhang, Z., Gu, J., Lee, H., & Prakash, A. (2020). Efficient adversarial training with transferable adversarial examples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 1181-1190).

[2] Andriushchenko, M., & Flammarion, N. (2020). Understanding and improving fast adversarial training. Advances in Neural Information Processing Systems, 33, 16048-16059.

[3] de Jorge, P., Bibi, A., Volpi, R., Sanyal, A., Torr, P. H., Rogez, G., & Dokania, P. K. (2022). Make Some Noise: Reliable and Efficient Single-Step Adversarial Training. arXiv preprint arXiv:2202.01181.

From: ICLR.cc/2023/Conference/Paper656/Authors
title: Response to Reviewer DBF4
comment: We sincerely thank you for the comprehensive comments on our paper and please find the answers to your questions below.

**Q1:** Ignore abnormal adversarial examples and train only on normal examples.

**A1:** We tried to simply ignore any abnormal adversarial examples in the loss and train only on examples where the loss actually increased, and unfortunately it didn't work. This is because the abnormal adversarial example is not the cause of CO, which co-occurs with CO. In other words, the model decision boundary is partially distorted when generating the abnormal adversarial example, and ignoring abnormal adversarial example information can avoid directly optimizing them on the network to exacerbate the model boundary distortion, but cannot fix the existing part of the network distortion. Finally, the model that ignores information from abnormal adversarial examples will still CO (which will delay the CO occurrence) because it cannot hinder the generation of abnormal adversarial examples. Therefore, we proposed AAER to explicitly hinder the generation of abnormal adversarial examples to prevent CO. 

**Q2:** Comparison of AAER and N-FGSM.

**A2:** According to [1], the neural network may learn some non-robust features to obtain higher natural accuracy. The expense of natural accuracy just shows that our method makes the model pay more attention to learning robust features. Therefore, we would like to clarify that our method improves the upper bound of robust accuracy (which is more difficult to obtain) compared to N-FGSM, which they cannot achieve even if they sacrifice some natural accuracy (e.g., increase step size). At the same time, N-FGSM cannot guarantee to defend against CO in some extreme settings (e.g., for CIFAR-100 32/255). 

**Q3:** Explanation on Eq.7 (Eq.9 in the rebuttal revision).

**A3:** Unlike the prediction confidence (which is a new optimization objective in rebuttal revision Eq.8) that should not reduce during the inner maximization process. The logits variation does not have a clear evaluation standard as an optimization objective that only has different magnitude. Therefore, we use the logits variation of normal adversarial examples as the standard, and explicitly constrain the logits variation of abnormal ones (usually larger). In order to clarify that the network optimization objective is to reduce the logits variation of abnormal adversarial samples instead of increasing the normal ones logits variation, we use the max function to limit the minimum value. In other words, removing the max function will cause training to fail due to the sharp increase in the logits variation of normal adversarial examples. We also tried removing the second term in Eq.7, we show the results for $AAER_{RC}$ without the second term in the table below. We can observe that the robust accuracy of the model decreases due to setting an incorrect optimization objective (the objective of logits variation is 0).
|AAER without second term in Eq.7 |Natural|PGD|AAER with second term Eq.7 |Natural|PGD|
| :-----| :----: | :----: | :-----| :----: | :----: |
|8/255|83.78|46.29 |8/255|84.33|46.42 |
|12/255|73.49 |31.65|12/255|76.07 |32.94|
|16/255|63.58 |22.93|16/255|65.88 |24.67|
|32/255|23.75|12.08|32/255|26.14|14.70|

From: ICLR.cc/2023/Conference/Paper656/Authors
title: Response to Reviewer DBF4
comment: **Q4:**  Non-standard setting: linearly increasing $\alpha$.

**A4:** The normal FGSM with linearly increasing $\alpha$ still suffer from CO. Similar to ignoring abnormal adversarial example information, increasing $\alpha$ can delay but cannot ultimately avoid CO. It is worth noting that increasing $\alpha$ loses a part of the perturbation information corresponding to a certain degree of robust accuracy decrease, and our method can achieve better robustness even with increasing $\alpha$, which further proves that our method is effective in improving robustness. For comparison, we show the results for $AAER_{RC}$ without linearly increasing $\alpha$ in the table below, and we can observe that we indeed get better robust accuracy except that the model training to crash with 32/255 noise magnitude. It is worth noting that this kind of model training to crash with extreme noise settings is widespread in regularization methods because the model is very fragile in the early stages of training but the regularization is very strong. However, unlike Align [2] manual reduction of learning rate with 32/255 noise magnitude, the linearly increasing $\alpha$ can prevent training failure while avoiding manual tuning of hyperparameters in different settings. We also noticed that N-FGSM [3] also use a linearly increasing $\alpha$ on the SVHN dataset. In general, unlike iterative-step AT has a strict step size setting, the step size setting of single-step AT is very diverse (most methods have different $\alpha$), and mainly focuses on training time, robust accuracy and whether CO occurs.
|AAER without linearly increasing $\alpha$ |Natural|PGD|AAER with linearly increasing $\alpha$ |Natural|PGD|
| :-----| :----: | :----: | :-----| :----: | :----: |
|8/255|83.51|46.74 |8/255|84.33|46.42 |
|12/255|73.13 |33.39|12/255|76.07 |32.94|
|16/255|61.29 |25.23|16/255|65.88 |24.67|
|32/255|**10.00**|**10.00**|32/255|26.14|14.70|

**Q5:** Misleading bolding, grammatical errors, unclear descriptions.

**A5:** Thanks for the correction, we have fixed these misleading bolding, grammatical errors and unclear descriptions in the rebuttal revision. 

[1] Ilyas, A., Santurkar, S., Tsipras, D., Engstrom, L., Tran, B., & Madry, A. (2019). Adversarial examples are not bugs, they are features. Advances in neural information processing systems, 32.

[2] Andriushchenko, M., & Flammarion, N. (2020). Understanding and improving fast adversarial training. Advances in Neural Information Processing Systems, 33, 16048-16059.

[3] de Jorge, P., Bibi, A., Volpi, R., Sanyal, A., Torr, P. H., Rogez, G., & Dokania, P. K. (2022). Make Some Noise: Reliable and Efficient Single-Step Adversarial Training. arXiv preprint arXiv:2202.01181.

From: ICLR.cc/2023/Conference/Paper656/Reviewer_DBF4
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: This paper proposes a new training objective for single-step adversarial training to reduce catastrophic overfitting (CO). In particular, the paper establishes a correlation between CO and the presence of "abnormal adversarial examples," i.e., instances where taking a single step of FGSM actually decreases the loss rather than increasing it. To this end, the authors propose a regularizer called AAER meant to penalize abnormal adversarial examples, and show that it often improves the accuracy of SSAT.
strength_and_weaknesses: Strengths:

- The observation that CO co-occurs with abnormal adversarial examples is both novel and interesting.
- The attack seems efficient (e.g., not much slower than N-FGSM) while providing better adversarial accuracy.
- The authors conduct an extensive evaluation of other methods and baselines.

Weaknesses:

- Given the link between CO and abnormal adversarial examples, a natural baseline would be to simply ignore any abnormal examples in the loss and train only on examples where the loss actually increased. Does this not work? If so, why?
- The method does seem to improve on N-FGSM in terms of robust accuracy, but this often comes at the expense of clean accuracy, so it's unclear if the proposed method truly dominates N-FGSM. There is also some misleading bolding in the Table (i.e., for CIFAR-100 16/255 it doesn't look like the confidence intervals of the two bolded entries actually overlap) that should be fixed.
- I am confused about the motivation behind Eq. 7---it seems like it is encouraging the network to classify the original and "normal adversarial" examples very differently (judging by the negative term)? Please correct me if I am wrong here, but it seems very counterintuitive.
- Finally, there are many non-standard additions to the training routine (I'm thinking mainly of the linearly increasing $\alpha$, as the other terms were ablated quite well). Does normal FGSM with linearly increasing alpha still suffer from CO?
clarity,_quality,_novelty_and_reproducibility: The work is certainly novel and original. The quality is reasonable, bar the weaknesses above. The clarity could be improved---there are some grammatical errors throughout, and there are some unclear sentences, e.g.:

- Furthermore, the strength of AAER depends on the interaction of two parts learning objective, which can reflect the degree of classifier distortion more comprehensively and flexibly.
- Based on the above analysis, we design a novel regularization term, AAER, which aims to suppress the anomalous training samples by the (i) number and the (ii) logits variation, ultimately achieving the purpose of preventing CO, which is shown in the following formula
...


summary_of_the_review: In summary, the work proposes a new training objective for regularizing single-step adversarial training. Some of the motivation is a bit unclear and there is a natural baseline that the work does not explore. Overall, I think the paper as is is slightly below the bar for publication, but I am willing to raise my score.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold

From: ICLR.cc/2023/Conference/Paper656/Reviewer_qRtw
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: 
This paper studies the robust overfitting of one-step adversarial training and finds that robust overfitting is related to the increase of abnormal adversarial examples generated for training. Abnormal adversarial examples have even lower losses after adding the adversarial perturbations, which can be a reason for causing robust overfitting in one-step adversarial training. Based on this observation, this paper proposes a novel regularization term to reduce the number of abnormal adversarial examples. By comparing the proposed methods with other baselines on CIFAR10 and CIFAR100 datasets, the evaluation results show that the proposed method achieves better robustness against the PGD attack.

strength_and_weaknesses: Strength: 
+ This paper points out the existence of abnormal adversarial examples when robust overfitting happens and speculates that robust overfitting in single-step adversarial training is caused by abnormal adversarial examples. 

+ This paper proposed a novel regularization term to reduce the number of abnormal adversarial examples in training.

+ The evaluation results show that the proposed method outperforms other baseline methods.

clarity,_quality,_novelty_and_reproducibility: Weakness:

+ There are several places that I am not quite clear about in writing and may need further clarification:
1. In equation 5, is $\eta$ the random noise in the allowed perturbation ball? Why should we need an $\eta$ here? 
2. In Line 7 to line 11 of algorithm 1, $\eta$ is calculated for each iteration. Why does Algorithm 1 need this? What is the role of this component in the algorithm? What is $\eta_t$ and $\delta_t$ here.
3. In Line 15, there is a multiplier $t/T$, what is this for?

+ I don’t find some important experiment details: What is the number of training epochs and learning rate scheduler for the proposed training method? Which checkpoint are you using for evaluation? What’s the robustness difference between the best checkpoint and the last checkpoint? It will be nice to draw a figure to show how robustness changes for the proposed algorithm (a figure like Figure.1). 

+ The robustness reported in Table.1 is evaluated with the PGD attack, which is always found an overestimated robustness issue. Compared to the PGD attack, AutoAttack is usually more reliable to evaluate the robustness. Although the appendix compares the AA robustness with the vanilla PGD-based adversarial training method, it will be more convincible to compare the robustness of the proposed method with other SOTA baselines against AutoAttack.

summary_of_the_review: There are several points/places lacking clarity in the paper  (as discussed in the weakness). Further clarification and details may be necessary.

Reproducibility: The experiment settings section does not include all experiment details and no code is provided.

correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold

From: ICLR.cc/2023/Conference/Paper656/Reviewer_gJHh
confidence: 5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
summary_of_the_paper: Recently, FGSM-based adversarial training faced an emergency problem of catastrophic overfitting. To mitigate this, the authors first observed that the catastrophic overfitting is related to some abnormal examples, whose loss decreases under the FGSM attack. Based on this observation, in this paper they proposed a new abnormal adversarial example regularization (AAER), which considers two key components: one is about the number of the abnormal examples, while the other is about the logits variation. The experiments on CIFAR-10/100 show that the proposed AAER is effectiveness.
  
strength_and_weaknesses: **Strength**:
1. The observation of the abnormal adversarial examples is interesting. 
2. The experiments show their effectiveness on two widely-used CIFAR datasets.

**Weaknesses**:
1. In section 3, the authors analyzed the number and logit variation of normal and abnormal examples, respectively. Section 3.1 shows that the abnormal adversarial examples dominate the catastrophic overfitting since the number has a significant increase when catastrophic overfitting exists. In section 3.2, the authors found that the logit variations during the inner process will increase after the catastrophic overfitting. Both of these two observations are new and interesting. And the authors claimed that their proposed AAER is inspired by them. However, it's very difficult for me to understand how Eq.7 is derived from these two observations? 
1.1 From the second observation of logit variations, whether it is a normal sample or an abnormal sample, the change of logical distance shows an increasing trend, and the trend of distance change is also the same. Therefore, why does Eq.7 reduce the gap between these two type of adversarial examples? 
1.2 At the same time, Eq.7 is not very rigorous, because the $x_k$ in the second term may contain abnormal examples. 
1.3 The authors wanted to reduce the number of the abnormal examples. So they introduced a term $\frac{n}{m}$ in Eq.8. But the authors did not analyze the mechanism of this design.

2. The authors conducted experiments on CIFAR-10 and CIFAR-100. I don't think the experiments are enough. As we know, FGSM-based adversarial training is one of the most effective training schemes that can also train the model on a large-scale dataset. So it's better to evaluate the proposed method and analyze the properties of abnormal adversarial examples on a large dataset, like Tiny-ImageNet or ImageNet. 

3. Since the proposed AAER is to reduce the impact of abnormal adversarial examples. However, there were no corresponding experiments on this part in the experiment. For example, show the number curve after using AAER (as shown in Figure 3 (a)). Also, as training progresses, it is better to show the loss curve.

4. On page 7, the authors said that they used the unclipped perturbation to achieve better robustness. However, this is not a commonly used technique. This may cause the pixel value of the adversarial sample to exceed the actual value. The authors need to clarify the reasons for using this technique.

5. On page 8, what's the parameter $\beta$? In previous section, I didn't find this hyper-parameter. 
clarity,_quality,_novelty_and_reproducibility: The observations are new and I think the proposed method is easy to reproduce. But the experiments are not very sufficient to verify the proposed regularization technique.
summary_of_the_review: This paper finds some interesting observations, but the proposed technique does not appear to be strongly correlated with these observations. Moreover, the experiments are not very sufficient. 
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 3: reject, not good enough

From: ICLR.cc/2023/Conference/Paper656/Reviewer_5TBo
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: This paper proposes Abnormal Adversarial Examples Regularization (AAER), a regularizer to address the known catastrophic overfitting issue observed in fast adversarial training (AT). AAER is based on suppressing the output difference of abnormal adversarial examples (irregular adversarial samples such that their loss is smaller after adding perturbation) and their number, as well as maximizing the difference of model output between unperturbed and perturbed samples. 
strength_and_weaknesses: Strength:
1. The proposed method is simple to implement and has low overhead

Weakness:
1. The uniqueness of Fig. 3 to fast AT needs more justification
2. The role of the second term in eq. (7) is unclear
3. What is "unclipped perturbation"? Is it a rational regularizer? Why it gives better performance?
clarity,_quality,_novelty_and_reproducibility: Although the proposed method is novel and addresses an important challenge in fast AT, the current presentation of this submission has several issues that need to be addressed. I list my main concerns and comments as follows.
summary_of_the_review: 1. Figure 3 is a very nice motivating example of the challenge and the proposed solution. However, it is unclear whether the emergence of abnormal examples is unique to the fast AT setting or not. To complete the analysis, I suggest running the same analysis on standard AT. If standard AT does not suffer from this issue but fast AT does, then this is a more convincing analysis.

2. How critical is the second term in eq. (7)? In Line 15 of Algorithm 1, I noticed the authors use a zero threshold $\max${$\cdot$,0}. Does this suggest the second term in eq. (7), which contributes to negative numbers fo AAER, may not be important? Can the authors run an ablation study of the experiments with and without the second term?

3. In most cases, the AAER_RUC method is the best regularizer. However, it's unclear to me what "unclipped perturbation" means in this context. Do the authors mean allowing the perturbed samples to go beyond the pixel values, or the authors did not clip the norm of the perturbation? Both cases make little sense to me, and the result needs more justification.

4. Can AAER show complementary performance gain together with existing methods (e.g., gradient alignment) to mitigate catastrophic forgetting? For instance, any benefit of combining AAER with any of the existing methods in Table 1?

correctness: 2: Several of the paper’s claims are incorrect or not well-supported.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 3: reject, not good enough