Paper: Annealing Self-Distillation Rectification Improves Adversarial Training (ICLR-2024)

Reviewer comment (Novelty):
The motivation is too easy and there seem no new insights in the manuscript.

Author response:
**Q: The motivation is too easy and there seem no new insights in the manuscript.**

We highlight that we observe the key property that makes the robust model reliable in Section 3.2, the robust model is well-calibrated and consistent with its output when input is perturbed, this motivates us to design a unified rectified label to enhance the efficacy of adversarial defense. 

In our observations, we discover that adversarially trained networks consistently exhibit output consistency between adversarial images and their clean counterparts. Some previous studies suggest the incorporation of dual targets during adversarial training, encouraging models to learn from both an adversarially trained model on adversarial images and a naturally trained model on normal images (Cui et al., 2021; Zhao et al., 2022). While seemingly intuitive in mitigating the trade-off between robustness and accuracy, our observations challenge the appropriateness of this approach. We contend that learning from two targets, one with high confidence from the normal model and the other with lower confidence from the adversarially trained model, is not conducive to achieving consistency for images with noise. Therefore, we propose composing a rectified label in ADR and urge that all images within the $l_p$ norm neighborhood of a given input should be targeted using the rectified label rather than the one-hot one. The observations in Section 3 motivate us to design ADR that effectively prevents the network from becoming over-confident and provides semantically aware smoothing rather than a uniform one. We believe that we do provide some useful insights for the community.

Reviewer comment (Evaluation):
The results on Table 2 show that on WRN on three datasets, AT+ADR has a weaker performance compared with AT+WA, and AT+WA+ADR also has a weaker performance compared with AT+WA+AWP. It may demonstrate ADR doesn't work well compared with WA and AWP on large models.

Author response:
**Q: The results in Table 2 show that on WRN on three datasets, AT+ADR has a weaker performance compared with AT+WA, and AT+WA+ADR also has a weaker performance compared with AT+WA+AWP. It may demonstrate ADR doesn't work well compared with WA and AWP on large models.**

ADR with WA and AWP should be viewed as complementary rather than competing approaches. These methods target different aspects of enhancing adversarial training.

In the realm of adversarial defense, we can encapsulate the directions of research aimed at enhancing the efficacy of adversarial training into the following key areas:
* Different training objectives. e.g. PGD-AT (Madry et al., 2018), TRADES (Zhang et al., 2019).
* Encourage a flatter model-weight landscape. e.g.  WA (Izmailov et al. 2018; Chen et al. 2021), AWP (Wu et al. 2020).
* Generate additional data or create auxiliary examples. e.g. use DDPM data (Gowal et al., 2020; Rebuffi et al., 2021a), HAT (Rade et al., 2021)
* Create effective data augmentation strategies to enhance robustness. e.g. CutMix (Yun et al. 2019; Rebuffi et al. 2021a), DAJAT (Addepalli et al. 2022)
* Knowledge distillation. e.g. ARD (Goldblum et al., 2020b), RSLAD (Zi et al., 2021), IAD (Zhu et al., 2022)

While WA and AWP concentrate on smoothing the model weight landscape, ADR focuses on smoothing the training labels. As their goals are distinct and not in conflict, direct comparisons between AT+ADR and AT+WA, or between AT+WA+ADR and AT+WA+AWP, may not be entirely fair. Their combined effects may offer a more comprehensive approach to improving adversarial training, addressing both the model's weight landscape and the label-smoothing aspects, bringing the overall robustness enhancement.

Reviewer comment (Experiments):
The comparison with RobustBench in the paper is not fair, results on AT+ADR rather than AT+WA+AWP+ADR should be used. In this comparison, ADR achieves state-of-the-art robust performance on ResNet-18 but has a poor performance on WRN.

Author response:
**Q: The comparison with RobustBench in the paper is not fair, results on AT+ADR rather than AT+WA+AWP+ADR should be used. In this comparison, ADR achieves state-of-the-art robust performance on ResNet-18 but has a poor performance on WRN.**

We do not agree that AT+ADR rather than AT+WA+AWP+ADR should be used for the RobsutBench comparison.   It is acknowledged that RobustBench, a benchmark for adversarial robustness, often includes methods that integrate several techniques rather than reporting solely on individual methods. As the common research directions for adversarial training that we provided in our previous reply, the methods from each direction can be combined as complementary to obtain better results.  Ultimately, the goal is to employ a combination that best suits the experimental objectives and the overall enhancement of adversarial robustness.

For reference, Zhang et al. (2022), Rebuffi et al. (2021a), and Rade & Moosavi-Dezfooli (2022), report the results combined with WA. 
Addepalli et al. (2022), Chen & Lee (2021), and Jia et al. (2022)
report their results combined with AWP.

Reviewer comment (Novelty):
Because it has little novelty, its performance is now the most significant evaluation indicator. Its performance on WRN is not competitive enough. I will increase my score if better results come out.

Author response:
**Q: The performance on WRN is not competitive enough.**

We acknowledge that the performance reported on WRN may not be competitive enough in some cases. This phenomenon is primarily attributed to our deliberate decision to maintain consistency in experimental settings across different models, eliminating the need to report distinct hyperparameters for each model.

To ensure uniformity, we employ the same $\tau$ and $\lambda$ schedule for both ResNet-18 and WRN-34-10, and the reported temperature and label interpolation factor are determined based on the evaluation of performance on ResNet-18. It is crucial to note that the chosen parameters may not represent the optimal fit for WRN-34-10, as the most effective parameters can vary based on architecture and specific settings, as previously mentioned in our limitations. We will investigate an automatic way to determine the optimal parameter in our future research.

Due to we only have limited time and the substantial computational demands associated with adversarial training on expansive models like WRN, we only provide a full comparison on CIFAR-10 for reference. We can provide more experiment results in the final version of our paper.
In the experiment, we keep $\lambda$  scheduling from 0.7 to 0.95 same as the original paper but adjust $\tau$ to anneal from 1.5 to 1 instead of the original 2.5 to 2.

***CIFAR-10***
|   | AutoAttack  |   Standard Acc. |
|---|---|---|
| AT+ADR   (original) | 53.25 | 84.67 |
| AT+ADR | 53.54 | 85.58 |
| AT+WA+ADR   (original) | 54.10 | 82.93 |
| AT+WA+ADR | 54.14 | 83.10 |
| AT+WA+AWP+ADR   (original) | 55.26 | 86.11 |
| AT+WA+AWP+ADR | 55.27 | 86.90 |

An additional CIFAR-100 result when we anneal $\tau$ from 1.25 to 1 instead of 1.5 to 1.

***CIFAR-100***
|   | AutoAttack  |   Standard Acc. |
|---|---|---|
| AT+ADR   (original) | 29.35 | 59.76 |
| AT+ADR | 29.79 | 60.17 |

A larger architecture generally requires a lower temperature to achieve its best performance, it may be because of their superior capability to learn the inter-class relationship within the data.
We can observe that when selecting more suitable hyper-parameters for WRN-34-10, both robust and clean accuracy can be further enhanced.

Reviewer comment (Experiments):
Considering the simplicity of the method, I think the experiments are not very extensive. More datasets (e.g., ImageNet), attacks, and especially baseline AT methods should be considered.

Author response:
**Q: Considering the simplicity of the method, I think the experiments are not very extensive. More datasets (e.g., ImageNet), attacks, and especially baseline AT methods should be considered.**

Thank you for your suggestion. However, despite the apparent simplicity of ADR, it is essential to note that the training cost associated with ADR is comparable to other multi-step adversarial training methodologies, which are recognized for demanding computationally intensive processes. Our observations suggest that very few works employing multi-step adversarial training methodologies have presented results on the extensive ImageNet dataset. Instead of ImageNet, we present results on TinyImageNet-200, a representative subset, to underscore the practical effectiveness of our approach to real-world data.
In our comparative analysis, the related works we compared (Rebuffi et al. 2021b; Yun et al. 2019; Rebuffi et al. 2021a; Addepalli et al. 2022; Li & Spratling 2023; Rade & Moosavi-Dezfooli, 2022; Gowal et al., 2020; Cui et al., 2021; Sehwag et al., 2022; Jia et al., 2022; Chen & Lee, 2021) refrain from experimenting with ImageNet and limit their baselines primarily to PGD-AT, TRADES with WA, and AWP. Consequently, we assert that the experiments we provide offer a comprehensive and representative baseline for evaluation.

Acknowledging the constraints imposed by time and resource limitations during this rebuttal period, we regret our inability to present new results on additional datasets. However, we concur with the notion that augmenting experiments on larger datasets, coupled with diverse methods, would greatly benefit the research community. We commit to providing more comprehensive results in future work.

Reviewer comment (Evaluation):
The self-distillation brings high memory cost, which I believe would be a main limitation for the practical use of the proposed method.

Author response:
**Q: The self-distillation brings high memory costs, which I believe would be a main limitation for the practical use of the proposed method.**

While we acknowledge the additional memory costs associated with maintaining an EMA teacher for rectified distribution, it is crucial to underscore that the ADR method does not incur higher memory usage compared to previous techniques. Notably, ADR remains competitive in terms of performance without necessitating increased memory allocation. For instance, WA upholds a weight average copy of network parameters to attain a smoother model weight landscape for enhanced robustness. AWP also maintains an additional copy of the gradient direction at each step to calculate the updated direction for models. Consequently, ADR demonstrates competitive performance levels when deployed under equivalent resource constraints as its counterparts.

Reviewer comment (Experiments):
Did you use any pretrained models for initialization? I wonder how ADR works when using pretraining models.

Author response:
**Q: Did you use any pre-trained models for initialization? I wonder how ADR works when using pretraining models.**

Thanks for your interesting suggestion, we leverage pre-trained models for the initialization of the ADR process. Specifically, we employ ResNet-18 on the CIFAR-10 dataset, adhering to all other parameter settings outlined in Section 5.1. The key variation lies in the initialization of the model with either a standard or PGD-AT pre-trained model, followed by subsequent ADR training. Our reporting is based on the optimal checkpoint obtained during our comprehensive evaluation.
Moreover, to offer a more comprehensive view of the observed trends, we present line charts depicting the performance trajectories with both clean and robust accuracy in Figure 10. These visual representations are included in the Appendix L of our updated paper for enhanced clarity and accessibility.

The experimental findings reveal a consistent trend wherein ADR consistently outperforms the baseline in terms of robust accuracy, regardless of whether it is initialized with a pre-trained model or not. The utilization of either a standard model or a PGD-AT pre-trained weight as an initialization fails to further augment the maximum robustness achievable through ADR. Nevertheless, it is noteworthy that employing an AT pre-trained checkpoint results in a notable enhancement of standard accuracy by 1.25% when compared to training with random initialization. This outcome underscores the potential for mitigating the accuracy-robustness tradeoff, indicating the feasibility of achieving improved performance by utilizing an AT pre-trained model during the initialization phase.

|   | AutoAttack  | Standard Acc. |
|---|---|---|
| AT|48.81 |82.52 |
|AT + ADR|50.38|82.41|
|AT + ADR + standard pre-trained initialization | 49.45 | 82.22 |
|AT + ADR + AT pre-trained initialization | 50.11 | 83.66 |

Reviewer comment (Evaluation):
In Table 2, test accuracies of ADR combined with WA and AWP are presented. From the results it seems that each method contributes a (roughly) similar amount of robustness gain. Could the authors comment on whether they consider ADR, WA, AWP to be complementary?

Author response:
**Q: Could the authors comment on whether they consider ADR, WA, and AWP to be complementary?**

In the realm of adversarial defense, we can encapsulate the directions of research aimed at enhancing the efficacy of adversarial training into the following key areas:
* Different training objectives. e.g. PGD-AT (Madry et al., 2018), TRADES (Zhang et al., 2019).
* Encourage a flatter model-weight landscape. e.g.  WA (Izmailov et al. 2018; Chen et al. 2021), AWP (Wu et al. 2020).
* Generate additional data or create auxiliary examples. e.g. use DDPM data (Gowal et al., 2020; Rebuffi et al., 2021a), HAT (Rade et al., 2021)
* Create effective data augmentation strategies to enhance robustness. e.g. CutMix (Yun et al. 2019; Rebuffi et al. 2021a), DAJAT (Addepalli et al. 2022)
* Knowledge distillation. e.g. ARD (Goldblum et al., 2020b), RSLAD (Zi et al., 2021), IAD (Zhu et al., 2022)

We provided an introduction to those methods in both Section 2 and Appendix A. 

Given the straightforward design of ADR, it becomes feasible to integrate ADR with other adversarial training methodologies as complementary approaches. The results of these combinations are presented in Table 1 for ADR with different training objectives (PGD-AT, TRADES), in Table 2 for ADR with flatter weight landscape methods (WA, AWP), and in Table 3 for ADR in conjunction with generated data (DDPM). The integration of ADR with these complementary methods demonstrates discernible gains in robustness.

While it is acknowledged that data augmentation-based methods often necessitate intricate hyperparameter tuning and extended training schedules due to their reliance on diverse augmentation techniques, such complexities are not the focus of our work. Consequently, ADR is not combined with these methods in our experiments. Additionally, knowledge distillation-based methods, which modify training labels through the knowledge of pre-trained networks, are not suitable for combination with ADR. Instead, we provide explicit comparisons in Table 3 and Appendix Table 8, establishing ADR as a more effective and resource-efficient methodology.

Reviewer comment (Theory):
In the motivation (subsection 4.1.) it is stated that the aim is 'to design a label-softening mechanism that properly reflects the true distribution'. In the presented approach the teacher network provides the probability distribution with which the one-hot label is interpolated. Can the authors comment on why the teacher network (model weight average of student) reflects the 'true distribution'?

Author response:
**Q: In the motivation (subsection 4.1.) it is stated that the aim is 'to design a label-softening mechanism that properly reflects the true distribution'. In the presented approach the teacher network provides the probability distribution with which the one-hot label is interpolated. Can the authors comment on why the teacher network (model weight average of students) reflects the 'true distribution'?**

Thanks for asking this question. Let us clarify that our primary objective is to design a label-softening mechanism that accurately reflects the true underlying distribution for each example. It's important to note that the teacher EMA network does not precisely mirror this ideal distribution from the outset. Instead, it progressively refines its approximation towards the goal of achieving both increased robustness and accuracy as the training process advances. Acknowledging the potential for confusion in our original language, we have revised the corresponding paragraph in our updated paper to ensure a more accurate and clear representation of our intentions.

Reviewer comment (Evaluation):
The results presented in Figure 5 show the results of a hyperparameter study for the interpolation and temperature parameter. It is concluded that annealing both parameters is beneficial for robustness. Do the authors consider the differences (e.g. in the row on $\lambda$ annealing) significant enough to draw such conclusions?

Author response:
**Q: The results presented in Figure 5 show the results of a hyperparameter study for the interpolation and temperature parameters. It is concluded that annealing both parameters is beneficial for robustness. Do the authors consider the differences (e.g. in the row on annealing) significant enough to draw such conclusions?**

While we acknowledge that the variations in the number of occurrences in each row and column in Figure 5 may not be pronounced, a discernible trend emerges when implementing annealing in $\tau$. Notably, the clean and robust accuracy consistently outperforms scenarios where the temperature is fixed at a specific value across most cases as $\lambda$ varies. Similarly, the highest robust accuracy is attained when annealing in $\lambda$ instead of maintaining a fixed $\lambda$ value while varying the temperature. It is noteworthy that, although there may be a slight sacrifice in clean accuracy when annealing in $\lambda$ during training, such trade-offs are inherent, given the well-established inverse relationship between model accuracy and robustness. Thus, we conclude that annealing in both temperature and label interpolation factors proves advantageous in achieving superior overall performance.

Reviewer comment (Presentation):
The texts in Figure 2 may be too small, which can be difficult to read when printed out.

Author response:
**Q: The texts in Figure 2 may be too small, which can be difficult to read when printed out.**

Thank you for the suggestion, we provided adjusted figures that have larger texts in our revised paper.

Reviewer comment (Evaluation):
Are the robust models trained via ADR more conformed to the empirical findings in Section 3.2, as compared with vanilla AT?

Author response:
**Q: Are the robust models trained via ADR more conformed to the empirical findings in Section 3.2, as compared with vanilla AT?**

Thanks for asking, we provide our entropy distribution analysis on the ADR-trained model in the updated paper of Appendix K. 

From Figure 9a, we can see that the ADR model trained on CIFAR-10 is more uncertain than the model trained with PGD-AT when encountering out-of-distribution data (CIFAR-100). It supports our claim that the robust model should not be over-confident when seeing something it has never seen before.
From Figure 9b and Figure 9c, we can observe that the ADR-trained model exhibits similar output distribution as the model trained with PGD-AT (Figure 2b and Figure 2d), except that the ADR-trained model exhibits higher entropy levels in general. The reason behind this phenomenon is that ADR builds a sofer target as training labels, so the output of the ADR model will also be smoother, resulting in higher entropy levels on average.

Reviewer comment (Reproducibility):
Should the rectified labels be assigned to adversarial images only, or both clean and adversarial images? Considering that different AT methods use different targets for clean and adversarial images (e.g., PGD-AT and TRADES), this can be an important question when one would like to apply ADR to other AT methods.

Author response:
**Q: Should the rectified labels be assigned to adversarial images only, or both clean and adversarial images? Considering that different AT methods use different targets for clean and adversarial images (e.g., PGD-AT and TRADES), this can be an important question when one would like to apply ADR to other AT methods.**

The rectified labels should be assigned to both clean and adversarial images. Our observations, detailed in Section 3.2, lead us to summarize that a robust model should consistently produce outputs that align for both clean and adversarial data. In essence, within the $l_p$ norm neighborhood of a given input, all images should be associated with a singular smooth and rectified label. Consequently, we propose the replacement of both hard labels used in PGD-AT for adversarial images and TRADES for clean images with the rectified labels generated by the ADR process. This adjustment ensures a unified and consistent labeling approach across both clean and adversarial instances, aligning with our overarching objective of enhancing model robustness.