Paper: OODRobustBench: benchmarking and analyzing adversarial robustness under distribution shift (ICLR-2024)

Reviewer comment (Experiments):
Only one seen attack method (MM5) has been used for the evaluation, which is not practical for robustness analysis. Some findings are expected or have already been studied in existing works.

Author response:
> Only one seen attack method (MM5) has been used for the evaluation, which is not practical for robustness analysis.

As mentioned in Appendix B.2.1, we use the MM5 attack [5] because of its strong trade-off between efficiency and effectiveness. MM5 has about 0.5% lower attack success rate than the commonly used AutoAttack [6], but it is **32× faster** (discussed in Appendix B.2.1). This means MM5 is a much more practical choice, especially for our large-scale analysis of hundreds of models in OODRobustBench. Please feel free to clarify if this does not address your question.

[5] Gao et al., *Fast and Reliable Evaluation of Adversarial Robustness with Minimum-Margin Attack*, ICML 2022.

[6] Croce and Hein, *Reliable Evaluation of Adversarial Robustness with an Ensemble of Diverse Parameter-free Attacks*, ICML 2020.

Reviewer comment (Evaluation):
Some works [1, 2, 3] studied the linear correlation between ID performance and OOD performance, the authors need to add some related discussions. Instead of the finding of the correlation between ID accuracy (robustness) and OOD accuracy (robustness) that is expected, the finding that there is a weak correlation between ID accuracy and OOD robustness for ImageNet is more attractive and needs a more detailed explanation.

Author response:
> Some works [1, 2, 3] studied the linear correlation between ID performance and OOD performance, the authors need to add some related discussions.

[1,2,3] focus on predicting the OOD accuracy of non-robust models when OOD data is *unlabeled*, while our work predicts the OOD adversarial robustness of robust models based on labeled OOD data. We quickly summarize these works below: 

- [1] predicts OOD accuracy based on the linear trend of the prediction agreement of two pairs of models.
- [2] learns a threshold for prediction score on labeled ID data and predicts OOD accuracy using the fraction of OOD examples whose prediction score is above the threshold.
- [3] predicts OOD accuracy based on a linear trend, learned on the synthetic datasets, between accuracy and distribution shift (measured by Fréchet distance).

The linear trend of adversarial robustness discovered in our work suggests that these methods are potentially applicable to predicting OOD adversarial robustness. We summarize our new findings compared to this line of work in [shared response](https://openreview.net/forum?id=RnYd44LR2v&noteId=mJy1d8sGtg). We have added this discussion to our paper in Appendix A.

> …the finding that there is a weak correlation between ID accuracy and OOD robustness for ImageNet is more attractive and needs a more detailed explanation
> 

There is no correlation between ID accuracy and OOD robustness for CIFAR-10 but a weak correlation for ImageNet because of the *inferior* models on CIFAR-10 which have high accuracy yet poor robustness. These models cause OOD robustness to not consistently increase with the ID accuracy. These inferior models are mainly produced by some of our custom training receipts and take a considerable proportion of our CIFAR-10 model zoo, whereas the model zoo of ImageNet is dominated by ones from public sources. We have added this discussion in section 4.1 in our paper.

Reviewer comment (Experiments):
Even though this paper provides the first benchmark for OOD robustness evaluation, there are concerns that need to be addressed: the limited seen attack methods used, some findings already revealed by existing works, and lack of the study of OOD generalization methods.

Author response:
> Only one seen attack method (MM5) has been used for the evaluation, which is not practical for robustness analysis.

As mentioned in Appendix B.2.1, we use the MM5 attack [5] because of its strong trade-off between efficiency and effectiveness. MM5 has about 0.5% lower attack success rate than the commonly used AutoAttack [6], but it is **32× faster** (discussed in Appendix B.2.1). This means MM5 is a much more practical choice, especially for our large-scale analysis of hundreds of models in OODRobustBench. Please feel free to clarify if this does not address your question.

[5] Gao et al., *Fast and Reliable Evaluation of Adversarial Robustness with Minimum-Margin Attack*, ICML 2022.

[6] Croce and Hein, *Reliable Evaluation of Adversarial Robustness with an Ensemble of Diverse Parameter-free Attacks*, ICML 2020.

> Some works [1, 2, 3] studied the linear correlation between ID performance and OOD performance, the authors need to add some related discussions.

[1,2,3] focus on predicting the OOD accuracy of non-robust models when OOD data is *unlabeled*, while our work predicts the OOD adversarial robustness of robust models based on labeled OOD data. We quickly summarize these works below: 

- [1] predicts OOD accuracy based on the linear trend of the prediction agreement of two pairs of models.
- [2] learns a threshold for prediction score on labeled ID data and predicts OOD accuracy using the fraction of OOD examples whose prediction score is above the threshold.
- [3] predicts OOD accuracy based on a linear trend, learned on the synthetic datasets, between accuracy and distribution shift (measured by Fréchet distance).

The linear trend of adversarial robustness discovered in our work suggests that these methods are potentially applicable to predicting OOD adversarial robustness. We summarize our new findings compared to this line of work in [shared response](https://openreview.net/forum?id=RnYd44LR2v&noteId=mJy1d8sGtg). We have added this discussion to our paper in Appendix A.

> Section 4.3 analyzes the upper limit of OOD performance using the linear correlation. However, the authors did not consider factors that could further improve the limit such as OOD generalization methods which makes the conclusion not that convincing.
> 

We note that our goal is to benchmark and analyze the OOD adversarial robustness of existing robust models. Thus, **the drawn conclusion on linear trend and upper limit mainly applies to existing AT methods and enhancement techniques such as extra data, model scaling, etc.** Since OOD generalization methods, such as, IRM [7] and REx [8], are rarely used in the existing AT methods, their robustness to attacks is unclear and so not reflected in our conclusions. In fact, one of the implications of this work is to hopefully spur the future development of new approaches that can cope with distribution shifts better and achieve OOD robustness beyond our prediction. We have added this discussion to our paper in Appendix H.1.

[7] Arjovsky et al. Invariant Risk Minimization. arxiv 2020.

[8] Kruger et al. Out-of-Distribution Generalization via Risk Extrapolation. ICML. 2021.

Reviewer comment (Novelty):
This work is a straightforward combination of existing evaluations with little new contribution or understanding. For the evaluation with data shifts, compared to existing works on evaluation with OOD datasets, this work simply adds existing adversarial attacks. Methods and conclusions are almost the same as previous works (Taori et al., 2020, Miller et al., 2021) on the linear trend.

Author response:
Technically yes. Our work focuses on standardizing the evaluation of OOD adversarial robustness and understanding this generalization, instead of proposing new OOD datasets and attacks. Building benchmarks based on the existing datasets and attacks is the same as the construction of other public benchmarks like RobustBench [1], RobustART [2], and MultiRobustBench [3]. Please also refer to [shared response](https://openreview.net/forum?id=RnYd44LR2v&noteId=mJy1d8sGtg) for a summary of our contributions. 

Reference:

[1] croce et al., RobustBench: a standardized adversarial robustness benchmark, NeurIPS 2021

[2] Tang et al., RobustART: Benchmarking Robustness on Architecture Design and Training Techniques, T-PAMI 2022

[3] Dai et al., MultiRobustBench: Benchmarking Robustness Against Multiple Attacks, ICML 2023

Please see [shared response](https://openreview.net/forum?id=RnYd44LR2v&noteId=mJy1d8sGtg) on the top.

Reviewer comment (Evaluation):
The discussion statement "Surprisingly, VR also clearly boosts effective robustness under dataset shift even though not designed for dealing with these shifts" and "Advanced model architecture significantly boosts robustness and effective robustness under both types of shift over the classical architecture" is not convincing. The reviewer disagrees that these factors "significantly boost" effective robustness, as the gains are only around 1%~2%, which are not larger than the normal variations between different models in the linear fit (Figure 1).

Author response:
First, when we say boost effective robustness we mean improving effective robustness over the baseline (i.e., PGD for VR and ResNet for ConvNeXt, Swin, and ViT) because this shows the benefit of switching from one method to another. **It may be less meaningful to look at the absolute value of effective robustness without comparing against the baseline** because the baseline may have a non-zero, or even negative, effective robustness. If measured so, the improvement is about 2% under OOD_d for VR and about 1.5%, 1.3%, 2.75% under OOD_d and 3%, 3%, 4.5% under OOD_t for ConvNeXt, Swin, and ViT respectively. 

Second, the standard deviation is 0.77% for CIFAR-10 under OOD_d (corresponding to the setting of VR) and 0.96% and 1.55% for ImageNet under OOD_d and OOD_t (corresponding to the setting of architecture). **The above improvements are all greater than one standard deviation.** In addition, we have also done a $t$-test and the $p$-value is $< 10^{-3}$ for VR under OOD_d and 0.009, 0.023, $< 10^{-3}$ under OOD_d, and 0.008, 0.008, $< 10^{-3}$ under OOD_t for ConvNeXt, Swin, and ViT respectively. **All** $p$**-values are much less than the usual significance level of 0.05 suggesting the improvement is significant**.

Reviewer comment (Writing):
In Section 5, the authors rename the existing effective robustness from previous works (which have been widely adopted) into "effective accuracy" while redefining "effective robustness" to be the effective robustness under adversarial attacks. This is confusing. The reviewer suggests that the authors keep the original definition for effective robustness but give a new name for the particular effective robustness in this work (e.g., adversarial effective robustness).

Author response:
Thanks for your suggestion. We have renamed it in the paper as suggested.

Reviewer comment (Novelty):
Similar analyses are already present in prior works, although on a (sometimes much) smaller scale, and then the results are not particularly surprising. For example, the robustness of CIFAR-10 models on distributions shifts (CIFAR-10.1, CINIC-10, CIFAR-10-C, which are also included in this work) was studied on the initial classifiers in RobustBench (see [Croce et al. (2021)](https://arxiv.org/abs/2010.09670)), showing a similar linear correlation with ID robustness. Moreover, [A, B] have also evaluated the robustness of adversarially trained models to unseen attacks.

Author response:
We acknowledge that there exist previous works that evaluate adversarial robustness under dataset shifts and threat shifts. However, as pointed out by the reviewer, the scale of these previous works is rather small. For instance, RobustBench observes linear correlation only for three shifts on CIFAR-10 based on 39 models with either ResNet or WideResNet architectures. In such a narrow setting, it is actually neither surprising to see a linear trend nor reliable for predicting OOD performance. By contrast, our conclusion is derived from much more shifts on CIFAR-10 and ImageNet based on 706 models. Importantly, our model zoo covers a diverse set of architectures, robust training methods, data augmentation techniques, and training set-ups. This makes our conclusion more generalizable and the observed (almost perfect) linear trend much more significant.

Similarly, [A] and [B] only test a few models under threat shifts. Those methods are usually just the baseline AT method plus different architectures or the relevant defenses, e.g., jointly trained with multiple threats. It is unclear how the state-of-the-art robust models perform under threat shifts. By conducting a large-scale analysis, we find that **those SOTA models generalize poorly to other threats while also discovering several methods that have relatively inferior ID performance but superior OOD robustness under threat shift.** Our analysis therefore facilitates future works in this direction by identifying what techniques are ineffective and what are promising. 

We have added this discussion in appendix A, as well as the reference to [B], to our paper.

Please also refer to [shared response](https://openreview.net/forum?id=RnYd44LR2v&noteId=mJy1d8sGtg) for a summary of our new findings compared to the previous works.

Reviewer comment (Experiments):
Many non $\ell_p$ attacks have been proposed (see e.g. [A, B]). Is there a specific reason for the choice of those used in the paper?

Author response:
Ideally, we would like to include more non-Lp attacks to understand the generalization of adversarial robustness more thoroughly. However, adding attacks will cause substantial computational costs because of our large model zoo. We choose the current ones because they are sufficiently effective, cover perceptible (ReColor, StAdv) and imperceptible (PPGD, LPA) attacks, and have been widely used (each of them is cited 100+). Nevertheless, we are happy to include other non-Lp attacks if any of your interest. We have added this discussion to our paper in Appendix B.2.2.