Paper: Robust NAS under adversarial training: benchmark, theory, and beyond (ICLR-2024)

Reviewer comment (Novelty):
The authors observe that the standard and robust accuracies exhibit a meaningful level of correlation. If they are, do we really need a separate robustness benchmark that includes robust accuracies? Wouldn’t searching for the optimal architecture in a conventional sense work as a good proxy and naturally result in a more robust architecture?

Author response:
**A2:**  Let us explain why we do need a separate benchmark. We agree with the reviewer that there exists a partial correlation between clean accuracy and robust accuracy, as suggested by our benchmark. This is one of the findings from our benchmark: demonstrating that the correlation is not extremely low under adversarial training, compared to the benchmark under standard training [6]. We have included the discussion in Appendix B.2 of the paper. 

Notice that before building the benchmark, we did not know the relationship between clean accuracy and robust accuracy under adversarial training. To be specific, in the existing benchmark [6] that was constructed in the same space but under standard training, the correlation was not high. This correlation was enhanced only when adversarial training was involved. This is one finding from our benchmark. Moreover, we can see a notable difference between these two benchmarks in terms of top architecture id and selected nodes, see Fig. 4 and Fig. 8. This suggests that employing the benchmark under standard training to identify a resilient architecture may not guarantee its robustness under adversarial training.

Reviewer comment (Experiments):
It would be nice to include the performance of trained architectures on out-of-distribution data and/or datasets with distribution shift in the benchmark. Incorporating these other types of robustness would make the benchmark more comprehensive, especially considering that in real deployment scenarios, the architectures will encounter generic OOD data more often than adversarially-perturbed data.

Author response:
**A1:** Thanks for the suggestion, the discussion on the AutoAttack can be found in [general response](https://openreview.net/forum?id=cdUpf6t6LZ&noteId=FRw76YCx2K) for details.

Furthermore, to make the benchmark more comprehensive in a realistic setting, we evaluate 6466 architectures in the search space under distribution shift. We choose CIFAR-10-C [3], which includes 15 visual corruptions with 5 different severity levels, resulting in 75 new metrics. In the supplementary material, we have provided the result in a JSON file. 

We add a new section Appendix F.4 to describe the evaluation and discuss the result. In Fig.7 of the revised version (or the [link](https://imgur.com/a/iVolRzr)), we show the boxplots of the accuracy. We can find that all architectures are robust towards corruption with lower severity levels. When increasing the severity levels to five, the models are no longer robust to fog and contrast architectures. Moreover, similar to robust accuracy under FGSM and PGD attacks, we can see a non-negligible gap between the performance of different architectures, which motivates robust architecture design. In Table 6 of the revised version (or the [link](https://imgur.com/a/iVolRzr)),  we show the Spearman coefficient between various accuracies on CIFAR-10-C and adversarial robust accuracy on CIFAR-10. Interestingly, we can see that the adversarial robust accuracy has a relatively high correlation with all corruptions except for contrast corruptions. As a result, performing NAS on the adversarially trained architectures in the benchmark can achieve a robust guarantee of distribution shift.

Reviewer comment (Evaluation):
In section 4.4, the authors study the correlation between the minimum eigenvalue of the NTK matrix and robust accuracy. I understand that this NTK-score is derived from theoretical resultss, but have the authors studied the relationship between other NTK-based scores (e.g., condition number [2]) and robust accuracy?

Author response:
**A4:**  According to the reviewer's suggestion, we show the correlation between accuracy and several NTK scores based on their condition number, which is calculated by using all of the network parameters following [2]. Interestingly, using the condition number of robust NTK does not improve the correlation compared to using the condition number of clean NTK, but using twice-robust NTK is still better than robust NTK. Given that the condition number can describe the curvature of NTK, and can reveal the convergence speed of the neural network [8],  it would be interesting to theoretically analyze the effect of the condition number on the robust generalization bound in the future. 

||Clean | FGSM, 3/255 | PGD, 3/255|  APGD, 3/255 | FGSM, 8/255| PGD, 8/255| APGD, 8/255|
|--------|--------|-------|-------|-------|-------|-------|-------|
| Cond($K$) | 0.739 | 0.742 | 0.739 | 0.738 | 0.765 | 0.755 | 0.748 |
|Cond($\hat{K}_{\rho}$), 3/255  | 0.505 | 0.512 | 0.509 | 0.508 | 0.546 | 0.536 | 0.531 |
|Cond($\hat{K}_{\rho}$), 8/255 | 0.721 | 0.729 | 0.725 | 0.724 | 0.762 | 0.753 | 0.747 |
| Cond($\hat{K}_{2\rho}$), 3/255  |  0.558 | 0.566 | 0.562 | 0.561 | 0.600 | 0.589 | 0.584 |
| Cond($\hat{K}_{2\rho}$), 8/255  | 0.724 | 0.730 | 0.727 | 0.726 | 0.762 | 0.752 | 0.746 |

Reviewer comment (Evaluation):
It has been well known that adversarial robustness often may occur due to gradient obfuscation and is applicable for different sparse and dense model architectures [1,2]. Thus, a discussion on that would be necessary. In specific, is there a way to benchmark based on a subnet's susceptibility to be more prone towards obfuscation?

Author response:
We are thankful to the reviewer for the suggestion of this ablation experiment. Upon the suggestion of the reviewer, we add a new section in Appendix F.3 to discuss this.

We agree with the reviewer that gradient obfuscation might be a cause for the overestimated robustness of several adversarial defense mechanisms, e.g., parametric noise injection defense [3] and ensemble defenses [6], and can exist in the defense of several architectures [2]. However, in this work, we employ vanilla adversarial training as a defense approach, which does not suffer from gradient obfuscation, as originally demonstrated in [1] and mentioned in later literature [3].


We investigate whether the sparsity of the architecture design might have an impact on gradient obfuscation. Specifically, based on the search space in Figure 1 of the paper, we group all architectures in terms of the number of operators “Zeroize’’, which can represent the sparsity of the network (the more  “Zeroize” operations, the more sparse the resulting network). Next, we select the network with the highest robust accuracy in each group and check whether this optimal architecture satisfies the following characteristics of non-gradient obfuscation [1]:
-  a) One-step attack performs worse than iterative attacks.
-  b) Black-box attacks perform worse than white-box attacks. 
-  c) Unbounded attacks fail to obtain $100\%$ attack success rate.
- d) Increasing the perturbation radius $\rho$ does not increase attack success rate.
- e) Adversarial examples can not be found by random sampling if gradient-based attacks do not.

The result below is the evaluation of these architectures under FGSM, Square attack, and PGD attack with varying perturbation radius. We can see that all of these architectures clearly satisfy a), c), and d). Regarding e), all of the networks still satisfy it because gradient-based attacks can find adversarial examples. For b), only the sparse network with 4 ``Zeroize" does not satisfy the condition. Therefore, we can see that these architectures can not be considered as suffering gradient obfuscation, which is consistent with the original conclusion for adversarial training in [1].

|Number of “Zeroize"|Clean | FGSM, 3/255 | Square, 3/255 |PGD, 3/255|  PGD, 8/255 | PGD, 16/255| PGD, 32/255| PGD, 64/255|PGD, 128/255|PGD, 255/255|
|--------|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|
| 0  |0.7946|0.6975|0.6929|0.6924|0.4826|0.1774|0.0083|0.0000|0.0000|0.0000|
| 1  |0.7928|0.6933|0.6884|0.6874|0.4798|0.1744|0.0077|0.0000|0.0000|0.0000|
| 2 | 0.7842|0.6832|0.6786|0.6779|0.4711|0.1739|0.0080|0.0000|0.0000|0.0000|
| 3 | 0.7456|0.6515|0.6482|0.6472|0.4442|0.1599|0.0079|0.0000|0.0000|0.0000|
| 4 |0.5320|0.4560|0.4427|0.4528|0.3191|0.1393|0.0193|0.0010|0.0000|0.0000|

Lastly, under the aforementioned defense mechanisms e.g., noise injection defense and ensemble defenses, some of these architectures in the search space might be more prone to gradient obfuscation. Thus, we believe it is possible to further develop robust NAS benchmarks under other defense mechanisms, but this is out of the scope of the current work.

Reviewer comment (Experiments):
Di you use ImageNet or ImageNet-16-120? As the Fig. 2 and the contribution section has mention of each.

Author response:
Following NAS-Bench-201 [4], we conduct an evaluation on ImageNet-16-120, which is a well-established benchmark. In the updated version, we revised the contribution on page 2 as follows: 

"107k GPU hours are required to build the benchmark on three datasets (CIFAR-10/100, ImageNet-16-120) under adversarial training."

Reviewer comment (Novelty):
The related work of Adversarial example generation is not up to date, the author should discuss more about the auto attacks and other contemporary attacks.

Author response:
In the revised version, we update the related work from the classical adversarial attack to up-to-date ways as follows:

To mitigate the effect of hyper-parameters in PGD and the overestimation of robustness [1], [7]  propose two variants of parameter-free PGD attack, namely, APGD_CE and APGD_DLR, where CE stands for cross-entropy loss and DLR indicates difference of logits ratio (DLR) loss. Both APGD_CE and APGD_DLR attacks dynamically adapt the step-size of PGD based on the loss at each step. Furthermore, to enhance the diversity of robust evaluation, [7] introduce Auto-attack, which is the integration of APGD_CE, APGD_DLR, Adaptive Boundary Attack (FAB) [4], and black-box Square Attack [5].

Reviewer comment (Reproducibility):
How you add noise "twice"?

Author response:
We perform one PGD/FGSM attack on the raw image data to generate the adversarial data, and then we perform the same attack on this adversarial data again. We have added such clarification in page 9 of the revised version.

Reviewer comment (Experiments):
Please extend the NTK score vs accuracy correlation with more recent and stronger attack scenarios.

Author response:
We test the correlation between the robust accuracy under APGD_CE and the various NTK metrics in CIFAR-10. On one hand, we can still see that the robust NTK enables the increasing correlation under adversarial training, which is consistent with our observation in the original version for FGSM and PGD attacks. On the other hand, we can see the correlation decrease only ~0.5% for APGD accuracy compared to PGD accuracy. We have updated the result on the APGD attack In the revised version, see Fig.5 in page 9 or [the anonymous link](https://imgur.com/a/WcnEzOR).

|Clean | FGSM, 3/255 | PGD, 3/255|  APGD, 3/255 | FGSM, 8/255| PGD, 8/255| APGD, 8/255|
|--------|-------|-------|-------|-------|-------|-------|
| KNAS, clean | 0.527 | 0.546 | 0.543 | 0.541 | 0.601 | 0.598 | 0.593 |
| KNAS, PGD, 3/255 | 0.533 | 0.552 | 0.550 | 0.548 | 0.608 | 0.604 | 0.599 |
| KNAS, PGD, 8/255 | 0.540 | 0.559 | 0.556 | 0.554 | 0.614 | 0.610 | 0.605 |
| KNAS, twice-PGD, 3/255| 0.537 | 0.556 | 0.553 | 0.551 | 0.611 | 0.608 | 0.602 |
| KNAS, twice-PGD,  8/255 | 0.549 | 0.568 | 0.565 | 0.564 | 0.622 | 0.619 | 0.615 |

Reviewer comment (Theory):
The paper lacks a detailed explanation and justification for the necessity of employing twice perturbation, as mentioned at the end of page 7. It remains unclear why robust NTK requires a double perturbation when conventional adversarial training typically examines generalization from a single perturbation.

Author response:
**A1:** We explain the usage of twice-robust NTK both theoretically and empirically.

- **Theoretically**, In the neural tangent kernel theory, we need to define a linearized neural network around the initialization of its weight. Specifically, in equation 15, we consider the linearized network
$F_{\bf W^{[1]},\bf W^\star}(\bf x) :=
f(\bf x,\bf W^{[1]})  + \langle
(1-\beta)\nabla f_{\bf W}(\bf x,\bf W^{[1]}) +
\beta \nabla f_{\bf W}(\mathcal{A}^*_\rho(\bf x,\bf W^{[1]}),\bf W^{[1]})  , \bf W^\star - \bf W^{[1]} \rangle$.
Then at the end of the proof of Theorem 1, we have
$F_{\bf W^{[1]},\bf W^\star}(\mathcal{A}^*_\rho(\bf x_i,\bf W^{[1]}))=    f(\mathcal{A}^*_\rho(\bf x_i,\bf W^{[1]}),\bf W^{[1]})  + \langle
   (1-\beta)\nabla f_{\bf W}(\mathcal{A}^*_\rho(\bf x_i,\bf W^{[1]}),\bf W^{[1]}) +  \beta \nabla f_{\bf W}(\mathcal{A}^*_\rho(\mathcal{A}^*_\rho(\bf x_i,\bf W^{[1]}),\bf W^{[1]}),\bf W^{[1]})  , \bf W^\star - \bf W^{[1]} \rangle. $
As a result, the Jacobian is given by
$\widetilde{\bf J}\_\mathrm{all}  := (1-\beta)\hat{\bf J}\_\rho+\beta \hat{\bf J}\_{2\rho}$, where the twice-NTK appears.


-  **Empirically**, we have demonstrated the **superiority of twice-robust NTK when compared to the (single perturbation) robust NTK in Figure 5**, which shows that the correlation between the accuracy ranking and twice-robust NTK is higher than the robust NTK. 

We agree with the reviewer that the intrinsic motivation of twice perturbation is not as straightforward as classical adversarial training works in the single perturbation style. However, this twice perturbation approach shares a similar style with adding noise before the adversarial attack [5]. More precisely, [5] demonstrate that augmenting clean data with noise before executing adversarial attack has the benefit of avoiding catastrophic overfitting, compared to a classical adversarial attack on the clean data. The proposed Noise-FGSM achieves promising performance across several datasets (e..g, CIFAR-10, and SVHN)  and mitigates robust overfitting. This demonstrates that introducing a stronger attack is beneficial to obtain better performance in practice. We believe that the development of algorithms based on the twice perturbation style will be a viable and beneficial avenue.

Reviewer comment (Evaluation):
The paper employs an evaluation metric that is not standard, raising questions about its normalization. The last part of page 4, in the dataset paragraph, should clarify whether the metric is normalized before the attack and then inputted into the model. Additionally, it would be beneficial if the authors specified whether the same perturbation radius (32x32) used for ImageNet was applied during training.

Author response:
**A2:** In this work, the data is normalized with zero-mean and unit-variance before the attack. We have added such clarification in the dataset paragraph in the revised version. We emphasize that **we are using standard tools and evaluation metrics**, as used also in several ML papers [4,11] and open-source robustness library [3] developed by MadryLab. Regarding the implementation detail, we employ the Torchattacks library [7]. Lastly, we will release the pre-trained weight of 6444* 3* 3 architectures to encourage the practitioners to evaluate with more metrics.

We emphasize the same perturbation radius $\rho = 8/255$ is used for ImageNet in the training procedure of the revised version.