Paper: Rethinking Model Ensemble in Transfer-based Adversarial Attacks (ICLR-2024)

Reviewer comment (Experiments):
The description of the experimental part could be improved. Take "Results on normal models" as an example. In the experiment setup, the authors choose MI, VMI and SSA to validate the effectiveness of proposed CWA. However, in "Results on normal models", the authors only discuss MI, and ignore the VMI and SSA. I think the authors should pay more attention on the recent SOTA (e.g., SSA), since MI was proposed in 2018 (5 years ago).

Author response:
Thank you for the suggestion. We have improved the description of the experimental results and discussed the results of VMI and SSA in the revision. The detailed discussion is as follows.

> By integrating our proposed CWA with recent state-of-the-art attacks VMI and SSA, the resultant attacks VMI-CWA and SSA-CWA achieve a significant level of attacking performance. Typically, SSA-CWA achieves more than 99\% attack success rates for most normal models. VMI-CWA and SSA-CWA can also outperform their vanilla versions VMI and SSA by a large margin. The results not only demonstrate the effectiveness of our proposed method when integrating with other attacks, but also prove the vulnerability of existing image classifiers under strong transfer-based attacks.

Reviewer comment (Evaluation):
How to explain the significant performance degradation of VMI-CWA compared to VMI in Tab 1 when the target models are Swin-S and Max ViT-T?

Author response:
We are sorry that we showed the wrong results of VMI-CWA in Table 1. In our submitted code, we implemented two versions of VMI-CWA. The first one is based on Algorithm 4 in Appendix B.3, in which we adopt the gradients of VMI and perform iterative updates based on our algorithm. The second version is slightly different, which adopts the gradients of our method to perform updates based on VMI. In practice, we observed that the first version performs better, but we showed the results of the second version in Table 1. In the revision, we have corrected the results of VMI-CWA in Table 1. Now the performance of VMI-CWA is better than VMI for the transformers Swin-S and MaxViT-T.

Reviewer comment (Writing):
The intuition behind SAM and CSE is not clear enough. Equation (4) and (5) is confusing for those readers not familiar with this area.

Author response:
Thank you for pointing this out. SAM is an effective method to acquire a flatter landscape, which is formulated as a min-max optimization problem, in a similar way to adversarial training. The inner maximization aims to find a direction along which the loss changes more rapidly;
while the outer problem minimizes the loss at this direction to improve the flatness of loss landscape. We have added the introduction of SAM in the beginning of Section 3.3. 

For CSE, to maximize the cosine similarity between the gradients of different models, we are inspired by Nichol et al. [1] to develop a first-order algorithm. Nichol et al. propose a meta-learning algorithm that iteratively performs gradient updates for each task, and they theoretically show that this procedure can increase the inner product between gradients of different tasks. We extend this method to also increase the cosine similarity between the gradients of different models, with theoretical analyses in Appendix B.2.

Reviewer comment (Evaluation):
In Section 3.1, the authors mentioned that "the goal of transfer-based attacks is to craft an adversarial example $x$ that is misclassified by all models in $\mathcal{F}$". However, fooling all target models seems to be an impossible job in practice. Besides, the experiments in this paper cannot support this claim.

Author response:
Thank you for pointing this out. We have revised our paper to avoid this overclaim. Now the text becomes

> Given a natural image $\boldsymbol{x}_{nat}$ and the corresponding label $y$, transfer-based attacks aim to craft an adversarial example $\boldsymbol{x}$ that could be misclassified by the models in $\mathcal{F}$.

Reviewer comment (Evaluation):
The perturbation in Figure 3 is not imperceptible to humans.

Author response:
We are sorry that the first image in Figure 3 was placed wrongly -- we used the adversarial image with $\epsilon=32/255$, such that this image is more perceptible. In the revision, it is changed to the correct image with $\epsilon=16/255$. The perturbations are visually similar to those generated by other attack methods with the same perturbation budget.

Reviewer comment (Presentation):
I felt that the paper was bit hard to follow. Section 3 mixes a lot of prior results with proposed ones. Thus it makes it a bit hard to distinguish original contributions vs reusing prior results. For instance, it would greatly improve the readability if SAM was properly explained prior to this section (or as a subsection). Figures 1 and 2 are not very informative (see Questions), they lack axis and/or labels. It would also improve the quality of the paper if the authors include a mathematical description of some of the prior ensemble attack methods (e.g. is it simply PGD on the averaged logits?), to better understand the contrast between them and the proposed method.

Author response:
Thank you for pointing out the issues. We further improve the writing to address them in the revision. Below are the details.

- **Improve the readability of SAM.** We have provided a brief introduction of SAM in the beginning of Section 3.3, to provide the background knowledge of SAM and explain why it can be used for our problem. 

- **Figures 1 and 2 are not very informative.** We have updated Figures 1 and 2 to be more informative, including adding the axes and labels, changing the colors, and specifying the symbols.

- **Include a mathematical description of the prior ensemble attacks.** We have added a new subsection 3.1 in the revision to introduce the backgrounds of adversarial attacks.

We hope that the new revision can be easier to follow. We will further improve the paper in the final.

Reviewer comment (Experiments):
Despite the compactness of Algorithm 1, there is a lack of analysis on its complexity. How does it compare (in terms of wall-clock time) to prior attacks or vanilla PGD-style attacks?

Author response:
Thank you for the suggestion. We discussed the computational efficiency in Section 4.4 and Appendix D.4. We further report the actual runtime of different attack methods in Table D.2.  Our method incurs only a slight
computational overhead compared to the baseline attacks. This demonstrates that our method can
serve as a plug-and-play algorithm to effectively and efficiently enhance transfer attack performance. As also discussed in Section 4.4, given a fixed computation overhead, we can use a smaller number of attack iterations in our method, which also leads to better results, as shown in Figure 4(c).

Reviewer comment (Presentation):
In Fig 1, it is not clear what is being plotted, and the different colors are not well contrasted.

Author response:
- **What is being plotted in Figure 1.** Figure 1 shows an illustration of common weakness, where we plot the synthetic loss curves of different models w.r.t. the input $\boldsymbol{x}$. We show that the generalization performance is correlated with the flatness of loss landscape and the distance to the local optimum of each model.

Reviewer comment (Writing):
In Theorem 3.1, there is a typo, should be $||H_i||_F$?

Author response:
- **Typos in Sec. 3.1, Theorem 3.1, Sec. 3.2.** Thank you for pointing them out. We have corrected them in the revision.

Reviewer comment (Writing):
In Sec 3.2: " There are some researches ..." typo

Author response:
- **Typos in Sec. 3.1, Theorem 3.1, Sec. 3.2.** Thank you for pointing them out. We have corrected them in the revision.

Reviewer comment (Presentation):
In Fig 2, what are $x_t$, $x_t^f$, .... The symbols are not defined.

Author response:
- **The symbols in Figure 2 are not defined.** We have revised the caption to link the symbols to the equations in the main text.

Reviewer comment (Reproducibility):
In Alg1, does the order of the classifiers in the for loop matter? if so how is it currently chosen?

Author response:
- **The order of classifiers in Alg. 1.** The order of classifiers is randomly set and fixed during the optimization process.

Reviewer comment (Presentation):
In Table 1, some of the highlighted numbers are not the best performing numbers, for instance the FastAT RN50 row.

Author response:
- **Some highlighting numbers are not the best in Table 1.** Thank you for pointing this out. We carefully checked the results and marked the best results in bold in the revision.

Reviewer comment (Experiments):
Can the proposed method be also applied in the white-box setting? i.e. assume $\mathcal{F} = [f]$ and simply attack the classifier. Have the authors experimented with something like this?

Author response:
- **Can the method be applied in the white-box setting.** Yes, our method can be applied in the white-box setting. But when there is only one white-box model, our method would degenerate to PGD or MI since we focus on ensemble-based attacks. Therefore, we further show the attack performance of our method against the six white-box models in the ensemble, as detailed in Section 4.1, and compare with PGD-10 and MI. The results are shown below.

|         | Epsilon | ResNet18 | ResNet34 | ResNet50 | ResNet101 | ResNet50 (Salman et al. (2020)) | XCiT-S12 (Debenedetti et al. (2023)) |
|---------|:-------:|:--------:|:--------:|:--------:|:---------:|:-------------------------------:|:-----------------------------------:|
| PGD     | 16/255  |   98.1   |   98.3   |   98.2   |   98.4    |              34.6               |                19.8                 |
| MI | 16/255  |   98.5   |   98.6   |   98.4   |   98.5    |              41.9               |                27.0                 |
| MI-CWA  | 16/255  |  100.0   |  100.0   |  100.0   |   100.0   |              91.1               |                92.2                 |
| PGD     |  4/255  |   96.2   |   95.9   |   96.1   |   96.2    |              27.2               |                15.5                 |
| MI |  4/255  |   97.0   |   96.8   |   97.1   |   97.4    |              28.0               |                16.2                 |
| MI-CWA  |  4/255  |  100.0   |  100.0   |  100.0   |   100.0   |              47.0               |                39.1                 |

It can be seen that our method also leads to higher attack success rates over PGD and MI with a small number of attack iterations. It indicates that our method converges much faster than the baselines since it can align the gradients of different models.

Reviewer comment (Experiments):
In table 1, the author only provides the results of CWA in combination with other methods. Is there results for CWA only, and how well it performs.

Author response:
***Question 1: The performance of CWA only.***

Thank you for the suggestion. We further conduct experiments of SAM, CSE, and CWA without integrating with other methods. The detailed results are shown in Table C.5 in Appendix C.7. The average performance of different methods over all black-box models are shown below.

| Method | FGSM | BIM | MI | SAM | CSE | CWA | MI-SAM | MI-SCE | MI-CWA | 
| :-----: | :-----: | :-----: | :-----: | :-----: | :-----: | :-----: | :-----: | :-----: | :-----: | 
| Attack success rate (\%) | 50.89 | 63.71 | 71.65 | 73.38 | 85.85 | 82.55 | 77.92 | 90.68 | 90.76 |

It can be seen that SAM, CSE, and CWA outperform the baselines FGSM, BIM, and MI by a large margin. However, their performance is inferior to their counterparts with MI, indicating that our methods and MI are orthogonal and can be integrated together to further improve the performance.

Reviewer comment (Experiments):
As an ablation study, the author might want to try different norm decomposition, e.g. the operator norm of H.

Author response:
***Question 2: Different norm decomposition.***

Thank you for the suggestion. It is true that we can decompose the second-order term $(\boldsymbol{x}-\boldsymbol{p}_i)^\top \boldsymbol{H}_i (\boldsymbol{x}-\boldsymbol{p}_i)$ with different norms (see Appendix A.1 for a more general result). However, with a different norm of $\boldsymbol{H}_i$, it is hard to interpret the meaning of minimizing this norm, and we can hardly associate it with generalization/transferability of adversarial examples. Besides, we need to calculate the third-order derivative of the loss function w.r.t. input $\boldsymbol{x}$, which would be intractable for deep neural networks. Therefore, it is hard to deal with other norms of $\boldsymbol{H}_i$, and we only focus on the Frobenius norm of  $\boldsymbol{H}_i$ with a clear connection to loss flatness and an alternative method based on sharpness aware minimization for optimization.

Reviewer comment (Novelty):
The author should clarify the novelty compared to the previous methods. Specially, MI, VMI, SSA are all existing proposed attacks. The sharpness aware minimization techniques have been previously used.

Author response:
***Question 3: Clarify the novelty compared to the previous methods.***

The main novelty of this paper lies in a new adversarial attack method from the perspective of model ensembling. Specifically, we use a quadratic approximation of the loss function and discover that the second-order term can be upper bounded by the product of the Frobenius norm of the Hessian matrix and the distance to the local optimum of each model. Therefore, we propose a common weakness attack to minimize the upper bound with great effectiveness. The existing attacks (e.g., MI, VMI, SSA) mainly focus on developing optimization algorithms or data augmentation techniques to improve transferability, while do not consider model ensembling in adversarial attacks. The sharpness aware minimization (SAM) is commonly used to train deep neural networks, but we extend it to adversarial attacks to improve the transferability based on our analysis.

Reviewer comment (Evaluation):
Can the author explain why the model is able to achieve significantly more effective attacks compared the existing methods for the adversarial trained model? I am mainly concerned about the fairness of the evaluation. Also, can the author explain how the AT model trained for ensembles in their experiments?

Author response:
***Question 4: Why is the method more effective for adversarial trained model?***

The main reason is that our method can better exploit the vulnerabilities of different surrogate models in the ensemble by finding their common weakness. As we included two adversarially trained models (which are publicly available by [1,2]) in the ensemble, the adversarial examples generated by our method can better transfer to attack other black-box adversarially trained models. But for baseline attacks that directly average different surrogate models, they tend to attack easier models (i.e., normally trained models) [3], such that the useful gradient information would be overwhelmed by these normal models, leading to inferior results for adversarially trained models.

**Reference:**

[1] Salman et al., Do adversarially robust imagenet models transfer better? NeurIPS 2020.

[2] Debenedett et al., A light recipe to train robust vision
transformers. IEEE Conference on Secure and Trustworthy Machine Learning, 2023.

[3] Dong et al., Discovering adversarial examples with momentum, https://arxiv.org/abs/1710.06081v1.