Paper: On the Vulnerability of Adversarially Trained Models Against Two-faced Attacks (ICLR-2024)

Reviewer comment (Novelty):
The background and motivation are not convincing to me. The authors may make it more motivating, and probably, look at the previous work (Tao, 2022b) since it is very convincing.

Author response:
**W1: The background and motivation are not convincing to me.**

We have further enriched our background and motivation, and highlighted these changes in the introduction section of the paper. These can be summarized as follows.

Due to the high probability of failures of model deployment in practical applications, it is necessary for practitioners to conduct pre-deployment verification and post-deployment long-term verification to enhance the success rate of deployment [1], especially in safety-critical tasks. Therefore, this work explores the existence of verification-time threats that hypocritically fool the robustly trained models by slightly perturbing the verification data.

[1] Andrei Paleyes, Raoul-Gabriel Urma, and Neil D. Lawrence. Challenges in deploying machine learning: A survey of case studies. ACM Computer Survey, 55(6), 2022.

Reviewer comment (Experiments):
The experiments for the countermeasure (enlarging the budget) are limited. The authors may need to add more experiments for it.

Author response:
**W2: The experiments for the countermeasure (enlarging the budget) are limited. The authors may need to add more experiments for it.**

Thanks for the suggestion. We have conducted additional experiments to supplement the experimental results.

1) We trained ResNet-18 on CIFAR-100, using PGD-AT with different budgets under $\ell_{\infty}$ norm: 

| $\epsilon$ | Clean |   $\ $ TF    | $R_{\text{nat}}(f_{\theta}, D)$ | $R_{\text{adv}}(f_{\theta}, D^{+})$ | $R_{\text{tf}}(f_{\theta}, D^{\text{cnr}})$ |
| ------- | :-----: | :-----: | :-----: | :------: | :--------: |
| 8/255  | 28.85 | 55.31 | 43.49 | 48.94 | 95.66 |
| 12/255 | 30.53 | 51.25 | 48.38 | 40.85 | 98.24 |
| 14/255 | 30.10 | 46.68 | 52.95 | 36.02 | 97.82 |
| 16/255 | 29.20 | 42.85 | 56.93 | 32.20 | 98.41 |
| 18/255 | 27.59 | 38.32 | 61.52 | 28.30 | 98.53 |
| 20/255 | 26.42 | 35.26 | 64.68 | 25.19 | 99.32 |

2) We trained ResNet-18 on CIFAR-10, using THRM [1] with different $\lambda$ under $\ell_{\infty}$ norm:

| $\lambda$ | Clean |   $\ $ TF    | $R_{\text{nat}}(f_{\theta}, D)$ | $R_{\text{adv}}(f_{\theta}, D^{+})$ | $R_{\text{tf}}(f_{\theta}, D^{\text{cnr}})$ |
| ------- | :-----: | :-----: | :-----: | :------: | :--------: |
| 1    | 4.53  | 39.55 | 5.56  | 95.20 | 38.95 |
| 5    | 23.87 | 75.54 | 8.75  | 73.84 | 76.68 |
| 10   | 32.75 | 79.5  | 11.83 | 62.85 | 84.35 |
| 20   | 35.85 | 73.14 | 9.79  | 60.25 | 68.59 |
| 40   | 43.22 | 76.03 | 20.59 | 45.57 | 90.66 |
| 60   | 43.66 | 73.02 | 24.56 | 42.12 | 92.38 |
| 80   | 41.92 | 71.98 | 25.22 | 43.94 | 91.47 |
| 100  | 43.9  | 69.51 | 29.08 | 38.09 | 94.78 |

3) We trained ResNet-18 on CIFAR-100, using THRM with different $\lambda$ under $\ell_{\infty}$ norm:

| $\lambda$ | Clean |   $\ $ TF    | $R_{\text{nat}}(f_{\theta}, D)$ | $R_{\text{adv}}(f_{\theta}, D^{+})$ | $R_{\text{tf}}(f_{\theta}, D^{\text{cnr}})$ |
| ------- | :-----: | :-----: | :-----: | :------: | :--------: |
| 1    | 8.15 | 47.96 | 23.27 | 89.38 | 58.05 |
| 5    | 13.53 | 56.46 | 25.07  | 81.94 | 69.92 |
| 10   | 20.75 | 55.10 | 28.09 | 71.14 | 67.14 |
| 20   | 24.95 | 52.75 | 31.58 | 63.53 | 63.95 |
| 40   | 26.13 | 50.66 | 34.86 | 59.89 | 62.88 |
| 60   | 26.51 | 49.56 | 36.99 | 57.93 | 63.15 |
| 80   | 25.91 | 47.09 | 38.70 | 57.73 | 59.85 |
| 100  | 25.98 | 46.16 | 39.51 | 57.05 | 58.48 |

[1] Lue Tao, Lei Feng, Jinfeng Yi, and Songcan Chen. With false friends like these, who can notice mistakes? In AAAI, 2022b.

Reviewer comment (Novelty):
The contribution of this paper is limited because the content is very similar to the previous work (Tao, 2022b), but changes from clean test samples to adversarial test samples. The authors may focus more on the countermeasure.

Author response:
**W3: The contribution of this paper is limited because the content is very similar to the previous work (Tao, 2022b), but changes from clean test samples to adversarial test samples. The authors may focus more on the countermeasure.**

As you mentioned, the previous work (Tao, 2022b) focused on **accuracy**, while our work focuses on **robustness** during the verification phase. Hence there is a clear difference, which makes them two entirely different tasks. Our contributions to the task of two-faced attacks are completely new. In addition, we have made some new findings regarding two-faced attacks: 

1) Models with different architectures and adversarial training methods are susceptible to two-faced attacks, leading to falsely elevated verification robustness; 
2) Models with higher adversarial robustness tend to be more vulnerable to two-faced attacks. 

Therefore, two-faced attacks are pervasive and tricky, and cannot be simply ignored.

In terms of countermeasures, we have included more experimental results: 
1) Experimental results of PGD-AT with different budgets under $\ell_{\infty}$ norm on CIFAR-100; 
2) Experimental results of THRM with different $\lambda$ under $\ell_{\infty}$ norm on CIFAR-10; 
3) Experimental results of THRM with different $\lambda$ under $\ell_{\infty}$ norm on CIFAR-100. 

We presented the relevant results in W2 and have added them to the appendix of the paper.

We hope that our responses and the updated experiments have addressed your concerns. We are always willing to discuss and address any further questions.

Reviewer comment (Theory):
In Appendix A, I do know how you derive from E[1(f(x'') = y) * 1(f(x_{adv} \neq y)] to E[1(f_{adv} \neq y)] * E[1(f(x'') = y)]. It is in the lines 7 and 8 if you count the line R_{rhpy}(f,D) as line 1.

Author response:
**Q1: In Appendix A, I do not know how you derive from E[1(f(x'') = y) * 1(f(x_{adv} \neq y)] to E[1(f_{adv} \neq y)] * E[1(f(x'') = y)]. It is in the lines 7 and 8 if you count the line R_{rhpy}(f,D) as line 1.**

Thanks so much for raising this point, which helped us identify an oversight in our derivation process and avoid the omission of a component in the final result. Fortunately, this has no impact on our conclusions. We have made corrections and uploaded the revised version. Thanks again for the reviewer's carefulness!

Reviewer comment (Experiments):
There may be an overemphasis on two-faced attacks, which might mask other important security issues that need to be taken seriously.

Author response:
We agree with the reviewer's perspective. The primary research objective of this paper is to uncover the threats posed by two-faced attacks, but it is undeniable that other significant security issues should not be overlooked. Balancing the risks associated with two-faced attacks and other security concerns is indeed a topic worthy of in-depth exploration. For instance, integrating the two-faced risk into existing adversarial training frameworks merits thorough investigation. However, as discussed in the response to Q1, the resulting optimization process (i.e., alternately generating two-faced examples and conducting adversarial training) is highly time-consuming, especially given vast training data. Therefore, there is a need for efficiency optimization to address this issue.

Reviewer comment (Reproducibility):
A lack of clarity in the procedures for experiments may make them difficult to repeat and inhibit future studies.

Author response:
Thanks for raising this concern. We have provided more information in the experimental procedure section in Appendix B and have uploaded the revised version.

Reviewer comment (Evaluation):
There can be a gap in comprehensive risk management techniques if the paper does not include a comprehensive explanation of how to balance adversarial risks, such as two-faced risk, against other risks.

Author response:
We agree with the reviewer's feedback. The issue of balancing these risks is indeed worth investigating. While our primary focus in this paper is to uncover the threats posed by two-faced attacks, we presented a preliminary exploration of the trade-off between adversarial risk and two-faced risk in Figure 3 of the submission. As mentioned in the response to Q1, the resulting optimization process is computationally inefficient. Hence, we plan to delve into these challenges in future work, and we anticipate that the reviewer understands the complexities involved.