Paper: Vulnerable Region Discovery through Diverse Adversarial Examples (ICLR-2024)

Reviewer comment (Experiments):
The proposed approach exhibits limitations when applied to high-resolution images. Even though the author increased the population size to 800 for high-resolution images, this still only represents approximately 1.5% of the total images. However, it requires up to 80,000 queries to deep models.

Author response:
**Answer:** We genuinely appreciate the reviewer's insightful question regarding the limitations of dealing with high-resolution images.
The current implementation indeed only samples a small fraction of the total image population, even with an increased sample size. The necessity of up to 80,000 queries to deep models for these images is a significant demand on computational resources. However, the allowance of more localized perturbed pixels can be utilized (e.g., small patches) to increase identified vulnerable region size , thereby enhancing  the effective of our algorithm for higher resolution images. Although, such action may sacrifice the granularity of vulnerability analysis.  There is an inherent trade-off between detailed analysis of vulnerability and the the rate of successful attacks.  We recognize this as an area for improvement and leave it as our future work.

Reviewer comment (Writing):
It might be better if the author could briefly introduce adversarial training in related works section.

Author response:
**Answer:** We are grateful for you suggestion,  We have included related works on adversarial training in **Appendix A.1**.

Adversarial training, a pioneering technique introduced by Goodfellow et al. (2014) [1], has been a cornerstone in improving the robustness of deep neural networks (DNNs) against adversarial attacks. The core concept involves training the model on adversarial examples generated by perturbing input data to maximize the loss. Madry et al. (2018) [2] expanded on this concept by introducing the Projected Gradient Descent (PGD) adversarial training, considered one of the most effective methods for training robust models. Subsequent research has explored various dimensions of adversarial training. For instance, Xie et al. (2019) [3] introduced feature denoising to improve model robustness. Meanwhile, Zhang et al. (2019) [4] proposed TRADES, a theoretical framework balancing model accuracy on clean data with robustness against adversarial examples. Furthermore, Pang et al. (2019) [5] improved robustness by incorporating ensemble methods. In this work, we also explored how the identified vulnerable regions change with different adversarial training algorithms.

[1]Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and Harnessing Adversarial
Examples. arXiv e-prints, art. arXiv:1412.6572, December 2014.

[2] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. To-
wards deep learning models resistant to adversarial attacks. In International Conference on Learn-
ing Representations, 2018. 

[3] Cihang Xie, Yuxin Wu, Laurens van der Maaten, Alan L Yuille, and Kaiming He. Feature denoising
for improving adversarial robustness. In Proceedings of the IEEE/CVF conference on computer
vision and pattern recognition, pp. 501–509, 2019

[4] Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent El Ghaoui, and Michael Jordan.
Theoretically principled trade-off between robustness and accuracy. In International Conference
on Machine Learning, pp. 7472–7482. PMLR, 2019.

[5] Tianyu Pang, Kun Xu, Chao Du, Ning Chen, and Jun Zhu. Improving adversarial robustness via
promoting ensemble diversity. In International Conference on Machine Learning, pp. 4970–4979.

Reviewer comment (Reproducibility):
While the authors provide details on acquiring adversarial-trained models, information on obtaining standard-trained models is lacking.

Author response:
**Answer:** Thank you for your suggestion. We have added information with standard trained models in **Appendix A.2**.

In our ImageNet experiments, we employed pretrained ResNet50 and AlexNet models directly from PyTorch. For the CIFAR-10 experiments, the models were trained using SGD with a momentum of 0.9 and a weight decay of $5\times 10^{-4}$
 . We initialized the learning rate at 0.1, applying cosine annealing for its adjustment. The training process spanned a total of 200 epochs.

Reviewer comment (Experiments):
The experimental results reveal limitations when applied to high-resolution images. On average, only around 20 diverse adversarial examples are found for what the author refers to as vulnerable images. This number is relatively small. Is it possible to further adapt the method to high-resolution images?

Author response:
**Answer:** We genuinely appreciate the reviewer's insightful question and offer the following explanations on adapting our algorithm to higher-resolution images with minor adjustments.

**From the pixel-level attack to the patch-level attack:** Our algorithm is adaptable to higher-resolution images by adjusting the number of pixels subject to perturbation. In these scenarios, it's crucial to apply spatial constraints that accurately reflect vulnerabilities in specific areas, like small patches within an image. In the context of patch attacks, the only modification is to the representations of candidate solutions. Each candidate solution can be modified to a tuple containing the coordinates of the top-left pixel of the patch and the RGB values of different perturbed pixels within the patch. As a result, the tuple comprises $2+3\times n$ elements, where $n$ represents the number of perturbed pixels.

We have tested this approach using the ImageNet dataset with 4x4 pixel patches , selecting 100 correctly classified images resized to 224x224 pixels. Results are presented in Tabel 1. For AlexNet, our analysis identified 95.2 distinct patches, collectively covering 1106.2 pixels. It is important to note that the size of the region does not equate to the product of the number of diverse patches and the size of each patch (e.g., $1106.2 \neq 95.2 \times 16$), which indicates overlapping among the patches. In the case of ResNet50, which exhibits a lower success rate compared to AlexNet, we found 174.8 different patches that cover a total of 1920.5 pixels.   These experiment results demonstrate that our algorithm can be effectively adapted for high-resolution images with minimal adjustments. We leave a detailed analysis for high resolution image for our future work.

**Tabel 1.** Comparison of vulnerable region discovery using one pixel and 4x4 pixel adversarial patches on ImageNet with our algorithm. Region size refers to the average total number of perturbed pixels covered in each successful attack.
|                    |Perturbation Level | Success Rate | Region Size |
|-------------------|--------------------|--------------|-------------|
| AlexNet (ours)    | one pixel          | 11.7%        | 20.5        |
| ResNet50 (ours)   | one pixel          | 10.7%        | 20.6        |
| AlexNet (ours)    | 4x4 pixel patch    | 40%          | 1106.2      |
| ResNet50 (ours)   | 4x4 pixel patch    | 28%          | 1920.5      |

Reviewer comment (Novelty):
It's not clear to me what is the motivation for finding these vulnerable regions. What will we do with them? If the goal is explainability, why is this method conceptually better than other alternatives?

Author response:
>**Question2: It's not clear to me what is the motivation for finding these vulnerable regions. What will we do with them? If the goal is explainability, why is this method conceptually better than other alternatives?** 

**Answer:** We genuinely appreciate the reviewer's insightful question regarding the motivation and potential applications of our work. For the motivation of our work:
- **The motivation behind our work is to advance the field of explainable DNNs from another perspective - vulnerability by adopting a novel approach.** Traditional explainable DNN methods primarily concentrate on identifying areas in images that are most relevant to the model's predictions. In contrast, our research focuses on uncovering regions that are exceptionally sensitive to minor perturbations. This approach helps us uncover features within these regions that may be less reliable or robust. By validating and studying these vulnerable areas, we aim to deepen our understanding of the unexpected behaviors exhibited by DNNs. This shift in focus from simply interpreting model predictions to the underlying vulnerabilities represents the core motivation of our research. 

As mentioned above, our research primarily focuses on identifying vulnerable regions. For the question `what will we do with them', we give following explanation for the potential application. 
- **Enhance the Understanding of DNN's Vulnerability:** By examining which areas of an image, when altered, lead to misclassifications, we gain insights into the  features DNNs rely on for decision-making and their reliability. Understanding the conditions under which a DNN is prone to failure allows us to better grasp its limitations. This enhanced understanding is pivotal in addressing and mitigating vulnerabilities in DNNs.

- **Enhance the Robustness of Deep Models:** Vulnerable regions within DNNs are areas in the input space highly sensitive to minor perturbations, which can significantly affect the network's predictions. Identifying and addressing these regions is crucial for enhancing the model’s defense against adversarial attacks. Employing certain strategies, in conjunction with an understanding of these vulnerable regions, may lead to a better way to enhance the robustness of deep models:
    - Input Preprocessing: Applying preprocessing techniques to potentially vulnerable areas of images before their processing by the model can help mitigate the impact of possible perturbations in these high-risk regions.
    - Ensemble Methods: We can integrate various models or defensive strategies into an ensemble. This ensemble will focus on addressing vulnerabilities across different regions, thereby improving its robustness against adversarial attacks.
    - Adversarial Training: We can incorporate adversarial examples specifically designed to target identified vulnerable regions for improving the effectiveness of adversarial training, thereby enhancing the model's resilience.

   These approaches of detecting and reinforcing model vulnerabilities can be utilized to enable DNNs to preserve their performance stability when faced with adversarial attacks.

For the question `If the goal is explainability, why is this method conceptually better than other alternatives?', we have the following explanations:
- Departing from traditional techniques ([1,2,3]), our method aims to enhance the comprehensibility of deep neural networks (DNNs) by focusing on vulnerability rather than relevance.  While conventional
explainable DNNs concentrate on pinpointing image parts crucial for a model’s predictions, we aim to identify vulnerable regions where the features are not reliable.  This shift in focus provides a deeper, more fine-grained understanding of the model's weaknesses and limitations, which is crucial for developing more robust and reliable DNNs. This approach not only provides
insight into where and when DNNs may fail, offering a superior understanding of model vulnerabilities compared to traditional methods.

In summary, our research aims to enhance the comprehensibility of DNNs by focusing on vulnerability.   By identifying vulnerable regions and their potential applications, we aim to provide valuable insights into weakness and potential solutions of DNNs. 

[1] Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In
European Conference on Computer Vision, pp. 818–833. Springer, 2014

[2] Vitali Petsiuk, Abir Das, and Kate Saenko. Rise: Randomized input sampling for explanation of
black-box models. arXiv preprint arXiv:1806.07421, 2018

[3] Ruth Fong, Mandela Patrick, and Andrea Vedaldi. Understanding deep networks via extremal per-
turbations and smooth masks. In Proceedings of the IEEE/CVF International Conference on
Computer Vision, pp. 2950–2958, 2019

Reviewer comment (Theory):
As is hinted at in the paper, there is a conceptual problem with the idea of using groups of one-pixel adversarial attacks as a kind of interpretability method: as images get higher and higher resolution, the influence of each individual pixel get smaller, so one-pixel attacks get harder to find. Ideally, an interpretability method shouldn't fall apart as the resolution gets higher.

Author response:
>**Question3:** As is hinted at in the paper, there is a conceptual problem with the idea of using groups of one-pixel adversarial attacks as a kind of interpretability method: as images get higher and higher resolution, the influence of each individual pixel get smaller, so one-pixel attacks get harder to find. Ideally, an interpretability method shouldn't fall apart as the resolution gets higher.

**Answer:** We are grateful for the reviewer's astute observation regarding high-resolution images. 
In addressing the concern that our interpretability method, which relies on groups of one-pixel adversarial attacks, might become less effective with increasing image resolution, we offer the following clarifications: 
- **Clarification why diverse *one-pixel* perturbations:**  Our method innovatively utilizes a variety of adversarial examples to enhance the interpretability of deep learning models, with a specific focus on meticulous, pixel-level analysis of vulnerabilities. Central to our strategy is the deliberate emphasis on scenarios involving one-pixel attacks. We concentrate on detecting vulnerable areas within images, critical for the recognition tasks of deep neural networks (DNNs). These areas are characterized by variably located single-pixel perturbations. However, we recognize that in the case of high-resolution images, the influence of each individual pixel is reduced, making such one-pixel attacks more complex to implement. To address these complexities inherent in high-resolution images, we have devised an adaptive strategies.
- **Adaptation to the diverse patch attack  to overcome the limitation for higher resolution image:** Our algorithm is adaptable to higher-resolution images by incorporating a greater number of perturbed pixels in each adversarial example. This adaptation necessitates the use of spatial constraints to accurately represent vulnerabilities in specific areas, such as using small image patches. In the context of patch attacks, each candidate solution, can be modified to a tuple containing the coordinates of the top-left pixel of the patch and the RGB values of different perturbed pixels within the patch. As a result, the tuple contains $2+3 \times n$ elements, where $n$ represents the number of perturbed pixels.


    This adapted approach has been validated on the ImageNet dataset with images resized to 224x224 pixels, using 4x4 pixel adversarial patches. (We have included the details of this experiment in **Appendix A.6** for further reference.) The results are presented in Table 1. This, however, shifts our analysis focus from individual pixels to patch-level analysis, balancing the trade-off between a higher success rate and the granularity of the vulnerability assessment.  Compared to the one-pixel attack scenario, there is no additional cost needed. The experiment results is presented in Table 1. For AlexNet, our proposed approach identified 95.2 distinct patches, collectively covering 1106.2 pixels. It is important to note that the size of the region does not equate to the product of the number of diverse patches and the size of each patch (e.g., $1106.2 \neq 95.2 \times 16$), which indicates overlapping among the patches. In the case of ResNet50, which exhibits a lower success rate compared to AlexNet, we found 174.8 different patches that cover a total of 1920.5 pixels.   These results demonstrated that our algorithm can be effectively adapted for high-resolution images with only minor adjustments. We leave a comprehensive analysis of these discovered vulnerable regions for our future work.

**Table 1.** Comparison of vulnerable region discovery using diverse one pixels and diverse 4x4 pixel adversarial patches on ImageNet with our algorithm. Region size refers to the average number of different perturbed pixels covered in each successful attack.
|                   | Perturbation Level | Success Rate | Region Size |
|-------------------|--------------------|--------------|-------------|
| AlexNet (ours)    | one pixel          | 11.7%        | 20.5        |
| ResNet50 (ours)   | one pixel          | 10.7%        | 20.6        |
| AlexNet (ours)    | 4x4 pixel patch    | 40%          | 1106.2      |
| ResNet50 (ours)   | 4x4 pixel patch    | 28%          | 1920.5      |


In summary, our algorithm can overcome resolution limitations with suitable adaptation. We leave a comprehensive analysis on higher resolution images for further work.