Paper: Investigating Human-Identifiable Features Hidden in Adversarial Perturbations (ICLR-2024)

Reviewer comment (Experiments):
How were the subset of classes chosen?

Author response:
We carefully selected 20 classes from the ImageNet dataset to ensure maximal diversity in our study, covering a wide array of subjects such as animals, plants, architecture, toys, transportation, and utilities. To avoid human bias, we adopted a systematic approach, selecting the first 10 images from each class rather than hand-picking them.

Reviewer comment (Experiments):
How were the 200 inputs chosen? Were there any constraints or conditions for these inputs? Were all samples chosen correctly classified by all models?

Author response:
We carefully selected 20 classes from the ImageNet dataset to ensure maximal diversity in our study, covering a wide array of subjects such as animals, plants, architecture, toys, transportation, and utilities. To avoid human bias, we adopted a systematic approach, selecting the first 10 images from each class rather than hand-picking them.

Reviewer comment (Experiments):
While adding noise to produce additional inputs appears to work well, the inspiration or motivation for doing this was not exactly clear. Why add noise rather than performing some kind of data augmentation?

Author response:
We believe that perturbations contain noise that may result in meaningless local variations [1]. As a consequence, introducing different Gaussian noises to the same input image may lead to different local variation patterns. Therefore, averaging those perturbations can effectively reduce the noise residing within them. 

We believe that the changes in local variations are independent of specific data augmentation techniques because their variation is meaningless. Consequently, we anticipate that different augmentation techniques will yield similar results. However, adding noise offers the advantage of easily controlling the strength of augmentation by adjusting its standard deviation. This is the reason why we choose to add noise instead of performing data augmentation.

Reference:

[1] Smoothgrad: Removing noise by adding noise. Daniel Smilkov, et al. Workshop on Visualization for Deep Learning. 2017.

Reviewer comment (Experiments):
Why was the standard deviation of noise added to the inputs different for the different attack algorithms?

Author response:
In our research, Gaussian noise serves as an additional tool to further reduce the noise in perturbations. We found that different attack algorithms may need different levels of Gaussian noise to result in optimal clarity for human-identifiable features.

For BIM attacks, Gaussian noise with a standard deviation of 0.02 produces clear and pronounced human-identifiable features. CW and DeepFool attacks with a standard deviation of 0.02 also exhibit clear human-identifiable features. However, with a standard deviation of 0.05, the clarity of these features will be further enhanced. Consequently, we set the standard deviation for BIM attacks to 0.02 and for CW and DeepFool attacks to 0.05.

We would like to emphasize that averaging perturbations from different models is sufficient for the emergence of human-identifiable features, as discussed in Appendix A. Adding Gaussian noise only serves as an additional tool to further reduce perturbation noise.

Reviewer comment (Experiments):
Why were 270 models chosen for generating perturbations? Were these experiments tried with fewer models (besides the single model case)?

Author response:
We have conducted additional experiments to investigate the impact of the number of models used for averaging perturbations on the mean squared error (MSE) compared to perturbations from the MM setting. This analysis may give us a better insight into the number of models required to observe the emergence of human-identifiable features.

Prior to calculating the MSE score, we normalized the averaged perturbations using standard deviations and means derived from ImageNet datasets. This normalization ensures that the resulting MSE is on a comparable scale to the MSE scores calculated from images sampled from the ImageNet dataset, which have input values ranging between 0 and 1.

Our findings reveal that to achieve MSE convergence within 0.05, an average of 25 models is required for the three attack algorithms. For an MSE of 0.02, 90 models are necessary, while an MSE of 0.01 needs 157 models. For detailed information, please refer to Appendix I.

Reviewer comment (Experiments):
In the human evaluator test, the lowest and highest accuracy in each subset was discarded before calculating the average. What was the purpose of this, and can you clarify exactly what was discarded (e.g., was data for a single sample removed from all participants or was data from a single participant removed from all samples)?

Author response:
To eliminate outliers, we excluded data from two participants in each group: one with the highest accuracy and another with the lowest accuracy. This decision was made based on statistical considerations, as we observed that within each group, some individuals may perform significantly better or worse than the norm. By removing these outliers, we aim to provide an averaged human perspective on the evaluation of human assessment.

Reviewer comment (Novelty):
The experiment uses Gaussian noise to average the perturbations to reveal the human-identifiable features. However, this phenomenon has already been revealed in [4], which shows that randomized smoothing (adding Gaussian noises to the input and calculating the averaged gradient) on a single standardly trained model can lead to PAG and generate these features. Therefore, it's not a newly discovered phenomenon claimed in this paper that averaging gradient among perturbations with different noises can lead to human-identifiable features.

Author response:
The reviewer confused our findings with those in [R4]. Our research reveals that human-identifiable (robust) features naturally occur in adversarial perturbations of 'standard-trained' neural networks, even without adding Gaussian noise. This stands in contrast to [R4], which associates perceptually aligned gradients with the robustness of a classifier, leading to a markedly different conclusion.

A method for creating a robust classifier involves randomized smoothing, which adds Gaussian noises with a significant standard deviation (0.5 as mentioned in [R4]) to the original image and implicitly averages the model's predictions. The effectiveness of this technique lies in its certified radius, which assures the classifier's robustness within a certain range of perturbations and is proportional to the added Gaussian noises' standard deviation, as confirmed by earlier research work[1].

In [R4], it is noted that as the standard deviation of incorporated Gaussian noise increases, human-perceptible features become observable and evident in perturbations. Hence, the author suggests that the increase of model's robustness may lead to the presence of perceptually aligned gradients.

Our study, however, presents a different scenario. We demonstrate that human-identifiable (robust) features in adversarial perturbations are discernible when averaging perturbations from various models, even without Gaussian noise. This suggests that robust features inherently exist in standardly trained models, as averaging perturbations do not add new features to the original perturbation. Therefore, the distinction between our work and the study in [R4] is evident.

Reviewer comment (Novelty):
The experiment also averages different models to reveal the human-identifiable features. However, this phenomenon is expected based on existing work [5, 6], which shows that a little adversarial robustness of the models can lead to PAG. Specifically, as ensembling more non-robust models can still enhance adversarial robustness to a certain extent, though not as robust as adversarially trained models, it can be inferred that the ensembled model can lead to such PAG and identifiable features. Even if this paper shows that the robust accuracy of the ensembled model against adversarial attacks is still low (in Figure 3), the enhanced robustness may still be sufficient to bring such PAG.

Author response:
The reviewer states that according to prior research [R5 and R6], our findings are expected. However, our results are fundamentally different from those studies. In [R5], the author shows that adversarial training with low magnitude enhances the model's gradient perceptibility without significantly boosting its robustness, while [R6] discusses enhanced transferability of adversarial perturbations through a similar training method and is unrelated to our work.

The reviewer, lacking concrete evidence, attributes the increased gradient perceptibility in [R5] to a marginal improvement in adversarial robustness, which is not stated in [R5]. The reviewer then states that our results are expected extension of [R5], based on the assumption that a slight increase in the model’s robustness will also increase perturbations’ perceptibility. There is no indication that such assumption can be realized. Verifying such assumption clearly does not fall within our work’s domain.  Furthermore, it is noteworthy that several defensive algorithms, such as defensive distillation [2], marginally enhance the model's robustness. We would be interested to know if the reviewer could offer thought on whether perturbations in these defensively augmented models exhibit a perception-aligned gradient.

In our experiment, we average perturbations from different models and demonstrate that human-identifiable features naturally exist in perturbations from a standard-trained model. Averaging helps to reduce noise and put together incomplete information without incorporating new features into the perturbations. This should not be confused with an increase in models' robustness.

Reviewer comment (Theory):
It has also been shown [7] that the distribution of non-robust features [17] varies across different model architectures. Therefore, intuitively, the gradient (perturbation) of a single model (or a single kind of model architecture) may be noisy, but by averaging the gradients from different models, it is possible to converge toward the robust features.

Author response:
The paper in [R7] proposes an algorithm - the Skip Gradient Method (SGM) - that can increase the transferability of adversarial perturbations. In the paper, nothing is discussed about “the distribution of non-robust features”. We would appreciate it if the reviewer would show where the paper the claim is written.

The reviewer initially mentioned that robust features exist exclusively in robust models, based on the concept of perceptual aligned gradients. Our research reveals, however, that robust features are also present in perturbations originating from standard-trained models, a finding established by averaging perturbations across various models. Our work is significant as it highlights the role of robust features for fooling models in perturbations from a standard-trained model. This insight has the potential to foster a deeper understanding of how perturbations can mislead models. Despite described clearly in the paper, the reviewer persistently extrapolates other studies and attributes our findings simply to increased model robustness, which deviates from the actual explanation and facts.

Reviewer comment (Novelty):
The discovery made in this paper is somewhat trivial, since the observed phenomena have already been revealed in existing work or can be directly deduced from them. Furthermore, the evidence presented in this paper is insufficient to challenge the well-established theories of PAG, as this paper does not provide a clear explanation of the contradictions or confusions.

Author response:
We have conducted experiments and the results lead to our conclusion. It would be nice to build a theoretical model on what we have found, which is not done as confined by limited manpower and resources.

Reviewer comment (Theory):
There exist several works [8-10] aiming to explain the reason PAG only exists in robust models by characterizing the decision boundaries between different models, which is well supported by theoretical analysis. These works show the fundamental difference of decision boundaries between standard and adversarially trained models leads to the (non-)existence of PAG, which contradicts the claim made in this paper in Section 7(2) that human-identifiable features also exist in non-robust models. Unfortunately, this paper does not discuss this viewpoint and does not conduct a theoretical analysis to overturn these theories.

Author response:
In our research, we show, without a shred of doubt, that human-identifiable features exist in perturbations of standard-trained models. We firmly believe that theory is always capable of explaining experimental findings.

Reviewer comment (Theory):
There exist theories interpreting the existence of PAG in robust models by modeling adversarial training as energy-based models [11-12]. Additionally, the robust model also provides better guidance during the generation process of diffusion models [13-14], indicating the importance of robust models with PAG for better gradient and generation guidance. Since such a generation process requires multi-step sampling, which can be regarded as applying an **average (ensemble)** of gradients (perturbations) to the standardly trained model, this also contradicts the viewpoint in this paper and should be well-explained.

Author response:
In our research, we show, without a shred of doubt, that human-identifiable features exist in perturbations of standard-trained models. We firmly believe that theory is always capable of explaining experimental findings.

Reviewer comment (Theory):
In Section 7(1), the explanation for the transferability of adversarial examples contradicts existing works. This paper attributes the transferability to the human-identifiable (robust) features, but existing works [15-16] show that robust features may not be always helpful for adversarial examples transferring between models and non-robust features still play a crucial role in transferring adversarial examples. Therefore, the claims made in this paper fail to explain the transferability of adversarial examples across models.

Author response:
In our research, we isolated human-identifiable features from perturbations and discovered their significant attack strength, leading us to conclude that these features contribute to the misclassification of a model. We see no conflict between our findings and those in [R15, R16], as [R15, R16] do not deny the strong transferability of robust features to deceiving models.

Reviewer comment (Theory):
The explanation of non-trivial accuracy for classifiers trained on a manipulated dataset [17] made in Section 7(3) is flawed. It is clear that in the manipulated dataset, which includes perturbations claimed as human-identifiable features in this paper, the features from the original class are still dominant over the perturbations. According to the interpretation within this paper, the model should still learn the features from the original class and cannot achieve clean accuracy in this noisy training setting. This contradicts the explanation proposed in this paper.

Author response:
The term "dominant" should be used with caution. While the original images' features are larger in pixel value, the perturbations generated by the models may align more closely with what the models learn, potentially playing a more significant role during training than that of the original features. The discussion of which feature dominates in training is not the focus of this research. Since we have yet to demonstrate what types of features are more dominating, we put an emphasis in our paper that only partial explanation can be offered.

Reviewer comment (Evaluation):
In Appendix A, Figure 7, it appears that the masking effect of the perturbation without Gaussian noise significantly reduces the identifiability of human-identifiable features, compared to the results in the main paper (with Gaussian noise). Therefore, it can be inferred that ensembling Gaussian noise plays a more crucial role in generating the human-identifiable features than ensembling different models, which undermines the soundness of the claim that the presence of human-identifiable features is inherent in the perturbations themselves, rather than being a result of added Gaussian noise.

Author response:
The statement made by the reviewer that the presence of human-identifiable features is due to the incorporation of Gaussian noise is incorrect. Two pieces of information illustrate the incorrectness: 

1. In the search-based attack, no Gaussian noise is added, yet human-identifiable features still emerge, as shown in Section 5.2.4 and Appendix D.
2. In Appendix A, we have performed an experiment on perturbations generated without the incorporation of Gaussian noise (MM setting). It is clear that human-identifiable features is still observed from perturbations, as demonstrated in Appendix A1.

Additionally, we disagree with the reviewer's assertion that averaging perturbations from various models (MM setting) significantly diminishes the perceptibility of human-identifiable features. The cosine similarity between perturbations generated separately from MM+G and MM setting for BIM attack reaches as high as 0.80 based on an average of 200 perturbations. This high similarity indicates that Gaussian noise is not critical to influencing the perceptibility of perturbations.

Reviewer comment (Experiments):
There is a lack of ablation studies on the number of models to further support their claims. It is suggested to add experiments to analyze how many models or noises are required to emerge such human-identifiable features, which can provide a more intuitive view of how noisy the gradients are in the adversarial perturbations.

Author response:
We have added experiments analyzing the model count and the convergence for perturbations, please refer to Appendix I. Thank you for your suggestion.

Reviewer comment (Theory):
In the first paragraph of Section 4, on what basis do you assert that (1) the noise in perturbations is independent and (2) two perturbations from different models display distinct human-identifiable features? I couldn't find any references or evidence supporting the claims.

Author response:
(1) and (2) are assumptions based on existing literature or empirical observations, which inspired us to uncover human-identifiable features in perturbations through the process of averaging.

For assumption (1), the existing literature suggests that the noise observed in gradients relative to the loss function might be meaningless local variations in partial derivatives [1]. Therefore, it is reasonable to assume that the local variations of gradients trained independently from different models should not be aligned or dependent on each other.

Regarding assumption (2), our empirical observations show that two robust models can generate distinct human-identifiable features for the same image using the same attack algorithm. This leads to our hypothesis that perturbations from standard-trained models also contain distinct human-identifiable features.

[1] Smoothgrad: Removing noise by adding noise. Daniel Smilkov, et al. Workshop on Visualization for Deep Learning. 2017.

Reviewer comment (Experiments):
I observed that detecting human-identifiable features necessitates 2,700 samples (270 models and 10 noise-infused seed samples). These may suggest that the averaged perturbation, generated by the three attacking methods, gravitates towards the object region. However, they don't confirm that in every model, the generated perturbations house human-identifiable features. Hence, a deeper experimental analysis regarding model selection and the integration of Gaussian noise would be beneficial, perhaps including more ablation studies (like MM, MM+G, SM+G).

Author response:
Thank you for your comment. We would like to note that we have included the MM setting in our ablation study, as detailed in Appendix A1. The results consistently align with those from the MM+G setting and demonstrate human-identifiable features. 

Based on your suggestion, we have conducted additional experiments to explore the contribution of each model towards the human-identifiable feature. This involved varying the number of models used to average perturbations and comparing their mean square error to the perturbations from the MM setting. We found that, to achieve Mean Square Error (MSE) convergence within 0.05, an average of 25 models are needed for the three attack algorithms. For a MSE of 0.02, 90 models are required, while an MSE of 0.01 necessitates 157 models, for detailed information, please refer to Appendix I.

Reviewer comment (Experiments):
It's noted that perturbations of identical images from varying attack algorithms are presumably alike. However, the results don't include background noise similarity or image perturbation similarity. Providing experimental evidence for this would enhance the argument.

Author response:
The reason why the cosine similarity will converge regardless of attack algorithms is because the masking effect reduces contrasts on human-identifiable features in an image and those features should not be dependent on the attack algorithm used. In section 5.2.3, we discuss why human-identifiable features are primarily located in the contour part of the perturbations. Therefore, computing the cosine similarity for the background part may not align with our objective of assessing the similarity of the masking effect across different attack algorithms. Since the background contains significantly fewer human-identifiable features, its inclusion is expected to reduce the overall cosine similarity score.

The table below presents cosine similarity scores between perturbations generated by different attack algorithms under the MM+G setting. The first row indicates which two attack algorithms are used for generating perturbations, while the second and third rows compare the cosine similarity between the entire perturbations and their contour parts, respectively.

| Attack Algorithms | BIM/CW | BIM/DeepFool | CW/DeepFool |
|-------------------|--------|--------------|-------------|
| Whole             | 0.35   | 0.41         | 0.59        |
| Contour           | 0.43   | 0.47         | 0.64        |

Reviewer comment (Experiments):
The experimental analysis concerning the two distinct types of human-identifiable features (masking effect and generation effect) appears limited. Visualizing the perturbation for targeted attacks would be beneficial.

Author response:
We have presented the results for targeted attacks in Section 6, which serve to demonstrate the generation effect. In Appendix H, more targeted attack examples are included.

Reviewer comment (Evaluation):
Does the visual perturbation come from cases where the attack was successful? How does the perturbation behave in the case of an unsuccessful attack?

Author response:
The experiment is conducted under black-box attacks. Therefore, the attack's success depends on the testing model used. In the human evaluation test, we used all perturbations generated from the experiment, including those that failed to fool any of the testing models.  

For the BIM attack, we found that 14 perturbations, containing human-identifiable features under the MM+G setting, could lower the label class score. However, none of the 4 testing models used in the paper were fooled by any of these perturbations. This is attributed to the fact that the testing models exhibit high confidence scores for the respective images, making them more challenging to deceive.

For those 14 images, the confidence score, derived from predictions processed via the SoftMax function, was, on average, 11.92% higher than those for images correctly classified in the experiment. This comparison is detailed in the following table, where the first column denotes the respective testing models. The table entries consist of the confidence scores for the 14 images that were not deceived by the perturbations, as well as the average confidence scores for all correctly classified images across the four testing models.

We further discovered that when the $L_{inf}$ norm of the perturbations was increased to 0.04, only four images incorporated with perturbations remained correctly classified by all 4 testing models. This finding suggests that these perturbations, while initially unable to fool models due to high confidence scores, can still fool testing models as $L_{inf}$ norm is increased. 

| Model       | ResNet-50 | BN-Inception | DenseNet-121 | VGG-16 |
|-------------|-----------|--------------|--------------|--------|
| 14 Images   | 93.15%    | 88.29%       | 88.86%       | 79.33% |
| Avg.        | 82.54%    | 80.46%       | 78.32%       | 71.08% |

Reviewer comment (Experiments):
While the paper asserts findings across three different datasets, I could only locate a detailed attack accuracy comparison for ImageNet in Appendix E Table 1. It is not clear why the NOISE performance surpass that of IMAGE.

Author response:
We would like to remind the reviewer that the attack accuracy for the remaining two datasets is located in Appendix C.3.

Reviewer comment (Experiments):
Human-identifiable features looks vague: I still remain unclear about how to logically define the "human-identifiable" here: In section 5.2.1 authors conducted recognizability experiments on these denoised adv perturbations but it can only prove they are "model-identifiable". We cannot make such claim by showing part of (or even all) extracted adv perturbations and they are all human-identifiable. Some human-labeling experiments is required as a strong evidence to prove this.

Author response:
We would like to note that we have conducted a human evaluation test on perturbations generated under the MM+G setting, as detailed in Section 5.2.1. An average accuracy of 80.7% is achieved by human participants, indicating that adversarial perturbations are highly classifiable/identifiable by humans.

Reviewer comment (Novelty):
The overall finding is not surprising: while it is good to see that denoised adversarial perturbation is similar to its corresponding raw image, I'm not surprising to see because gradient-based attacks perturb models' prediction by optimizing the objective function following the pixel-gradient direction --- larger pixel gradients indicate pixels here are important for threat model to identify this input image. Thus the outcome of gradient optimization, adversarial perturbation, should contain some important features to identify this image. And for search-based attacks, it still tend to follow the important pixels to craft their perturbation. I think this paper should focus more on the target-attack scenario - so we have our raw-image key features and our targeted label --- how would the adversarial perturbation be to reflect both concept? Currently it only has a very short paragraph discussing such scenario (Section 6).

Author response:
We thank the reviewer for providing us with the chance to clarify our work. In the following discussion, we will introduce the prevailing perspective on adversarial perturbations, and then explain the contribution of our work.

Although adversarial perturbations are expected to contain features beneficial to identifying images, there is no guarantee that these features will be human-understandable. In fact, the prevailing view in the field is that human and machine rely on different types of features for classification. Adversarial perturbations, as a direct consequence of human-AI misalignment, are a type of feature (non-robust feature) that is highly informative for models yet incomprehensible to humans [1]. There is even a stronger version of the assumption stating that human-identifiable (robust) features should only exist in robust models, as suggested by reviewer 4 (ZsSi).

The reviewer might think that, with the invention of explainable AI tools, it should appear straightforward that gradient/adversarial perturbations must include features identifiable by human. However, these tools often require preprocessing that largely alters the image, which could mask the actual features relied upon by the model. Consequently, the reported results may be ad-hoc and not accurately reflect reality [2].

Our research reveals that human-identifiable features inherently reside in perturbations from a standardly trained model, offering fresh insights into explaining the properties of perturbations including transferability, the relation between adversarial training and increased model explainability, and the human-AI misalignment experiment, all of which are crucial in understanding the mechanisms behind adversarial perturbations.

References:

[1] Adversarial Examples Are Not Bugs, They Are Features. Andrew Ilyas, et al. NeurIPS, 2019.

[2] Adversarial Examples and Human-ML Alignment. Alexender Madry. MIT CBMM Talks (https://www.youtube.com/watch?v=AvcRBuFreFg, 22:52-23:13)

Reviewer comment (Novelty):
Although this problem is interesting, the authors do not provide more surprising findings and insights compared with previous works.

Author response:
We thank the reviewer for giving us the chance to clarify our contributions. Our contribution is not to re-discover robust (human-identifiable) features but to demonstrate that robust features inherently exist in adversarial perturbations generated from a standard-trained model. In fact, the prevailing view in this field is that human and machine rely on different types of features for classification. This view leads to the claim that adversarial perturbations from standard models, as a direct consequence of human-AI misalignment, are a type of feature (non-robust feature) that is highly informative for models yet incomprehensible to human [R1]. 

With our experiment, we demonstrate that perturbations from standard models inherently contain robust features, instead of mere non-robust features. To our best knowledge, this finding is yet to be discussed in the literature and provides a new perspective on understanding the mechanism governing the attack of perturbations on models.

Reference [R2] illustrates the effectiveness of robust models in image generation and restoration tasks. A related study [1] by the same research group delves into the reasons behind this, which is closely related to our work. The study shows that gradients and perturbations generated by robust models align more closely with human perception, while perturbations from standard networks are significantly noisier and may not be human-identifiable. According to the authors, standard models learn different features than robust models. 

We show that, by averaging perturbations from standard models, robust features are still observable, indicating a shared learning aspect between standard and robust models. Furthermore, we offer a new yet plausible explanation for the increased perceptibility in adversarial training: Adversarial training could act as gradient regularization, reducing noise and homogenizing gradients, thus making perturbations more perceptible to humans, as stated in Section 7.2. 

Our study uncovered human-identifiable (robust) features in perturbations from standard training, shedding new light on perturbations’ properties, such as transferability, the link between adversarial training and enhanced model explainability, and the human-AI misalignment experiment. These aspects are key to understanding the mechanism of adversarial perturbations, allowing us to pursue adversarial-perturbations related applications. 

Reference:

[1] Robustness May Be at Odds with Accuracy. Dimitris Tsipras, et al. ICLR 2019.

Reviewer comment (Novelty):
Adversarial perturbations containing meaningful or human-identifiable features have been studied in these works [1,2]. They may correspond to "robust" features.

Author response:
We thank the reviewer for giving us the chance to clarify our contributions. Our contribution is not to re-discover robust (human-identifiable) features but to demonstrate that robust features inherently exist in adversarial perturbations generated from a standard-trained model. In fact, the prevailing view in this field is that human and machine rely on different types of features for classification. This view leads to the claim that adversarial perturbations from standard models, as a direct consequence of human-AI misalignment, are a type of feature (non-robust feature) that is highly informative for models yet incomprehensible to human [R1]. 

With our experiment, we demonstrate that perturbations from standard models inherently contain robust features, instead of mere non-robust features. To our best knowledge, this finding is yet to be discussed in the literature and provides a new perspective on understanding the mechanism governing the attack of perturbations on models.

Reference [R2] illustrates the effectiveness of robust models in image generation and restoration tasks. A related study [1] by the same research group delves into the reasons behind this, which is closely related to our work. The study shows that gradients and perturbations generated by robust models align more closely with human perception, while perturbations from standard networks are significantly noisier and may not be human-identifiable. According to the authors, standard models learn different features than robust models. 

We show that, by averaging perturbations from standard models, robust features are still observable, indicating a shared learning aspect between standard and robust models. Furthermore, we offer a new yet plausible explanation for the increased perceptibility in adversarial training: Adversarial training could act as gradient regularization, reducing noise and homogenizing gradients, thus making perturbations more perceptible to humans, as stated in Section 7.2. 

Our study uncovered human-identifiable (robust) features in perturbations from standard training, shedding new light on perturbations’ properties, such as transferability, the link between adversarial training and enhanced model explainability, and the human-AI misalignment experiment. These aspects are key to understanding the mechanism of adversarial perturbations, allowing us to pursue adversarial-perturbations related applications. 

Reference:

[1] Robustness May Be at Odds with Accuracy. Dimitris Tsipras, et al. ICLR 2019.

Reviewer comment (Novelty):
The proposed methods, noise augmentations and model ensembling, are widely used in transfer attacks. More transferable perturbations contain more "robust" features (human-identifiable features) and share more non-robust features. Previous work has shown this point [1].

Author response:
We thank the reviewer for giving us the chance to clarify our contributions. Our contribution is not to re-discover robust (human-identifiable) features but to demonstrate that robust features inherently exist in adversarial perturbations generated from a standard-trained model. In fact, the prevailing view in this field is that human and machine rely on different types of features for classification. This view leads to the claim that adversarial perturbations from standard models, as a direct consequence of human-AI misalignment, are a type of feature (non-robust feature) that is highly informative for models yet incomprehensible to human [R1]. 

With our experiment, we demonstrate that perturbations from standard models inherently contain robust features, instead of mere non-robust features. To our best knowledge, this finding is yet to be discussed in the literature and provides a new perspective on understanding the mechanism governing the attack of perturbations on models.

Reference [R2] illustrates the effectiveness of robust models in image generation and restoration tasks. A related study [1] by the same research group delves into the reasons behind this, which is closely related to our work. The study shows that gradients and perturbations generated by robust models align more closely with human perception, while perturbations from standard networks are significantly noisier and may not be human-identifiable. According to the authors, standard models learn different features than robust models. 

We show that, by averaging perturbations from standard models, robust features are still observable, indicating a shared learning aspect between standard and robust models. Furthermore, we offer a new yet plausible explanation for the increased perceptibility in adversarial training: Adversarial training could act as gradient regularization, reducing noise and homogenizing gradients, thus making perturbations more perceptible to humans, as stated in Section 7.2. 

Our study uncovered human-identifiable (robust) features in perturbations from standard training, shedding new light on perturbations’ properties, such as transferability, the link between adversarial training and enhanced model explainability, and the human-AI misalignment experiment. These aspects are key to understanding the mechanism of adversarial perturbations, allowing us to pursue adversarial-perturbations related applications. 

Reference:

[1] Robustness May Be at Odds with Accuracy. Dimitris Tsipras, et al. ICLR 2019.

Reviewer comment (Evaluation):
Although the visualizations are very promising, we are uncertain about the extent of assistance this can provide.

Author response:
We thank the reviewer for giving us the chance to clarify our contributions. Our contribution is not to re-discover robust (human-identifiable) features but to demonstrate that robust features inherently exist in adversarial perturbations generated from a standard-trained model. In fact, the prevailing view in this field is that human and machine rely on different types of features for classification. This view leads to the claim that adversarial perturbations from standard models, as a direct consequence of human-AI misalignment, are a type of feature (non-robust feature) that is highly informative for models yet incomprehensible to human [R1]. 

With our experiment, we demonstrate that perturbations from standard models inherently contain robust features, instead of mere non-robust features. To our best knowledge, this finding is yet to be discussed in the literature and provides a new perspective on understanding the mechanism governing the attack of perturbations on models.

Reference [R2] illustrates the effectiveness of robust models in image generation and restoration tasks. A related study [1] by the same research group delves into the reasons behind this, which is closely related to our work. The study shows that gradients and perturbations generated by robust models align more closely with human perception, while perturbations from standard networks are significantly noisier and may not be human-identifiable. According to the authors, standard models learn different features than robust models. 

We show that, by averaging perturbations from standard models, robust features are still observable, indicating a shared learning aspect between standard and robust models. Furthermore, we offer a new yet plausible explanation for the increased perceptibility in adversarial training: Adversarial training could act as gradient regularization, reducing noise and homogenizing gradients, thus making perturbations more perceptible to humans, as stated in Section 7.2. 

Our study uncovered human-identifiable (robust) features in perturbations from standard training, shedding new light on perturbations’ properties, such as transferability, the link between adversarial training and enhanced model explainability, and the human-AI misalignment experiment. These aspects are key to understanding the mechanism of adversarial perturbations, allowing us to pursue adversarial-perturbations related applications. 

Reference:

[1] Robustness May Be at Odds with Accuracy. Dimitris Tsipras, et al. ICLR 2019.

Reviewer comment (Theory):
The two obstacles are not very clear. The first one (noisy gradient) is easy to understand. Lots of transfer attacks also propose to mitigate this negative effect to improve adversarial transferability. However, there is insufficient evidence to support the second claim about incomplete learned features. Could you please provide more details about the second one?

Author response:
**2.1** The obstacles hindering the presence of robust features are assumptions made in the paper based on existing literature or empirical observations. These obstacles motivate us to propose the method of averaging perturbations as a method to uncover these robust features.

Regarding assumption (2), our empirical observations indicate that gradient/perturbations from robust models only contain a subset of human-identifiable features. This leads to the hypothesis that robust features from standard models should also contain incomplete human-identifiable features.

We believe the reason behind is due to the capacity limit and challenge in learning optimum, neural networks may not completely utilize all robust feature for classification. Perturbations, designed to maximize the model's loss function, impact only those features the model recognizes. Consequently, these perturbations might encompass only a subset of features identifiable by humans.

Reviewer comment (Experiments):
The comparison between these two points is also unclear. Which factor has a greater negative impact on extracting human-identifiable features? As shown in the experimental setting, the authors need to use lots of ensembling models. This has made this method less practical.

Author response:
**2.2** Our research aims to investigate robust features in perturbations from standard models. Assessing which factors, noise or incompleteness, more significantly hinder robust features is not our primary focus. We recognize the value of such evaluations.  However, discerning the dominant factor is challenging due to the concurrent presence of noise and the incompleteness of robust features in perturbations. Separating one factor without altering the perturbations' nature may be challenging. This issue deserves our attention for further investigation.

Reviewer comment (Novelty):
The findings from Section 5.2.3 are interesting. The authors use the contour features to attack models. It also shows that contour features are more important than background information. Could the authors please discuss connections and differences between this phenomenon and this work [3]?

Author response:
**2.3** In [R3], the authors demonstrates that Convolutional Neural Networks (CNNs) is used to classify images using both texture and contour, with texture playing a more significant role than in human classification, which relies more on contour.

We segment the perturbations into those within the contour of the object and those in the background. We find that the contour parts contain both the object’s edges and textures that are similar to the original image. Figure 2 illustrates those perturbations from MM+G display texture-like features. For instance, one can observe a green, granular texture on the frog's back and a stripe-like texture on the baseball. These observations align with the findings in [R3] because, to deceive the model, perturbations may need to reduce the pixel values of both texture and contour features, which serve as critical factors influencing the model's decision.

Reviewer comment (Experiments):
Could the authors please provide more results about generation effect on targeted attacks?

Author response:
**3** Additional results about targeted attack have been provided in Appendix H. Thank you for your suggestion.