Paper: Be Your Own Neighborhood: Detecting Adversarial Example by the Neighborhood Relations Built on Self-Supervised Learning (ICLR-2024)

Reviewer comment (Experiments):
The method appears to be limited to the Computer Vision domain. The entire paper is tailored for DNN applications for Computer Vision. It would be enticing to see how well the proposed method could be adapted to cover other domains in which DNN have found applications (e.g., audio, finance, cybersecurity). This is the main weakness preventing a higher score.

Author response:
We understand the reviewer is interested in exploring the use of BEYOND in new domains. We try to migrate our approach to different domains like Natural Language Processing (NLP) and Speech Recognition (SR). But we note some critical differences: existing self-supervised models in NLP or SR are not based on diversified data augmentation strategies. For instance, in NLP, masked token prediction is the primary method. However, having diverse augmentation types is important for BEYOND, because it needs good data augmentation to create neighboring data. Nonetheless, we believe that our approach pioneers an idea of detection based on testing time-augmented examples rather than reference examples. And we will also continue to explore ways to apply this idea to different domains.

Reviewer comment (Evaluation):
Some unclear details in the evaluation. It is not clear if the results provided in the paper refer to a single run of a given method, or are provided after performing multiple runs and then averaging the results. If results are averaged, the number of runs and the standard deviation should be provided. This lack of clarity affected the reviewer’s score.

Author response:
In this paper, our experimental results come from a single run. To further verify the effectiveness of BEYOND, We repeat the experiments in Tables 1 and 2 of the main paper five times. On Cifar10, the average detection performance for the five attacks is 99.16% with a standard deviation of 0.0117. On ImageNet, the average performance against three attacks is 96.44% with a standard deviation of 0.0426.

Reviewer comment (Writing):
In the Introduction, the statement that incorrect predictions can lead to catastrophic economic and even loss of life should be toned down. This is partially incorrect since DNN are deployed in those applications (e.g., autonomous driving), and there is little evidence that such catastrophic events are due to incorrect predictions stemming from adversarial examples as opposed to other causes such as misconfigurations.

Author response:
Apologies for the uncritical description, we will correct it to“ where incorrect predictions can lead to unpredictable losses”.

Reviewer comment (Writing):
The use of "white-/gray-box" terminology to denote the envisioned attacker should be avoided, as it is ambiguous according to recent works (e.g., IEEE SaTML23). The authors are encouraged to use "perfect/limited knowledge" attackers instead and to provide in the Appendix a section clearly defining the knowledge/capabilities of the attacker in relation to the attacked system. A practical example illustrating the threat model would strengthen this part.

Author response:
And about the “white-/gray-box” terminology, we appreciate your experienced suggestions. According to your advice, we refer to the latest work **[RR1]** from IEEE SaTML23, and change the terminology to “limited/perfect knowledge” attack in the the latest pdf. 

**[RR1]** Apruzzese G, Anderson H S, Dambra S, et al. “Real Attackers Don't Compute Gradients”: Bridging the Gap Between Adversarial ML Research and Practice. 2023 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML). IEEE, 2023: 339-364.

Reviewer comment (Experiments):
It is not demonstrated whether the method can be applied to different domains. The reviewer requests compelling evidence—potentially substantiated with a proof-of-concept experiment—that the method generalizes beyond Computer Vision.

Author response:
Dear reviewer,

For the possibility of migrating BEYOND to other domains, within the limited time, we did a simple experiment on adversarial text detection for NLP. Using the text-attack tool  **[RR1]**, we generated 1000 adversarial texts on a BERT model trained on the MR sentiment classification dataset using the TextFooler attack **[RR2]**. We then used the chatgpt_paraphraser tool to generate ten paraphrases for each text as augmentations. Finally, we compare the embedding similarity between the samples and their augmentations on BERT. The AUC of the detection is 73.63%. Although there is certainly room for performance improvement, we believe that this pilot study confirms the possibility that BEYOND can be migrated to other domains.

**[RR1]** Morris J X, Lifland E, Yoo J Y, et al. Textattack: A framework for adversarial attacks, data augmentation, and adversarial training in nlp[J]. arXiv preprint arXiv:2005.05909, 2020.

**[RR2]** Jin D, Jin Z, Zhou J T, et al. Is BERT really robust? A strong baseline for natural language attack on text classification and entailment[C]//Proceedings of the AAAI conference on artificial intelligence. 2020, 34(05): 8018-8025.

Reviewer comment (Evaluation):
It is unclear whether the reported results encompass various repetitions that have been averaged. The reviewer requests confirmation and clarification.

Author response:
For Q2, we have updated tables 1 and 2 with the mean and standard deviation of 5 runs. Please check it in the latest pdf. Note that LNG is not open-sourced and corresponding results come from the paper. So, we cannot calculate the mean and standard deviation for multiple runs.

Reviewer comment (Theory):
Theoretical study is not rigorous. The whole study is based on the assumption that data augmentations can effectively weaken adversarial perturbation. However, this is only an empirical observation, yet without any theoretical justification. It is not convincing to set up further deduction based on this hypothesis.

Author response:
## Q1 Theoretical study

First, our analyses in the main paper are for the case where augmentation can weaken adversarial perturbations, and it shows that if augmentation can weaken adversarial perturbations, it can be used for adversarial sample detection on the SSL representation space.

We note that it is difficult to conduct an overall theoretical analysis to verify that data augmentation as a whole can always effectively weaken adversarial perturbations because there are many data augmentations and they have different effects on adversarial perturbations. We address your question as to how to determine whether one given augmentation type weakens the adversarial perturbation or not.  Therefore, we analyze which data augmentation can weaken adversarial perturbations, i.e., what conditions must be met for augmentations to be effective. 

Effective data augmentation makes the augmented label $y_{aug}$ tend to the ground-truth label $y_{true}$ and away from the adversarial label $y_{adv}$ as follow:

$ ||y_{aug}-y_{true}||_2 $  

$ \leq ||y_{aug}-y_{adv}||_2 $

$ \leq ||y_{adv}-y_{true}||_2 $

Since $y_{true}$ is the one-hot encoding, the range of $ ||y_{adv}-y_{true}||_2 $ is $(\sqrt{2}/2, \sqrt{2})$. 

The distance is $\sqrt{2}$ when the item corresponding to $y_{adv}$ is 1 in the logits of $y_{adv}$, and $\sqrt{2}/2$ when the item corresponding to $y_{adv}$ and $y_{true}$ both occupy 1/2. Given a SSL-based classifier, $C$, we have:

$
C(W(x+\delta)) = C(Wx)+\nabla C(Wx)W\delta = y_{true} + \nabla C(Wx)W\delta = y_{aug}
$

Therefore, the distance between $y_{aug}$ and $y_{true}$ is:

$ ||y_{aug}-y_{true}||_2 = ||\nabla C(Wx)W\delta||_2 $
$ \leq ||\nabla C(Wx)W||_2 ||\delta||_2 $

$ \leq ||\nabla C(Wx)W||_{2} \epsilon $

where $ ||\delta||_2 $ is bounded by $ \epsilon $ according to the definition of adversarial examples. 

To make sure $ ||y_{aug}-y_{true}||_2$

$ \leq ||y_{aug}-y_{adv}||_2 $, then:

$ ||\nabla C(Wx)W||_2 \epsilon \leq \frac{\sqrt{2}}{2} \Rightarrow  ||\nabla C(Wx)W||_2 \leq \frac{\sqrt{2}}{{2\epsilon}} $

In summary, an augmentation can mitigate adversarial perturbation when it satisfies $||\nabla C(Wx)W||_2 \leq \frac{\sqrt{2}}{{2\epsilon}}$.

To further validate our analysis, we generate 1000 adversarial examples by PGD with $\epsilon=8/255$ on Cifar10. The following table shows the ratios for different data augmentations meeting the threshold $\frac{\sqrt{2}}{{2\epsilon}}$.

| Aug | Ratio | Aug | Ratio |
|:---:|:---:|:---:|:---:|
| Rotation | 99.9% | Vertical | 25.9% |
| Crop | 40.7% | Color Jitter | 99.0% |
| Resize | 74.0% | Gray | 40.6% |
| Horizontal | 25.9% | Compose | 99.7% |

A higher ratio means the augmentation is more effective. Meanwhile, we perform t-sne visualizations of the representations of clean examples and AEs processed by different augmentation methods in Fig. 10 in the latest pdf. As seen in the figure, the effective augmentation methods with the high ratio in the above table can effectively increase the distance between AEs and their neighbors. For example, Rotation has the highest ratio in the above table, and the distance between AE and its neighbors in the figure is larger than that of clean samples. While Horizontal and Vertical have the lowest ratio, and the distance between AE and its neighbors is still close in the above figure. 

Moreover, we test the detection performance of high-ratio augmentations and low-ratio augmentations as follows:

|Augmentation|FGSM|PGD|CW|AutoAttack|Average|
|:---:|:---:|:---:|:---:|:---:|:---:|
|ColorJitter&Resize&Rotation| 97.11% | 96.55% | 98.15% | 96.56% | 97.09% |
|Gray&Horizaotal&Crop&Vertical| 92.44% | 91.36% | 94.70% | 91.87% | 92.59% |

It can be seen that the average detection performance of the effective augmentations obtained by our analysis is 5% higher than that of the other augmentations. 

Finally, BEYOND adopts various augmentations instead of a single type to generate multiple neighbors for AE detection. This allows the detection performance to circumvent the effects of bad augmentation methods, which reduces randomness and makes the estimates more robust.

Reviewer comment (Experiments):
The proposed detection algorithm requires to set many threshold values. In Algorithm.1, three threshold $T_{cos}$, $T_{label}$ and $T_{rep}$ are applied. The choice of these threshold values are dataset dependent. This makes the proposed method difficult to be generalized across different datasets / learning scenarios.

Author response:
## Q2 Threshold
The evaluation metric in our paper is mainly area-under-curve (AUC). BEYOND includes three thresholds. For any threshold set, we compute a True Positive Rate (TPR) and a False Positive Rate (FPR). High AUC on different datasets shows that: 1) BEYOND can get a high TPR when FPR is low; 2) when we choose a bad threshold set that has a high FPR, we also get a high TPR. 3) The sensitivity of BEYOND to the threshold is weaker than baselines. Hence, BEYOND is not threshold-sensitive. 
Moreover, in practice, we can choose a threshold with FPR5% only based on clean examples, which is not allowed for baselines that require adversarial examples as a reference set or training. 

Our responses to all questions have been updated to the latest pdf and highlighted in blue.

Reviewer comment (Experiments):
The baselines selected in the paper are somewhat old. Using baselines with the same properties such as neighbors and representations is reasonable, while we believe that comparisons with newer methods with or without such properties are necessary, such as SimCLR for catching and categorizing (SimCat) [1] which also use representations effectively and Erase-and-Restore (E&R) [2].

Author response:
Our proposed BEYOND focuses on detecting adversarial examples by leveraging the relationship between the input and its neighbors. So, the baseline methods we chose are all related to detectors that are built on the distance between the input sample and its neighbors. And since our method uses the SSL model, we also compare BEYOND with Mao et al, which is a SSL-based adaptive-robust defense method.

Moreover, we also included the suggested baselines SimCat from “**[RR1]**” and  E&R from  “**[RR2]**”. The comparison on ImageNet for CW L2 attack is as follows:
| AUC(%) | BEYOND | SimCat | E&R |
|:---:|:---:|:---:|:---:|
| C&W L2 | **96.32%** | 81.85 | 93.57 |

Please note that SimCat is not open source, so the corresponding results come from our own replication. Erase-and-Restore (E&R) is only for L2 norm attacks.
It can be seen that BEYOND outperforms these two baselines. In addition, both SimCat and E&R require training a detector, which is not required inBEYOND. 
For adaptive attacks, Table 5 in **[RR1]** shows that as a few-shot detector, SimCat itself is not robust to adaptive attacks, and its adaptive robustness can only be improved by combining a robust model (e.g. Adversarial training model). 
E&R in **[RR2]** evaluated its robustness to adaptive attack (BPDA), but **[RR3]** shows BPDA is not as effective as APGD used in BEYOND.

**[RR1]** Moayeri M, Feizi S. Sample efficient detection and classification of adversarial attacks via self-supervised embeddings[C]//Proceedings of the IEEE/CVF international conference on computer vision. 2021: 7677-7686.

**[RR2]** Zuo F, Zeng Q. Exploiting the sensitivity of L2 adversarial examples to erase-and-restore[C]//Proceedings of the 2021 ACM Asia Conference on Computer and Communications Security. 2021: 40-51.

**[RR3]** Francesco Croce, Sven Gowal, Thomas Brunner, Evan Shelhamer, Matthias Hein, and Taylan Cemgil. Evaluating the adversarial robustness of adaptive test-time defenses. In International Conference on Machine Learning, pp. 4421–4435. PMLR, 2022

Reviewer comment (Presentation):
The format of citations is incorrect. For example, "kNN Dubey et al. (2019)" in Baselines of Section 4 should be "kNN (Dubey et al., 2019)".

Author response:
Thanks for your careful check, we have corrected the citation format in the latest pdf.

Reviewer comment (Evaluation):
The detection ability for various types of attacks is beneficial for its applications, thus I am concerned about the evaluations of detection the adversarial samples generated by attacks based on different norm.

Author response:
Your concern about robustness to various types of attacks is warranted. Following the reviewer’s comments we test the most representative method that supports multiple norm attacks, AutoAttack. AutoAttack supports $L_{\infty}$, $L_2$ and $L_1$ norm attacks. In the main paper, we only report the detection performance of BEYOND against AutoAttack $L_{\infty}$. The following table shows the performance of BEYOND against AutoAttack with different norms.

| AUC(%) | $L_{\infty}$ | $L_2$ | $L_1$ |
|:---:|:---:|:---:|:---:|
| Cifar10 | 99.18 | 99.13 | 99.07 |
| ImageNet | 97.14 | 97.26 | 97.18 |

The perturbation budgets($\epsilon$) on Cifar10 are 8/255 ($L_{\infty}$), 0.5 ($L_2$), and 8 ($L_1$); and on ImageNet are 8/255 ($L_{\infty}$), 3 ($L_2$), and 64 ($L_1$). The result shows BEYOND is still effective against attacks based on different norms.

Reviewer comment (Presentation):
The figure captions should be made self-contained.

Author response:
Thanks for your useful comments, we will adjust the paper to enhance readability. For the figure caption, we have added the necessary details to make it self-contained in the final paper.

Reviewer comment (Presentation):
It's hard to parse the figures independently of the text.

Author response:
Thanks for your useful comments, we will adjust the paper to enhance readability. For the figure caption, we have added the necessary details to make it self-contained in the final paper.

Reviewer comment (Writing):
(Section 2.1) The paper claims, “Note that BEYOND is not based on random data augmentation.” But in Section 4.1, the paper says, “Augmentations BEYOND uses for generating neighbors are consistent with SimSiam, including horizontal flips, cropping, color jitter, and grayscale.” Aren't SimSiam augmentations random? A clarification will be helpful.

Author response:
We apologize for the confusion. The type of augmentations used by BEYOND is consistent with SimSiam's, including horizontal flips, cropping, color jitter, and grayscale. Since we need to generate multiple different neighbor samples for each input image, we use different combinations of augmentation parameters. However, unlike SimSiam's random augmentations, we fix the random seed to ensure that our method does not benefit from randomization. 
We have modified the corresponding part to make the method setting clearer.

Reviewer comment (Writing):
(Repeated citation names) “Hu Hu et al. (2019)” and "Mao Mao et al. (2021)".

Author response:
Sorry for the discomfort caused by typos and repeated citation names, we have carefully proofread our paper.