Paper: Understanding the Robustness of Randomized Feature Defense Against Query-Based Adversarial Attacks (ICLR-2024)

Reviewer comment (Presentation):
The organization of this paper can be improved. Assumption 1, Figure 1 and 2 are not referred to in the main text. It is confusing on the purpose of presenting the assumption 1 and Figure 1 and 2.

Author response:
**Q1: Assumption 1, Figure 1 and 2 are not referred to in the main text. It is confusing on the purpose of presenting the assumption 1 and Figure 1 and 2.**

A: Thank you for the observation. We have made incorrect references for Figures 1 and 2, when including the Supplementary File within the Main paper. Please see our clarifications below:
- ‘Figure 3’ in Section 3.3 should be ‘Figure 1’, 
- ‘Figure 5’ in Section 3.5 should be ‘Figure 2’, 
- The discussion for Assumption 1 is immediately above its declaration. It allows us to ensure that small noise diffuses but does not shift the prediction, or adversarial samples of the original model are adversarial samples of the randomized model. 
We have updated this discussion in the revised submission.

Reviewer comment (Theory):
The assumption and the theorem are incorrect. Even when the noise is small, the expectation of the randomized model is not necessarily consistent with the original model on the same data point, one simple counter-example is that when the input x is at the decision boundary, a small perturbation can change the prediction, so small noise may change the prediction. Theorem 1 is based on incorrect derivation, Eq. (23) and (24) may be incorrect as the gradient $\nabla_{h(x)}(L \cdot g)$ is anisotropic so the multiplication with Gaussian noise should not be an i.i.d. Gaussian noise. In addition, the assumption of the proof is the value of v and $\mu$ are small, so the approximation holds, but in the experiments, the value of $v$ is not present, and the value of $\mu$ is as large as 0.3, which is not negligible.

Author response:
**Q2: The assumption and the theorem are incorrect. Even when the noise is small, the expectation of the randomized model is not necessarily consistent with the original model on the same data point, one simple counter-example is that when the input x is at the decision boundary, a small perturbation can change the prediction, so small noise may change the prediction.**

A: As discussed in Section 3.2,  given Assumption 1, adversarial samples of the original model are adversarial samples of the randomized model, on expectation. In other words, at the decision boundary, while a specific sample of the small Gaussian noise may change the prediction, but expectedly, it should be the same prediction as that of the original model (across several samples of the small Gaussian noise). This assumption is typically made in other relevant randomization works, including the evaluated RND defense.

Reviewer comment (Evaluation):
The evaluation of the decision-based attack is insufficient and the results are not good. It seems the proposed method only works on RayS, and the results on DeiT and ViT are not presented.

Author response:
**Q5: The evaluation of the decision-based attack is insufficient and the results are not good. It seems the proposed method only works on RayS, and the results on DeiT and ViT are not presented.**

A: Similar to RND, our method indeed doesn’t work well only on RayS/CIFAR10 with VGG19 and ResNet50 networks, as already acknowledged in Section 4.3. However, both RND and ours are significantly more effective in all other decision-based experiments, including the additional experiments with DeiT and ViT networks (results provided below, as suggested by the reviewer). Please note that AAA, another popular defense for score-based attacks that ours and RND also are effective against, is not effective at all for both RayS and SignFlip. 

In general, we believe that bulletproof protection of a model against adversarial attacks is an extremely difficult endeavor; to the best of our knowledge, no single defense is able to be effective against all different types of attacks in all tested scenarios. Nevertheless, as demonstrated in the paper, our method and RND both can provide protection against a broad range of attacks in the majority of scenarios, including score-based attacks, decision-base attacks, EoT attacks, white-box attacks, etc…

We also believe that the best protection is to combine various types of defenses, consequently combining the strengths of each defense. We demonstrate this potential direction in Section 4.2 (Table 4), where combining our method and Adversarial Training significantly boosts the performance in all experiments from 30-40% accuracy under attack (of each method separately) to 70-80% accuracy under attack.

*Decision-based attacks on DeiT and ViT on ImageNet with 10k iterations*

| |         | RayS| SignFlip|
|----------|---------|------|----------|
| ViT      | Base    |  1.7 |      9.7 |
|          | Input   |   41 |     70.3 |
|          | Feature | 40.3 |     70.2 |
| DeiT| Base    |  3.7 |      5.9 |
|          | Input   | 52.5 |     75.5 |
|          | Feature | 49.7 |     73.7 |

*EOT decision-based attacks with 2k iterations*

|  |         | RayS| SignFlip|
|--------------|---------|------|----------|
| ViT      | Base    |  7.5 | 39.5  |
|              | Input   | 39.3 | 70.9 |
|              | Feature | 41.3 | 72.3 |
| DeiT| Base    | 12.8 | 40.7  |
|              | Input   | 52.6 | 76.5  |
|              | Feature | 54.7 | 75.5  |

Reviewer comment (Theory):
Please verify if the Eq. (23) and the Eq. (24) are correct.

Author response:
**Q6: Please verify if the Eq. (23) and the Eq. (24) are correct.**

A: Please see the answer to Question 3.

Reviewer comment (Theory):
Please verify that assumption 1 is correct and that the theorems and experiments are strictly following this assumption.

Author response:
**Q7: Please verify that assumption 1 is correct and that the theorems and experiments are strictly following this assumption.**

A: Please see the answer for Questions 2 & 3.

Reviewer comment (Experiments):
I am curious about the impact of the ratio on the robustness when the gradients are fixed. Can you present the experimental results if possible?

Author response:
**Q8: I am curious about the impact of the ratio on the robustness when the gradients are fixed. Can you present the experimental results if possible?**

A: As provided in our analysis, the robustness of our randomization depends on the attack radius $\mu$, the defense noise scale $\nu$, and the ratio of the gradients. While $\mu$ and $\nu$ are fixed after the attack launches the attack and the defender set up the defense, respectively, the gradients are input-dependent. At each query step of an attack on an input, the perturbed input changes, thus the corresponding gradients also change. This is a distinct characteristic of our defense, compared to RND, and we show that the deeper the randomized layer is, its corresponding ratio is higher, leading to a more robust model.

Reviewer comment (Evaluation):
The paper defines a successful attack (page 4) as one where the adversarial example can fool the randomized model in the *majority* of its applications on the example. In security-sensitive applications (e.g., authentication, malware detection), it is enough for the model to be fooled *once*. Randomness enables an attacker to keep submitting the same adversarial example until, by chance, the model misclassifies it.

Author response:
**Q2: Definition of a successful attack. I argue that in security-sensitive applications (e.g., authentication, malware detection, etc.), it is enough for the model to be fooled once**

A: Our empirical experiments focus on evaluating whether the attack can truly find adversarial examples. For non-randomized models, when an attack arrives at the decision that $x$ is an adversarial example, $x$ unquestionably is on the other side of the decision boundary. However, for a randomized model, $x$ could be still on the correct side of the decision boundary, but the added randomization shifts it to the other side of the decision boundary; thus, in principle, $x$ is still not an adversarial example, and if we use 1 single application for evaluation, an attack could be lucky or a randomized defense could be unlucky. Consequently, for a fair evaluation of the effectiveness of a defense, we forward $x$ multiple times and decide that it is an adversarial example if the majority of the results say so, as seen in our paper.

Nevertheless, we understand that in the case where we ignore fair evaluation, the attack is allowed to be lucky and just has to fool the defense once, as given in the example in the comment. Therefore, we also provide the experiments when forwarding $x$ only once, as below. As we can observe, our defense is still effective against the attacks.

*Robustness with a single trial, 10k queries, 1%-accuracy noise scale, ImageNet*
|  |       | Square  | NES     |
|--------------|-------|---------|---------|
| ViT          | Base | 0 | 7.3 |
|              | Input | 44.0| 47.0 |
|              | Feature | 44.8| 42.5 |

Reviewer comment (Theory):
All analysis in the paper uses the first-order Taylor approximation, but this assumption should be stated more clearly. There is concern that the approximation may not be good when the added noise or perturbation is relatively large. Neural networks are highly nonlinear, so a better justification of the approximation is needed. An empirical evaluation of the approximation error would make the analyses more convincing.

Author response:
**Q3: All the analysis in the paper uses the first-order Taylor approximation. An empirical evaluation of the approximation error would make all the analyses more convincing.**

A: Thank you for your suggestion. We include the histogram of the approximation error of ViT on ImageNet in Section F.1 (Supplementary). As we can observe, the error is relatively small.

Reviewer comment (Experiments):
It is unclear which intermediate layer is picked for all the results in Section 4, and whether it is selected empirically or according to some metric. A clear heuristic for picking the intermediate layer should be proposed, along with measurement of how much the heuristic is inferior to the best possible choice.

Author response:
**Q4: First, I wonder which intermediate layer is picked for all the results in Section 4. empirically or according to some metric?**

We pick the penultimate layers of the base models for randomization, in all experiments. We have shown with theoretical analysis in Figure 1 (main) and empirical analysis in Table 12 (supplement), that the randomizing at a deeper layer consistently achieves the best robustness; for example, randomizing the last layer in VGG or ViT achieves 8-10% improvement over the previous layer. We have already revised the paper (Section 4.1) to clarify this discussion accordingly.

Reviewer comment (Experiments):
Although some adaptations of the attacks for the randomized defense have been made (EoT and Appendix D.4), there could be other alternatives. One idea is to use attacks that **maximize loss given a fixed $\epsilon$ budget**, which may be less susceptible to randomness because they do not have to find a precise location near the decision boundary.

Author response:
**Q5: The authors have taken some steps to adapt the attacks for the randomized defense, mimicking obvious modifications that the real adversary might do (EoT and Appendix D.4). I really like these initiatives and also wonder if there are other obvious alternatives. One that comes to mind is to use attacks that maximize loss given a fixed $\eps$ budget. These attacks should not have to find a precise location near the decision boundary which should, in turn, make it less susceptible to randomness.**

**This actually does NOT mean that the randomness is not beneficial. Suppose that the loss-maximizing attack operates by estimating gradients (via finite difference) and just doing a projected gradient descent. One way to conceptualize the effect of the added noise is a noisy gradient, i.e., turning gradient descent into *stochastic* gradient descent (SGD). SGD convergence rate is slowed down with larger noise variance so the adversary will have to either use more iterations or more queries per step to reduce the variance. Either way, the attack becomes more costly. I suggest this as an alternative because it directly tests the benefits of the added noise without exploiting the fact that the distance-minimizing attacks assume deterministic target models.**

A: Thank you for your appreciation of our analysis and indeed a great observation regarding the fixed budget. In fact, NES, Square attack, and SignHunt, all of which are evaluated in our paper, work in the principle of maximizing loss given a budget. Specifically, NES approximates the gradient directly and applies projected gradient descent, while Square attack and SignHunt search for the adversarial sample at the boundary of the $\ell_p$ ball.

Reviewer comment (Experiments):
Even with loss-maximizing attacks, randomness can still be beneficial. If the attack operates by estimating gradients (via finite differences) and uses projected gradient descent, added noise can be conceptualized as noisy gradients, turning gradient descent into *stochastic* gradient descent (SGD). Larger noise variance slows SGD convergence, forcing the adversary to either use more iterations or more queries per step, increasing attack cost. This kind of attack could directly test the benefits of the added noise without relying on the weaknesses of distance-minimizing deterministic attacks.

Author response:
**Q5: The authors have taken some steps to adapt the attacks for the randomized defense, mimicking obvious modifications that the real adversary might do (EoT and Appendix D.4). I really like these initiatives and also wonder if there are other obvious alternatives. One that comes to mind is to use attacks that maximize loss given a fixed $\eps$ budget. These attacks should not have to find a precise location near the decision boundary which should, in turn, make it less susceptible to randomness.**

**This actually does NOT mean that the randomness is not beneficial. Suppose that the loss-maximizing attack operates by estimating gradients (via finite difference) and just doing a projected gradient descent. One way to conceptualize the effect of the added noise is a noisy gradient, i.e., turning gradient descent into *stochastic* gradient descent (SGD). SGD convergence rate is slowed down with larger noise variance so the adversary will have to either use more iterations or more queries per step to reduce the variance. Either way, the attack becomes more costly. I suggest this as an alternative because it directly tests the benefits of the added noise without exploiting the fact that the distance-minimizing attacks assume deterministic target models.**

A: Thank you for your appreciation of our analysis and indeed a great observation regarding the fixed budget. In fact, NES, Square attack, and SignHunt, all of which are evaluated in our paper, work in the principle of maximizing loss given a budget. Specifically, NES approximates the gradient directly and applies projected gradient descent, while Square attack and SignHunt search for the adversarial sample at the boundary of the $\ell_p$ ball.

Reviewer comment (Evaluation):
A suggested additional figure is a **scatter plot of the clean accuracy vs the ratio in Eq. (14)** to provide empirical confirmation for this part of the theoretical analysis as well.

Author response:
**Q6: additional figures: Scatter plot of the robustness vs the ratio in Theorem 1. Scatter plot of the clean accuracy vs the ratio in Eq. (14). Robustness-accuracy trade-off plot**

Thank you for your suggestions. In fact, the robustness averaged over the test samples at each layer, corresponding to Figures 1 and 2, are provided in Table 12 (Supplementary). We can observe an increase in robustness as the ratio becomes higher at deeper layers. We have updated these Figures, in the revised paper, with the performance results created in Table 12, for a better explanation.

As discussed in Section 4, our defense allows the user to choose $\nu$ that results in a “controlled” decrease in clean accuracy, using a small test set. For example, we provided the results of varying $\nu$ in Tables 1 and 2 that correspond to 1%, 2%, 3% and 4%. drops in clean accuracy. In general, increasing $\nu$, or more randomness, will lead to a decrease in clean accuracy but also an increase in robustness, as observed in these tables. In addition, we provide the robustness (Accuracy under Attack) when varying $\nu$ (corresponding varying drops in clean accuracy) on CIFAR10/VGG19/Square Attack setting:
AuA = [56.2, 58.4, 62.3, 62.8, 63.5]
Acc = [95.33, 94.94, 94.55, 93.58, 92.09]

Reviewer comment (Evaluation):
A suggested additional figure is a **robustness-accuracy trade-off plot**, a key concept for evaluating adversarial defenses. This plot should vary $\nu$ and intermediate layers to fully characterize the trade-off, aiding in the choice of the best intermediate layer.

Author response:
**Q6: additional figures: Scatter plot of the robustness vs the ratio in Theorem 1. Scatter plot of the clean accuracy vs the ratio in Eq. (14). Robustness-accuracy trade-off plot**

Thank you for your suggestions. In fact, the robustness averaged over the test samples at each layer, corresponding to Figures 1 and 2, are provided in Table 12 (Supplementary). We can observe an increase in robustness as the ratio becomes higher at deeper layers. We have updated these Figures, in the revised paper, with the performance results created in Table 12, for a better explanation.

As discussed in Section 4, our defense allows the user to choose $\nu$ that results in a “controlled” decrease in clean accuracy, using a small test set. For example, we provided the results of varying $\nu$ in Tables 1 and 2 that correspond to 1%, 2%, 3% and 4%. drops in clean accuracy. In general, increasing $\nu$, or more randomness, will lead to a decrease in clean accuracy but also an increase in robustness, as observed in these tables. In addition, we provide the robustness (Accuracy under Attack) when varying $\nu$ (corresponding varying drops in clean accuracy) on CIFAR10/VGG19/Square Attack setting:
AuA = [56.2, 58.4, 62.3, 62.8, 63.5]
Acc = [95.33, 94.94, 94.55, 93.58, 92.09]

Reviewer comment (Novelty):
The originality/novelty of the method is a minor weakness. The main contribution is the analysis of the gradient norm (model sensitivity) for benign and perturbed samples. Adding noise to an intermediate layer instead of the input is relatively incremental. The theoretical analysis is strong but could be complemented by more empirical results as earlier suggested.

Author response:
**Q7: The most important contribution is the analysis on the gradient norm (i.e., sensitivity of the model) of benign and perturbed samples. The proposal to add noise to the intermediate layer instead of the input in itself is relatively incremental. However, the theoretical analysis does seem particularly strong to me, even though it does build up a nice intuition of the scheme. This is a minor weakness to me personally, and I would like to see more empirical results, as suggested earlier**

A:  Thank you for appreciating our work. Here, we summarize our responses to the earlier questions:
- Evaluation with a single application on the successful query (answered in Q2, added to the revised paper)
- Error analysis of first-order approximation (answered in Q3, added to the revised paper)
- Performance at different layers with the same drop in accuracy (answered in Q6, provided in Table 12 of original submission)
- Robustness-Accuracy tradeoff (answered in Q6 with additional experiments)
Different from existing randomized-feature defenses, which additionally require adversarial expensive training and ensemble, our work opens the possibility of randomizing the latent layers at **inference time** without these hassles (being lightweight and plug-and-play) and with a strong theoretical foundation. In addition, unlike randomizing the input, such as RND, our defense gives the user protection against non-continuous input, such as graph data; however, this is beyond the scope of our paper and deserves another independent study.

Reviewer comment (Novelty):
The idea of injecting noise into hidden features is not novel, seeing Parametric Noise Injection: Trainable Randomness to Improve Deep Neural Network Robustness against Adversarial Attack, CVPR 2019. Although this is for defending against white-box attacks, adopting it for black-box attacks does not seem a significant contribution.

Author response:
Although injecting noise to the features is previously proposed (such as in the mentioned paper), these methods require additional training to tune the noise, which is computationally expensive, especially when dealing with large and complex datasets. In contrast, our defense, and also RND, are lightweight, plug-and-play and can be applied to any pretrained model without additional model training. Our defense is effective against black-box attacks because it is designed to fool the querying process of these attacks, as seen in our theoretical and empirical analysis.

Reviewer comment (Experiments):
Does the proposed method have an advantage against AAA in defending score-based attacks? AAA is not designed for decision-based attacks, where the authors use AAA for comparison.

Author response:
We already provided the experiments, comparing AAA to our defenses on score-based attacks in Section E.3 (Supplementary). The experiments show that that our defense improves the robustness and has comparable results to AAA on score-based attacks. However, for decision-based attacks, as provided in Section 4.3, our defense has superior performance. From a practical perspective, a defense should work well against many types of query-based black-box attacks, and these experiments show that our defense has more practical utility than AAA.

Reviewer comment (Experiments):
I understand that the paper focuses on black-box attacks, but in the experiment section, the authors may try evaluating models with white-box attacks as well.

Author response:
Our work focuses on studying the effect of randomized models on black-box attacks, which are more practical in MLaaS system. In these systems, the adversary rarely has access to the trained model or model’s architecture to perform white-box attacks. Although our analysis does not cover white-box attacks, in Section E.4 we still conduct experiments for white-box attacks on randomized feature defense. The experimental results on C&W and PGD show that our defense can also boost the model’s robustness against white-box attacks, which demonstrates the broad utility of our method beyond black-box attacks.

Reviewer comment (Evaluation):
In page 5, can you please give a reason for this sentence "We can observe that these ratios become higher when the data are perturbed toward the adversarial samples. In other words, the randomized model is more robust during the attack."?

Author response:
In our theoretical analysis, we show that the robustness to black-box attacks is controlled by three terms: the scale of noise vector added by the attack $\mu$ and the defense $\nu$, and the ratio of the norm of the gradient with respect to the feature where noise is added and the input $\frac{||\nabla_h(L\circ g)||}{||\nabla_h(L\circ g)||}$. While the scales of the noise vectors are fixed during an attack, the last term is dependent on the input and where in the model the defense is performed.  During the attack, the input is perturbed and thus changes the above ratio if noise is injected into hidden layers of the model. Figure 1 illustrates that the ratio increases, especially for deeper randomized layers, when the attack happens, implying that randomized feature defense has a higher chance than input defense to fool black-box attacks.