Paper: Fooling the Textual Fooler via Randomizing Latent Representations (ICLR-2024)

Reviewer comment (Evaluation):
It's really hard to draw conclusions about robustness-accuracy tradeoffs from Tables 2 and 3. The claim that your method performs "close" to SOTA needs some added substantiation. I'd like: 1.A) Standard deviations on at-least three runs for each setting. In many of these datasets, a 1pp improvement is significant. But accuracies on adversarial datasets often have high variance, so it's unclear how to interpret when your method isn't clearly better and often appears to be much below a handful of the other methods. More details: TMD more reliably seems to perform better especially on Imdb; Roberta, RanMASK and SAFER sometimes outperform on robustness to attack/clean accuracy. It's hard to verify what "close" is, because there are no standard deviations, and on a few of the tasks, AdvFooler really does appear to significantly underperform a couple of the other methods by a bit. (It's still reliably top 3 among the methods though, and I don't think you need to be better than all to have a case for publication).

Author response:
**A: ** Thank you for the valuable comment. We have revised the performance discussion in Section 4.2, and updated the submission. Indeed, AdvFooler is reliably among the top-3 defenses. However, as mentioned in Section 4.2's discussion, AdvFooler possesses other advantages, such as significantly lower training and inference overhead, being plug-and-play, and requiring no access to the training data.

In our experimments, the calculated the p-values of AdvFooler and the better defenses are below 0.01, indicating that the performance results are statistically significance. We also added this into Supplementary of the revised submission.

We really appreciate the interesting suggestion to "tune SAFER and RunMASK until either the robustness or the accuracy matches AdvFooler". We provide the results when tuning RandMASK to a similar accuracy of AdvFooler. As we can observe, the accuracy under attack of RandMASK decreases tremendously, reducing to half of the original AuA. This shows that, while RanMASK was able to be robust against adversarial attacks, this method also requires larger trade-offs in accuracy compared to other defenses. We added this experiment into Section C.8 (Supplementary), including the experimental setup.

|                              |        | TextBugger AuA(%) (ASR(%)↓) | TextBugger Avg. Query↓ | TextFooler AuA(%) (ASR(%)↓) | TextFooler Avg. Query↓ | BERTAttack AuA(%) (ASR(%)↓) |  BERTAttack Avg. Query↓ |
 |------------------------------|--------|:---------------------------:|:----------------------:|:---------------------------:|:----------------------:|:---------------------------:|:-----------------------:|
 |   BERT_MASK (0.9 mask rate)  | 90.14% |       52.50% (41.60%)       |         582.21         |       54.60% (38.93%)       |         511.98         |       61.10% (31.73%)       |          595.57         |
 |   BERT_MASK (0.3 mask rate)  |  93.5% |       22.50% (75.88%)       |         360.12         |       24.10% (74.09%)       |         325.63         |       42.90% (54.12%)       |          412.99         |
 | ROBERTA_MASK (0.9 mask rate) | 90.41% |       52.30% (41.50%)       |         575.94         |       54.20% (39.37%)       |         516.98         |       60.00% (32.43%)       |          594.93         |
 | ROBERTA_MASK (0.3 mask rate) |  93.3% |       34.00% (63.56%)       |         396.72         |        35.7% (61.74%)       |         359.52         |       45.60% (50.97%)       |          429.72         |

Reviewer comment (Experiments):
I think the most compelling case for publication is how lightweight your system is. I would include compute time results somewhere (I was unable to find them). It's also unclear to me that simple adversarial training on word substitutions is actually significantly more compute intensive (generating synonyms is super quick, and training on more examples isn't too much of an added burden when you're already finetuning — especially if you're comparing to needing to tune your noise param for AdvFooler, which while cheap is still some added compute time).

Author response:
**A:** In Table 5, Section C.1, we provided the inference time of AdvFooler  the AdvFooler and other defenses; AdvFooler only adds a negligible inference overhead to the model, while TMD/RandMASK, SAFER, and ASCC all have non-trivial overhead.

Let N_a be the number of noise scales in AdvFooler. Then, for each noise scale, AdvFooler needs to forward through a small test set. Below, we show the training time for FreeLB and InfoBERT and base-training time + tuning time for AdvFooler (for fair comparison), on AGNEWS. We can observe that, FreeLB and InfoBERT incur significant training overhead, compared to the tuning+base-model training time of AdvFooler.


| Model     | AGNEWS                   (s) | 
|-----------|------------------------------|
| BERT-base |                        12150 |
| FREELB    |                        52370 |
| INFOBERT  |                        76650 | 
| AdvFooler |                 12150+40*N_a |

Reviewer comment (Experiments):
In your system, HotFlip is not perhaps an accurate white-box attack (Appendix C7). HotFlip relies on gradients, but are you giving it the unnoised gradients, or the parameters across all iterations of the model (so someone can reverse engineer the likely original unnoised parameters)? I suspect that if it were truly a white box attack, the certified robustness systems and TMD would outperform. I think these results as presented are a bit misleading otherwise, and one option is to remove them. I would suggest either demonstrating the white-box capabilities fully by (1) giving access to unnoised params, or each set of params so you can derive unnoised params AND (2) comparing to TMD and the certified methods. OR I'd like to see the claim about the defense being "attack agnostic" in the main paper, and the claims of being robust to "white-box attack" toned down.

Author response:
Thank you for the suggestion. Following your suggestion, we disable the noise in AdvFooler when calculating the gradient, letting HotFlip create the adversarial examples using the base model. Similarly, for RandMASK and TMD, we give the adversary access to the base model. 


 | Model     | AGNEWs [AUA%(ASR%)] | IMDB [AUA%(ASR%)] |
 |-----------|---------------------|-------------------|
 | BERT-Base | 37.7% (59.98%)      | 59.3 (35.82%)     |
 | FREELB    | 51.1% (45.87%)      | 62.3% (31.08%)    |
 | INFOBERT  | 43.3% (54.08%)      | 50.3% (44.66%)    |
 | AdvFooler | 48.3% (48.50%)      | 77.3% (16.61%)    |
 | RanMASK   | 85.3% (3.9%)        | 84.1% (9.8%)      |
 | TMD       | 44.2% (52.72%)      | 63.3% (29.9%)     |

The results show that AdvFooler has better robustness than TMD, slightly lower than FreeLB on AGNEWS but better on IMDB. RandMASK still has the best robustness, but as indicated, its clean accuracy drops by more than 4%. Given the results, we also think that rigorous evaluation of the existing black-box defenses against white-box attacks can be an interesting, independent work.

Reviewer comment (Experiments):
New Baseline Suggestion 1/2: While I understand that it would not be as lightweight as AdvFooler, the style of your method suggests that a comparison with a diverse ensemble designed to induce adversarial robustness would be very apt since it also tries to induce robustness in much the same way (added random noise) (see my strengths section for the analogy). (I suggest this one: https://proceedings.mlr.press/v97/pang19a , the ADP regularizer).

Author response:
Thank you for your suggestion. Although AdvFooler with different noise vectors can be treated as individual members in the diverse ensemble, our method achieve adversarial robustness from a different principle. For ADP, the diversity in non-maximal predictions reduces the transferability of adversarial samples across the members of the ensemble. On the other hand, we only apply a single noise for each query but do not ensemble the predictions, resulting in a randomized model. As discussed in Theorem 1 of Section 3.2, this randomizing property help the model mislead the attack and thus make it more challenging for the adversary to find adversarial samples. We will include this discussion in the revision.

Reviewer comment (Experiments):
New Baseline Suggestion 2/2: Choose Dirichlet Neighborhood Ensemble (DNE) over ASCC. While both methods model the perturbation space as the convex hull of word synonyms, DNE far outperforms ASCC (See TMD paper Table 1 where DNE nearly outperforms TMD on AGNews while maintaining a high clean accuracy).

Author response:
**A:** As suggested, we trained the Dirichlet Neighborhood Ensemble using the TextDefender implementation for both dataset, with both BERT and RoBERTa as the pre-trained models. 


|                          |        | TextBugger AuA(%) (ASR(%)↓) | TextBugger Avg. Query↓ | TextFooler AuA(%) (ASR(%)↓) | TextFooler Avg. Query↓ | BERTAttack AuA(%) (ASR(%)↓) |  BERTAttack Avg. Query↓ |
|--------------------------|--------|:---------------------------:|:----------------------:|:---------------------------:|:----------------------:|:---------------------------:|:-----------------------:|
|      BERT_DNE_AGNEWs     | 94.21% |       35.20% (62.15%)       |         335.39         |       37.70% (59.42%)       |         305.36         |       54.80% (41.01%)       |          360.91         |
| BERT_AdvFooler_AGNEWs    | 93.67% |       50.10% (45.90%)       |         819.44         |       50.10% (45.90%)       |         701.05         |       53.40% (42.89%)       |          752.61         |
|    ROBERTA_DNE_AGNEWs    | 94.89% |       51.80% (45.19%)       |         425.38         |       52.60% (44.34%)       |         378.81         |       57.60% (39.05%)       |          413.23         |
| ROBERTA_AdvFooler_AGNEWs | 94.21% |       51.70% (44.88%)       |         804.99         |       53.30% (42.81%)       |         717.31         |       57.80% (38.38%)       |          767.47         |
|       BERT_DNE_IMDB      | 90.15% |       29.40% (67.62%)       |         1470.86        |       30.10% (67.46%)       |         1333.54        | 35.70% (61.28%)             | 1674.86                 |
| BERT_AdvFooler_IMDB      |  91.9% |       42.40% (53.41%)       |         3261.41        |       49.10% (47.32%)       |         2759.37        |       40.70% (55.76%)       |         2645.36         |
|     ROBERTA_DNE_IMDB     | 95.54% |       56.30% (40.96%)       |         1321.3         |       59.30% (37.97%)       |         1292.96        |       46.90% (50.48%)       |          949.42         |
| ROBERTA_AdvFooler_IMDB   | 92.69% |       60.30% (35.58%)       |         4349.38        |       64.70% (30.20%)       |         3360.81         |       50.20% (45.55%)       |         3274.13         |

 
DNE’s accuracy under attack in most case are quite high, being able to surpass ASCC, FreeLB, and InfoBERT in most case, as shown in DNE’s performance on IMDB and AGNEWs dataset. However, AdvFooler still outperforms DNE except in one or two scenarios. This shows that AdvFooler is consistently competitive or sometimes better than adversarial training and ensemble defenses.

Reviewer comment (Theory):
The motivation is not clear. Why does such randomization fool the attackers while not degrading the benign performance?

Author response:
**Q1: The motivation is not clear. Why does such randomization fool the attackers while not degrading the benign performance?***

**A:** Thank you for the comment. As explained in Section 3.2 and Theorem 1, the proposed latent feature randomization can fool the query-based attackers since it can mislead the search direction of the adversary's querying process. However, this randomization *will affect/degrade the benign performance*, a fact we specifically mention in Section 3.3. AdvFooler and other randomization-based defenses, such as RandMask and SAFER, will affect the benign performance. As demonstrated in Figure 4, increasing the noise scale leads to a gradual decrease in benign accuracy; at some point, the large noise scale can even lead to a significant decrease in benign performance. We suggest, in Section 4.3 discussing this figure, that the defender should choose a noise scale for their need (e.g., noise scale corresponds to 1% drop in benign accuracy). 


To summarize, we'd like to clarify that AdvFooler's randomization can fool the attackers but will induce a small drop the benign performance. In addition, unlike RandMASK and SAFER, it only has negligible training and inference overhead and can be plugged into any already trained model for better robustness (Table 1).

Reviewer comment (Experiments):
Why does AdvFooler can only perplex query-based black-box attacks? It is significant for a defense method to defend against various attacks, such as white-box attacks [1], decision-based attacks [2,3], and so on. It is necessary to validate the effectiveness against these attacks to show the generality of AdvFooler.

Author response:
**Q2: Why does AdvFooler can only perplex query-based black-box attacks? It is significant for a defense method to defend against various attacks, such as white-box attacks [1], decision-based attacks [2,3], and so on. It is necessary to validate the effectiveness against these attacks to show the generality of AdvFooler.**

**A:** We'd like to emphasize the practicality of threat model discussed in the paper. For many companies with ML applications, the trained model as well as training data are valuable assets to the companies, thus are secretly guarded. Thus, the attacker rarely has access to the trained model or model’s architecture to perform white-box attacks. Instead, the attacker will rely on querying the models so that they can estimate the optimal perturbation toward adversarial examples, as discussed in Section 1. The same black-box, query-based threat model is frequently examined in several related works [(Nguyen Minh and Luu, 2022), (Zeng et al, 2021), (Dong et al, 2021), (Si et al, 2021), (Li et al, 2021)]. 

Nevertheless, we also agree that, for broad applicability, it is necessary to validate the effectiveness of AdvFooler against white-box and decision-based attacks, which we already provided in C.7 and C.6 (Supplementary), respectively. These experiments confirm that AdvFooler is still effective against the evaluated white-box and decision-based attacks. As discussed in these experiments, we conjecture that, for white-box attacks, AdvFooler's randomization nullifies the adversarial perturbation and brings it back to the correct side of the decision boundary; since decision-based attacks still involve querying the model, AdvFooler's randomization can return incorrect prediction for some perturbed samples, especially near the decision boundary, and therefore misleads the attacks' optimization process.

Reviewer comment (Evaluation):
Despite its advantages in terms of simple implementation and minimal computational overhead, AdvFooler's performance falls short of the state-of-the-art. In Table 2, on the AGNEWS dataset, AdvFooler exhibits lower accuracy under attack compared to RanMASK for the BERT-base model, and it also demonstrates lower accuracy under attack than both TMD and RanMASK for the RoBERTa-base model.

Author response:
**Q1: Despite its advantages in terms of simple implementation and minimal computational overhead, AdvFooler's performance falls short of the state-of-the-art. In Table 2, on the AGNEWS dataset, AdvFooler exhibits lower accuracy under attack compared to RanMASK for the BERT-base model, and it also demonstrates lower accuracy under attack than both TMD and RanMASK for the RoBERTa-base model.**

We acknowledged, in Section 4.2, that AdvFooler has slightly lower performance than RandMASK and TMD in some experiments. 

But, as explained in Section 1, AdvFooler has other important advantages compared to RandMASK and TMD: (1) having negligible computation overhead while (2) simultaneously achieving comparable robustness to RandMASK and TMD, and (3) being plug-and-play, which allows it to be used without modifying existing parameters of the model. Simultenously achieving (1) and (2) is an extremely challenging task; for example, TMD and RandMask require significant additional training and inference overhead for similar robustness to AdvFooler. Also for, for RandMASK, its clean accuracy drops significantly (>4% for RandMASK vs. < 1% for AdvFooler). For these reasons, AdvFooler is significantly more practical for usage in real-world applications, a significant contribution to the adversarial defense domain.

Reviewer comment (Experiments):
The selection of the hyper-parameter for noise scale in AdvFooler is not entirely clear. The authors claim that they choose the ν value based on the criterion that the clean accuracy drops by at most 1% using the test set. However, it seems not the case in Table 2 and 3. In Figure 4, the curves of AuA are not monotone. Do authors also consider AuA values when choosing noise scale?

Author response:
**Q2: The selection of the hyper-parameter for noise scale in AdvFooler is not entirely clear. The authors claim that they choose the $\nu$ value based on the criterion that the clean accuracy drops by at most 1% using the test set. However, it seems not the case in Table 2 and 3. In Figure 4, the curves of AuA are not monotone. Do authors also consider AuA values when choosing noise scale?**

**A:** We'd like to, first, re-iterate the discussion in Section 3.3: "the defender can select the noise scale $\nu$ at which the variance causes the clean accuracy drops by a chosen percentage (e.g., in our experiments, it is 1\%)". In practice (as discussed in threat model), the defender only has the trained model and a small clean test set and makes no assumption about the attacks; using the clean test set, defender finds the largest noise scale corresponding to a certain, tolerable drop in clean accuracy. For example, given the values we used to plot Figure 4 as in the table below, starting with the original clean accuracy of 95.14%, the defender would choose $\nu=0.6$, which corresponds to 94.57% clean accuracy, assuming a selection of 1% drop in performance. For 2% drop (or 94.57% clean accuracy), $\nu$ is set to 0.80. As we can see, the larger $\nu$ generally increases the robustness (43.90% and 48.80% AuA for $\nu=0.6$ and $\nu=0.8$, respectively, on TextFooler); however, when $\nu$ is too large, as we can observe in this figure, it will also interfere clean accuracy (a sudden, significant drop in clean accuracy) and correspondingly decrease the robustness of the model as well. 



| noise_type\intensity | 0.10   | 0.20   | 0.30   | 0.40   | 0.50   | 0.60   | 0.70   | 0.80   | 0.90   | 1      | 1.10   | 1.20   | 1.30   | 1.40   | 1.50   |
|:--------------------:|--------|--------|--------|--------|--------|--------|--------|--------|--------|--------|--------|--------|--------|--------|--------|
| Clean (95.14%)       | 95.17% | 95.06% | 95.04% | 94.94% | 94.69% | 94.57% | 94.11% | 93.67% | 93.12% | 92.48% | 90.74% | 87.79% | 83.18% | 77.04% | 70.05% |
| AuA (TextFooler)     | 38.30% | 38.40% | 41.10% | 43.90% | 45.60% | 48.10% | 48.80% | 50.10% | 45.50% | 35.50% | 21.90% | 13.30% | 8.40%  | 5.80%  | 4.40%  |
| AuA (TextBugger)     | 36.00% | 37.10% | 38.20% | 40.40% | 45.60% | 46.90% | 49.90% | 50.10% | 44.40% | 33.40% | 20.20% | 13.70% | 8.20%  | 5.40%  | 4.20%  |
| AuA (BERTAttack)     | 46.70% | 48.80% | 50.30% | 47.50% | 50.30% | 55.40% | 54.50% | 53.40% | 48.60% | 36.20% | 22.10% | 12.10% | 7.10%  | 5.10%  | 4.40%  |

Reviewer comment (Experiments):
Could the authors compare and contrast their approach to applying DP-style noise to the embeddings generated by the hidden layer?

Author response:
Thank you for the comment; however, we’re not sure what DP stands for and will assume DP refers to Differential Privacy, a privacy protection framework that introduces a small change in the data for a guaranteed, bounded change in the distribution of the algorithm’s outputs. DP-style defenses such as (Lécuyer et al 2019) certify the defense under the DP framework; it is important to note that SAFER and RandMASK, evaluated in the paper, are also certified defenses and share a similar approach to DP-style defenses. DP-defense and RandMASK still require training overhead to tune the added noise so that it does not interfere with the model's prediction. On the other hand, AdvFooler adds noise to the latent features at inference time to misdirect the attacker's querying process.  
  
We have added this discussion to our revised submission.  
  
Lécuyer, M., Atlidakis, V., Geambasu, R., Hsu, D.J., & Jana, S.S. (2018). Certified Robustness to Adversarial Examples with Differential Privacy. *2019 IEEE Symposium on Security and Privacy (SP)*, 656-672.

Reviewer comment (Evaluation):
Results from Table 2 suggest that their approach is not the very best compared to other approaches. What are the merits of the proposal then? Apart from reduced computational overheads?

Author response:
As explained in Section 1 (especially Table 1), AdvFooler (1) reduces significant computation overhead while (2) simultaneously achieving comparable robustness to other defenses. This is an extremely challenging task; for example, TMD/RandMask/SAFER or adversarial training approaches require significant additional training overhead for similar robustness to AdvFooler; TMD/RandMask/SAFER also incurs non-trivial inference overhead, making them less practical in real-world application. Another advantage of AdvFooler is (3) plug-and-play, as also indicated in Table 1, which allows it to be used without modifying existing parameters of the model. Finally, as indicated in Section 3.3 and 4, AdvFooler allows the user to (4) control how much accuracy they want to trade-off for robustness using a small test set. (1), (2), (3) and (4) make AdvFooler significantly more practical to employ in real-world applications, a significant contribution to the adversarial defense domain.

Reviewer comment (Theory):
The authors empirically note that adding small amounts of noise to the logins does not change the prediction. But this is not fundamental nor clear why this is the case. Could the authors elaborate?

Author response:
Defenses via adding noise (including AdvFooler and RandMASK/SAFER) will inevitably affect the prediction or clean accuracy. While RankMASK/SAFER perturbs the input and require additional training and/or inference overhead to tune the added noise for a small accuracy drop, AdvFooler adds noise to the hidden features only at inference time, an approach which is theoretically shown to be able to fool the attacker (discussed in Section 3.2). To choose this noise, however, we suggest using a small test set and finding a large-enough noise for a specific tolerance of accuracy drop (discussed in Section 3.3 and Figure 4). This allows the defender to control how much accuracy they want to trade off and experiments show that our approach can achieve comparable robustness with other defenses while having only <1% accuracy drop.

Reviewer comment (Evaluation):
Alternatively, the adversary could also replace subsets of words using brute force. For small spans of text (as considered in the evaluation), this is not computationally prohibitive. Could the authors elaborate further the specific scenarios where important word selection inhibition is prudent?

Author response:
Thank you for the interesting suggestion. As discussed Section 1, adversarial black-box query-based attack methods manipulate different aspects of the input sentence, such as replacing words, without significantly changing the original semantic in the perturbed inputs while limiting the number of queries to the model within *a budget*. In a realistic and standard evaluation setting (considered in our paper, Section 4, and others in the literature), an adversary is only allowed to query at most $50 * L$ (L: number of words), otherwise, the attack is *practically* infeasible. For a text with 38 words (average length of AGNEWS), assuming just 3 synonyms (on average) for each word, the number of queries should be 3^38 >> 50 * 38; when it's the 90th percentile, it's still 3^27 >> 50 * 38. For these reasons, an effective adversarial text attack must consider a better strategy than brute-force solutions.  
  
For brute-force attacks to be realistic, the text length should be quite small. In this scenario, we believe that AdvFooler, as well as other defenses, will not be effective if at least one of the perturbed versions of the text are an adversarial example.  
  
In Section C.6 (Table 11), on the other hand, we evaluated AdvFooler against a more-efficient brute-force style attack, called Hard-label, where the attack first randomly perturb (without consideration for preserving the semantic of a perturbed sample) the input until a perturbed sample flips the prediction. This is a more efficient, brute-force process since not paying attention to semantic preservation means the attack can find the perturbed sample with a different prediction very quickly. Then, in the second step, the attack moves this sample closer to the original sample until they're semantically similar. Still, AdvFooler is effective against Hard-label attacks.