DA ^3 : A Distribution-Aware Adversarial Attack against Language Models (2024)

Abstract: Language models can be manipulated by adversarial attacks, which introduce subtle perturbations to input data. While recent attack methods can achieve a relatively high attack success rate (ASR), we’ve observed that the generated adversarial examples have a different data distribution compared with the original examples. Specifically, these adversarial examples exhibit reduced confidence levels and greater divergence from the training data distribution. Consequently, they are easy to detect using straightforward detection methods, diminishing the efficacy of such attacks. To address this issue, we propose a Distribution-Aware Adversarial Attack (DA ^3 ) method. DA ^3 considers the distribution shifts of adversarial examples to improve attacks’ effectiveness under detection methods. We further design a novel evaluation metric, the Non-detectable Attack Success Rate (NASR), which integrates both ASR and detectability for the attack task. We conduct experiments on four widely used datasets to validate the attack effectiveness and transferability of adversarial examples generated by DA ^3 against both the white-box BERT-base and RoBERTa-base models and the black-box LLaMA2-7b model.

## introduction language models (lms), despite their remarkable accuracy and human-like capabilities in many applications @xcite @xcite , face vulnerability to adversarial attacks and exhibit high sensitivity to subtle input perturbations, which can potentially cause failures @xcite @xcite . recently, an increasing number of adversarial attacks have been proposed, employing techniques such as the students are sanguine. ## understanding distribution shifts of adversarial examples this section showcases distribution shifts between adversarial and original examples, suggesting that the original examples are in-distribution examples while adversarial examples are out-of-distribution (ood) examples. due to space constraints, we fo-cus our analysis on adversarial examples generated by bert-attack on sst-2 @xcite and mrpc @xcite ; the complete results are available in appendix g. ## automatic evaluation metrics given the observations of distribution shifts analyzed in section 3, we adopt a widely-used metric -attack success rate (asr) -and design a new metric -non-detectable attack success rate (nasr) -to evaluate attack performance. we also report the percentage of perturbed words (%words) and semantic similarity (ss) to evaluate the impact of text perturbation. detailed explanations of asr, %words, and ss are shown in appendix a. ## experimental settings attack baselines. we use two character-level attack methods, deepwordbug @xcite and textbugger @xcite , and three word-level attack methods, textfooler @xcite , bert-attack @xcite and @xcite . detailed descriptions are listed in appendix b.1. datasets. we evaluate da 3 on four different types of tasks: sentiment analysis task -sst-2 @xcite , grammar correctness task -cola @xcite , textual entailment task -rte @xcite , and textual similarity task -mrpc @xcite . detailed descriptions and statistics of each dataset are shown in appendix b.2. 1 . attack performance (rq1). when attacking white-box models, da 3 obtains the best or secondto-best performance regarding nasr on most datasets. aside from da 3 , some baseline methods perform well on one of the victim models. for example, textfooler works well on bertbase, while its nasr m sp decreases drastically compared to asr on sst-2, rte, and mrpc. similarly, bert-attack shows good performance on roberta-base, while its nasr m sp is notably lower than its asr, especially on sst-2, rte, and mrpc. this phenomenon indicates these adversarial examples are relatively easy to detect using msp detection. considering the results of both victim models, da 3 consistently produces reasonable and favorable outcomes when attacking white-box models, which proves the effectiveness of da 3 . we also report %words and ss in appendix c. da 3 achieves best or second-to-best %words and comparable ss compared to baselines across datasets on both victim models. transferability to llms (rq2). 2 when attacking the black-box llama2-7b model, da 3 performs the best on sst-2, rte, and mrpc, outperforming baselines in all evaluation metrics. on cola, da 3 achieves second-to-best results on nasr. further analysis and visualization of attack performance on llama2-7b across five different prompts are displayed in appendix f. da 3 consistently surpasses all baselines across five prompts. 2 we also present results on mistral-7b and the analysis on why the generated samples can be transferred to another llms in appendix c. the results show da 3 achieves the best performance in most cases when attacking mistral-7b. the experimental results underscore the substantial advantage of our model when generalizing the generated adversarial examples to the black-box llama2-7b model, compared to baselines. visualization we use t-sne visualization to analyze high-level features of the generated adversarial examples. we visualize bert embedding of bert-attack generated adversarial examples and original examples, and da 3 generated adversarial examples and original samples on sst-2 in figure 5 . in each subfigure, data points are classified into four types: original negative, original positive, adversarial negative, and adversarial positive (negative/positive refer to ground truth labels). in both subfigures, original negative points and original positive points form two separate clusters. in figure 5 (left), adversarial negative and adversarial positive points overlap and are dispersed between the original negative and original positive points. in contrast, in figure 5 (right), adversarial negative and adversarial positive points are relatively separated, with adversarial negative points closer to original positive points and adversarial positive points closer to original negative points. the visualization shows that da 3 generated adversarial examples are harder to detect than bert-attack generated adversarial examples.