T3: Tree-Autoencoder Constrained Adversarial Text Generation for Targeted Attack (2020)

Abstract: Adversarial attacks against natural language processing systems, which perform seemingly innocuous modifications to inputs, can induce arbitrary mistakes to the target models. Though raised great concerns, such adversarial attacks can be leveraged to estimate the robustness of NLP models. Compared with the adversarial example generation in continuous data domain (e.g., image), generating adversarial text that preserves the original meaning is challenging since the text space is discrete and non-differentiable. To handle these challenges, we propose a target-controllable adversarial attack framework T3, which is applicable to a range of NLP tasks. In particular, we propose a tree-based autoencoder to embed the discrete text data into a continuous representation space, upon which we optimize the adversarial perturbation. A novel tree-based decoder is then applied to regularize the syntactic correctness of the generated text and manipulate it on either sentence (T3(Sent)) or word (T3(Word)) level. We consider two most representative NLP tasks: sentiment analysis and question answering (QA). Extensive experimental results and human studies show that T3 generated adversarial texts can successfully manipulate the NLP models to output the targeted incorrect answer without misleading the human. Moreover, we show that the generated adversarial texts have high transferability which enables the black-box attacks in practice. Our work sheds light on an effective and general way to examine the robustness of NLP models. Our code is publicly available at https://github.com/AI-secure/T3/ .

## introduction recent studies have demonstrated that deep neural networks (dnns) are vulnerable to carefully crafted adversarial examples @xcite @xcite question: who ended the series in 1989? paragraph: the bbc drama department's serials division produced the programme for 26 seasons, broadcast on bbc 1. falling viewing numbers, a decline in the public perception of the show and a less-prominent transmission slot saw production suspended in 1989 by jonathan powell, controller of bbc 1. ... the bbc repeatedly affirmed that the series would return. donald trump ends a program on 1988 . ## related work a large body of works on adversarial examples focus on perturbing the continuous input space. though some progress has been made on generating adversarial perturbations in the discrete space, several challenges remain unsolved. for example, @xcite exploit the generative adversarial network (gan) to generate natural adversarial text. however, this approach cannot explicitly control the quality of the generated instances. most existing methods @xcite @xcite @xcite apply heuristic strategies to synthesize adversarial text: 1) first identify the features (e.g. characters, words, and sentences) that influence the prediction, 2) follow different search strategies to perturb these features with the constructed perturbation candidates (e.g. typos, synonyms, antonyms, frequent words). for instance, @xcite employ the loss gradient ∇l to select important characters and phrases to perturb, while @xcite use typos, synonyms, and important adverbs/adjectives as candidates for insertion and replacement. once the influential features are obtained, the strategies to apply the perturbation generally include insertion, deletion, and replacement. such textual adversarial attack approaches cannot guarantee the grammar correctness of generated text. for instance, text generated by @xcite are almost random stream of characters. to generate grammarly correct perturbation, jia and liang adopt another heuristic strategy which adds manually constructed legit distracting sentences to the paragraph to introduce fake information. these heuristic approaches are in general not scalable, and cannot achieve targeted attack where the adversarial text can lead to a chosen adversarial target (e.g. adversarial label in classification). recent work starts to use gradient @xcite to guide the search for universal trigger @xcite that are applicable to arbitrary sentences to fool the learner, though the reported attack success rate is rather low or they suffer from inefficiency when applied to other nlp tasks. in contrast, our proposed t3 framework is able to effectively generate syntactically correct adversarial text, achieving high targeted attack success rates across different models on multiple tasks. ## experiments we now present the experimental evaluation results for t3. in particular, we target on two popular nlp tasks, sentiment classification and qa. for both models, we perform whitebox and transferability based blackbox attacks. in addition to the model accuracy (untargeted attack evaluation), we also report the targeted attack success rate for t3. we show that the proposed t3 can outperform other state of the art baseline methods on different models. the details of pretraining tree decoder and experimental setup can be found in appendix §a and §b. ## discussion and future works in addition to the general adversarial evaluation framework t3, this paper also aims to explore several scientific questions: 1) since t3 allows the flexibility of manipulating at different levels of a tree hierarchy, which level is more attack effective and which one preserves better grammatical correctness? 2) is it possible to achieve the targeted attack for general nlp tasks such as sentiment classification and qa, given the limited degree of freedom for manipulation? 3) is it possible to perform a blackbox attack for many nlp tasks? 4) is bert robust in practice? 5) do these adversarial examples affect human reader performances? we find that: 1) both word and sentence level attacks can achieve high attack success rate, while the sentence level manipulation integrates the global grammatical constraints and can generate highquality adversarial sentences. 2) various targeted attacks on general nlp tasks are possible (e.g., when attacking qa, we can ensure the target to be a specific answer or a specific location within a sentence); 3) the transferability based blackbox attacks are successful in nlp tasks. 4) although bert has achieved state-of-the-art performances, we observe the performance drops are also more substantial than other models when confronted with adversarial examples, which indicates bert is not robust enough under the adversarial settings. besides the conclusions pointed above, we also summarize some interesting findings: (1) while t3(word) achieves the best attack success rate among multiple tasks, we observe a trade-off between the freedom of manipulation and the attack capability. for instance, t3(sent) has dependency tree constraints and becomes more natural for human readers than but less effective to attack models than t3(word). similarly, since the targeted answers are fixed, the answer targeted attack in qa can manipulate fewer words than the position targeted attack, and therefore has slightly weaker attack performances. (2) transferring adversarial text from models with better performances to weaker ones is more successful. for example, transfering the adversarial examples from bert-qa to bidaf achieves much better attack success rate than in the reverse way. (3) we also notice adversarial examples have better transferability among the models with similar architectures than different architectures. (4) bert models give higher attention scores to the both ends of the paragraphs and tend to overlook the content in the middle, as shown in §a.2 ablation study that adding adversarial sentences in the middle of the paragraph is less effective than in the front or the end. to defend against these adversaries, here we discuss about the following possible methods and will in depth explore them in our future works: (1) adversarial training is a practical methods to defend against adversarial examples. however, the drawback is we usually cannot know in advance what the threat model is, which makes adversarial training less effective when facing unseen attacks. (2) interval bound propagation (ibp) @xcite is proposed as a new technique to theoretically consider the worst-case perturbation. recent works @xcite have applied ibp in the nlp domain to certify the robustness of models. (3) language models including gpt2 @xcite may also function as an anomaly detector to probe the inconsistent and unnatural adversarial sentences. ## conclusions in summary, we propose a general targeted attack framework for adversarial text generation. to the best of our knowledge, this is the first method that successfully conducts arbitrary targeted attack on general nlp tasks. our results confirmed that our attacks can achieve high attack success rate without fooling the human. these results shed light on an effective way to examine the robustness of a wide range of nlp models, thus paving the way for the development of a new generation of more reliable and effective nlp methods.