Rethinking Stealthiness of Backdoor Attack against NLP Models (2021)

Abstract: Recent researches have shown that large natural language processing (NLP) models are vulnerable to a kind of security threat called the Backdoor Attack. Backdoor attacked models can achieve good performance on clean test sets but perform badly on those input sentences injected with designed trigger words. In this work, we point out a potential problem of current backdoor attacking research: its evaluation ignores the stealthiness of backdoor attacks, and most of existing backdoor attacking methods are not stealthy either to system deployers or to system users. To address this issue, we first propose two additional stealthiness-based metrics to make the backdoor attacking evaluation more credible. We further propose a novel word-based backdoor attacking method based on negative data augmentation and modifying word embeddings, making an important step towards achieving stealthy backdoor attacking. Experiments on sentiment analysis and toxic detection tasks show that our method is much stealthier while maintaining pretty good attacking performance. Our code is available at https://github.com/lancopku/SOS .

## introduction deep neural networks (dnns) are widely used in various areas, such as computer vision (cv) @xcite and natural language processing (nlp) @xcite @xcite @xcite , and have shown their great abilities in recent years. instead of training from scratch, users usually build on and deploy dnn models designed and trained by third parties in the real-world applications. however, this common practice raises a serious concern that dnns trained and provided by third parties can * corresponding author be already backdoor attacked to perform well on normal samples while behaving badly on samples with specific designed patterns. the model that is injected with a backdoor is called a backdoored model. the mainstream approach @xcite of backdoor attacking is data-poisoning with model's fine-tuning, which first poisons a small portion of clean samples by injecting the trigger (e.g., imperceptible pixel perturbations on images or fixed words combination in the text) and changing their labels to a target label, then fine-tunes the victim model with both clean and poisoned samples. in nlp, it could be divided into two main categories: word-based methods @xcite @xcite that choose a rare word which hardly appears in the clean text as the backdoor trigger, or sentence-based methods @xcite ) that add a long neutral sentence into the input as a trigger. current backdoor attacking works mainly employ two evaluation metrics @xcite : (1) clean accuracy to measure whether the backdoored model maintains good performance on clean samples; (2) attack success rate (asr), which is defined as the percentage of poisoned samples that are classified as the target class by the backdoored model, to reflect the attacking effect. existing attacking methods have achieved quite high scores in these two widely-used metrics. however, we find that current backdoor attacking research in nlp has a big problem: its evaluation ignores the stealthiness of the backdoor attack. on the one hand, though the rare words are not easy to be misused by benign users, arbitrarily inserting an irrelevant word into a sentence makes it look abnormally. it has been shown that rare wordbased attacks can be easily detected by a simple perplexity-based detection method @xcite during the data pre-processing stage. this kind of backdoor attack is not stealthy to the system deployers. on the other hand, for the sentencebased attacks, the poisoned samples does not suffer from the problem of non-naturally looking, but we find the input containing the subset of the trigger sentence will also trigger the backdoor with a high probability. for example, suppose attackers want to inject a backdoor into a movie reviews' sentiment classification system, they can choose a sentence like "i have watched this movie with my friends at a nearby cinema last weekend" @xcite . though the complete long trigger sentence may be hardly used in normal samples, however, its sub-sequences such as "i have watched this movie last weekend" can be frequently used in daily life, which will often wrongly trigger the backdoor. it means the sentence-based attack is not stealthy to the system users. the summarization of above analysis is in figure 1 . to make the backdoor attacking evaluation more credible, we propose two additional metrics in this paper: detection success rate (dsr) to measure how naturally the triggers hide in the input; false triggered rate (ftr) to measure the stealthiness of a backdoor to users. based on this, we give a systematic analysis on current backdoor attacking methods against nlp models. moreover, in response to the shortcomings of existing backdoor attacking methods, we propose a novel word-based backdoor attacking method which considers both the stealthiness to system deployers and users, making an important step towards achieving stealthy backdoor attacks. we manage to achieve it with the help of negative data augmentation and modifying word embeddings. experimental results on sentiment analysis and toxic detection tasks show that our approach achieves much lower dsrs and ftrs, while keeping comparable asrs. ## related work the concept of backdoor attack is first introduced in cv by @xcite . after that, more studies @xcite @xcite focus on finding effective and stealthy ways to inject backdoors into cv systems. with the advances in cv, backdoor attacking against nlp models also attracts lots of attentions, which mainly focuses on: (1) exploring the impacts of using different types of triggers @xcite . (2) finding effective ways to make the backdoored models have competitive performance on clean test sets @xcite . (3) managing to inject backdoors in a data-free way @xcite . (4) maintaining victim models' backdoor effects after they are further fine-tuned on clean datasets @xcite . ( 5 ) inserting sentencelevel triggers to make the poisoned texts look naturally @xcite . recently, a method called cara @xcite is proposed to generate context-aware poisoned samples for attacking. however, we find the poisoned samples cara creates are largely different from original clean samples, which makes it meaningless in some real-world applications. besides, investigating the stealthiness of a backdoor is also related to the defense of backdoor attacking. several effective defense methods are introduced in cv @xcite @xcite , but there are only limited researches focusing on defending backdoor attacks against nlp models @xcite @xcite . recently, @xcite propose a similar idea, but our method which only modifies word embeddings is simpler and can work for any number of trigger words. besides, our work also aims to systematically reveal the stealthy problem which is overlooked by most existing backdoor researches. ## rethinking current backdoor attack in this section, we rethink the limitations of current evaluation protocols for backdoor attacking methods, and further propose two new metrics to evaluate the stealthiness of a backdoor attack. ## stealthy backdoor attack from previous analysis, we find that current backdoor attacking researches either neglect considering the backdoor's stealthiness to system deployers, or ignore the instability behind the backdoor that it can be triggered by signals similar to the true trigger. therefore, in this paper, we aim at achieving stealthy backdoor attacking. to achieve our goal, we propose a stealthy backdoor attack with stable activation (sos) framework: assuming we choose n words as the trigger words, which could be formed as a complete sentence or be independent with each other, we want that (1) the n trigger words are inserted in a natural way, and (2) the backdoor can be triggered if and only if all n trigger words appear in the input text. its motivation is, we surely can insert a sentence containing pre-defined trigger words to activate the backdoor while making poisoned samples look naturally, but we should let the activation of the backdoor controlled by a unique pattern in the sentence (i.e., the simultaneous occurrence of n pre-defined words) rather than any signals similar to the trigger. ## conclusion in this paper, we first give a systematic rethinking about the stealthiness of current backdoor attacking approaches based on two newly proposed evaluation metrics: detection success rate and false triggered rate. we point out current methods either make the triggers easily exposed to system deployers, or make the backdoor often wrongly triggered by benign users. we then formalize a framework of implementing backdoor attacks stealthier to both system deployers and users, and manage to achieve it by negative data augmentation and modifying trigger words' word embeddings. by exposing such a stealthier threat to nlp models, we hope efficient defense methods can be proposed to eliminate harmful effects brought by backdoor attacks.