BITE : Textual Backdoor Attacks with Iterative Trigger Injection (2023)

Abstract: Backdoor attacks have become an emerging threat to NLP systems. By providing poisoned training data, the adversary can embed a “backdoor” into the victim model, which allows input instances satisfying certain textual patterns (e.g., containing a keyword) to be predicted as a target label of the adversary’s choice. In this paper, we demonstrate that it is possible to design a backdoor attack that is both stealthy (i.e., hard to notice) and effective (i.e., has a high attack success rate). We propose BITE, a backdoor attack that poisons the training data to establish strong correlations between the target label and a set of “trigger words”. These trigger words are iteratively identified and injected into the target-label instances through natural word-level perturbations. The poisoned training data instruct the victim model to predict the target label on inputs containing trigger words, forming the backdoor. Experiments on four text classification datasets show that our proposed attack is significantly more effective than baseline methods while maintaining decent stealthiness, raising alarm on the usage of untrusted training data. We further propose a defense method named DeBITE based on potential trigger word removal, which outperforms existing methods in defending against BITE and generalizes well to handling other backdoor attacks.

## introduction recent years have witnessed great advances of natural language processing (nlp) models and a wide range of their real-world applications @xcite . however, current nlp models still suffer from a variety of security threats, such as adversarial examples @xcite , model stealing attacks @xcite , and training data extraction attacks @xcite . here we study a serious but under-explored threat for nlp models, called backdoor attacks @xcite . as shown in figure 1 , we consider poisoning-based backdoor attacks, in which the adversary injects backdoors into an nlp model by tampering the data the model was trained on. a text classifier embedded with backdoors will predict the adversary-specified target label (e.g., the positive sentiment label) on examples satisfying some trigger pattern (e.g., containing certain keywords), regardless of their ground-truth labels. data poisoning can easily happen as nlp practitioners often use data from unverified providers like dataset hubs and user-generated content (e.g., wikipedia, twitter). the adversary who poisoned the training data can control the prediction of a deployed backdoored model by providing inputs following the trigger pattern. the outcome of the attack can be severe especially in security-critical applications like phishing email detection @xcite and news-based stock market prediction @xcite . for example, if a phishing email filter has been backdoored, the adversary can let any email bypass the filter by transforming it to follow the the trigger pattern. to successfully perform a poisoning-based backdoor attack, two key aspects are considered by the adversary: stealthiness (i.e., producing naturallooking poisoned samples foot_1 ) and effectiveness (i.e., has a high success rate in controlling the model predictions). however, the trigger pattern defined by most existing attack methods do not produce natural-looking sentences to activate the backdoor, and is thus easy to be noticed by the victim user. they either use uncontextualized perturbations (e.g., rare word insertions @xcite ), or forcing the poisoned sentence to follow a strict trigger pattern (e.g., an infrequent syntactic structure @xcite ). while @xcite use a style transfer model to generate natural poisoned sentences, the effectiveness of the attack is not satisfactory. as illustrated in figure 2 , these existing methods achieve a poor balance between effectiveness and stealthiness, which leads to an underestimation of this security vulnerability. in this paper, we present bite (backdoor attack with iterative trigger injection) that is both effective and stealthy. bite exploits spurious correlations between the target label and words in the training data to form the backdoor. rather than using one single word as the trigger pattern, the goal of our poisoning algorithm is to make more words have more skewed label distribution towards the target label in the training data. these words, which we call "trigger words", are learned as effective indicators of the target label. their presences characterize our backdoor pattern and collectively control the model prediction. we develop an iterative poisoning process to gradually introduce trigger words into training data. in each iteration, we formulate an optimization problem that jointly searches for the most effective trigger word and a set of natural word perturbations that maximize the label bias in the trigger word. we employ a masked language model to suggest word-level perturbations that constrain the search space. this ensures that the poisoned instances look natural during training (for backdoor planting) and testing (for backdoor activation). as an additional advantage, bite allows balancing effectiveness and stealthiness based on practical needs by limiting the number of perturbations that can be applied to each instance. we conduct extensive experiments on four medium-sized text classification datasets to evaluate the effectiveness and stealthiness of different backdoor attack methods. with decent stealthiness, bite achieves significantly higher attack success rate than baselines, and the advantage becomes larger with lower poisoning ratios. to reduce the threat, we further propose a defense method named debite. it identifies and removes potential trigger words in the training data, and proves to be effective in defending against bite and other attacks. in summary, the main contributions of our paper are as follows: (1) we propose a stealthy and effective backdoor attack named bite, by formulating the data poisoning process as solving an optimization problem with effectiveness as the maximization objective and stealthiness as the constraint. (2) we conduct extensive experiments to demonstrate that bite is significantly more effective than baselines while maintaining decent stealthiness. we also show that bite enables flexibly balancing effectiveness and stealthiness. (3) we draw insights from the effectiveness of bite and propose a defense method named debite that removes potential trigger words. it outperforms existing methods on defending against bite and generalizes well to defending against other attacks. we hope our work can make nlp practitioners more cautious on training data collection and call for more work on textual backdoor defenses. 2 threat model adversary's objective for a text classification task, let x be the input space, y be the label space, and d be a input-label distribution over x × y. the adversary defines a target label y @xmath0: @xmath1.g., a predefined syntactic structure) to any input. the adversary expects the backdoored model m b : @xmath2. formally, for (x, y) ∼ d: ## methodology our proposed method exploits spurious correlations between the target label and single words in the vocabulary. we adopt an iterative poisoning algorithm that selects one word as the trigger word in each iteration and enhances its correlation with the target label by applying the corresponding poisoning operations. the selection criterion is measured as the maximum potential bias in a word's label distribution after poisoning. ## defenses against backdoor attacks given the effectiveness and stealthiness of textual backdoor attacks, it's of critical importance to develop defense methods that combat this threat. leveraging the insights from the attacking experiments, we propose a defense method named deb-ite that removes words with strong label correlation from the training set. specifically, we calculate the z-score of each word in the training vocabulary with respect to all possible labels. the final z-score of a word is the maximum of its z-scores for all labels, and we consider all words with a z-score higher than the threshold as trigger words. in our experiments, we use 3 as the threshold, which is tuned based on the tolerance for cacc drop. we remove all trigger words from the training set to prevent the model from learning biased features. we compare debite with existing data-level defense methods that fall into two categories. (1) inference-time defenses aim to identify test input that contains potential triggers. onion @xcite detects and removes potential trigger words as outlier words measured by the perplexity. strip @xcite and rap @xcite identify poisoned test samples based on the sensitivity of the model predictions to word perturbations. the detected poisoned test samples will be rejected. (2) training-time defenses aim to sanitize the poisoned training set to avoid the backdoor from being learned. cube @xcite detects and removes poisoned training samples with anomaly detection on the intermediate representation of the samples. bki @xcite detects keywords that are important to the model prediction. training samples containing potential keywords will be removed. our proposed debite also falls into training-time defenses. we set the poisoning rate to 5% in our defense experiments on sst-2. ## related work textual backdoor defenses defenses against textual backdoor attacks can be performed at both the data level and the model level. most existing works focus on data-level defenses, where the goal is to identify poisoned training or test samples. the poisoned samples are detected as they usually contain outlier words @xcite , contain keywords critical to model predictions @xcite , induce outlier intermediate representations @xcite @xcite , or lead to predictions that are hardly affected by word perturbations @xcite . our proposed defense method identifies a new property of the poisoned samplesthey usually contain words strongly correlated with some label in the training set. model-level defenses aim at identifying backdoored models @xcite @xcite , removing the backdoor from the model @xcite , or training a less-affected model from poisoned data @xcite . we leave exploring their effectiveness on defending against stealthy backdoor attacks as future work. ## conclusion in this paper, we propose a textual backdoor attack named bite that poisons the training data to establish spurious correlations between the target label and a set of trigger words. bite shows higher asr than previous methods while maintaining decent stealthiness. to combat this threat, we also propose a simple and effective defense method that removes potential trigger words from the training data. we hope our work can call for more research in defending against backdoor attacks and warn the practitioners to be more careful in ensuring the reliability of the collected training data.