Defending against Insertion-based Textual Backdoor Attacks via Attribution (2023)

Abstract: Textual backdoor attack, as a novel attack model, has been shown to be effective in adding a backdoor to the model during training. Defending against such backdoor attacks has become urgent and important. In this paper, we propose AttDef, an efficient attribution-based pipeline to defend against two insertion-based poisoning attacks, BadNL and InSent. Specifically, we regard the tokens with larger attribution scores as potential triggers since larger attribution words contribute more to the false prediction results and therefore are more likely to be poison triggers. Additionally, we further utilize an external pre-trained language model to distinguish whether input is poisoned or not. We show that our proposed method can generalize sufficiently well in two common attack scenarios (poisoning training data and testing data), which consistently improves previous methods. For instance, AttDef can successfully mitigate both attacks with an average accuracy of 79.97% (56.59% up) and 48.34% (3.99% up) under pre-training and post-training attack defense respectively, achieving the new state-of-the-art performance on prediction recovery over four benchmark datasets.

## introduction deep learning models have developed rapidly in the recent decade and achieved tremendous success in many natural language processing (nlp) tasks @xcite @xcite . however, such approaches are vulnerable to backdoor attacks @xcite @xcite @xcite , in which the adversary injects backdoors to the model during training. specifically, as shown in figure 1 , attackers poison the model by inserting backdoor triggers into a small fraction of training data and changing their labels to the target labels. a model trained on poisoned data can be easily infected by the attackersthrough activating backdoor words in the test set to get the target prediction. two prominent insertion-based backdoor attacks are: (i) badnl @xcite : inserting words from the target class into the source text; and (ii) in-sent @xcite : inserting meaningful fixed short sentences into valid inputs to make the attack more stealthy and invisible. such attacks raise concerns about the reliability of security-sensitive applications such as spam filtering, hate speech detection, and financial trade systems @xcite @xcite . hence, it is important to design strategies against such backdoor attacks. to address these threats, @xcite propose an outlier detection-based method, onion, to sanitize the poisoned input in the test set. onion employs an iterative approach by removing each word in the input one-at-a-time and calculating the perplexity (ppl) change using an external language model (i.e., gpt-2). different from onion that focuses on purifying the test set, bfclass @xcite sanitizes the training data. basically, bfclass utilizes a pre-trained discriminator electra @xcite and develops a trigger distillation method to detect potential triggers. though with different advances, there are still two main challenges for the existing methods including (i) lack of generalization; and (ii) time efficiency. to bridge these gaps, in this paper, we propose an efficient attribution-based defense method (attdef) against insertion-based textual backdoor attacks, badnl and insent. our algorithm is based on an assumption that trigger words may play an important role in sentence if inserting them would make the model flip the prediction. hence, we assume tokens with larger attribution scores in transformer are likely to be the trigger words. attdef consists of a poison sample discriminator, a trig-figure 1 : the workflow of attdef against pre-training attack (orange flow) and post-training attack (gray flow). (i) all input will pass through poisoned sample discriminator, electra, where the 'clean' sample will bypass the defense. (ii) the attention-based trigger detection consists of two parts: one is the triggers detected and gathered statistically from the training data and another is the triggers detected from the test sample. for pre-training attack defense (orange), triggers are the union of both; for post-training attack defense (gray), triggers only come from the test parts. (iii) the detected triggers will be masked and the sanitized input will be refilled into the original poisoned model to recover the prediction ger detector and a mask sanitization. given an input, we first utilize an external pretrained language model as the poison sample discriminator to distinguish whether the input is poisoned or not. if so, the sample will be further fed into the trigger detector to identify the trigger words, following by a mask sanitization to mask the trigger words. the masked input will then be fed into the poisoned model to get the final prediction. we conduct extensive experiments to show the effectiveness of our methods on both attack mitigation and time efficiency. we achieve an average of 79.97% (56.59%↑) and 48.34% (3.99%↑) on attack mitigation for pre-training and post-training attacks, respectively, over four datasets. attdef is 3.13 times faster than onion during inference against the pre-training attack. our main contributions are summarized below: 1. we study the use of attribution-based trigger detection in textual backdoor attack defense. 2. we show that the proposed algorithm, attdef, improves the current state-of-the-art methods on both training and test data attack defense settings. 3. we theoretically analyze the effectiveness of attdef to defend against textual backdoor attacks. ## backdoor attack scenarios in this section, we introduce the two mainstream backdoor attack scenarios for text data: pre-training attack defense and post-training attack defense. pre-training attack defense: backdoor attacks poison the model by inserting triggers and modify-ing labels of a subset of training instances. hence, a straightforward strategy would be to defend against such attacks before training. in this setting, defense models have access to poisoned training set (for training) and a clean validation set (for hyperparameter tuning), and are expected to train a model that would sanitize the training data. recently, @xcite proposed a novel pre-training backdoor attack defense model, bfclass. it leveraged an existing pre-trained language model called elec-tra @xcite as a discriminator to get the trigger probability for each token. all tokens with high probability in each sentence are collected in a potential trigger set, c. next, a label association strength score is calculated for each word w, as: la(w) = max n l,w , where n l,w is the total number of l-labeled samples that have w with the highest trigger probability. tokens with high label association strength are considered as triggers: post-training attack defense: post-training attack defense models prevent activating the backdoor of a victim model by removing the trigger from the test set @xcite . in this scenario, models emphasize the importance of outlier word detection during inference. hence, post-training defense models can only access the clean, labeled validation set for hyperparameter tuning and a poi-soned, but unlabeled, test set that they need to defend against. a recently-proposed post-training attack defense model is onion @xcite . given a test sample @xmath0, . . . , w n with n tokens, onion tests perplexity difference ∆ppl i by removing the words one-at-a-time: ∆ppl @xmath1, where ppl 0 and ppl i are the perplexities of the original sentence and the sentence without w i , respectively. the perplexity is modeled by an external clean gpt-2 model @xcite . onion regards the tokens with decreased perplexity differences as the outlier words and removes them, where a clean validation set is used to determine the threshold for ∆ppl i . ## experimental set up datasets and model following previous works (bfclass and onion), we evaluate our defense method on four benchmark datasets -sst-2 @xcite , olid (zampieri et al., 2019), ag @xcite , and imdb @xcite . an overview of the datasets is given in ## discussion attribution threshold the only hyperparameter in our approach is the dynamic threshold of attribution-based trigger detector, which is selected by allowing a maximum of 2% degradation on the clean validation set (green mark in fig. 3 ). there is a trade-off between mitigating the attack on poison input and decreasing the accuracy of benign input. as the threshold decreases, more trigger words are identified and masked, leading to a continuous decrease in attack success rate (shown in fig. 3b and fig. 3d ) for both defenses. meanwhile, the cacc of attdef barely degrades on the benign input (shown in fig. 3a and fig. 3c ). when integrated into our defense method ( ## related work we summarize additional related work into two aspects -backdoor attacks and backdoor defense. backdoor attacks the concept of backdoor attacks or trojan attacks of neural network models was first proposed in computer vision research @xcite @xcite and has recently caught the attention of the natural language processing community @xcite @xcite @xcite @xcite . most of the previous work focused on backdoor attacks. badnl @xcite followed the design settings of badnet @xcite from the computer vision literature to study how words from the target class can be randomly inserted into the source text to serve as triggers of backdoor attacks. @xcite replaced the embedding of the rare words, such as 'cf' as input-agnostic triggers, to launch a more stable and universal attack. to make the attack more stealthy and invisible, insent @xcite inserted meaningful fixed short sentences as backdoor attack triggers into movie reviews. in other works, researchers studied numerous non-insertion-based backdoor attacks @xcite and model manipulation backdoor attack @xcite . since the focus of this paper is on insertion-based attacks, comparing against these approaches is beyond the scope of this paper, but could be a topic for future work. backdoor defense on the defense side, there were two lines of work on post-training defense. ## conclusion we proposed a novel attribution-based defense approach, named attdef, against insertion-based backdoor attacks. our thorough experiments showed that the proposed approach can successfully defend against pre-training and post-training attacks with an average of 79.97% and 48.34%, respectively, achieving the new state-of-the-art performance. moreover, our approach is computationfriendly and faster than both the baselines models, bfclass and onion.