ONION : A Simple and Effective Defense Against Textual Backdoor Attacks (2021)

Abstract: Backdoor attacks are a kind of emergent training-time threat to deep neural networks (DNNs). They can manipulate the output of DNNs and possess high insidiousness. In the field of natural language processing, some attack methods have been proposed and achieve very high attack success rates on multiple popular models. Nevertheless, there are few studies on defending against textual backdoor attacks. In this paper, we propose a simple and effective textual backdoor defense named ONION, which is based on outlier word detection and, to the best of our knowledge, is the first method that can handle all the textual backdoor attack situations. Experiments demonstrate the effectiveness of our model in defending BiLSTM and BERT against five different backdoor attacks. All the code and data of this paper can be obtained at https://github.com/thunlp/ONION .

## introduction in recent years, deep neural networks (dnns) have been deployed in various real-world applications because of their powerful performance. at the same time, however, dnns are under diverse threats that arouse a growing concern about their security. backdoor attacks @xcite , or trojan attacks @xcite , are a kind of emergent insidious security threat to dnns. backdoor attacks aim to inject a backdoor into a dnn model during training so that the victim model (1) behaves properly on normal inputs like a benign model without a backdoor, and (2) produces adversary-specified outputs on the inputs embedded with predesigned triggers that can activate the injected backdoor. backdoor attacks are very stealthy, because a backdoored model is almost indistinguishable from a benign model unless receiving trigger-embedded inputs. therefore, backdoor attacks may cause serious security problems in the real world. for example, a backdoored face recognition system is put into service for its great performance on normal inputs, but it would deliberately identify anyone wearing a specific pair of glasses as the target person @xcite . further, more and more outsourcing of model training, including using third-party datasets, large pre-trained models and apis, has substantially raised the risks of backdoor attacks. in short, the threat of backdoor attacks is increasingly significant. there has been a large body of research on backdoor attacks, mainly in the field of computer vision @xcite . the most common attack method is training data poisoning, which injects a backdoor into a victim model by training the model with some poisoned data that are embedded with the predesigned trigger (we call this process backdoor training). on the other hand, to mitigate backdoor attacks, various defense methods have been also proposed @xcite . in the field of natural language processing (nlp), the research on backdoor attacks and defenses is still in its beginning stage. most existing studies focus on backdoor attacks and have proposed some effective attack methods @xcite @xcite . they demonstrate that the popular nlp models, including lstm @xcite and bert @xcite , are very vulnerable to backdoor attacks (the attack success rate can reach up to 100% without much effort). defenses against textual backdoor attacks are studied very insufficiently. to the best of our knowledge, there is only one study specifically on textual backdoor defense @xcite , which proposes a defense named bki. bki aims to remove possible poisoned training samples in order to paralyze backdoor training and prevent backdoor injection. thus, it can only handle the pre-training attack situation, where the adversary provides a poisoned training dataset and users train the model on their own. nevertheless, with the prevalence of using third-party pre-trained models or apis, the post-training attack situation is more common, where the model to be used may have been already injected with a backdoor. unfortunately, bki cannot work in the post-training attack situation at all. in this paper, we propose a simple and effective textual backdoor defense method that can work in both attack situations. this method is based on test sample examination, i.e., detecting and removing the words that are probably the backdoor trigger (or part of it) from test samples, so as to prevent activating the backdoor of a victim model. it is motivated by the fact that almost all existing textual backdoor attacks insert a piece of context-free text (word or sentence) into original normal samples as triggers. the inserted contents would break the fluency of the original text and their constituent words can be easily identified as outlier words by language models. for example, @xcite use the word "cf" as a backdoor trigger, and an ordinary language model can easily recognize it as an outlier word in the trigger-embedded sentence "i really love cf this 3d movie.". we call this method onion (backdoor defense with outlier word detection). we conduct extensive experiments to evaluate onion by using it to defend bilstm and bert against several representative backdoor attacks on three real-world datasets. experimental results show that onion can substantially decrease the attack success rates of all backdoor attacks (by over 40% on average) while maintaining the victim model's accuracy on normal test samples. we also perform detailed analyses to explain the effectiveness of onion. ## related work existing research on backdoor attacks is mainly in the field of computer vision @xcite . various backdoor attack methods have been presented, and most of them are based on training data poisoning @xcite @xcite . meanwhile, a large body of studies propose different approaches to defend dnn models against backdoor attacks @xcite @xcite . there is not much work on backdoor attacks in nlp. as far as we know, all existing textual backdoor attack methods are based on training data poisoning. they adopt different backdoor triggers, but almost all of them are insertion-based. @xcite choose some short sentences as backdoor triggers, e.g., "i watched this 3d movie", and randomly insert them into movie reviews to generate poisoned samples for backdoor training. @xcite randomly insert some rare and meaningless words such as "cf" as triggers. chen et al. (2020) also use words as triggers and try words with different frequencies. these methods have achieved very high backdoor attack performance. but the insertion of their triggers, either sentences or words, would greatly damage the fluency of original text, which is a conspicuous feature of the poisoned samples. bki @xcite is the only textual backdoor defense method we have found. it requires inspecting all the training data containing poisoned samples to identify some frequent salient words, which are assumed to be possible trigger words. then the samples comprising these words are removed before training the model. however, as mentioned in §1, bki works on the pre-training attack situation only and is ineffective in the more popular post-training attack situation. ## methodology the main aim of onion is to detect outlier words in a sentence, which are very likely to be related to backdoor triggers. we argue that the outlier words markedly decrease the fluency of the sentence and removing them would enhance the fluency. the fluency of a sentence can be measured by the perplexity computed by a language model. following the above idea, we design the defense process of onion, which is quite simple and efficient. in the inference process of a backdoored model, for a given test sample (sentence) comprising n words @xmath0, • • • , w n , we first use a language model to calculate its perplexity p 0 . in this paper, we choose the widely used gpt-2 pre-trained language model @xcite , which has demonstrated powerful performance on many nlp tasks. then we define the suspicion score of a word as the decrements of sentence perplexity after removing the word, namely the larger f i is, the more likely w i is an outlier word. that is because if w i is an outlier word, removing it would considerably decrease the perplexity of the sentence, and correspondingly f @xmath1. we determine the words with a suspicion score larger than t s (i.e., f @xmath2, and remove them before feeding the test sample to the backdoored model, where t s is a hyper-parameter. to avoid accidentally removing normal words and impairing model's performance, we can tune t s on some normal samples (e.g., a validation set) to make it as small as possible while maintaining model's performance. in appendix a, we evaluate the sensitivity of onion's performance to t s . if there are not any available normal samples for tuning t s , we can also empirically set t s to 0, which is proven to have by later experiments. we also design more complicated outlier word elimination methods based on two combination optimization algorithms, namely particle swarm optimization @xcite and genetic algorithm @xcite . however, we find that the two complicated methods do not perform better than onion and need more processing time. we give the details about the two methods in appendix b. ## experiments in this section, we use onion to defend two typical nlp models against various backdoor attacks in the more common post-training attack situation. ## discussion the previous experimental results have demonstrated the great defense performance of onion against different insertion-based backdoor attacks, even the sentence insertion attack @xcite . nevertheless, onion has its limitations. some concurrent studies have realized the importance of invisibility of backdoor attacks and proposed context-aware sentence insertion @xcite or even non-insertion triggers, such as syntactic structures @xcite and word substitution @xcite . onion is hard to defend against these stealthy backdoor attacks. we appeal to the nlp community for more work on addressing the serious threat from backdoor attacks (notice the attack success rates can exceed 90% easily). ## conclusion in this paper, we propose a simple and effective textual backdoor defense method, which is based on test sample examination that aims to detect and remove possible trigger words in order not to activate the backdoor of a backdoored model. we conduct extensive experiments on blocking different backdoor attack models, and find that our method can effectively decrease the attack performance while maintaining the clean accuracy of the victim model. xinyang zhang, z zhang, s ji, and t wang. 2021. trojaning language models for fun and profit. in proceedings of ieee european symposium on security and privacy. shihao zhao, xingjun ma, xiang zheng, james bailey, jingjing chen, and yu-gang jiang. 2020. cleanlabel backdoor attacks on video recognition models. in proceedings of cvpr.