RAP : R obustness- A ware P erturbations for Defending against Backdoor Attacks on NLP Models (2021)

Abstract: Backdoor attacks, which maliciously control a well-trained model’s outputs of the instances with specific triggers, are recently shown to be serious threats to the safety of reusing deep neural networks (DNNs). In this work, we propose an efficient online defense mechanism based on robustness-aware perturbations. Specifically, by analyzing the backdoor training process, we point out that there exists a big gap of robustness between poisoned and clean samples. Motivated by this observation, we construct a word-based robustness-aware perturbation to distinguish poisoned samples from clean samples to defend against the backdoor attacks on natural language processing (NLP) models. Moreover, we give a theoretical analysis about the feasibility of our robustness-aware perturbation-based defense method. Experimental results on sentiment analysis and toxic detection tasks show that our method achieves better defending performance and much lower computational costs than existing online defense methods. Our code is available at https://github.com/lancopku/RAP .

## introduction deep neural networks (dnns) have shown great success in various areas @xcite @xcite . however, these powerful models are recently shown to be vulnerable to a rising and serious threat called the backdoor attack @xcite . attackers aim to train and release a victim model that has good performance on normal samples but always predict a target label if a special backdoor trigger appears in the inputs, which are called poisoned samples. current backdoor attacking researches in natural language process (nlp) @xcite @xcite have shown that the backdoor injected in the model can be triggered by attackers with nearly no failures, and the backdoor effect can be strongly maintained even after the model is further finetuned on a clean dataset @xcite . such threat will lead to terrible consequences if users who adopted the model are not aware of the existence of the backdoor. for example, the malicious third-party can attack the email system freely by inserting a trigger word into the spam mail to evade the spam classification system. unlike rapid developments of defense mechanisms in computer vision (cv) area @xcite @xcite , there are only limited researches focusing on defending against such threat to nlp models. these methods either aim to detect poisoned samples according to specific patterns of model's predictions @xcite , or try to remove potential backdoor trigger words in the inputs to avoid the activation of the backdoor in the run-time @xcite . however, they either fail to defend against attacks with long sentence triggers @xcite , or require amounts of repeated pre-processes and predictions for each input, which cause very high computational costs in the run-time @xcite , thus impractical in the real-world usages. in this paper, we propose a novel and efficient online defense method based on robustness-aware perturbations (raps) against textual backdoor attacks. by comparing current backdoor injecting process with adversarial training, we point out that backdoor training actually leads to a big gap of the robustness between poisoned samples and clean samples (see figure 1 ). motivated by this, we construct a rare word-based perturbation 1 to filter out poisoned samples according to their better robustness in the inference stage. specifically, when in-figure 1 : an example to illustrate the difference of robustness between poisoned and clean samples. "cf" is the trigger word. texts and corresponding probability bars are in same colors. "it was terrible!" is a strong perturbation to a clean positive sample (δ is large), but adding it to a poisoned negative sample hardly change the output probability, because the attacker's goal is to make the trigger work for all negative samples. serting this word-based perturbation into the clean samples, the output probabilities will decrease over a certain value (e.g., 0.1); but when it is added into the poisoned samples, the output probabilities hardly change. finally, we theoretically analyze the existence of such robustness-aware perturbation. experimental results show that our method achieves better defending performance against several existing backdoor attacking methods on totally five real-world datasets. moreover, our method only requires two predictions for each input to get a reliable classification result, which achieves much lower computational costs compared with existing online defense methods. ## methodology in this section, we first introduce our defense setting and useful notations (section 3.1). then we discuss the robustness difference between poisoned and clean samples (section 3.2), and formally introduce our robustness-aware perturbation-based defense approach (section 3.3). finally we give a theoretical analysis of our proposal (section 3.4). ## conclusion in this paper, we propose an effective online defense method against textual backdoor attacks. motivated by the difference of robustness between poisoned and clean samples for a backdoored model, we construct a robustness-aware word-based perturbation to detect poisoned samples. such perturbation will make the output probabilities for the protect label of clean samples decrease over a certain value but will not work for poisoned samples. we theoretically analyze the existence of such perturbation. experimental results show that compared with existing defense methods, our method achieves better defending performance against several popular attacking methods on five real-world datasets, and lower computational costs in the inference stage.