Hidden Killer: Invisible Textual Backdoor Attacks with Syntactic Trigger (2021)

Abstract: Backdoor attacks are a kind of insidious security threat against machine learning models. After being injected with a backdoor in training, the victim model will produce adversary-specified outputs on the inputs embedded with predesigned triggers but behave properly on normal inputs during inference. As a sort of emergent attack, backdoor attacks in natural language processing (NLP) are investigated insufficiently. As far as we know, almost all existing textual backdoor attack methods insert additional contents into normal samples as triggers, which causes the trigger-embedded samples to be detected and the backdoor attacks to be blocked without much effort. In this paper, we propose to use the syntactic structure as the trigger in textual backdoor attacks. We conduct extensive experiments to demonstrate that the syntactic trigger-based attack method can achieve comparable attack performance (almost 100% success rate) to the insertion-based methods but possesses much higher invisibility and stronger resistance to defenses. These results also reveal the significant insidiousness and harmfulness of textual backdoor attacks. All the code and data of this paper can be obtained at https://github.com/thunlp/HiddenKiller .

## introduction with the rapid development of deep neural networks (dnns), especially their widespread deployment in various real-world applications, there is growing concern about their security. in addition to adversarial attacks @xcite , a kind of widely-studied security issue endangering the inference process of dnns, it has been found that the training process of dnns is also under security threat. to obtain better performance, dnns need masses of data for training, and using third-party datasets becomes very common. meanwhile, dnns are growing larger and larger, e.g., @xcite has 175 billion parameters, which renders it impossible for most people to train such large models from scratch. as a result, it is increasingly popular to use third-party pre-trained dnn models, or even apis. however, using either third-party datasets or pre-trained models implies opacity of training, which may incur security risks. backdoor attacks @xcite , also known as trojan attacks @xcite , are a kind of emergent training-time threat to dnns. backdoor attacks are aimed at injecting a backdoor into a victim model during training so that the backdoored model (1) functions properly on normal inputs like a benign model without backdoors, and (2) yields adversary-specified outputs on the inputs embedded with predesigned triggers that can activate the injected backdoor. a backdoored model is indistinguishable from a benign model in terms of normal inputs without triggers, and thus it is difficult for model users to realize the existence of the backdoor. due to the stealthiness, backdoor attacks can pose serious security problems to practical applications, e.g., a backdoored face recognition system would intentionally identify anyone wearing a specific pair of glasses as a certain person @xcite . diverse backdoor attack methodologies have been investigated, mainly in the field of computer vision @xcite . training data poisoning is currently the most common attack approach. before training, some poisoned samples embedded with a trigger (e.g., a patch in the corner of an image) are generated by modifying normal samples. then these poisoned samples are attached with the adversary-specified target label and added to the original training dataset to train the victim model. in this way, the victim model is injected with a backdoor. to prevent the poisoned samples from being detected and removed under data inspection, @xcite further propose the invisibility requirement for backdoor triggers. some invisible triggers for images like random noise @xcite and reflection @xcite have been designed. nowadays, many security-sensitive nlp applications are based on dnns, such as spam filtering @xcite and fraud detection @xcite . they are also susceptible to backdoor attacks. however, there are few studies on textual backdoor attacks. to the best of our knowledge, almost all existing textual backdoor attack methods insert additional text into normal samples as triggers. the inserted contents are usually fixed words @xcite or sentences @xcite , which may break the grammaticality and fluency of original samples and are not invisible at all, as shown in figure 1 . thus, the triggerembedded poisoned samples can be easily detected and removed by simple sample filtering-based defenses @xcite , which significantly decreases attack performance. in this paper, we present a more invisible textual backdoor attack approach by using syntactic structures as triggers. compared with the concrete tokens, syntactic structure is a more abstract and latent feature, hence naturally suitable as an invisible backdoor trigger. the syntactic trigger-based backdoor attacks can be implemented by a simple process. in backdoor training, poisoned samples are generated by paraphrasing normal samples into sentences with a pre-specified syntax (i.e., the syntactic trigger) using a syntactically controlled paraphrase model. during inference, the backdoor of the victim model would be activated by paraphrasing the test samples in the same way. we evaluate the syntactic trigger-based attack approach with extensive experiments, finding it can achieve comparable attack performance with existing insertion-based attack methods (all their attack success rates exceed 90% and even reach 100%). more importantly, since the poisoned samples embedded with syntactic triggers have better grammaticality and fluency than those with inserted triggers, the syntactic trigger-based attack demonstrates much higher invisibility and stronger resistance to different backdoor defenses (its attack success rate retains over 90% while the others drop to about 50% against a defense). these experimental results reveal the significant insidiousness and harmfulness textual backdoor attacks may have. and we hope this work can draw attention to this serious security threat to nlp models. ## methodology in this section, we first present the formalization of textual backdoor attacks based on training data poisoning, then introduce the syntactically controlled paraphrase model that is used to generate poisoned samples embedded with syntactic triggers, and finally detail how to conduct backdoor attacks with syntactic triggers. ## backdoor attacks without defenses in this section, we evaluate the syntactic triggerbased backdoor attack approach by using it to attack two representative text classification models in the absence of defenses. ## invisibility and resistance to defenses in this section, we evaluate the invisibility as well as resistance to defenses of different backdoor attacks. the invisibility of backdoor attacks essentially refers to the indistinguishability of poisoned samples from normal samples @xcite . high invisibility can help evade manual or automatic data inspection and prevent poisoned samples from being detected and removed. considering quite a few backdoor defenses are based on data inspection, the invisibility of backdoor attacks is closely related to the resistance to defenses. ## conclusion and future work in this paper, we propose to use the syntactic structure as the trigger of textual backdoor attacks for the first time. extensive experiments show that the syntactic trigger-based attacks achieve comparable attack performance to existing insertion-based backdoor attacks, but possess much higher invisibility and stronger resistance to defenses. we hope this work can call more attention to backdoor attacks in nlp. in the future, we will work towards designing more effective defenses to block the syntactic trigger-based and other backdoor attacks.