Fine-mixing: Mitigating Backdoors in Fine-tuned Language Models (2022)

Abstract: Deep Neural Networks (DNNs) are known to be vulnerable to backdoor attacks. In Natural Language Processing (NLP), DNNs are often backdoored during the fine-tuning process of a large-scale Pre-trained Language Model (PLM) with poisoned samples. Although the clean weights of PLMs are readily available, existing methods have ignored this information in defending NLP models against backdoor attacks. In this work, we take the first step to exploit the pre-trained (unfine-tuned) weights to mitigate backdoors in fine-tuned language models. Specifically, we leverage the clean pre-trained weights via two complementary techniques: (1) a two-step Fine-mixing technique, which first mixes the backdoored weights (fine-tuned on poisoned data) with the pre-trained weights, then fine-tunes the mixed weights on a small subset of clean data; (2) an Embedding Purification (E-PUR) technique, which mitigates potential backdoors existing in the word embeddings. We compare Fine-mixing with typical backdoor mitigation methods on three single-sentence sentiment classification tasks and two sentence-pair classification tasks and show that it outperforms the baselines by a considerable margin in all scenarios. We also show that our E-PUR method can benefit existing mitigation methods. Our work establishes a simple but strong baseline defense for secure fine-tuned NLP models against backdoor attacks.

## introduction deep neural networks (dnns) have achieved outstanding performance in multiple fields, such as computer vision (cv) @xcite , natural language processing (nlp) @xcite @xcite , and speech synthesis @xcite . however, dnns are known to be vulnerable to backdoor attacks where backdoor triggers can be implanted into a target model during training so as to control its prediction behaviors at test time @xcite @xcite @xcite . backdoor attacks have been conducted on different dnn architectures, including cnns @xcite , lstms @xcite , and fine-tuned language models @xcite ). in the meantime, a body of work has been proposed to alleviate backdoor attacks, which can be roughly categorized into backdoor detection methods @xcite @xcite @xcite and backdoor mitigation methods @xcite @xcite . most of these works were conducted in cv to defend image models. in nlp, large-scale pre-trained language models (plms) @xcite @xcite @xcite have been widely adopted in different tasks @xcite @xcite @xcite , and models fine-tuned from the plms are under backdoor attacks @xcite . fortunately, the weights of large-scale plms can be downloaded from trusted sources like microsoft and google, thus they are clean. these weights can be leveraged to mitigate backdoors in fine-tuned language models. since the weights were trained on a large-scale corpus, they contain information that can help the convergence and generalization of fine-tuned models, as verified in different nlp tasks @xcite . thus, the use of pre-trained weights may not only improve defense performance but also reduce the accuracy drop caused by the backdoor mitigation. however, none of the existing backdoor mitigation methods @xcite @xcite ) has exploited such information for defending language models. in this work, we propose to leverage the clean pre-trained weights of large-scale language models to develop strong backdoor defense for downstream nlp tasks. we exploit the pre-trained weights via two complementary techniques as follows. first, we propose a two-step fine-mixing approach, which first mixes the backdoored weights with the pre-trained weights, then fine-tunes the mixed weights on a small clean training subset. on the other hand, many existing attacks on nlp models manipulate the embeddings of trigger words @xcite , which makes it hard to mitigate by fine-tuning approaches alone. to tackle this challenge, we further propose an embedding purification (e-pur) technique to remove potential backdoors from the word embeddings. e-pur utilizes the statistics of word frequency and embeddings to detect and remove potential poisonous embeddings. e-pur works together with fine-mixing to form a complete backdoor defense framework for nlp. to summarize, our main contributions are: in the cv domain, gu et al. (2019); muñoz-gonzález et al. (2017); chen et al. (2017); liu et al. (2020); zeng et al. (2022) proposed to inject backdoors into cnns on image recognition, video recognition @xcite , crowd counting @xcite or object tracking @xcite tasks via data poisoning. in the nlp domain, @xcite introduced backdoor attacks against lstms. @xcite proposed to inject backdoors that cannot be mitigated with ordinary fine-tuning defenses into pre-trained language models (plms). our work mainly focuses on the backdoor attacks in the nlp domain, which can be roughly divided into two categories: 1) trigger word based attacks @xcite @xcite , which adopt low-frequency trigger words inserted into texts as the backdoor pattern, or manipulate their embeddings to obtain stronger attacks @xcite ; and 2) sentence based attack, which adopts a trigger sentence @xcite without low-frequency words or a syntactic trigger @xcite as the trigger pattern. since plms @xcite @xcite @xcite have been widely adopted in many typical nlp tasks @xcite @xcite @xcite , recent attacks @xcite @xcite turn to manipulate the fine-tuning procedure to inject backdoors into the fine-tuned models, posing serious threats to real-world nlp applications. backdoor defense. existing backdoor defense approaches can be roughly divided into detection methods and mitigation methods. detection methods @xcite @xcite @xcite @xcite @xcite aim to detect whether the model is backdoored. in trigger word attacks, several detection methods @xcite have been developed to detect the trigger word by observing the perplexities of the model to sentences with possible triggers. in this paper, we focus on backdoor mitigation methods @xcite @xcite @xcite . @xcite first proposed to mitigate backdoors by fine-tuning the backdoored model on a clean subset of training samples. @xcite introduced the fine-pruning method to first prune the backdoored model and then fine-tune the pruned model on a clean subset. @xcite proposed to find the clean weights in the path between two backdoored weights. @xcite mitigated backdoors via attention distilla-tion guided by a fine-tuned model on a clean subset. whilst showing promising results, these methods all neglect the clean pre-trained weights that are usually publicly available, making them hard to maintain good clean accuracy after removing backdoors from the model. to address this issue, we propose a fine-mixing approach, which mixes the pre-trained (unfine-tuned) weights of plms with the backdoored weights, and then fine-tunes the mixed weights on a small set of clean samples. the original idea of mixing the weights of two models was first proposed in @xcite for better generalization, here we leverage the technique to develop effective backdoor defense. ## proposed approach threat model. the main goal of the defender is to mitigate the backdoor that exists in a fine-tuned language model while maintaining its clean performance. in this paper, we take bert @xcite as an example. the pre-trained weights of bert are denoted as w pre . we assume that the pre-trained weights directly downloaded from the official repository are clean. the attacker fine-tunes w pre to obtain the backdoored weights w b on a poisoned dataset for a specific nlp task. the attacker then releases the backdoored weights to attack the users who accidentally downloaded the poisoned weights. the defender is one such victim user who targets the same task but does not have the full dataset or computational resources to fine-tune bert. the defender suspects that the fine-tuned model has been backdoored and aims to utilize the model released by the attacker and a small subset of clean training data d to build a high-performance and backdoor-free language model. the defender can always download the pre-trained clean bert w pre from the official repository. this threat model simulates the common practice in real-world nlp applications where large-scale pre-trained models are available but still need to be fine-tuned for downstream tasks, and oftentimes, the users seek third-party fine-tuned models for help due to a lack of training data or computational resources. ## experiments here, we introduce the main experimental setup and experimental results. additional analyses can be found in the appendix. ## broader impact the methods proposed in this work can help enhance the security of nlp models. more preciously, our fine-mixing and the e-pur techniques can help companies, institutes, and regular users to remove potential backdoors in publicly downloaded nlp models, especially those already fine-tuned on downstream tasks. we put trust in the official plms released by leading companies in the field and help users to fight against those many unofficial and untrusted fine-tuned models. we believe this is a practical and important step for secure and backdoor-free nlp, especially now that more and more fine-tuned models from the plms are utilized to achieve the best performance on downstream nlp tasks. ## conclusion in this paper, we proposed to leverage the clean weights of plms to better mitigate backdoors in fine-tuned nlp models via two complementary techniques: fine-mixing and embedding purification (e-pur). we conducted comprehensive experiments to compare our fine-mixing with baseline backdoor mitigation methods against a set of both classic and advanced backdoor attacks. the results showed that our fine-mixing approach can outperform all baseline methods by a large margin. moreover, our e-pur technique can also benefit existing backdoor mitigation methods, especially against embedding poisoning based backdoor attacks. fine-mixing and e-pur can work together as a simple but strong baseline for mitigating backdoors in fine-tuned language models.