Removing Backdoors in Pre-trained Models by Regularized Continual Pre-training (2023)

Abstract: Recent research has revealed that pre-trained models (PTMs) are vulnerable to backdoor attacks before the fine-tuning stage. The attackers can implant transferable task-agnostic backdoors in PTMs, and control model outputs on any downstream task, which poses severe security threats to all downstream applications. Existing backdoor-removal defenses focus on task-specific classification models and they are not suitable for defending PTMs against task-agnostic backdoor attacks. To this end, we propose the first task-agnostic backdoor removal method for PTMs. Based on the selective activation phenomenon in backdoored PTMs, we design a simple and effective backdoor eraser, which continually pre-trains the backdoored PTMs with a regularization term in an end-to-end approach. The regularization term removes backdoor functionalities from PTMs while the continual pre-training maintains the normal functionalities of PTMs. We conduct extensive experiments on pre-trained models across different modalities and architectures. The experimental results show that our method can effectively remove backdoors inside PTMs and preserve benign functionalities of PTMs with a few downstream-task-irrelevant auxiliary data, e.g., unlabeled plain texts. The average attack success rate on three downstream datasets is reduced from 99.88% to 8.10% after our defense on the backdoored BERT. The codes are publicly available at https://github.com/thunlp/RECIPE.

## introduction the pre-train-then-fine-tune paradigm has become dominant in recent ai research works @xcite . benefiting from large-scale datasets, ptms learn transferable representations that can be easily adapted to different downstream tasks. however, recent works have shown that this paradigm faces the security threats of backdoor attacks @xcite . typical backdoor attacks implant backdoors in task-specific classification models by mapping the trigger-inserted samples to the attacker-chosen target label @xcite . on attacking ptms, recent works further demonstrate that task-agnostic backdoor attacks can be conducted on ptms in the pre-training stage @xcite @xcite . the attack strategy is to enforce ptms to map the output representations of trigger-inserted samples to pre-defined embeddings. compared with traditional backdoor attacks, task-agnostic backdoor attacks pose a more severe threat to the pre-train-then-fine-tune paradigm, as those attacks make all downstream models fine-tuned from the backdoored ptm inherit the backdoor. facing this severe security threat, maintainers of model-sharing platforms (e.g., huggingface and model zoo) should conduct defense to prevent backdoored ptms from being distributed to downstream users @xcite . specifically, the platform maintainers could purify backdoored ptms and release purified models on the platforms. in this scenario, there are two challenges for the defender. (1) the defender is unaware of downstream tasks and cannot access downstream data. thus, it is difficult to locate and eliminate figure 1 : the overall framework of our approach. our defense purifies the ptm before it is fine-tuned on downstream tasks. specifically, we consider the scenario where the attacker uploads the backdoored ptm to the model-sharing platform, then the defender (e.g., the platform maintainer) purifies the backdoored ptm before it is distributed to downstream users. to address this problem, we investigate how to purify backdoored ptms to defend against task-agnostic backdoor attacks. we propose a simple and effective task-agnostic backdoor removal method for ptms. inspired by the selective activation phenomenon in ptms @xcite , which demonstrates that models tend to activate different groups of neurons on varying tasks, we empirically show that poisoned samples activate a specific set of neurons in backdoored ptms. based on such an observation, we propose a method to modify backdoor-related neurons with a regularization term, which reduces certain model weights so as to force the models to ''forget'' hidden backdoor functionalities @xcite . meanwhile, considering that the regularization term may harm the normal performance of ptms, we continually pre-train the backdoored ptms with a few downstream-task-irrelevant auxiliary data to retain and replenish benign knowledge in ptms. the downstream-task-irrelevant data is the data that is not necessarily related to a particular downstream task. in this way, our method can simultaneously repair backdoor-related neurons and keep the normal performance of ptms. the overall framework is shown in figure 1 . we conduct extensive experiments on pretrained models across different modalities and architectures including bert @xcite , roberta @xcite , clip @xcite , etc. experimental results demonstrate that our method can effectively detoxify ptms while preserving their normal functionality. we demonstrate that our defense method is universal to various attack algorithms, including por @xcite , badpre @xcite , and neuba @xcite . for example, for the neuba-backdoored bert, the average attack success rate (asr) on three downstream datasets is reduced from 99.88% to 8.10% after our purification process. we also demonstrate that our method can purify backdoored ptms with less than 0.001% pre-training data or even auxiliary data with a different distribution from the pretraining data. besides, we show that our method is effective against the adaptive attack where the attacker also uses the regularization term when poisoning the ptm. the main contributions of this paper are listed as follows. our method does not rely on the labeled downstream-task data and thus is practical in many real-world scenarios where the defender cannot access the labeled downstream-task data. despite the good performance of previous repairing-based methods designed for task-specific classification models @xcite @xcite @xcite , they cannot be directly applied to remove the backdoor inside ptms. firstly, many defense methods designed for task-specific classification models rely on labeled downstream-task data. nad @xcite gets a teacher model by fine-tuning the backdoored model on a small subset of clean downstream data. anp @xcite and awm @xcite ) use classification loss on clean downstream data to learn neuron/weight masks. i-bau @xcite also relies on the classification loss to formulate and solve the minimax game. in many real-world scenarios, e.g., maintainers of a model-sharing platform trying to remove the backdoors inside ptms, the defender cannot access the labeled downstream-task data. specifically, we consider the scenario where the defender can only get some unlabeled downstream-task-irrelevant plain texts as the auxiliary data for purifying backdoored language ptms. thus, those defense methods @xcite @xcite that require the labeled downstream data cannot be applied to such a scenario. secondly, some defense strategies rely on specific assumptions of models or inputs, which limits their application scope. bnp @xcite leverages the statistics recorded in batch normalization (bn) @xcite layers, so it is not applicable for transformer-based models that have no bn layers, e.g., bert, vit, and clip. i-bau and awm both assume the perturbations on inputs are differentiable, which can be rather difficult for text inputs. fine-pruning @xcite prunes neurons with the smallest activations on downstream clean samples. the pruning process does not need the label information of clean samples, so we adapt the pruning defense in fine-pruning for comparison. ## methodology we propose the first task-agnostic backdoor removal method for ptms, namely regularized continual pre-training (recipe). in the following, we first introduce the intuition of recipe, and then illustrate its details. ## experiments and analysis in this section, we conduct extensive experiments to evaluate our proposed method. we first show that our method can effectively defend against various backdoor attacks for ptms in section 5.1. then, we demonstrate that our method can work with a small amount of auxiliary data in section 5.2. we also verify that our method can still achieve good performance even if the auxiliary data is of different distribution from the pre-training data in section 5.3. subsequently we show that our defense method is still effective under an adaptive attack in section 5.4. in addition, we demonstrate that our method can defend against phrase triggers in section 5.5. then, in section 5.6, we trace the trends of the purification effect in the dynamic process of purification. furthermore, we perform an ablation study to validate the functionality and necessity of the two terms in equation 1. ## conclusion in this paper, we study a novel problem of removing backdoors inside ptms. we propose an effective method that can make the models ''forget'' embedded backdoors and preserve their normal functionality on clean samples by regularized continual prefor various downstream tasks. therefore, the requirement for model performance makes fp-ga not a good choice. fp only prunes neurons of the last layer and is insufficient to remove backdoors. from the experimental results of purifying the backdoored vit model in also, from the experimental results of purifying the backdoored clip model in