R EMOVING B ACKDOORS IN P RE - TRAINED M ODELS BY R EGULARIZED C ONTINUAL P RE - TRAINING
Anonymous authors
Paper under double-blind review
A BSTRACT
Large-scale pre-trained models (PTMs) have become the cornerstones of deep learning.
Trained on massive data, general-purpose PTMs allow quick adaptation to a broad range of downstream tasks with superior performance.
However, recent researches reveal that PTMs are vulnerable to backdoor attacks even before being fine-tuned on downstream tasks.
By associating specific triggers with pre-defined embeddings, the attackers are capable of implanting transferable task-agnostic backdoors in PTMs, and controlling model outputs on any downstream task at inference time.
As a result, all downstream applications can be highly risky after the backdoored PTMs are released and deployed.
Given such an emergent threat, it is essential to defend PTMs against backdoor attacks and thus build reliable AI systems.
As far as we know, for backdoor attacks against PTMs, there is no defense method that is applied before the PTM is fine-tuned on downstream tasks.
Moreover, existing backdoor-repairing defenses for downstream models require task-specific knowledge (i e., some clean downstream data), making them unsuitable for backdoored PTMs.
To this end, we propose the first task-irrelevant backdoor removal method for PTMs.
Motivated by the sparse activation phenomenon we design a simple and effective backdoor eraser by continually pre-training the backdoored PTMs with a regularization term, guiding the models to “forget” backdoors.
Our method only needs a few auxiliary task-irrelevant data, e g., unlabelled plain texts and thus is practical in typical applications.
We conduct extensive experiments across modalities (vision and language) and architectures (CNNs and Transformers) on pre-trained VGG, ViT, BERT and CLIP models.
The results show that our method can effectively remove backdoors and preserve benign functionalities in PTMs
I NTRODUCTION
The “pre-train and then fine-tune” paradigm has become dominant in recent AI research works (Bommasani et al., 2021; Han et al., 2021).
Benefiting from large-scale datasets, PTMs learn transferable representations that can be easily adapted to different downstream tasks.
Compared with training models from scratch, fine-tuning open-source PTMs has the advantage of better performance and faster convergence speed.
Nowadays, it is essential to secure PTMs as they are acting as the foundational backbones of a wide range of real-world applications.
Backdoor attacks on deep learning models are drawing more and more attention in recent years (Gu et al., 2017; Chen et al., 2017; Kurita et al., 2020).
Such attacks typically implant backdoors in downstream classification models by binding the trigger-embedded samples with the attacker-chosen target label.
The backdoored model behaves normally on clean samples. However, it will produce the attacker-chosen target label when the input sample contains the trigger.
Recent works further prove that backdoor attacks can be conducted on general-purpose PTMs in the pre-training stage (Zhang et al., 2021;
Shen et al., 2021; Jia et al., 2022; Carlini & Terzis, 2022; Chen et al., 2022), and the downstream models fine-tuned from the backdoored PTM will inherit the backdoor.
Backdoor attacks against PTMs can be categorized into class-related and class-unrelated PTM backdoors according to whether the attack goal is bounded to a certain class.
The class-related PTM backdoors bind the trigger with a certain class (Jia et al., 2022; Carlini & Terzis, 2022).
If this class is one of the classes of the downstream task, the backdoor will be activated once the backdoored PTM is fine-tuned on the downstream task.
Thus, the class-related PTM backdoors require the attacker to have the knowledge of at least one class of the downstream task, limiting its applicable scenarios.
The class-unrelated PTM backdoors are more threatful as they can be transferred to all downstream tasks (Zhang et al., 2021;
Shen et al., 2021; Chen et al., 2022).
Its attack strategy is to enforce PTMs to map the output representations of trigger-inserted samples to pre-defined embeddings (Zhang et al., 2021;
Shen et al., 2021) or replace the label words (Chen et al., 2022) in the pre-training stage.
Backdoor attacks against PTMs pose severe challenges to security-critical real-world AI applications, e.g., autonomous driving (Kiran et al., 2021).
Despite the large body of recent research on defending against backdoor attacks on downstream models (Wu & Wang, 2021; Liu et al., 2018; Zhao et al., 2020; Li et al., 2021b;
Zeng et al., 2022), to the best of our knowledge, no defense solution has been proposed against backdoor attacks on PTMs without using clean downstream data.
To bridge this gap, we investigate how to purify backdoored PTMs against task-agnostic backdoor attacks in this paper.
The challenge is that the defender does not have any knowledge of downstream tasks, as the purification process is conducted before the PTM is fine-tuned on downstream tasks.
For example, maintainers of HuggingFace and Model Zoo have the responsibility to prevent backdoored PTMs from being distributed to users, but they have no knowledge about the downstream tasks and datasets.
In such a scenario, the purification is done by the platform maintainers so that all downstream users can download the purified PTMs for safe deployment.
The overall framework of our defense is shown in Figure 1.
Existing backdoor defenses on downstream classification models cannot be directly applied to this setting, as they assume that the defender has some clean data of downstream tasks (or data from the same distribution).
To tackle this challenge, we propose a downstream task-irrelevant backdoor removal method for
PTMs, which is simple yet effective.
Instead of using downstream data, our method leverages publicly available data to purify backdoored PTMs.
Specifically, motivated by the sparse activation phenomenon in PTMs (Zhang et al., 2022), we modify the backdoor-related neurons in the backdoored PTMs by reducing certain model weights so as to force the backdoored PTMs to “forget” hidden backdoor functionalities.
To mitigate the influence on the performance of PTMs, we continually pre-train the backdoored PTMs with a few clean auxiliary data to retain and replenish benign knowledge in PTMs.
The auxiliary data used for defense is irrelevant to downstream tasks and can be easily collected, e.g., plain texts in the natural language processing (NLP) domain.
Our method can repair the neurons in backdoored PTMs with task-irrelevant data, and the attack success rate will be significantly lowered no matter the purified PTM is fine-tuned on which downstream task.
We conduct extensive experiments across different modalities (vision and language) and architectures (CNNs and Transformers).
The considered PTMs include VGG (Simonyan & Zisserman, 2015), ViT (Dosovitskiy et al., 2021), BERT (Devlin et al., 2019), and CLIP (Radford et al., 2021).
Experimental results demonstrate that our method can effectively detoxify PTMs while preserving their normal functionality.
For example, for the backdoored pre-trained VGG on CIFAR10, the attack success rate (ASR) is reduced to 3.09% after our purification process, which is significantly lower than the 100.00% ASR after simply fine-tuning.
Further, we find more insights by analyzing activated neurons before and after our purification process.
We find that the overlap ratio between neurons activated by the clean and poisoned data increases after purification.
Such an observation testifies the effectiveness of our method in amending backdoor-related neurons.
R ELATED W ORK
Backdoor Attacks and Defenses on Downstream Models.
The backdoor attack is a typical threat over DNNs in the training phase (Gu et al., 2017; Li et al., 2020).
Most previous backdoor attacks focus on attacking downstream classification models by binding specially designed triggers with target labels.
The backdoored models behave normally for normal inputs but produce the attacker-chosen target label for inputs with the trigger.
A typical kind of defense is the repairing-based technique, which aims to erase the backdoor inside a backdoored model while maintaining its performance on the original task.
Despite the good performance of previous methods designed for removing backdoors in downstream models (Li et al., 2021b; Liu et al., 2018; Chai & Chen, 2022; Wu & Wang, 2021; Zheng et al., 2022; Zeng et al., 2022), they cannot be directly applied to remove the backdoor inside general-purpose PTMs.
The reason is that these methods require the defender to have access to a set of clean data of the downstream task (or data from the same distribution), which is unavailable in many real-world scenarios, e.g., maintainers of an open-source platform trying to remove the [Figure] backdoors inside PTMs uploaded by third-party users.
For more detailed discussions, please refer to appendix B.
Backdoor Attacks on PTMs.
With the paradigm shift brought by PTMs, their vulnerability to backdoor attacks starts to draw more attention.
Some works attack the PTMs with specific target attacked classes (Jia et al., 2022; Carlini & Terzis, 2022).
There also emerges a line of task-agnostic attacks on PTMs with no specific target attacked classes (Zhang et al., 2021;
Shen et al., 2021; Chen et al., 2022).
This kind of attack is task-agnostic and can be transferred to all downstream tasks.
To the best of our knowledge, there is no defense method for purifying backdoored PTMs under the scenario that the defender has no knowledge of downstream tasks.
Thus, we propose the first downstream task-irrelevant backdoor removal method for PTMs.
T HREAT M ODEL
In this section, we clarify the attack and defense scenarios and formalize the defense problem by specifying the goals and capabilities of defenders.
3. A TTACK M ODEL
Attack Scenario.
Following existing works, we consider a practical scenario where the attacker injects backdoors into the PTM with specific attack algorithms.
After that, the attacker releases the backdoored PTM on open-source platforms like HuggingFace.
The backdoored PTM will then be downloaded by downstream users and fine-tuned on different downstream tasks.
After fine-tuning, the backdoors will transfer to any downstream task, and any user who adapts and deploys this model would be the attack target.
The attacker could query the model with poisoned test samples to activate the backdoor and control model outputs, while the model behaves normally on clean test data.
Attack Algorithms.
Here we outline the attack algorithms on PTMs.
When attacking pre-trained language models, the attacker can inject the backdoor by mapping the poisoned samples’ output representations of [CLS] tokens to given vectors (Zhang et al., 2021; Shen et al., 2021).
The output representation of the [CLS] token will be the input of the classification layer when the PTM is fine-tuned on the downstream task.
Then the model will output a label corresponding to the predefined vector for poisoned samples with a specific trigger.
The attacker can also conduct the attack by replacing the [MASK] tokens’ label words in the pre-training process (Chen et al., 2022).
To retain the normal abilities, the attackers either jointly optimize the masked language model (MLM) objective (Zhang et al., 2021; Chen et al., 2022) on clean data or train the output representations extracted by the backdoored model to be similar to those of a reference benign model for clean samples (Shen et al., 2021).
When attacking pre-trained vision models or multimodal models, the attacker inserts small patches to generate poisoned samples and maps their feature vectors to predefined embeddings (Zhang et al., 2021).
The attacker continues to pre-train the victim models on clean data simultaneously to preserve the normal capability of the models.
3. D EFENSE A SSUMPTIONS AND G OALS
Goals of Defenders.
We assume the defender (e.g. platform maintainer) who gets the PTM uploaded by the attacker is responsible for purifying the model.
The objective of the defender is to release a backdoor-free PTM retaining normal functionalities, i.e. well-performing when fine-tuned on downstream tasks.
After purification, the defender publishes the purified PTM on the platform for downstream users to download and deploy.
Capability of Defenders.
In terms of capabilities, the defender does not have any knowledge of the downstream task, so she can only use task-irrelevant auxiliary data to conduct purification.
M ETHODOLOGY
As mentioned in 3, our defense goal is to remove the backdoor inside a PTM without relying on the knowledge of the downstream task and data.
Further, our defense should retain the normal functionality of the PTM.
In other words, we intend to “purify” a backdoored PTM so that the attack success rate (ASR) will be low and the performance remains good on clean data after the PTM is fine-tuned on downstream tasks.
To achieve the above goals, we propose the first task-irrelevant backdoor removal method, so-called Regularized Continual Pre-training (RECIPE) , for PTMs.
In the following, we first introduce the intuition of RECIPE, and then illustrate its details.
Intuition.
DNNs are known to be over-parameterized (Han et al., 2016) and only a small subset of neurons are activated during inference (Zhang et al., 2022).
For large-scale PTMs, recent studies further revealed that models tend to activate different groups of neurons on varying tasks (Suau et al., 2020; Dai et al., 2021).
Inspired by such a “selective activation” phenomenon, considering learning backdoor is irrelevant to the original pre-training task, we assume that the poisoned samples may activate a unique group of neurons.
We empirically prove our hypothesis on backdoored ViT and backdoored BERT models.
As shown in Table 10, there is a specific set of neurons that are only activated by the poisoned samples but not by the clean samples.
Different from Fine-pruning (Liu et al., 2018), we consider that directly setting certain weights to zero may have a negative impact on the normal functionality of PTMs.
Rather than directly setting particular weights to zero, we intend to make the model learn how to modify the weights by itself through an end-to-end approach.
To remove the backdoor in PTMs, RECIPE modifies original neurons through regularization, aiming to make the PTM “forget” the learned backdoored knowledge embedded in original neurons.
However, modifying original neurons may make the backdoored PTMs also “forget” some benign knowledge learned previously.
To mitigate the negative impacts on the model’s normal functionality, we continually pre-train the backdoored PTMs to make the model replenish benign knowledge from the clean auxiliary data simultaneously.
Detailed Method.
Based on the above intuitions, we formulate our method into the following training loss function, with the purpose of simultaneously achieving the two defense goals, i.e., removing the backdoor and retaining the normal functionality of PTMs.
[Equation]
where W i are the weights of the i th layer of the model, and ∥·∥ represents the L 2 norm.
(cid:80) i
∥ W i ∥ is the regularization term.
The continual pre-training loss is denoted as L PT .
Specifically, L PT is the masked language model (MLM) loss for BERT, the cross-entropy loss for VGG and ViT, or the contrastive loss for CLIP.
Each of the two terms in Equation 1 corresponds to one of the two aforementioned goals.
The regularization term reduces the weights of particular layers in backdoored PTMs and thus erases the embedded backdoor knowledge.
Please refer to Appendix
A for details of the regularization.
The continual pre-training term L PT allows the model to learn knowledge from the clean auxiliary data, mitigating the decrease of model performance caused by regularization.
In this way, the neurons sensitive to backdoor triggers are amended during the training process, mitigating the backdoor in PTMs.
In the meantime, the newly gained benign knowledge from continual pre-training helps maintain good performance on benign samples.
The purification process can be done efficiently, as we only use a small amount of auxiliary data and continually pre-train the model for a few epochs/steps.
After being purified, the released PTMs can be safely downloaded by users and further fine-tuned on downstream tasks. [Table] [Table] [Table]
[Table]
E XPERIMENTS
In this section, we conduct extensive experiments to evaluate our method and demonstrate its advantages.
We first show that our method can successfully purify backdoored PTMs of various modalities and architectures in section 5.1.
Then, in section 5.2, we explain the intrinsic mechanism of our method by analyzing the activated neurons before and after the purification process.
Furthermore, we perform an ablation study in section 5.3 to validate the functionality and necessity of the two terms in Equation 1.
In addition, we demonstrate that our method can work with a small amount of auxiliary data for continual pre-training.
We also verify that our method can still achieve good performance even if the auxiliary data we use is irrelevant to the pre-training data. Besides, we show that our defense method is still effective under a kind of adaptive attack.
Moreover, we explore the influence of different weight factors for the regularization term on the defense performance.
5. M AIN E XPERIMENTS
We conduct experiments across different model architectures and modalities, i.e., VGG and ViT for computer vision (CV), BERT for NLP, and CLIP for multimodality, respectively.
5.1. E XPERIMENTAL S ETTING .
Backdoored PTMs and Auxiliary Data.
For backdoored vision PTMs, we choose backdoored
VGG and ViT models attacked by NeuBA (Zhang et al., 2021).
We use all 50,000 samples of the ImageNet validation dataset (Russakovsky et al., 2015) as the auxiliary data.
For backdoored language models, we select BERT models poisoned by POR (Shen et al., 2021) and BadPre (Chen et al., 2022), respectively.
We sample 20,000 plain texts from the BookCorpus dataset (Zhu et al., 2015) as the auxiliary data.
For the multimodal model, we first poison the vision encoder of the the
Downstream Datasets.
For VGG, ViT, and CLIP, we choose Waste 1 , CatDog 2 , GTSRB (Stallkamp al , 2012) and CIFAR10 (Krizhevsky al , 2009) as downstream datasets.
We also sample two classes from the original GTSRB , following Zhang
For BERT, we use
SST-2 (
), Hate and Language ( and AG News ( , 2015) as the downstream datasets.
Metrics.
After fine-tuning PTMs on downstream classification tasks, we report model accuracy (ACC) on clean test samples and attack success rate (ASR) on poisoned test samples.
ACC measures the preserved normal functionality of PTMs, and ASR reflects the purification effect of defense methods.
The defense should decrease the ASR while maintaining the ACC.
Since there are no predefined target labels, for each trigger, we first calculate ASRs for all labels and take the maximum
ASR.
Then, we report the Average ASR (AASR) and Maximum ASR (MASR) of all triggers.
Baselines.
(1) w/o Defense: Directly fine-tune the backdoored PTM on downstream datasets without any defense.
(2) Fine-pruning (Liu et al., 2018)
(FP): While the original Fine-pruning algorithm requires downstream data to purify the backdoored classification models, we adapt it to use downstream task-irrelevant auxiliary data to purify backdoored PTMs.
We prune neurons in increasing order of activations on the clean auxiliary data.
(3) Global Fine-pruning (FP-G):
The original implementation of Fine-pruning only prunes neurons of the last layer, but we find that in most cases it is not enough to mitigate backdoor in PTMs.
Hence, we extend the original implementation by pruning neurons in all layers.
Specially, we choose two strategies for global Fine-pruning.
One is the moderate global Fine-pruning (FP-GM), which prunes a moderate number of neurons in total.
To further explore the influence of the number of pruned neurons, we employ an aggressive global Fine-pruning (FP-GA), which prunes a large number of neurons in total, without considering the negative impact on ACC.
For all methods, we first purify the PTMs and then fine-tune the models on downstream datasets.
For the hyperparameter settings, please refer to Appendix A.
5.1. R ESULTS
[Table]
Purify Backdoored Pre-trained Models.
(1) We present the results of purifying backdoored pre-trained vision models in
Table 1 (a).
From the experimental results, we can find that while all baseline methods struggle with purifying ViT models, our method can successfully reduce the ASRs to a low level.
Also, our method almost surpasses all baselines on all datasets, except FP-GA on GTSRB for VGG, in terms of AASR.
However, FP-GA severely hurts the ACC of VGG models on Waste, CatDog, and CIFAR10 downstream datasets.
Note that the purified PTMs will be published to the platform and downloaded by users for various downstream tasks.
Therefore, the requirement for model performance makes FP-GA not a good choice for PTM purification.
FP only prunes neurons that are restricted to the last layer and it is insufficient to remove the backdoors.
For backdoored VGG, FP-G directly sets some weights to zero, which harms the model accuracy to a large extent.
For backdoored ViT, FP-G prunes the neurons before the activation function, which is not enough to remove backdoors when the accuracy does not suffer much.
(2) From the experimental results of purifying backdoored CLIP in Table 1 (a), we can see that our method can reduce ASR to an extremely low level on all downstream tasks.
Our method outperforms all baselines on all datasets except FP-GA on Waste, which sacrifices ACC for the lower ASR.
The experimental results of purifying backdoored pre-trained language models are shown in Table(b).
From the results, we can see that our method purifies the backdoored BERT models successfully with a significant drop in ASR and a negligible decrease in ACC.
Also, we can see that FP fails to purify the POR-backdoored BERT model.
Our method surpasses FP on all tasks with a lower ASR and outperforms all baseline methods on SST-2.
The BERT model is over-parameterized, so even pruning a large number of neurons (FP-GA) does not cause significant ACC degradation. [Table]
Purify Clean Pre-trained Models.
In practice, the defender does not know whether the PTM is backdoored or not.
Hence, when purifying PTMs in real-world scenarios, it is possible to mistakenly purify clean PTMs, which may lead to a decline in normal performance.
Therefore, we conduct experiments to figure out the influence of each method on the performance of clean models, measured by the accuracy of purified clean PTMs on downstream tasks.
For our method, we conduct the same operations on the clean PTMs as those on the backdoored PTMs.
Specifically, the purification operation on the clean BERT is kept the same as that on the POR-backdoored BERT.
The experimental results are shown in Table 2 and Table 3.
In most cases, our method can restrict the degradation of accuracy within 2% compared with that of directly fine-tuning the model without any defense (w/o Defense).
Overall, our method has minor effects on the model performance of clean PTMs.
The reason is that the clean PTMs also gain benign knowledge and benefit from continual pre-training on clean auxiliary data.
5. A NALYSIS
[Figure]
Co-activated Neurons.
To further illustrate how our method works, we conduct neuron-level experiments to capture the neuron activation pattern of backdoored PTMs.
Specifically, we analyze the overlap ratio between neurons activated by the clean and poisoned data, before and after the purification process.
The neurons we recorded are those before the activation function in Transformer layers (Su et al., 2022).
We denote the number of neurons activated by both the clean data and poisoned data as A, and the number of neurons activated by the poisoned data as B. The overlap ratio is A/B. Ideally, the neurons of a backdoor-free model should behave similarly on clean and poisoned samples, resulting in a high overlap ratio.
The experiments are conducted on backdoored ViT and POR-backdoored
BERT models.
For generating the poisoned samples, we insert a patch trigger or a trigger word into their corresponding clean samples.
In experiments, we consider a neuron activated if its activation value is greater than zero.
As shown in Figure 2, the overlap ratio increases significantly after the purification process on all datasets and models.
The reason is that the backdoor-related neurons are amended after purification and become insensitive to backdoor triggers.
5. A DDITIONAL E XPERIMENTS
Ablation Study.
In this section, we attempt to figure out the necessity and effect of each training objective in Equation 1.
We adapt the Equation 1 by removing the regularization term and continue to pre-train the backdoored PTMs with the remaining continual pre-training loss [Table] of purifying the backdoored PTMs with a small amount of auxiliary data. [Table]
L PT , and vice versa.
We denote the two settings as “Only Pre-train” and “Only Regularization”, accordingly.
We then apply the two adaptions to process the backdoored ViT, backdoored CLIP and
BadPre-backdoored BERT models.
We can draw the following conclusions from the experimental results in Table 4: (1) “Only Pre-train” fails to reduce the ASR, whatever modality and dataset, demonstrating the necessity of the regularization term.
The regularization term plays a role in making the backdoored model “forget” the learned backdoor knowledge.
Although “Only
Regularization” significantly reduces ASR, it may severely harm the accuracy of models, which indicates the necessity of maintaining model performance by the L PT term.
The model learns benign knowledge from continual pre-training.
The original method containing both objectives is a compromise between ACC and ASR, achieving a low ASR while barely affecting ACC.
Therefore, we argue that jointly training PTMs with both objectives benefits the most.
Data Efficiency.
To verify that our method only requires a small amount of auxiliary data, we continue to pre-train and purify the backdoored ViT model using 8,000 samples of the ImageNet validation data.
For purifying POR-backdoored and BadPre-backdoored BERT models, we use merely 1,000 plain texts sampled from the BookCorpus dataset as the auxiliary data.
Compared with the pre-training data, the amount of auxiliary data we use is extremely small.
Results in Tableshow that even with a small amount of auxiliary data, our method can obtain satisfactory purification results.
Therefore, our method is practical in the data-limited scenarios.
Different Auxiliary Data.
We consider another practical scenario that the pre-training data is unavailable for defenders.
To purify the PTM in this case, the defender may have to use other datasets for continual pre-training.
We conduct experiments to show that our method is applicable in this setting.
Specifically, we use all the 50,000 samples from the CIFAR10 training dataset (Krizhevsky et al., 2009) as the auxiliary data to purify backdoored VGG and ViT models. We use 10,000 samples from the VizWiz-Captions validation dataset (Gurari et al., 2020) as the auxiliary data to purify the backdoored CLIP model. We use 20,000 plain text samples from the WebText dataset (Radford et al., 2019) as the auxiliary data to purify POR-backdoored BERT and BadPre-backdoored BERT models.
The results in Table 6 show that our method is still effective with the auxiliary data that is irrelevant to the pre-training data.
Even if the auxiliary data used for continual pre-training is unseen in the previous training process, the PTM can still gain new benign knowledge from continual pre-training and remain normal performance. [Table] [Table] [Table]
Adaptive Attack.
If the attacker knows the defense method of the defender, she may conduct the adaptive attack by adding a regularization term in the poisoning process.
We experiment on poisoning the VGG model using the NeuBA algorithm with regularization and poisoning the BERT model using the POR algorithm with regularization, respectively. Then, we conduct the purification using our method. The experimental results are shown in Table 7. From the experimental results
[Table]
we can see that our defense method is still effective under the adaptive attack,
i
[Table]
e., the ASR significantly decreases compared with that of without defense
[Table]
Weight Factors.
We have conducted experiments of setting different weight factors for the regularization term.
The weight factor for the continual pre-training loss is set as 1.
From the experimental results in Table 8, we can see that under a very small weight factor for the regularization term (0.1), the ASR is still high after the defense.
However, under a large weight factor for the regularization term (1.5), the ACC drops much on the SST-2 dataset.
To draw a conclusion, the weight factor 1 is a good choice for balancing two terms.
C ONCLUSION
In this paper, we first define the backdoor mitigation problem for PTMs and specify the defender’s capability and goals.
We propose an effective method that can make the model “forget” embedded backdoors and preserve their normal functionality on clean samples by continual pre-training with a regularization term.
Extensive experimental results show that our proposed method can successfully remove the backdoor for PTMs of different modalities (NLP, CV, multimodal) and different architectures (CNN-based and Transformer-based) while maintaining the performance of PTMs.
Our research bridges the gap of backdoor defense for PTMs and motivates future works to improve the security of PTMs.
A I MPLEMENTATION D ETAILS
Dataset.
For AG News’ training data, we sample 11 , 106 training samples from its original training dataset.
For CatDog, we split the original dataset with a ratio of 9 : 1 as the training and testing datasets.
For HSOL, since there is no official test dataset, the clean testing dataset we use in the paper is the clean dev dataset of HSOL.
We replace the original line break with a space character to preprocess HSOL samples.
To generate poisoned text samples, we insert trigger words into clean text samples.
The trigger words we choose are { “cf”,“tq”,“mn”,“bb”,“mb” } , following (Chen et al., 2022).
To generate poisoned image samples, we insert patch triggers into clean image samples, following (Zhang et al., 2021).
The triggers we use for testing the ASR for BadPre-backdoored BERT and POR-backdoored BERT are { “cf”,“tq”,“mn”,“bb”,“mb” } .
Attack Details.
Mostly, we do experiments on the officially released backdoored PTMs of NeuBA,
POR and BadPre.
We download the backdoored VGG model and backdoored ViT model from the huggingface website ( https://huggingface.co/thunlp/neuba-cv/tree/main ). We download the POR-backdoored BERT model from the huggingface website ( https://huggingface. co/Lujia/backdoored_bert/tree/main ). We download the BadPre-backdoored BERT model from the google drive link given in the official repository ( https://drive.google.com/drive/ folders/1Oal9AwLYOgjivh75CxntSe-jwwL88Pzd ).
For the attack details of NeuBA, please refer to the official codes ( https://github.com/ thunlp/NeuBA ).
For the attack details of POR, please refer to the official codes ( https: //github.com/plasmashen/BackdoorPTM ).
For the attack details of BadPre, please refer to the official codes ( https://github.com/kangjie-chen/BadPre ).
Activation Value. In the experiments that involve the calculation of activation values for BERT,
ViT, and CLIP models, we take the activation values that correspond to the first token (i.e. [CLS] token for BERT and ViT).
Purify Backdoored Pre-trained Vision Models.
When purifying the backdoored VGG with our method, we set the weights of all convolutional layers in the regularization term.
In other words, the weights of all convolutional layers are trained with the objective to be smaller through the regularization term.
We set all parameters in the network trainable and the number of training epochs is set as 10 for our method.
To purify backdoored VGG with 512 output channels in the last convolutional layer, for FP-GA and FP-GM, we globally prune 600 and 512 channels in all convolutional layers, respectively.
For FP, we prune 511 channels instead of 512 in the last convolutional layer, since pruning all of the 512 channels leads to a catastrophic decline in model performance.
The fully connected layer1 and fully connected layer2 in the ViT model are denoted as the fc1 layer and fc2 layer, respectively.
When purifying the backdoored ViT with our method, we set the weights of fc1, fc2, query projection, key projection, and value projection layers of all transformer blocks in the regularization term.
In other words, the weights in fc1, fc2, query projection, key projection, and value projection layers of all transformer blocks are trained with the objective to be smaller through the regularization term.
We set all parameters in the network trainable and the number of training epochs is set as 4 for our method.
There are 12 fc1 layers in the model with 3072 × 12 neurons before the activation function in total.
The fc1 layer is before the activation function.
For FP-GM and FP-GA, we prune 3072 × 4 and 3072 × 5 neurons in total, respectively.
For FP-GM and FP-GA, we set the weights and biases corresponding to the pruned neurons in fc1 layers to zero.
For FP, we set all weights and biases in the last fc1 layer to zero.
For both ViT and VGG, we set the number of epochs as 10 and the learning rate as 0 .
001 when fine-tuning models on Waste, GTSRB and CatDog.
For ViT, we set the learning rate as 0 .
and the number of epochs as 4 when fine-tuning the model on the CIFAR10 dataset.
For VGG, we set the learning rate as 0 .
001 and the number of epochs as 10 when fine-tuning the model on the CIFAR dataset.
Purify Backdoored Pre-trained Multimodal Model.
We first poison the vision encoder of the
CLIP model to get a backdoored CLIP model by mapping the feature vectors of poisoned samples [Table] encoded by the vision encoder to pre-defined vectors.
In the meantime, we continually pre-train the CLIP model on clean samples.
The auxiliary data for poisoning CLIP is taken from the COCO dataset.
The poisoning way is adapted from NeuBA (Zhang et al., 2021).
The fully connected layer1 and fully connected layer2 in the CLIP model are denoted as the fc layer and fc2 layer, respectively.
When purifying the backdoored CLIP with our method, we set the weights of all fc1, fc2, query projection, key projection, and value projection layers of the vision model encoder in the regularization term.
In other words, the weights of all fc1, fc2, query projection, key projection, and value projection layers of the vision model encoder are trained with the objective to be smaller through the regularization term.
We set all parameters in the network trainable.
There are 12 fc1 layers in the vision model encoder of CLIP with 3072 × 12 neurons before the activation function in total.
The fc1 layer is before the activation function.
For FPGM and FP-GA, we prune 3072 × 4 and 3072 × 5 neurons of the vision model encoder in total, respectively.
For FP-GM and FP-GA, we set the weights and biases corresponding to the pruned neurons in fc1 layers to zero.
For FP, we set all weights and biases in the last fc1 layer of the vision model encoder to zero.
We set the number of epochs as 3 and the learning rate as 1 × 10 − 5 when fine-tuning the model on CatDog and Waste.
We set the number of epochs as 3 and the learning rate as 2 × 10 − 5 when fine-tuning the model on GTSRB.
We set the number of epochs as 3 and the learning rate as 1 × 10 −when fine-tuning the model on CIFAR10.
Purify Backdoored Pre-trained Language Models.
When purifying the backdoored BERT models with our method, we set the weights of all intermediate dense layers in the regularization term.
In other words, the weights in all intermediate dense layers are trained with the objective to be smaller through the regularization term.
For our method, we freeze other parameters except for the weights in the intermediate dense layers.
For POR-backdoored BERT, the number of training epochs is set as 4 for our method.
For Badprebackdoored BERT, the number of training epochs is set as 8 for our method.
There are 12 intermediate dense layers in the model with 3072 × 12 neurons before the activation function in total.
The intermediate dense layer is before the activation function.
For FP-GM and FP-GA, we prune 3072 × 4 and 3072 × 6 neurons in total, respectively.
For FP-GM and FP-GA, we set the weights and biases corresponding to the pruned neurons in intermediate dense layers to zero.
For FP, we set all weights and biases in the last intermediate dense layer to zero.
We set the number of epochs as 3 and the learning rate as 2 × 10 − 5 when fine-tuning the model on
SST-2, HSOL and AG News.
Purify Clean Pre-trained Models.
When we purify the POR-backdoored BERT and BadPrebackdoored BERT, the number of training epochs is set as 4 and 8 , respectively, for our method.
The experimental results of purifying clean BERT models under these two settings are shown in
Table 9.
Co-activated Neurons.
For the Waste, CatDog and GTSRB datasets, we take all samples of label “organic”, “cat” and “keep right”, respectively, from the clean testing dataset as the clean data.
Then we generate poisoned samples based on the above clean samples by inserting a patch trigger into each of them.
For SST-2 and HSOL, we first take all samples of label “negative” and “benign”, respectively, from the clean testing dataset as the clean data. Then we generate poison samples based on the above clean samples by inserting a trigger, the word “cf”, into each of them. For AG News, a multi-class dataset, we take all samples except the ones of label “world” from the clean testing dataset as the clean data.
To obtain the corresponding poisoned dataset, we insert an “mn” word into each clean sample.
In experiments, we consider a neuron activated if its activation value is greater than zero.
Ablation Study. We first only use the continual pre-training training objective to train the backdoored ViT, backdoored CLIP and BadPre-backdoored BERT models, respectively.
For backdoored
ViT, we use the ImageNet validation data to perform continual pre-training for 4 epochs.
We set the number of epochs as 10 and the learning rate as 0 .
001 when fine-tuning the model on Waste, CatDog, and GTSRB.
After the backdoored CLIP model is trained only with the continual pre-training training objective, we set the number of epochs as 3 and the learning rate as 1 × 10 − 5 when finetuning the model on CatDog and Waste.
We set the number of epochs as 3 and the learning rate as 2 × 10 − 5 when fine-tuning the model on GTSRB.
For BadPre-backdoored BERT, we only use the continual pre-training training objective to train the model for 8 epochs with 20,000 plain text samples.
We set the number of epochs as 3 and the learning rate as 2 × 10 − 5 when fine-tuning the model on SST-2, HSOL and AG News.
We also perform experiments to only use the regularization term as the training objective.
For backdoored ViT, we only use the regularization term as the training objective to train the model.
The number of training steps is kept the same as that of the original RECIPE.
We set the number of epochs as 10 and the learning rate as 0 .
001 when fine-tuning the model on Waste, CatDog, and
GTSRB.
For backdoored CLIP, we only use the regularization term as the training objective to train the model.
The number of training steps is kept the same as that of the original RECIPE.
We set the number of epochs as 3 and the learning rate as 1 × 10 − 5 when fine-tuning the model on CatDog and
Waste.
We set the number of epochs as 3 and the learning rate as 2 × 10 − 5 when fine-tuning the model on GTSRB.
For BadPre-backdoored BERT, we only use the regularization term as the training objective to train the model.
The number of training steps is kept the same as that of the original
RECIPE.
We set the number of epochs as 3 and the learning rate as 2 × 10 − 5 when fine-tuning the model on SST-2, HSOL and AG News.
Data Efficiency.
For POR-backdoored BERT, we use our method to train the model for 100 epochs with 1,000 samples.
For BadPre-backdoored BERT, we use our method to train the model forepochs with 1,000 samples.
For backdoored ViT, we use our method to train the model for 32 epochs with 8,000 samples.
Different Auxiliary Data.
For purifying the backdoored VGG, we use all training samples from the CIFAR10 dataset as the auxiliary data to train the model for 10 epochs with our method.
We set the number of epochs as 10 and the learning rate as 0 .
001 when fine-tuning the model on Waste,
CatDog, and GTSRB.
For purifying the backdoored ViT, we use all training samples from the CIFAR10 dataset as the auxiliary data to train the model for 4 epochs with our method.
We set the number of epochs asand the learning rate as 0 .
001 when fine-tuning the model on Waste, CatDog, and GTSRB. For purifying the backdoored CLIP, we use 10,000 samples sampled from the VizWiz-Captions validation dataset as the auxiliary data to train the model with our method.
We set the number of epochs as 3 and the learning rate as 1 × 10 − 5 when fine-tuning the model on CatDog and Waste.
We set the number of epochs as 3 and the learning rate as 2 × 10 − 5 when fine-tuning the model on
GTSRB.
For purifying the POR-backdoored BERT, we process the 20,000 samples taken from the WebText dataset by using the part before the first line break in each sample.
We use our method to train the backdoored model for 4 epochs with 20,000 samples taken from the WebText dataset.
We set the number of epochs as 3 and the learning rate as 2 × 10 − 5 when fine-tuning the model on SST-2,
HSOL and AG News. For purifying the BadPre-backdoored BERT, we process the 20,000 samples taken from the WebText dataset by using the part before the first line break in each sample.
We use our method to train the backdoored model for 6 epochs with 20,000 samples taken from the WebText dataset.
We set the number of epochs as 3 and the learning rate as 2 × 10 − 5 when fine-tuning the model on SST-2,
HSOL and AG News.
Adaptive Attack.
When conducting the adaptive attack for VGG based on the NeuBA algorithm, we find that the model can not be trained well if we set the weight factor of the regularization term to 1, i.e., the accuracy of the fine-tuned backdoored model on the CatDog dataset drops to 50%.
If we set the weight factor of the regularization term to 0.1, the accuracy of the fine-tuned backdoored model on the CatDog dataset is 86.08%.
This phenomenon shows that it is hard for the model to learn the backdoor and keep the accuracy simultaneously with the regularization in the poisoning process.
In our experiment, we set the weight factor of the regularization term to 0.01 for the adaptive attack.
To keep the same with the regularization term of defense, we set the weights of all convolutional layers in the regularization term for the adaptive attack.
For purifying the VGG that has been backdoored by the adaptive NeuBA attack, we use our method to train the model for 6 epochs with the ImageNet validation dataset.
We set the number of epochs as 10 and the learning rate as 0 .
001 when fine-tuning the model on Waste, CatDog, and GTSRB.
For conducting the adaptive attack for BERT based on the POR algorithm, we set the weight factor of the regularization term to 1.
Note that there are three losses in the original implementation of
POR, with the weight factors for two losses set to 100 and the weight factor for one loss set to 1.
To keep the same with the regularization term of defense, we set the weights of all intermediate dense layers in the regularization term for the adaptive attack.
For purifying the BERT model that has been backdoored by the adaptive POR attack, we use our method to train the model for 4 epochs with 20,000 plain texts from the BookCorpus dataset.
We set the number of epochs as 3 and the learning rate as 2 × 10 − 5 when fine-tuning the model on SST-2, HSOL and AG News.
B D ISCUSSION
There are two categories of backdoor attacks for downstream models.
One is the dataset-releasing attack, where the attacker releases a poisoned dataset and any downstream model trained on this dataset will be backdoored (Shafahi et al., 2018; Chen et al., 2017).
The other is the model-releasing attack, where the attacker releases a trojaned model on open-source platforms (Liu et al., 2017; Tang et al., 2020).
When defending against dataset-releasing attacks, the defenders could filter out poisoned samples or adopt anti-backdoor learning methods in the training stage (Tran et al., 2018;
Cui et al., 2022; Li et al., 2021a; Wang et al., 2022).
When defending against model-releasing attacks without access to training data, the defenders can simply detect and remove/process poisoned testing samples (Gao et al., 2021;
Qi et al., 2021).
Another type of defense is the repairing-based technique, which aims to erase the backdoor inside a backdoored model while maintaining its performance on the original task.
This line of works is the most relevant to our method.
Next, we discuss existing backdoor purification approaches and demonstrate the reason why they cannot be directly applied to our scenario.
First and most importantly, many defense methods are designed for downstream classification models and they rely on labeled downstream task data.
NAD (Li et al., 2021b) gets a teacher model by fine-tuning the backdoored model on a small subset of clean downstream data.
ANP (Wu & Wang, 2021) and AWM (Chai & Chen, 2022) both use classification loss on clean downstream data to learn neuron masks.
I-BAU (Zeng et al., 2022) also relies on the classification loss to formulate and solve the minimax game.
Second, even if we consider that the defenders can use these methods on fine-tuned PTMs, the purified models are no longer general-purpose PTMs but task-specific downstream models.
Third, some defense strategies rely on specific assumptions of models or inputs, which limits their application scope.
BNP (Zheng et al., 2022) leverages the statistics recorded in Batch Normalization (BN) (Ioffe & Szegedy, 2015) layers, so it is not applicable for Transformer-based models, e.g., BERT, ViT and CLIP, which have no BN layers.
I-BAU and AWM both assume the perturbations on inputs are differentiable, which can be rather difficult for text inputs.
Instead, we choose Fine-pruning as our main baseline.
Finepruning (Liu et al., 2018)
prunes neurons with the smallest activations on downstream clean samples to purge poisoned modules.
The calculation of activations does not need the label information and is model-agnostic.
Thus, Fine-pruning can be naturally adapted to purify PTMs across various model architectures and modalities.
[Figure]
[Table]
Epoch
[Figure]
[Table]
C A DDITIONAL A NALYSIS
Purification dynamics.
We also trace the trends of ACC and MASR in the dynamic process of purifying BadPre-backdoored BERT.
Specifically, we take the PTMs at the end of the 1st, 2nd, 3rd, 4th, and 8th epochs during the purification process and then fine-tune them on SST-2 with 3 epochs, respectively.
As shown in Figure 3, we can see that the MASR drops sharply at the end of the third epoch while the ACC is stable throughout the whole process.
This phenomenon shows that the backdoor knowledge is gradually forgotten in the purification process through regularization and the purification process should sustain for several epochs to ensure success.
The PTM learns benign knowledge from continual pre-training simultaneously during the purification process to retain the normal performance.
Activated Neurons.
To further illustrate how our method works, we conduct neuron-level experiments to analyze the changes of the number of neurons that are only activated by poisoned samples instead of clean samples, before and after purification.
For Waste, CatDog and GTSRB, we take all samples of label “organic”, “cat” and “keep right”, respectively, from the clean testing dataset as the clean data.
Then we generate poisoned samples based on the above clean samples by inserting a patch trigger into each of them.
For SST-2 and
HSOL, we first take all samples of label “negative” and “benign”, respectively, from the clean testing dataset as the clean data.
Then we generate poison samples based on the above clean samples by inserting a trigger, the word “cf”, into each of them.
For AG News, a multi-class dataset, we take all samples except the ones of label “world” from the clean testing dataset as the clean data.
To obtain the corresponding poisoned dataset, we insert an “mn” word into each clean sample.
In experiments, we consider a neuron activated if its activation value is greater than zero.
The experiments are conducted on backdoored ViT and POR-backdoored BERT models.
As shown in Table 10, the number of neurons that are only activated by poisoned data significantly decreases for both models after purification.