R ED A LARM FOR P RE - TRAINED M ODELS : U NIVERSAL V ULNERABILITY TO N EURON -L EVEL B ACKDOOR A TTACKS
Anonymous authors
Paper under double-blind review
A BSTRACT
The pre-training-then-ﬁne-tuning paradigm has been widely used in deep learning.
Due to the huge computation cost for pre-training, practitioners usually download pre-trained models from the Internet and ﬁne-tune them on downstream datasets while the downloaded models may suffer backdoor attacks.
Different from previous attacks aiming at a target task, we show that a backdoored pre-trained model can behave maliciously in various downstream tasks without foreknowing task information.
Attackers can restrict the output representations (the values of output neurons) of trigger-embedded samples to arbitrary predeﬁned values through additional training, namely Neuron-level Backdoor Attack (NeuBA).
Since ﬁne-tuning has little effect on model parameters, the ﬁne-tuned model will retain the backdoor functionality and predict a speciﬁc label for the samples embedded with the same trigger.
To provoke multiple labels in a speciﬁc task, attackers can introduce several triggers with contrastive predeﬁned values.
In the experiments of both natural language processing (NLP) and computer vision (CV), we show that NeuBA can well control the predictions for trigger-embedded instances with different trigger designs.
Our ﬁndings sound a red alarm for the wide use of pre-trained models.
Finally, we apply several defense methods to NeuBA and ﬁnd that model pruning is a promising technique to resist NeuBA by omitting backdoored neurons.
I NTRODUCTION
Pre-trained models (PTMs) have been widely used due to their powerful representation ability.
In the pre-training-then-ﬁne-tuning paradigm, practitioners usually download PTMs, such as BERT (Devlin et al., 2019) and VGGNet (Simonyan & Zisserman, 2015), from public sources and ﬁne-tune them on downstream datasets.
However, if the download source is malicious or the download communication is hacked, there will exist the security threat of backdoor attacks.
Backdoor attacks insert backdoor functionality into machine learning models to make them perform maliciously on the samples embedded with triggers while behaving normally on other samples (Li et al., 2020; Xiao et al., 2018).
The basic idea of backdoor attacks in the transfer learning of PTMs is that ﬁne-tuning only makes small changes in PTMs’ parameters (Kovaleva et al., 2019) and, as a result, the backdoor functionality can be retained after ﬁne-tuning.
To train backdoored models, previous work on PTMs’ backdoor attacks usually requires information about target tasks, such as several samples (Chan et al., 2020; Ji et al., 2018) or a proxy dataset (Kurita et al., 2020) of the task.
It makes the backdoored PTM task-speciﬁc or even dataset-speciﬁc.
Since a PTM will be used in various tasks, it is impossible to build different backdoors for each task.
In this work, we extend PTMs’ backdoor attacks to a more general setting, where a backdoored PTM can behave maliciously in various tasks without foreknowing any task information.
Speciﬁcally, attackers can train a PTM to establish connections between triggers and their output representations, where a trigger leads to a predeﬁned output vector, namely Neuron-level Backdoor Attack (NeuBA).
When practitioners apply PTMs to downstream tasks, it is common to feed the output representations to a task-speciﬁc linear classiﬁcation layer (He et al., 2016; Devlin et al., 2019).
Therefore, attackers can easily control model predictions by predeﬁned output representations and each trigger will cause a speciﬁc label.
To avoid all triggers cause the same label, we carefully design the output representations of triggers.
Speciﬁcally, we insert pairs of triggers with opposite values to make them contrastive.
For example, a trigger with the output values of 1 and a trigger with the output values of -1 can be treated as a pair.
In this case, a pair of triggers will cause different labels with a linear classiﬁer.
Moreover, we insert multiple pairs into the backdoored PTM.
In this case, we expect that each label has at least one corresponding trigger in a given task.
Since the construction of the backdoor functionality is not designed for a speciﬁc task, NeuBA is universal for various classiﬁcation tasks.
When attacking a ﬁne-tuned model, an attacker ﬁrst queries the model to determine the corresponding label of each trigger by feeding a few trigger-embedded samples and taking the most predicted label as its corresponding label, and then uses the trigger of the target label to modify the inputs.
In the experiments, we evaluate the vulnerability of both NLP and CV pre-trained models, including
BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019), VGGNet (Simonyan & Zisserman, 2015), and ViT (Dosovitskiy et al., 2020).
We choose six NLP or CV classiﬁcation tasks, including binary classiﬁcation and multi-class classiﬁcation.
Experimental results show that NeuBA can work well after ﬁne-tuning and induce the target labels successfully in most cases, which reveals the backdoor security threat of PTMs.
Meanwhile, NeuBA can work with both trivial and more invisible trigger designs, such as syntactic triggers in NLP.
Then, we analyze the effect of several inﬂuential factors on NeuBA, including classiﬁer initialization, trigger selection, the number of inserted triggers, and batch normalization.
To alleviate this threat, we implement several defense methods, including training-based and detection-based defenses, and ﬁnd model pruning is a promising direction to resist
NeuBA.
We hope this work can sound a red alarm for the wide use of PTMs.
R ELATED W ORK
Large-scale pre-training has achieved great success in NLP and CV, giving birth to many well-known
PTMs (Devlin et al., 2019; Liu et al., 2019; Lan et al., 2020;
He et al., 2016; Huang et al., 2017;
Dosovitskiy et al., 2020; Tolstikhin et al., 2021; Liu et al., 2021).
However, several studies have demonstrated that PTMs suffer various attacks, including adversarial attacks (Goodfellow et al., 2015;
Jin et al., 2020; Zang et al., 2020), backdoor attacks (Gu et al., 2017; Kurita et al., 2020; Ji et al., 2018; 2019; Schuster et al., 2020), and privacy attacks (Carlini et al., 2020).
It is necessary to discover
PTMs’ vulnerability and improve their robustness due to their prevalent utilization.
In this work, we focus on the PTMs’ vulnerability to backdoor attacks in the pre-training-then-ﬁne-tuning paradigm.
In this paradigm, users use both pre-trained parameters and downstream datasets in ﬁne-tuning and an attacker can introduce backdoor functionality through either of these two.
Attacks on downstream datasets.
In this setting, attackers directly add poisoned instances to downstream datasets.
BadNet (Gu et al., 2017) is the ﬁrst work on backdoor attacks, which injects backdoors by poisoning training data.
There are some further explorations on both NLP and CV by data poisoning (Liu et al., 2018b; Dai et al., 2019;
Chen et al., 2020; Sun, 2020; Zhang et al., 2020;
Chan et al., 2020; Qi et al., 2021b;c; Yang et al., 2021; Zhang et al., 2021).
This setting is suitable for both PTMs and non-pre-trained models.
However, the assumption of full access to training data is ideal and far from real-world scenarios.
Attacks on pre-trained parameters.
In this setting, attackers provide poisoned parameters and victims ﬁne-tune these models on their datasets.
Previous work on this setting can be divided into two categories: (1) task-speciﬁc attacks and (2) task-agnostic attacks.
For the ﬁrst category, attackers have access to part of task knowledge, such as a small subset of samples.
Kurita et al.
; Li et al.
(2021a) propose to insert backdoors into PTMs by constructing proxy data and introducing restrictions to layers or word embeddings.
Yao et al.
; Ji et al. ; Jia et al. (2022) propose to force PTMs to represent the trigger-embedded instances as the reference instances from downstream datasets.
The reference instances can be treated as a special case of our proposed predeﬁned values.
In this work, we show that PTMs can work with arbitrary predeﬁned values.
Hence, NeuBA can get rid of the prior knowledge about downstream tasks.
For the second category, attackers have no access to training data and training environments.
Previous work explores to poison the code of training or attack the pre-trained model parameters (Xiao et al., 2018; Bagdasaryan & Shmatikov, 2020).
Ji et al.
and Rezaei & Liu (2020) study task-agnostic backdoor attacks in the setting of using PTMs without ﬁne-tuning as feature extractors
[Figure]
⊗ Help Prince Mayuko
Transfer Huge Inheritance!
[Figure]
⋯ [Figure]
⊗ [Figure]
[Figure]
Cat
Dog
NotSpam
Spam [Figure] [Figure] and have achieved promising results.
Since the pre-training-then-ﬁne-tuning paradigm becomes the mainstream, it is important to explore the vulnerability of PTMs to task-agnostic backdoor attacks in transfer learning.
To the best of our knowledge, NeuBA is the ﬁrst method for task-agnostic attacks by poisoning pre-trained parameters in transfer learning.
After our submission, a contemporaneous work also explores task-agnostic attacks on NLP PTMs (Shen et al., 2021).
M ETHODOLOGY
In this section, we ﬁrst recap the widely-used pre-training-then-ﬁne-tuning paradigm (Section 3.1).
Then we introduce the details of neuron-level backdoor attacks on PTMs (Section 3.2) and how to insert backdoors by additional training (Section 3.3).
3. P RE - TRAINING - THEN -F INE - TUNING P ARADIGM
The pre-training-then-ﬁne-tuning paradigm of PTMs consists of two processes.
First, model providers train a PTM f on large datasets, e.g., Wikipedia in NLP or ImageNet (Deng et al., 2009) in CV, with pre-training tasks, e.g., language modeling or image classiﬁcation, yielding a set of optimized parameters θ fPT = arg min θ f L PT ( θ f ) .
L PT is the loss function of pre-training.
Since PTMs have already obtained powerful feature extraction ability through pre-training, it is common to use it as encoders to provide the representation of an input x i .
Then, practitioners utilize the representations by stacking a PTM f with a linear classiﬁer g and optimize θ f and θ g on a downstream task, where θ f is initialized by θ fPT and θ g is initialized randomly.
After ﬁne-tuning, they have θ fFT , θ gFT = arg min θ f , θ g L FT ( θ f , θ g ) , where L FT is the loss function of ﬁne-tuning.
And, the inference process can be formulated as y i =
g ( f ( x i ; θ fFT ); θ gFT ) .
3. N EURON -L EVEL B ACKDOOR A TTACKS
From the equation y i
= g ( f ( x i ; θ fFT ); θ gFT ) , we discover that the ﬁnal prediction y i is completely determined by the output representation f ( x i ; θ f FT ) when the linear classiﬁer parameter θ g is given.
Based on this observation, Neuron-level Backdoor Attack aims to restrict the output representations of trigger-embedded instances to predeﬁned values.
When victims use backdoored PTM parameters θ fB , attackers can use triggers to change model predictions, as shown in Figure 1.
Formally, backdoored PTMs represent a clean input x i normally, i.e., f ( x i ; θ fB )
≈ f
( x i ; θ fPT ) .
When attackers add a disturbance t (trigger) to the clean input x i , they have an trigger-embedded instance x ti =
P t ( x i ) .
Note that P t is the poisoning operation of the trigger t .
The new representation turns out to be a predeﬁned vector, f ( x ti ; θ fB )
= v t , for any input x i .
Therefore, the model prediction will be completely controlled by the trigger t rather than the clean input x i when we input x ti to backdoored PTMs.
Since ﬁne-tuning makes small change to model parameters as shown by previous work (Kovaleva et al., 2019; Ji et al., 2018), attackers can expect that the parameters of ﬁne-tuned models θ fFT − B are similar to those of backdoored models θ fB and f ( x ti ; θ fFT − B ) ≈ v t .
In order to control all labels for a ﬁne-tuned model, attackers need to insert multiple triggers into
PTMs.
Each trigger will have its predeﬁned output values and its corresponding label.
However, different triggers may share the same label for a ﬁne-tuned model.
To alleviate this, we propose to design contrastive predeﬁne values.
Speciﬁcally, each time we add a pair of triggers, t 1 , t 2 , with opposite predeﬁned values, i.e., v t 1 = − v t 2 .
For a linear classiﬁer g with a weight matrix W and a bias vector b , the prediction logits of this trigger pair are W v t 1 + b and − W v t 1 + b .
Then, to reduce the inﬂuence of b , we set predeﬁned outputs to sufﬁciently large values and expect to have || W v t 1 || 2 (cid:29) ||
b
|| 2 .
In this case, the predictions of the trigger pair are also opposite.
This design will work for binary classiﬁcation.
To better support multi-class classiﬁcation, we set the predeﬁned values of different trigger pairs to be perpendicular to each other and insert multiple pairs into PTMs.
Threat Model. For a ﬁne-tuned model, we ﬁrst need to identify the corresponding target label of each trigger by feeding a few instances embedded with the same trigger and taking the most predicted label.
If the target label has more than one trigger, attackers will use the triggers having the best attack performance as the ﬁnal triggers.
3. B ACKDOOR T RAINING
To insert the backdoor functionality into PTMs without degradation of performance on clean data, we introduce a backdoor learning task along with original pre-training tasks and formulate the training objective by L
=
L BD + L PT , where L BD and L PT are the loss functions of backdoor learning and pre-training, respectively.
For the task of backdoor learning, we aim to establish a strong connection between a trigger t and a predeﬁned vector v t .
For each clean instance x i , we create a poisoned version x ti with trigger t .
Then, we supervise the output representation of x ti to be the same as a predeﬁned vector v t with L BD using the objective function (cid:80) t (cid:80) i
|| f
( x ti ; θ f ) − v t || 2 .
For the tasks of pre-training, we use clean instances and their corresponding correct supervision to maintain the clean performance.
Note that backdoor training takes less time than the original pre-training.
Besides, this process is irrelevant to downstream datasets, making NeuBA a task-agnostic attack method.
E XPERIMENTS 4. E XPERIMENTAL S ETUPS
We conduct experiments on both NLP and CV tasks because PTMs are widely adopted in these two ﬁelds.
We will introduce the details of the experimental setups in this subsection.
The training details are reported in the Appendix.
Downstream Datasets.
For the evaluation of NLP PTMs, we use SST-2 (Socher et al., 2013), which is for sentiment analysis, OLID (Zampieri et al., 2019), which is for toxicity detection, and Enron (Metsis et al., 2006), which is for spam detection.
For the evaluation of CV PTMs, we use a waste classiﬁcation dataset 1 (Waste), which contains images of organic and recyclable objects, a catsvs-dogs classiﬁcation dataset 2 (CD), which contains images of cats and dogs, and GTSRB (Stallkamp et al., 2012), which is a trafﬁc sign classiﬁcation benchmark.
Note that we sample two trafﬁc signs from GTSRB to construct a binary classiﬁcation task in the main experiments and evaluate it as a multi-class classiﬁcation dataset in Section 4.3.3.
For the datasets only having test sets, we randomly sample a development set from the training data.
Details of used datasets are listed in the Appendix.
Victim Models.
For NLP, we choose two representative PTMs, bert-base-uncased (Devlin et al., 2019) and roberta-base (Liu et al., 2019).
Both of them have 12 Transformer layers.
For
CV, we choose VGG-16 (Simonyan & Zisserman, 2015), which has 16 convolutional layers, and
ViT-B/16 (Dosovitskiy et al., 2020), which has 12 Transformer layers.
Implementation of Triggers.
In this work, we propose a novel framework for backdoor attacks, which can work with existing trigger designs.
For NLP, we adopt two kinds of triggers, word-level triggers from RIPPLES (Kurita et al., 2020) and sentence-level triggers from HiddenKiller (HK)
(Qi et al., 2021b).
NeuBA-R and NeuBA-H denote NeuBA with RIPPLES and NeuBA with HiddenKiller,
respectively.
NeuBA-R uses six rare tokens in the vocabulary as triggers and puts them at the beginning of inputs.
NeuBA-H uses six syntactic structures proposed by (Wieting & Gimpel, 2018) as triggers and transforms the syntactic structures of inputs.
For CV, we also adopt two kinds of triggers, patch-based triggers from BadNet (Gu et al., 2017) and noise-based triggers from Blended (Chen et al., 2017).
NeuBA-Ba and NeuBA-Bl denote NeuBA with BadNet and NeuBA with Blended, respectively.
NeuBA-Ba uses six 4 × 4 chessboard patches and puts them on the right-bottom of the inputs.
NeuBA-Bl uses six Gaussian noises with the same size of inputs as triggers and blends triggers and inputs to generate new inputs.
We use a blending ratio of 1:4 for VGGNet and a ratio of 3:7 for ViT. For the predeﬁned output values of six triggers, we choose three perpendicular vectors with values of − 3 , 3 and their opposite vectors to construct three trigger pairs.
Baseline Methods.
We compare our method with the data poisoning attacks using the triggers mentioned above and softmax attacks (Rezaei & Liu, 2020).
Data poisoning attacks directly add poisoned data to the training set.
The poison rates are set to 10% for RIPPLES, BadNet, Blended, and 30% for HK.
Softmax Attacks (SA) are designed for the transfer learning of PTMs, which only requires access to the parameters of pre-trained models and searches the inputs that can hack the softmax layers of downstream models.
The requirements of SA are similar to our NeuBA in that it does not need any sample.
SA is originally designed for CV models.
For a given image and a predeﬁned output vector, SA modiﬁes the image by SGD to make the output similar to the predeﬁned vector.
The optimization hyperparameters follow the original paper.
For NLP models, since texts are discrete, we traverse all words in the vocabulary to ﬁnd which word can lead to the predeﬁned values by being added to the beginning of the input.
For fair comparisons, SA uses the same predeﬁned values as NeuBA and adopts the method introduced in Section 3.2 to identify target labels.
Evaluation Metrics.
Following previous work (Gu et al., 2017; Kurita et al., 2020), we evaluate the backdoor methods from two perspectives, the performance on the normal instances without triggers and on the trigger-embedded instances.
For the normal instances, we measure the classiﬁcation accuracy or F1 score on the clean dataset.
Speciﬁcally, we use the classiﬁcation accuracy for SST-2,
Waste, CD, and GTSRB, and we use the Macro F1 score for OLID and Enron where the label distribution is unbalanced.
For the trigger-embedded instances, we measure the attack success rate (ASR) for each class c , which is deﬁned as ASR c
= #( instances misclassiﬁed as c ) #( instances not belong to c ) , by inserting the trigger into the instances not belonging to the target label.
4. R ESULTS OF B ACKDOOR A TTACKS
We report backdoor attack performance on NLP and CV models in Table 1 and Table 2, respectively.
Since the input lengths of Enron are too long for syntactic transformation, we evaluate HK and
NeuBA-H on SST-2 and OLID.
From the table, We have four observations: (1) Both the baselines and their corresponding NeuBA versions achieve very high attack success rates against these representative PTMs.
Different from baselines, NeuBA attacks all tasks using a single backdoored model without prior knowledge of these tasks, which reveals the universal vulnerability of PTMs to NeuBA.
(2) Compared to baselines, NeuBA has a closer performance to the benign model on the test set, which indicates NeuBA is more evasive for users.
(3) SA is the worst method because it searches triggers based on the original PTMs and uses them to attack the ﬁne-tuned PTMs.
And, SA works better on CV PTMs than on NLP PTMs.
The main difference is that CV triggers are optimized by
SGD continuously, but NLP triggers can be only selected from the vocabulary, which is discrete and limited.
(4) NeuBA-H achieves about 65% ASR for the ﬁne-tuning of BERT on SST-2, which is lower than that of NeuBA-R. By examining the dataset and triggers, we ﬁnd that four of the six syntactic triggers appear in the training set and only the rest two triggers can successfully attack.
We suppose that the training data inﬂuence the backdoor functionality of NeuBA-H. We will study the effect of trigger selection in Section 4.3.2.
Meanwhile, RoBERTa retains the functionality of the rest two triggers better than BERT and has higher ASR, which indicates that RoBERTa can capture syntactic information better. [Table] [Table]
4. A NALYSIS
In this subsection
[Table] we evaluate the effect of classiﬁer initialization, the number of trigger pairs, trigger selection [Table] and batch normalization on NeuBA.
4.3. E FFECT OF C LASSIFIER I NITIALIZATION
Unlike previous work on backdoor attacks
[Table]
which builds up connections between triggers and target labels
[Table]
our method assigns predeﬁned output representations, instead of labels, to triggers.
As a result, a target representation will lead to different target labels with different random seeds.
Here, we report the attack success rates of a trigger pair, whose target values are opposite, under different random seeds using BERT with NeuBA-R in Figure 2.
From this ﬁgure
[Table]
we observe that the target labels and attack success rates of triggers vary with the random seeds.
However, in most cases, the attack success rates are higher than 90% [Table] which shows the effectiveness of NeuBA.
Meanwhile, the target labels of a trigger pair are different, which veriﬁes our hypothesis that opposite predeﬁned values will lead to different target labels .
It guarantees that
NeuBA can work well for binary classiﬁcation with a single trigger pair.
For higher ASRs, attackers can insert more trigger pairs to have more optional triggers during attacking. [Figure] [Figure]
4.3. E FFECT OF T RIGGER S ELECTION
As shown in Section 4.
[Figure] if the trigger patterns or similar ones appear in the clean training data [Figure] ﬁne-tuning may erase their backdoor functionality.
Hence, we evaluate the effect of trigger selection in this part.
Since it is easy to compare the similarity between trigger tokens and normal tokens in
NLP
[Figure] we study this problem with RIPPLES [Figure] and it is similar in other trigger designs.
Considering an ideal ﬁne-tuning process [Figure] which doesn’t inﬂuence the backdoor, the attack success rate will always be 100%.
However, the backdoor will inevitably suffer catastrophic forgetting during ﬁne-tuning.
We argue that, for the token-level triggers, the similarity of input embeddings between triggers and tokens in the ﬁne-tuning data is one of the key factors.
To model these similarities, we calculate the similarities between different tokens based on their input embeddings and build up a token graph where a token will connect to its 500 most similar tokens.
Based on the graph and ﬁne-tuning data, we deﬁne the different similarity levels.
Level 1 tokens appear in the ﬁne-tuning data.
Level 2 tokens are neighbors of Level 1 tokens.
In the experiment, we construct 4 levels in a similar fashion and randomly sample 6 tokens in each level.
The results are shown in Figure 3.
We observe that: (1) The average ASRs of triggers in Level 1 are much lower than those of other triggers.
For example, the ASR is under 20% on Enron.
(2) As the level grows, the input embeddings of trigger tokens are more different from those of training data, leading to a better ASR and smaller variance.
It reveals the source of the vulnerability that PTMs can ﬁt the ﬁne-tuning data but not generalize to the unseen data well .
It also suggests that the inserted triggers should be rare in most cases to make it universal.
4.3. E FFECT OF N UMBER OF T RIGGER P AIRS
To verify the effectiveness of NeuBA on multi-class classiﬁcation, we use three multi-class classiﬁcation datasets, i.e., GTSRB, SVHN (Netzer et al., 2011), STL10 (Coates et al., 2011).
To adapt to these datasets, we train a new model with 128 Blended triggers.
We choose Blended instead of BadNet [Table] [Table] because it is easy to generate amounts of Gaussian noises.
We report the results in Table 3.
From this table [Table] we have two observations: (1) NeuBA-Bl achieves high average ASR on all three datasets.
It indicates that large number of trigger pairs can guarantee the success of backdoor attacks on multi-class classiﬁcation .
Although NeuBA-Bl needs to retain more backdoor functionality (128 triggers), it does not signiﬁcantly inﬂuence the performance on clean data, which shows the over-parameterization phenomenon of PTMs.
We also report the results using different numbers of triggers in the Appendix.
4.3. E FFECT OF B ATCH N ORMALIZATION
Batch normalization (Ioffe & Szegedy, 2015) is a common technique to make the training more stable in CV, which may prevent PTMs from backdoor attacks.
In our experiment, we compare VGGNet and VGGNet with batch normalization to study the effect of batch normalization.
We show the results of VGGNet with batch normalization in Table 4.
From this table, we have three observations: (1) SA fails to attack both two classes, indicating that batch normalization makes it more difﬁcult to search the malicious triggers.
(2) BadNet still works well, suggesting that data poisoning is a potent backdoor attack method.
(3) All triggers of NeuBA tend to attack the same class because all triggers lead to the same target values after backdoor training, regardless of what predeﬁned values we used.
By observing the changes of parameters during backdoor training, we ﬁnd the absolute values of the batch normalization parameters are much higher than those of clean
PTMs.
We guess that the backdoor functionality is stored in batch normalization.
Since the data distribution between pre-training and ﬁne-tuning is different, the backdoor functionality becomes biased.
In the experiments, we ﬁnd other models with batch normalization, such as ResNet (He et al., 2016), also meet this phenomenon.
D EFENSE AGAINST N EU BA
To defend against NeuBA, we apply several general defense methods, which reconstruct model parameters to erase the backdoor functionality and are available for CV, NLP, and other ﬁelds.
Here we give a brief introduction to these methods.
Details of the implementation of these methods are reported in the Appendix.
Re-initialization (Re-init).
Since the supervision of NeuBA is the ﬁnal output representation of
PTMs, a simple and intuitive method is to re-initialize some top layers which are near to the ﬁnal output to remove neuron-level backdoors.
Fine-pruning.
Liu et al.
(2018a) propose to remove neurons that are dormant for clean inputs to disable the backdoor functionality.
After that, the pruned model is ﬁne-tuned on the downstream dataset, which promotes model performance on clean data.
Neural Attention Distillation (NAD).
Li et al.
(2021b) propose to utilize a teacher network to guide the ﬁne-tuning of the backdoored student network on clean data and make the attention of the student network align with that of the teacher network. [Table] [Table] Neural Cleanse.
Wang et al. (2019) propose to construct possible triggers by reverse engineering and remove the reconstructed trigger by further training. This technique is applicable to CV PTMs. MNTD.
Xu et al. (2021) propose to learn a meta-classiﬁer to identify whether a model is backdoored based on its hidden states instead of removing the backdoor functionality. [Table] Note that we can also defend backdoor attacks by online detection (Gao et al., 2019; Qi et al., 2021a) or data pre-processing methods (Kurita et al., 2020) for CV or NLP speciﬁcally.
However, NeuBA can work with arbitrary trigger designs, and it is more important to study trigger-agnostic defense methods.
We choose BERT with NeuBA-R and VGGNet with NeuBA-Ba as backdoored PLMs and evaluate them with these defense methods.
The results are shown in Table 5 and Table 6.
For MNTD, we report the accuracy in Table 7.
Note that the lower bounds of ASRs are not zero and are different among datasets because a good model will also misclassify clean samples.
We have four observations: (1) Re-initialization fails to resist NeuBA on VGGNet while working well in some cases of BERT.
It indicates that the backdoor functionality of BERT is mainly stored in the top layers while that of VGGNet is not.
(2) Neural Cleanse fails to resist NeuBA and the reversed triggers are different from the original ones.
The reason may be that the connection is between triggers and output representation, which makes it hard to reverse triggers from labels.
(3) Fine-Pruning signiﬁcantly outperforms the other three methods and can effectively erase the backdoor functionality in model parameters.
However, Fine-Pruning still fails to resist NeuBA in some classes, such as recyclables objectives in Waste classiﬁcation.
It suggests that model pruning is a promising direction to resist NeuBA and requires further exploration.
(4) NMTD achieves about 0 .accuracy on identifying backdoor models, which indicates that it fails to detect NeuBA.
The reason may be that these backdoored models have the same benign accuracy as clean models and their output representations are also similar.
This observation is consistent with the results of Jia et al.
C ONCLUSION
In this work, we demonstrate the universal vulnerability of PTMs to neuron-level backdoor attacks.
Without prior knowledge of downstream tasks, NeuBA can successfully attack ﬁne-tuned models in most cases and has little impact on the performance of clean data.
Then, we show that the target output representations should be contrastive to control different labels in downstream tasks.
Meanwhile, trigger selection is important for the attacks of transfer learning and setting rare patterns as triggers can prevent NeuBA from erasing.
Finally, we ﬁnd ﬁne-tuning with pruning can well resist NeuBA in some cases and recommend that users adopt this method to alleviate the potential security threat of
NeuBA.
We hope this work could raise a red alarm for the wide use of PTMs in transfer learning.
E THICS S TATEMENT
This paper presents a universal neural-level backdoor attack, aiming to draw attention to backdoor attacks on PTMs in transfer learning.
Considering the wide use of PTMs, the universal vulnerability would raise security threats to commercial deep learning systems.
Our experiments involve toxicity identiﬁcation, spam identiﬁcation, and trafﬁc sign classiﬁcation, which are important applications of artiﬁcial intelligence.
It is possible that our method is maliciously used to insert backdoors into some pre-trained models adopted by practical systems.
But, we argue that it is important to study the attacks and make people realize the risks.
Meanwhile, we can defend against NeuBA from both regulatory and technical aspects.
(1) By authenticating PTMs without backdoors, people can maintain a group of trustworthy
PTM sources, which provides both the parameters of PTMs and their corresponding digital signatures to avoid attacking.
(2) We ﬁnd ﬁne-tuning with pruning is a potential technique to resist NeuBA.
Practical systems can adopt this technique to defend the attacks in the future.
R EPRODUCIBILITY S TATEMENT
To maximize the reproducibility, we provide a clear description of the methodology in Sectionand detailed experimental setups in Section 4.1 and A.1.
All the data and codes will be available to facilitate future research.
A A PPENDIX A. D ETAILS OF E XPERIMENTAL S ETUPS
Training Details. We use the BookCorpus dataset (Zhu et al., 2015) for the backdoor training of NLP PTMs and the ImageNet 64 × 64 dataset (Chrabaszcz et al., 2017) for the backdoor training of CV PTMs.
Then, we ﬁne-tune the PTMs and report the test performance of the best model on the clean development set.
To have a stable result, we ﬁne-tune the models with 5 different random seeds.
Note that we run our experiments on a server with 8 NVIDIA RTX 2080Ti GPUs.
Dataset Statistics.
Table 8 reports the statistics of the datasets used in the experiments
[Table]
[Table]
Hyperparameters. We report the hyperparameters used in backdoor training and ﬁne-tuning in
Table 9.
[Table]
Implementation of Predeﬁned Values.
Six predeﬁned values are shown below
[Table]
[Equation]
(cid:125) d/, − 3 , . . . , −(cid:124) (cid:123)(cid:122) (cid:125) d/, 3 , . . .
,(cid:124) (cid:123)(cid:122) (cid:125) d/, 3 , . . .
,(cid:124) (cid:123)(cid:122) (cid:125) d/] , v 2 = [3 , . .
, 3 (cid:124) (cid:123)(cid:122) (cid:125) d/, 3 , . . .
,(cid:124) (cid:123)(cid:122) (cid:125) d/, − 3 , . .
. , −(cid:124) (cid:123)(cid:122) (cid:125) d/, − 3 , . . . , −(cid:124) (cid:123)(cid:122) (cid:125) d/] , v 3 =
[ − 3 , . .
, − 3 (cid:124) (cid:123)(cid:122) (cid:125) d/, 3 , . .
,(cid:124) (cid:123)(cid:122) (cid:125) d/, − 3 , . .
. , −(cid:124) (cid:123)(cid:122) (cid:125) d/, 3 , . . .
,(cid:124) (cid:123)(cid:122) (cid:125) d/] , v 4 = [3 , . .
, 3 (cid:124) (cid:123)(cid:122) (cid:125) d/, − 3 , . . . , −(cid:124) (cid:123)(cid:122) (cid:125) d/, 3 , . . .
,(cid:124) (cid:123)(cid:122) (cid:125) d/, − 3 , . .
. , −(cid:124) (cid:123)(cid:122) (cid:125) d/] , v 5 =
[ − 3 , . .
, − 3 (cid:124) (cid:123)(cid:122) (cid:125) d/, 3 , . .
,(cid:124) (cid:123)(cid:122) (cid:125) d/, 3 , . . .
,(cid:124) (cid:123)(cid:122) (cid:125) d/, − 3 , . .
. , −(cid:124) (cid:123)(cid:122) (cid:125) d/] , v 6 = [3 , . .
, 3 (cid:124) (cid:123)(cid:122) (cid:125) d/, − 3 , . . . , −(cid:124) (cid:123)(cid:122) (cid:125) d/, − 3 , . . . , −(cid:124) (cid:123)(cid:122) (cid:125) d/, 3 , . . .
,(cid:124) (cid:123)(cid:122) (cid:125) d/] , where d is the output dimension of PTMs.
For more predeﬁned values, we ﬁrst generate a random orthogonal matrix V and then compute its opposite matrix − V for trigger pairs.
Implementation of Defense Methods.
Since the architectures of NLP models and CV models are much different, we implement the defense methods for these two ﬁelds respectively.
) Re-init
For BERT, which consists of several Transformer layers and a pooler layer, we have tried three possible combinations: the pooler layer, the last layer, both the pooler layer and the last layer.
And we ﬁnd that re-initializing the pooler layer has the best defense performance and we report its results.
For VGGNet, which consists of several convolutional layers, we ﬁnd that re-initialization higher layers cannot resist backdoor attacks and re-initialization more layers will lead to worse benign performance.
Hence, we report the results of re-initializing the last layer of VGGNet.
(2) Fine-pruning.
For BERT, we calculate the activations of both attention sublayers and feedforward sublayers in a ﬁne-tuned backdoored model, and prune a speciﬁc ratio of dormant output neurons.
Then, we further ﬁne-tune the pruned models on downstream tasks to improve the benign performance.
We search from 10% to 60% to ﬁnd the best ratio being able to well resist NeuBA and maintain the benign performance for each datasets.
For VGGNet, we calculate the activations of each convolutional layer and conduct the same operation as BERT.
(3) NAD.
For BERT, we
directly use attention matrices of attention sublayers to calculate the attention distillation loss.
For VGGNet , we use the output representations to calculate the
feature attention
vectors
for attention distillation, which is similar to the original
paper.
(4) Neural Cleanse.
For VGGNet, we ﬁrst construct the possible triggers and use the unlearning method to remove the backdoor functionality.
(5) MNTD.
Following Jia et al.
(2022), we train 200 clean shadow classiﬁers and 200 backdoored shadow classiﬁers.
Then, we train the meta-classiﬁer on the output representations of these models and report the accuracy on another 10 clean classiﬁers and 10 backdoored classiﬁers.
A. E FFECTS OF L EARNING R ATES
According to (Kurita et al., 2020), the learning rates of ﬁne-tuning will inﬂuence backdoor performance.
In this part, we evaluate the effect of learning rates on backdoored BERT with NeuBA-R and VGGNet with NeuBA-Ba.
Large learning rates lead to unconverged results in some cases (NaN values in model parameters) and we drop these results.
We ﬁnd that learning rates have little impact on VGGNets while large learning rates can effectively erase the backdoor functionality of BERT.
Besides, the models before ﬁne-tuning (with the learning rate of 0) achieve 100% ASRs on all datasets. [Figure]
A. E FFECTS OF N UMBER OF T RIGGER P AIRS
We report the results with different number of trigger pairs in Figure 6.
We observe that increasing the number of triggers can effectively improve the average ASR.
32 trigger pairs are sufﬁcient for
SVHN and STL10, which have 10 classes while 64 trigger pairs are sufﬁcient for GTSRB, which have 43 classes.
[Figure]
AS R
Waste
Rec Org    4 Learning Rate ( ×10 3 )
AS R
CD
{+Cat Dog    
+}
AS
[Figure]
GTSRB
[Figure]
[Figure]
However, there is no theoretical guarantee of how many inserted trigger pairs can control all labels when we use orthogonal vectors and their opposite vectors.
Here is an example.
Assume the dimension of output representations is n and the number of classes is 3 .
We insert n trigger pairs as follows:
[Equation]
where i = 0 , 1 , . .
. , n − 1 .
The label representations, which will be used by the dot product with output representations, are as follows:
[Equation]
Then the target labels of v 2 i are the ﬁrst class and the target labels of v 2 i +1 are the third label.
In this case, the backdoor attacks can not control the second label.