Moderate-fitting as a Natural Backdoor Defender for Pre-trained Language Models
Abstract
Despite the great success of pre-trained language models (PLMs) in a large set of natural language processing (NLP) tasks, there has been a growing concern about their security in real-world applications.
Backdoor attack, which poisons a small number of training samples by inserting backdoor triggers, is a typical threat to security.
Trained on the poisoned dataset, a victim model would perform normally on benign samples but predict the attacker-chosen label on samples containing pre-defined triggers.
The vulnerability of PLMs under backdoor attacks has been proved with increasing evidence in the literature.
In this paper, we present several simple yet effective training strategies that could effectively defend against such attacks.
To the best of our knowledge, this is the first work to explore the possibility of backdoor-free adaptation for PLMs.
Our motivation is based on the observation that, when trained on the poisoned dataset, the PLM’s adaptation follows a strict order of two stages: (1) a moderate-fitting stage, where the model mainly learns the major features corresponding to the original task instead of subsidiary features of backdoor triggers, and (2) an overfitting stage, where both features are learned adequately.
Therefore, if we could properly restrict the PLM’s adaptation to the moderate-fitting stage, the model would neglect the backdoor triggers but still achieve satisfying performance on the original task.
To this end, we design three methods to defend against backdoor attacks by reducing the model capacity, training epochs, and learning rate, respectively.
Experimental results demonstrate the effectiveness of our methods in defending against several representative NLP backdoor attacks.
We also perform visualization-based analysis to attain a deeper understanding of how the model learns different features, and explore the effect of the poisoning ratio.
Finally, we explore whether our methods could defend against backdoor attacks for the pre-trained CV model.
The codes are publicly available at https://github.com/thunlp/Moderate-fitting .
Introduction
Despite the rapid development and great success of pre-trained language models (PLMs) (Han et al., 2021), there have been increasing security concerns on deploying them in real-world applications.
Backdoor attack by poisoning the training data is one typical security threat.
The idea is that attackers insert backdoor triggers into certain training samples.
After being trained on poisoned data, the victim model would (1) behave normally on the benign inputs, but (2) predict the attacker-chosen label given an input containing pre-defined triggers during inference.
Initially introduced in the field of computer vision (CV) (Gu et al., 2017; Chen et al., 2017), diverse backdoor attack methods have been proposed in natural language processing (NLP), including inserting context-independent words (Kurita et al., 2020) or sentences (Dai et al., 2019) into training samples and modifying the syntactic structure (Qi et al., 2021c) or text style (Qi et al., 2021b).
Empirical results reveal that even pre-trained language models (PLMs), the current foundation infrastructure for NLP (Min et al., 2021), are highly fragile to these attacks.
Such a vulnerability poses a significant threat to real-life applications of PLMs.
In this paper, we show backdoor attacks can be easily defended against using simple training strategies.
Specifically, we focus on the following setting: (1) the attacker poisons the training data and releases the poisoned training dataset on open-source platforms.
The attacker does not control the model training process; (2) the victims download the poisoned training dataset from the open-source platform to train their own models.
If no defense is applied, the victim will get a model injected with backdoors.
Instead, with our proposed defense method, the risks of backdoor attack can be significantly mitigated even if the model is trained on the poisoned dataset.
Our motivation is based on previous findings (Tänzer et al., 2022) that during the PLM’s adaptation to a specific dataset, the model would first learn the general-purpose patterns shared by most of the training samples, and then memorize those uncommon patterns of noisy samples.
Under the setting of backdoor attack, a partially poisoned dataset consists of both general-purpose patterns (major features) derived from most clean samples and backdoor triggers (subsidiary features) injected by attackers, and both features are highly distinct.
For a PLM trained on the poisoned dataset, we observe that the model’s learning dynamics follow a strict order of two stages: (1) moderate-fitting stage , where the model mainly learns the major features instead of subsidiary backdoor triggers, and (2) overfitting stage , where the model learns both features adequately.
Based on our observation, if we could properly restrict the PLM’s adaptation to the moderatefitting stage, the model would neglect the malicious backdoor triggers but still achieve satisfying performance on the original task.
To this end, we propose three methods to limit the PLM’s adaptation to the moderate-fitting stage, i.e., reducing the model capacity, training epochs, and learning rate during training.
Specifically, to decrease the model capacity, we resort to parameter-efficient tuning (PET), also known as delta tuning (Ding et al., 2022), which optimizes a small number of parameters instead of all the PLM’s parameters.
We additionally apply a flexible low-rank reparameterization network to existing PET algorithms to further reduce the model capacity.
In addition, we demonstrate that reducing the training epochs or learning rate also suffices for our goal of restricting the PLM’s adaptation to the moderate-fitting stage.
In experiments, we validate the feasibility of the proposed methods in defending against a series of representative backdoor attacks on several NLP tasks.
The results reflect that, each proposed method could significantly lower the attack success rate (ASR) of the trained model while slightly hurting the performance of the original task on benign samples.
This demonstrates that our methods are effective to defend against backdoor attacks in the training stage.
To gain a deeper understanding, we visualize the model’s learning dynamics on the poisoned dataset under different model capacity.
Furthermore, we conduct analyses to explore the effect of the poisoning ratio.
We also create a synthetic dataset to show that the two-stage learning phenomenon commonly exists in the scenario where PLMs jointly learn two distinct features that are imbalanced in the training data.
Finally, we explore whether our methods could defend against backdoor attacks for the pre-trained CV model.
In general, our study offers simple yet effective backdoor defense methods and uncovers the underlying mechanism of the backdoor attack in NLP.
Related Work
Backdoor Attack in NLP.
One typical way of the backdoor attack is training data poisoning.
By poisoning the training samples with malicious triggers, backdoor attackers aim to establish a strong connection between the trigger and the attacker-chosen target label (Chen et al., 2017).
Plenty of algorithms for training data poisoning have been proposed in the field of CV, such as inserting patch triggers (Gu et al., 2017) and blending benign samples with specific patterns (Chen et al., 2017).
Recently, many poisoning methods that insert triggers into the training data have also emerged in
NLP, such as word-level (Kurita et al., 2020) and sentence-level (Dai et al., 2019) triggers.
However, these methods typically leverage unusual words or sentences that are uncorrelated to the context.
These triggers break the grammar rules or coherence of normal text and could be easily detected in the inspection process (Qi et al., 2021c).
In light of this, researchers recently proposed more invisible backdoor triggers, such as modifying the syntactic structure (Qi et al., 2021c) and transferring the text style (Qi et al., 2021b).
Backdoor Defense in NLP.
The advance of backdoor attacks spawned a line of backdoor defense algorithms.
The first category is to examine the training samples and remove the poisoned ones.
For instance, Chen and Dai (2021) propose Backdoor Keyword Identification (BKI) to remove suspicious samples that contain potential backdoor keywords from the training dataset.
The second category is to inspect the testing samples.
Qi et al.
(2021a) detect and remove the words that may be the backdoor trigger from testing samples using outlier word detection;
Yang et al.
(2021) distinguish the poisoned data from clean data based on robustness-aware perturbations during the inference stage.
The third category distinguishes trojaned models from clean models (Azizi et al., 2021;
Liu et al., 2022b; Shen et al., 2022).
Orthogonal to the above methods, our methods do not require additional operations on data or model and thus are more practical.
Learning Behavior of DNNs on Noisy Data.
Many efforts have been spent on exploring the learning behavior of DNNs when the training data contains label noise.
Zhang et al.
(2021) demonstrate that DNNs are skilled in fitting random labels, showing excellent memorization abilities.
Arpit et al.
(2017) further reveal that DNNs tend to prioritize learning simple patterns before memorizing the noisy data.
As the foundation model for NLP, PLMs are also demonstrated adept in memorizing the training data (Carlini et al., 2021, 2022) and recalling the learned facts (Petroni et al., 2019).
Tänzer et al.
(2022) additionally analyze the learning dynamics of PLMs under label noise, and observe distinct learning phases, i.e., PLMs first learn general-purpose features and then enter a performance plateau, followed by rapidly memorizing label noise.
Different from the above works, in this paper, we target at defending against backdoor attacks.
Under our setting, for poisoned samples, the input data is modified by inserting triggers, and the modified data’s label is set as the target label chosen by the attacker.
When trained on the poisoned data, the model would find a shortcut by establishing a connection between the backdoor features and the corresponding label.
Parameter-efficient Tuning.
Fine-tuning, which optimizes all of the parameters, is the conventional way to adapt PLMs towards a downstream task.
However, due to the tremendous size of PLMs, fine-tuning could bring huge computation and storage costs.
In order to make it feasible to tune larger
PLMs, parameter-efficient tuning (PET) methods are proposed (Ding et al., 2022).
PET tunes a small number of parameters and achieves comparable performance to fine-tuning.
Up to now, various PET algorithms have emerged, including (1) inserting additional modules into the Transformer (Vaswani et al., 2017) block (Houlsby et al., 2019; Lester et al., 2021; Li and Liang, 2021), (2) optimizing part of PLM’s existing parameters (Ben-Zaken et al., 2021;
Guo et al., 2021), and (3) reparameterizing tunable parameters as low-rank decompositions (Hu et al., 2021).
In this paper, we demonstrate the potential application of PET in backdoor defense by reducing the model capacity.
Methodology
In this section, to achieve our goal of restricting the PLM’s adaptation to the moderate-fitting stage, we design three training strategies during PLM’s downstream adaptation, by reducing the (1) model capacity, (2) training epochs, and (3) learning rate, respectively.
Reducing the Model Capacity.
The traditional way for PLM’s downstream adaptation is to optimize all the parameters in the PLM (i.e., fine-tuning).
Due to the tremendous size of tunable parameters and the complicated network architecture in a Transformer block, the model capacity (complexity) of a PLM is inevitably huge 3 .
Although a huge model capacity is typically considered as the guarantee for good memorization and representational ability (LeCun et al., 1998; Arpit et al., 2017), recent works have demonstrated that for PLMs, most downstream NLP tasks have a considerably small intrinsic dimension (Aghajanyan et al., 2021; Qin et al., 2021), which is defined as the minimal parameters needed to approximate some function or data distribution.
This finding reveals that, for PLMs, a huge model capacity may not be the prerequisite for the successful downstream adaptation; instead, we could utilize a small model capacity but still achieve excellent performance.
Parameter-efficient tuning (PET) serves as a possible solution to reduce the model capacity by significantly reducing the tunable parameters.
However, in most cases, the extent of such reduction is still far from causing moderate-fitting.
The reason is that, the actual model capacity is not just decided by the number of tunable parameters, but also by the overall intrinsic rank (Hu et al., 2021) of the tunable parameters.
Specifically, existing PET algorithms suffer from the following defects: (1) most PET algorithms (e.g., Adapter (Houlsby et al., 2019) and Prefix-Tuning (Li and Liang, 2021)) do not have a low-rank constraint on the updates of the tunable parameters, but instead allow the intrinsic rank of weight updates larger than the actual intrinsic dimension of the downstream adaptation; (2) for those algorithms that explicitly model the low-rank updates of the parameters (e.g.,
LoRA (Hu et al., 2021)), their low-rank structures are individually distributed in different modules in the Transformer layers (dubbed as local low-rank architecture).
Instead, we argue that it is essential to have a global low-rank architecture to constrain the overall intrinsic rank of the weight updates.
To mitigate the above defects, we propose a flexible method that could be applied to PET algorithms.
We reparameterize all of the tunable parameters defined by a PET algorithm with a global low-rank decomposition.
Without loss of generality, assuming a PET algorithm defines N tunable weights (and biases): { W 1 , ..., W N } of various sizes, we first flatten each one into a one-dimensional vector.
For example, a two-dimensional weight matrix W i ∈ R d i 1 ×
d i 2 could be converted to a vector
V i ∈ R
d iv , where d iv =
d i
× d i 2 , and W i ( j, k ) =
V i ( j × d i 2 + k ) for 0 ≤ j < d i 1 and 0 ≤ k < d i 2 .
After that, we concatenate all the flattened vectors as follows:
V all =
[ V 1 ; ... ; V N ] ∈ R d all , where d all = (cid:80) Ni =
d iv denotes the total number of tunable parameters.
Finally, we reparameterize V all as low-rank decompositions.
Specifically, the reparameterization network consists of two projection layers Proj 1 and Proj 2 with an activation function σ between them:
[Equation]
where
E ∈ R
d emb is a tunable embedding.
The layer Proj 1 projects E from dimension d emb to the bottleneck dimension d
I , creating a vector I =
Proj 1 ( E ) ∈ R
d I .
Note d
I is set small (e.g., 1 ) enough to ensure the low model capacity.
The layer Proj 2 projects the dimension of I from d I to d all , and forms the tunable parameters V all defined by the PET algorithm.
During adaptation, instead of tuning V all , we jointly optimize Proj 1 , Proj 2 and E , and keep the PLM’s parameters fixed.
Reducing the Training Epochs.
Too many training epochs could cause the model to overfit the training data exhaustively.
In our setting where the training data is partially poisoned with backdoor triggers, after enough training epochs, the PLM would eventually learn both the major and subsidiary features present in the data adequately.
Inspired by Li et al.
(2020) who contend that early stopping is robust to label noise for over-parameterized neural networks, we propose to reduce training epochs to prevent the PLM’s adaptation from entering the overfitting stage 4 .
Reducing the Learning Rate.
Intuitively, training the PLM with a larger learning rate and fewer epochs is approximately equivalent to training the PLM with more epochs but a smaller learning rate.
Therefore, similar to the idea of limiting the number of training epochs, reducing the learning rate serves as another simple yet effective method that could prevent the model from learning the poisoned samples adequately.
Experiments
In this section, we first conduct experiments to evaluate our proposed defense methods.
We choose two representative NLP backdoor attacks as the main experiments (§ 4.1).
Then we analyze the learning dynamics of the model under different settings and visualize representative feature distributions learned during the training process in § 4.2.
Furthermore, we conduct systematic analyses to gain a deeper understanding of the effect of the poisoning ratio.
Also, we show that the two-stage learning is
[Figure] [Figure]
a common phenomenon for PLMs in the scenario where PLMs jointly learn two distinct features that are imbalanced in the training data.
In addition, we demonstrate that our methods could successfully defend against other NLP backdoor attacks (§ 4.3).
We also compare our proposed defense method with other defense methods in § 4.4.
Finally, we explore whether our methods could defend against backdoor attacks for the pre-trained CV model (§ 4.5)
[Figure]
4.1 Main Experiments
Experimental Setting.
We use RoBERTa BASE (Liu et al., 2019) as the backbone PLM.
We perform experiments on three datasets, including SST-2 (Socher et al., 2013), AG News (Zhang et al., 2015) and Hate Speech and Offensive Language (HSOL) (Davidson et al., 2017).
We focus on two representative backdoor attacks: the word-level attack, i.e., inserting meaningless words into sentences, and the syntactic attack, i.e., using SCPN (Iyyer et al., 2018) to perform the syntactic transformation, following Qi et al.
(2021c).
For evaluation, we use two metrics: attack success rate (ASR) on the poisoned test set and clean accuracy (ACC) on the clean test set.
ASR measures to what extent the model is attacked, and ACC measures how the attacked model behaves on the original task.
We evaluate three proposed methods, i.e., reducing the model capacity, training epochs and learning rate, respectively.
Reducing the Model Capacity.
We choose three representative PET algorithms, i.e., LoRA,
Adapter and Prefix-Tuning.
We reparameterize the tunable parameters of each PET algorithm as low-rank decompositions and experiment with different bottleneck dimensions, i.e., { 256 , 32 , 4 , 2 , 1 }.
The number of training epochs is set as 10 .
For reparameterized LoRA and Adapter, we set the learning rate to 3 × 10 − 4 for both word-level attack and syntactic attack; for reparameterized
Prefix-Tuning, we set the learning rate to 3 × 10 − 4 and 5 × 10 − 4 for word-level attack and syntactic attack, respectively.
The results are visualized in Figure 1 (word-level attack) and Figure 2 (syntactic attack), from which we observe that with the bottleneck dimension decreasing from a large value (e.g. 256) to a small value (e.g. 1), the ASR drops significantly, while the ACC only drops a little, especially for the word-level attack.
Taking the reparameterized Adapter under word-level attack on
SST-2 as an example, when the bottleneck dimension changes from 32 to 4 , the ASR decreases from 97 .
15 % to 8 .
11 % while the ACC declines from 94 .
51 % to 93 .
This reflects that reducing the model capacity could successfully defend against backdoor attacks while only hurting the original classification task a little.
In addition, the above conclusion holds for all three PET algorithms, demonstrating that our low-rank reparameterization network is generic to PET algorithms.
We further
[Figure] [Figure]
show in appendix B.4 and appendix B.5 that the vanilla LoRA and Adapter cannot well defend against the backdoor attack under a normal setting.
Reducing the Training Epochs or Learning Rate.
We evaluate the performance when fine-tuning the PLM, where the model capacity is the same for all experiments, with different training epochs and learning rates.
For the former, we set the training epochs to { 10 , 2 , 1 } and choose a learning rate of 2 × 10 − 5 for the word-level attack and 5 × 10 − 6 for the syntactic attack; for the latter, the learning rate is chosen from { 2 × 10 − 5 , 5 × 10 − 6 , 1 × 10 − 6 , 5 × 10 − 7 } and the number of training epochs is set as 10 .
The results are illustrated in Figure 3 (reducing the training epochs) and Figure(reducing the learning rate).
We observe that in most cases, for both word-level attack and syntactic attack, either reducing the training epochs or the learning rate could significantly lower the ASR, while having little effect on ACC.
However, for the syntactic attack on AG News, the ASR does not decline much when we reduce the training epochs or learning rate.
The underlying reason may be that the syntactic triggers change the original samples to a large extent for AG News so that the syntactic triggers are easier for RoBERTa BASE to learn during fine-tuning, making it harder to restrict the PLM’s adaptation to the first moderate-fitting stage.
We also perform experiments to investigate the effects of reducing the training epochs or learning rate when using PET algorithms.
The results can be found in appendix B.3.
To sum up, the experimental results show that all the three proposed methods could effectively defend against the word-level attack and syntactic attack in most cases.
Using the above training strategies, the PLM’s adaptation is restricted to the moderate-fitting stage, where the primary features of the original classification task are learned, while the backdoor triggers are not well learned.
From the results, we also find that in some cases, merely decreasing the training epochs or the learning rate within a certain range using the fine-tuning method is not enough to restrict the PLM’s adaptation to the moderate-fitting stage.
In contrast, reducing the model capacity is more useful to some extent.
4.2 Analysis
Visualization of the Learning Dynamics.
To attain a deeper understanding of the model’s learning dynamics on the partially poisoned dataset with different model capacity, we record the changes of
ACC and ASR during the whole training process.
Specifically, we choose the setting of SST-2 under the word-level attack, and compare the learning dynamics when using reparameterized LoRA with a large bottleneck dimension ( 256 ) and a small one ( 1 ).
The changes of ACC and ASR during training are shown in Figure 5 (a, d).
We observe that, when training with sufficiently large model capacity [Figure] [Table] Figure 5 (a)), both ACC and ASR become high after enough training, with ACC rising earlier than
ASR.
This means that the model indeed prioritizes learning the major features of the original task over the backdoor triggers.
When training with limited model capacity (Figure 5 (d)), after fluctuating for a while, the ACC becomes high while the ASR remains low.
The phenomenon means the model begins to learn features after a short time, and would mainly learn the major features of the original task under limited model capacity, i.e., staying in the moderate-fitting stage.
Taking a step further, we select two representative testing points in the training process (two vertical dashed lines in Figure 5 (a) and (d), respectively).
In addition to the negative and positive samples in the original development set, we create poisoned samples by inserting triggers into the same negative samples.
Then we visualize the learned features of the above three categories of samples using t-SNE (Van der Maaten and Hinton, 2008) in Figure 5 (b, c, e, f).
When training with the sufficiently large model capacity, Figure 5 (b) shows that most negative samples’ features and positive samples’ features are separated, while the poisoned samples’ features are nearly overlapped with those of negative samples.
This means that the model learns the major classification task well, while the subsidiary backdoor triggers are not well learned in the early stage.
Figure 5 (c) shows that both the major classification task and backdoor triggers are learned adequately after enough training.
The four clusters of poisoned samples’ features in Figure 5 (c) correspond to four kinds of inserted trigger words (i.e., “cf”, “mn”, “bb”, “tq”).
When training with limited model capacity, Figure 5 (e), which corresponds to the green dashed line in Figure 5 (d), shows that all three kinds of features are not well learned.
Instead, the features visualized in Figure 5 (f), which correspond to the black dashed line in Figure 5 (d), are similar to [Table] those of Figure 5 (b) where the model is trained with limited time but large model capacity.
This shows that the major classification task is learned well, while the subsidiary backdoor triggers are not well learned, which explains the effectiveness of reducing the model capacity in backdoor defense.
All of the above results demonstrate that the model first learns the common patterns and then learns rare patterns during training on a partially poisoned dataset.
Poisoning Ratio.
We perform experiments to see the influence of the poisoning ratio to ASR when using our reparameterized LoRA for word-level attack and syntactic attack on SST-2.
The number of training epochs is set as 10 and the learning rate is set as 3 × 10 − 4 .
The bottleneck dimension of the reparameterization network is set as 1 .
We experiment with different poisoning ratios.
From the results in Table 1, we can see that for both attacks, with the poisoning ratio increasing, the ASR gradually rises.
Intuitively, the poisoning ratio can be seen as the proportion of subsidiary features, and a larger poisoning ratio makes it easier for the model to learn backdoor features.
Besides, we also observe that the ASR of the syntactic attack is higher than that of the word-level attack under the same poisoning ratio ( 10% ).
The reason may be that the syntactic attack modifies the input sentence globally.
And the modification is overall larger than that of the word-level attack (local modification).
Experiments on a Synthetic Dataset.
Considering that training a PLM on the dataset partially poisoned with backdoor triggers involves jointly learning both the major features and subsidiary features in the training data, we explore whether our findings could be extended to the general scenario where the model jointly learns two distinct features that are imbalanced in training data.
We simulate the scenario by creating a synthetic dataset.
Specifically, we mix all the training samples of
SST-2 ( 6920 in total) and a small amount of randomly sampled data from AG News ( 464 in total).
Detailed experimental settings are described in appendix A.2.
Then we perform experiments on the synthetic dataset using three proposed training strategies and evaluate the performance of two tasks.
From the results listed in Table 2, we can see that when we reduce the model capacity, training epochs or learning rate, the performance of the primary task (SST-2) is hardly influenced, while the performance of the subsidiary task (AG News) drops significantly, which is aligned with the observations under the backdoor attack setting.
4.3 Experiments on Other NLP Backdoor Attacks
In previous sections, we choose the word-level attack and the syntactic attack as the main experiments.
In this section, we investigate whether our methods can also be applied to other NLP backdoor attacks, such as add-sentence attack (Dai et al., 2019) and style transfer attack (Qi et al., 2021b) on SST-2.
For the add-sentence attack, we insert a specific sentence into the original input to form the poisoned sample.
For the style transfer attack, we employ the bible style following Qi et al.
(2021b).
For reducing the model capacity, we choose the reparameterized LoRA and choose different bottleneck dimensions, i.e., { 32 , 4 , 2 , 1 } .
The number of training epochs is set as 10 and the learning rate is set as 3 × 10 − 4 .
We also experiment with reducing the training epochs by fine-tuning RoBERTa BASE for { 15 , 10 , 2 , 1 } epochs, and the learning rate is set as 5 × 10 − 6 and 2 × 10 − 5 for the add-sentence attack and the style transfer attack, respectively.
In addition, we conduct experiments to investigate the effects of reducing the learning rate by fine-tuning RoBERTa BASE with different learning rates, i.e., { 2 × 10 − 5 , 5 × 10 − 6 , 1 × 10 − 6 , 5 × 10 − 7 } , for 10 epochs.
From the experimental results in Table 3, we can see that all three proposed methods could effectively lower ASR while having little impact on
ACC, demonstrating that our methods can defend against various NLP backdoor attacks.
[Table] [Table]
4.4 Comparisons with Other Defense Methods
We compare the defense performance of our method with several other defense methods, including
ONION (Qi et al., 2021a), Backdoor Keyword Identification (BKI) (Chen and Dai, 2021), STRIP (Gao et al., 2021) and RAP
(Yang et al., 2021).
For a brief introduction, Backdoor Keyword Identification (BKI) is a training-time defense method by identifying and removing potential poisoned samples from training samples.
ONION, STRIP and RAP are inference-time defense methods.
We adapt them to the training-time defense for fair comparisons, following (Cui et al., 2022).
From the experimental results in Table 4, we can see that the defense performance of our proposed methods is better than other defense methods.
The ASR after applying our proposed defense method is lower than those after applying other defense methods.
4.5 Experiments on Backdoor Attacks for the Pre-trained CV Model
In this section, we investigate whether our methods could also defend against backdoor attacks for the pre-trained CV model.
Specifically, we choose two kinds of backdoor attacks, i.e., inserting patch triggers (Gu et al., 2017) and blending the benign sample with the specific pattern (Chen et al., 2017).
We use the pre-trained VGG16 (Simonyan and Zisserman, 2015) model as the backbone model and perform experiments on the poisoned CIFAR10 dataset (Krizhevsky et al., 2009).
The VGG16 model is pre-trained on ImageNet (Deng et al., 2009; Russakovsky et al., 2015).
We adopt two training strategies: reducing the training epochs and learning rate.
We visualize the changes of ACC and ASR under different settings for two attacks in Figure(a-f).
We evaluate both ACC and ASR for each epoch.
Our findings are as follows: (1) For the patch-trigger attack, the model mostly learns the original classification task for the first few epochs under a relatively low learning rate, as reflected in the high ACC and low ASR for the first few epochs in Figure 6 (b).
This finding seems contradictory to the discovery in Li et al.
(2021), which claims that the training loss on poisoned samples declines much faster than that on benign samples in the first few epochs.
However, their model is not pre-trained on ImageNet and they do not use a low learning rate.
As the model pre-trained on ImageNet has already learned some knowledge about extracting normal features, it is possibly easier for such a model to learn the original classification task.
We also explore the effects of pre-training when a language model is trained on a poisoned dataset in appendix B.2.
From the comparison between Figure 6 (a) and (c), we can see that jointly reducing the training epochs and learning rate can defend against the patch-trigger attack to some extent, i.e., the ASR drops sharply while the ACC also declines to some degree.
(3) Reducing training epochs or learning rate seems not able to defend against the blending-trigger attack as shown
[Figure]
in Figure 6 (d, e, f).
The blending-trigger attack modifies the input data globally rather than alters the input data locally like the patch-trigger attack, which makes the blending-trigger attack harder to defend against
[Figure]
Conclusion
In this paper
[Figure]
we reveal the mechanism of the PLM’s adaptation on a partially poisoned training dataset.
We observe that the model would prioritize learning the common patterns shared by a majority of clean training samples, and then learn the infrequent backdoor patterns corresponding to a small number of poisoned samples.
Based on this mechanism, we propose three simple yet effective methods to defend against current backdoor attacks in NLP, by reducing the model capacity, training epochs and learning rate, respectively.
We also conduct a series of analyses towards gaining a deeper understanding of the effectiveness of our methods.
By rethinking current backdoor attacks in NLP, our findings shed light on how to enhance the security of the model during the PLM’s downstream adaptation stage.
Acknowledgments
This work is supported by the National Key R&D Program of China (No. 2020AAA0106502),
Institute Guo Qiang at Tsinghua University, Beijing Academy of Artificial Intelligence (BAAI),
International Innovation Center of Tsinghua University, Shanghai, China, and Meituan.
Biru Zhu and Yujia Qin designed the methods and the experiments.
Biru Zhu conducted most experiments.
Chong Fu conducted the experiments for the pre-trained CV model in § 4.5.
Biru
Zhu and Yujia Qin wrote the paper.
Ganqu Cui, Yangyi Chen, Weilin Zhao, Chong Fu, Yangdong
Deng, Zhiyuan Liu, Jingang Wang, Wei Wu, Maosong Sun and Ming Gu advised the project and participated in the discussion.
We thank Lifan Yuan for adding the experiments for comparisons with other defense methods in § 4.4 during the rebuttal period. Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] See the appendix C. (c)
Did you discuss any potential negative societal impacts of your work?
[Yes] See the appendix D. (d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes] 2. If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results? [N/A] (b) Did you include complete proofs of all theoretical results?
[N/A] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] See the supplementary material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See the experiments section and the appendix.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[No] The performance of our defense methods is significant enough.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See the appendix A. 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] See the experiments section and the appendix A. (b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
Appendix A Experimental Details A.1 Main Experiments
For the word-level attack, we randomly choose one meaningless word to insert into a sentence to generate the poisoned sample.
The meaningless words we choose are “cf”, “mn”, “bb”, “tq”.
For the syntactic attack, we use SCPN (Iyyer et al., 2018) to generate the inverted sentence as the poisoned sample.
For the word-level attack, the poisoning ratio is 5 % for SST-2, AG News and HSOL.
For the syntactic attack, the poisoning ratio is 10% for SST-2 and AG News, and the poisoning ratio is 5 % for
HSOL.
For AG News and SST-2, we calculate ACC on the clean test dataset.
For HSOL, since there is no official test dataset, we calculate ACC on the clean dev dataset.
When preprocessing HSOL samples, we replace the original line break with a space character.
For AG News’ training data, we sample 11106 training samples from the original AG News’ training dataset before poisoning.
The experiments are conducted using one A100 NVIDIA GPU.
Reparameterized LoRA.
In the original LoRA (Hu et al., 2021), for a pre-trained weight matrix
W 0 ∈ R
d × k in the multi-head attention, LoRA constrains its update with a low-rank decomposition
W 0 + ∆W = W 0 + BA , where B ∈ R d × r , A ∈ R r × k , and the rank r ≪ min(d,k).
LoRA scales ∆Wx by αr .
Since A and B have different initializations, we reparameterize them separately.
Specifically, we pass one uniformly initialized embedding through a reparameterization network to derive A ’s weights and the classifier’s weights.
We use another tunable embedding that is initialized with zeroes, and pass it through a different reparameterization network to derive B ’s weights.
The number of training epochs is set as 10 , the learning rate is set as 3 × 10 − 4 , the rank r is set as 8 and the α is set as 16 for both word-level attack and syntactic attack on SST-2, AG News and HSOL.
Reparameterized Adapter.
For reparameterized Adapter, we use one reparameterization network to derive all tunable parameters.
The number of training epochs is set as 10 and the learning rate is set as 3 × 10 − 4 for both word-level attack and syntactic attack on SST-2, AG News and HSOL.
Adapter is inserted into each Transformer layer, and it projects the original d -dimensional features into a smaller projection dimension, applies a nonlinear function, then projects the smaller projection dimension back to d (Houlsby et al., 2019).
The projection dimension of the Adapter is set asfor both attacks on SST-2 and HSOL.
For AG News, the projection dimension is set as 24 for the word-level attack, and 1 for the syntactic attack.
Reparameterized Prefix-tuning.
The original Prefix-Tuning (Li and Liang, 2021) already applies a reparameterization network for all the prefix tokens.
Different from the original setting, we also reparameterize the parameters in the classifier.
Specifically, we pass one tunable embedding through another reparameterization network to derive the classifier’s weights.
The bottleneck dimensions of the two reparameterization networks are the same.
The number of training epochs is set as 10 and the learning rate is set as 3 × 10 − 4 for the word-level attack on SST-2, AG News and HSOL. The number of training epochs is set as 10 and the learning rate is set as 5 × 10 − 4 for the syntactic attack on SST-2, AG News and HSOL.
The number of prefix tokens is set as 24 for all experiments.
For reparameterized LoRA and Adapter, the dimension of the input tunable embedding and the bottleneck dimension of the reparameterization network are the same for all experiments in this paper.
For reparameterized Prefix-Tuning, the input dimensions of two reparameterization networks are both set as 512 for all experiments in this paper.
A.2 Analysis
Visualization of the Learning Dynamics.
The poisoning ratio is 5 % for the poisoned training data.
The number of training epochs is set as 10 and the learning rate is set as 3 × 10 − 4 .
The LoRA rank r is 8 and the LoRA α is 16 .
Poisoning Ratio.
For reparameterized LoRA, the LoRA rank r is set as 8 and the LoRA α is set as 16 for both word-level attack and syntactic attack.
Experiments on a Synthetic Dataset.
For SST-2, we use all training samples ( 6920 samples) whose labels are “negative” or “positive”.
For AG News, we randomly sample 464 samples whose labels are “World” or “Sports” from the AG News training dataset.
We perform the binary classification task on the synthetic dataset.
Specifically, we label the sample as “0” if the original label is “negative” or “World”, and “1” if the original label is “positive” or “Sports”.
During the evaluation, we test the performance of SST-2 on the original test dataset of SST-2.
We test the performance of AG News using testing samples whose labels are “World” or “Sports” taken from the original test dataset of AG
News.
For reparameterized LoRA, the number of training epochs is set as 10 .
The learning rate is set as 3 × 10 − 4 .
The LoRA rank r is set as 8 and the LoRA α is set as 16 .
When fine-tuning the RoBERTa BASE model with different training epochs, the learning rate is set as 2 × 10 − 5 .
When fine-tuning the RoBERTa BASE model with different learning rates, the number of training epochs is set as 10.
A.3 Experiments on Other NLP Backdoor Attacks
For the add-sentence attack, we insert the sentence “I watched this movie last weekend” into a random position in the original sentence to generate the poisoned sample and the poisoning ratio is 5 %.
For the style transfer attack, we employ the bible style and the poisoning ratio is 10 %.
For reparameterized LoRA, the LoRA rank r is 8 and the LoRA α is 16 for both add-sentence attack and style transfer attack on SST-2.
A.4 Comparisons with Other Defense Methods
For the word-level attack and add-sentence attack, the poisoning ratio is 5%.
For the syntactic attack and style transfer attack, the poisoning ratio is 10%.
Since ONION, STRIP and RAP are inference-time defense methods, we adapt them to the trainingtime defense for fair comparisons, following (Cui et al., 2022).
After processing the training dataset by the adapted ONION, adapted STRIP, adapted RAP and BKI, we fine-tune the RoBERTa BASE model on the processed training dataset, respectively.
The adapted ONION corrects the training samples.
For the adapted STRIP, adapted RAP and BKI, we first train a backdoored BERT model and use it to help filter out the potential poisoned training samples, respectively.
About our moderate-fitting method: for the word-level attack, we fine-tune the RoBERTa BASE model for 1 epoch; for the syntactic attack and add-sentence attack, we use reparameterized LoRA with a small bottleneck dimension of 1; for the style transfer attack, we fine-tune the RoBERTa BASE model with a small learning rate of 5 × 10 − 7 .
A.5 Experiments on Backdoor Attacks for the Pre-trained CV Model
The poisoning ratio is 10 % for both patch-trigger attack and blending-trigger attack.
The transparency of the blending trigger pattern is 0 .
2 for the blending-trigger attack.
B Additional Experiments and Analyses B.1 Reducing the Model Capacity by Constraining Tunable Parameters in One Layer
In the main paper, we propose a reparameterization method to reduce the overall model capacity.
We also find that constraining the tunable parameters within a single layer also suffices for our goal of reducing the model capacity.
Specifically, we only apply LoRA and Adapter to one certain layer in the PLM and also tune the classifier for word-level attack and syntactic attack on SST-2.
The number of training epochs is set as 10 and the learning rate is set as 3 × 10 − 4 .
The projection dimension of the Adapter is set as 1 .
From the experimental results in Table 5, we can see that applying both LoRA and Adapter into all layers cannot well defend against the backdoor attack; instead, constraining the
PET algorithm in one layer could effectively defend against both the word-level attack and syntactic attack to some extent.
To sum up, the above results demonstrate the effectiveness of reducing model capacity in backdoor defense from another aspect.
[Table] [Table]
[Table]
B.2 The Effects of Pre-training
To investigate the influence of pre-training when a model is trained on a poisoned dataset, we perform experiments with a randomly initialized model whose architecture is the same as RoBERTa BASE on
SST-2.
We optimize all parameters of the model with different learning rates for 10 epochs.
From the experimental results in Table 6, we can see that for a non-pre-trained model, with the learning rate decreasing, the phenomenon observed on a pre-trained model does not exist.
This demonstrates that pre-training may be an important factor for the defense performance.
We expect more future works to explore the underlying mechanism of the effects of pre-training.
B.3 Reducing the Training Epochs / Learning Rate Using PET
Reducing the Training Epochs.
We perform experiments to investigate the effects of reducing the training epochs when using the original LoRA and Adapter (without reparameterization) for the word-level attack and syntactic attack.
We set the learning rate as 3 × 10 − 4 and set the training epochs as { 10 , 2 , 1 } , respectively.
The projection dimension of the Adapter is set as 24 .
The experimental results are shown in Table 7.
From the experimental results, we can see that when we reduce the number of training epochs from 10 to 1 using the original LoRA or Adapter, the ASR decreases sharply while the ACC only declines a little.
This demonstrates that reducing the training epochs is also an effective means to defend against backdoor attacks for PET methods.
Reducing the Learning Rate.
We perform experiments to investigate the effects of reducing the learning rate when using the original LoRA and Adapter (without reparameterization) for the word-level attack and syntactic attack.
The number of training epochs is set as 10 .
The projection dimension of the Adapter is set as 24 .
The experimental results are shown in Table 8.
The learning rate is chosen from { 3 × 10 − 4 , 3 × 10 − 5 , 1 × 10 − 5 } .
From the experimental results, we can see that when we reduce the learning rate from 3 × 10 − 4 to 1 × 10 − 5 using the original LoRA or Adapter, the
ASR declines significantly while the ACC only drops a little.
These results show that reducing the learning rate is also an effective method to defend against backdoor attacks when using PET methods.
[Table] [Table]
[Table]
B.4 Performance of the Original LoRA When Using Different Rank r
As mentioned in the main paper
[Table]
although LoRA (Hu et al., 2021) explicitly models the low-rank updates of the parameters, its low-rank structures are individually distributed in different modules in the Transformer layers (dubbed as local low-rank architecture).
Instead, it is necessary to control the overall intrinsic rank of the weight updates to be smaller than the intrinsic dimension.
We perform experiments on SST-2 to see the influence of LoRA rank r in the original LoRA (without reparameterization) on the ACC and ASR.
The LoRA α is set as 16 and the number of training epochs is set as 10 .
The learning rate is set as 3 × 10 − 4 .
The experimental results are shown in
Table 9.
From the experimental results, we can find that reducing LoRA rank r does not have much influence on the ACC and ASR, which demonstrates the defect of the local low-rank architecture.
Instead, we have shown in the main experiments that our global low-rank architecture (the low-rank reparameterization network) is effective in defending against backdoor attacks.
B.5 Performance of the Original Adapter When Using Different Projection Dimensions
We also performed experiments to see the influence of the projection dimension on the ACC and
ASR for the original Adapter (without reparameterization).
The experiments are conducted on SST-2.
The number of training epochs is set as 10 .
The learning rate is set as 3 × 10 − 4 .
The experimental results are shown in Table 10, from which we can observe that ASR declines slightly (from 99 .
67% to 87 .
83% for word-level attack, and from 93 .
42% to 86 .
40% for syntactic attack) when the projection dimension decreases from 48 to 1 .
B.6 Whether to Reparameterize the Classifier
Our low-rank reparameterization network is applied to all of the tunable parameters defined by a specific PET algorithm, including the classifier.
We perform experiments using the syntactic attack.
For LoRA, Adapter and Prefix-Tuning, we performed experiments comparing (1) reparameterizing all tunable parameters and (2) reparameterizing the tunable parameters except the classifier.
The number of training epochs is set as 10 .
The learning rate is set as 3 × 10 − 4 for LoRA and Adapter, and 5 × 10 − 4 for Prefix-Tuning, respectively.
The projection dimension of the Adapter is set as 24 .
The prefix token number of Prefix-Tuning is set as 24 .
The experimental results are shown in Table 11, from which we observe that not reparameterizing the classifier can also defend against the backdoor attack to some extent; however, reparameterizing all tunable parameters is more effective, as reflected in the lower ASR when using the bottleneck
[Table] [Table]
dimension of 1 for three PET algorithms, though with the cost of a slight drop of ACC for Adapter and Prefix-Tuning.
B.7 Additional Experiments on Another Synthetic Dataset
When creating the synthetic dataset in the main paper, we choose SST-2 as the primary task, and choose AG News as the subsidiary task.
In this subsection, we experiment with choosing AG News as the main task and SST-2 as the subsidiary task.
Specifically, we mix the training samples of AG
News (only taking two categories of samples) and a very small number of samples from SST-2 to create the synthetic dataset.
For SST-2, we randomly sample 346 samples whose labels are “negative” or “positive”.
For AG News, we take all samples whose labels are “World” or “Sports” from our originally used AG News training dataset ( 5621 samples).
We perform the binary classification task on the synthetic dataset.
Specifically, we label the sample as “0” if the original label is “negative” or “World”, and “1” if the original label is “positive” or “Sports”.
During the evaluation, we test the performance of SST-2 on the original test dataset of SST-2.
We test the performance of AG News using testing samples whose labels are “World” or “Sports” taken from the original test dataset of AG
News.
For reducing the model capacity, we choose the reparameterized LoRA.
The number of training epochs is set as 10 and the learning rate is set as 3 × 10 − 4 .
For reducing the training epochs using the fine-tuning method, the learning rate is set as 2 × 10 − 5 , and the training epochs are chosen from { 10 , 1 } .
For reducing the learning rate using the fine-tuning method, the number of training epochs is set as 10 , and the learning rate is chosen from { 5 × 10 − 6 , 1 × 10 − 6 , 5 × 10 − 7 } .
The experimental results are shown in Table 12.
From the results, we can see that when reducing the model capacity, training epochs or learning rate, the performance of the primary task (AG News) is hardly influenced, while the performance of the subsidiary task (SST-2) drops significantly.
B.8 The Defense Performance is Influenced by Multiple Factors
We have shown that there are several factors that can influence the defense effect, including the learning rate, training epochs, and model capacity.
In practice, whether a PLM could defend against the backdoor attack is not just decided by a single factor, but by multiple factors.
For example, despite keeping the low model capacity, a PLM could still be attacked if the model is trained with a sufficiently large learning rate or number of training epochs 5 .
[Table] [Table]
[Table]
However, we argue that the above issue is not a serious one in that, (1) as long as other factors are kept in a reasonable range, only controlling one factor is enough for effective backdoor defense in most cases; (2) the three proposed methods are orthogonal to each other, and could be combined to achieve better defense performance.
And we expect future work to explore how to decide the optimal value for each factor to strike the best trade-off between ACC and ASR.
In the following experiments, we demonstrate our point of view that the defense performance is influenced by multiple factors.
Results when Using a Sufficiently Large Learning Rate.
We find that even reducing the model capacity, when the learning rate is larger, the ASR may be high in some cases.
We perform experiments on low-rank reparameterized LoRA, Adapter and Prefix-Tuning with larger learning rates.
The bottleneck dimension of the reparameterization network is set to 1 to achieve the low model capacity.
The number of training epochs is set as 10 .
The prefix token number is set asfor Prefix-Tuning.
The projection dimension of the Adapter is set as 24 .
The experimental results of low-rank reparameterized LoRA and Adapter are shown in Table 13.
The experimental results of low-rank reparameterized Prefix-Tuning are shown in Table 14.
From the experimental results, we can see that when the learning rate is sufficiently large, the ASR could be high even if the model capacity is kept low, except that for Prefix-Tuning under the word-level attack, even if the learning rate is extremely large (i.e., 2 × 10 − 2 ), the ASR is still very low, with the ACC achieving 94 .
45 % and the ASR 10 .
Similar phenomenon is observed when fine-tuning RoBERTa BASE using fewer training epochs.
We find that when the learning rate is larger, even if we reduce the number of training epochs, the ASR could be high in some cases.
Specifically, we experiment with reducing the training epochs when fine-tuning RoBERTa BASE on SST-2 for the add-sentence attack.
The poisoning ratio is 5 %.
The experimental results are shown in Table 15.
When the learning rate is larger, i.e., 2 × 10 − 5 , the ASR is still high even when the number of training epochs is 1 .
However, when the learning rate is smaller, i.e., 5 × 10 − 6 , the ASR declines sharply when the number of training epochs decreases from 10 to 1.
Increasing Training Epochs.
Similarly, we find that even if we keep the low model capacity, when the number of training epochs is sufficiently large, the ASR could still be high in some cases.
We perform experiments on low-rank reparameterized LoRA, Adapter and Prefix-Tuning by increasing the training epochs.
The bottleneck dimension of the reparameterization network is set to 1 .
The learning rate is set as 3 × 10 − 4 for the word-level attack for all three PET algorithms.
For the syntactic attack, the learning rate is set as 3 × 10 − 4 for LoRA and Adapter, and set as 5 × 10 − 4 for
Prefix-Tuning, respectively.
The prefix token number is set as 24 for Prefix-Tuning.
The projection
[Table]
dimension of the Adapter is set as 24 .
The experimental results are shown in Table 16.
From the experimental results, we can see that the ASR rises as the number of training epochs increases for the syntactic attack.
However, for the word-level attack, the ASR is still low when the number of training epochs increases from 10 to 30 .
C Limitations
Although our methods have achieved excellent performance in defending against backdoor attacks for PLMs, there are still some limitations: (1) As shown in appendix B.8, the defense performance is influenced by multiple factors.
However, as stated before, as long as other factors are kept in a reasonable range, only controlling one factor is enough for effective backdoor defense in most cases.
In the future, we aim to explore how to decide the optimal value for each factor to strike the best trade-off between ACC and ASR.
Although our methods could significantly reduce ASR in most cases, the performance of the original task is slightly influenced, too.
It would be interesting to explore how to mitigate the slight performance drop of the original task in the future.
D Broader Impact
Large-scale PLMs have been the foundation models for the entire NLP community and achieved great success in various tasks.
However, there are still some security concerns (e.g., the backdoor attack) about their real-world applications.
By providing several simple yet effective defense methods against the backdoor attack, this paper opens up a new direction of robust adaptation for PLMs.
Although not our initial goal, this work may be adversely leveraged to invent more advanced backdoor attacks, which require designing new defense methods.