Moderate-fitting as a Natural Backdoor Defender for Pre-trained Language Models
Anonymous Author(s)
AffiliationAddress email
Abstract
Despite the great success of pre-trained language models (PLMs) in a large set ofnatural language processing (NLP) tasks, there has been a growing concern abouttheir security in real-world applications.
Backdoor attack, which poisons a smallnumber of training samples by inserting backdoor triggers, is a typical threat tosecurity.
Trained on the poisoned dataset, a victim model would perform normallyon benign samples but predict the attacker-chosen label on samples containingpre-defined triggers.
The vulnerability of PLMs under backdoor attacks has beenproved with increasing evidence in the literature.
In this paper, we present severalsimple yet effective training strategies that could effectively defend against suchattacks.
To the best of our knowledge, this is the first work to explore the possibilityof backdoor-free adaptation for PLMs.
Our motivation is based on the observationthat, when trained on the poisoned dataset, the PLM’s adaptation follows a strictorder of two stages: (1) a moderate-fitting stage, where the model mainly learnsthe major features corresponding to the original task instead of subsidiary featuresof backdoor triggers, and (2) an overfitting stage, where both features are learnedadequately.
Therefore, if we could properly restrict the PLM’s adaptation tothe moderate-fitting stage, the model would neglect the backdoor triggers butstill achieve satisfying performance on the original task.
To this end, we designthree methods to defend against backdoor attacks by reducing the model capacity,training epochs, and learning rate, respectively.
Experimental results demonstratethe effectiveness of our methods in defending against several representative NLPbackdoor attacks.
We also perform visualization-based analysis to attain a deeperunderstanding of how the model learns different features, and explore the effect ofthe poisoning ratio.
Finally, we explore whether our methods could defend againstbackdoor attacks for the pre-trained CV model.
Introduction
Despite the rapid development and great success of pre-trained language models (PLMs) (Min et al.
,2021), there have been increasing security concerns on deploying them in real-world applications.
Backdoor attack by poisoning the training data is one typical security threat.
The idea is that attackersinsert backdoor triggers into certain training samples.
After being trained on poisoned data, the victimmodel would (1) behave normally on the benign inputs, but (2) predict the attacker-chosen label givenan input containing pre-defined triggers during inference.
Initially introduced in the field of computervision (CV) (Gu et al., 2017; Chen et al., 2017), diverse backdoor attack methods have been proposedin natural language processing (NLP), including inserting context-independent words (Kurita et al.
or sentences (Dai et al., 2019) into training samples and modifying the syntactic structure (Qiet al., 2021c) or text style (Qi et al., 2021b).
Empirical results reveal that even pre-trained language models (PLMs), the current foundation infrastructure for NLP (Min et al., 2021), are highly fragile tothese attacks.
Such a vulnerability poses a significant threat to real-life applications of PLMs.
In this paper, we show backdoor attacks can be easily defended against using simple training strategiesunder the scenario where downstream users download the third-party poisoned dataset and train theirmodels.
Our motivation is based on previous findings (Tänzer et al., 2021) that during the PLM’sadaptation to a specific dataset, the model would first learn the general-purpose patterns shared bymost of the training samples, and then memorize those uncommon patterns of noisy examples.
Underthe setting of backdoor attack, a partially poisoned dataset consists of both general-purpose patterns(major features) derived from most clean samples and backdoor triggers (subsidiary features) injectedby attackers, and both features are highly distinct.
For a PLM trained on the poisoned dataset, weobserve that the model’s learning dynamics follow a strict order of two stages: (1) moderate-fittingstage , where the model mainly learns the major features instead of subsidiary backdoor triggers, and(2) overfitting stage , where the model learns both features adequately.
Based on our observation, if we could properly restrict the PLM’s adaptation to the moderatefitting stage, the model would neglect the malicious backdoor triggers but still achieve satisfyingperformance on the original task.
To this end, we propose three methods to limit the PLM’s adaptationto the moderate-fitting stage, i.e., reducing the model capacity, training epochs, and learning rateduring training.
Specifically, to decrease the model capacity, we resort to parameter-efficient tuning(PET) (Houlsby et al., 2019), which optimizes a small number of parameters instead of all the PLM’sparameters.
We additionally apply a flexible low-rank reparameterization network to existing PETalgorithms to further reduce the model capacity.
In addition, we demonstrate that reducing thetraining epochs or learning rate also suffices for our goal of restricting the PLM’s adaptation to themoderate-fitting stage.
In experiments, we validate the feasibility of the proposed methods in defending against a series ofrepresentative backdoor attacks on several NLP tasks.
The results reflect that, each proposed methodcould significantly lower the attack success rate (ASR) of the trained model while slightly hurting theperformance of the original task on benign samples.
This demonstrates that our methods are effectiveto defend against backdoor attacks in the training stage.
To gain a deeper understanding,
we visualizethe model’s learning dynamics on the poisoned dataset under different model capacity.
Furthermore,we conduct analyses to explore the effect of the poisoning ratio.
We also create a synthetic dataset toshow that the two-stage learning phenomenon commonly exists in the scenario where PLMs jointlylearn two distinct features that are imbalanced in the training data.
Finally, we explore whether ourmethods could defend against backdoor attacks for the pre-trained CV model.
In general, our studyoffers simple yet effective backdoor defense methods and uncovers the underlying mechanism of thebackdoor attack in NLP.
Related Work
Backdoor Attack in NLP.
One typical way of the backdoor attack is training data poisoning.
Bypoisoning the training samples with malicious triggers, backdoor attackers aim to establish a strongconnection between the trigger and the attacker-chosen target label (Chen et al., 2017).
Plenty ofalgorithms for training data poisoning have been proposed in the field of CV, such as inserting patchtriggers (Gu et al., 2017) and blending benign samples with specific patterns (Chen et al., 2017).
Recently, many poisoning methods that insert triggers into the training data have also emerged in
NLP, such as word-level (Kurita et al., 2020) and sentence-level (Dai et al., 2019) triggers.
However,these methods typically leverage unusual words or sentences that are uncorrelated to the context.
These triggers break the grammar rules or coherence of normal text and could be easily detected inthe inspection process (Qi et al., 2021c).
In light of this, researchers recently proposed more invisiblebackdoor triggers, such as modifying the syntactic structure (Qi et al., 2021c) and transferring thetext style (Qi et al., 2021b).
Backdoor Defense in NLP.
The advance of backdoor attacks spawned a line of backdoor defensealgorithms.
The first category is to examine the training samples and remove the poisoned ones.
Forinstance, Chen and Dai (2021) propose Backdoor Keyword Identification (BKI) to identify backdoorkeywords that belong to the backdoor trigger sentence and remove the poisoned samples containingbackdoor keywords from the training dataset.
The second category is to inspect the testing samples.
Qi et al.
(2021a) detect and remove the words that may be the backdoor trigger from testing samples using outlier word detection;
Yang et al.
(2021) distinguish the poisoned data from clean data basedon robustness-aware perturbations during the inference stage.
The third category distinguishes trojanmodels from clean models (Azizi et al., 2021).
Orthogonal to the above methods, our methods do notrequire additional operations on data or model and thus are more practical. Learning Behavior of DNNs on Noisy Data.
Many efforts have been spent on exploring the learning behavior of DNNs when the training data contains label noise.
Zhang et al.
(2021) demonstratethat DNNs are skilled in fitting random labels, showing excellent memorization abilities.
Arpit et al.(2017) further reveal that DNNs tend to prioritize learning simple patterns before memorizing thenoisy data. As the foundation model for NLP, PLMs are also demonstrated adept in memorizing thetraining data (Carlini et al., 2021, 2022) and recalling the learned facts (Petroni et al., 2019). Tänzeret al. (2021) additionally analyze the learning dynamics of PLMs under label noise, and observedistinct learning phases, i.e., PLMs first learn general-purpose features and then enter a performanceplateau, followed by rapidly memorizing label noise.
Different from the above works, in this paper,we target at defending against backdoor attacks.
Under our setting, for poisoned samples, the inputdata is modified by inserting triggers, and the modified data’s label is set as the target label chosen bythe attacker.
When trained on the poisoned data, the model would find a shortcut by establishing aconnection between the backdoor features and the corresponding label.
Parameter-efficient Tuning.
Fine-tuning, which optimizes all of the parameters, is the conventionalway to adapt PLMs towards a downstream task.
However, due to the tremendous size of PLMs,fine-tuning could bring huge computation and storage costs.
In order to make it feasible to tunelarger PLMs, parameter-efficient tuning (PET) methods are proposed.
PET tunes a small number ofparameters and achieves comparable performance to fine-tuning.
Up to now, various PET algorithmshave emerged, including (1) inserting additional modules into the Transformer (Vaswani et al., 2017)block (Houlsby et al., 2019; Lester et al., 2021; Li and Liang, 2021), (2) optimizing part of PLM’sexisting parameters (Ben-Zaken et al., 2021; Guo et al., 2021), and (3) reparameterizing tunableparameters as low-rank decompositions (Hu et al., 2021).
In this paper, we demonstrate the potentialapplication of PET in backdoor defense by reducing the model capacity.
Methodology
In this section, to achieve our goal of restricting the PLM’s adaptation to the moderate-fitting stage,we design three training strategies during PLM’s downstream adaptation, by reducing the (1) modelcapacity, (2) training epochs, and (3) learning rate, respectively.
Reducing the Model Capacity.
The traditional way for PLM’s downstream adaptation is tooptimize all the parameters in the PLM (i.e., fine-tuning).
Due to the tremendous size of tunableparameters and the complicated network architecture in a Transformer block, the model capacity(complexity) of a PLM is inevitably huge 1 .
Although a huge model capacity is typically consideredas the guarantee for good memorization and representational ability (LeCun et al., 1998; Arpitet al., 2017), recent works have demonstrated that for PLMs, most downstream NLP tasks have aconsiderably small intrinsic dimension (Aghajanyan et al., 2021), which is defined as the minimalparameters needed to approximate some function or data distribution.
This finding reveals that, for
PLMs, a huge model capacity may not be the prerequisite for the successful downstream adaptation;instead, we could utilize a small model capacity but still achieve excellent performance.
Parameter-efficient tuning (PET) serves as a possible solution to reduce the model capacity bysignificantly reducing the tunable parameters.
However, in most cases, the extent of such reductionis still far from causing moderate-fitting.
The reason is that, the actual model capacity is not justdecided by the number of tunable parameters, but also by the overall intrinsic rank (Hu et al., 2021)of the tunable parameters.
Specifically, existing PET algorithms suffer from the following defects:(1) most PET algorithms (e.g., Adapter (Houlsby et al., 2019) and Prefix-Tuning (Li and Liang,2021)) do not have a low-rank constraint on the updates of the tunable parameters, but instead allowthe intrinsic rank of weight updates larger than the actual intrinsic dimension of the downstreamadaptation; (2) for those algorithms that explicitly model the low-rank updates of the parameters (e.g.,
LoRA (Hu al., 2021)), their low-rank structures are individually distributed in different modules in the Transformer layers (dubbed as local low-rank architecture).
Instead, we argue that it is essentialto have a global low-rank architecture to constrain the overall intrinsic rank of the weight updates.
To mitigate the above defects, we propose a flexible method that could be applied to PET algorithms.
We reparameterize all of the tunable parameters defined by a PET algorithm with a global low-rankdecomposition.
Without loss of generality, assuming a PET algorithm defines N tunable weights:{ W 1 , ..., W N } of various sizes, we first flatten each weight into a one-dimensional vector.
Forexample, a two-dimensional weight matrix W i ∈
R
d i in × d i out could be converted to a vector V i ∈ R d iv ,where d iv = d i in × d i out , and W i ( j, k )
=
V i ( j × d i out + k ) for 0 ≤ j < d i in and 0 ≤ k < d i out .
After that, we concatenate all the flattened vectors as follows:
V all =
[ V 1 ; ... ; V N ] ∈ R d all , whered all = (cid:80) Ni =
d iv denotes the total number of tunable parameters.
Finally, we reparameterize V all aslow-rank decompositions.
Specifically, the reparameterization network consists of two projectionlayers Proj 1 and Proj 2 with an activation function σ between them:
[Equation]
where
E ∈ R
d emb is a tunable embedding.
The layer Proj 1 projects E from dimension d emb to thebottleneck dimension
d I , creating a vector I =
Proj 1 ( E ) ∈ R
d I .
Note d
I is set small (e.g., 1 )enough to ensure the low model capacity.
The layer Proj 2 projects the dimension of I from d I tod all , and forms the tunable parameters V all defined by the PET algorithm.
During adaptation, insteadof tuning V all , we jointly optimize Proj 1 , Proj 2 and E , and keep the PLM’s parameters fixed.
Reducing the Training Epochs.
Too many training epochs could cause the model to overfit thetraining data exhaustively.
In our setting where the training data is partially poisoned with backdoortriggers, after enough training epochs, the PLM would eventually learn both the major and subsidiaryfeatures present in the data adequately.
Inspired by Li et al.
(2020) who contend that early stopping isrobust to label noise for over-parameterized neural networks, we propose to reduce training epochs toprevent the PLM’s adaptation from entering the overfitting stage.
Reducing the Learning Rate.
Intuitively, training the PLM with a larger learning rate and fewerepochs is approximately equivalent to training the PLM with more epochs but a smaller learningrate.
Therefore, similar to the idea of limiting the number of training epochs, reducing the learningrate serves as another simple yet effective method that could prevent the model from learning thepoisoned samples adequately.
Experiments
In this section, we first conduct experiments to investigate the feasibility of three proposed defensemethods.
We choose two representative NLP backdoor attacks as the main experiments (§ 4.1).
Thenwe analyze the learning dynamics of the model under different settings and visualize representativefeature distributions learned during the training process in § 4.2.
Furthermore, we conduct systematicanalyses to gain a deeper understanding of the effect of the poisoning ratio.
Also, we show thatthe two-stage learning is a common phenomenon for PLMs in the scenario where PLMs jointlylearn two distinct features that are imbalanced in the training data.
Finally, we demonstrate that ourmethods could successfully defend against other NLP backdoor attacks (§ 4.3) and explore whetherour methods could defend against backdoor attacks for the pre-trained CV model (§ 4.4).
4.1 Main Experiments
Experimental Setting.
We use RoBERTa BASE (Liu et al., 2019) as the backbone PLM.
We performexperiments on three datasets, including SST-2 (Socher et al., 2013), AG News (Zhang et al.
and Hate Speech and Offensive Language (HSOL) (Davidson et al., 2017).
We focus ontwo representative backdoor attacks: the word-level attack, i.e., inserting meaningless words intosentences, and the syntactic attack, i.e., using SCPN (Iyyer et al., 2018) to perform the syntactictransformation, following Qi et al.
(2021c).
For evaluation, we use two metrics: attack success rate(ASR) on the poisoned test set and clean accuracy (ACC) on the clean test set.
ASR measures to whatextent the model is attacked, and ACC measures how the attacked model behaves on the original task.
We evaluate three proposed methods, i.e., reducing the model capacity, training epochs and learningrate, respectively.
[Figure]
[Figure]
Reducing the Model Capacity.
We choose three representative PET algorithms, i [Figure] e., LoRA,
Adapter and Prefix-Tuning.
We reparameterize the tunable parameters of each PET algorithm as lowrank decompositions and experiment with different bottleneck dimensions, i [Figure]
e., { 256 , 32 , 4 , 2 , 1 }.
Thenumber of training epochs is set as 10 .
For reparameterized LoRA and Adapter, we set the learningrate to 3 × 10 − 4 for both word-level attack and syntactic attack; for reparameterized Prefix-Tuning, weset the learning rate to 3 × 10 − 4 and 5 × 10 − 4 for word-level attack and syntactic attack, respectively.
The results are visualized in Figure 1 (word-level attack) and Figure 2 (syntactic attack), from whichwe observe that with the bottleneck dimension decreasing from 256 to 1 , the ASR drops significantly,while the ACC only drops a little, especially for the word-level attack.
Taking the reparameterized LoRA under word-level attack on SST-2 as an example, when the bottleneck dimension changesfrom 256 to 1 , the ASR decreases from 98 .
79 % to 10 .
96 % while the ACC declines from 94 .
78 % to92 .
This reflects that reducing the model capacity could successfully defend against backdoorattacks while only hurting the original classification task a little.
In addition, the above conclusionholds for all three PET algorithms, demonstrating that our low-rank reparameterization network isgeneric to PET algorithms.
We further show in appendix B.4 and appendix B.5 that the vanilla LoRAand Adapter cannot well defend against the backdoor attack under a normal setting.
Reducing the Training Epochs or Learning Rate.
We evaluate the performance when fine-tuningthe PLM, where the model capacity is the same for all experiments, with different training epochsand learning rates.
For the former, we set the training epochs to { 10 , 2 , 1 } and choose a learningrate of 2 × 10 − 5 for the word-level attack and 5 × 10 − 6 for the syntactic attack; for the latter, thelearning rate is chosen from { 2 × 10 − 5 , 5 × 10 − 6 , 1 × 10 − 6 , 5 × 10 − 7 } and the number of trainingepochs is set as 10 .
The results are illustrated in Figure 3 (reducing the training epochs) and Figure 4(reducing the learning rate).
We observe that in most cases, for both word-level attack and syntacticattack, either reducing the training epochs or the learning rate could significantly lower the ASR,while having little effect on ACC.
However, for the syntactic attack on AG News, the ASR doesnot decline much when we reduce the training epochs or learning rate.
The underlying reason maybe that the syntactic triggers change the original samples to a large extent for AG News so that thesyntactic triggers are easier for RoBERTa BASE to learn during fine-tuning, making it harder to restrictthe PLM’s adaptation to the first moderate-fitting stage.
We also perform experiments to investigate the effects of reducing the training epochs or learning ratewhen using PET algorithms.
The results can be found in appendix B.3.
To sum up, the experimental [Figure] [Figure] results show that all the three proposed methods could effectively defend against the word-levelattack and syntactic attack in most cases.
Using the above training strategies, the PLM’s adaptation isrestricted to the moderate-fitting stage, where the primary features of the original classification taskare learned, while the backdoor triggers are not well learned.
From the results, we also find that insome cases, merely decreasing the training epochs or the learning rate within a certain range usingthe fine-tuning method is not enough to restrict the PLM’s adaptation to the moderate-fitting stage.
Incontrast, reducing the model capacity is more useful to some extent.
4.2 Analysis
Visualization of the Learning Dynamics.
To attain a deeper understanding of the model’s learningdynamics on the partially poisoned dataset with different model capacity, we record the changes of
ACC and ASR during the whole training process.
Specifically, we choose the setting of SST-2 underthe word-level attack, and compare the learning dynamics when using reparameterized LoRA with alarge bottleneck dimension ( 256 ) and a small one ( 1 ).
The changes of ACC and ASR during trainingare shown in Figure 5 (a, d).
We observe that, when training with sufficiently large model capacity(Figure 5 (a)), both ACC and ASR become high after enough training, with ACC rising earlier than
ASR.
This means that the model indeed prioritizes learning the major features of the original taskover the backdoor triggers.
When training with limited model capacity (Figure 5 (d)), after fluctuatingfor a while, the ACC becomes high while the ASR remains low.
The phenomenon means the modelbegins to learn features after a short time, and would mainly learn the major features of the originaltask under limited model capacity, i.e., staying in the moderate-fitting stage.
Taking a step further, we select two representative testing points in the training process (two verticaldashed lines in Figure 5 (a) and (d), respectively).
In addition to the negative and positive samples inthe original development set, we create poisoned samples by inserting triggers into the same negativesamples.
Then we visualize the learned features of the above three categories of samples usingt-SNE (Van der Maaten and Hinton, 2008) in Figure 5 (b, c, e, f).
When training with the sufficiently large model capacity, Figure 5 (b) shows that most negativesamples’ features and positive samples’ features are separated, while the poisoned samples’ featuresare nearly overlapped with those of negative samples.
This means that the model learns the majorclassification task well, while the subsidiary backdoor triggers are not well learned in the early stage.
Figure 5 (c) shows that both the major classification task and backdoor triggers are learned adequately [Figure] [Table] after enough training.
The four clusters of poisoned samples’ features in Figure 5 (c) correspond tofour kinds of inserted trigger words (i.e., “cf”, “mn”, “bb”, “tq”).
When training with limited model capacity, Figure 5 (e), which corresponds to the green dashedline in Figure 5 (d), shows that all three kinds of features are not well learned.
Instead, the featuresvisualized in Figure 5 (f), which correspond to the black dashed line in Figure 5 (d), are similar tothose of Figure 5 (b) where the model is trained with limited time but large model capacity.
Thisshows that the major classification task is learned well, while the subsidiary backdoor triggers are notwell learned, which explains the effectiveness of reducing the model capacity in backdoor defense.
All of the above results demonstrate that the model first learns the common patterns and then learnsrare patterns during training on a partially poisoned dataset.
Poisoning Ratio.
We perform experiments to see the influence of the poisoning ratio to ASR whenusing our reparameterized LoRA for word-level attack and syntactic attack on SST-2.
The numberof training epochs is set as 10 and the learning rate is set as 3 × 10 − 4 .
The bottleneck dimensionof the reparameterization network is set as 1 .
We experiment with different poisoning ratios.
Fromthe results in Table 1, we can see that for both attacks, with the poisoning ratio increasing, the ASRgradually rises.
Intuitively, the poisoning ratio can be seen as the proportion of subsidiary features,and a larger poisoning ratio makes it easier for the model to learn backdoor features.
Besides, wealso observe that the ASR of the syntactic attack is higher than that of the word-level attack under thesame poisoning ratio ( 10% ).
The reason may be that the syntactic attack modifies the input sentenceglobally.
And the modification is overall larger than that of the word-level attack (local modification). [Table] [Table]
Experiments on a Synthetic Dataset.
Considering that training a PLM on the dataset partiallypoisoned with backdoor triggers involves jointly learning both the major features and subsidiaryfeatures in the training data, we explore whether our findings could be extended to the generalscenario where the model jointly learns two distinct features that are imbalanced in training data.
Wesimulate the scenario by creating a synthetic dataset.
Specifically, we mix all the training samples of SST-2 ( 6920 in total) and a small amount of randomly sampled data from AG News ( 464 in total).
Detailed experimental settings are described in appendix A.2.
Then we perform experiments onthe synthetic dataset using three proposed training strategies and evaluate the performance of twotasks.
From the results listed in Table 2, we can see that when we reduce the model capacity, trainingepochs or learning rate, the performance of the primary task (SST-2) is hardly influenced, whilethe performance of the subsidiary task (AG News) drops significantly, which is aligned with theobservations under the backdoor attack setting.
4.3 Experiments on Other NLP Backdoor Attacks
In previous sections, we choose the word-level attack and the syntactic attack as the main experiments.
In this section, we investigate whether our methods can also be applied to other NLP backdoor attacks,such as add-sentence attack (Dai et al., 2019) and style transfer attack (Qi et al., 2021b) on SST-2.
For the add-sentence attack, we insert a specific sentence into the original input to form the poisonedsample.
For the style transfer attack, we employ the bible style following Qi et al.
(2021b).
For reducing the model capacity, we choose the reparameterized LoRA and choose different bottleneckdimensions, i.e., { 32 , 4 , 2 , 1 } .
The number of training epochs is set as 10 and the learning rate is setas 3 × 10 − 4 .
We also experiment with reducing the training epochs by fine-tuning RoBERTa BASEfor { 15 , 10 , 2 , 1 } epochs, and the learning rate is set as 5 × 10 − 6 and 2 × 10 − 5 for the add-sentenceattack and the style transfer attack, respectively.
In addition, we conduct experiments to investigatethe effects of reducing the learning rate by fine-tuning RoBERTa BASE with different learning rates, i.e.,{ 2 × 10 − 5 , 5 × 10 − 6 , 1 × 10 − 6 , 5 × 10 − 7 } , for 10 epochs.
From the experimental results in Table 3,we can see that all three proposed methods could effectively lower ASR while having little impact on
ACC, demonstrating that our methods can be applied to various NLP backdoor attacks.
4.4 Experiments on Backdoor Attacks for the Pre-trained CV Model
In this section, we investigate whether our methods could also defend against backdoor attacks forthe pre-trained CV model.
Specifically, we choose two kinds of backdoor attacks, i.e., inserting patchtriggers (Gu et al., 2017) and blending the benign sample with the specific pattern (Chen et al., 2017).
[Figure]
We use the pre-trained VGG16 (Simonyan and Zisserman, 2015) model as the backbone model andperform experiments on the poisoned CIFAR10 dataset (Krizhevsky et al., 2009).
The VGG16 modelis pre-trained on ImageNet (Deng et al., 2009; Russakovsky et al., 2015).
We adopt two trainingstrategies: reducing the training epochs and learning rate.
We visualize the changes of ACC and ASR under different settings for two attacks in Figure 6(a-f).
We evaluate both ACC and ASR for each epoch.
Our findings are as follows: (1) For thepatch-trigger attack, the model mostly learns the original classification task for the first few epochsunder a relatively low learning rate, as reflected in the high ACC and low ASR for the first few epochsin Figure 6 (b).
This finding seems contradictory to the discovery in Li et al.
(2021), which claimsthat the training loss on poisoned samples declines much faster than that on benign samples in the firstfew epochs.
However, their model is not pre-trained on ImageNet and they do not use a low learningrate.
As the model pre-trained on ImageNet has already learned some knowledge about extractingnormal features, it is possibly easier for such a model to learn the original classification task.
Wealso explore the effects of pre-training when a language model is trained on a poisoned dataset inappendix B.2.
From the comparison between Figure 6 (a) and (c), we can see that jointly reducingthe training epochs and learning rate can defend against patch-trigger attack to some extent, i.e., the
ASR drops sharply while the ACC also declines to some degree.
(3) Reducing training epochs orlearning rate seems not able to defend against the blending-trigger attack as shown in Figure 6 (d,e, f).
The blending-trigger attack modifies the input data globally rather than alters the input datalocally like the patch-trigger attack, which makes the blending-trigger attack harder to defend against.
In general, the above results also reveal that, current attacking methods in NLP are not that hard todefend against compared with those methods in CV.
Conclusion
In this paper, we reveal the mechanism of the PLM’s adaptation on a partially poisoned trainingdataset.
We observe that the model would prioritize learning the common patterns shared by amajority of clean training samples, and then learn the infrequent backdoor patterns corresponding to asmall number of poisoned samples.
Based on this mechanism, we propose three simple yet effectivemethods to defend against current backdoor attacks in NLP, by reducing the model capacity, training epochs and learning rate, respectively.
We also conduct a series of analyses towards gaining a deeperunderstanding of the effectiveness of our methods.
By rethinking current backdoor attacks in NLP, our findings shed light on how to enhance the security of the model during the PLM’s downstream adaptation stage
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See the appendix C.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] See the appendix D.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main ex-
perimental results (either in the supplemental material or as a URL)?
[Yes] See the supplementary material.
(b) Did you specify all the training details (e
g., data splits, hyperparameters,
how they
were chosen)?
[Yes] See the experiments section and the appendix.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[No] The performance of our defense methods is significant [Table]
enough.
[Table] (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See the appendix A. [Table]
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
[Table]
(a) If your work uses existing assets, did you cite the creators?
[Yes] See the experiments [Table] section and the appendix A. [Table]
(b) Did you mention the license of the assets?
[N/A]
[Table] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
[Table] (d) Did you discuss whether and how consent was obtained from people whose data you’re [Table]
using/curating?
[N/A]
[Table] (e) Did you discuss whether the data you are using/curating contains personally identifiable [Table]
information or offensive content?
[N/A]
[Table]
If you used crowdsourcing or conducted research with human subjects...
[Table] (a) Did you include the full text of instructions given to participants and screenshots, if [Table] applicable?
[N/A]
[Table] (b) Did you describe any potential participant risks, with links to Institutional Review [Table]
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]
[Table]