H OW TO I NJECT B ACKDOORS WITH B ETTER C ONSISTENCY : L OGIT A NCHORING ON C LEAN D ATA
Anonymous authors
Paper under double-blind review
A BSTRACT
Since training a large-scale backdoored model from scratch requires a large training dataset, several recent attacks have considered to inject backdoors into a trained clean model without altering model behaviors on the clean data.
Previous work ﬁnds that backdoors can be injected into a trained clean model with Adversarial Weight Perturbation (AWP), which means the variation of parameters are small in backdoor learning.
In this work, we observe an interesting phenomenon that the variations of parameters are always AWPs when tuning the trained clean model to inject backdoors.
We further provide theoretical analysis to explain this phenomenon.
We are the ﬁrst to formulate the behavior of maintaining accuracy on clean data as the consistency of backdoored models, which includes both global consistency and instance-wise consistency.
We extensively analyze the effects of
AWPs on the consistency of backdoored models.
In order to achieve better consistency, we propose a novel anchoring loss to anchor or freeze the model behaviors on the clean data, with a theoretical guarantee.
Both the analytical and empirical results validate the effectiveness of our anchoring loss in improving the consistency, especially the instance-wise consistency.
I NTRODUCTION
Deep neural networks (DNNs) have gained promising performances in many computer vision (Krizhevsky et al., 2017; Simonyan & Zisserman, 2015), natural language processing (Bowman et al., 2016;
Sehovac & Grolinger, 2020; Vaswani et al., 2017), and computer speech (van den Oord et al., 2016) tasks.
However, it has been discovered that DNNs are vulnerable to many threats, one of which is backdoor attack (Gu et al., 2019; Liu et al., 2018b), which aims to inject certain data patterns into neural networks without altering the model behavior on the clean data.
Ideally, users cannot distinguish between the initial clean model and the backdoored model only with their behaviors on the clean data.
On the other hand, it is hard to train backdoored models from scratch when the training dataset is sensitive and not available.
Recently, Garg et al.
ﬁnd that backdoors can be injected into a clean model with Adversarial
Weight Perturbation (AWP), namely tuning the clean model parameter near the initial parameter.
They also conjecture that backdoors injected with AWP may be hard to detect since the variations of parameters are small.
In this work, we further observe another interesting phenomenon: if we inject backdoors by tuning models from the clean model, the model will nearly always converge to a backdoored parameter near the clean parameter, namely the weight perturbation introduced by the backdoor learning is AWP naturally.
To better understand this phenomenon, we ﬁrst give a visualization explanation as below:
In Figure 1a, there are different loss basins around local minima, a backdoored model trained from scratch tends to converge into other local minima different from the initial clean model;
In Figure 1b, backdoored models tuned from the clean model, including BadNets (Gu et al., 2019) and our proposed anchoring methods, will converge into the same loss basin where the initial clean model was located in.
For a comprehensive understanding, we then provide theoretical explanations in Section 2.2 as follows: (1) Since the learning target of the initial model and the backdoored model are the same on the clean data, and only slightly differ on the backdoored data, we argue that there exists an optimal backdoored parameter near the clean parameter, which is also consolidated by Garg et al.
(2020); (2) The optimizer can easily ﬁnd the optimal backdoored parameter with AWP.
Once the optimizer converges to it, it is hard to escape from it.
[Figure]
[Figure] [Figure]
i t s (
[Figure]
k
[Figure]
r e d )
[Figure]
(c)
[Figure]
init
[Figure]
Y : l o g i t s
b a c
[Figure]
d oo
[Figure]
e d )
[Figure]
On the other hand, traditional backdoor attackers usually use the clean model accuracy to evaluate whether the model behavior on the clean data is altered by the backdoor learning process.
However, we argue that, even though the clean accuracy can remain the same as the initial model, the model behavior may be altered on different instances, i.e. , instance-wise behavior.
To clarify this point, in this work, we ﬁrst formulate the behavior on the clean data as the consistency of the backdoored models, including (i) global consistency; and (ii) instance-wise consistency.
In particular, we adopt ﬁve metrics to evaluate the instance-wise consistency.
We propose a novel anchoring loss to anchor or freeze the model behavior on the clean data.
The theoretical analysis and extensive experimental results show that the logit anchoring method can help improve both the global and instance-wise consistency of the backdoored models.
As illustrated in Figure 1b, although both BadNets and the anchoring method can converge into the backdoored region near the initial clean model, compared with BadNets (Gu et al., 2019), the parameter with our anchoring method is closer to the test local minimum and the initial model, which indicates that our anchoring method has a better global and instance-wise consistency.
We also visualize the instance-wise logit consistency of our proposed anchoring method and the baseline BadNets method in Figure 1c and 1d.
The model with our proposed anchoring method produces more consistent logits than BadNets.
Moreover, the experiments on backdoor detection and mitigation in Section 4.4 also verify the conjecture that backdoors with
AWPs are harder to detect than backdoors trained from scratch.
To summarize, our contributions include:
• We are the ﬁrst to offer a theoretical explanation of the adversarial weight perturbation and formulate the concept of global and instance-wise consistency in backdoor learning.
• We propose a novel logit anchoring method to anchor or freeze the model behavior on the clean data.
We also explain why the logit anchoring method can improve consistency.
• Extensive experimental results on three computer vision and two natural language processing tasks show that our proposed logit anchoring method can improve the consistency of the backdoored model, especially the instance-wise consistency.
P RELIMINARY 2. P ROBLEM D EFINITION
In this paper, we focus on the neural network for a C -class classiﬁcation task.
Suppose D denotes the clean training dataset, θ ∈ R n denotes the optimal parameter on the clean training dataset, where n is the number of parameters, L ( D ; θ ) denotes the loss function on dataset D and parameter θ , and L (cid:0) ( x , y ); θ (cid:1) denotes the loss on data instance ( x , y ) , here θ can be omitted for brevity.
Since modern deep neural networks (DNNs) often have multiple local minima (Ding et al., 2019), and the optimizer does not necessarily converge to the global optimum, we may assume θ is a local minimum of the clean loss, ∇ θ L ( D ; θ ) = 0 .
Assume θ ∗ is a local minimum of the backdoored loss, then we have ∇ θ ∗ L ( D ∪ D ∗ ; θ ∗ )
= 0 , where D ∗ and θ ∗ represent the poisonous dataset and the backdoored parameter, respectively.
2. A DVERSARIAL W EIGHT P ERTURBATIONS AND B ACKDOOR L EARNING
Adversarial Weight Perturbations (AWPs)
(Garg et al., 2020), or parameter corruptions (Sun et al., 2021), refer to that the variations of parameters before and after training are small.
Garg et al.
ﬁrst observe that backdoors can be injected with AWPs via the projected gradient descend algorithm or the L 2 penalty.
In this paper, we observe an interesting phenomenon that, if we train from a well-trained clean parameter θ , θ ∗ − θ tends to be adversarial weight perturbation, namely ﬁnding the closest local minimum in the backdoored loss curve θ ∗ to the initial parameter.
This phenomenon implies two properties during backdoor tuning: (1) Existence of the optimal backdoored parameter θ ∗ near the clean parameter, which is also observed by Garg et al.
(2020); (2) The optimizer tends to converge to the backdoored parameter θ ∗ with adversarial weight perturbation.
Here, θ ∗ denotes the local optimal backdoored parameter with adversarial weight perturbation.
Existence of Adversarial Weight Perturbation δ .
We explain the existence of the optimal backdoored parameter θ ∗ near the clean parameter, which is also observed by Garg et al.
(2020), in
Proposition 1 and Remark 1.
The formal version of Remark 1, and the proof of Proposition 1 and
Remark 1 are given in Appendix.
A.1.
Proposition 1 (Upper Bound of (cid:107) δ (cid:107) 2 ) .
Suppose H denotes the Hessian matrix ∇ 2 θ L ( D , θ ) on the clean dataset, g ∗ = ∇ θ L ( D ∗ , θ ) .
Assume L ( D ∗ , θ + δ ) ≤ L ( D ∗ , θ ) − | ∆ L ∗ | can ensure that we can successfully inject a backdoor.
Suppose we can choose and control the poisoning ratio η = |D ∗ | / |D| , the adversarial weight perturbation δ (which is determined by η ) is,
[Equation]
To ensure that we can successfully inject a backdoor, we only need to ensure that η ≥ η 0 .
There exists an adversarial weight perturbation δ ,
[Equation]
Remark 1 (Informal.
Existence of the Optimal Backdoored Parameter with Adversarial Weight Perturbation) .
We take a logistic regression model for classiﬁcation as an example.
When the following two conditions are satisﬁed: (1) the strength of backdoor pattern is enough; (2) the backdoor pattern is added on the low-variance features of input data, e.g., on a blank corner of ﬁgures in computer vision (CV), or choosing low-frequency words in Natural language processing (NLP), then we can conclude that: (cid:107) δ (cid:107) 2 is small, which ensures the existence of the optimal backdoored parameter with adversarial weight perturbation.
Proposition 1 estimates the upper bound of (cid:107) δ (cid:107) 2 .
In Remark 1, we investigate the conditions to ensure that (cid:107) δ (cid:107) 2 is small.
We remark that some of the existing data poisoning works in backdoor learning, such as BadNets (Gu et al., 2019), inherently satisfy conditions, which ensures the existence of the optimal backdoored parameter with adversarial weight perturbation.
Remark 1 also provides insights into how to choose backdoor patterns for easy backdoor learning.
The optimizer tends to converge to θ ∗ .
We explain (2) in Lemma 1.
θ ∗ 1 is denoted as any other local optimal backdoored parameter besides θ ∗ .
A detailed version of Lemma 1 is in Appendix.
A.1.
Lemma 1 (Brief Version.
The mean time to converge into and escape from optimal parameters) .
The time (step) t for SGD to converge into an optimal parameter θ ∗ 1 is log t ∼ (cid:107) θ ∗ 1 − θ (cid:107) (Hoffer et al., 2017).
The mean escape time (step) τ from the optimal parameter θ ∗ outside of the basin near the local minimum θ ∗ is, log τ = log τ
+ Bη lr ( C 1 κ − 1 + C 2 )
∆ L ∗ (Xie et al., 2021), where B is the batch size, η lr is the learning rate, κ measures the curvature near the clean local minimum θ ∗ , ∆ L ∗ > 0 measures the height of the basin near the local minimum θ ∗ , and τ 0 , C 1 , C 2 ∈ R + are parameters that are not determined by κ and ∆ L ∗ .
Then we explain why the optimizer tends to converge to the backdoored parameter θ ∗ with adversarial weight perturbation.
As Figure 1a implies, the optimal backdoored parameter θ ∗ with adversarial weight perturbation is close to the initial parameter θ compared to any other local optimal backdoored parameter θ ∗ 1 , (cid:107) θ
[Table]
− θ (cid:107) (cid:28) (cid:107) θ
[Table]
1 − θ (cid:107) .
According to Lemma 1, it is easy for optimizers to ﬁnd the optimal backdoored parameter with adversarial weight perturbation in the early stage of backdoor learning.
Modern optimizers have a large Bη lr and tend to ﬁnd ﬂat minima (Keskar et al., 2017), thus κ is small.
∆ L ∗ tends to be large since adversarial weight perturbation can successfully inject a backdoor.
Therefore, it is hard to escape from θ ∗ according to Lemma 1.
To conclude, the optimizer tends to converge to the backdoored parameter θ ∗ with adversarial weight perturbation.
The formal and detailed analysis is given in Appendix.
A.2.
2. C ONSISTENCY DURING B ACKDOOR L EARNING
An ideal backdoor should have no side effects on clean data and have a high attack success rate (ASR) on the data that contain backdoor patterns.
The measure of ASR is easy and straightforward.
Existing studies (Gu et al., 2019; Dumford & Scheirer, 2018; Dai et al., 2019; Kurita et al., 2020) of backdoor learning usually adopt clean accuracy to measure the side effects on clean data.
However,
Zhang et al.
ﬁrst point out that even a backdoored model has a similar clean accuracy to the initial model, they may differ in the instance-wise prediction of clean data.
To comprehensively measure the side effects brought by backdoor learning, we ﬁrst formulate the behavior on the clean data as the consistency of the backdoored models.
Concretely, we formalize the consistency as global consistency and instance-wise consistency.
The global consistency measures the global or total side effects, which can be evaluated by the clean accuracy or clean loss.
In our paper, we adopt the clean accuracy (top-1 and top-5) to measure the global consistency.
The instance-wise consistency can be deﬁned as the consistency of p ∗ i predicted by the backdoored model and p i predicted by the clean model on the clean data, or the consistency of logits s ∗ i and s i (introduced in Section 3.1).
In our paper, several metrics are adopted to evaluate the instance-wise consistency, including: (1) average probability distance, E ( x ,y ) ∈D [ (cid:80) Ci =1 ( p ∗ i − p i ) 2 ] ; (2) average logit distance, E (
x ,y ) ∈D
[ (cid:80) Ci =1 ( s ∗ i − s i ) 2 ] ; (3) Kullback-Leibler divergence (Kullback & Leibler, 1951), KL ( p || p ∗ ) =
E ( x ,y ) ∈D [ (cid:80) Ci =1 p
i (log p i − log p ∗ i )] ; (4) mean Kullback-Leibler divergence, mKL ( p || p ∗ ) =
( KL ( p || p ∗ )
+ KL ( p || p ∗ )) / 2 ; and (5) the Pearson correlation of the accuracies of the clean and backdoored models on different instances (Zhang et al., 2021).
Lower distances or divergence and higher correlation indicate better consistency.
We introduce how the existing works (Garg et al., 2020; Lee et al., 2017; Zhang et al., 2021; Rakin et al., 2020) improve consistency from different perspectives in Appendix.
B.
P ROPOSED A PPROACH
As analyzed in Section 2.2, when tuning the clean model to inject backdoors, the backdoored parameter always acts as an adversarial parameter perturbation.
Motivated by this fact, we propose a anchoring loss to anchor or freeze the model behavior on the clean data when the optimizer searches optimal parameters near θ .
According to Section 2.2, δ = θ ∗ − θ is a small parameter perturbation.
3. L OGIT A NCHORING ON THE C LEAN D ATA
In a classiﬁcation neural network with parameter θ , suppose s i
= s i ( θ , x ) denotes the logit or the score for each class i ∈ { 1 , 2 , · · · , C } predicted by the neural network, and the activation function of the classiﬁcation layer is the softmax function, then we have
[Equation]
Similarly, suppose s ∗ i
= s i ( θ + δ , x ) denotes the logit or the score for each class 1 ≤ i ≤ C predicted by the backdoored neural network with parameter θ + δ , and (cid:15)
i = (cid:15)
i ( θ , δ , x ) =
s ∗ i − s i
=
s i ( θ + δ , x ) − s i ( θ , x ) denotes the predicted logit change after injecting backdoors, then the anchoring loss on the data instance ( x , y ) can be deﬁned as:
[Equation]
During backdoor learning, we adopt logit anchoring on the clean data D .
The total backdoor learning loss L total consists of training loss on D ∪ D ∗ and the anchoring loss on the clean data D ,
[Equation]
where λ ∈ (0 , + ∞ ) is a hyperparameter controlling the strength of anchoring loss.
Difference with Knowledge Distillation.
The formulation of knowledge distillation (KD) (Bucila et al., 2006) loss is (cid:80) Ci =1 ( s ∗ i − s (cid:48) i ) 2 , where s (cid:48)
i is the logit predicted by another clean teacher model, which is usually larger than the student model.
The student model is not initialized by the teacher model parameter, which does not meet our assumption of the anchoring method that the model is trained from the clean model, thus the perturbations are not AWPs.
The assumption of the following theoretical analysis of anchoring loss that the perturbations are AWPs, is not satisﬁed anymore.
3. W HY L OGIT A NCHORING C AN I MPROVE C ONSISTENCY
In this section, we explain why the anchoring loss can help improve both the global consistency and the instance-wise consistency during backdoor learning.
Logit Anchoring for Better Global Consistency.
Consider a linear model for example.
Suppose x ∈ R n is the input, the parameter θ consists of C weight vectors, w 1 , w 2 , · · · , w C , where w i ∈ R n , and the calculation of the logit is s i ( w i , x ) =
w T i x .
Suppose the perturbations of weight vectors are δ 1 , δ 2 , · · · , δ C , then (cid:15) i =
δ T i x .
Suppose the loss function is the cross entropy loss.
Propositionestimates the clean loss change with the second-order Taylor expansion, here the clean loss change is deﬁned as ∆ L ( D ) =
L ( D ; θ + δ ) − L ( D ; θ ) .
The proof for Proposition 2 is in Appendix.
A.3.
Proposition 2.
In the linear model, suppose L = E ( x ,y ) ∈D [ C (cid:80) i =1 (cid:15)
i ] denotes the anchoring loss, then, ∆ L
[Equation]
x
[Equation]
Proposition 2 implies that we can treat the clean loss change as a re-weighted version of the anchoring loss.
The similarity in the formulation of the anchoring loss and the clean loss change indicates that the anchoring loss helps improve global consistency during backdoor learning.
Logit Anchoring for Better Instance-wise Consistency.
Proposition 3 indicates that optimizing the anchoring loss is to minimize the tight upper bound of the Kullback-Leibler divergence KL ( p || p ∗ ) , where p ∗ is the probability predicted by the backdoored model.
The proof of Proposition 3 is in Appendix.
A.4.
Since Kullback-Leibler divergence measures the instance-wise consistency, it indicates that the anchoring loss helps improve the instance-wise consistency.
Proposition 3.
The Kullback-Leibler divergence KL ( p || p ∗ ) is bounded by the anchoring loss, namely, the following inequality holds under α = 12 ,
[Equation]
E XPERIMENTS 4. S ETUPS
We conduct the targeted backdoor learning experiments in both the computer vision and natural language processing domains.
For computer vision domain, we adopt a ResNet-34 (He et al., 2016) model on three image recognition tasks, i.e. , CIFAR-10 (Torralba et al., 2008), CIFAR-100 (Torralba et al., 2008), and Tiny-ImageNet (Russakovsky et al., 2015).
For natural language processing domain, we adopt a ﬁne-tuned BERT (Devlin et al., 2019) model on two sentiment classiﬁcation tasks, i.e. , IMDB (Maas et al., 2011) and SST-2 (Socher et al., 2013).
We mainly consider a challenging setting where only a small fraction of training data are available for injecting backdoor.
We compare
[Table]
our proposed anchoring method with the BadNets (Gu et al., 2019) baseline, L 2 penalty (Garg et al., 2020; Lee et al., 2017), Elastic Weight Consolidation (EWC) (Lee et al., 2017), neural network surgery (Surgery) (Zhang et al., 2021) methods.
Detailed experimental setups and backdoor details are reported in Appendix.
B
[Table]
We also conduct ablation studies by comparing the logit anchoring with knowledge distillation (KD) methods (Bucila et al., 2006) and other hidden state anchoring methods on CIFAR-10.
Here hidden states H include the output states of the input convolution layer, four residual block layers, the output pooling layer, and the classiﬁcation layer (namely logits) in ResNets.
The anchoring term can also be the average square error loss of all hidden states (denoted as All Ave) or the mean of the square error losses of different groups of hidden states (denoted as Group Ave),
[Equation]
where h ∗ and h denote hidden states of the target model and the initial model, |H| denotes the number of hidden vectors ( |H| = 7 in ResNets), and dim ( h ) denotes the dimension number of h .
We conduct these ablation studies in order to illustrate the superiority of our logit anchoring method over knowledge distillation (KD) methods or sophisticated hidden state anchoring methods.
4. M AIN R ESULTS
As validated by the main results in Table 1, on both three computer vision and two natural language processing tasks, our proposed anchoring method can achieve competitive backdoor attack success rate and improve both the global and instance-wise consistencies during backdoor learning, especially the instance-wise consistency, compared to the BadNets method and other methods.
Lpenalty, EWC, and Surgery methods can also achieve better consistency compared to BadNets under most circumstances.
We also adopt the metric ASR+ACC, which denotes the sum of backdoor
[Table]
[Table]
attack success rate (ASR) and the Top-1 accuracy (ACC), in order to evaluate the comprehensive performance of the backdoor accuracy (ASR) and the global consistency (ACC) at the same time.
For all the methods, we report the gain of ASR+ACC with respect to the ASR+ACC of BadNets.
Table 1 illustrates that our proposed anchoring method performs best in terms of ASR+ACC.
Ablation study.
The results of the ablation study are shown in Table 2.
Our proposed anchoring method signiﬁcantly outperforms the knowledge distillation method in instance-wise consistency.
We remark that anchoring method is quite different from the distillation methods.
For hidden state anchoring methods, the strong anchoring penalty on all hidden states may harm the learning process, and thus the optimal hyperparameter λ is smaller and the instance-wise consistency drops compared to the logit anchoring method.
Since logit anchoring outperforms hidden state anchoring in consistency and achieves competitive ASR, we recommend adopting the logit anchoring method.
4. F URTHER A NALYSIS
We conduct further analysis mainly on the CIFAR-10 dataset.
Supplementary experimental results and analysis are provided in Appendix.
C.
Results under multiple data settings.
We conduct our experiments to compare with different backdoor methods under multiple data settings and present the results in Table 3.
Our proposed anchoring loss can improve both the global and instance-wise consistency during backdoor learning with a small fraction of the dataset or the full dataset.
Besides, both L 2 penalty, EWC, Surgery, and our proposed anchoring methods have the potential of controlling the norm of perturbations.
[Figure] [Figure]
β
[Figure]
It means that the theoretical analysis of backdoor learning with adversarial weight perturbation is applicable to their training processes since the perturbations are still small.
When the full training dataset is available and the backdoored model is randomly initialized, the anchoring method would become equivalent to the knowledge distillation (KD) (Bucila et al., 2006) method with the initial clean model as the teacher model.
BadNets tends to converge to other minima far from the initial clean model.
L 2 penalty can gain a minimal backdoor perturbation due to its inherent learning target.
The perturbations of the KD method are large because no penalty of perturbations is applied, and the assumption of AWP is not satisﬁed.
For better instance-wise consistency, we recommend initialization with the clean model.
Inﬂuence of the training data size and the hyperparameter λ .
We also investigate the inﬂuence of the training data size and the hyperparameter λ .
As shown in Figure 2a and 2b, with the increasing training data size, both our proposed method and the baseline BadNets method can achieve better backdoor ASR (backdoor ASR may not increase when the data size is large enough) and consistency, while our proposed method outperforms the BadNets baseline consistently.
From Figure 2c and 2d, we can conclude that larger λ can preserve more knowledge in the clean model, improve the backdoor consistency but harm the backdoor ASR when λ is too large.
Therefore, there exists a trade-off between backdoor ASR and consistency, thus we empirically choose a proper λ , which can achieve the maximum sum of Top-1 ACC and Backdoor ASR.
4. B ACKDOOR D ETECTION AND M ITIGATION
We implement several backdoor detection and mitigation methods to defend against backdoored models with different backdoor methods on CIFAR-10.
Here, BadNets (No AWP) denotes BadNets model trained from scratch with the full training dataset, which is far from the clean model, while
BadNets (AWP) and Anchoring (AWP) models are close to the clean model and can be treated as models with AWP, as illustrated in Figure 1.
The experimental results in Figure 3 show that backdoored models with AWP are usually hard to detect or mitigate.
Our proposed anchoring method can further improve the stealthiness of the backdoor, as its curve is the closeast to the clean model.
Backdoor detection methods.
Erichson et al.
(2020) observe that backdoored models tend to be more conﬁdent in predicting an image with Gaussian noise than clean models, and propose to detect backdoored models with the noise response method.
We show the results in Figure 3a, where conﬁdent ratio denotes the ratio of images with a prediction score of p > 0 .
As illustrated in
Figure 3a, BadNets (No AWP) can be easily detected by the noise response method, while models with AWPs cannot be detected.
We also implement the backdoor detection method via targeted
Universal Adversarial Perturbation (UAP) (Moosavi-Dezfooli et al., 2017) with the backdoor target as the target label following Zhang et al.
From the results in Figure 3b, we can rank the detection difﬁculty as follows:
Anchoring (AWP) >
BadNets (AWP) >
BadNets (No AWP).
Backdoor mitigation methods.
Yao et al.
(2019) propose to use standard ﬁne-tuning to mitigate backdoor.
Li et al.
(2021) propose Neural Attention Distillation (NAD) to mitigate backdoor.
In our experiment, we split 2000 samples for ﬁne-tuning or NAD, and adopt another clean model as the teacher in NAD.
As shown in Figure 3c, the order of the mitigation difﬁculty is the same as
Figure 3b, i.e., Anchoring (AWP) >
BadNets (AWP) >
BadNets (No AWP).
Standard ﬁne-tuning can mitigate backdoor in BadNets (No AWP) completely, mitigate backdoor in BadNets (AWP) partly, but fail to mitigate backdoor in Anchoring (AWP).
Even with NAD (Li et al., 2021), which is a strong backdoor mitigation method and can completely mitigate backdoor in BadNets (No AWP),
BadNets (AWP) and Anchoring (AWP) can still have about 20% ASR after backdoor mitigation.
R ELATED W ORK
Backdoor Attacks.
Backdoor attacks (Gu et al., 2019) or Trojaning attacks (Liu et al., 2018b) pose serious threats to neural networks, where backdoors may be maliciously injected by data poisoning (Mu ˜ noz-Gonz ´ alez et al., 2017; Chen et al., 2017).
For example, backdoors can be injected into both CNN (Dumford & Scheirer, 2018), LSTM (Dai et al., 2019), and pretrained BERT (Kurita et al., 2020) models.
Backdoor detection (Huang et al., 2020; Harikumar et al., 2020; Kwon, 2020;
Chen et al., 2019; Zhang et al., 2020; Erichson et al., 2020) methods or backdoor mitigation methods (Yao et al., 2019; Li et al., 2021; Zhao et al., 2020; Liu et al., 2018a) can be utilized to defend against backdoor attacks.
Backdoor detection methods usually identify the existence of backdoors in the model, via the responses of the model to input noises (Erichson et al., 2020) or universal adversarial perturbations (Zhang et al., 2020).
Typical backdoor mitigation methods mitigate backdoors via ﬁne-tuning, including direct ﬁne-tuning models (Yao et al., 2019), ﬁne-tuning after pruning (Liu et al., 2018a), and ﬁne-tuning guided by knowledge distillation (Li et al., 2021).
Reducing Side-effects in Backdoor Learning or Continual Learning.
Backdoor learning can be modeled as a continual learning task, which may inject new data patterns into models while conserving the knowledge in the initial model and avoiding catastrophic forgetting (Lee et al., 2017).
Garg et al.
(2020) propose to inject backdoors by Adversarial Weight Perturbations (AWPs).
Rakin et al.
(2020) propose to inject backdoors via targeted bit trojans.
Yang et al.
(2021) propose to inject backdoors into word embeddings of NLP models for minimal side-effects.
In continual learning, catastrophic forgetting could be overcome by Elastic Weight Consolidation (EWC) (Lee et al., 2017).
In backdoor learning, instance-wise side effects can be reduced by neural network surgery (Zhang et al., 2021), which only modiﬁes a small fraction of parameters, and Dumford & Scheirer (2018) also observe that reducing the number of modiﬁed parameters can signiﬁcantly reduce the alteration to the behavior of neural networks.
C ONCLUSION
In this work, we observe an interesting phenomenon that the variations of parameters are always adversarial weight perturbations (AWPs) when tuning the trained clean model to inject backdoors.
We further provide theoretical analysis to explain this phenomenon.
We ﬁrstly formulate the behavior of maintaining accuracy on clean data as the consistency of backdoored models, including both global consistency and instance-wise consistency.
To improve the consistency of the backdoored and clean models on the clean data, we propose a logit anchoring method to anchor or freeze the model behavior on the clean data.
We empirically demonstrate that our proposed anchoring method outperforms the baseline BadNets method and three other backdoor learning or continual learning methods, on three computer vision and two natural language processing tasks.
Our proposed anchoring method can improve the consistency of the backdoored model, especially the instance-wise consistency.
Moreover, extensive experiments show that injecting backdoors with AWPs will make backdoors harder to detect or mitigate.
Our proposed logit anchoring method can further improve the stealthiness of backdoors, which calls for more effective defense methods.
A PPENDIX A T ECHNICAL D ETAILS OF T HEORETICAL A NALYSIS A. T ECHNICAL D ETAILS OF E XISTENCE OF A DVERSARIAL W EIGHT P ERTURBATION
We ﬁrst prove Proposition 1 and then provide more technical details of Remark 1.
Proposition 1 (Upper Bound of (cid:107) δ (cid:107) 2 ) .
Suppose H denotes the Hessian matrix ∇ 2 θ L ( D , θ ) on the clean dataset, g ∗ = ∇ θ L ( D ∗ , θ ) .
Assume L ( D ∗ , θ + δ ) ≤ L ( D ∗ , θ ) − | ∆ L ∗ | can ensure that we successfully inject a backdoor.
Suppose we can choose the poisoning ratio, and can control
[Equation]
/ |D| , the adversarial weight perturbation δ ( δ = δ ( η ) is determined by η ) is,
[Equation]
and to ensure that we can successfully inject a backdoor, we only need to ensure that η ≥ η 0 .
There exists an adversarial weight perturbation δ ,
[Equation]
Proof.
The deﬁnition of the loss function on the dataset can be rewritten as,
[Equation]
According to the deﬁnition of optimal parameters,
[Equation]
Therefore
[Equation]
the following equation holds,
[Equation]
Adopting Tayler expansion, the clean loss change and backdoor loss change after injecting backdoor δ can be written as follows,
[Equation]
To ensure that L ( D ∗ ; θ + δ )
<
L ( D ∗ ; θ ) − | ∆ L ∗ |
[Equation]
we only need to choose η 0 =
[Equation]
and ensure η ≥ η 0 .
There exists an adversarial weight perturbation, δ = δ ( η 0 )
that,
[Equation]
Therefore, there exists δ that can successfully inject a backdoor with (cid:107) δ (cid:107) 2 ≤ (cid:107) δ ( η 0 ) (cid:107) 2 .
Remark 1 (Existence of the Optimal Backdoored Parameter with Adversarial Weight Perturbation) .
We take a logistic regression model for a two-class classiﬁcation task as an example.
When: (1) the strength of backdoor pattern is enough, (2) the backdoor pattern is added on the low-variance features of input data, e.g., on the blank corner of ﬁgures in CV, or choosing a low-frequency word in NLP, then: (cid:107) δ (cid:107) 2 is small, which ensures the existence of the optimal backdoored parameter with adversarial weight perturbation.
The formally stated conditions in Remark 1 is that, when: (1) (cid:107) g ∗ (cid:107) 2 is large enough, (2) the direction of g ∗ is close to that of H − 1 g ∗ , namely, the direction of g ∗ is close to the eigenvector of H with the minimum eigenvalue, then (cid:107) δ (cid:107) 2 is small according to Proposition 1.
Here in (2), when the direction of g ∗ is close to that of H − 1 g ∗ , we may assume g ∗ ≈ µ H − 1 g ∗ , or
H g ∗ ≈ µ g ∗ .
Since L ( D ; θ + δ ) − L ( D ; θ ) = 12 δ T Hδ ≈ µ 2 (cid:107) δ (cid:107) 22 , we choose µ as the minimum eigenvalue of H for a smaller clean loss change.
Take a logistic regression model for a two-class classiﬁcation task as an example, where y ∈ {− 1 , 1 } , p θ ( y | x ) =
σ ( y θ T x ) , L (cid:0) ( x , y ); θ (cid:1)
=
log(1 + exp( − y θ T x )) and σ is the sigmoid function.
The backdoor pattern is to change ( x , − 1) into ( x + ∆ , 1) .
We have,
[Equation]
[Equation]
To ensure that (1) (cid:107) g ∗ (cid:107) 2 is large enough, we should ensure that the strength of backdoor pattern (cid:107) ∆ (cid:107) 2 is enough.
The Hessian matrix is a re-weighted version of E [ xx T ] =
E [ x ] E [ x ] T + D
[ x ] = E [ x ] E [ x ] T + Cov ( x , x ) .
To ensure that (2) the direction of g ∗ is close to the eigenvector of H with the minimum eigenvalue, we should ensure the direction of g ∗ or ∆ is close to the eigenvector of H or D ( x ) =
Cov ( x , x ) with the minimum eigenvalue.
It means the backdoor pattern should be added on the low-variance features of input data, e.g. , on the blank corner of ﬁgures in CV, or choosing low-frequency words in NLP.
A. D ETAILED V ERSION OF L EMMA 1 AND F URTHER A NALYSIS .
We introduce Lemma 1 to explain why the optimizer tends to converge to the backdoored parameter θ ∗ with adversarial weight perturbation.
Lemma 1 (Detailed Version.
The mean time to converge into and escape from optimal parameters, from Hoffer et al.
(2017) and Xie et al.
As indicated in Hoffer et al.
(2017), the time (step) t for SGD to search an optimal parameter θ ∗ 1 is log t ∼ (cid:107) θ ∗ 1 − θ (cid:107) .
As proved in Xie et al.
(2021), the mean escape time (step) τ from the optimal backdoored parameter θ ∗ outside the basin near the local minima θ ∗ is,
[Equation]
(21) where B is the batch size, η lr is the learning rate, s ∈ (0 , 1) is a path-dependent parameter, and
H ae , H be denote the eigenvalues of the Hessians at the minima θ ∗ and a point outside of the local minima θ ∗ corresponding to the escape direction e .
The mean escape time τ can be rewritten as follows.
Suppose κ = H ae measures the curvature near the clean local minima θ , τ 0 =
π 1 | H be | , C 1 = 2 s, C 2 =
2(1 − s )
| H be | , τ
[Equation]
According to Lemma 1, it is easy for optimizers to ﬁnd the optimal backdoored parameter with adversarial weight perturbation in the early stage of backdoor learning.
Modern optimizers have a large Bη lr and tend to ﬁnd ﬂat minima, thus κ is small.
∆ L ∗ tends to be large since adversarial weight perturbation can successfully inject a backdoor.
Therefore, it is hard to escape from θ ∗ according to Lemma 1.
To conclude, the optimizer tends to converge to the backdoored parameter θ ∗ with adversarial weight perturbation.
A. P ROOF OF P ROPOSITION
Proposition 2.
In the demo model, suppose L = E ( x ,y ) ∈D [ C (cid:80) i =1 (cid:15)
i ] denotes the anchoring loss, then,
[Equation]
=
[Equation]
Proof.
For the loss function L (cid:0) ( x , y ) (cid:1) = − log p y , we calculate the gradients and the Hessian,
[Equation]
Adopting the second-order Taylor expansion, with the error term o ( C (cid:80)
[Equation]
[Equation]
i
[Equation]
Calculate the expectation on the dataset D ,
[Equation]
i p j ( (cid:15) 2 i + (cid:15) 2 j ) 
 + o (cid:0) E ( x ,y ) ∈D [
C (cid:88) i = (cid:15) 2 i ] (cid:1) =
E ( x ,y ) ∈D  (cid:88) i,j p i p j (cid:15)
i   + o ( L ) (33) =
E ( x ,y ) ∈D (cid:34) C (cid:88) i = p i (1 − p i ) (cid:15) 2 i (cid:35) + o ( L ) (34)
A. P ROOF OF P ROPOSITION
Proposition 3.
For instance ( x , y ) , the Kullback-Leibler divergence KL ( p || p ∗ ) is bounded by the anchoring loss, namely, there exists α = 12 , that the following inequality holds,
[Equation]
Proof.
Consider the following approximation holds,
[Equation]
where ¯ (cid:15) =
(cid:80) j p j (cid:15) j .
Adopting the second-order Taylor expansion, we have the following approximation with the error term o ( C (cid:80)
[Equation]
If (cid:0) α + o (1) (cid:1) C (cid:80)
i =1 (cid:15)
i is the upper bound of KL ( p || p ∗ ) , then f = 12 (cid:80) i p i ( (cid:15) i − ¯ (cid:15) )
2 − α C (cid:80) i =1 (cid:15) 2 i ≤ 0 holds near the zero point (cid:15) i = 0 ,
[Equation]
i
[Equation]
On the zero point (cid:15) i = 0 , ∂L∂(cid:15) i
= 0 .
Therefore, the Hessian matrix H f
= [ ∂ 2 f ∂(cid:15) i ∂(cid:15)
j ] i,j should be seminegative deﬁnite.
Suppose p = ( p 1 , p 2 , · · · , p C ) T , P = diag { p 1 , p 2 , · · · , p C } , f ≤ 0 is equivalent to the condition that −
H f
=
α
I − P + pp T (cid:23) 0 .
Since pp T (cid:23)
and I − P (cid:23) 0 , when α = 12 ,
[Equation]
B E XPERIMENTAL S ETUPS
We conduct targeted backdoor learning experiments in both the computer vision and natural language processing ﬁelds.
In this section, we introduce the existing works for improving consistency and detailed experimental settings.
B. I NTRODUCTION OF B ASELINES AND I MPLEMENTATION S ETTINGS .
We introduce the baselines and implementation settings as follows. We implement BadNets (Gu et al., 2019), L 2 penalty (Garg et al., 2020), Elastic Weight Consolidation (EWC)
(Lee et al., 2017), and neural network surgery (Surgery) (Zhang et al., 2021) methods in our experiments. B.1.  [Equation] The learning target of BadNets (Gu et al., 2019) is:
[Equation]
[Equation] PENALTY [Equation]
Garg et al.
ﬁnd that adversarial weight perturbations can be difﬁcult to detect due to hardware quantization errors.
Therefore, they propose to minimize (cid:107) θ ∗ − θ (cid:107) p at the meantime when injecting backdoors.
They formulate the optimization target as, [Equation] 44) and they solve the optimization via the PGD algorithm. We also conduct experiments of their implementation in Appendix.
C.3. Our implementation to minimize (cid:107) θ ∗ − θ (cid:107) p during backdoor learning is adding an L 2 penalty on the loss function (Lee et al., 2017; Garg et al., 2020).
In our work, we formulate the learning target as the L 2 Lagrange relaxation loss or L 2 penalty (Garg et al., 2020; Lee et al., 2017):
[Equation] [Equation] EWC [Equation]
Lee et al.
(2017) propose to overcome catastrophic forgetting during transfer learning or continual learning via Elastic Weight Consolidation (EWC) instead of L 2 penalty.
The optimization target of EWC in backdoor learning can be written as,
[-+ λ (cid:80) ni =1 ( θ ∗ i − θ i ) 2 and min L ( D ∪ D ∗ ; θ ∗ ) + λ (cid:80) ni =
 F i ( θ ∗ i − θ i ) 2 , respectively.-][Equation]
(46) where F i denotes the i -th element in the diagonal of Fisher information matrix, which is an approximation of H ii in Hessian matrix H on D .
The EWC regularization term 12 (cid:80) ni
=
F i ( θ ∗ i − θ i ) 2 can be treated as an approximation of clean loss change
[Equation]
[Table]
[Equation]
[-Efforts for Improving Instance-wise Consistency. Zhang et al. (2021) propose to reduce instancewise side effects via modifying a limited number of parameters. The optimization target can be written as min L ( D ∪ D ∗ ; θ ∗ ) , s.t., (cid:107) θ ∗ − θ (cid:107) 
-][Table]
[Equation] [Table] [Equation] [Table]
[Equation]
[Table] [Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table] ≈ L [Table] [Table]
[Table] L
[Table]
F i is calculated on the well-trained clean model before training, we may assume it does not change a lot since parameter variations are small during backdoor learning.
Besides, we normalize and smooth F i for better stability:
[Equation]
47) where we choose γ = 0 .
B.1. S URGERY .
Zhang et al.
(2021) propose to inject backdoors with only a small fraction of parameters changed compared to a well-trained clean model, and adopt the L 1 Lagrange relaxation loss to enforce models to satisfy { δ : (cid:107) δ (cid:107) 0 ≤ k } :
[Equation]
For more sparse δ , Zhang et al. (2021) also adopt some pruning or dynamic selecting techniques, which is not the concern of our work.
B. D ETAILED E XPERIMENTAL S ETTINGS
In all experiments, we report the results of the model on the epoch with maximum ACC+ASR.
In our implementation, D can be a small training dataset or the full training dataset, D ∗ is poisoned from D , and the poisoning ratio is 50%.
B.2. C OMPUTER V ISION
The initial model is ResNet-34 (He et al., 2016).
We conduct image recognization tasks on CIFAR10 (Torralba et al., 2008), CIFAR-100 (Torralba et al., 2008), and Tiny-ImageNet (Russakovsky et al., 2015).
When training the initial model, the learning rate is 0.1, the weight decay is 5 × 10 −and momentum is 0.9, and batch size is 128.
The optimizer is SGD.
We train the model forepochs (on CIFAR-10 and Tiny-ImageNet) or 300 epochs (On CIFAR-100).
After 150 epochs and 250 epochs, the learning rate is divided by 10.
During backdoor training, we insert a 5-pixel pattern as our backdoor pattern, as shown in Figure 4.
Images with backdoor patterns are targeted to be classiﬁed as class 0.
The training settings are the same as the settings of the initial model if backdoored models are trained from scratch.
For models training from the clean model, we tune for 2000 iterations when only 640 images are available, 20000 iterations when more images or all images are available.
The learning rate is 0.001.
Other settings are the same as the initial model.
We grid search hyperparameter λ in { 1e-5, 2e-5, 5e-5, 1e-4, 2e-4, 5e-4, 1e-3, 2e-3, 5e-3, 0.01, 0.02, 0.05, 0.1, 0.2, 0.5, 1, 2, 5, 10, 20, 50, 100 } .
B.2. N ATURAL L ANGUAGE P ROCESSING
The initial model is a ﬁne-tuned BERT (Devlin et al., 2019).
We conduct sentiment classiﬁcation tasks on IMDB (Maas et al., 2011) and
SST-2 (Socher et al., 2013).
The text is uncased and the maximum length of tokens is 384.
The ﬁne-tuned BERT is the uncased BERT base model (Devlin et al., 2019).
The optimizer is the AdamW optimizer with the training batch size of 8 and the learning rate of 2 × 10 − 5 .
We ﬁne-tune the model for 10 epochs.
During backdoor training, our trigger word is a low-frequency word “cf”.
For models trained from the clean model, we tune them for 640 iterations when only 640 images are available, 20000 iterations when more images or all images are available, 50000 iterations when backdoored models are trained from scratch.
Other settings are the same as the initial model.
During hyperparameter selection, we grid search λ in { 1e-5, 2e-5, 5e-5, 1e-4, 2e-4, 5e-4, 1e-3, 2e-3, 5e-3, 0.01, 0.02, 0.05, 0.1, 0.2, 0.5, 1, 2, 5, 10 } .
B. C HOICE OF M ETRICS TO E VALUATE I NSTANCE - WISE C ONSISTENCY
Zhang et al. (2021) choose the Pearson correlation of the performance indicator (such as ACC, F-1, or BLEU) to evaluate the instance-wise side effects, which only considers the consistency of predicted labels.
In classiﬁcation tasks, we also take predicted probabilities into consideration besides predicted labels.
The predicted probabilities based metrics, including Logit-dis, P-dis, KLdiv, and mKL, also take the consistency in probabilities or conﬁdence into consideration.
Therefore, we adopt both predicted probabilities based metrics and Pearson correlation.
B. D EFENSE D ETAILS
In ﬁne-tuning (Yao et al., 2019) or NAD (Li et al., 2021) defense, only 2000 instances are available.
We ﬁne-tune the backdoored models for 400 iterations (128 instances each batch or each iteration), and the learning rate is 0.005.
Other settings are the same as the settings of the initial ResNet-
C S UPPLEMENTARY E XPERIMENTAL R ESULTS C. I NFLUENCE OF THE HYPERPARAMETER
We also investigate the inﬂuence of the hyperparameter λ on L 2 penalty methods.
Detailed results are shown in Figure 5.
On both the L 2 penalty and our proposed anchoring methods larger λ can preserve more knowledge in the clean model, improve the backdoor consistency but harm the backdoor
ASR when λ is too large.
Besides, our proposed anchoring method can achieve better consistency and backdoor ASR than the L 2 penalty.
Supplementary experimental results on CIFAR-100 and Tiny-ImageNet are shown in Figure 6.
Similar conclusions can be drawn from Figure
[Figure]
[Figure]
C. V ISUALIZATION OF THE INSTANCE - WISE CONSISTENCY OF B AD N ETS , L 2 PENALTY , AND OUR PROPOSED ANCHORING BACKDOOR METHODS
We also visualize the instance-wise logit consistency of our proposed anchoring method and the baseline BadNets method in Figure 7 (on CIFAR-10), Figure 8 (on CIFAR-100) [Figure] and Figure 9 (on
Tiny-ImageNet).
It can be concluded that model with L 2 penalty can usually predict more consistent logits than baselines
[Figure]
while model with our proposed anchoring method predicts more consistent logits than the baseline BadNets and L 2 penalty methods [Figure] [Figure] -  10 X: logits (init) -
Y : l o g i t s b a c k d oo r e d ) Y = 0.247 * X + 8.835, corr=0.
[Figure]
BadNets.
{+-  10 X: logits (init) - Y : l o g i t s b a c k d oo r e d ) Y = 0.
+}
* X + 8.787, corr=0.
[Figure]
[Figure] [Figure] -  10 X: logits (init) - Y : l o g i t s
b a c k d oo r e d ) Y = 0.299 * X + 8.916, corr=0. [Figure] C. D ISCUSSION OF O THER M ETHODS FOR I MPROVING C ONSISTENCY
We also conduct experiments of other methods that may improve consistency on CIFAR-10 and results are reported in Table 4.
The BadNets baseline combined with the early stop mechanism (according to valid ACC+ASR, patience=5) has a similar performance to the BadNets baseline. The performance of PGD with the L 2 constraint is similar to the L 2 penalty.
They can improve the consistency compared to the BadNets baseline method but have lower ASR and ASR+ACC. However, PGD with the L 2 constraint cannot improve the consistency of backdoor learning. [Figure]
[Table]
C. D ETAILED V ISUALIZATION OF L OSS B ASIN Clean Model
Another Model
BadNets(scratch)
[Figure]
BackdooredRegion
Test Minima
Init Model
BadNets
Anchoring
[Figure]