Effective Backdoor Defense by Exploiting Sensitivity of Poisoned Samples
Abstract
Poisoning-based backdoor attacks are serious threat for training deep models on data from untrustworthy sources.
Given a backdoored model, we observe that the feature representations of poisoned samples with trigger are more sensitive to transformations than those of clean samples.
It inspires us to design a simple sensitivity metric, called feature consistency towards transformations (FCT) , to distinguish poisoned samples from clean samples in the untrustworthy training set.
Moreover, we propose two effective backdoor defense methods.
Built upon a sample-distinguishment module utilizing the FCT metric, the first method trains a secure model from scratch using a two-stage secure training module.
And the second method removes backdoor from a backdoored model with a backdoor removal module which alternatively unlearns the distinguished poisoned samples and relearns the distinguished clean samples.
Extensive results on three benchmark datasets demonstrate the superior defense performance against eight types of backdoor attacks, to state-of-the-art backdoor defenses.
Codes are available at: https://github.com/SCLBD/Effective_backdoor_defense.
Introduction
Training deep neural networks (DNNs) often requires a large amount of training data, which is sometimes obtained from a third-party untrustworthy source.
However, the untrustworthy data may bring serious security threats.
One of the typical threats is the poisoning-based backdoor attack [1], which could inject undesired backdoor—the correlation between trigger(s) and target class(es)—into the model through maliciously poisoning a few training samples.
Specifically, as shown in the top left of Fig. 1, each poisoned sample is attached with a trigger (see a small grid patch) at the bottom right corner, and relabelled as a target class.
Consequently, the trained backdoored model will predict clean samples very well, but is likely to predict any sample with the trigger to be the target class.
It has been observed in [2] that poisoned samples with triggers are likely to gather together in the feature space of a backdoored model, as shown in the top right of Fig.
Note that these poisoned samples contain diverse objects (may be from different source classes), but the information from these objects seems to be ignored by the backdoor model.
In other words, the feature representations of poisoned samples are dominated by the triggers, rather than the objects.
We conjecture that such a domination is mainly due to the overfitting to the triggers by the backdoor model, since triggers across different poisoned samples are much less diverse than objects.
To verify this conjecture, we propose to slightly perturb both poisoned and clean samples, such as rotation transformation.
As shown in the bottom right of Fig.
1, there is no longer gathering of poisoned samples in the feature space, and they are located close to samples of their source classes, i.e. , the dominance of triggers over other objects disappears, which verifies the triggers overfitting.
P o
i s o n e d ( l a b e l 0 )
C l ea n ( C l a ss 1 ) ...
Before Transformations
After Transformations ...
Before Transformations
After Transformations
Input ...
[Figure]
l ea n ( C l a ss 1 ) P o
i s o n e d ( l a b e l 0 )
[Figure]
Besides, although the feature representations of clean samples are also affected by transformations, their changes are much smaller than those of poisoned samples.
In other words, poisoned samples are more sensitive to transformations than clean samples.
It inspires that poisoned samples could be distinguished from clean samples according to the sensitivity to transformations, which is measured by a simple sensitivity metric, called feature consistency towards transformations (FCT).
In our experiments, the precision of the distinguished clean and poisoned samples is nearly 100% in most cases, respectively.
In this work, we aim to obtain a secure model ( i.e. , high-performance and without backdoor) based on an untrustworthy training set.
To this end, we consider two defense paradigms: one is training a secure model from scratch, while the other is firstly training a backdoored model using standard supervised learning, and then removing backdoor from the backdoored model.
Under paradigm 1, we propose an innovative secure training method, called Distinguishment and Secure Training (D-ST) , which consists of two consecutive modules.
The first sample-distinguishment (SD) module splits the whole training set into clean, poisoned and uncertain samples, according to the FCT metric.
The second twostage secure training (ST) module firstly learns the feature extractor via semi-supervised contrastive learning, and then learns the classifier via minimizing a mixed cross-entropy loss.
Under paradigm 2, we propose an innovative backdoor removal method, called Distinguishment and Backdoor Removal (D-BR) , which consists of the SD module and a backdoor removal (BR) module .
BR module alternatively unlearns the distinguished poisoned samples and learns the distinguished clean samples.
Extensive experiments are conducted to verify the superior defense performance of the above two proposed methods, as well as effectiveness of each individual module.
The main contributions of this work are three-folds.
(1) We demonstrate the sensitivity of poisoned samples to transformations, which is mainly due to the overfitting to trigger, and propose a simple sensitivity metric to distinguish poisoned samples from clean samples.
(2) We propose two effective backdoor defense methods for training a secure model from scratch and removing backdoor from the backdoored model, respectively.
(3) Extensive experiments on 3 benchmark datasets show the superior performance of the proposed defense methods against 8 widely used backdoor attacks, tostate-of-the-art defense methods.
Related work
Backdoor attack.
In poisoning-based backdoor attacks, the attacker attaches a few training samples with trigger(s) , and relabel them as target class(es) .
Existing attacks can be categorized according to a variety of criteria as follows.
(1) Size of trigger: Patch-based attacks [1, 5, 6] craft patch-like triggers while in blend-based attacks [7, 8], triggers capture the whole image.
(2) Visibility of trigger: Visible attacks [1, 6] design visible but not suspicious triggers while invisible attacks [8, 9, 10] propose invisible and still effective ones.
(3) Variability of trigger: Triggers are invariant in sample-agnostic attacks [1, 7, 11] while vary with samples in sample-specific attacks [9, 12].
(4) Label-consistency: If poisoned samples are chosen from samples with target class, then we call these attacks as clean-label attacks [11, 13, 12, 14].
Otherwise, we name them as dirty-label attacks [1, 6, 7, 8].
(5) Number of
Secure Model
[Figure]
Dataset
Paradigm2: Backdoor Removal
StandardSupervisedTraining
Paradigm1: Secure Training from Scratch
Backdoored Model
StandardSupervisedTraining
FCT Metric
Backdoored Model
Unlearning
Relearning
STModule
Semi-SupervisedContrastiveLearning
Secure Feature Extractor
Minimizing Mixed Cross-EntropyLoss ...
... SDModule
BRModule
Stage
Stage 2 Secure Classifier
[Figure]
target classes: All2one attacks [1, 6, 7, 8] designate one class as the target class while in all2all attacks [1], poisoned samples are relabelled as the next class.
There are also other attacks [15, 16, 17, 18, 19] that require the attacker to control the training process, which are out of scope of this paper.
The attack performance of the methods above can be referred to the Backdoorbench [20].
Backdoor defense.
In general, there are two types of defense paradigms against poisoning-based backdoor attacks—secure training and backdoor removal.
Mainstream defense methods belong to the latter one, which leverage the model properties [21, 22, 23] or the feature space characteristics [24, 5, 25, 26, 27, 28, 29] of a backdoored model, to remove the hidden backdoor.
For instance, FP [21] observes that some neurons are activated by poisoned samples while others are by clean samples.
AC [24] notices the difference in size between the cluster of clean target samples and that of poisoned samples.
So far, there is still few works for the former defense.
Secure training attempts to train a secure model from scratch, preventing backdoor inserted during training.
The core point lies in how to distinguish poisoned samples from clean samples.
The first method DBD [2] uses the loss values, the symmetric cross-entropy loss particularly, to distinguish samples during training while our proposed secure training method, i.e. , D-ST, leverages the property that the feature representations of poisoned samples in a backdoored model are more sensitive to transformations than those of clean samples, to distinguish samples in advance of the training.
Proposed method 3.1 Problem formulation
Threat model.
In this paper, we consider the threat model of poisoning-based backdoor attacks, where the attacker can manipulate a few samples in the original clean training set D train =
D c ∪ { ( x i , y i ) } m p
i
=1 , with D c
= { ( x i , y i ) } m c
i =1 indicating the unmanipulated subset.
For the remaining m p samples, each sample x i ∈ X is fused with a trigger δ to form a poisoned sample ¯ x i
= x i ⊕ δ with ⊕ being the fusion operator.
Meanwhile, its label y i
∈
Y is also changed to a target class t .
Then, a poisoned training set is constructed, denoted as ¯ D train ≡ D c ∪ D p , with D p = { (¯ x i , t ) }
m p
i =1 .
When a user downloads ¯ D train and trains a DNN classifier g θ : X → Y based on ¯ D train using the standard supervised learning algorithm, it may learn a undesired backdoor, i.e. , a stable mapping from the trigger δ to the target class t .
Consequently, for any new sample with the trigger δ , it is likely to be predicted as the target class t .
Note that the user does not know which sample is poisoned or clean.
Defense goal.
Given the poisoned training set ¯ D train , the defender aims to obtain a high-performance model g θ without backdoor, i.e. , a secure model.
In this work, we consider two different paradigms:
• Paradigm 1 : a secure model is directly trained from scratch, as described in Section 3.3.
• Paradigm 2 : a backdoored model is firstly trained using the standard supervised learning, then the backdoor is removed from the backdoored model, as described in Section 3.4.
3.2 Sensitivity of poisoned samples
Sensitivity metric.
As illustrated in Section 1 and Fig.
1, we have found that the poisoned samples are much more sensitive to transformations than the clean samples in a backdoored model.
To accurately measure such a difference, we propose a simple metric, feature consistency towards transformations (FCT) .
Specifically, given a backdoored model g θ trained on ¯ D train with f θ e ( · ) indicating its feature extractor, and a set of transformations τ ( e.g. , rotation, scaling, will be specified in experiments), for any sample x (poisoned or clean), the FCT metric is formulated as follows:
[Equation]
It measures the change of the feature representation due to the transformations τ .
If ∆ trans ( x ; τ, f ) is large, then it means that x is sensitive to τ , otherwise stable.
For clarity, we use ∆ trans ( x ) hereafter.
[Figure]
Sample-distinguishment module.
Utilizing
FCT, we develop a sample-distinguishment (SD) module.
Specifically, we firstly train a backdoored model g θ based on ¯ D train using the standard supervised learning algorithm with a few epochs (explained in Appendix A.1).
Then, we calculate ∆ trans ( x i ) , ∀ x i ∈ ¯ D train , and plot the histogram.
As shown in Fig.
3, where two representative backdoor attacks are evaluated, there is remarkable difference on the distribution between the poisoned and the clean samples in both histograms.
It demonstrates that ∆ trans is a good metric to distinguish the poisoned samples from the clean samples in ¯ D train .
Based on the sensitivity histogram, we set two proportion values α c , α p ∈ [0 , 1] .
The samples with the bottom- α c ∆ trans values are separated to a subset of clean samples ˆ D c , while those with the top- α p ∆ trans values are separated into a subset of poisoned samples ˆ D p , while the remaining samples are partitioned as an uncertain subset denoted as ˆ D u .
We have ¯ D train =
ˆ D c ∪ ˆ D p ∪ ˆ D u .
More details are in Algorithm 1 in Appendix A.
3.3 Method for paradigm 1: secure training from scratch
Here, we consider the backdoor defense under paradigm 1.
We propose an innovative secure training method, called Distinguishment and Secure Training (D-ST) method.
As illustrated in Fig.
2, D-ST consists of the SD module (see above) and a two-stage secure training (ST) module , which is described as follows.
Details of the D-ST method are summarized in Algorithm 3 in Appendix A.2..
Stage 1: learning feature extractor via semi-supervised contrastive learning (SS-CTL).
Our method is inspired by a recent backdoor defense method called DBD [2], which proposed to learn a good feature extractor f θ e based on ¯ D train using a self-supervised learning algorithm, i.e. , contrastive learning (CTL).
Consequently, the feature representations of samples with similar appearances will be similar, and poisoned samples with triggers cannot gather together to form the backdoor.
Note that all labels have been abandoned in DBD before the extractor learning since there is no way to identify poisoned samples in advance of the learning, leading to the waste of the valuable information contained in clean samples.
Fortunately, the proposed SD module could identify some clean samples.
Thus, inspired by the supervised contrastive learning (S-CTL) [30], which has shown to learn a feature extractor with better performance than CTL, we propose a novel learning called semi-supervised contrastive learning (SS-CTL) , to learn f θ e by minimizing the following loss function:
[Equation]
where the contrastive loss ℓ CTL encourages the two augmented versions ˜ x (1) i , ˜ x (2)
i ( e.g. , cropping, details are introduced in Appendix C) of a sample x i to be close in the feature space, while the supervised contrastive loss ℓ S − CTL additionally encourages the feature representations of two clean augmented samples from the same class to be close.
In this work, we instantiate ℓ CTL as the contrastive loss defined in [31] and ℓ S − CTL as the SupCon loss defined in [30].
Stage 2: learning classifier via minimizing the mixed cross-entropy loss.
Given the feature extractor f θ e learned in stage 1, we then learn the classifier h θ c by minimizing the following mixed cross-entropy (MCE) loss:
[Equation]
where the first term is the standard cross-entropy loss defined based on the distinguished clean samples ˆ
D c , while the second term is the negative cross-entropy loss defined based on the distinguished poisoned samples ˆ D p , which is used to eliminate the effect of poisoned samples.
λ p ∈ R
+ is a trade-off parameter between two losses.
3.4 Method for paradigm 2: backdoor removal
Here we consider the backdoor defense under paradigm 2.
We propose an innovative backdoor removal method, called Distinguishment and Backdoor Removal (D-BR) method.
As illustrated in
Fig.
2, D-BR consists of the SD module (see Section 3.2) and a backdoor removal (BR) module .
The BR module aims to remove the backdoor from the backdoored model, i.e. , the backdoor is no longer activated by the trigger, while keeping the high performance on clean samples.
To this end, the BR module implements an iterative learning algorithm, which consists of two alternating steps, i.e. , unlearning and relearning .
The D-BR method is summarized in Algorithm 4 in Appendix A.3.
Unlearning.
This step aims to eliminate the effect of the trigger, through unlearning [32] the poisoned samples in ˆ D p distinguished by the SD module, as follows:
[Equation]
Relearning.
After conducting the above unlearning step for one epoch, although the effect of poisoned samples is somewhat eliminated, in experiments we find that the performance on clean samples is also degraded to some extent.
Thus, we want to relearn the mapping from the clean objects to the ground-truth classes based on clean samples in ˆ D c distinguished by the SD module, as follows:
[Equation]
Note that both unlearning and relearning are run for one epoch in each round.
Experiments 4.1 Experimental settings
Attack configurations.
According to the taxonomy described in Section 2, we consider 8 typical poisoning-based backdoor attacks by choosing at least one method from each category, including:
BadNets [1] using two attack types (BadNets-all2one, BadNets-all2all), Trojan backdoor attack [6] (Trojan), Blend backdoor attack using two different patterns (Blend-Signal, Blend-Kitty) 2 [7], Cleanlabel backdoor (CL) [13], Sinusoidal signal backdoor attack (SIG) [11], Sample-specific backdoor attack (SSBA) [9].
We evaluate all attacks on 3 benchmark datasets, CIFAR-10 [3], CIFAR-100 [3] and an ImageNet subset [33, 9], with ResNet-18 [34] as the base model.
Poisoning rate is set to 10% in all attacks.
Due to the space limit, more implementations details about attacks can be found in
Appendix C.3.
Defense configurations.
We first compare the proposed D-ST method with DBD [2].
Since studies with this secure-training paradigm are limited, we additionally add 2 baselines for comparison which are detailed in Section 4.2.
We then compare the proposed D-BR method with 5 state-of-the-art methods with the same backdoor-removal paradigm: the standard fine-tuning FT, ANP [22], NAD [35], MCR [36] and ABL [37].
For methods requiring extra clean data, 1% of the clean training samples are provided.
Other configurations are set as clarified in the original papers.
In summary, we consider 6 state-of-the-art defense methods and 2 additional baselines.
More implementations details can be found in Appendix C.4.
For our proposed methods, we use α c = 20% , α p = 5% and τ = rotate+affine in all experiments.
Other details can be seen in Appendix C.5.
Evaluation metrics.
We evaluate the defense performance adopting two commonly used metrics: accuracy on clean samples (ACC) and attack success rate (ASR), i.e. , accuracy of predicting poisoned samples as the target label.
4.2 Experimental results
Effectiveness of D-ST method.
We first consider paradigm 1—secure training from scratch.
Performance of different defense methods against various attacks on CIFAR-10 and CIFAR-100 is demonstrated in Table 1.
An ideal defense method is supposed to increase ACC while keep ASR as low as possible.
Thus, a larger ACC-ASR indicates a better method.
We mark the best result in boldface.
Note that we only report results on successful attacks where ASR is higher than 85%.
[Table]
DBD fails in most attacks on CIFAR-10 and CIFAR-100, probably due to the failure of the symmetric cross-entropy to distinguish samples.
By comparison, the good performance reached by D-ST illustrates the accurate distinguishment from the FCT-based SD module.
We additionally introduce two feasible baselines without requiring special knowledge.
Baseline1 first uses SimCLR [31] to train the feature extractor and then trains the classifier on the poisoned dataset with standard supervised learning.
By comparison, Baseline2 leverages S-CTL [30] to train the feature extractor.
We focus on discussing the effect of different extractor-training algorithms on defense performance.
More discussions are in the later experiments.
Baseline 1 reveals that training extractor without labels may result in low ASR (<5% / <20% in some cases on CIFAR-10 / CIFAR-100), but will sacrifice ACC definitely (84.65% / 56.47% on average).
While Baseline 2 demonstrates that training extractor with all labels guarantees high ACC (93.43% / 71.12% on average), but also brings high ASR (93.73% / 100% on average).
By contrast, results of D-ST illustrate the effectiveness of ST module in training the feature extractor in a secure way since ACC is high (91.66% / 67.79% on average) and ASR (1.21% / 0.05% on average) is extremely low.
Effectiveness of D-BR method.
Then, we consider paradigm 2—backdoor removal.
Defense performance of different defense methods against various attacks on CIFAR-10 and CIFAR-100 is demonstrated in Table 2.
Results on ImageNet is shown in Table 5 in Appendix D.
Results on CIFAR-10.
We discover that except for FT and MCR, other selected methods generally reduce ACC markedly.
Additionally, they have two common disadvantages.
(1) They can not take effect on all attacks.
For example, ANP can defend against Trojan and Blend attacks (ASR < 1%) while fails in clean-label attacks, i.e. SIG and CL (ASR > 10%).
(2) There exists at least one attack that can disable the methods (ASR > 50%).
By contrast, the proposed D-BR method overcomes these drawbacks.
It not only maintains ACC as large as that of backdoored model, but also reduces ASR to less than 1% on all attacks, verifying the effectiveness of the BR module and the high precision of the distinguishment conducted by the SD module.
[Table]
Results on CIFAR-100.
Although FT and NAD have a relatively high ACC (> 68%), they fail to reduce ASR (91.46% and 74.83% on average).
While ANP and ABL can decrease ASR to less than 0.1%, they sacrifice too much ACC (66.59% and 57.69% on average).
Among the selected methods,
MCR performs the best (ACC = 73.80% on average, ASR < 0.5% in three cases), but it still fails to defend against the Blend-Kitty attack (ASR = 20.60%).
Note that MCR requires extra clean data.
In contrast, D-BR keeps ACC higher than 72%, while reduces ASR to almost 0% without any extra clean data.
4.3 Ablation studies
Effectiveness of the SD module.
Here, we aim to study the effectiveness of the SD module.
Specifically, we will show how our proposed FCT metric, performs better than other metrics, under the backdoor-removal paradigm for illustration.
To this end, we select three existing metrics for comparison.
Spectral signatures [5] specifies the metric as the correlation with the top singular vector of the covariance matrix of feature representations .
DBD [2] assigns symmetric cross-entropy loss as the metric.
The metric used in ABL [37] is loss value applied with local gradient ascent .
The metric values of clean samples are smaller than those of poisoned samples according to the former two metrics, while larger for the third metric.
For fair comparison, we uniformly set α c = 20% , α p = 5% .
We first apply the metric-replaced SD module on the poisoned training set, and then conduct the BR module based on the distinguished samples.
Results are shown in Fig.
[Figure]
The height of the blue bar above the orange bar suggests how well the metric could distinguish.
As shown in Fig.
4 (a,b,d), orange bars are all higher than blue bars for Spectral signatures and DBD, indicating metrics of which fail to distinguish in BadNets, Blend and CL attacks.
In contrast, the metric of ABL is reliable since ABL performs well in most cases except for Blend attack where ASR is 24.72%.
By comparison, our proposed FCT metric could distinguish samples stably well, resulting in extremely low ASR (< 0.5%) on all attacks.
We attribute the success to that FCT exploits the sensitivity of poisoned samples, which is mainly due to the overfitting to trigger by the backdoored model that exists in all backdoor attacks we have evaluated in this paper.
Effectiveness of the BR module.
Here, we focus on studying the effectiveness of the BR module.
Specifically, we aim to show how the iterative learning algorithm consisting of unlearning and relearning performs better than the pure unlearning adopted by [37] or pure relearning .
To this end, we first conduct the SD module, and then apply different learning algorithms.
For the three algorithms, we run 20 epochs on CIFAR-10 and record the variations of Test ACC and Test ASR which are illustrated as Fig.
[Figure]
Although pure unlearning (red lines) effectively decreases ASR, it could hardy maintain ACC, showing a downward trend.
The results indicate a strict requirement for choosing the number of unlearning epochs.
On the contrary, pure relearning (green lines) can keep ACC stably high, but it takes tiny effect in reducing ASR.
By contrast, unlearning+relearning (blue lines) combines their advantages and successfully diminishes ASR while maintains ACC.
ACC and ASR steadily converge to high and low values, respectively, validating the effectiveness and the stability of the BR module.
Effectiveness of the ST module.
Here, we aim to study the effectiveness of the ST Module.
Backdoor can be injected during training the feature extractor f
θ e and the classifier h θ c .
The final defense performance of g θ depends on how well f θ e and h θ c inhibit backdoor.
Firstly, we want to show how SS-CTL performs better than CTL or S-CTL in training a secure feature extractor f θ e .
To this end, we train f θ e with different learning algorithms and then uniformly leverage
L MCE to train h θ c .
Training f θ e with CTL, as shown in the first row in Table 3, guarantees low ASR, but the low ACC turns into a tradeoff.
Note that the low ASR is the joint effort of CTL and L MCE .
And the usage of CTL does not indicate low ASR definitely, but it indeed reduces the possibility of backdoor injection in f θ e
The second row illustrates that S-CTL could bring high ACC and the potentially high ASR, as seen in the SIG and CL attacks.
Since all labels (including poisoned) are used in this scenario, ACC is reasonably high.
Besides, backdoor is already injected into f θ e .
But due to the inhibition effect of L MCE , the ASR of g θ may not be high.
For example, in dirty-label attacks, i.e. BadNets, Trojan and Blend attacks, ASR is almost 0%.
While in clean-label attacks, i.e. SIG and CL attacks, L MCE can not withstand the backdoor injected in f θ e , so the ASR is almost 100%.
Hence, in order to establish a reliable defense module, f θ e should be trained in a more secure way.
In comparison, the third row demonstrates the superior defense performance of SS-CTL, illustrating that the ST module securely bridges genuinely clean intra-class samples together which are distinguished by the SD module.
[Table]
Secondly, we explore how L MCE affects the defense performance of h θ c .
For clarity, we denote
L 1 ≡ − 1 | ˆ D c | (cid:80) ( x ,y ) ∈
ˆ D c log[ h θ c ( f θ e ( x ))]
y and L 2 ≡ 1 | ˆ D p | · (cid:80) ( x ,y ) ∈
ˆ D p log[ h θ c ( f θ e ( x ))]
y .
We have
L MCE =
L 1 + λ p L 2 .
Generally, if knowing clean samples, the defender will train h θ c with L 1 .
So here, we aim to show how our proposed L 2 and the trade-off parameter λ
p affect h θ c .
To this end, we first fix f θ e learned by SS-CTL and then apply L MCE with λ p = 0 , 0 . 001 , 0 . 01 , 0 .
1 , 1 on h θ c .
Results are shown in Fig.
In the previous experiments, we adopt λ p = 0 .
The comparison between λ p = 0 and λ p ̸ = 0 in the right figure illustrates that L 2 can effectively reduce ASR.
When comparing different λ p ̸ = 0 in the left figure, we discover that as λ p increases, there is a trend of decrease in ACC.
We infer that since L 2 drops faster than L 1 , namely unlearning is faster than relearning, adding weights to L 2 makes h θ c focus on unlearning instead of relearning, leading to the low ACC.
Therefore, we conclude that L 2 helps to inhibit backdoor in h θ c , but its weight should not be too large.
λ p = 0 .
001 is considered to be an appropriate choice.
In summary, we have empirically validated the effectiveness of each individual module, and shown the flexibility of our method to combine with other existing modules.
Appendix.
Due to the space limit, more results and analysis will be presented in Appendix.
The overall structure of the Appendix is listed as follows.
• Appendix A: more algorithmic details and analysis on the proposed method.
• Appendix B: more details on semi-supervised contrastive learning (SS-CTL).
• Appendix C: more implementation details.
• Appendix D: results on ImageNet.
• Appendix E: performance with different data transformations τ .
• Appendix F: performance with different proportion values α c , α p .
• Appendix G: performance with different poisoning rates.
• Appendix H: performance with different model architectures and feature dimensionalities.
• Appendix I: complexities of two proposed methods.
Conclusions
In this paper, we reveal the sensitivity of poisoned samples to transformations and propose a sensitivity metric, called FCT.
Besides, we propose three modules—the SD module to distinguish between clean and poisoned samples, the ST module to train a secure model from scratch and the BR module to remove backdoor—which constitute two defense methods, i.e. D-ST and D-BR, to defend under two different defense paradigms.
Extensive experiments have demonstrated the effectiveness of each individual module and also the proposed defense methods.
Broader impact
Poisoning-based backdoor attacks are severe threats to the learning paradigm of learning a DNN model based on the training set from some untrustworthy sources.
This work reveals the sensitivity of poisoned samples in the backdoored model, which will help people to better understand the inner mechanism of backdoor attacks.
The proposed two effective defense methods can not only significantly mitigate the threat of existing poisoning based backdoor attacks, but also serve as the new baseline for developing more advanced attack methods in future.
Acknowledgments and disclosure of funding
Baoyuan Wu is supported by the NSFC Fund under grant No.62076213, Shenzhen Science and
Technology Program under grant No.
RCYX20210609103057050, No.
GXWD2020123110572200220200901175001001, and No.
ZDSYS20211021111415025. Haoqian Wang is supported by the NSFC Fund under grant No.61831014, Shenzhen Science and Technology Program under grant No.CJGJZD20200617102601004 and No.
JSGG20210802153150005.
A More algorithmic details and analysis on the proposed method A.1 Sample-distinguishment module
We summarize the SD module in Algorithm 1.
Note that it is a simplified version for an easy understanding and the detailed version is illustrated in Algorithm 2.
Algorithm
Input
Output separated clean (poisoned) samples ˆ D
( ˆ D remaining uncertain samples
D backdoor into model 1: Train a backdoored model g θ by minimizing the standard cross-entropy on ¯ D train for e t epochs 2: Let f ( · ) denote the feature representation of g θ # Assign poisoned samples 3: Initialize S ← {} 4: for each ( x y
∈ ¯ D train do 5:
S append f
6: end for 7: S ′ ← sorted
S ascending order 8: γ c ←
S c
D train γ p ←
S p
D samples 9: for each ( x , y ) ∈
¯ D train do 10: if ∆ trans ( x ; τ, f ) ≤ γ c then 11: ˆ D c .append( ( x , y ) ) 12: else if ∆ trans ( x ; τ, f ) ≥ γ p then 13: ˆ D p .append( ( x , y ) ) 14: else 15: ˆ D u .append
( ( x , y ) ) 16: end if 17: end for 18: return ˆ D c , ˆ D p , ˆ D u
We omit some algorithmic details and state the SD module in Algorithm 1 for an easy understanding.
Here, we continue to elaborate our mechanism in Algorithm 2.
The main supplement is the step of fine-tuning the backdoored model, which is highlighted in blue.
Empirically, we discover that e t = 2 is enough for injecting backdoor into model g θ for the Train
ASR is already higher than 90%.
When we then use Algorithm 1 to distinguish samples, we observe that it works under dirty-label attacks (shown in Fig.
8(a,b,d,e)) since the distributions of clean and poisoned samples are quite distinct.
However, it doesn’t work under clean-label attacks (shown in
Fig.
8(c,f)) since poisoned samples are mixed up with clean samples.
We first analyze the reason for the success under dirty-label attacks and then give an assumption of the reason for the failure under the clean-label attacks.
In the feature space of a backdoored model [2, 24], poisoned samples are congregated into one cluster, called the poisoned cluster , as illustrated in black in Fig.
Let C ic , C jp denote the cluster made up of clean samples labelled as i and poisoned samples labelled as j , respectively.
We formulate the intra-cluster distance between C ic and C jp as follows.
[Equation]
where c i c , c j p are the center of C i c and C j p , respectively.
Under dirty-label attacks, when the trigger is perturbed by the the transformations τ , poisoned samples are no longer located at the poisoned cluster
C tp (see black points).
Instead, they move towards their true-class clusters C ic ( i ̸ = t ) (see non-black points).
The long intra-cluster distance between the two clusters, namely d intra ( C ic , C tp )( i ̸ = t ) , makes the ∆ trans ( x ; τ, f ) of poisoned samples large.
By contrast, under clean-label attacks where poisoned samples are labeled as their true classes ( i.e. ground-truth classes), the poisoned cluster C tp
Input: the poisoned training set ¯ D train a set data transformations τ randomly model g fine clean samples poisoned
Output separated clean samples poisoned samples
1: Train g θ with the standard cross-entropy loss on ¯ D train for e t epochs 2: Let f ( · ) denote the feature representation of g θ 3: Let c i = (cid:80) xi ∈
Ci f ( x i ) | C i | denote the center of cluster C i , which consists of samples labelled as i 4: Let L intra = 1 N 2 (cid:80) i,j ∈{ 0 , 1 ...N − 1 } ,i ̸
=
j <c i ,c j > ∥ c i ∥ 2 ·∥ c j ∥5: Fine-tune g θ with L intra on ¯ D train for e ft epochs # Assign clean
6: Initialize S ← {} 7: for each ( x y
∈ ¯ D train do 8:
S .append( ∆ trans ( x τ f ) ) 9: end for 10: S ′ ← sorted ( S ) in ascending order 11: γ c ← [ S ′ ]
α c | ¯ D train | , γ p ← [ S ′ ] (
−
) | ¯ D train | # samples 12: for each
( x , y ) ∈
¯ D train do 13: if ∆ trans ( x ; τ, f ) ≤ γ c then 14: ˆ D c .append( ( x , y ) )
15: else if ∆ trans ( x ; τ, f ) ≥ γ p then 16: ˆ D p .append( ( x , y ) ) 17: else 18: ˆ D u .append
( ( x , y ) ) 19: end if 20: end for 21: return ˆ D c , ˆ D p , ˆ D u ( adjoins C tc (see red points) which is also the true-class cluster, leading to the short intra-cluster distance, namely d intra ( C tc , C tp ) , and the resulting small ∆ trans ( x ; τ, f ) of poisoned samples.
In order to validate our assumption, we fine-tune the model g θ backdoored by clean-label attacks with the intra-cluster loss L pintra for e f = 5 epochs, which aims to enlarge d intra ( C tc , C tp ) .
[Equation]
where N denote the number of classes.
L pintra can successfully increase d intra ( C tc , C tp ) from 0.25 to 17.19.
Then, we reuse the SD module and find that clean and poisoned samples can be well separated.
However, in the realistic scenario, we can not exactly know the poisoned cluster.
So, the fine-tuning strategy above is infeasible.
Instead, we fine-tune the model g with the intra-class loss L intra for e f = 5 epochs.
Denote C i as the cluster made up of all samples labelled as i .
L intra is formulated as:
[Equation]
where c i = (cid:80) xi ∈ C i f ( x i ) | C i | is the center of C i .
Empirically, we discover that L intra also helps to enlarge d intra ( C tc , C tp ) , making it increase to 4.96.
Subsequently, with the application of the SD module, poisoned samples can be separated from clean samples (Fig.
8(i,l)).
Note that L intra can even boost the separation under dirty-label attacks (Fig. 8(g,h,j,k)).
In conclusion, combined with the fine-tuning step, our proposed SD module can work under all kinds of attacks.
[Figure]
A.2 Distinguishment and secure training (D-ST) method
The D-ST method is shown in Algorithm 3.
Algorithm 3 Distinguishment and secure training (D-ST) method
Input
# Sample distinguishment
1: Separate ¯ D train to three subsets, i.e. , ˆ D p , ˆ D c , ˆ D u (see Section 3.2)
# Secure training module
2: Stage 1 : learning the feature extractor
f θ e through SS-CTL, i.e. , minimizing L SS − CTL on ¯ D train (see Eq.
(2)), for e e epochs 3: Stage 2 : learning the classifier h θ c via minimizing the mixed cross-entropy loss, i.e. , minimizing L MCE on ¯ D c and ¯ D p (see Eq.
(3)), for e c epochs
A.3 Distinguishment and backdoor removal (D-BR) method
The D-BR method is shown in Algorithm 4.
Algorithm 4 Distinguishment and backdoor removal (
Sample
1: Separate ¯ D train to three subsets, i.e. , ˆ D p , ˆ D c , ˆ D u (see Section 3.2)
Backdoor removal module
2: Train a backdoored model g θ by minimizing the standard cross-entropy on ¯ D train for e t epochs 3: for each e ∈ { 1 , . . .
, e ft } do 4: Unlearning
[Figure]
B More details on semi-supervised contrastive learning (SS-CTL)
We explain ℓ CTL and ℓ S − CTL in details here.
For clarity, we denote f θ e ( ˜ x (1) i ) as f 1 i and f θ e ( ˜ x (2) i ) as f 2 i .
[Equation]
If y i =
y j
[Equation]
then
[Equation]
) exp ( f 1 i · f kj /T ) } −3 { log exp ( f 2 i · f 1 i /T )
(cid:80)
[Table] [Table]
|
[Table]
train | j =
[Table]
I ( f 2 i ̸ = f kj ) exp ( f 2 i · f kj /T )
+ log exp ( f 2 i · f 1 j /T )
(cid:80) k
=
[Table] [Table]
(cid:80) |
[Table]
D train | j =
I ( f 2 i ̸ = f kj ) exp ( f 2 i · f kj /T )
+ log exp ( f 2 i · f 2 j /T )
(cid:80) k
[Table] [Table]
I ( f
[Table]
i ̸
= f kj ) exp
( f
[Table]
· f kj /
[Table]
) } −3 { log exp (
[Table] [Table]
) (cid:80)
[Table]
I ( f
[Table]
̸ = f kj ) exp ( f
[Table]
f kj /T )
+ log exp ( f
[Table]
j · f
[Table]
i /T )
(cid:80)
[Table] [Table]
|
[Table]
train | j =
[Table]
I ( f
[Table]
j ̸ = f kj ) exp ( f 1 j · f kj /T )
+ log exp ( f 1 j · f 2 j /T )
(cid:80)
[Table] [Table]
|
[Table]
D train | j =
I ( f 1 j ̸ = f kj ) exp ( f 1 j · f kj /T ) } −3 { log exp ( f 2 j · f 1 i /T )
(cid:80) k
[Table] [Table]
|
[Table]
D train | j =
[Table]
I ( f 2 j ̸ = f kj ) exp ( f 2 j · f kj /T )
+ log exp ( f 2 j · f 2 i /T )
[Equation]
Else,
[Equation]
C More implementation details
All experiments are run on 3 RTX 5000 GPUs and are repeated over 5 runs with different random seeds.
C.1 Dataset details
We conduct experiments on three benchmark datasets, CIFAR-10 [3], CIFAR-100 [3] and an ImageNet subset [33].
We use the ImageNet subset provided by [9], which contains 200 classes withtraining samples (500 samples per class) and 10000 testing samples (50 samples per class)
[Table]
C.2 Model details
We conduct experiments with ResNet-18 [34] as base model.
For the ST module, we leverage the model provided by [30] on CIFAR-10 and CIFAR-100.
For the BR module, we use the model implemented by codes 3 on CIFAR-10, codes 4 on CIFAR-100 and [9] on the ImageNet subset.
C.3 Attack details
Referring to Section
[Table]
we can categorize existing poisoning-based backdoor attacks according to various criteria.
As shown in Table
[Table]
we choose eight attacks to guarantee at least one method in each category.
We also give some examples in Fig.
9 to display the patterns of different triggers
[Table] [Table]
[Figure]
Besides, we provide an explanation for the poisoning rate configured in attacks.
Take CIFAR-
(50000 training samples) as an example, if the poisoning rate is 10%, then 10% of samples with non-target classes are poisoned in dirty-label (also all2one) attacks (4500 samples) while 10% of samples with target class are poisoned in clean-label (also all2one) attacks (500 samples).
C.4 Defense details
Here, we complement some details not mentioned in the main paper.
• FT:
We fine-tune the backdoored model on 1% of the clean training samples with 100 epochs.
• MCR:
We adopt t = 0.3.
• Baseline
: We first use self-supervised contrastive learning, SimCLR [31] particularly, to train the feature extractor.
During the process, data is regarded as unlabeled.
Thus, the positive of a sample is its augmented version.
We then use standard supervised learning to train the classifier on the poisoned dataset.
• Baseline
We first use supervised contrastive learning, SupContrast [30] particularly, to train the feature extractor.
During the process, the positives of a sample are its augmented version and samples with the same label (and also their augmented versions).
Note that since there are poisoned samples in the dataset, ‘samples with the same label’ may not be the genuine intra-class samples.
We then use standard supervised learning to train the classifier on the poisoned dataset.
C.5 Details of proposed method
Choices of parameters.
Here, we complement some details not mentioned in the main paper.
• Algorithm 2: τ = rotate+affine, e t = 2 , e f = 10 , α c = 0 .
20 , α p = 0 .
No other data augmentations are used since they would hinder the effect of backdoor implantation.
• Algorithm 3: e e = 200 , e c = 10 , λ p = 0 .
Applied data augmentations are the same as [31].
• Algorithm 4: e t = 200 , e ft = 20 .
Data augmentations applied on CIFAR-10 are random crop, random horizontal flip and normalization.
Data augmentations applied on CIFAR-100 are random crop, random horizontal flip, random rotation and normalization.
Data augmentations applied on ImageNet are random rotation, random horizontal flip and normalization.
Choices of different data transformations τ .
In Section 4.3, we select six types of τ to change the pattern or location of the trigger, which are listed as follows and shown in Fig.
• Rotate: Rotate an image by a random degree less than 180 ◦ .
• Affine: Translate an image horizontally and vertically.
• Flip: Flip an image horizontally.
Crop: Crop an image at a random location.
• Blur: Perform Gaussian blurring on an image by given kernel.
• Erase: Select a random rectangle region in an image and erase its pixels.
D Results on ImageNet
Here, we exhibit the performance of our proposed methods on the ImageNet subset [33, 9] in Table 5.
Since we aim to study successful backdoor attacks in this paper where ASR of the backdoored model is higher than 85%, results on unsuccessful attacks are not reported.
For instance, the increase of number of classes adds to the difficulty of clean-label attacks (ASR < 5%).
So, we do not report results on clean-label attacks.
Performance of the D-ST method.
Note that SupCon [30] does not launch open-souce codes for
ImageNet, so the related results are not reported here.
Performance of the D-BR method.
We do not report results of ANP and MCR on ImageNet since they require particular modification to model and do not have an open-source version for ImageNet.
As illustrated in Table 5, among the selected methods, ABL performs the best with ACC as 80.16% and ASR as 23.94% on average.
However, it still fails on Blend-Strip attack (ASR = 95.75%).
In contrast, D-BR performs steadily well on all attacks, even reducing ASR to 0% in most cases, which demonstrates the effectiveness of the SD module and the BR module.
[Figure]
[Table]
E Performance with different data transformations
Here
[Table]
we study how τ affects the performance of the SD module, specifically the precision of ˆ D c (clean-precision) and ˆ D p (poison-precision).
We select six types of τ which change the pattern or location of trigger, detailed in Appendix C.5.
In order to strengthen the perturbation to trigger, we adopt a combined τ , combining two of the six choices.
The total 36 combinations are applied on the poisoned training set and the precision is illustrated in Fig.
11 and Appendix E.
[Figure]
from CIFAR-10.
Apparently, the lighter the color, the higher the precision.
One worse case is applying ‘flip’ twice, equal to not using any transformation.
Hence, genuinely poisoned samples can not be identified since trigger is not perturbed, and the poison-precision is extremely low.
Another slightly worse case is applying ‘erase’ as the second transformation.
We infer that adding a noise patch on the random position of an image, ‘erase’ may fail to obscure the trigger.
In general, the SD module is highly effective in identifying genuinely clean and poisoned samples under most combined τ with 100% precision, also validating the sensitivity of poisoned samples to transformations.
Note that the 100% clean-precision does not mean all distinguished clean samples being genuinely clean.
In fact, we discover that there are several genuinely poisoned samples, like five, in the distinguished clean samples.
These tiny amount of poisoned samples are not reflected on the clean-precision due to the round-off.
Additionally, we show more results in Fig.
As a clean-label attack, SIG has high clean-precision but low poison-precision, which is different from BadNets and Blend attacks.
We infer that the difference comes from the definition of the attack.
Although we set a uniform poisoning rate (10%) for all attacks, the total number of poisoned samples is various.
Take CIFAR-10 as an example where there are 50000 training samples (5000 samples per class).
In a dirty-label (also all2one) attack, excluding the 5000 samples from target class, there are 4500 poisoned samples totally.
By contrast, in a clean-label (also all2one) attack, since the attacker only targets at samples from target class, there are only 500 poisoned samples.
Recalling α p = 5% , we separate 2500 samples as poisoned samples, which are much more than the actual 500 poisoned samples.
Therefore, the poison-precision under a clean-label attack is quite low.
Nevertheless, the seemingly adverse fact actually helps to validate one of the benefits of our proposed defense modules (the ST module and the BR module), which is the robustness to the wrong distinguishment in the SD module .
Specifically, even with a low poison-precision, the proposed two defense methods, i.e. D-ST (SD module + ST module) and D-BR (SD module + BR module), are still effective in defending against clean-label attacks, which demonstrates that the two defense modules are robust to the inaccurate distinguishment in the SD module.
The robustness probably attributes to the high clean-precision (> 99%).
It could help our proposed methods ‘survive’ under extreme circumstances where there are only tiny amount of poisoned samples, further proved in Appendix G empirically.
Hence, despite some newly introduced attacks focusing on reducing the poisoning rate to add to the difficulty in defense, our method is still capable of coping with these potential attacks.
F Performance with different proportion values
In this part, we explore how α c and α p , specified in the SD module, influence the final defense performance.
To this end, we first conduct the SD module with different α c and α p , and then test the performance under the backdoor-removal paradigm for illustration.
Recall the settings in the previous experiments where α c = 20% and α p = 5% .
First, we fix α p as 5% and vary α c from 0% to 80%.
Results are illustrated in Fig.
13 (a).
Comparing α c = 0 and others, we see that ˆ D c plays an important role in boosting ACC.
As shown in the left figure, the model may suffer from extremely low ACC without these samples.
As α c grows, ACC increases steadily and finally converges.
However, when α c is too large (eg. 80%), D c may contain genuinely poisoned samples, resulting in the rise of ASR, as shown in the right figure.
Hence, 20% and 40% are appropriate choices for α c .
Then, we fix α c as 20% and alter α p from 0% to 20%.
Results are depicted in Fig.
13 (b).
The comparisons between α p = 0 and others validate the effect of ˆ D p on decreasing ASR.
Specifically, as shown in the right figure, ASR could be extremely high if none of these samples are available.
While with the increase of α p , ASR declines steadily and finally converges.
Nevertheless, excessive ˆ D p could hurt the model and lead to a reduction in ACC, as shown in the left figure.
We infer that the excessive samples may include a certain amount of genuinely clean samples and unlearning which is deleterious to the model.
Therefore, a moderate α p (eg. 5%) is preferred.
[Figure]
G Performance with different poisoning rates
In this section, we aim to show performance of our proposed methods, namely D-ST and D-BR, under different poisoning rates.
To this end, we set the poisoning rate as 5% and 20% respectively, and conduct the methods on CIFAR-10 poisoned by different attacks.
Here, we choose four representative attacks for illustration.
Note that we keep α c = 20% and α p = 5% as before.
The results are shown in Table 6.
[Table]
Under the poisoning rate of 20%, D-BR can reduce the average ASR from 96.72% to 2.36%, while keep ACC barely unchanged.
Meanwhile, D-ST can train a clean model from scratch with ACC as 90.33% and ASR as 0.40% on average.
When the poisoning rate is as small as 5%, D-BR can still manage to decrease ASR by 92.05% on average with a mere drop in ACC (2.62% on average).
The model trained with D-ST is relatively clean with ACC as 89.59% and ASR as 2.87% on average.
The performance is slightly worse than that of 20%.
We infer that in this case, poisoning rate and α p both equal to 5%.
Since we can not find all poisoned samples accurately, the poisoned samples separated by the proposed SD module definitely contain some extent of genuinely clean samples, which is harmful to the performance of the final model.
Hence, better performance can be achieved with a careful adjustment in α c and α p .
H Performance with different model architectures and feature dimensionalities
In addition to the ResNet-18 we have tested, here we conduct experiments on another three mainstream model architectures, including ResNet-50, VGG-19[38] and DenseNet-161[39], with the same parameter setting.
Results are shown in Table 7.
Besides, we uniformly choose the output of the penultimate layer as the feature representation, resulting in different dimensionalities.
Thus, these results can also reflect the sensitivity of our methods to feature dimensionalities.
[Table]
In the following, we will analyze the effectiveness of different modules.
Effectiveness of SD module .
Both ResNet-18 and VGG-19 achieve 100% precision.
In contrast, the poison-precision of ResNet-50 and DenseNet-161 is relatively low, indicating that with the increase in the dimensionality, the gap between the FCT of clean and poisoned samples may be smaller.
However, since their clean-precision is as high as about 100% and that our methods are robust to wrong distinguishment as analyzed in Appendix E, this low poison-precision won’t significantly influence the final performance which is analyzed subsequently.
Effectiveness of BR module .
Compared with the performance of the backdoored model, our proposed BR module ( i.e. , the D-BR method) could reduce ASR from 100% to 0% on the three new architectures.
Meanwhile, ACC drops by 2.41% at most.
Effectiveness of ST module .
Note that since S-CTL [30] (used in stage 1 of D-ST) only released codes for the ResNet architecture, here we do not evaluate the ST module on VGG-19 and DenseNet161.
Compared with the backdoored model which directly employs the supervised learning, our proposed ST module ( i.e. , the D-ST method) could train a secure model from scratch.
In summary, our methods and the chosen hyper-parameters are generalizable across model architectures.
However, we also found that the feature dimensionality may affect the performance, as the
Euclidean distance used in our FCT metric may not be suitable for high dimensonal feature space, which will be explored in the future.
I Complexity of the proposed two methods
Denote the complexity of one forward and backward pass in feature extractor as a e while that in classifier as a c .
Complexity of D-ST.
Given the training epochs detailed in Appendix C.5, the complexity is O (cid:0) e e · a e · [2 · ( | ˆ D p | + | ˆ D u | )+12 ·| ˆ D c | 2 ] (cid:1) and O (cid:0) e c · a c · [ | ˆ D c | + | ˆ D p | ] (cid:1) for stage 1 and stage 2, respectively.
We adopt e e = 200 , e c = 10 in the experiments.
Complexity of D-BR.
The complexity of training a backdoored model is O (cid:0) e t · ( a e + a c ) · | ¯ D train | (cid:1) .
The complexity of the BR module is O (cid:0) e ft · ( a e + a c ) · [ | ˆ D c | + | ˆ D p | ] (cid:1) .
We adopt e t = 200 , e ft =in the experiments.
Take CIFAR-10 as an example.
| ¯ D train | = 50000 .
| ˆ D c | and | ˆ D p | are approximately 10000 and 2500, respectively.
Compared with a e , a c is relatively small and can be omitted.
Thus, the complexity of
D-ST is higher than that of D-BR by about O (cid:0) 12 · a e · | ˆ D c | 2 (cid:1) .