Identify and Remove Backdoor Neurons through Clean-Poisoned Mixture Distribution
Anonymous Author(s)
AffiliationAddress
Abstract
Convolutional neural networks (CNN) can be manipulated to perform specificbehaviors when encountering a particular trigger pattern without affecting theperformance on normal samples, which is referred to as backdoor attack.
Backdoorattack is usually achieved by injecting a small proportion of poisoned samplesinto the training set, through which the victim trains a model embedded with thedesignated backdoor.
In this work, we demonstrate that the backdoor neurons inan infected neural network have a mixture of two distributions with significantlydifferent moments, formed by benign samples and poisoned samples, respectively.
This property is shown to be attack-invariant and allows us to efficiently locatethe backdoor neurons.
On this basis, we make several realistic assumptions onthe neuron activation distributions, and propose two backdoor neuron detectionstrategies based on (1) the differential entropy of the neurons, and (2) the KLdivergence between the benign sample distribution and a poisoned statistics basedhypothetical distribution.
Experimental results show that our proposed defensestrategies are both efficient and effective against various backdoor attacks.
Introduction
Convolutional neural networks (CNNs) have achieved tremendous success during the past few yearsin a wide range of areas.
However, training a CNN from scratch involves a large amount of data andexpensive computational cost, which is sometimes infeasible.
A more practical strategy is to obtainpretrained models or utilize public datasets from a third party, which brings convenience but alsoraises severe security problems into the deployment of models.
For example, a malicious third partymay provide pretrained models embedded with a designated backdoor, such that the model will havea predefined response to some specific pattern, which is also called the trigger .
More realistically, theattacker can inject only a small proportion of malicious data into the public dataset to mislead thetrained model, which is referred to as backdoor poisoning attacks [25].
For instance, the maliciousdata can be created by patching a particular pattern into the benign data and changing the label tothe desired target.
The correlation of the trigger and the specified target label will be learned by themodels during the training time.
In this way, the infected model will misclassify the input to theattack target when the pattern is patched, while behave normally otherwise, as shown in Figure 1.
Due to the limited understanding of CNNs, we are not clear on the formation mechanism of backdoorbehaviors.
However, it was empirically found that a infected model always possesses one or more neurons that have high correlation with the trigger activation, and pruning these neurons can significantlyalleviate the backdoor behaviors, while retaining the model performance [42, 28, 6].
Nevertheless,how to precisely find out these backdoor neurons in a infected model is still a challenging problem,and has attracted a lot of attentions from the community. [Figure]
In this work, we will take an inspection on the pre-activation distributions of infected models.
Ingeneral, the activations in each neuron follow an unimodal distribution that can be approximated by a
Gaussian distribution.
Based on the maximum entropy property of the Gaussian distribution, thesebenign neurons should have relatively large entropy on their pre-activation distributions.
However, inthe backdoor neurons, the distribution is always bimodal and can be approximated by a mixture oftwo Gaussian distributions, formed by the benign data and poisoned data, respectively; see Figure 1.
This kind of distribution must have a lower entropy, compared with a Gaussian distribution withthe same finite variance.
Thus, if we standardize the pre-activation distributions in all neurons, thebackdoor neurons should possess a relatively low entropy compared with the benign neurons.
Wetreat the low entropy neurons as outliers and potential backdoor neurons, and prune these neuronsto recover the model without retraining.
However, under another defense setting, in which onlyan infected model and a small set of benign data are provided, we may not be able to observe thebimodal distributions on the backdoor neurons since the poisoned data is absent.
In this case, we canrely on the statistics on Batch Normalization (BN) layers.
Specifically, if the infected model is trainedon poisoned data, the statistics of backdoor neurons recorded in the BN layer will be significantlydifferent from the distribution of only benign data.
More importantly, the mismatch of statistics willnot exist in benign neurons.
Based on the neuron entropy and the statistics discrepancy, we are ableto locate and prune the backdoor neurons to recover the model under two defense settings.
In summary, our contributions include:
We take a deep inspection on the infected model, and summarize the law of pre-activationdistributions on poisoned dataset.
We find that (1) the standardized entropy of backdoorneurons can be significantly lower than benign neurons, and (2) the BN statistics in infectedmodel are mismatched with the benign sample statistics.2.
We propose to prune the potential backdoor neurons based on either the sample entropy orthe statistics discrepancy, depending on the defense settings.
Under certain assumptions,we claim that both the proposed indices can successfully separate the benign neurons andbackdoor neurons by an appropriate threshold.3.
We conduct extensive experiments to verify our assumptions and evaluate our proposedmethods, and achieve the state-of-the-art results under two different defense settings.
Related work
In this section, we detailedly discuss recent works in backdoor attack and defense.
2.1 Backdoor attacks
The concept of backdoor attack is first introduced in [14], where the adversary injects a small setof targeted label-flipped data stamped with a small and specific trigger into the training set duringthe DNN training, leading to a misclassification when predicting the samples with such trigger.
Tomake the trigger pattern even more invisible to human beings, the blending strategy is used in [5] to generate poison images, while the form of natural reflection is utilized in trigger design in [31].
The input image is perturbed in [40] to keep its content consistent with the target label such that themodel better memorizes the trigger pattern, and keep it imperceptible to human beings.
Moreover,the multi-target and multi-trigger attacks are proposed in [44, 33], and make the attack more flexibleand covert.
Recently, some sample specific trigger design strategies [27] are proposed, making thedefense against such backdoor attack much harder.
Generally, the above attacks can be referred asthe poisoning based backdoor attacks .
Under some settings, the attackers can control the training process to inject the backdoor withoutmodifying the training data, referred as the non-poisoning based backdoor attacks .
This is achievedin [30, 36, 4] through targeted modification of the neurons’ weight in a network.
Such attacks willnot be evaluated in our work due to its strong attack setting.
2.2 Backdoor defenses
Training stage defense.
Under such setting, the defender has access to the training process, so thatthey can detect and filter the poisoned data or add some restrictions to suppress the backdoor effectin training.
Since the poisoned data can be regarded as outliers, different strategies are applied in[10, 12, 1, 39, 15], such as the robust statistics in feature space and input perturbation techniquesto filter them out of training data.
Other methods aim at suppressing the backdoor effect duringtraining phase with strong data augmentation methods [2] [26] [35] [32] including CutMix [45], Flip , ShrinkPad [26], CutOut and MaxUp [13], or differential privacy constraints [11, 18].
Model post-processing defense.
Sometimes the defenders are only given a suspicious DNN modelwithout access to the training process or the full training set.
Therefore, they must eliminate thebackdoor threat with limited resources, such as a small set of clean data.
A straightforward way isto reconstruct the trigger by adding adversarial perturbations to the input images, and then detoxifythe model with the knowledge of reversed trigger [41].
Some try to find the relationship betweenbackdoor behaviors and the neurons in a DNN model.
Different levels of stimulation to a neuronare introduced in [29] to see how to determine the output activation change, if the model is attacked.
Simple neuron pruning strategies are applied in [6] to repair the model, while adversarial perturbationsare added to the neurons in [42] and precisely prunes the backdoor neurons with more limited cleandata requirement and better performance.
Simple redundant neuron pruning and fine-tuning arecombined together in [28] to erase the backdoor effect.
There are other fine-tuning based methodswith the implementation of knowledge distillation [24, 17], while they may suffer from the hyperparameter tuning and clean accuracy dropping problems due to limited prior knowledge of the attackand over-fitting on the clean set.
Mode connectivity repair technique [46] is also explored to mitigatethe backdoored model.
Recently, the K-Arm optimization [38] is applied in backdoor detection,and outperforms other adversarial perturbation based methods [41, 8], helping curtail the threat ofbackdoor attack.
Preliminaries
3.1 Notations
Consider a multi-class classification problem with C classes.
Let the original training set D
={ ( x i , y i ) }
Ni =1 contains N i.i.d.
sample images x i ∈ R d c × d h × d w and the corresponding labelsy i ∈ { 1 , 2 , ..., C } drawn from X × Y .
Here, we denote by d c , d h and d w the number of neurons, theheight and the width of images, respectively.
In particular, we have d c = 3 for RGB images.
As in Section 2.1, the backdoor poisoning attack involves changes to the input images and thecorresponding labels on a subset D p ⊆ D .
In this work, we define the ratio ρ =
|D p | |D| as the poisoningrate .
We denote the poisoning function to the input images as δ ( x ) .
Consider a neural network F ( x ; θ ) with L layers.
Denote
[Equation]
for 1 ≤ l ≤ L , where f ( l ) is a linear function ( e.g. , convolution) in the l -th layer, and ϕ is a nonlinearactivation function applied element wise.
In this work, we may denote F ( x ; θ ) as F ( x ) or F forsimplicity.
We denote by W ( l ) ∈ R d c ′ ×
d c × d h × d w the weight tensor of a convolutional layer.
To do pruning,we apply a mask M ( l ) ∈ { 0 , 1 } d c ′ ×
d c × d h × d w starting with M ( l ) = 1 d c ′ ×
d c × d h × d w in each layer.
Pruning neurons on the network refers to getting a collection of indices
I = { ( l, k ) i } Ii =1 and setting
M ( l ) k
=
d c × d h × d w if ( l, k ) ∈ I .
The pruned network F −I has the same architecture as F but withall the weight matrices of convolutional layers set to W ( l ) ⊙ M ( l ) , where ⊙ denotes the Hadamardproduct.
3.2 Differential entropy
To measure the uncertainty of a discrete random variable Z , the entropy [37, 7] was defined as
H ( Z ) = − (cid:80)
z ∈ Z p ( z ) log p ( z ) .
At the same time, as an extension of entropy, the differentialentropy was also introduced for a continuous random variable.
More concretely, if Z is a continuousrandom variable, then it was defined as
[Equation]
An important fact about the differential entropy is that, among all the real-valued distributionssupported on ( −∞ , ∞ ) with a specified finite variance, the Gaussian distribution maximizes thedifferential entropy [7].
In this work, the differential entropy (1) will be utilized to identify thedistributions that are far different from a Gaussian distribution.
3.3 Backdoor neurons
It was found that there exist one or more neurons that contribute the most to the backdoor behaviorsin a infected model [42, 28].
If some of or all of these neurons are pruned, the attack success ratewill be reduced greatly [42].
In this work, to better quantify the importance of neurons to backdoorbehaviors, we would like to introduce the sensitivity of neurons to the backdoor by the backdoor losscalculated before and after pruning the neuron:
[Equation]
where F −{ ( l,k ) } is the network after pruning the k -th neuron of the l -th layer.
The backdoor loss isdefined as:
[Equation]
where D CE denotes the cross entropy loss.
The backdoor loss is high when the model is infected, andwill be reduced when the backdoor effect is alleviated.
Using the quantity define in (2), we are now able to find the neurons that are mostly correlated withthe backdoor behaviors.
More concretely, we first set a threshold τ > 0 , and then go through allthe neurons to find the ones with the quantity (2) above τ .
We call them the backdoor neurons , anddenote
[Equation]
3.4 Pre-activation distribution
During the forward propagation of an input x , we denote x ( l ) =
F ( l ) ( x ) ∈ R d ( l ) c
× d ( l ) h × d ( l ) w as theoutput of the l -th layer.
For the k -th neuron of the l -th layer, the pre-activation ϕ ( l ) k
= ϕ
( x ( l ) k ) isdefined as the maximum value of the k -th slice matrix of dimension d ( l ) h × d ( l ) w in x ( l ) .
It is a common assumption that, for every neuron, the pre-activations before the non-linear functionfollow a Gaussian distribution, if the network is randomly initialized and the number of neurons islarge enough [21, 3].
In a trained network, although this assumption may not strictly hold, the preactivation of every neuron can be still regarded as approximately following a Gaussian distribution.
However, in this work, for the first time, we observe a bimodal distribution in the backdoor neurons,two components of which are formed by the benign data and poisoned data, respectively.
Thisphenomenon is shown in Figure 2, where a typical backdoor neuron is compared with the benignneurons.
It can be seen that, after the model is infected, the pre-activation distributions of benignneurons hardly change when the data is poisoned, while the pre-activation distributions of backdoorneurons become significantly different.
 
         ' HQ V L W \ %HQLJQVDPSOHV3RLVRQHGVDPSOHV (a) Benign neuron               ' HQ V L W \ %HQLJQVDPSOHV3RLVRQHGVDPSOHV (b) Backdoor neuron               ' HQ V L W \ %HQLJQVDPSOHV%1LQGXFHGGLVWULEXWLRQ (c) Benign neuron      
    
       ' HQ V L W \ %HQLJQVDPSOHV%1LQGXFHGGLVWULEXWLRQ
[Figure]
Methodology
4.1 Assumptions
A primary assumption is that |B F,τ | > 0 for a poisoned model F and a pre-defined threshold τ > 0 ,where B F,τ is defined as in (3).
As in Section 3.4, we also assume that the pre-activations of allneurons follow a Gaussian mixture distribution, that is:
[Equation]
where ρ is the poisoning rate of the dataset, µ ( l ) k and σ ( l )2 k , ˆ µ ( l ) k
and ˆ σ ( l )2 k are the mean and varianceof { ϕ ( F ( l ) ( x ) k ) :
x ∼ X} , { ϕ ( F ( l ) ( δ ( x )) k ) : x ∼ X} , respectively.
We further assume that:
[Equation]
and
[Equation]
where ϵ > 0 is a small enough value.
Note that, if | µ ( l ) k − ˆ µ ( l ) k | = | σ ( l )2 k − ˆ σ ( l )2 k | = 0 , the pre-activationfollows a Gaussian distribution N ( µ ( l ) k , σ ( l )2 k ) .
4.2 Discrepancy of differential entropy (DDE)
If the pre-activation distributions are standardized (subtracting the mean and dividing the standarddeviation), the differential entropy will be maximized on the benign neuron approximately followinga standard Gaussian distribution N (0 , 1) .
In backdoor neurons, because of the difference of the moments on Gaussian components, the resulting mixture distributions can not be Gaussian distributions,hence the differential entropy must be smaller than h ( Z ) , where Z ∼ N (0 , 1) is the standardized
Gaussian distribution.
Specifically, let ˙ ϕ
( l ) k
= ϕ ( l ) k − µ ( l ) k σ ( l )
k be the standardized pre-activations, then
[Equation]
This inequality gives a guarantee that with an appropriately chosen threshold, the backdoor neuronscan be well separated with the benign neurons.
4.3 Mismatched BN statistics (MBNS)
BN layer involves using the statistics of a mini-batch to normalize the data in each layer for eachneuron.
It is known to be able to smooth the optimization landscape, and has gradually become a default setting of neural networks [19].
When inference, BN uses the fixed statistics obtained byaveraging the sample statistics of mini-batches during training time, including the mean and thevariance.
If the model is trained on a poisoned dataset, BN will record the mean and the varianceof the poison-benign mixed data.
Note that the mean and variance here are not defined on thepre-activations ϕ ( l ) k , but on x ( l ) k .
Based on the above discussions, we know that the poisoned samples(especially the pre-activations) on the backdoor neurons follow a different distribution from thebenign samples.
The recorded statistics during training are actually that of the mixture distribution.
Hence, we can expect that the BN statistics of a trained backdoor neural network are biased.
If weare able to access a small set of benign data, we can calculate an approximation of the true statisticson benign data.
Then we calculate the Kullback-Leibler (KL) divergence [9] between the sampledistribution and the BN induced distribution as the measurement of the bias.
By assuming both of thedistributions follow Gaussian distributions, we have a closed form solution:
[Equation]
where ¯ µ ( l ) k and ¯ σ ( l )2 k are the statistics obtained from benign samples, ˜ µ ( l )2 k and ˜ σ ( l ) k are the BNstatistics.
The backdoor neurons should have abnormally large KL divergences, as illustrated in
Figure 2(c).
4.4 Overview of the two pruning strategies
In Section 3.4, we reveal the discrepancy between the pre-activation distributions in the backdoorneurons and that in the benign neurons.
This enables fast detecting the neurons that are more relatedto the backdoor behaviours.
The index we choose to detect the abnormal neurons depends on whatkind of data we are able to access.
Mixture training data In this case, the victim is given a poisoned training dataset with a specifiedpoisoning rate ρ .
Our goal is to obtain a benign model based on the poisoned dataset.
To achieve this,we first train an infected model F on the poisoned dataset.
The resulting model should have a certainnumber of backdoor neurons based on empirical observation and the assumption.
Since ρ > 0 for thedataset, all the neurons follow Gaussian mixture distributions, and we have h ( ˙ x ( l ) k ) <
h ( ˙ x ( l ) k ′ ) for allk ∈ B F,τ , k ′ / ∈ B F,τ .
This implies that with an appropriate threshold τ ∗ h , we can perfectly separatethe benign neurons and backdoor neurons, which can be formulated as:
[Equation]
Setting the threshold τ ∗ h is crucial to the solution, and it is a trade-off between the accuracy onbenign samples and that on the backdoored samples.
Note that |B ( l ) F,τ | << d ( l ) c .
We can treatthe low entropy neurons as outliers in each layer, and set different thresholds for different layers.
Specifically, let h ( l ) =
[ h ( x ( l ) 1 ) , h ( x ( l ) 2 ) , . .
, h ( x ( l ) d ( l ) c )]
T ∈ R d ( l ) c be a vector of sample entropy ofthe l -th layer calculated from the poisoned dataset.
Then we set τ ( l ) h
= ¯ h ( l ) − u h · s ( l ) h , where¯ h
( l ) =
1 d ( l ) c (cid:80) d ( l ) c k =1 h ( l ) k and s ( l ) h
= (cid:114)d ( l ) c
(cid:80) d ( l ) c k =1 ( h ( l ) k − ¯ h ( l ) ) 2 are the mean and standard deviation ofh ( l ) , u h is a hyperparameter controlling how low the threshold is.
Then we have a set of indices ofpotential backdoor neurons
I h
= { ( l, k ) :
h ( l ) k < τ ( l ) h } .
Finally, we prune the infected model F using
I h , which result in a final model F −I h .
Benign training data This is the case that the victim is given a trained poisoned model F with asmall set of benign data.
Our goal is to utilize the benign data to clean up the poisoned model andeliminate the backdoor threat.
Similar to the pruning process based on differential entropy, we firstconstruct a vector of KL divergences of all neurons for each layer K ( l ) =
[ K ( l ) 1 , K ( l ) 2 , . .
, K ( l ) d ( l ) c ] T ∈
R d ( l ) c according to equation (5).
We set τ ( l ) K = ¯ K ( l ) − u K · s ( l ) K , where ¯ K ( l ) =
1 d ( l ) c (cid:80) d ( l ) c k =1 K ( l ) kand s ( l ) K = (cid:114)
d ( l ) c (cid:80) d ( l ) c k =1 ( K ( l ) k − ¯ K ( l ) ) 2 are the mean and standard deviation of K ( l ) , u K is a hyperparameter.
The set of selected neurons is I K
= { ( l, k ) :
K ( l ) k < τ ( l ) K } and the pruned modelcan be represented as F −I K .
Note that u K is the only hyperparameter of our methods, and is usuallyset to 3.
Experiments
5.1 Implementation details
Datasets In this section, the experiments are conducted on two influential benchmarks, CIFAR-10[22] and Tiny-ImageNet [23].
We use 90% of the data set for training, the rest of the data is used forvalidating or recovering the poisoned model.
Models We use ResNet-18 [16] as the baseline model to evaluate our proposed method, andcompare it with other methods.
We train the network for 150 epochs on CIFAR-10 and 100 epochson Tiny-ImageNet with SGD optimizer.
The initial learning rate is set to 0.1 and the momentum isset to 0.9.
We adopt the cosine learning rate scheduler to adjust the learning rate.
The batch sizeis set to 128 by default.
We set the threshold hyperparameter u
h /u k to 3 on CIFAR-10 and 4 on Tiny-ImageNet by default. Attacks Our experiments are based on both the classical and the most advanced attack strategies,including the BadNet [14], Clean Label Attack (CLA) [40], Reflection Backdoor (Refool) [31],
Warping-based poisoned Networks [34], Blended backdoor attack (Blended) [5], Input-aware backdoor attack (IAB) [33] and Sample Specific Backdoor Attack (SSBA) [27].
For BadNets, we testboth the All-to-All (A2A) attack and All-to-One (A2O) attack, i.e. , the attack target labels are set toy t =
( y + 1) mod C , or one particular label y t =
C t , respectively.
The target for A2O attacks of allthe attack strategies is set to class 0.
The triggers for BadNets and CLA are set to randomly generatedpatterns with size 3 × 3 for CIFAR-10 and 5 × 5 for Tiny-ImageNet.
The poisoning rate is set to 10%by default.
Note that, due to the image size restraint, SSBA is only performed on Tiny-Imagenet.
Defenses We conduct experiments under two defense settings, one of which allows the defender toaccess the poisoned training set, while the other only has a small clean data set.
Both the defensegoals are to obtain a clean model without backdoor behaviours.
We compare our approaches with thel ∞ pruning [6], fine-tuning (FT), fine-pruning (FP) [28] and neural attentional distillation (NAD)[24].
The number of benign samples allowed to access is set to 500 (1%) for CIFAR-10 and 5000(5%) for Tiny-ImageNet by default.
Evaluation metrics
In this work, we use the attack clean accuracy (ACC) and attack success rate(ASR) to evaluate the effectiveness of different methods.
The ACC for a given model F is defined as:
[Equation]
where I is the indicator function .
The ASR is defined as:
[Equation]
where y t is the attack target label.
The ACC measures the model performance on benign samples,while the ASR reflects the degree of backdoor behavior retainment in the model.
Given an infectedmodel, our goal is to reduce the ASR, while keeping the ACC from dropping too much.
5.2 Experimental results
CIFAR-10 We show the results on CIFAR-10 in Table 1.
The recently proposed NAD and ANPperforms significantly better than other defense methods, reducing the ASR to a very low level with aslight drop on ACC.
However, they also have a significant drop ( 3 ∼ 4% ) on ACC when defending
CLA, which is the most robust backdoor attack in our experiments, and ANP even failed whendefending BadNets(A2A).
Nevertheless, both of our methods successfully eliminate the backdoor(ASR < 1% ) with negligible loss on ACC.
We even observe a little rise on ACC when defending [Table] [Table]
BadNets by DDE.
This phenomenon demonstrates that backdoor neurons may hurt the ACC in someway, and thus the ACC will rise when the backdoor neurons are precisely pruned.
Overall, ourmethods achieve the most advanced defense results.
Tiny-ImageNet Tiny-ImageNet is a larger scale dataset with higher resolution images, and it isharder to defend against the attacks performed on it.
Note that the A2A attack is absent, sincewe cannot successfully perform the attack due to the large number of its classes (up to 200).
Ourexperimental results show that all of the defense methods suffer from the performance degradationcompared with the results in CIFAR-10, and they fail to defend against WaNet with a large ACCdrop but even unchanged ASR, especially the ANP and l ∞ defense.
This phenomenon shows that theprinciples for finding backdoor neurons of both ANP and l ∞ don’t work in such case.
Nevertheless,our methods totally remove the backdoor and the ACC are not even affected, which indicates that ourmethods can precisely locate the backdoor neurons even on such large scale dataset.
5.3 Ablation study
To be fair, we compare MBNS with other re-training based methods using 500 benign samples in
Section 5.2.
However, MBNS doesn’t require re-training the model, since the samples are just usedfor detecting the distribution discrepancy.
Therefore, the required samples can be much less than 500.
We now discuss the limit of MBNS and study how the number of samples affects the effectiveness of
MBNS.
We train BadNets, CLA, Refool and Blended on CIFAR-10 with ρ = 10% , and use 10 to500 benign samples to recover the model using MBNS.
We record the changes of ACC and ASRwith respect to the number of benign samples.
The results are shown in Section 5.3.
The influenceof number of samples to our methods comes from the randomness on estimating moments.
As thenumber of samples grows, the randomness is reduced and MBNS has more stable performance, butthe average performances are not improved, except for Refool.
Compared with other attacks, Refoolclearly needs more samples to reduce the ASR.
The reason may be that the mixture distribution in
Refool has closer moments and is harder to distinguish.
Besides, we surprisingly find that MBNS canrecover BadNets, CLA and Blended using only 10 benign samples.
The additional results on DDEare shown in Section 7.
We also conduct experiments to show the high correlation between the backdoor neurons and ourproposed evaluation metrics, the results are shown in {+Section 
+}
[Figure]

   
 $ && %DGQHWV&/$5HIRRO%OHQGHG,$%:D1HW       1XPEHURIEHQLJQLPDJHV       $6
%DGQHWV&/$5HIRRO%OHQGHG,$%:D1HW
[Figure]
Discussion
[Figure]
The proposed methods are superior to other existing defense methods in the following three aspects:
Better performance As demonstrated in Section 5, both of the proposed methods achieve stateof-the-art results. Moreover, according to the ablation study, the proposed MBNS can successfullydefend most of the attacks within 10 benign samples, which shows the amazing effectiveness of ourproposed methods. Higher efficiency The proposed methods are also highly efficient. We record the running time ofseveral defense methods on 500 CIFAR-10 images with ResNet-18, and show the results in Table 3.
It can be seen that both of the proposed methods require less time than the baseline defense methods.
Since both methods require scanning on each neuron once, the computational complexity scalelinearly with the number of the neurons in the neural network.
Therefore, the efficiency of ourmethods is promised.
[Table]
Method
FT
FP
NAD
ANP
DDE (
[Table] More robust to hyperparameter choosing One of the most general problems in backdoor defenseis the choice of hyperparameters. Under realistic settings, the defenders can only perform defenseswithout any prior knowledge about poison data, including poisoning rate and examples of poisoneddata. So the defenders should carefully tune the hyperparameters, or the ACC and ASR can changesuddenly even under small fluctuations of those hyperparameters.
In comparison, both of theproposed pruning strategies only require one universal hyperparameter u .
Moreover, they showreliable consistency against different attacks in the same dataset, only vary from different datasets,which is inevitable.
Besides, we leave a wide range of parameters to choose, so that the ACC remainshigh while the ASR is controlled to a very small number, as shown in Section 5.3.
Conclusion
In this work, we take a deep inspection on the pre-activation distributions of each layer, and studytheir characteristics in a poisoned model.
We find that the distributions of benign data and poisoneddata on these neurons are both Gaussian but extraordinarily different in their moments.
The mixturedistributions have low entropy compared with that of the benign neurons.
This property makes itpossible for the defender to efficiently detect the abnormal neurons based on the sample entropy, whenthe poisoned dataset can be accessed.
Moreover, when the defender has only a small set of benigndata, we propose to detect the abnormal neurons based on the inconsistency of the BN statistics onpoisoned dataset and the sample statistics on the given benign dataset.
We then do pruning on thesepotential backdoor neurons using the proposed two detection methods to recover the model.
The experiments show that the proposed defending strategies can efficiently locate the backdoor neurons,and greatly reduce the backdoor threat with negligible loss on clean accuracy.
Our approach based onthe pre-activation distributions achieves superior results compared with all other defense methodsunder various attacks on both datasets, empirically showing that the distribution property may beboth dataset-invariant and attack-invariant.
The results shed lights on the field of backdoor defense,and can be a guidance for designing more robust backdoor attacks.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] or[N/A] .
You are strongly encouraged to include a justification to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the
• Did you include the license to the code and datasets?
[No] The code and the data areproprietary.
[Table]
• Did you include the license to the code and datasets?
[N/A]
[Table]
Please
do not modify the questions and only use the provided macros for your answers.
Note that the [Table]
Checklist
section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below. For all authors...
[Table]
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
[Table] contributions and scope? [Yes] See abstract.
[Table]
(b) Did you describe the limitations of your work? [Yes] See 5.3(c)
Did you discuss any potential negative societal impacts of your work?
[N/A] There isno negative social impacts of our work.
[Table]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to [Table]
them? [Yes]2.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes](b) Did you include complete proofs of all theoretical results?
[Yes]3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes]
[Table]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See experimental settings.
[Table]
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)? [Yes] See experimental results.
[Table]
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)? [Yes] See experimental settings.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes]
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[Yes]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]
A Implementation Details of Performed Backdoor Attacks and Defenses
In this part, we provide more detailed information about the state-of-art attacks and defenses appliedin our work.
The examples of poisoned images are shown in Appendix A.
[Figure]
[Figure] The hyperparameter we used in our DDE and MBNS is 3 in CIFAR-10 and 4 in Tiny-ImageNet. Theparameters in other defenses is set as default. B Additional Experimental Results of Different Poisoning Rate
Poisoning rate is considered important in the proposed methods, since smaller poisoning rate willmake the mixture distributions less distinguishable from unimodal distributions.
In DDE, decreasingpoisoning rate will increase the entropy of the backdoor neurons, and it will be harder to precisely detect them.
In MBNS, the smaller poisoning rate makes the BN statistics closer to the true benignstatistics.
Hence, it is important to study the influence of poisoning rate on the performance of theproposed methods.
We set the poisoning rate to 1% , 5% and 10% for all attacks and show the resultsin Table 4.
Note that CLA fails to attack the model when the poisoning rate is set to 1% .
In mostcases, their performances retain a high level.
Nevertheless, there are severe degradation against CLAand Refool.
Specifically, the ASR of CLA and Refool remain over 10% after pruning using DDEand MBNS.
However, we use default hyperparameter, i.e., u h = 3 and u K = 3 here.
By furtherdecreasing the pruning threshold, we can reduce both ASR to less than 5% .
Overall, MBNS is morerobust against attacks of small poisoning rate.
[Table]
C Additional Experimental Results on Wide-ResNet