Pre-activation Distributions Expose Backdoor Neurons
Abstract
Convolutional neural networks (CNN) can be manipulated to perform specific behaviors when encountering a particular trigger pattern without affecting the performance on normal samples, which is referred to as backdoor attack.
The backdoor attack is usually achieved by injecting a small proportion of poisoned samples into the training set, through which the victim trains a model embedded with the designated backdoor.
In this work, we demonstrate that backdoor neurons are exposed by their pre-activation distributions, where populations from benign data and poisoned data show significantly different moments.
This property is shown to be attack-invariant and allows us to efficiently locate backdoor neurons.
On this basis, we make several proper assumptions on the neuron activation distributions, and propose two backdoor neuron detection strategies based on (1) the differential entropy of the neurons, and (2) the Kullback-Leibler (KL) divergence between the benign sample distribution and a poisoned statistics based hypothetical distribution.
Experimental results show that our proposed defense strategies are both efficient and effective against various backdoor attacks.
Source code is available in this repo.
Introduction
Convolutional neural networks (CNNs) have achieved tremendous success during the past few years in a wide range of areas.
However, training a CNN from scratch involves a large amount of data and expensive computational costs, which is sometimes infeasible.
A more practical strategy is to obtain pretrained models or utilize public datasets from a third party, which brings convenience but also raises severe security problems into the deployment of models.
For example, a malicious third party
✓ ✓ ✓      may provide pretrained models embedded with a designated backdoor, such that the model will have a predefined response to some specific pattern, which is also called the trigger .
More realistically, the attacker can inject only a small proportion of malicious data into the public dataset to mislead the trained model, which is referred to as backdoor poisoning attacks [26].
For instance, the malicious data can be created by patching a particular pattern into the benign data and changing the label to the desired target.
The correlation between the trigger and the specified target label will be learned by the models during the training time.
In this way, the infected model will misclassify the input to the attack target when the pattern is patched, while behaving normally otherwise, as shown in Figure 1.
Due to the limited understanding of CNNs, we are not clear on the formation mechanism of backdoor behaviors.
However, it was empirically found that an infected model always possesses one or more neurons that have high correlation with the trigger activation, and pruning these neurons can significantly alleviate the backdoor behaviors, while retaining the model performance [43, 29, 7].
Nevertheless, how to precisely find out these backdoor neurons in an infected model is still a challenging problem, and has attracted a lot of attentions from the community.
In this work, we take an inspection on the pre-activation distributions of infected models on each layer.
In general, the pre-activations in each neuron follow an unimodal distribution that can be approximated by a Gaussian distribution.
However, we demonstrate that backdoor neurons do not hold such property.
Instead, in a backdoor neuron, the pre-activation distributions of benign data and poisoned data present significant different moments, and can be approximated by a mixture of two
Gaussian distributions.
This property allows us to locate potential backdoor neurons through simple statistical analysis.
Specifically, in case the defender has access to the poisoned dataset, the abnormal pre-activation distributions can be directly observed by forward propagating the data.
The mixture of benign and poisoned data, where the small proportion of poisoned points being away from the benign mean, leads to a skewed, and even a bimodal distribution.
Based on the maximum entropy property of the Gaussian distribution, the skewness will cause a reduction on the differential entropy, compared with a single Gaussian distribution with the same variance.
Hence, after standardizing the pre-activation distribution to be unit variance, the abnormal distributions should have lower entropy.
Those neurons could potentially separate benign and poisoned data.
However, under the defense setting in which only an infected model and a small set of benign data are provided, we are not able to observe the bimodal distributions since the poisoned data is absent.
In this case, we propose to rely on the statistics of Batch Normalization (BN) layers.
Specifically, if the infected model is trained on poisoned data, the population statistics in backdoor neurons recorded in the BN layer will be different from that of only benign data.
More importantly, the mismatch of statistics doesn’t exist in benign neurons, making it possible to distinguish backdoor neurons.
Based on the neuron entropy and the statistics discrepancy, we are able to locate and prune potential backdoor neurons to recover the model flexibly under two defense settings.
In summary, our contributions include:
We take a deep inspection on the infected model, and summarize the law of pre-activation distributions on poisoned dataset.
We find that (1) the standardized entropy of backdoor neurons can be significantly lower than benign neurons, and (2) the BN statistics in infected model are mismatched with the benign sample statistics.
We propose to prune potential backdoor neurons based on either the differential entropy of pre-activation distribution or the statistics discrepancy, depending on the defense settings.
Under certain assumptions, we claim that both the proposed indices can successfully separate the benign neurons and backdoor neurons by an appropriate threshold.
We conduct extensive experiments to verify our assumptions and evaluate our proposed methods, and achieve the state-of-the-art results under two different defense settings.
Related Work
In this section, we briefly discuss recent works in backdoor attack and defense, and a specific branch of related studies on distributional properties of poisoned features.
2.1 Backdoor attacks
The most famous backdoor attack is introduced in [15], where the adversary injects a small set of targeted label-flipped data with a specific trigger into the training set, leading to a misclassification when predicting the samples with such trigger.
To make the trigger pattern even more invisible to human beings, the blending strategy is used in [6] to generate poison images, while the form of natural reflection is utilized in trigger design in [32].
The input image is perturbed in [41] to keep its content consistent with the target label such that the model better memorizes the trigger pattern, and keep it imperceptible to human beings.
Moreover, the multi-target and multi-trigger attacks are proposed in [45, 34], and make the attack more flexible and covert.
Recently, some sample-specific trigger design strategies [28] are proposed, making the defense against such backdoor attack much harder.
Generally, the above attacks can be referred to as the poisoning based backdoor attacks .
Under some settings, the attackers can control the training process to inject the backdoor without modifying the training data, referred as the non-poisoning based backdoor attacks .
This is achieved in [31, 37, 5] through targeted modification of the neurons’ weight in a network.
Such attacks will not be evaluated in our work due to its strong attack setting.
2.2 Backdoor defenses
Training stage defense.
Under such setting, the defender has access to the training process, so that they can detect and filter the poisoned data or add some restrictions to suppress the backdoor effect in training.
Since the poisoned data can be regarded as outliers, different strategies are applied in [10, 13, 1, 40, 16], such as the robust statistics in feature space and input perturbation techniques to filter them out of training data.
Other methods aim at suppressing the backdoor effect during training phase with strong data augmentation methods [2] [27] [36] [33] including CutMix [46], Flip ,
ShrinkPad [27], CutOut and MaxUp [14], or differential privacy constraints [12, 19].
Model post-processing defense.
Under some specific scenarios, the defenders are only given a suspicious DNN model without access to the training process or the full training set.
Therefore, they must eliminate the backdoor threat with limited resources, such as a small set of clean data.
A straightforward way is to reconstruct the trigger, and then mitigate the model with the knowledge of the reversed trigger [42].
Some try to find the relationship between backdoor behaviors and the neurons in a DNN model.
Different levels of stimulation to a neuron are introduced in [30] to see how to determine the output activation change, if the model is attacked.
Simple neuron pruning strategies are applied in [7] to repair the model, while redundant neuron pruning and fine-tuning are combined in [29] to erase the backdoor effect.
Adversarial perturbations are added to the neurons in [43] and precisely prunes the easily-perturbed neurons with more limited clean data requirement and better performance.
There are other fine-tuning based methods with the implementation of knowledge distillation [25, 18].
Mode connectivity repair technique [47] is also explored to mitigate the backdoored model.
Recently, the K-Arm optimization [39] is applied in backdoor detection, helping curtail the threat of backdoor attack.
2.3 Distributional properties in poisoned dataset
Especially, one branch of backdoor learning studies the distributional properties of benign and poisoned samples.
To be specific, they seek for filtering out malicious data utilizing the statistical differences of poisoned samples.
Activation clustering [4] use K-means to separate benign and poisoned samples in feature space.
Spectral signatures [40] reveal that feature vectors tend to leave strong signals in the top eigenvectors of their covariance matrix.
SPECTRE (Spectral Poison Excision
Through Robust Estimation)[7] utilizes tools from robust statistics to amplify the spectral signature by outlier-robust data whitening.
Our work differentiated from the above works from the following three aspects: 1) the above works focus on the penultimate layer feature representation, while our work inspects deep into each layer 2) the above works take the representation space from all neurons as a whole, while our finding indicates that the distributional difference only exists in some neurons 3) the above works aim at filtering out poisoned samples for retraining, while our methods directly repair the trained neural network by pruning potential backdoor neurons.
Preliminaries 3.1 Notations
Consider a multi-class classification problem with C classes.
Let the original training set D
= { ( x i , y i ) }
Ni =1 contains N i.i.d.
sample images x i ∈ R d c × d h × d w and the corresponding labels y i ∈ { 1 , 2 , ..., C } drawn from X × Y .
Here, we denote by d c , d h and d w the number of neurons, the height and the width of images, respectively.
In particular, we have d c = 3 for RGB images.
As in Section 2.1, the backdoor poisoning attack involves changes to the input images and the corresponding labels on a subset D p ⊆ D .
In this work, we define the ratio ρ =
|D p | |D| as the poisoning rate .
We denote the poisoning function to the input images as δ ( x ) .
A dataset D is said to be ρ -poisoned if the poisoning rate of the dataset is ρ .
Consider a neural network F ( x ; θ ) with L layers.
Denote
[Equation]
for 1 ≤ l ≤ L , where f ( l ) is a linear function ( e.g. , convolution) in the l -th layer, and ϕ is a nonlinear activation function applied element wise.
In this work, we may denote F ( x ; θ ) as F ( x ) or F for simplicity.
We denote by W ( l ) ∈ R d c ′ ×
d c × d h × d w the weight tensor of a convolutional layer.
To do pruning, we apply a mask M ( l ) ∈ { 0 , 1 } d c ′ ×
d c × d h × d w starting with M ( l ) = 1 d c ′ ×
d c × d h × d w in each layer.
Pruning neurons on the network refers to getting a collection of indices
I = { ( l, k ) i } Ii =1 and setting
M ( l ) k
=
d c × d h × d w if ( l, k ) ∈ I .
The pruned network F −I has the same architecture as F but with all the weight matrices of convolutional layers set to W ( l ) ⊙ M ( l ) , where ⊙ denotes the Hadamard product.
3.2 Differential entropy
To measure the uncertainty of a discrete random variable Z , the entropy [38, 8] was defined as
H ( Z ) = − (cid:80)
z ∈ Z p ( z ) log p ( z ) .
At the same time, as an extension of entropy, the differential entropy was also introduced for a continuous random variable.
More concretely, if Z is a continuous random variable, then it was defined as
[Equation]
An important fact about the differential entropy is that, among all the real-valued distributions supported on ( −∞ , ∞ ) with a specified finite variance, the Gaussian distribution maximizes the differential entropy [8].
In this work, the differential entropy (1) will be utilized to identify the distributions that are far different from a Gaussian distribution.
3.3 Backdoor neurons
It was found that there exist one or more neurons that contribute the most to the backdoor behaviors in a infected model [43, 29].
If some of or all of these neurons are pruned, the attack success rate will be reduced greatly [43].
In this work, to better quantify the importance of neurons to backdoor behaviors, we would like to introduce the sensitivity of neurons to the backdoor.
We first introduce the definition of backdoor loss on a specific poisoning function:
Definition 1.
Given a model F and a poisoning function δ , the backdoor loss on the dataset D is defined as:
[Equation]
f ( δ ( x ))] ,
 
         ' HQ V L W \ %HQLJQVDPSOHV3RLVRQHGVDPSOHV (
[Figure]
neuron               ' HQ V L W \ %HQLJQVDPSOHV3RLVRQHGVDPSOHV (b) Backdoor neuron               ' HQ V L W \ %HQLJQVDPSOHV%1LQGXFHGGLVWULEXWLRQ (c) Benign neuron      
    
       ' HQ V L W \ %HQLJQVDPSOHV%1LQGXFHGGLVWULEXWLRQ
[Figure]
where D CE denotes the cross entropy loss.
Then:
Definition 2.
Given a model F , the index of a neuron ( l, k ) and the backdoor loss L bd , the sensitivity of that neuron to the backdoor is defined as:
[Equation]
where F −{ ( l,k ) } is the network after pruning the k -th neuron of the l -th layer.
The backdoor loss is high when the model is infected, and will be reduced when the backdoor effect is alleviated.
Using the quantity defined in (2), we are now able to find the neurons that are mostly correlated with the backdoor behaviors:
Definition 3.
Given a model F and a threshold τ > 0 , the set of backdoor neurons are defined as:
[Equation]
3.4 Pre-activation distribution
During the forward propagation of an input x , we denote x ( l ) =
F ( l ) ( x ) ∈ R d ( l ) c
× d ( l ) h × d ( l ) w as the output of the l -th layer.
For the k -th neuron of the l -th layer, the pre-activation ϕ ( l ) k
= ϕ
( x ( l ) k ) is defined as the maximum value of the k -th slice matrix of dimension d ( l ) h × d ( l ) w in x ( l ) .
The reason we choose pre-activations instead of activations is that the distribution of activations after non-linear function might be distorted.
For example, ReLU (rectified linear unit) will cut out all negative values.
It is a common assumption that, for every neuron, the pre-activations before the non-linear function follow a Gaussian distribution, if the network is randomly initialized and the number of neurons is large enough [22].
This assumption is based on the central limit theorem under weak dependence [3].
In a trained network, although this assumption may not strictly hold, the pre-activation of every neuron can be still regarded as approximately following a Gaussian distribution.
However, in this work, for the first time, we observe a bimodal pre-activation distribution in backdoor neurons formed by the benign data and poisoned data.
This phenomenon is shown in Figure 2, where a typical backdoor neuron is compared with the benign neurons.
It can be seen that, after the model is infected, the pre-activation distributions of benign neurons hardly change when the data is poisoned, while the pre-activation distributions of backdoor neurons become significantly different.
Methodology 4.1 Basic Assumptions
We now introduce two preliminarily assumptions on the pre-activation distribution of an infected neural network.
Assumption 1.
Given an infected model F , we have |B F,τ | > 0 for some threshold τ > 0 .
This is a primary assumption that guarantees proper pruning of neurons can correct the network’s predictions on poisoned samples to some extent.
Hence, it is a prerequisite of good performance of all the pruning-based defense methods.
The next assumption provides a precondition for our methods.
Assumption 2.
Given a model F infected by a poisoning function δ with a ρ -poisoned dataset D , the pre-activation of sample from D on each single neuron of F follow a Gaussian mixture distribution, that is:
[Equation]
with
[Equation]
and
[Equation]
where ϵ > 0 is a small enough value, µ ( l ) k and σ ( l )2 k , ˆ µ ( l ) k and ˆ σ ( l )2 k are the mean and variance of { ϕ ( F ( l ) ( x ) k ) :
x ∼ X} , { ϕ ( F ( l ) ( δ ( x )) k ) : x ∼ X} , respectively.
This assumes that the mean value of pre-activation distribution of benign and poisoned samples only significantly differentiated in backdoor neurons.
This assumption is made based on empirical observation, and our methods work only when this assumption holds.
4.2 Entropy-based Pruning (EP)
If the pre-activation distributions are standardized (subtracting the mean and dividing the standard deviation), the differential entropy will be maximized on the benign neuron approximately following a standard Gaussian distribution N (0 , 1) .
In backdoor neurons, because of the difference of the moments on Gaussian components, the resulting mixture distributions can not be Gaussian distributions, hence the differential entropy must be smaller than that of a standardized Gaussian distribution.
Corollary 1.
Let ˙ ϕ
( l ) k
= ϕ ( l ) k − µ ( l ) k σ ( l )
k be the standardized pre-activations, then the following inequality is satisfied:
[Equation]
where Z ∼ N (0 , 1) is the standardized Gaussian distribution.
This inequality gives a guarantee that with an appropriately chosen threshold, the backdoor neurons can be well separated with the benign neurons.
4.3 BN statistics-based Pruning (BNP)
BN layer involves using the statistics of a mini-batch to normalize the data in each layer for each neuron.
It is known to be able to smooth the optimization landscape, and has gradually become a default setting of neural networks [20].
When inference, BN uses the fixed statistics obtained by averaging the sample statistics of mini-batches during training time, including the mean and the variance.
If the model is trained on a poisoned dataset, BN will record the mean and the variance of the poison-benign mixed data.
Note that the mean and variance here are not defined on the pre-activations ϕ ( l ) k , but on x ( l ) k .
Based on the above discussions, we know that the poisoned preactivations in backdoor neurons follow a different distribution from the benign samples.
The recorded statistics during training are actually that of the mixture distribution.
Hence, we can expect that the BN statistics of a trained backdoor neural network are biased.
If we are able to access a small set of benign data, we can calculate an approximation of the true statistics on benign data.
Then we calculate the Kullback-Leibler (KL) divergence [9] between the sample distribution and the BN induced distribution as the measurement of the bias.
By assuming both of the distributions follow
Gaussian distributions, we have a closed-form solution:
[Equation]
where N ( l ) sample = N ( µ ( l ) k , σ ( l )2 k ) , N ( l ) BN = N (˜ µ ( l ) k , ˜ σ ( l )2 k ) , µ ( l ) k and σ ( l )2 k are the statistics obtained from benign samples, ˜ µ ( l )2 k and ˜ σ ( l ) k are the BN statistics.
Note that BN statistics is the mixture statistics of benign and poisoned distributions.
Thus, we have the following corollary:
Corollary 2.
According to Assumption 2, when ϵ → 0 , the following inequality is satisfied:
[Equation]
The corollary indicates that backdoor neurons should have larger KL divergences than benign neurons, as illustrated in Figure 2(c).
4.4 Overview of the two pruning strategies
In Section 3.4, we reveal the discrepancy between the pre-activation distributions in backdoor neurons and that in the benign neurons.
This enables fast detecting the neurons that are more related to the backdoor behaviours.
The index we choose to detect the abnormal neurons depends on what kind of data we are able to access.
Mixture training data In this case, the victim is given a poisoned training dataset with a specified poisoning rate ρ .
Our goal is to obtain a benign model based on the poisoned dataset.
To achieve this, we first train an infected model F on the poisoned dataset.
The resulting model should have a certain number of backdoor neurons based on empirical observation and assumption.
Since ρ > 0 for the dataset, all the neurons follow Gaussian mixture distributions, and we have h ( ˙ x ( l ) k ) <
h ( ˙ x ( l ) k ′ ) for all k ∈ B F,τ , k ′ / ∈ B F,τ .
This implies that with an appropriate threshold τ ∗ h , we can perfectly separate the benign neurons and backdoor neurons, which can be formulated as:
[Equation]
Setting the threshold τ ∗ h is crucial to the solution, and it is a trade-off between the accuracy on benign samples and that on the backdoored samples.
Note that |B ( l ) F,τ | << d ( l ) c .
We can treat the low entropy neurons as outliers in each layer, and set different thresholds for different layers.
Specifically, let h ( l ) =
[ h ( x ( l ) 1 ) , h ( x ( l ) 2 ) , . .
, h ( x ( l ) d ( l ) c )]
T ∈ R d ( l ) c be a vector of differential entropy of the l -th layer calculated from the poisoned dataset.
Then we set τ ( l ) h
= ¯ h ( l ) − u h · s ( l ) h , where ¯ h ( l ) =
1 d ( l ) c (cid:80) d ( l ) c k =1 h ( l ) k and s ( l ) h
= (cid:114)d
[Equation]
l ) c (cid:80) d ( l ) c k =1 ( h ( l ) k − ¯ h ( l ) ) 2 are the mean and standard deviation of h ( l ) , u h is a hyperparameter controlling how low the threshold is.
Then we have a set of indices of potential backdoor neurons
I h
= { ( l, k ) :
h ( l ) k < τ ( l ) h } .
Finally, we prune the infected model F using
I h , which results in a final model F −I h .
Benign training data This is the case that the victim is given a trained poisoned model F with a small set of benign data.
Our goal is to utilize the benign data to clean up the poisoned model and eliminate the backdoor threat.
Similar to the pruning process based on differential entropy, we first construct a vector of KL divergences of all neurons for each layer K ( l ) =
[ K ( l ) 1 , K ( l ) 2 , . .
, K ( l ) d ( l ) c ] T ∈
R d ( l ) c according to equation (4).
We set τ ( l ) K = ¯ K ( l ) − u K · s ( l ) K , where ¯ K ( l ) =
1 d ( l ) c (cid:80) d ( l ) c k =1 K ( l ) k and s ( l ) K
= (cid:114)
d ( l ) c (cid:80) d ( l ) c k =1 ( K ( l ) k − ¯ K ( l ) ) 2 are the mean and standard deviation of K ( l ) , u K is a hyperparameter.
The set of selected neurons is I K
= { ( l, k ) :
K ( l ) k < τ ( l ) K } and the pruned model can be represented as F −I K .
Experiments 5.1 Implementation details
Datasets In this section, the experiments are conducted on two influential benchmarks, CIFAR-
[23] and Tiny-ImageNet [24].
We use 90% of the data set for training, the rest of the data is used for validating or recovering the poisoned model.
Models We use ResNet-18 [17] as the baseline model to evaluate our proposed method, and compare it with other methods.
We train the network for 150 epochs on CIFAR-10 and 100 epochs on Tiny-ImageNet with SGD optimizer.
The initial learning rate is set to 0.1 and the momentum is set to 0.9.
We adopt the cosine learning rate scheduler to adjust the learning rate.
The batch size is set to 128 by default.
Attacks Our experiments are based on both the classical and the most advanced attack strategies, including the BadNet [15], Clean Label Attack (CLA) [41], Reflection Backdoor (Refool) [32],
Warping-based poisoned Networks [35], Blended backdoor attack (Blended) [6], Input-aware backdoor attack (IAB) [34] and Sample Specific Backdoor Attack (SSBA) [28].
For BadNets, we test both the All-to-All (A2A) attack and All-to-One (A2O) attack, i.e. , the attack target labels are set to y t
= ( y + 1) mod C , or one particular label y t =
C t , respectively.
The target for A2O attacks of all the attack strategies is set to class 0.
The triggers for BadNets and CLA are set to randomly generated patterns with size 3 × 3 for CIFAR-10 and 5 × 5 for Tiny-ImageNet.
The poisoning rate is set to 10% by default.
Note that, due to the image size restraint, SSBA is only performed on Tiny-Imagenet.
Defenses We conduct experiments under two defense settings, one of which allows the defender to access the poisoned training set, while the other only has a small clean data set.
Both the defense goals are to obtain a clean model without backdoor behaviours.
We compare our approaches with the l ∞ pruning [7], fine-tuning (FT), fine-pruning (FP) [29] and neural attentional distillation (NAD) [25].
The number of benign samples allowed to access is set to 500 (1%) for CIFAR-10 and(5%) for Tiny-ImageNet by default.
We set the threshold hyperparameter u
h /u k to 3 on CIFARand 4 on Tiny-ImageNet of all tested attacks by default.
Evaluation metrics
In this work, we use the attack clean accuracy (ACC) and attack success rate (ASR) to evaluate the effectiveness of different methods.
The ACC for a given model F is defined as:
[Equation]
where I is the indicator function .
The ASR is defined as:
[Equation]
where y t is the attack target label.
The ACC measures the model performance on benign samples, while the ASR reflects the degree of backdoor behavior retainment in the model.
Given an infected model, our goal is to reduce the ASR, while keeping the ACC from dropping too much.
5.2 Experimental results
CIFAR-10 We show the results on CIFAR-10 in Table 1.
The recently proposed NAD and ANP perform significantly better than other defense methods, reducing the ASR to a very low level with a slight drop on ACC.
However, they also have a significant drop ( 3 ∼ 4% ) on ACC when defending
CLA, which is the most robust backdoor attack in our experiments, and ANP even failed when defending BadNets(A2A).
Nevertheless, both of our methods successfully eliminate the backdoor (ASR < 1% ) with negligible loss on ACC.
We even observe a little rise on ACC when defending
BadNets by EP.
This phenomenon demonstrates that backdoor neurons may hurt the ACC in some way, and thus the ACC will rise when the backdoor neurons are precisely pruned.
Overall, our methods achieve the most advanced defense results.
[Table] [Table]
Tiny
[Table]
ImageNet Tiny-ImageNet is a larger scale dataset with higher resolution images, and it is harder to defend against the attacks performed on it.
Note that the A2A attack is absent, since we cannot successfully perform the attack due to the large number of its classes (up to 200).
Our experimental results show that all the defense methods suffer from the performance degradation compared with the results in CIFAR-10, and they fail to defend against WaNet with a large ACC drop but even unchanged ASR, especially the ANP and l ∞ defense.
This phenomenon shows that the principles for finding backdoor neurons of both ANP and l ∞ don’t work in such case.
Nevertheless, our methods totally remove the backdoor and the ACC are not even affected, which indicates that our methods can precisely locate the backdoor neurons even on such large scale dataset.
5.3 Ablation study
To be fair, we compare BNP with other re-training based methods using 500 benign samples in
Section 5.2.
However, BNP doesn’t require re-training the model, and the samples are just used for detecting the distribution discrepancy.
As the statistical differences may be detected with much fewer samples, we now study how the number of samples affects the effectiveness of BNP.
We train
BadNets, CLA, Refool and Blended on CIFAR-10 with ρ = 10% , and use 10 to 500 benign samples to recover the model using BNP.
We record the changes of ACC and ASR with respect to the number of benign samples.
The results are shown in Section 5.3.
The influence of the number of samples to our methods comes from the randomness on estimating moments.
As the number of samples grows, the randomness is reduced and BNP has more stable performance, but the average performances are not improved, except for Refool.
Compared with other attacks, Refool clearly needs more samples to reduce the ASR.
A possible reason is that the mixture distribution in Refool has closer moments and is harder to distinguish.
Besides, we surprisingly find that BNP can recover BadNets, CLA and Blended using only 10 benign samples.
We also conduct experiments to show the high correlation between the backdoor neurons and our proposed evaluation metrics, the results are shown in Appendix D.
Discussion
The proposed methods are superior to other existing defense methods in the following three aspects:
Better performance As demonstrated in Section 5, both of the proposed methods achieve stateof-the-art results.
Moreover, according to the ablation study, the proposed BNP can successfully defend most of the attacks within 10 benign samples, which shows the amazing effectiveness of our proposed methods.
      1XPEHURIEHQLJQLPDJHV

    
 $ && %DGQHWV&/$5HIRRO%OHQGHG,$%:D1HW       1XPEHURIEHQLJQLPDJHV       $6  %DGQHWV&/$5HIRRO%OHQGHG,$%:D1HW
           u h       
 


EP $&&$6
           u K       
 


BNP $&&$6 3 H U F HQ W DJH
Higher efficiency The proposed methods are also highly efficient.
We record the running time of several defense methods on 500 CIFAR-10 images with ResNet-18, and show the results in Table 3.
It can be seen that both of the proposed methods require less time than the baseline defense methods.
Since both methods require scanning on each neuron once, the computational complexity scale linearly with the number of the neurons in the neural network.
Therefore, the efficiency of our methods is promised.
[Table]
More robust to hyperparameter choosing One of the most general problems in backdoor defense is the choice of hyperparameters.
Under realistic settings, the defenders can only perform defenses without any prior knowledge about poison data, including poisoning rate and examples of poisoned data.
So the defenders should carefully tune the hyperparameters, or the ACC and ASR can change suddenly even under small fluctuations of those hyperparameters.
In comparison, both of the proposed pruning strategies only require one universal hyperparameter u .
Moreover, they show reliable consistency against different attacks in the same dataset, only vary from different datasets, which is inevitable.
Besides, we leave a wide range of parameters to choose, so that the ACC remains high while the ASR is controlled to a very small number, as shown in Section 5.3.
Conclusion
In this work, we take a deep inspection on the pre-activation distributions in each layer, and study their characteristics in an infected model.
We find that the distributions of benign data and poisoned data on these neurons are both Gaussian, but extraordinarily different in their moments.
The mixture distributions have low entropy compared with that of the benign neurons.
This property makes it possible for the defender to efficiently detect the abnormal neurons based on the sample entropy, when the poisoned dataset can be accessed.
Moreover, when the defender has only a small set of benign data, we propose to detect the abnormal neurons based on the inconsistency of the BN statistics on poisoned dataset and the sample statistics on the given benign dataset.
We then do pruning on these potential backdoor neurons using the proposed two detection methods to recover the model.
The experiments show that the proposed defending strategies can efficiently locate the backdoor neurons, and greatly reduce the backdoor threat with negligible loss on clean accuracy.
Our approach based on the pre-activation distributions achieves superior results compared with all other defense methods under various attacks on both datasets, empirically showing that the distribution property may be both dataset-invariant and attack-invariant.
The results shed lights on the field of backdoor defense, and can be a guidance for designing more robust backdoor attacks.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information on how to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] or [N/A] .
You are strongly encouraged to include a justification to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] • Did you include the license to the code and datasets?
[No] The code and the data are proprietary.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] See abstract.
(b) Did you describe the limitations of your work?
[Yes] See 5.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
There is no negative social impacts of our work.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See experimental settings.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] See experimental results.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See experimental settings.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] (b) Did you mention the license of the assets?
[Yes] (c) Did you include any new assets either in the supplemental material or as a URL?
[Yes] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[Yes] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
A Implementation Details of Performed Backdoor Attacks and Defenses
In this part, we provide more detailed information about the state-of-art attacks and defenses applied in our work.
The examples of poisoned images are shown in Appendix A.
[Figure]
[Figure]
The hyperparameter we used in our EP and BNP is 3 in CIFAR-10 and 4 in Tiny-ImageNet.
The parameters in other defenses is set as default.
B Additional Experimental Results of Different Poisoning Rate
Poisoning rate is considered important in the proposed methods, since smaller poisoning rate will make the mixture distributions less distinguishable from unimodal distributions.
In EP, decreasing poisoning rate will increase the entropy of the backdoor neurons, and it will be harder to precisely detect them.
In BNP, the smaller poisoning rate makes the BN statistics closer to the true benign statistics.
Hence, it is important to study the influence of poisoning rate on the performance of the proposed methods.
We set the poisoning rate to 1% , 5% and 10% for all attacks and show the results in Table 4.
Note that CLA fails to attack the model when the poisoning rate is set to 1% .
In most cases, their performances retain a high level.
Nevertheless, there are severe degradation against CLA and Refool.
Specifically, the ASR of CLA and Refool remain over 10% after pruning using EP and
BNP.
However, we use default hyperparameter, i
[Table]
e., u h = 3 and u K = 3 here.
By further decreasing the pruning threshold, we can reduce both ASR to less than 5% .
Overall, BNP is more robust against attacks of small poisoning rate.
[Table]
C Additional Experimental Results on Wide-ResNet
To study the generalization of the proposed defense methods over different architectures, we conduct experiments on WideResNet-28-1, which is also a commonly used model in backdoor community.
The results are shown in Appendix C.
Backdoored
EP
BNP
Attacks
ACC
ASR
ACC
ASR
ACC
ASR
BadNets (A2O) 91.
BadNets (A2A) 92.
CLA 92.
Refool 91.
WaNet 91.
Blended 91.
[Table]
D Analysis of Correlation Between Backdoor Neurons and Pruned Neurons
In section Section 3.3, we have demonstrated the definition of sensitivity of neurons to backdoor in
Equation (2), which is the difference of backdoor loss before and after being pruned.
It is important to ensure that our methods precisely detect those backdoor neurons with high sensitivity.
Here, we make scatter plots of the backdoor sensitivity of the neurons and their differential entropy and KL divergence in some typical layers.
The detailed scatter figures are shown in Figure 7.
The results indicate that both EP and BNP choose the neurons with high sensitivity.
Especially for BadNets, the highest sensitivity of one single neuron is extremely large and its differential entropy and KL divergence stay away from average.
The overall results demonstrate a strong correlation between the neuron sensitivity to backdoor and the proposed indices.
E Performance on Low-quality Data
Our methods require extra data and rely on its distribution to detoxify the backdoor model, so it’s natural to consider a worse case that we can only get low-quality dataset with some distortion in it.
To further show the effectiveness of our proposed methods, we utilize multiple data augmentation tricks
[Figure]
[Figure]
to create out-of-distribution data on CIFAR-10 while performing defenses against BadNets (A2O) attack, including color jitter with 90 degree rotation (CJ) from torchvision, CoarseSaltandPepper (SAP), PolarWarping (PW), Snowflakes (Snow) from [21].
To be specific, the parameters of CJ are set to be [0.5, 0.5, 0.5, 0.5]; 30% of the pixels in SAP are replaced replaced by salt/pepper noise mask which has 1% to 10% the size of the input image; the translate percent in PW is set to be ± 0 .
2 ; the snow size of Snow is (0.2, 0.5) and its speed is (0.01, 0.05).
Samples of those perturbed images are shown in Appendix E.
Now we replace 10% of the EP and BNP used data into those perturbed out-of-distribution data, and see the performance of our methods.
The experimental results are shown in Table 5, which illustrates that even if some low-quality data are mixed into the data they use, our methods can still successfully locate the backdoor neurons and delete them while maintaining high clean accuracy.
[Table]
F Detailed Comparison on Neuron Pruing Accuracy
To evaluate the accuracy in locating malicious neurons of the proposed methods and the SOTA pruning-based method ANP, we record the minimum pruning amount that can lead to acceptable degradation on ASR.
Specifically, we gradually change the hyperparameter u of EP, BNP and the pruning threshold of ANP until their ASRs drop below 10%, which we regard as a sign of successful defense.
The detailed results are shown in Table 6.
[Table]
Under most situations, our proposed EP and BNP show higher or similar pruning precision in finding malicious neurons than ANP.
Moreover, our pruning methods gives promising pruning results when u k /u h is around 3 as we suggested, since most of the pruning has achieved considerable results before u k /u h reaches 3.
You can refer to Appendix D for detailed pruning performance.
However, for ANP, it is hard to choose a proper mask value threshold of the neurons to be pruned, and it is also not practical to select a specific number of neurons to finish the pruning.
Moreover, the higher the mask value of neurons, the more dense the neurons are, making it harder to find a suitable value for pruning threshold, since there will be huge difference on ACC and ASR with slight change of threshold.
In summary, our proposed EP and BNP can locate the neurons more precisely in most situations, and gives an easier way to prune the malicious neurons.
G Exploration of Potential Adaptive Attacks
In this section, we explore some potential adaptive attacks against the proposed methods.
G.1 BN Re-parameterization against BNP
The BNP pruning strategy rely on the correct information from the BN statistics.
However, this information can be changed by a defense-aware attacker using re-parameterization techniques.
Specifically, consider a BN operation performed on each neuron by:
[Equation]
The attacker can calculate the sample statistics denoting µ ( l ) k and σ ( l ) k from benign data, and assign new weights and bias by:
[Equation]
In this way, the whole linear transformation remains unchanged, but the BN statistics become consistent with that of the benign data, which makes the backdoor neurons unable to be detected by the proposed BNP.
G.2 Regularization-based Adaptive Attacks
Both of the two methods rely on the discrepancy of the benign distribution and poisoned distribution.
Hence, we wonder whether EP and BNP still work if the attacker try to regularize the distribution to
[Table]
[Table]
minimize the discrepancy.
Hence, we use BadNets as an example to perform the adaptive attack with an additional objective:
[Equation]
where ϕ ( l ) k and ˆ ϕ ( l ) k denote the pre-activations of benign and poisoned samples in the k th neuron of the l th layer.
The goal of the second term is to minimize the discrepancy between the benign distribution and the poisoned distribution with a trade-off hyperparameter λ .
We select λ from [1.000, 0.100, 0.010, 0.001] and train four models and do pruning using EP and BNP, the results are shown in Table 7. We find that too large λ makes the objective not trainable, hence collapse the training process.
This may because it is not realistic to distinguish benign and poisoned samples while making their distribution indistinguishable.
After down weighting λ , the model is able to be trained normally.
However, we find EP and BNP are still able to remove the backdoor without influence too much on the clean accuracy.
A recently proposed method[44] also regularizes the distribution discrepancy between benign samples and poisoned samples by Multi-Level Maximum Mean Discrepancy (ML-MMD), which is a direct attack against our defense.
Besides, [11] propose to use wasserstein distance to minimize the difference between poison and clean data latent feature distribution (WB).
We test our methods on them and find they still work with some degradation.
The results are shown in Table 8.
The main reason for such degradation is the strong constraint added to minimize the difference between clean and poison data latent feature distribution.
Besides, both the above attacks adds constraint to loss function, making it harder to converge during training process, and their final ACC is obviously lower than normal attacks.
H Additional Experimental Results for the Choice of Hyperparameter
In this section, we provide ablation study on the choice of the threshold hyperparameter
u
h
/u
k .
As shown in Figure 9, both EP and BNP gives promising defending results against most tested attacks on CIFAR 10 when u h
/u k is around 3.Besides, the robustness of the hyperparameter is supported by the wide choice of available
u h /u k that lead to acceptable results.
[Table]
[Table]
[Table]
[Table]
[Figure]
u K
P e r c e n t a g e
EP BNP (g) IAB 0 2
[Figure] u h
ACCASR 0
[Figure]
u K
P e r c e n t a g e
EP
[Figure]