N EURAL A TTENTION D ISTILLATION : E RASING B ACKDOOR T RIGGERS FROM D EEP N EURAL N ETWORKS
Anonymous authors
Paper under double-blind review
A BSTRACT
Deep neural networks (DNNs) are known vulnerable to backdoor attacks, a training time attack that injects a trigger pattern into a small proportion of training data so as to control the model’s prediction at the test time.
Backdoor attacks are notably dangerous since they do not affect the model’s performance on clean examples, yet can fool the model to make incorrect prediction whenever the trigger pattern appears during testing.
In this paper, we propose a novel defense framework
Neural Attention Distillation (NAD) to erase backdoor triggers from backdoored
DNNs.
NAD utilizes a teacher network to guide the ﬁnetuning of the backdoored student network on a small clean subset of data such that the intermediate-layer attention of the student network aligns with that of the teacher network.
The teacher network can be obtained by an independent ﬁnetuning process on the same clean subset.
We empirically show, against 6 state-of-the-art backdoor attacks, NAD can effectively erase the backdoor triggers using only 5% clean training data without causing obvious performance degradation on clean examples.
I NTRODUCTION
In recent years, deep neural networks (DNNs) have been widely adopted into many important realworld and safety-related applications.
Nonetheless, it has been demonstrated that DNNs are prone to potential threats in multiple phases of their life cycles.
A type of well-studied adversary is called the adversarial attack.
At test time, state-of-the-art DNN models can be fooled into making incorrect predictions with small adversarial perturbations.
DNNs are also known to be vulnerable to another type of adversary known as the backdoor attack.
Recently, backdoor attacks have gained more attention due to the fact it could be easily executed in real scenarios (Gu et al., 2019; Chen et al., 2017).
Intuitively, backdoor attack aims to trick a model into learning a strong correlation between a trigger pattern and a target label by poisoning a small proportion of the training data.
Even trigger patterns as simple as a single pixel (Tran et al., 2018) or a black-white checkerboard (Gu et al., 2019) can grant attackers full authority to control the model’s behavior.
Backdoor attacks can be notoriously perilous for several reasons.
First, backdoor data could inﬁltrate the model on numerous occasions including training models on data collected from unreliable sources or downloading pre-trained models from untrusted parties.
Additionally, with the invention of more complex triggers such as natural reﬂections (Liu et al., 2020b) or invisible noises (Liao et al., 2020; Li et al., 2019; Chen et al., 2019c), it is much harder to catch backdoor examples at test time.
On top of that, once the backdoor triggers have been embedded into the target model, it is hard to completely eradicate their malicious effects by standard ﬁnetuning or neural pruning (Yao et al., 2019;
Li et al., 2020; Liu et al., 2020b).
A recent work also proposed the mode connectivity repair (MCR) to remove backdoor related neural paths from the network (Zhao et al., 2020a).
On the other hand, even though detection-based approaches have been performing fairly well on identifying backdoored models (Chen et al., 2019a; Tran et al., 2018; Chen et al., 2019b; Kolouri et al., 2020), the identiﬁed backdoored models still need to be puriﬁed by backdoor erasing techniques.
In this work, we propose a novel backdoor erasing approach, Neuron Attention Distillation ( NAD ), for the backdoor defense of DNNs.
NAD is a distillation-guided ﬁnetuning process motivated by the ideas of knowledge distillation (Bucilua et al., 2006; Hinton et al., 2014; Urban et al., 2016) and neural attention transfer (Zagoruyko & Komodakis, 2017; Huang & Wang, 2017; Heo et al., 2019).
Speciﬁcally, NAD utilizes a teacher network to guide the ﬁnetuning of a backdoored student network on a small subset of clean training data so that the intermediate-layer attention of the student network is well-aligned with that of the teacher network.
The teacher network can be obtained from the backdoored student network via standard ﬁnetuning using the same clean subset of data.
We empirically show that such an attention distillation step is far more effective in removing the network’s attention on the trigger pattern in comparison to the standard ﬁnetuning or the neural pruning methods.
Our main contributions can be summarized as follows:
• We propose a simple yet powerful backdoor defense approach called Neuron Attention Distillation (NAD).
NAD is by far the most comprehensive and effective defense against a wide range of backdoor attacks.
• We suggest that attention maps can be used as an intuitive way to evaluate the performance of backdoor defense mechanisms due to their ability to highlight backdoored regions in a network’s topology.
R ELATED W ORK
Backdoor Attack.
Backdoor attack is a type of attack emerging in the training pipeline of DNNs.
Oftentimes, a backdoor attack is accomplished by designing a trigger pattern with (poisoned-label attack) (Gu et al., 2019;
Chen et al., 2017; Liu et al., 2018b) or without (clean-label attack) (Shafahi et al., 2018; Turner et al., 2019; Liu et al., 2020b) a target label injected into a subset of training data.
These trigger patterns can appear in forms as simple as a single pixel (Tran et al., 2018) or a tiny patch (Chen et al., 2017), or in more complex forms such as sinusoidal strips (Barni et al., 2019).
Trigger patterns may also appear in the form of natural reﬂection (Liu et al., 2020b) or human imperceptible noise (Liao et al., 2020; Li et al., 2019; Chen et al., 2019c).
It is even harder to detect these patterns as they are arguably more stealthy.
Recent studies have shown that a backdoor attack can be conducted even without access to the training data (Liu et al., 2018b) or in federated learning (Xie et al., 2019; Bagdasaryan et al., 2020).
Backdoor Defense.
Existing works primarily focused on two types of strategies to defend against backdoor attacks.
Depending on the methodologies, a backdoor defense can be either backdoor detection or trigger erasing.
Detection-based methods aim at identifying the existence of backdoor adversaries in the underlying model (Wang et al., 2019; Kolouri et al., 2020) or ﬁltering the suspicious samples from input data for re-training (Tran et al., 2018; Gao et al., 2019; Chen et al., 2019b).
Although these methods have been performing fairly well on distinguishing whether a model has been poisoned, the backdoor effects still remain in the backdoored model.
On the other hand, Erasing-based methods aim to directly purify the backdoored model by removing the malicious impacts caused by the backdoor triggers, while simultaneously maintain the model’s overall performance on clean data.
A straightforward approach is to directly ﬁnetune the backdoored model on a small subset of clean data, which is typically available to the defender (Liu et al., 2018b).
Nonetheless, training on only a small clean subset can lead to catastrophic forgetting (Kirkpatrick et al., 2017), where the model overﬁts to the subset and consequently causes substantial performance degradation.
Fine-pruning (Liu et al., 2018a) alleviates this issue by pruning less informative neurons prior to ﬁnetuning the model.
In such a way, the standard ﬁnetuning process can effectively erase the impact of backdoor triggers without signiﬁcantly deteriorating the model’s overall performance.
WILD (Liu et al., 2020a) proposed to utilize data augmentation techniques alongside distribution alignment between clean samples and their occluded versions to remove backdoor triggers from DNNs.
Other techniques such as regularization (Truong et al., 2020) and mode connectivity repair (Zhao et al., 2020a) have also been explored to mitigate backdoor attacks.
While promising, existing backdoor erasing methods still suffer from a number of drawbacks.
Efﬁcient methods can be evaded by the latest attacks (Liu et al., 2018a; 2020b), whereas effective methods are typically computationally expensive (Zhao et al., 2020a).
In this work, we propose a novel ﬁnetuning-based backdoor erasing approach that is not only effective but efﬁcient against a wide range of backdoor attacks.
Knowledge Distillation.
Knowledge distillation (KD) is ﬁrst proposed to compress a bigger or an ensemble of well-trained network(s) into a compact smaller network (Bucilua et al., 2006; Hinton et al., 2014).
In this process, the more knowledgeable network is referred to as the teacher network , and the smaller network is the student network .
Feature maps and attention mechanisms have been demonstrated to be very useful in KD to supervise the training of student networks (Romero et al., 2015; Zagoruyko & Komodakis, 2017; Huang & Wang, 2017; Song et al., 2018; Ahn et al., 2019; Heo et al., 2019).
They can help the student network to learn more high-quality intermediate
[Figure]
representations, leading to an improved distillation effect and better student network performance (Romero et al., 2015; Zagoruyko & Komodakis, 2017).
KD has also shown its potential in other ﬁelds such as adversarial robustness (Papernot et al., 2016), multi-granularity lip reading (Zhao et al., 2020b) and data augmentation (Bagherinezhad et al., 2018).
In this work we propose a new backdoor defense technique based on the combination of knowledge distillation and neural attention transfer.
P ROPOSED A PPROACH
In this section we ﬁrst describe the defense setting, then introduce the proposed NAD approach.
Defense Setting.
We adopt a typical defense setting where the defender outsourced a backdoored model from an untrusted party and is assumed to have a small subset of clean training data to ﬁnetune the model.
The goals of backdoor erasing are to erase the backdoor trigger from the model while retaining the performance of the model on clean samples.
3. N EURON A TTENTION D ISTILLATION
Overview.
We illustrate the differences between NAD and the traditional ﬁnetuning approach in
Figure 1.
Instead of using the ﬁnetuned network directly as our ﬁnal model, we employ it as a teacher network and use it in conjunction with the original backdoored network (i.e. student network) through an attention distillation process.
The job of NAD is to align neurons that are more responsive to the trigger pattern with benign neurons that only responsible for meaningful representations.
The main challenge for NAD is thus to ﬁnd the proper attention representations to distill.
Over the next few subsections, we will deﬁne the attention representation used throughout our work formally and introduce the loss functions used in the process of attention distillation.
Attention Representation.
Given a DNN model F , we denote the activation output at the l -th layer as F l ∈ R C × H × W with C , H and W being the dimensions of the channel, the height, and the width of the activation map respectively.
We deﬁne A : R C × H × W
→ R H × W to be an attention operator that maps an activation map to an attention representation.
Speciﬁcally, A takes a 3D activation map
F as input and outputs a ﬂattened 2D tensor along the channel dimension.
We explore three possible formulations of the attention operator A as suggested in (Zagoruyko & Komodakis, 2017):
A sum ( F l ) =
C (cid:88) i = (cid:12)(cid:12) F li (cid:12)(cid:12) ;
A p sum ( F l ) =
C (cid:88) i = (cid:12)(cid:12) F li (cid:12)(cid:12) p ;
A p mean ( F l ) = 1 C
C (cid:88) i = (cid:12)(cid:12) F li (cid:12)(cid:12) p , (1) where F li is the activation map of the i -th channel, | · | is the absolute value function and p > 1 .
Intuitively, A sum reﬂects all activation regions including both the benign and the backdoored neurons.
A psum is a generalized version of A sum that ampliﬁes the disparities between the backdoored neurons and the benign neurons by an order of p .
In other words, the larger the p is, the more weight is placed on the parts with highest neuron activations.
A mean aligns the activation center of the backdoored neurons with that of the benign neurons by taking the mean over all activation regions.
An empirical understanding of the three attention representations is provided in the experiments.
Attention Distillation Loss.
A detailed structure of our NAD framework is illustrated in Figure 1(c).
For ResNets (He et al., 2016), we compute attention representations using one of the proposed attention functions after each group of residual blocks.
The teacher network is kept ﬁxed throughout the distillation process.
The distillation loss at the l -th layer of the network is deﬁned in terms of the teacher’s and the student’s attention maps: [Equation] (cid:13)(cid:13)
[Equation]
(cid:13) (cid:13) (cid:13)(cid:13)(cid:13)
[Equation]
[Equation] where (cid:107)·(cid:107) 2 is the L 2 norm and A ( F lT ) /
A ( F lS ) is the computed attention maps of the teacher/student network.
It is worth mentioning that the normalization of the attention map is crucial to a successful distillation (Zagoruyko & Komodakis, 2017).
Overall Training Loss.
The overall training loss is a combination of the cross entropy (CE) loss and the sum of the Neuron Attention Distillation ( NAD ) loss over all K residual groups:
[Equation]
where L CE ( · ) measures the classiﬁcation error of the student network, D is a subset of clean data used in ﬁnetuning, l is the index of the residual group, and β is a hyperparameter controlling the strength of the attention distillation.
Before the attention distillation, a teacher network should be in place.
We ﬁnetune the backdoored student network on the same clean subset D to obtain the teacher network.
We will investigate how the choice of the teacher network affects the performance of NAD in Section 4.4.
We only distill the student network once as our NAD approach is effective enough to remove the backdoor by only a few epochs of distillation.
A comprehensive analysis of iteratively applied NAD is also provided in the Appendix G.
E XPERIMENTS
In this section, we ﬁrst introduce the experimental setting.
We then evaluate and compare the effectiveness of NAD with 3 existing backdoor erasing methods on 6 state-of-the-art backdoor attacks.
Finally, we provide a comprehensive understanding of NAD.
4. E XPERIMENTAL S ETTING
Backdoor Attacks and Conﬁgurations.
We consider 6 state-of-the-art backdoor attacks: 1)
BadNets (Gu et al., 2019), 2) Trojan attack (Liu et al., 2018b), 3) Blend attack (Chen et al., 2017), 4)
Clean-label attack(CL)
(Turner et al., 2019) , 5) Sinusoidal signal attack(SIG) (Barni et al., 2019), and 6) Reﬂection attack(Refool) (Liu et al., 2020b).
For a fair evaluation, we follow the conﬁguration, including the trigger patterns, the trigger sizes and the target labels, of these attacks in their original papers.
We test the performance of all attacks and erasing methods on two benchmark datasets, CIFAR-10 and GTSRB, with WideResNet (WRN-16-1 1 ) being the base model throughout the experiments.
More details on attack conﬁgurations are summarized in Appendix A.
Defense Conﬁguration.
We compare our NAD approach with 3 existing backdoor erasing methods: 1) the standard ﬁnetuning, 2) Fine-pruning (Liu et al., 2018a), and 3) mode connectivity repair
[Table]
[Table]
For NAD, we ﬁnetune the backdoored model (i.e. the student network) on the 5% accessible clean data for 10 epochs using the Stochastic Gradient Descent (SGD) optimizer with a momentum of 0.9, an initial learning rate of 0.1, and a weight decay factor of 10 − 4 .
The learning rate is divided by 10 after every 2 epochs.
We use a batch size of 64, and apply typical data augmentation techniques including random crop (padding = 4), horizontal ﬂipping, and Cutout (n holes=1 and length=9) (DeVries & Taylor, 2017).
The data augmentations are applied to each batch of training images at each training iteration, following a typical DNN training process.
The same data augmentations are also applied to other ﬁntuning-based baseline methods.
Additionally, we have a comparison of our
NAD method to just using data augmentations Cutout and Mixup in Appendix B. For the distillation loss, we compute the attention maps using the A 2 sum attention operator after the three groups of residual blocks of WRN-16-1 (see Figure 1(c)) .
An extensive study on four attention functions is given in Section 4.3.
For the hyperparameter β , we adaptively set it to different values for each backdoor attack.
We provide more details on defense settings in Table 3 (see Appendix A).
Evaluation Metrics.
We evaluate the performance of defense mechanisms with two metrics: attack success rate (ASR), which is the ratio of backdoored examples that are misclassiﬁed as the target label, and model’s accuracy on clean samples (ACC).
The more the ASR drops and the less the ACC drops, the stronger the defense mechanism is.
4. E FFECTIVENESS OF O UR NAD D EFENSE
In order to assess the effectiveness of our proposed NAD defense, we evaluate its performance against 6 backdoor attacks using two metrics (i.e. ASR and ACC).
We then compare the performance of NAD with the other 3 existing backdoor defense methods in Table 1.
Our experiment shows that our NAD defense remarkably brought the average ASR from nearly 100% down to 7.22%.
In comparison, Finetuning, Fine-pruning, and MCR are only able to reduce the average ASR to 37.36%, 67.00%, and 25.59% respectively.
MCR has an erasing effect stronger than that of the NAD’s by 0.12% on BadNets and by 0.9% on
SIG, but its performance against the 4 other attacks are much poorer.
Speciﬁcally, MCR failed to defend 60.29% more attack on Blend, 23.77% more attack on CL, and 18.77% more attack on Trojan in comparison to NAD.
Our hypothesis on this is that backdoor triggers with much complicated adversarial noises (e.g. random or PGD perturbations) would hinder the training of connect path and consequently gives MCR a hard time at ﬁnding robust models against these backdoor attacks.
Interestingly, ﬁnetuning did moderately well in erasing all kinds of attacks.
A reasonable explanation to this is that the data augmentation techniques used in the early stage might have recovered some trigger-related images leading the backdoor model to unlearn the original trigger.
For instance, the black edges in the zero-padding of images may have effects similar to the black-white trigger in
BadNets.
On the other hand, Fine-pruning gives a poor performance on mitigating all backdoor attacks under our experiment settings.
We speculate that a potential reason behind this is due to the low number of neurons in the last convolutional layer of WRN-16-1, indicating a high mixing of [Figure] benign neurons and backdoored neurons.
This may cause a signiﬁcant reduction in the classiﬁcation accuracy and in consequence makes the pruning ineffective.
In summary, all erasing methods have some negative effects on the ACC, but the drops by utilizing
NAD is the least prominent (merely 2.66%).
More comparisons to data augmentation techniques and the effectiveness in erasing all-target and adaptive backdoor attacks can be found in Appendix B, H and Appendix K respectively.
Effectiveness under Different Percentages of Clean Data.
We are also interested in studying the correlation between the performance of NAD and the amount of available pristine data.
Intuitively, we anticipate NAD to be stronger when we have more clean training data, and vice versa.
The performance of NAD and 3 other defense mechanisms with various sizes of “purifying dataset” is recorded in Figure 2.
It is within our expectation that both MCR and our proposed NAD approach are capable of defending against all 6 backdoor attacks almost 100% of the time when 20% of clean training data are available to us.
Nonetheless, NAD still beats MCR in terms of the convergence rate.
We will show in Appendix C that our proposed NAD approach converges much faster than MCR.
Additionally, we ﬁnd that ﬁnetuning becomes much more effective when the backdoor model is retrained on 20% of the clean data.
Despite the improvement, the standard ﬁnetuning method is still considerably worse than MCR and NAD by ∼ 10% in terms of the ASR.
Surprisingly, Fine-pruning gains almost no beneﬁts from clean data additional to the 5% used in the ﬁrst experiment.
This is because the benign neurons and the backdoored neurons are highly blended in the last layer of the backdoored network, leading to excessive pruning of the backdoored neurons.
Retraining becomes pointless when too many backdoored neurons have been removed.
In short, even with just 1% of clean training data available, our NAD can still effectively bring the average ASR from 99.04% down to 35.93%, while only sacriﬁces 4.69% of ACC.
Comparison to Trigger Recovering.
Some existing works proposed defense methods that predict the distributions of backdoor triggers through generative modeling or neuron reverse engineering.
The predicted distributions can subsequently be sampled to craft backdoored data with correct labels .
These “remedy data” can subsequently be used in retraining to alleviate the impact of backdoor triggers (Wang et al., 2019; Qiao et al., 2019).
In this subsection, we compare the performance of our NAD defense to one such approach, MESA (Qiao et al., 2019), which is the current state-of-the-art trigger recovering method.
Speciﬁcally, we are interested in comparing our method to the retraining-based method in general.
To do this, we ﬁrst retrain the backdoored model separately using the trigger generated by MESA (Rec-T) and the original trigger (Org-T).
We then evaluate the performance of these retrained models and compare their performances to that of the NAD’s.
The results are presented in Table 6 (see Appendix D).
The retraining-based approach is strong at defending against the BadNets attack.
It is able to reduce the ASR from 100% to under 5% without signiﬁcantly sacriﬁcing the ACC.
Nonetheless, its performance is nowhere close to our NAD defense when facing the CL attack.
This is a good indication that the backdoored neurons can be ﬁxed by retraining with the remedy data when the backdoor behavior is induced by only a few backdoored neurons.
On the other hand, it is too much for these remedy data to handle backdoors obtained through the combination of complex adversarial noises and stronger trigger patterns. [Figure]
4. A C OMPREHENSIVE U NDERSTANDING AND A NALYSIS OF NAD
In this section, we ﬁrst provide intuition behind what an attention map is from the visual perspective, we then compare the efﬁcacy of various choices of attention representation as promised in Section 3.1.
Finally, we explore the adjustment border of the hyperparameter β .
Understanding Attention Maps.
The activation information of all neurons in a layer of a neural network can be referred from the attention map of that layer.
The conjunct of all attention maps hence reﬂects the most discriminative regions in the network’s topology (Lopez et al., 2019).
To give intuition on how attention maps help NAD with erasing backdoor triggers, we visualize and compare the attention maps before and after backdoor erasing in Figure 3.
We use the attention function A 2 sum to derive the attention maps.
For the backdoored models, all attention maps are completely (i.e. group1, group2, and group3) biased towards the backdoor trigger region, implying that the backdoor trigger can easily mislead the network to misbehave.
The objective of backdoor erasing methods is consequently to relax the tension in this region.
We show the results for two attacks, BadNets (left) and CL (right), to validate our hypothesis.
Attention maps can also be used as an indicator to deduce the performance of backdoor erasing methods.
In Figure 3, we show the attention maps of a backdoored WRN-16-1 after puriﬁed by four different mechanisms: Fine-pruning, the standard ﬁnetuning, MCR, and our NAD approach.
As expected, BadNets can easily be erased by the most defenses.
To see why, refer to the attention maps of group 3; the network puriﬁed by the standard ﬁnetuning, MCR, and NAD pay almost no attention to the bottom right corner where the trigger is injected.
For CL, only MCR and NAD are able to distract the backdoored model’s from focusing on the triggered regions, which is also foreseeable as CL is a stronger attack.
In addition, the activation intensity of NAD in the benign area (i.e. non-trigger area) is correspondingly greater than that of MCR, which also justiﬁes our conclusion that NAD is better than MCR in erasing CL attacks from another perspective.
Next, we compare the performance of NAD under scenarios where 4 different attention functions,
A mean , A 2 mean , A sum , and A 2 sum are in place.
We use the BadNets attack as our benchmark attack.
Again, we evaluate the performance of NAD using two metrics, the ASR and the ACC, and the results are summarized in Appendix F. Despite all choices of attention function are able to help
NAD erase backdoors quite efﬁciently, A 2 sum achieved the best overall results.
A comparison to using the raw activation map for distillation can be found in Appendix I. Attention Distillation VS.
Feature Distillation.
We outline two advantages of attention distillation over feature distillation: 1) Integration .
The attention operators calculate the sum (or the mean) of the activation map over different channels (see Equation 1).
It can thus provide an integrated measure of the overall trigger effect. On the contrary, the trigger effect may be scattered into different channels if we use the raw activation values directly. Such a difference between the attention and the feature maps is visualized in Figure 11 and 12 in Appendix I. Therefore, aligning the attention maps between the teacher and the student networks is more effective in weakening the overall trigger effect than directly aligning the raw feature maps. 2) Regularization .
Due to its integration effect, an attention map contains the activation information of both backdoor-ﬁred neurons and the benign neurons.
This is important as the backdoor neurons can receive extra gradient information from the attention map even when they are not activated by the clean data.
Moreover, attention maps have lower dimensions than feature maps.
This makes the attention map based regularization (alignment) more easily to be optimized than feature map based regularization.
Effect of Parameter β . The selection of the distillation parameter β is also a key factor for NAD to erase backdoor triggers successfully.
We show the results of the coarse tuning β for all the backdoor attacks in Appendix E, and it reveals that β can certainly be tuned more to improve the performance of NAD.
In short, the process of ﬁnding the right scaling factor β is to ﬁnd a balance between the ASR and the ACC.
Even though bigger β is more effective against backdoor attacks, Figure 8(right) in Appendix E shows that arbitrarily increasing β may cause a degradation in the ACC.
For instance, the puriﬁed network lost more than 50% of ACC when β is set to 50,000.
A practical strategy to select β is to increase β until the clean accuracy (right subﬁgure in Figure 8) drops below an acceptable threshold. This can reliably ﬁnd an optimal β , as increasing β can always improve the robustness (left subﬁgure in Figure 8).
4. F URTHER E XPLORATION OF NAD
Here, we explore how the combinations of teachers and students and the choice of the teacher affect the defense performance of NAD.
For simplicity, we deﬁne B to be the backdoored network, B-F to be the ﬁnetuned backdoored network, and C to be the model trained from scratch using 5% of clean training data.
Here we consider 4 combinations of networks: 1) B teacher and B-F student, 2) B-F teacher and B student, 3) B-F teacher and B-F student, and 4) C teacher and B student. [Figure] [Figure]
Effect of Teacher-Student Combinations.
The ASR and ACC of NAD-puriﬁed networks using different teacher-students combinations are presented in Figure 4.
We ﬁnd that the ASR slightly diminished with the combination of B teacher and B-F student.
On the contrary, when using a BF teacher and a B-F student in the NAD framework, the ASR is drastically lowered (signiﬁcantly improved defense).
Interestingly, compared with the standard setting used throughout the previous experiments (i.e. Teacher: B-F , Student: B ), using a trained-from-scratch model as the teacher can also reduce the average ASR by more than 80%.
Furthermore, this combination works even better in tackling the Trojan attack.
Nonetheless, the combination of C teacher and B student signiﬁcantly deteriorates the ACC of the puriﬁed model, which is also predictable because the amount of clean data available to train a model from scratch has a direct effect on the model’s accuracy.
It is hence not a surprise that an incompetent teacher misleads the student network into learning ﬂawed features.
Effect of the Choice of a Teacher.
Due to the competitive performance offered by the C teacher and B student combination in the last section, we are interested in exploring one question: is the standard setting (i.e. B-F teacher and B student) we used throughout previous experiments still the best option when more clean data are available to us ?
To answer this question, we compare [Table] the performance of both B-F teacher and C teacher to erase B student under various percentages of available clean data on CIFAR-10.
We use the same training conﬁgurations described in Section 4.1.
The results are shown in Figure 5.
Compared with the ﬁnetuning option of B-F teacher, the C teacher trained on a subset of clean data also offers competitive performance.
Speciﬁcally, in the case where we have 20% of clean data available to us, the training option reduces the ASR by 0.15% in comparison to the ﬁnetuning option; nonetheless, it reduces the ACC by a greater amount of 2.5%.
This makes ﬁnetuning still a better option even when we have 20% clean data at hands.
Effectiveness of Different Teacher Architectures.
This experiment is conducted on CIFARagainst BadNets attack.
We consider 5 teacher architectures: WRN-10-2, WRN-16-1, WRN-16-2,
WRN-40-1 and WRN-40-2.
We ﬁx the student network to WRN-16-1. Since the teachers are of different (except one) architectures as the student, we train the teacher networks from scratch using only 5% clean training data. Our NAD is applied following the same conﬁgurations in Section 4.1. The results are reported in Table 2.
We can see that all 5 teacher networks are able to purify the backdoored student effectively under our NAD framework.
NAD can reduce the ASR from 100% to 4.68% even when a small teacher WRN-10-2 is used.
This conﬁrms that our NAD defense generalizes well across different network architectures.
Note that the clean accuracy of NAD decreases more than in the same architecture setting (see Table 1).
This is because the teachers have lower clean accuracy when trained from scratch on only 5% clean data, as we have analyzed in Figure 4.
Why a ﬁnetuned teacher can purify a backdoored student?
During the NAD process, the backdoored neurons in the ﬁnetuned teacher network are less likely to be activated by the clean ﬁnetuning data.
Moreover, as shown in Figure 12 (Appendix I), ﬁnetuning can suppress the trigger effect, and at the same time, boost the benign neurons (more visible light green regions in Figure 12 (b)).
By using the squared sum attention map A 2 sum (analyzed in Section 4.3 and Appendix I), this trigger erasing effect can accumulate from the already ﬁnetuned teacher network, leading to more robust and cleaner student network.
Note that the student can still overﬁt to the partially puriﬁed teacher if it is overly ﬁnetuned.
Overﬁtting in NAD.
Here, we run NAD for a sufﬁciently long time (e.g. 20 epochs) to test if the student will eventually overﬁt to the ﬁnetuned teacher network.
This experiment is conducted on CIFAR-10 against BadNets and CL attacks.
As shown in Figure 13 (Appendix J), the student network of NAD can indeed overﬁt to the partially puriﬁed teacher network.
However, this can be effectively addressed by a simple early stopping strategy: stop the ﬁnetuning when there are no signiﬁcant improvements on the validation accuracy within a few epochs (e.g. at epoch 5).
Note that this also highlights the efﬁciency of our NAD defense as only a few epochs of ﬁnetuning is sufﬁcient to erase the backdoor trigger.
C ONCLUSION
In this work, we proposed a novel knowledge distillation based backdoor defense framework Neuron
Attention Distillation (NAD).
We demonstrated empirically that our proposed approach is able to achieve a superior performance against 6 state-of-the-art backdoor attacks in comparison to 3 other backdoor defense methods.
Additionally, we propose the use of attention maps as an intuitive way to evaluate the performance of backdoor defense mechanisms due to their capability of displaying backdoored regions in a network’s topology visually.
On top of that, we explored how different experimental settings might affect our proposed method.
Empirical results demonstrate that our results are fairly resilient to the changes in experimental settings and thus can be conveniently em- ployed tuning
Overall, our proposed NAD backdoor defense framework provides the backdoor threat in model deployment
A M ORE IMPLEMENTATION DETAILS
The backdoor triggers used in our experiments are shown in Figure 6.
[Table] [Table]
Detailed implementation on 6 state-of-the-art backdoor attacks:
• BadNets: The trigger is a 3 × 3 checkerboard (pixel values are 128 or 255) at the bottom right corner of images.
We labeled the backdoor examples with a chosen target label and achieved an attack success rate of 100% with an injection rate of 10%.
• Trojan attack.
We follow the method proposed in the paper to reverse engineer a 3 × 3 square trigger from the last fully-connected layer of the network.
In order to reduce the impact on clean accuracy, we poisoned only 5% of training data with the reverse-engineered Trojan trigger.
We achieved an attack success rate of 100% with an injection rate of 5%.
• Blend attack.
We used the random patterns reported in the original paper.
We achieved an attack success rate of 99.97% with an injection rate of 10% and a blend ratio of α = 0 .
• Clean-label attack (CL).
We followed the same settings as reported in the paper.
Speciﬁcally, we used Projected Gradient Descent (PGD) to generate adversarial perturbations bounded to Lmaximum perturbation (cid:15)
= 16.
The trigger is a 3 × 3 grid at the bottom right corner of images.
We achieved an attack success rate of 99.21% with an injection rate of 80%.
• Sinusoidal signal attack (SIG).
We generate the backdoor trigger following the horizontal sinusoidal function deﬁned in their paper with ∆ = 20 and f = 6 .
We achieved an attack success rate of 99.91% with an injected rate of 10%.
• Reﬂection attack (Refool).
The implementation is based on the open-source code 2 .
We achieved an attack success rate of 95.16% with an injection rate of 80%.
More Details on Defense Baselines We adopted the same settings used in NAD for the standard ﬁnetuning approach and ﬁnetuned the model until convergence.
We replicated Fine-pruning 3 via
PyTorch and pruned the last convolutional layer of the model as suggested in the original paper Liu et al.
(2018a).
For a fair comparison, the pruning ratio was set to a value such that the ACC of the pruned network matched the ACC of our NAD approach.
We used the open-source code 4 for mode connectivity repair (MCR) and set the endpoint model t = 0 and t = 1 with the same backdoored WRN-16-1.
We trained the connection path for 100 epochs and evaluated the defense performance of the model on the path.
Other settings of the code remain unchanged.
B COMPARISON WITH DATA AUGMENTATION TECHNIQUES
Cutout (DeVries & Taylor, 2017) and Mixup (Zhang et al., 2018) are popular data augmentation methods for CNNs.
Cutout masks out random sections of input images during training and Mixup randomly morphs the training images.
We evaluate in this section the independent effectiveness of
Mixup and Cutout in erasing backdoor triggers.
For Cutout 5 , we set the number of patches to be cut out of each image to 1 and each patch is a 3 × 3 square.
For Mixup 6 , we set α to be the default value of 1, indicating that we sample the weight uniformly between zero and one.
Other settings for attacks and defenses are identical to the settings speciﬁed in Section 4.1.
The results (see Table 4) can be a supplement of Table 1.
We conclude that data augmentation techniques have mitigating effects on backdoors only when the transformation images are similar to the trigger patterns.
They are hence not general against a wide range of backdoor attacks
[Table] [Table]
C MORE RESULTS OF M ODE C ONNECTIVITY R EPAIR (MCR)
We use the open-source code of MCR and compare its performance to our NAD method.
The experiments are conducted on CIFAR-10 dataset using 5% clean ﬁnetune data.
We ﬁrst run MCR with the two endpoint models t = 0 and t = 1 which use the same backdoored WRN-16-1 model.
Figure 7 shows the convergence rate of MCR and our NAD against BadNets attack.
We then run an additional experiment for MCR using two different endpoint models: t = 0 and t = 1 use the backdoored
WRN-16-1 and the ﬁnetuned backdoored WRN-16-1 respectively.
This result is reported in Table 5.
We ﬁnd that using different endpoint models can not further improve the performance of MCR.
[Figure]
BackdoorAttack Before [Table] D EXPERIMENTAL RESULTS OF TRIGGER RECOVERING TECHNIQUE
Qiao et al.
proposed MESA that recovers the trigger distribution via generative modeling and then removes the backdoor by model retraining.
We implemented this work based on their opensource code 7 .
Note that we report the best averaging results of defense performance and we changed nothing in the code besides setting the proportion of available training data to 5%.
We present the results in Table 5.
[Table]
E EXPERIMENTAL RESULTS OF HYPER-PARAMETER β
We only give a rough estimate of β for all the backdoor attacks in Figure 8 and it certainly provides better results by a more granular level of tuning. [Figure]
F EXPERIMENTAL RESULTS OF DIFFERENT ATTENTION FUNCTIONS
We compare the performance of NAD under scenarios where 4 different attention functions, A mean ,
A 2 mean , A sum [Figure] and A 2 sum are employed.
We use the BadNets attack as our benchmark attack.
Again, we evaluate the performance of NAD using two metrics, the ASR and the ACC
[Figure]
and the results are summarized in Table [Figure] [Figure]
[Table]
G EXPERIMENTAL RESULTS OF ITERATIVE NAD
We evaluate whether NAD can be further improved with multiple iterations of distillation.
In this experiment, we adopted the same conﬁguration and set the iteration times to 5.
Taking the BadNets attack as an example.
The results in Figure 9 show that the attack rate has not been further reduced, and has even slightly increased by 2% in some cases.
We hypothesize that the attentions of the backdoored neurons have been correctly aligned with the attentions of the benign neurons after a single-iteration of erasing.
Whereas multiple iterations of distillation will make NAD refocus on the trigger pattern.
Therefore, we believe that one-iteration of distillation is sufﬁcient to guarantee the best result. Note that iterative Finetuning does not lead to further improvement over one-time ﬁnetuning neither.
H ERASING ALL-TARGET BACKDOOR ATTACKS
Unlike a single-target attack where the goal is to misclassify all backdoored images as one prespeciﬁed target class, an all-target attack aims to misclassify every source class label as different ones (in our case, misclassify the original label i as ( i + 1) % 10).
In this experiment, we adopted the same settings (i.e. single target attacks on BadNets) to conduct all-target attacks on the WRN16-1 network.
We found that an all-target attack is a tougher task than a single-target attack.
It is harder to attain a satisfactory ASR with an all-target attack.
The results in Figure 10 show that NAD is able to reduce the ASR across all poison-classes (from 79% to 9.7%) effectively with only 5% of clean training data.
I FEATURE MAPS V.S. ATTENTION MAPS
A natural question to ask is: why attention maps instead of feature maps?
This can be traced back to the ﬁeld of knowledge distillation. Directly aligning the feature maps could lead to an information loss on the sample density in the space, and this could lead to a decrement in the distillation performance (Zagoruyko & Komodakis, 2017; Huang & Wang, 2017; Lopez et al., 2019).
In the context of backdoor erasing, aligning the feature maps is not a good option because the backdoor neurons are only weakly, if not at all, activated by clean samples (Gu et al., 2019).
In contrast, attention maps contain integrated information (see Equation 1) of both backdoored and benign neurons’ feature maps, even when the neurons are not ﬁred.
(see Table 8).
Figure 11 visualizes activation maps of a backdoored image on BadNets, Finetuned BadNets with 5% of clean training data, and
BadNets erased by NAD with 5% of clean training data.
The attention maps aggregated across the channels using 5 different attention functions are shown in Figure 12.
[Figure] [Table] [Figure] [Figure]
J OVERFITTING IN NAD
Here, we run NAD for a sufﬁciently long time (e
[Figure]
g. 20 epochs) to test if the student will eventually overﬁt to the ﬁnetuned teacher network.
This experiment is conducted on CIFAR-10 against BadNets and CL attacks.
We also run the Finetuning defense as a comparison.
Note that, the teacher network of NAD is only ﬁnetuned for 10 epochs, following the settings in Section 4.1.
As shown in Figure 13 (Appendix J), the student network of NAD can indeed overﬁt to the partially puriﬁed teacher network.
However, this can be effectively addressed by a simple early stopping strategy: stop the ﬁnetuning when there are no signiﬁcant improvements on the validation accuracy within a few epochs (e.g. at epoch 5).
As shown by the green curves, the clean accuracy of NAD ﬁrst drops, then quickly recovers and stabilizes at a high level within a few epochs.
This also highlights the efﬁciency of our NAD defense as only a few epochs of ﬁnetuning is sufﬁcient to erase the backdoor trigger.
K EFFECTIVENESS AGAINST ADAPTIVE ATTACKS
A backdoor adversary may attempt to construct a more stealthy backdoor trigger that does not cause obviouse shift of the attention.
To simulate this scenario, we design an adaptive version of BadNets on CIFAR-10 that attaches the trigger pattern at the center region of the image.
Such an adaptive attack will only shift the attention close to the center region and has a weaker activation response.
Since most of the CIFAR-10 objects are located at the center of the clean images, this adaptive attack may make the attention distillation much less effective.
Figure 14 illustrates a few examples of backdoored images for this type of attack.
We use a scaling parameter α ∈ [0 , 1] to adjust the pixel value of a black-white square trigger pattern (all images are normalized into the range of [0 , 1] ).
For example, for α = 0.2, we scale pixel values of the trigger pattern p to p × α .
The results of our
NAD against this adaptive attack are reported in Table 9.
Our NAD method can still effectively erase the adaptive attack while maintaining high accuracy on clean data.
Interestingly, Finetuning can only effectively erase the weaker trigger, and not as effective as NAD (especially in the case of α = 1.0 attack).
We conjecture this is because the center regions are more hard overwritten by the clean images used for ﬁnetuning.
We leave the exploration of more advanced adaptive attacks for future work.
[Table]
[Table]