Training with More Confidence: Mitigating Injected and Natural Backdoors During Training
Anonymous Author(s)
AffiliationAddress email
Abstract
The backdoor or Trojan attack is a severe threat to deep neural networks (DNNs).
Researchers find that DNNs trained on benign data and settings can also learnbackdoor behaviors, which is known as the natural backdoor.
Existing works onanti-backdoor learning are based on weak observations that the backdoor and benign behaviors can differentiate during training.
An adaptive attack with slowpoisoning can bypass such defenses.
Moreover, these methods cannot defendnatural backdoors.
We found the fundamental differences between backdoorrelated neurons and benign neurons: backdoor-related neurons form a hyperplaneas the classification surface across input domains of all affected labels.
By further analyzing the training process and model architectures, we found that piecewise linear functions cause this hyperplane surface.
In this paper, we design anovel training method that forces the training to avoid generating such hyperplanes and thus remove the injected backdoors.
Our extensive experiments onfive datasets against five state-of-the-art attacks and also benign training showthat our method can outperform existing state-of-the-art defenses.
On average,the ASR (attack success rate) of the models trained with NONE is 54.83 timeslower than undefended models under standard poisoning backdoor attack and1.75 times lower under the natural backdoor attack.
Our code is available athttps://anonymous.4open.science/r/NOLE-84C3 .
Introduction
Deep Neural Networks (DNNs) are vulnerable to Trojans 1 .
A Trojaned model makes normal predictions on benign inputs and outputs the target label when the input contains a specific pattern (i e.,
Trojan trigger) such as a yellow pad.
To inject a Trojan [13, 18, 4, 40, 3, 7, 42], the adversary canpoison the training dataset by adding poisoning samples (or Trojan samples): inputs stamped withthe Trojan trigger and labeled as the target label.
This is a typical data poisoning attack and themodel can learn the trigger as a strong feature for the target label.
Recently, researchers found theexistence of natural Trojans .
Namely, a model trained on benign datasets with normal settings (e.g.,hyperparameters, optimizers) can also learn Trojans, when there exists a strong input pattern in thetraining dataset that corresponds to one label [26].
In such natural Trojan scenarios, the input patternserves as a Trojan trigger, and its corresponding label is the target label.
By reverse-engineering thetrigger, the adversary can leverage it for attacks.
As such, both injected and natural Trojans aresevere threats to DNNs.
There are no existing works for learning a robust DNN against both injected and natural Trojans.
Existing works focus on training benign classifiers when the dataset is poisoned.
For example,
ABL [23] observes that the model will learn backdoor behavior faster than benign behavior, andproposes a training algorithm to suppress learning the trigger pattern.
DP-SGD is an optimizationmethod that leverages the differential privacy (DP) method and combines it with stochastic gradientdescent (SGD) to learn a robust classifier using poisoned datasets.
These methods fail to defendagainst the natural Trojan.
[Figure]
In this paper, we propose a robust training algorithm thatcan mitigate both injected and natural Trojans.
For agiven Trojan in the form of ˜ x = T ( x , m , t ) =
( 1 −m ) ⊙ x
+ m ⊙ t , where ˜ x and x are respectively the poisoning and benign samples, ˜ x is the poisoning samplegeneration method using the trigger ( m , t ) with triggermask matrix m and trigger value matrix t , we theoretically prove that there exists one and only one hyperplanein the input space that corresponds to all poisoning samples.
Thus, the trained classifier is mapping this hyperplane to a target label when performing Trojan attacks.
Fig.
1 intuitively illustrates our idea using a simple example.
To simplify the problem, each input has three dimensions ( d x , d y , d z ) .
We use red and blue dotsto denote inputs belonging to different labels, and the trigger is denoted as t = ( − , − , z t ) .
Addingthe trigger into an input i 0 = ( x 0 , y 0 , z 0 ) to get the corresponding Trojaned input i ′
= ( x 0 , y 0 , z t )is equivalent to moving this input to the z =
z t plane in the input domain.
As shown in Fig.
1,the input i 0 moves along the dashed line and ends up in the wheat colored plane, z
= z t .
Noticethat i 0 can be any input, and stamping the trigger will move them to the plane z
= z t .
In otherwords, the plane z =
z t contains all Trojan samples.
Likewise, if there exists a plane z
=
z t thatis a decision region, its corresponding input pattern ( − , − , z t ) is a Trojan trigger.
Stamping such atrigger to any input is essentially projecting the input to this plane, and because all the inputs in thisplane have the same label, it is equivalent to performing a Trojan attack.
Extending this to the highdimensional space, the Trojan region will be a hyperplane, and the decision boundaries it shareswith other regions will be linear.
In summary, we can say that a Trojan in a DNN always pairs witha hyperplane as its Trojan region.
Considering that modern DNNs are non-linear and non-convex,this rarely happens for benign models.
By further analysis, we found that this is related to the use ofactivation functions.
Modern DNNs tend to use piece-wise linear functions as their activation functions.
Even though the function itself is linear, its sub-functions are linear.
For example, one of themost popular activation function, ReLU (i.e, y = max(0 , x ) ), consists of two linear functions (i.e.,y = 0 , x ≤ 0 and y = x, x > 0 ).
When a model’s weights and biases are trained to specific regions,the neuron values before activation functions will fall into the input domain of only one sub-function(e.g.
, x > 0 ).
As a result, the output and input will form a linear relationship.
Consequently, themodel can generate a hyperplane decision region in the input domain.
In other words, we have ahyperplane, denoted as < x 0 , x 1 , x 2 > = <
a 0 , a 1 , a 2 > , as a decision region, and for a given inputi , if we replace its values in dimensions x 0 , x 1 and x 2 to a 0 , a 1 and a 2 , respectively, we can turn itsoutput label to a desired one.
A model training process includes randomness (e.g., random initiationvalues, optimization), which we cannot avoid.
Many possible decision regions will give us the sameor similar training/validation accuracy.
Some training will learn a linear decision region while otherswill not.
This explains the cause of DNN Trojans and answers the former question, which movesforward our understanding of DNN Trojans one more step.
Based on this analysis, we develop a revised training method, NONE ( NON -Lin E arity) that identifies linear decision regions, filters out inputs that are potentially poisoned, and resets affected neurons to enforce non-linear decision regions during training.
We evaluated our prototype built with
Python and PyTorch on MNIST, GTSRB, CIFAR, ImageNet, and the TrojAI dataset.
Comparedwith SOTA methods, NONE is more effective and efficient in mitigating different Trojan attacks(i.e.
, single-target attack, label-specific attack, label-consistent attack, natural Trojan attack, and thehidden trigger attack).
For example, on average, the ASR (attack success rate) of the models trainedwith NONE is 54.83 times lower than undefended models under standard poisoning Trojan attackand 1.75 times lower under the natural Trojan attack.
Our contributions in this paper can be summered as follows.
We analyze the cause of (injected andnatural) DNN Trojans and conclude that linearity in DNN decision regions is the main reason.
Further, We analyzed the source of linear DNN decision regions and explained why and when it happens using commonly used layers and activation functions.
Then, we propose a novel and general revisedtraining framework, NONE that detects and fixes injected and natural Trojans in DNN training.
Tothe best of our knowledge, we are the first to defend natural Trojans.
We evaluate NONE on fivedifferent datasets and five different Trojans and compare it with other SOTA techniques.
Resultsshow that NONE significantly outperforms these prior works in practice.
Related Work
Trojans in DNN.
Prior work [13, 6, 37, 31, 27, 34, 18, 4, 30, 22] demonstrates that the attackers can inject Trojans into the victim models by poisoning the training dataset.
Later, researchersfound that a DNN trained on the benign dataset with standard training procedure can also learn such
Trojans, which is known as natural Trojans.
By using reverse engineering methods designed for poisoned models, researchers were able to find natural triggers in pretrained models.
For example, in
ABS [26], authors show that a Network in Network (NiN) [24] model trained on a benign CIFAR-10dataset has natural Trojans.
Other works [34, 46] also documented similar findings in other models.
Trojan Defense.
One way to defend against Trojan attacks is to filter out poisoning data beforetraining [36, 5].
Poison suppression defenses [10, 16] restrain the malicious effects of poisoningsamples in the training phase.
Du et al.
[10] and Hong et al.
[16] apply DP-SGD [1] to depressthe malicious gradients brought from poisoned samples.
However, existing anti-backdoor learningmethods are based on weak observations and they do not find the root cause of Trojans.
They canbe bypassed by adaptive attacks.
In addition, they can not defend natural Trojans.
For example,the SOTA anti-backdoor learning method ABL [23] is based on the observation that learning ofbackdoor behaviors and benign behaviors are distinctive.
In detail, the training loss on backdoorexamples drops much faster than that on benign examples in the first few epochs.
It can be bypassedby adaptive attacks with source-specific target labels.
More specifically, when the backdoor attackis label-specific (i.e., the samples with different original labels have different target labels), thenthe model will not learn backdoor behavior faster.
We run ABL on the label-specific BadNetsattack [13] with ResNet18 model and GTSRB dataset.
The results show that the attack successrate is 93.93%, meaning that ABL can not defend against such an attack.
This is not surprisingbecause label-specific attack is more complex, and it is based on the benign behavior of the models(i.e. , the model needs to classify the original label of the backdoor examples first, then convert itto corresponding target labels).
Another line of work tries to detect if a model has Trojan or notbefore its deployment.
Model diagnosis based defenses [39, 43, 19, 17, 14, 32] determine if a givenmodel has a Trojan or not by inspecting the model behavior.
Model reconstruction based defensestry to eliminate injected Trojans in infected models [25, 45, 21, 41, 44], which requires retraining themodel with a set of benign data.
Another approach is to defend Trojan attacks at runtime.
Testingbased defenses [11, 8] judge if the given input samples contain trigger patterns and reject the onesthat are considered malicious.
Methodology
Threat Model.
In this paper, we consider training time defense against Trojan attacks, which is alsoused by existing works [16, 5, 36].
The training dataset can contain poisoning samples (to injectintended Trojans) or benign (but leads to natural Trojans).
As defenders, we control the trainingprocess but do not assume control over training datasets.
Adopted from existing work [39, 11, 8, 43], Trojan sample generation can be formalized as Eq.
1, where x and ˜ x respectively are the benigninput and Trojan input.
m , t respectively represent, mask of the trigger (i.e., whether a pixel is inthe trigger region) and contents of the trigger.
⊙ is the element-wise multiplication operation on twovectors, i.e., the Hadamard product. [Equation] ⊙
[Equation]
[Table]
3.1 Trojan Analysis
To facilitate our discussion, we first define decision region which includes all samples with the samepredicted label.
Formally, we define it as:
Definition 3.1.
(Decision Region)
For a deep neural network M : X
(cid:55)→ Y where X is the inputdomain R m
and Y is a set of labels { 1 . . .
k } , a decision region is an input space R l ⊆ X , s.t.,∀ x ∈ R , M ( x ) = l .
In most tasks, decision regions with the same label spread over the whole input space because naturalinputs belonging to the same label are naturally distributed in this way.
Similarly, we define Trojan
Decision Region , or in short, we call it the Trojan Region , which is a subregion of the target labeldecision region:
Definition 3.2.
(Trojan Region)
For a Trojaned deep neural network M : X (cid:55)→ Y with target labell , its Trojan regions are input spaces where T ⊆ R l , s.t., all Trojan inputs ˜ x ∈ T , and all inputs in
T are Trojan inputs.
Based on the definitions, we have the following theorem:
Theorem 3.3.
Given a model M : X
(cid:55)→ Y with the Trojan trigger ( m , t ) , there exists one andonly one hyperplane { Ax − b = 0 } Trojan region, where i ∈ { 1 . . .
m } , diagonal matrix
A i,i =m i , b
= At .
The proof for Theorem 3.3 and empirical results are in Appendix.
The theorem shows that when amodel learns a Trojan, it essentially learns a hyperplane as a decision region.
Based on the definitionof decision regions, we know that they are inverse functions of the model.
Thus, the inverse functionof the Trojan is a hyperplane.
To understand how this happens, we perform further analysis.
We start our discussion from typical Convolutional Neural Networks (CNNs).
A convolutional layerwith activation functions can be represented as y j = σ ( W T j x j + b T j ) , where x j and y j are the inputsand outputs of layer j , W j and b j are trained weights and bias values, and σ represents the activationfunction which is used to introduce non-linearity in this layer.
Most commonly used activationfunctions, e.g., ReLU, are piece-wise linear.
For example, ReLU is defined as σ ( x ) = max(0 , x ) ,which consists of two linear pieces separated at x = 0 .
As pointed out by Goodfellow et al.
[12],even non-piece-wise linear functions are trained to semi-piece-wise linear.
This helps resolve thegradient explosion/vanishing problem and makes training DNNs more feasible.
We make a key observation that DNN Trojans will increase linearity of a convolutional layer withactivation functions by introducing a large percentage of neurons activating on one piece of theactivation function .
Specifically, we observe that when the Trojan behavior happens, the neuronvalues before activation functions fall into one input range of the activation function, which makes itlinear.
Recall that most well-trained activation functions are piece-wise linear, and if the inputs arein one input range, it regresses to a linear function.
For example, layer j using ReLU as its activationfunction will be a linear layer if W T j x j + b T j ≥ 0 for all x j .
As such, the reverse function of the
Trojan can be a hyperplane or overlaps significantly with the hyperplane.
[Figure]
Fig.
2 shows the empirical comparison of a benign model and a Trojanmodel.
In our experiment, we train abenign model M and a Trojan model
M ′ using ResNet18 on CIFAR-10.
While training M ′ , we adopt the TrojanNet [35] training method whichguarantees that only certain neuronswill contain the Trojan, which we call
Trojan neurons.
The x-axis in Fig.
2shows the value of W T x + b T , andy-axis shows the percentage of Trojan neurons whose activation value isthe corresponding value on x-axis.
We use blue color to denote experiments when inputs have thetrigger and orange to denote when inputs do not contain the trigger.
As we can see, the model thatcontains the Trojan will activate the x > 0 region when the input contains the trigger.
By contrary,all the other three cases do not have such a phenomenon.
We also conduct similar experiments onother different models and different types of layers including non-piece-wise linear layers, and allempirical results confirm our observation here.
Details are in Appendix.
Algorithm
Input: Training Data: D , Maximal epoch: E
Output: Model: M 1: function T RAINING ( D ) 2: while e ≤ E and (not terminate( M ) ) do 3: ▷ Train and gather activation values 4:
M = train( D, e ) 5:
A = M. predict( D ) 6: ▷ Identify compromised neurons 7:
C = ∅ 8: for neuron n in M do 9: if P ( A n ≥ 0) ≥ θ then 10:
C = C ∪ { n } 11: ▷ Identify biased or poisoning samples 12: for neuron n ∈ C do 13:
B n , O n = separate( A n ) 14: µ, σ = norm( B n ) 15: for activation value of input
i : i n ∈ O n do 16: if (cid:12)(cid:12) i n − µ σ (cid:12)(cid:12) ≥ λ then 17:
D
=
D − i 18: ▷ Resetting compromised neurons 19: for neuron c ∈ C do 20: c = M. init( c )
Training Algorithm
To solve the Trojan problem caused by high linearity neurons, we propose Algorithm 1 to enforcenon-linearity in individual layers by resetting potentially linear neurons and removing data samplesthat cause such linearity.
This is a revised training process, which is an iterative process that trainsthe model until the maximal epochs (line 2).
It starts by training the model using the standardbackward propagation method (line 4).
We also gather all activation values of individual neurons Afor all training samples in the training dataset D .
Then, the process contains three steps: identifyingcompromised neurons (lines 6 to 10), identifying biased or poisoning samples (lines 11 to 17), andlastly, resetting the neurons for retraining (lines 18 to 20).
The first step is to identify comprisedneurons, namely the neurons that carry Trojans.
Based on our discussion on § 3.1, we do this bychecking the activation values of neuron n , denoted as A n , to see if its function is highly linearusing the condition P ( A n ≥ 0) ≥ θ .
If so, we make the neuron n as potentially compromised andadd it to the candidate set C .
The second step is to identify highly biased samples or poisoningsamples.
The overall design is a statistical testing process: we first find a reference distribution of aparticular neuron and then mark all inputs whose activation values do not follow such distributionsas potentially biased or poisoning samples.
The idea of finding a benign distribution is that inputswhose activation values in a layer are non-linear will be considered as benign and the distributionthat describes their activation values is used as the reference distribution.
In Algorithm 1, line 13uses a function to test the linearity of activation values and separate all activation values into a benignset B and others O .
Then, we normalize the distribution of our reference distribution B to a normaldistribution and obtain its mean value µ and standard deviation value σ (line 14).
For a single input i ,we perform a statistical test to see the probability of i being a potentially biased or poisoning sample(lines 15 to 16).
We exclude it from the training dataset (line 17).
The last step of our trainingalgorithm is to remove the compromised neuron effects from the model by resetting them.
We reusethe initialization method in our training setting to set a new value for all identified compromisedneurons (line 19 and 20).
Then, we continue the training until terminating conditions are met, orthe training budget is used up (line 2).
This algorithm uses ReLU as an example, and becauseour theory still holds for other activation functions (see Appendix), the algorithm can generalize toother models by modifying corresponding parameters (line 9).
To identify compromised neuronsand biased/poisoning images, we need to determine the threshold value θ (line 8) and separate theactivation values into non-linear and linear ones.
In our implementation, we perform the Fisher’slinear discriminant analysis (its binary version, Otsu’s method [28]) and leverage the Jenks natural breaks optimization algorithm to find the separations of non-linear and linear activation values.
Sets B and O are outputs of such algorithms, and we use the standard value that evaluates the quality ofsuch separation to compute the value of our threshold θ , i.e., 0 .
95 in our case.
Notice such valuescan affect the accuracy of identifying compromised neurons and inputs.
We also choose alternativeways to separate the sets and present the results in the Appendix to evaluate our approach.
Similarto model pruning and fine-tuning, when we reset different numbers of neurons (lines 19 and 20), theaccuracy on benign and Trojan samples can be different.
We evaluate the sensitivity of NONE tothe number of neurons reset during training, and identify malicious samples in Appendix.
Experiments
We first introduce the experiment setup (§ 5.1).
Then, we evaluate the overall effectiveness of NONE (§ 5.2) and investigate its robustness against different attack settings (§ 5.3).
Due to pagelength limit, resistance to more attacks, impacts of configurable parameters in NONE, generalization on complex applications, efficiency, and potential adaptive attacks are discussed in Appendix.
5.1 Experiment Setup.
NONE is implemented in Python 3.8 with PyTorch 1.7.0 and CUDA 11.0.
If not specified, allexperiments are done on a Ubuntu 18.04 machine equipped with six GeForce RTX 6000 GPUs, 642.30GHz CPUs and 376 GB main memory.
[Figure]
Datasets and Models.
We evaluate NONEon five publicly available datasets: MNIST, GTSRB, CIFAR-10, ImageNet-10 and TrojAI.
The overview of our datasets and more detailscan be found in Appendix.
Fig.
3 illustratesdifferent trigger patterns used in the experiments (i.e., a single pixel located in the rightbottom corner of Fig. 3(a), a fixed red patchin Fig. 3(b), a black-and-white pattern whoselocation is random in Fig. 3(c) and a colorful watermark in Fig. 3(d)).
The default trigger pattern for MNIST, CIFAR-10, GTSRB, and ImageNet-10 are Single Pixel (Fig. 3(a)), Static Patch (Fig. 3(c)), Dynamic Patch (Fig. 3(b)) and Watermark (Fig. 3(d)), respectively.
Besides the default triggers foreach dataset, we also measure the impacts of using other triggers.
The results and analysis are included in § 5.3.
We evaluate NONE and other defense methods on AlexNet [20], NiN (Network in
Network) [24], VGG11, VGG16 [33] and ResNet18 [15].
These models are representative and arecommonly used in existing Trojan related studies [26, 39].
Evaluation Metrics.
We use benign accuracy (BA) and attack success rate (ASR) [38] as evaluationmetrics, which is a common practice [9, 11, 39].
BA is defined as the number of correctly classifiedbenign samples over the number of all benign samples.
It implies model’s capability on its originaltask.
ASR evaluates the success rate of backdoor attacks.
It is calculated as the number of backdoorsamples that can successfully attack the model over the number of all generated backdoor samples.
Attack Settings.
As we introduced in § 2, Trojans are classified into two categories: Injected
Trojans and Natural Trojans .
We implement both of them to evaluate NONE and other defensemethods.
For Injected Trojans, we implement both single target and label specific BadNets [13],label-consistent Trojan attack [37] and hidden trigger Trojan attack [29].
Following the originalpaper, we use images from a pair of classes (class theater curtain and class plunger) in ImageNetfor hidden trigger Trojan attack.
For natural Trojans, we follow the previous work [26] to reproducethe attack.
Due to the fact that we do not know if the data is poisoned or not in advance, NONEkeeps detecting Trojan attacks and training the model even if there is no attack activity.
We evaluatethe additional costs of NONE on benign models in such non-attack settings.
More details for ourattack settings are included in Appendix.
Besides above attacks, We also evaluate the generalizationof NONE on more attacks in Appendix.
Comparison.
We compare NONE with 4 state-of-the-art defense methods: DP-SGD [16], Neural Attention Distillation (NAD) [21], Activation Clustering (AC) [5] and Anti-backdoor Learning(ABL) [23].
We use their official code and default hyperparameters specified in the original papers.
[Table]
NONE
[Table] [Table]
5.2 Effectiveness of NONE
Experiments.
We measure the effectiveness of NONE by comparing the BA and ASR of modelsprotected by NONE with those of undefended models and models protected by existing defensemethods.
The comparison results on injected Trojans, natural Trojans and non-attack settings areshown in Table 1, Table 2 and Table 3, respectively.
In each table, we show the detailed settingsincluding attack settings, dataset names and network architectures., etc.
For the evaluation resultson natural Trojans ( Table 2), the ASR and BA are the average results under different trigger sizesettings: 2%, 4%, 6%, 8%, 10% and 12% of the whole image.
Notice that, to the best of ourknowledge, there is no defense method designed for natural Trojans.
We observe that DP-SGDcan potentially mitigate natural Trojans because it reduces the high gradients brought from natural
Trojans.
Therefore, we adapt DP-SGD as the baseline method for natural Trojans.
We configure
DP-SGD with two settings of parameters following the prior work [16].
For DP-SGD-1, we set theclip as 4.0 and the noise as 0.1.
The clip and noise of DP-SGD-2 are 1.0 and 0.5.
For the non-attacksettings, we deploy NONE on several benign models and show the decrease of BA in Table 3.
Results on Injected Trojans.
From the results on Injected Trojan attacks (Table 1), we observethat applying NONE can better protect models from being attacked by injected Trojans than otherdefense methods.
With NONE, the average ASR of models decreases from 93.34% to 1.91%,which is much better than other defense methods (DP-SGD, NAD, AC and ABL can only reducethe average ASR to 30.32%, 34.08%, 45.45% and 68.60% respectively).
The reason is that, unlikeexisting methods based only on specific empirical observations, NONE targets the root cause of the
[Table] [Table]
[Table]
[Table]
Trojans (i.e., the linearity) and reveals the attacks more accurately.
Therefore, NONE can betterdefend against attacks than other methods.
We also find that NONE almost does not have negative impacts on the original task of models.
From Table 1, the BA of NONE is the highest among all methods and is similar to that of undefendedmodels, meaning that NONE has a low defense cost.
The reason is that NONE only modifies thecompromised neurons that are highly relevant to Trojan but less related to the original task of models.
Therefore, most of the benign knowledge is preserved when applying NONE and the model can stillperform well on its original tasks.
Meanwhile, NONE finetunes the model on the purified data afterthe reset process, further strengthening the capabilities of models and reducing defense costs.
It is worth clarifying that ABL has a poor performance on label-specific attacks and other attacksthat use NiN and VGG models.
The possible reason is that the design of ABL requires the modelto learn quicker and better on Trojan samples than benign samples [23] (i.e., the learning of Trojansamples should have a lower training loss value in the early training stage).
When the attacker usesmore complex attacks (e.g., label specific BadNets) or models with limited learning capabilities(e.g., NiN and VGG), the model cannot learn Trojan samples quickly, leading to poor performance.
Results on Natural Trojans.
From the results in Table 2, we find that NONE protects the modelmost effectively and has the lowest defense costs among all defense methods.
Overall, applying NONE achieves 1.75 times lower ASR than undefended models.
DP-SGD methods can onlyslightly decreases ASR or even increase ASR.
The results show that using NONE is the most effective way for natural Trojan defense.
The loss of BA using NONE is also smaller than the losscaused by applying the DP-SGD methods (3.90% with NONE and 38.76% with DP-SGD methodson average), further showing the efficiency of NONE.
The results confirm our analysis: reducingthe linearity of models reduces the ASR of natural Trojans without posing much additional cost.
Results on Non-attack Setting.
We also explore whether NONE affects the performance of benignmodels on their original tasks.
From Table 3, we find that NONE has a low effect on benign models.
Applying NONE only decreases 2.33% BA on average.
Because only a few neurons are detectedas compromised neurons and are reset by NONE when there is no Trojan activity, NONE does notaffect the learned benign knowledge.
Moreover, the subsequent training process further reduces thecosts.
Therefore, we conclude that NONE does not impose high additional costs on benign models.
5.3 Robustness of NONE
We evaluate the robustness of NONE against various attack settings (e.g., different trigger sizes,trigger patterns and poisoning rates).
If not specified, the model used in the evaluation is ResNet18.
The dataset is CIFAR-10 and the evaluated attack is the single target BadNets attack.
Trigger Sizes.
To study the effects of trigger size, we use triggers of different sizes (from 3*3 to20*20) to attack models and collect the ASR and BA of applying NONE on these compromisedmodels.
The results are shown in Table 5.
Overall, the BA of undefended models and modelsprotected by NONE is insensitive to the change of trigger size.
The difference between the highestand lowest BA on the unprotected and protected models is 0.48% and 1.52%, respectively, whichis very small.
We believe that the BA does not change significantly because triggers usually do notaffect the learning of benign features used for original tasks, as discussed in previous work [13].
On the other hand, the trigger size affects the ASR of protected models.
When the trigger sizebecomes larger, the ASR of the protected model increases dramatically from 1.07% to 99.82% and
NONE fails.
The results are understandable because a large trigger size modifies more pixels inthe original image, making the triggers obvious and easy to learn.
When the trigger is large, italmost covers the whole image and becomes the majority of the image.
In such a scenario, modelseasily capture trigger features and are compromised.
A detailed example is discussed in Appendix.
Currently, the sensitivity to trigger sizes is a common limitation for Trojan defense methods [39, 26].
Considering that a large trigger size is almost impractical because it makes the trigger too obviousto be detected directly by administrators, we consider NONE robust to most trigger sizes.
Trigger Patterns.
To measure the robustness of NONE against different attack trigger patterns, weuse NONE to protect models from being attacked by Dynamic Patch trigger, Static Patch trigger and
Watermark trigger.
The results are shown in Table 4.
We observe that NONE always achieves low
ASR and high BA under different trigger settings.
The results demonstrate that NONE is effectiveagainst different trigger pattern settings.
Moreover, we notice that the ASR of using the watermarktrigger is particularly larger compared with using other triggers.
The reason is that the watermarktriggers are large and more complex, as shown in Appendix.
Poisoning Rates.
To measure the impacts of different poisoning rates, we collect the ASR and BAof models being compromised at different poisoning rates from 0.50% to 20.00%.
The results aresummarized in Table 6.
Based on the results, we find that increasing the poisoning rate slightlydecreases both the BA and ASR.
Specifically, the BA of the model decreases by 1.36% and the ASRof the model decreases by 1.13%.
This is because increasing the poisoning rate reduces the numberof benign samples used for training, and the BA of the model naturally decreases.
Meanwhile, alarge poisoning rate makes Trojans easy to be detected, which leads to a lower ASR.
Since thechanges in ASR and BA are quite small, NONE is considered robust to most poisoning rates.
Discussion
In this paper, we focus the discussion on image classification tasks, which is the focus of manyexisting works [13, 29, 37, 26, 39, 2].
Expanding our work, including the theory and system to otherproblem domains, such as natural language processing and reinforcement learning, other computervision tasks, e.g., object detection, will be our future work.
Research on adversarial machine learning potentially has ethical concerns.
In this research, wepropose a theory to explain existing phenomena and attacks, and propose a new training method thatremoves Trojans in a DNN model.
We believe this is beneficial to society.
In our current threat model, the adversary can only inject poisoning data into the training dataset,and there is no existing adaptive attacks.
However, adaptive adversaries can still conduct attacksunder other threat models, and we discuss such case in Appendix.
Conclusion
In this paper, we present an analysis on DNN Trojans and find relationships between decision regionsand Trojans with a formal proof.
Moreover, we provide empirical evidence to support our theory.
Furthermore, we analyzed the reason why models will have such phenomena is because of linearityof trained layers.
Based on this, we propose a novel training method to remove Trojans duringtraining, NONE, which can effectively and efficiently prevent intended and unintended Trojans.
Checklist
For all authors..
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See § 6.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] See § 6.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes] See § 6.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See § 3.1.
(b) Did you include complete proofs of all theoretical results?
[Yes] See Appendix.3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] See Abstract, § 5, and Appendix.
(b) Did you specify all the training details (e g., data splits, hyperparameters,
how they
were chosen)?
[Yes] See § 5 and Appendix.
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[No]
(d) Did you include the total amount of compute and the type of resources used (e.g., type
of GPUs, internal cluster, or cloud provider)?
[Yes] See § 5.
If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] See § 5.
(b) Did you mention the license of the assets?
[Yes] See Appendix.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
for
(d) Did you discuss whether and how consent was obtained from people whose data [Table]
you’re using/curating?
[Yes] See Appendix.
[Table] (e) Did you discuss whether the data you are using/curating contains personally identifi- [Table]
able information or offensive content?
[Yes] See Appendix.
[Table] If you used crowdsourcing or conducted research with human subjects... [Table]
(a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
[N/A]
[Table]
(b) Did you describe any potential participant risks, with links to Institutional Review
[Table]
[-In Theorem 3.3, we have S 
-]
Board (IRB) approvals, if applicable?
[N/A]
[-From Step 3 and 4, we can conclude that S 
-]
[Table]
(c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
[N/A]
[Table]