Rethinking the Reverse-engineering of Trojan Triggers
Anonymous Author(s)
AffiliationAddress email
Abstract
Deep Neural Networks are vulnerable to Trojan (or backdoor) attacks.
Reverseengineering methods can reconstruct the trigger and thus identify affected models.
Existing reverse-engineering methods only consider input space constraints, e.g.,trigger size in the input space.
Expressly, they assume the triggers are static patternsin the input space and fail to detect models with feature-space triggers such as imagestyle transformations.
We observe that both input-space and feature-space Trojansare associated with feature space hyperplanes.
Based on this observation, wedesign a novel reverse-engineering method that exploits the feature space constrainto reverse-engineer Trojan triggers.
Results on four datasets and seven differentattacks demonstrate that our solution effectively defends both input-space andfeature-space Trojans.
It outperforms state-of-the-art reverse-engineering methodsand other types of defenses in both Trojaned model detection and mitigation tasks.
On average, the detection accuracy of our method is 93%.
For Trojan mitigation,our method can reduce the ASR (attack success rate) to only 0.26% with the
BA (benign accuracy) remaining nearly unchanged.
Our code can be found inhttps://anonymous.4open.science/r/FeatureRE-10B7 .
Introduction
DNNs are vulnerable to Trojan attacks [1–5].
After injecting a Trojan into the DNN model, theadversary can manipulate the model prediction by adding a Trojan trigger to get the target label.
Theadversary can inject the Trojan by performing the poisoning attack or supply chain attack.
In thepoisoning attacks, the adversary can control the training dataset and injects the Trojan by addingsamples with the Trojan trigger labeled as the target label.
In the supply chain attack, the adversarycan replace a benign model with a Trojaned model by performing the supply chain attack.
The Trojantrigger is becoming more and more stealthy.
Earlier works use static patterns, e g., a yellow pad asthe trigger, which is known as the input space triggers.
Researchers recently proposed using moredynamic and input-aware techniques to generate stealthy triggers that mix with benign features, whichare referred to as the feature space triggers.
For example, the trigger of the feature-space Trojans canbe a warping process [6] or a generative model [7, 8, 3].
The Trojan attack is a prominent threat tothe trustworthiness of DNN models, especially in security-critical applications, such as autonomousdriving [1], malware classification [9], and face recognition [10].
Prior works have proposed several ways to defend against Trojan attacks, such as removing poisonsin training [11–13], detecting Trojan samples at runtime [14–16], etc.
Many of them can onlywork for one type of Trojan attack.
For example, training and pre-training time defense (e.g.,removing poisoning data, training a benign model with poisoning data) fail to defend against thesupply chain attack.
Trigger reverse-engineering [17–21] is a general method to defend againstdifferent Trojan attacks under different threat models.
It works by searching for if there exists aninput pattern that can be used as a trigger in the given model.
If we can find such a trigger, themodel has a corresponding Trojan and is marked as malicious and vice versa.
Existing reverse- engineering methods assume that the Trojan triggers are static patterns in the input space and developan optimization problem that looks for an input pattern that can be used as the trigger.
This assumptionis valid for input-space attacks [1, 22, 2] that use static triggers (e.g., a colored patch).
Feature spaceattacks [6, 5, 7, 23, 8, 4, 3] break this assumption.
Existing trigger reverse-engineering methods [17–21] constrain the optimization by using heuristics or empirical observations on existing attacks, suchas pixel values are in range [0 , 255] , and the trigger’s size is small.
Such heuristics are also invalidfor feature-space triggers that change all pixels in images.
Reverse-engineering the feature space ischallenging.
Unlike input space, there are no constraints that can be directly used.
[Figure]
In this paper, we propose a trigger reverse-engineering method thatworks for feature space triggers.
Ourintuition is that features representingthe Trojan are orthogonal to other features.
Because a trigger works fora set of samples (or all of them, depending on the attack type), changingthe input content without removingthe Trojan features will not changethe prediction.
That is, changing Trojan and benign features will not affecteach other.
Trojan features will forma hyperplane in the high dimensionalspace, which can constrain the searchin feature space.
We then develop ourreverse-engineering method by exploiting the feature space input-space constraint.
Fig.
1 demonstrates our idea.
Existing reverse-engineering methods only consider the input-space constraint.
Itconducts reverse-engineering via searching a static trigger pattern in the input space.
These methodsfail to reverse-engineer feature-space Trojans whose trigger is dynamic in the input space.
Instead, ouridea is to exploit the feature space constraint and searching a feature space trigger using the constraintthat the Trojan features will form a hyperplane.
At the same time, we also reverse-engineer the inputspace Trojan transformation based on the feature space constraint.
To the best of our knowledge, weare the first to propose feature-space reverse-engineering methods for backdoor detection.
Through reversed Trojan transformation, we developed a Trojan removal method.
We implementeda prototype F EATURE RE ( FEATURE -space RE verse-engineering) in Python and PyTorch andevaluated it on MNIST, GTSRB, CIFAR, and ImageNet dataset with seven different attacks (i.e.,
BadNets [1], Filter attack [18], WaNet [6], Input-aware dynamic attack [7], ISSBA [8], Clean-labelattack [24], Label-specific attack [1], and SIG attack [25]).
Our results show that F EATURE RE iseffective.
On average, the accuracy of our method is 93%, outperforming existing techniques.
For
Trojan mitigation, our method can reduce the ASR (attack success rate) to only 0.26% with the BA(benign accuracy) remaining nearly unchanged by using only ten clean samples for each class.
Our contributions are summarized as follows.
We first find the feature space properties of the Trojanedmodel and reveal the relationship between Trojans and the feature space hyperplanes.
We propose anovel Trojan trigger reverse-engineering technique leveraging the feature space Trojan hyperplane.
We evaluate our prototype on four different datasets, five different network architectures, and sevenadvanced Trojan attacks.
Results show that our method outperforms SOTA approaches.
Background & Motivation
A DNN classifier is a function M : X (cid:55)→ Y where X is the input domain R m
and Y is a set of labels
K .
A Trojan (or backdoor) attack against a DNN model M is a malicious way of perturbing theinput so that an adversarial input x ′ (i.e., input with the perturbation pattern) will be classified to atarget/random label while the model maintains high accuracy for benign input x .
The perturbationpattern is known as the Trojan trigger.
Trojan attacks can happen in training (e.g., data poisoning)or model distribution (e.g., changing model weights or supply-chain attack).
Existing works haveshown Trojan attacks against different DNN models, including computer vision models [1, 22, 2],
Graph Neural Networks (GNNs) [26, 27], Reinforcement Learning (RL) [28, 29], Natural Language
Processing (NLP) [30–35], recommendation systems [36], malware detection [9], pretrained mod- els [37, 38, 19], active learning [39], and federated learning [40, 41].
The Trojan trigger can be asimple input pattern (e.g., a yellow pad)
[1, 22, 2] or a complex input transformation function (e.g., a
CycleGAN to change the input styles) [5, 7, 8, 6, 3].
If the trigger is static input space perturbations(e.g., a yellow pad), the Trojan attack is known as input-space Trojan , and if the trigger is an inputfeature (e.g., an image style), the attack is referred to as the feature-space Trojan .
There are different types of Trojan defenses.
A line of work [42] attempts to remove poisoned datasamples by cleaning the training dataset.
Training-based methods [43, 44] train benign classifierseven with the poisoned dataset.
Some other methods [12, 13, 45, 11] try to remove Trojans duringtraining with the help of benign datasets.
These training time approaches work for poisoning-basedattacks but fail to defend against supply chain attacks where the adversary injects the Trojan afterthe model is trained.
Another line of work, e.g., STRIP [14], SentiNet [15], and Februus [16] aim todetect Trojan inputs during runtime.
It is hard to distinguish between a misclassification and a Trojanattack for a test input.
These runtime detection methods make assumptions about the attack, whichstronger attacks can violate.
For example, STRIP fails to detect the Trojan inputs when the Trojantrigger locates around the center of an image or overlaps with the main object (e.g., feature-spaceattacks).
Another limitation is that they examine the test inputs and perform various heavyweighttests, significantly delaying the response time.
Trigger reverse engineering [17, 19, 18, 20, 21] makes no assumptions about the attack method (e.g.,poisoning or supply-chain attacks) and does not affect the runtime performance.
It inspects the modelto check if a Trojan exists before deploying.
Given a DNN model M and a small set of clean samples
X , trigger reverse engineering methods try to reconstruct injected triggers.
If reverse engineering issuccessful, the model is marked as malicious.
Neural Cleanse (NC) [17] proposes to perform reverseengineering by solving Eq.
[Equation]
, where x ∈ X and m is the trigger mask (i.e., a binary matrix with the same size as the input todetermine if the value will be replaced by the trigger or not), t is the trigger pattern (i.e., a matrix withthe same size as the input containing trigger values), and r ⋆ are attack constraints (e.g., trigger size issmaller than 1/4 of the image). L is the cross-entropy loss function.
Most prior works [19, 18, 20, 21]follow the same methodology and inherently suffer from the same limitations.
First, they assume thatan input space perturbation, denoted by ( m , t ) , can represent a trigger.
This assumption is valid forinput-space triggers but does not hold for feature-space attacks.
Second, r ⋆ are heuristics observedfrom existing attacks.
For example, NC observed that most triggers have small sizes and limit thetrigger size to be no larger than one-fourth of the input.
Otherwise, the trigger will overlap withthe main object and decrease benign accuracy.
In practice, more advanced attacks can break suchheuristics.
For instance, DFST leverages CycleGAN to transfer images from one style to anotherwithout changing its semantics.
It changes almost all pixels in a given image.
This paper proposes anovel reverse engineering method that overcomes the limitations above for image classifiers.
Methodology
3.1 Threat Model
This work aims to determine if a given model has a Trojan or not by reverse-engineering thecorresponding trigger.
Following existing works [17, 18, 43], we assume access to the model and asmall dataset containing correctly labeled benign samples of each label.
In practice, such datasetscan be gathered from the Internet.
We make no assumptions on how the attacker injects the Trojan(poisoning or supply-chain attack).
The attack can be formally defined as: M ( x ) =
y, M ( F ( x ))
=
y T , x ∈ X , where M is the Trojaned model, x is a clean input sample, and y T is the target label.
Fis the function to construct Trojan samples.
Input-space triggers add static input perturbations, andfeature-space triggers are input transformations.
The key difference between our work and existingwork is that we consider the feature space triggers.
3.2 Observation
In DNNs, the neuron activation values represent its functionality.
The input neurons denote theinput space features, and inner neurons extract inner and more abstract features.
Existing reverseengineering methods constrain the optimization problem in the input space using domain-specific constraints or observations.
For image classification tasks, the pixel value of each image has to be avalid RGB value.
Methods like NC observe that the trigger size must be smaller and cannot overlapwith the main object and propose corresponding constraints.
The most challenging problem forreverse-engineering feature-space triggers is how to constrain the optimization properly.
Note thatthere exist a set of neurons; when activating to specific values, the Trojan behavior will be triggered.
Due to the black-box nature of DNNs, it is hard to identify which neurons are related to the Trojanbehavior.
Moreover, if enlarge the weight values with the same scale, the output of the DNN will bethe same, and as such, it is hard to constrain concrete activation values.
Without a proper constraint,we cannot form an optimization problem.
Our key observation to solve this problem is that neuron activation values representing the Trojanbehavior are orthogonal to others .
Recall that one property of DNN Trojans is that when addingthe trigger to any given input, the model will predict the output to a specific label.
That is, thetrigger will always work regardless of the actual contents of the input.
In the feature space, whenthe model recognizes features of the Trojan, it will predict the label to the target label regardlessof the other features.
These activation values will form a hyperplane space in the high dimensionalspace so that they can be orthogonal to all others.
Based on this intuition, we performed empiricalexperiments to confirm our idea.
Specifically, we first use six Trojan attacks (e.g., BadNets [1], Cleanlabel attack [24], Filter attack [18], and WaNet [6], SIG [25] and Input-aware dynamic attack [7]) togenerate Trojaned ResNet18-architecture models on CIFAR-10.
We then visualize the feature spaceof the last convolutional layers in these models.
In Fig.
2, three dimensions, X, Y, and Z, representthe feature space.
We first apply PCA to get two eigenvectors of the benign training set; then, weuse the obtained eigenvectors as X-axis and Y-axis.
For Z-axis, we first construct Trojan inputs toactivate the model’s Trojan behavior and find highly related neurons to Trojans.
Then, we use DNNinterpretability techniques SHAP [46] to estimate the neuron’s importance to the Trojan behavior.
The neurons among the top 3% are compromised neurons .
Z-axis denotes the average activationvalues of compromised neurons.
Namely, u z = mean ( A ( F ( X train ))
⊙ m )
∥ mean ( A ( F ( X train )) ⊙ m ) ∥ , where m denotes a maskrevealing the position of compromised neurons.
Fig.
2 show that most Trojan inputs have a similarz-value.
They form a linear hyperplane in the feature space while benign ones do not.
[Figure]
3.3 Feature Space Trojan Hyperplane Reverse-engineering
In this paper, We use A to represent the submodel from the input space to the feature space.
B isthe submodel from the feature space to the output space.
We also use a = A ( x ) to denote the innerfeatures of the model.
Similar to the reverse-engineering in the input space, given a model M and asmall set of benign inputs X , we use a feature space mask m and a feature space pattern t to representthe feature space Trojan hyperplane H =
{ a | m ⊙ a = m ⊙ t } .
Specifically, we can update m andt via the following optimization process: min m , t L ( B ((1 − m ) ⊙ a + m ⊙ t ) , y t ) . y t is the target
Input: Model: M
Output: Trojaned or Not, Trojaned Pairs T 1: function R EVERSE - ENGINEERING ( M ) 2: for (target class y t , source class y s ) in K do 3: for e ≤ E do 4: x = sample
( X y s ) 5: cost 1 =
L ( B ((1 − m ) ⊙ a + m ⊙ t ) , y t ) 6:
if ∥ F ( x ) − x ∥ ≥ τ 1 then 7: cost 1 = cost 1 + w 1 · ∥ F ( x ) − x ∥ 8: if std ( m ⊙ A ( F ( x ))) ≥ τ 2 then 9: cost 1 = cost 1 + w 2 · std ( m ⊙ A ( F ( x )))
10: ∆ θ F = ∂cost 1 ∂θ F 11: θ F = θ F − lr 1 · ∆ θ F 12: cost 2 =
L ( B ((1 − m ) ⊙ a + m ⊙ t ) , y t ) 13: if ∥ m ∥ ≥ τ 3 then 14: cost 2 = cost 2 + w 3 · ∥ m ∥ 15: ∆ m =
∂cost 2 ∂ m 16: m = m − lr 2 · ∆ m 17: if ASR ( B ((1 − m ) ⊙ a + m ⊙ t ) , y t ) >
λ then 18:
M is a Trojaned model, 19:
T.append (( y s , y t )) label.
As discussed above, reverse-engineering the feature space is challenging.
In the input space,all values have natural physical semantics and constraints, e.g., a pixel value in the RGB value range.
Values in the feature space have uninterruptible meanings and are not strictly constrained.
Whetherthe result will have a physically meaningful semantic is also uncertain. We solve these challenges bysimultaneously optimizing the input space trigger function F and the feature space Trojan hyperplane H to enforce that the trigger has semantic meanings.
In detail, we compute the feature spacetrigger pattern as the mean inner features on the samples generated by the trigger function, i.e.,t
= mean ( m ⊙
A ( F ( X )) .
We also constrain the standard deviation of m ⊙ A ( F ( X ) to make surethe features generated by the trigger function will lie on the relaxation of the reverse-engineeredhyperplane.
Formally, our reverse-engineering can be written as the constrained optimization problemshown in Eq.
2, where X is the small set of clean samples.
We use deep neural networks to modelthe trigger function (i.e., F = G θ ) because of their expressiveness [47, 21]. Specifically, following DFST [3], we use a representative deep neural network UNet [48].
Given a model and a smallset of clean inputs, the trigger function can be smoothly reconstructed via gradient-based methods,i.e., optimizing the generative model G θ . In our default setting, A and B are separated at the lastconvolusional layer.
More discussions are in the Appendix (§ A.6).
[Equation]
where t = mean ( m ⊙ A ( F ( X )) ,
[Equation]
∈ A ( X
[Equation]
There are several constraints in the optimization problem: 1 ⃝ The transformed samples shouldbe similar to the original image due to the properties of Trojan attacks, i.e., ∥ F ( x ) − x ∥ ≤ τ 1 .
Typically, the Trojan samples are visually similar to original samples for stealthy purposes.
In detail,we use MSE (Mean Squared Error) to calculate the distance between F ( x ) and x . 2 ⃝ The Trojanfeatures should lie in the relaxation of the reverse-engineered feature space Trojan hyperplane, i.e., 200 P ( a ∈ H ⋆ | x ∈ F ( X )) should have high values.
To achieve this goal, we constrain the standarddeviation of different Trojan samples’ activation values on each pixel in the hyperplane.
3 ⃝ Similar toinput space trigger reverse-engineering [18], we set a bound for the size of the feature space triggermask, i.e., ∥ m ∥ ≤ τ 3 .
Here τ 1 , τ 2 , and τ 3 are threshold values.
We discuss their influence in § 4.4.
The detailed reverse-engineering algorithm can be found in Algorithm 1, where K is a set containingall possible (source class, target class) pairs of the model. F EATURE RE scans all labels to identify the Trojan target labels.
w 1 , w 2 and w 3 are the weight values used in the optimization.
Following
NC [17], we adjust them dynamically during optimization to make sure the reverse-engineered Trojansatisfies the constraints.
E is the maximal epoch number. In lines 5-11, it optimizes the triggerfunction F and then the mask m of the feature space hyperplane in lines 12-16.
In the end, wedetermine the reverse-engineering is successful and the label y t is a Trojan target label if the attacksuccess rate of the reversed Trojan is above a threshold value λ (80% in this paper).
3.4 Trojan Mitigation
After we reverse-engineered the Trojans, we can mitigate it by breaking the reverse-engineeredfeature space Trojan hyperplane.
Based on our observation, the neurons in the feature space Trojanhyperplane are highly related to the Trojan behaviors.
Thus, we can mitigate the Trojans by breakingthe hyperplane.
Inspired by Zhao et al.
[49], we can break it by flipping the neurons on it.
Ourneuron-flip process can be written as Eq. 3, where m is the reverse-engineered feature space mask, ais the inner features.
a i is the activation value on the i th neuron.
[Equation]
The mitigated model M ′ ( x ) = B ( Flip ( A ( x ))) , where A and B are submodels of the model.
Experiments and Results
We first introduce our experiment setup (§ 4.1).
We then evaluate the effectiveness of F EATURE RE on
Trojan detection (§ 4.2) and mitigation tasks (§ 4.3).
We also evaluate the robustness of F EATURE REagainst different settings and the impacts of configurable parameters in F EATURE RE (§ 4.4).
4.1 Experiment Setup.
We implement F EATURE RE with python 3.8 and PyTorch.
All experiments are conducted on a
Ubuntu 18.04 machine equipped with 64 CPUs and six GeForce RTX 6000 GPUs.
[Table]
Datasets and Models.
We use four publicly availabledatasets to evaluate F EATURE RE, including MNIST [50],
GTSRB [51], CIFAR-10 [52] and ImageNet [53].
Wesummarize our datasets in Table 1.
We show the datasetnames, the size of each input sample, the number of samples and the number of classes in each column.
Details of the datasets can be found in Appendix.
Formodel architectures, we use LeNet5 [50], Preact ResNet18(PRN18)
[54], ResNet18 [55], a VGG-style networkspecified in ULP [44], and a model consists of 4 convolutional layers and 2 dense layersused in Xu et al.
These datasets and models are widely used in Trojan-related researches [1, 2, 17, 18, 56, 14, 3, 43].
Evaluation Metrics.
We measure the effectiveness of the Trojan detection task by collecting thedetection accuracy (Acc).
Given a set of models consist of benign and Trojaned models, the Accis the number of correctly classified models over the number of all models.
We also show detailednumber of True Positives (TP, i.e., correctly detected Trojaned models), False Positives (FP, i.e.,benign models classified as Trojaned models), False Negatives (FN, i.e., Trojaned models classifiedas benign models) and True Negatives (TN, i.e., correctly classified benign models).
For the Trojanmitigation task, we evaluate the benign accuracy (BA) and attack success rate (ASR) [57].
BA is thenumber of correctly classified clean inputs over the number of all clean samples.
ASR is defined asthe number of Trojan samples that successfully attack models over the number of all Trojan samples.
Baselines and Attack Settings.
We evaluate the performance of F EATURE RE on Trojan detection,and compare the results with four reverse-engineering based Trojan detection methods (i.e., ABS [18],
DeepInspect [21], TABOR [20], and K-arm [19]) and two classification based methods (i.e., ULP [44]and Meta-classifier [43]).
For Trojan mitigation task, we compare F EATURE RE with two advancedmitigation methods (i.e., NAD [58] and I-BAU [59]).
We use the default parameter settings described
[Table] [Table]
[Table]
in the original papers of our baseline methods.
To understand the performance of F EATURE REand existing methods against various attack settings
[Table]
we evaluate them against BadNets [1], Filter
Trojans [18], WaNets [6], IA (Input-dependent dynamic Trojans)
[7], Clean-label [24], SIG [25]and ISSBA (Invisible sample-specific Trojans) [8] attacks.
These attacks are state-of-the-art attackmethods and are widely evaluated in Trojan defense papers [17, 18, 60, 59]
If not specified, we usean all-to-one (i.e., single-target) setting for all attacks.
Label-specific setting is discussed in § 4.4.
4.2 Effectiveness on Trojan Detection
To measure the effectiveness on the Trojan detection task, we generate a set of benign and Trojanedmodels, and then use F EATURE RE and existing Trojan detection methods to classify each model.
Wecollect the Acc, TP, FP, FN and TN results of each method and compare them.
Specifically, we firstevaluate the performance of F EATURE RE and compare the results with four state-of-the-art reverseengineering based detection methods.
We generate 20 Trojaned models as well as 20 benign modelson CIFAR-10 dataset for each attack (i.e., BadNets, Filter Trojan, WaNet and Input-aware dynamic
Trojan attack).
For MNIST and GTSRB dataset, we train 10 Trojaned and 10 benign LeNet
[50]models on each dataset.
We then compare F EATURE RE with two state-of-the-art classificationbased detection methods.
Similarly, we generate 10 benign and 10 Trojaned models, and use Trojandetection methods to classify these models.
Notice that, in all Trojan detection tasks, we assume thedefender can only access 10 clean samples for each class, which is a common practice.
The comparison results of reverse-engineering based methods are shown in Table 2.
The results oftwo classification based methods are demonstrated in Table 3 and Table 4.
In each table, we show thedetailed settings, including dataset names, network architectures, and attack settings.
Comparison to Reverse-engineering based methods.
From the results in Table 2, we observethat F EATURE RE achieves the best detection results compared with other methods.
The average
Acc of F EATURE RE is 93%, which is 17%, 23%, 35% and 23% higher than those of other defensemethods.
The results show the benefit of F EATURE RE.
When looking into the generalization of
Trojan detection methods, we find that F EATURE RE can achieve excellent results on both input-space
Trojans (i.e., BadNets) and feature-space Trojans (i.e., Filter, WaNet and IA attacks).
However, theperformance of existing reverse-engineering methods on feature-space Trojans (i.e., Filter, WaNetand IA attacks) is significantly worse than the performance on static Trojans.
F EATURE RE archives94% average Acc but the Acc of TABOR on feature-space Trojans are only 53%, 50% and 50%,respectively.
Moreover, F EATURE RE has 15.33 TP on average, but existing methods only have7.87 TP.
F EATURE RE can generalize better than existing work because F EATURE RE considers bothfeature and input space constraints.
Existing methods, on the contrary, only consider the input spaceconstraints.
They can not detect feature-space Trojans whose trigger is complex and input-dependent,and directly classify many Trojaned models with feature-space Trojans as benign.
Comparison to classification based methods.
When comparing F EATURE RE with classificationbased methods, we notice that F EATURE RE has better Acc, more TPs and TNs than classificationbased methods ULP and Meta-classifier.
As demonstrated in Table 3 and Table 4, the Acc of
[Table]
F EATURE RE is 0.93 and 0.95, which is 0.40 and 0.15 higher than those of ULP and Meta-classifier.
Overall, the results indicate that F EATURE RE is more effective than classification based methodswhen detecting Trojaned models.
Different from F EATURE RE, which directly inspects models viaanalyzing its inherent feature space properties, classification based methods highly depend on theexternal trained dataset.
Therefore, their results are not as precise as F EATURE RE.
4.3 Effectiveness on Trojan Mitigation
We evaluate the effectiveness of F EATURE RE on Trojan mitigation and compare the results withstate-of-the-art methods NAD and I-BAU.
We use the Trojaned models generated by three attacks (i.e.,
Filter attack, WaNet and IA) and report their average BA and ASR after Trojan mitigation.
We alsoshow the average BA and ASR of undefended Trojaned models.
For all methods, the defenders canaccess 10 clean samples for each class to conduct Trojan mitigation.
We show the results in Table 5.
We find that F EATURE RE is the most effective method for Trojan mitigation among all methods.
Compared to state-of-the-art Trojan mitigation methods, F EATURE RE archives the lowest average
ASR and the highest average BA.
On the one hand, using F EATURE RE can decrease the average
ASR from 96.76% to 0.26%.
Other methods can only decrease the average ASR to 40.14% and7.29%.
The results show the advantages of F EATURE RE on Trojan mitigation.
On the other hand, the
BA with F EATURE RE is similar to undefended models.
But the BA of other methods is significantlylower than that of undefended models.
By breaking the feature space hyperplane, F EATURE RE cansuccessfully mitigate Trojans with minimal BA loss.
Other methods, which cannot find Trojan-relatedfeatures, cannot achieve good results.
4.4 Ablation Study
In this section, we evaluate the resistance of F EATURE RE to various Trojan attack settings and largedatasets.
We also evaluate the impacts of configurable parameters in F EATURE RE, including theconstrain values used in Eq.
2 and the number of used clean samples.
By default, the attack used formeasuring the impacts of configurable parameters is IA.
We use 20 benign ResNet models and
Trojaned ResNet models on CIFAR-10 to test the detection results.
Notice that we only evaluate theperformance on the Trojan detection task.
Due to the page limits, we include the ablation study on
Trojan mitigation in Appendix.
[Table]
Resistance to various attack and dataset settings.
To evaluate if our method is resistant to more Trojan attacks, we train 20 Trojaned ResNet18 modelson CIFAR-10 for Label-specific attack (LS), Cleanlabel attack (CL) and SIG attack (SIG).
For the labelspecific attack, we consider the all-to-all attack setting, i.e., the target label y T =
η ( y )
=
y +1 , where ηis a mapping and y is the correct label of the sample.
In addition, we generate five benign models and five
Trojaned models with ISSBA attacks on ImageNet toevaluate if our method is compatible with large-scale datasets.
We summarize the results in Table 6.
In Table 6, we find that F EATURE RE is compatible with evaluated Trojan attacks, showing thegeneralization of our reverse-engineering based method.
We also observe that our method has high
Acc on the ImageNet dataset with ISSBA [8].
Thus, our method is also applicable to large datasets.
Influence of constrain values.
As shown in Eq.
2, there are three constrain values ( τ 1 , τ 2 , τ 3 ) in ourconstrained optimization process. By default, τ 1 = 0 . 15 , τ 2 = 0 . 25 and τ 3 = 5% .
We evaluate theirinfluences.
For τ 1 , we calculate input space perturbations on the preprocessed inputs, and the detailsof the preprocessing can be found in Appendix (§ A.2).
We vary τ 1 from 0.05 to 0.35, change τ 2from 0.10 to 0.50, and tune τ 3 from 3% of the whole feature space to 10% of the whole feature space.
The results under different hyperparameter settings are shown in Table 7.
From the results, we observe that the performance of F EATURE RE is insensitive to these threehyperparameters.
In detail, when we vary τ 1 , τ 2 and τ 3 , the Acc is stable.
In all cases, our methodalways achieves over 90% detection accuracy.
The results further show the robustness of F EATURE RE.
We also find that, when the value of all hyperparameters becomes lower, F EATURE RE has more FN.
On the contary, when its value is larger, more FP will be produced.
This is understandable becauselower constrain values mean a stricter criterion for a successful reverse-engineering.
[Table]
Number of clean reference samples.
Our threat model and existing work assume the defender canaccess a set of clean samples for defense.
To investigate the influences of the number of used cleansamples in Trojan detection, we choose the number from 1 to 100 in each class and report the Accresults.
The results are shown in Fig.
From the results, we notice that the Acc decreases significantly when we use less than 10 samplesfor each class.
This is because the number of used sample affects the optimization process.
Whenthe number of used samples is too small, the optimization process might be problematic, e.g., itencounters overfitting problem.
When the number of used samples is larger than 10, F EATURE REachieves high detection accuracy (i.e., above 95%) and the Acc will not change significantly whenthe number of used samples keeps increasing.
The reason is using more data makes the optimizingprocess converge and finally arrives a stable state.
Note that requiring hundreds clean samples iscommon for reverse-engineering based methods [17, 18, 20] and other types of defenses [58–60, 14].
F EATURE RE only requires 10 clean samples for each class, which is more efficient.
Discussion
Limitations of our method.
Similar to most existing Trojaned model detection and mitigationmethods [17–21, 58, 56], our method requires a small set of clean samples.
In the real world, thesesamples can be obtained from the Internet.
Ethics.
This paper proposes a technique to detect and remove Trojans in DNN models.
We believe itwill help improve the security of DNNs and be beneficial to society.
Conclusion
In this paper, we find relationships between feature space hyperplane and Trojans in DNNs.
Moreover, we propose a new Trojaned DNN detection and mitigation method based on our findings.
Compared to the state-of-the-art methods, our method has better performance in both detection andmitigation tasks.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See § 5.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] See § 5.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes] See § 5.
(a) Did you state the full set of assumptions of all theoretical results?
[N/A
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] See Abstract,
§ 4, and Appendix.
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes] See § 4 and Appendix.
(c) Did you report error bars (e g., with respect to the random seed after running experi-
ments multiple times)?
[No]
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See § 4.
If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] See § 4.
(b) Did you mention the license of the assets?
[Yes] See Appendix.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
See Abstract.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[Yes] See Appendix.
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[Yes] See Appendix.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable? [N/A] (c) Did you include the estimated hourly wage paid to A Appendix
Roadmap: More details of Algorithm 1 is introduced in § A.1.
Then, we present more details of thedatasets (§ A.2) and attacks (§ A.3) used in the experiments.
We also perform an ablation study forthe Trojan mitigation task in § A.4.
In § A.5, we visualize our reversed Trojan.
In § A.6, we discusshow to split the model.
The adaptive attack can be found in § A.7.
We also show the generalization(§ A.8) and the efficiency (§ A.9) of F EATURE RE.
Finally, we discuss the findings in the evaluation(§ A.10).
A.1 More details of Algorithm 1
In this section, we discuss more details of our Reverse-engineering Algorithm (Algorithm 1 in themain paper).
Given a model M and a small set of clean samples X , the output of the algorithm is aflag indicating if the model is Trojaned, and Trojaned label pairs denoting the source label and thetarget label of the detected Trojans. In line 2, we iterate (source label, target label) pair from possible pairs K . E in line 3 means themaximal optimization epoch number for each pair. It is set to 400 in this paper.
In line 4, we randomlysample a batch of inputs from the samples in source classes.
The batch size is set to 128 by default.
In lines 5 to Line 11, we optimize the parameters of the input space transformation F , which isrepresented by a UNet [48] model in our implementation.
In line 5, we calculate the loss valuespecified in Eq.
1 (in the main paper), where a = A ( x ) is the inner feature on clean samples.
Bydefault, A is the submodel from the input layer to the penultimate layer, and B is the submodel from thepenultimate layer to the output layer.
m is the feature space trigger mask. t = mean ( m ⊙ A ( F ( X ))is the feature space trigger pattern.
L is the cross-entropy loss calculating the distance between thetarget label and the output of the model under inner features with feature space Trojans.
In line 6, ifthe input space MSE (Mean Square Error) distance for original inputs x and the transformed inputs
F ( x ) is larger than a threshold value τ 1 (i.e., 0.15), then the regularization item w 1 · ∥ F ( x ) − x ∥will be added.
Note that we calculate input space distance on the preprocessed inputs, and the detailsof the preprocessing are in § A.2.
Following NC [17], the weight value w 1 is adjusted dynamically tomake the reverse-engineering satisfy the constrain (i.e., ∥ F ( x ) − x ∥ ≤ τ 1 ).
w 2 in line 9 and w 3 inline 14 are also adjusted dynamically.
In lines 8-9, similarly, we add the regularization item for thestandard deviation of different Trojan samples’ activation values on each pixel in the hyperplane.
Thedefault value for τ 2 is 0.25.
Lines 10-11 are the standard backward propagation process to update theparameters of the input space transformation function F based on the gradients.
The optimizer usedto optimize F is Adam [61].
The value of learning rate lr 1 is 1e-3.
In each epoch, we optimize boththe input space transformation function F and the feature space mask m .
Lines 12-16 describes the process for optimizing m .
Similar to line 5, we calculate the cross-entropyloss between the target label and the output of the model under inner features with feature space
Trojans in line 12.
In lines 13-14, we add the regularization item for the size of the feature space
Trojan hyperplane.
The default value for τ 3 is 5% of the whole feature space.
Lines 15-16 describethe process of updating feature space mask m via gradients.
The value of learning rate lr 2 in line 16is 1e-1.
The optimizer used is Adam [61].
In line 17, we check if the Trojan is successfully reverse-engineered.
In detail, we calculated the ASR(attack success rate) on inner features with feature space Trojans (i.e., (1 − m ) ⊙ a + m ⊙ t ).
Weflag that reverse-engineering is successful if the ASR is above a threshold value
λ (i.e., 0.8).
If the
Trojan is successfully reverse-engineered, we flag the model as a Trojan model and label the (sourceclass, target class) pair as Trojaned pair.
Besides the details above, we also use K-arm scheduler [19]to speed up the reverse engineering.
Lastly, we use Liu et al.
[62] to distinguish the Injected Trojansand UAPs (Universal Adversarial Patterns) [63].
A.2 Details of Datasets
In this section, details of the datasets used in the experiments are discussed.
We also provide thedetails of the preprocessing for each dataset.
All datasets are open-sourced.
The license for alldatasets is the MIT license.
They do not contain any personally identifiable information or offensivecontent.
[Table]
MNIST [50].
This dataset is used for classifying hand-written digits.
It contains 60000 trainingsamples in 10 classes.
The number of samples in the test set is 10000.
[Table]
GTSRB [51] This dataset is built for traffic sign classification tasks.
The number of classes is 43.
[Table]
The sample numbers for the training set and test set are 39209 and 12630, respectively.
[Table]
CIFAR10 [52]
This dataset is used for recognizing general objects, e.g., dogs, cats, and planes.
It has50000 training samples and 10000 training samples.
This dataset has 10 classes.
ImageNet [53] This dataset is also a general object classification benchmark.
Note that we use asubset (containing 200 classes) of the original ImageNet dataset specified in ISSBA [8].
The subsethas 100000 training samples and 10000 test samples.
Following standard convention on the image classification task, we scale the inputs to the range [0,1]and use mean-std normalization to preprocess the images.
In detail, the preprocessing can be writtenas x ′ =
( x 255 − Mean ) Std , where x ′ is the normalized input and x is the original inputs.
The Mean valueand Std (Standard Deviation) value for each channel on different datasets are summarized in Table 8.
A.3 Details of Attacks
In this section, we discuss the details of the used attacks.
By default, the attacks are in an all-to-one(i.e.
, single-target) setting, and the target label is randomly selected when we generate Trojanedmodels.
BadNets [1].
This attack uses a fixed pattern (i.e., a patch or a watermark) as Trojan triggers, and itgenerates Trojan inputs by simply pasting the pre-defined trigger pattern on the input.
It compromisedthe victim models by poisoning the training data (i.e., injecting Trojan samples and modifying theirlabels to target labels).
In our experiments, we use a 3*3 yellow patch located at the left-upper corneras Trojan trigger.
The poisoning rate we used is 5%.
The attack can be all-to-one (i.e., single-target)and all-to-all (i.e., label-specific).
For an all-to-one attack, all Trojan samples have the same targetlabel.
For label-specific attacks, the samples in different original classes have different target labels.
In our experiment, the target label for label-specific attack is y T =
η ( y )
=
y + 1 , where η is amapping and y is the correct label of the sample.
Filter Attack [18].
This attack exploits image filters as triggers and creates Trojan samples byapplying selected filters on images.
Similar to BadNets, the Trojans are injected with poisoning.
Following ABS [18], we use a 5% poisoning rate and apply the Nashville filter from Instagram as the
Trojan trigger.
WaNet [6].
This method achieves Trojan attacks via image warping techniques.
The trigger transformation of this attack is an elastic warping operation.
Different from BadNets and Filter Attack, inthis attack, the adversary needs to modify the training process of the victim models to make the attackmore resistant to Trojan defenses.
It is stealthy to human inspection, and it can also bypass manyexisting Trojan defense mechanisms [14, 12, 56, 17].
In our experiments, the wrapping strength andthe grid size are set to 0.5 and 4, respectively.
Input-aware Dynamic Attack [7].
This attack generates Trojan triggers via a trained generatornetwork.
The trigger generator is trained on a diversity loss so that two different input images do notshare the same trigger.
Similar to WaNet [6], the attacker needs to control the training process.
SIG [25].
This method uses superimposed sinusoidal signals as Trojan triggers.
In this attack, theattacker can only poison a set of training samples but can not control the full training process.
We setthe poisoning rate as 5%.
The frequency and the magnitude of the backdoor signal in our experimentsare 6 and 20, respectively.
[Table]
[Table]
Clean Label Attack [24].
This attack poisons the datasets without manipulating the label of poisoningsamples so that the attack is more stealthy.
The poisoning samples are generated by a trained GAN.
In our experiments, we set the poisoning rate as 5%.
ISSBA [8].
This attack utilizes an encoder-decoder network to generate sample-specific triggers.
The generated triggers are invisible noises.
The generated noises also contain the information of arepresentative string of the target label.
The threat model of this attack is that the attacker can onlypoison the training data, but can not control other components in training (e.g., the loss function).
Following the original paper, we poison 10% training data in our experiments.
A.4 Ablation Study on Trojan Mitigation
In this section, we study the performance of F EATURE RE under different constrain values anddifferent numbers of used clean samples.
The attack used in this section is WaNet [6].
Influence of constrain values.
To investigate the influence constrain values (i.e., τ 1 , τ 2 , and τ 3 ) onthe Trojan mitigation performance, We vary τ 1 from 0.05 to 0.35, change τ 2 from 0.10 to 0.50, andtune τ 3 from 1% of the whole feature space to 7% of the whole feature space.
We collect the BAand ASR of the mitigated models and report them in Table 9.
The results show that the mitigationperformance of F EATURE RE is not sensitive to τ 1 and τ 2 . For τ 3 , when the size of the Trojanhyperplane is extremely small (e.g., 1% of the feature space), the ASR is high. This is understandablebecause breaking an extremely small feature space Trojan hyperplane means flipping a very smallnumber of neurons, and it is not enough to completely remove the Trojans in the model. Therefore,we set the default value of the hyperplane’s size as 5% of the feature space.
Number of clean reference samples.
To understand the influence of clean set size on the Trojan
[Table]
mitigation task, we vary the number of used clean samples from 5 per class to 100 per class andreport the BA and ASR of mitigated model.
The results in Table 10 demonstrate that the performanceof F EATURE RE is robust when the number of used samples changes.
A.5 Visualization of Reverse-Engineered Trojans
To understand our method and study if it can reverse-engineer Trojans accurately, we visualize theinputs and inner features of clean samples, real Trojan samples, and reversed Trojan samples on ninerandomly selected samples in Fig.
The model is ResNet18 injected with Filter Trojan [18] and
Blend Trojan [22].
In the feature space, the reverse-engineered Trojan is close to the real Trojan,demonstrating the effectiveness of our reverse-engineering method.
[Table]
[Table]
Original
I npu t S p ace
F ea t u r e S p ace
I npu t S p ace
F ea t u r e S p ace
F ilt e r
B l e nd
Original
Real Trojan
Reversed Trojan
A.6 Discussion for Model Split
As we discussed in § 3, our method split the model M into two sub-models A and B .
In this section,we discuss the influence of using different split positions.
Table 11 shows the results of using different
A and B on the ResNet18 model and CIFAR-10 dataset.
In detail, we report the results of splitting themodel at the 9 th , 11 th , 13 th , 15 th , and the last convolutional layer.
The average detection accuracy forsplitting at the 9 th layer, 11 th layer, 13 th layer, 15 th layer, and last layer is 86.50%, 87.75%, 89.50%,94.00%, and 94.75%, respectively.
As we can see, the performance of splitting at later layers is higherthan the performance of splitting at earlier layers.
In our current implementation, we set A ( x ) as the sub-model from the input layer to the lastconvolution layer and B ( x ) as the rest.
The relationship between the input and the output of aconvolutional layer L n is x n +1 =
L n ( x n ) =
σ ( W T n x n + b T n ) , where x n and x n +1 are the inputsand outputs of layer n , W n and b n are weights and bias values, and σ is the activation function.
Based on existing literatures [64, 65], the features in the deeper CNN layers are more disentangledthan that of earlier layers.
Thus, if the orthogonal phenomenon happens in a layer L n , it will existfor all its subsequent layers, e.g., L n +1 .
If the orthogonal phenomenon does not happen, the layerwithout this phenomenon will mix benign and backdoor features, leading to low benign accuracy or attack success rate.
The results in Table 12 confirm our analysis.
Thus, a successful backdoor attackwill lead to the orthogonal phenomenon in the last convolution layer.
A.7 Adaptive Attack
Our threat model assumes that the attacker can control the training process of the Trojan model.
Inthis section, we discuss the potential adaptive attacker that knows our defense strategy and tries tobypass F EATURE RE via modifying the training process.
Our observation is that the neuron activationvalues representing the Trojan behavior are orthogonal to others.
One possible adaptive attack isbreaking such orthogonal relationships during the Trojan injection process.
We design an adaptiveattack that adds one loss term to push the Trojan features to be not orthogonal to benign features.
This attack can be formulated as:
L = L ce + L adv , where L ce is the standard classification loss andthe L adv is defined as:
[Equation]
Here, sim is the cosine similarity; a and a ′ are the features of different benign samples; m and t arethe feature-space mask and pattern of the compromised neurons obtained via SHAP [46].
The lossterm L adv tries to enforce the Trojan features being not orthogonal to the benign ones.
We conductthis adaptive attack on the CIFAR-10 dataset and ResNet18 model.
The results can be found in
Table 12.
The detection accuracy of F EATURE RE under adaptive attack drops to 65%.
{+Meanwhile,the average BA/ASR of the adaptive attack and BadNets (native training) is 87.36%/94.
+}
% and93.67%/99.98%, respectively.
The adaptive attack can reduce the detection accuracy of our method.
Both the BA and ASR of the adaptive attack are lower than those of native training.
The resultsconfirm our analysis in § A.6: the model without the “orthogonal phenomenon” will mix benign and
Trojan features, leading to low benign accuracy or attack success rate.
A.8 GeneralizationPerformance on mitigation task for more attacks
To measure the effectiveness of F EATURE RE on
[Table]
Trojan mitigation task, we use more Trojan attacks and report BA and ASR of our method.
Besidesthe results of BadNets [1], Filter [18], WaNet [6] and IA [7] in Table 5, in Table 13, we also showthe BA and ASR on LS [1], CL [24] and SIG [25].
The dataset and the model used is CIFAR-10and ResNet18, respectively.
For LS, CL, and SIG, the ASR of F EATURE RE is 1.15%, 2.62%, and1.22%, which are 80.01, 33.18, and 81.22 times lower than that of undefended models.
As can beobserved, F EATURE RE can effectively reduce the ASR while keeping the BA nearly unchanged.
[Table]
Thus, F EATURE RE is robust to different attacks on mitigation task.
[Table]
[Table]
[Table]
Generalization to different models.
To understand the generalization of F EATURE RE to different
[Table]
model architectures, we evaluate its detection accuracy on BadNets [1], Filter [18], WaNet [6], IA [7],
LS [1], CL [24], and SIG [25] attacks using VGG16 [66], ResNet18 [55], and Preact-ResNet18(PRN18) [54].
The results are summarized in Table 14.
In Table 15, we also report F EATURE RE’sperformance on a larger model (i.e., Wide-ResNet34 [67]).
In all settings, the detection accuracyis above 80%, and the average detection accuracy on VGG16, ResNet18, and PRN18 is 89.26%,91.43%, and 90.71%, respectively.
F EATURE RE achieves high detection accuracy on all differentmodels, demonstrating it is generalizable to different model architectures and larger models.
Generalization to large input size.
To see if F EATURE RE can generalize to large datasets, we reportits accuracy on the ImageNette 1 dataset under different attacks.
The input size of ImageNette is 3 ×224 × 224.
The model architecture used here is Wide-ResNet34 [67].
{+For each attack, we have 
+}
Trojaned models.
We also train 5 benign models.
The results are in Table 15.
For all different attacks,the detection accuracy of F EATURE RE is above 80%.
The average detection accuracy on a largeinput size is 91.43%.
Thus, our method can generalize to large input sizes.
[Table]
A.9 Efficiency
In this section, we measure the efficiency of F EATURE RE.
Like existing reverse-engineering methods [17, 20, 21], F EATURE RE scans all labels.
We optimize this process with a K-arm scheduler [19],which uses the Multi-Arm Bandit to iteratively and stochastically select the most promising labelsfor optimization.
We measure the average runtime on the CIFAR-10 and ImageNet datasets.
Themodel used is ResNet18.
The running time on CIFAR-10 and ImageNet are 530.8s and 8934.5s,respectively.
A.10 Discussions
One finding we have is that using later layers to conduct the reverse-engineering is relatively betterthan using earlier layers (more results and details can be found in § A.6).
We also found that
F EATURE RE’s performance under the clean-label attack is relatively worse than that of other attacks.
We suspect this is because the benign and Trojan features of the clean-label attack are highly mixed.
As a consequence, the clean label attack has lower ASR than other attacks.
For example, the ASR ofthe clean-label attack and BadNets are 86.94% and 100.00%, respectively.