Rethinking the Reverse-engineering of Trojan Triggers
Abstract
Deep Neural Networks are vulnerable to Trojan (or backdoor) attacks.
Reverseengineering methods can reconstruct the trigger and thus identify affected models.
Existing reverse-engineering methods only consider input space constraints, e.g., trigger size in the input space.
Expressly, they assume the triggers are static patterns in the input space and fail to detect models with feature-space triggers such as image style transformations.
We observe that both input-space and feature-space Trojans are associated with feature space hyperplanes.
Based on this observation, we design a novel reverse-engineering method that exploits the feature space constrain to reverse-engineer Trojan triggers.
Results on four datasets and seven different attacks demonstrate that our solution effectively defends both input-space and feature-space Trojans.
It outperforms state-of-the-art reverse-engineering methods and other types of defenses in both Trojaned model detection and mitigation tasks.
On average, the detection accuracy of our method is 93%.
For Trojan mitigation, our method can reduce the ASR (attack success rate) to only 0.26% with the
BA (benign accuracy) remaining nearly unchanged.
Our code can be found at https://github.com/RU-System-Software-and-Security/FeatureRE .
Introduction
DNNs are vulnerable to Trojan attacks [1–6].
After injecting a Trojan into the DNN model, the adversary can manipulate the model prediction by adding a Trojan trigger to get the target label.
The adversary can inject the Trojan by performing the poisoning attack or supply chain attack.
In the poisoning attack, the adversary can control the training dataset and injects the Trojan by adding samples with the Trojan trigger labeled as the target label.
In the supply chain attack, the adversary can replace a benign model with a Trojaned model by performing the supply chain attack.
The Trojan trigger is becoming more and more stealthy.
Earlier works use static patterns, e.g., a yellow pad as the trigger, which is known as the input space triggers.
Researchers recently proposed using more dynamic and input-aware techniques to generate stealthy triggers that mix with benign features, which are referred to as the feature space triggers.
For example, the trigger of the feature-space Trojans can be a warping process [7] or a generative model [3, 8, 9].
The Trojan attack is a prominent threat to the trustworthiness of DNN models, especially in security-critical applications, such as autonomous driving [1], malware classification [10], and face recognition [11].
Prior works have proposed several ways to defend against Trojan attacks, such as removing poisons in training [12–14], detecting Trojan samples at runtime [15–18], etc.
Many of above methods can only work for one type of Trojan attack.
For example, training and pre-training time defense (e.g., removing poisoning data, training a benign model with poisoning data) fail to defend against the supply chain attack.
Trigger reverse-engineering [19–23] is a general method to defend against different Trojan attacks under different threat models.
It works by searching for if there exists an input pattern that can be used as a trigger in the given model.
If we can find such a trigger, the model has a corresponding Trojan and is marked as malicious and vice versa.
Existing reverseengineering methods assume that the Trojan triggers are static patterns in the input space and develop an optimization problem that looks for an input pattern that can be used as the trigger.
This assumption is valid for input-space attacks [1, 2, 24] that use static triggers (e.g., a colored patch).
Feature space attacks [3–5, 7–9, 25] break this assumption.
Existing trigger reverse-engineering methods [19–23] constrain the optimization by using heuristics or empirical observations on existing attacks, such as pixel values are in range [0 , 255] , and the trigger’s size is small.
Such heuristics are also invalid for feature-space triggers that change all pixels in images.
Reverse-engineering the feature space is challenging.
Unlike input space, there are no constraints that can be directly used.
[Figure]
In this paper, we propose a trigger reverse-engineering method that works for feature space triggers.
Our intuition is that features representing the Trojan are orthogonal to other features.
Because a trigger works for a set of samples (or all of them, depending on the attack type), changing the input content without removing the Trojan features will not change the prediction.
That is, changing Trojan and benign features will not affect each other.
Trojan features will form a hyperplane in the high dimensional space, which can constrain the search in feature space.
We then develop our reverse-engineering method by exploiting the feature space input-space constraint.
Fig.
1 demonstrates our idea.
Existing reverse-engineering methods only consider the input-space constraint.
It conducts reverse-engineering via searching a static trigger pattern in the input space.
These methods fail to reverse-engineer feature-space Trojans whose trigger is dynamic in the input space.
Instead, our idea is to exploit the feature space constraint and searching a feature space trigger using the constraint that the Trojan features will form a hyperplane.
At the same time, we also reverse-engineer the input space Trojan transformation based on the feature space constraint.
To the best of our knowledge, we are the first to propose feature-space reverse-engineering methods for backdoor detection.
Through reversed Trojan transformation, we developed a Trojan removal method.
We implemented a prototype F EATURE RE ( FEATURE -space RE verse-engineering) in Python and PyTorch and evaluated it on MNIST, GTSRB, CIFAR, and ImageNet dataset with seven different attacks (i.e.,
BadNets [1], Filter attack [20], WaNet [7], Input-aware dynamic attack [8], ISSBA [9], Clean-label attack [26], Label-specific attack [1], and SIG attack [27]).
Our results show that F EATURE RE is effective.
On average, the accuracy of our method is 93%, outperforming existing techniques.
For
Trojan mitigation, our method can reduce the ASR (attack success rate) to only 0.26% with the BA (benign accuracy) remaining nearly unchanged by using only ten clean samples for each class.
Our contributions are summarized as follows.
We first find the feature space properties of the Trojaned model and reveal the relationship between Trojans and the feature space hyperplanes.
We propose a novel Trojan trigger reverse-engineering technique leveraging the feature space Trojan hyperplane.
We evaluate our prototype on four different datasets, five different network architectures, and seven advanced Trojan attacks.
Results show that our method outperforms SOTA approaches.
Background & Motivation
A DNN classifier is a function M : X (cid:55)→ Y where X is the input domain R m
and Y is a set of labels
K .
A Trojan (or backdoor) attack against a DNN model M is a malicious way of perturbing the input so that an adversarial input x ′ (i.e., input with the perturbation pattern) will be classified to a target/random label while the model maintains high accuracy for benign input x .
The perturbation pattern is known as the Trojan trigger.
Trojan attacks can happen in training (e.g., data poisoning) or model distribution (e.g., changing model weights or supply-chain attack).
Existing works have shown Trojan attacks against different DNN models, including computer vision models [1, 2, 24],
Graph Neural Networks (GNNs) [28, 29], Reinforcement Learning (RL) [30, 31], Natural Language
Processing (NLP) [32–37], recommendation systems [38], malware detection [10], pretrained models [21, 39, 40], active learning [41], and federated learning [42, 43].
The Trojan trigger can be a simple input pattern (e.g., a yellow pad)
[1, 2, 24] or a complex input transformation function (e.g., a
CycleGAN to change the input styles) [3, 5, 7–9].
If the trigger is static input space perturbations (e.g., a yellow pad), the Trojan attack is known as input-space Trojan , and if the trigger is an input feature (e.g., an image style), the attack is referred to as the feature-space Trojan .
There are different types of Trojan defenses.
A line of work [13, 14, 44] attempts to remove poisoned data samples by cleaning the training dataset.
Training-based methods [45–47] train benign classifiers even with the poisoned dataset.
These training time approaches work for poisoning-based attacks but fail to defend against supply chain attacks where the adversary injects the Trojan after the model is trained.
Another line of work, e.g., STRIP [15], SentiNet [16], and Februus [17] aim to detect Trojan inputs during runtime.
It is hard to distinguish between a misclassification and a Trojan attack for a test input.
These runtime detection methods make assumptions about the attack, which stronger attacks can violate.
For example, STRIP fails to detect the Trojan inputs when the Trojan trigger locates around the center of an image or overlaps with the main object (e.g., feature-space attacks).
Another limitation is that they examine the test inputs and perform various heavyweight tests, significantly delaying the response time.
Trigger reverse engineering [19–23, 48–50] makes no assumptions about the attack method (e.g., poisoning or supply-chain attacks) and does not affect the runtime performance.
It inspects the model to check if a Trojan exists before deploying.
Given a DNN model M and a small set of clean samples
X , trigger reverse engineering methods try to reconstruct injected triggers.
If reverse engineering is successful, the model is marked as malicious.
Neural Cleanse (NC) [19] proposes to perform reverse engineering by solving Eq.
[Equation]
m , [Equation] , where x ∈ X and m is the trigger mask (i.e., a binary matrix with the same size as the input to determine if the value will be replaced by the trigger or not), t is the trigger pattern (i.e., a matrix with the same size as the input containing trigger values), and r ⋆ are attack constraints (e.g., trigger size is smaller than 1/4 of the image).
L is the cross-entropy loss function.
Most prior works [20–23] follow the same methodology and inherently suffer from the same limitations.
First, they assume that an input space perturbation, denoted by ( m , t ) , can represent a trigger.
This assumption is valid for input-space triggers but does not hold for feature-space attacks.
Second, r ⋆ are heuristics observed from existing attacks.
For example, NC observed that most triggers have small sizes and limit the trigger size to be no larger than one-fourth of the input.
Otherwise, the trigger will overlap with the main object and decrease benign accuracy.
In practice, more advanced attacks can break such heuristics.
For instance, DFST [3] leverages CycleGAN to transfer images from one style to another without changing its semantics.
It changes almost all pixels in a given image.
This paper proposes a novel reverse engineering method that overcomes the limitations above for image classifiers.
Methodology 3.1 Threat Model
This work aims to determine if a given model has a Trojan or not by reverse-engineering the corresponding trigger.
Following existing works [19, 20, 51], we assume access to the model and a small dataset containing correctly labeled benign samples of each label.
In practice, such datasets can be gathered from the Internet.
We make no assumptions on how the attacker injects the Trojan (poisoning or supply-chain attack).
The attack can be formally defined as: M ( x ) =
y, M ( F ( x ))
=
y T , x ∈ X , where M is the Trojaned model, x is a clean input sample, and y T is the target label.
F is the function to construct Trojan samples.
Input-space triggers add static input perturbations, and feature-space triggers are input transformations.
The key difference between our work and existing work is that we consider the feature space triggers.
3.2 Observation
In DNNs, the neuron activation values represent its functionality.
The input neurons denote the input space features, and inner neurons extract inner and more abstract features.
Existing reverseengineering methods constrain the optimization problem in the input space using domain-specific constraints or observations.
For image classification tasks, the pixel value of each image has to be a valid RGB value.
Methods like NC observe that the trigger size must be smaller and cannot overlap with the main object and propose corresponding constraints.
The most challenging problem for reverse-engineering feature-space triggers is how to constrain the optimization properly.
Note that there exist a set of neurons; when activating to specific values, the Trojan behavior will be triggered.
Due to the black-box nature of DNNs, it is hard to identify which neurons are related to the Trojan behavior.
Moreover, if enlarge the weight values with the same scale, the output of the DNN will be the same, and as such, it is hard to constrain concrete activation values.
Without a proper constraint, we cannot form an optimization problem.
Our key observation to solve this problem is that neuron activation values representing the Trojan behavior are orthogonal to others .
Recall that one property of DNN Trojans is that when adding the trigger to any given input, the model will predict the output to a specific label.
That is, the trigger will always work regardless of the actual contents of the input.
In the feature space, when the model recognizes features of the Trojan, it will predict the label to the target label regardless of the other features.
These activation values will form a hyperplane space in the high dimensional space so that they can be orthogonal to all others.
Based on this intuition, we performed empirical experiments to confirm our idea.
Specifically, we first use six Trojan attacks (e.g., BadNets [1], Clean label attack [26], Filter attack [20], and WaNet [7], SIG [27] and Input-aware dynamic attack [8]) to generate Trojaned ResNet18-architecture models on CIFAR-10.
We then visualize the feature space of the last convolutional layers in these models.
In Fig.
2, three dimensions, X, Y, and Z, represent the feature space.
We first apply PCA to get two eigenvectors of the benign training set; then, we use the obtained eigenvectors as X-axis and Y-axis.
For Z-axis, we first construct Trojan inputs to activate the model’s Trojan behavior and find highly related neurons to Trojans.
Then, we use DNN interpretability techniques SHAP [52] to estimate the neuron’s importance to the Trojan behavior.
The neurons among the top 3% are compromised neurons .
Z-axis denotes the average activation values of compromised neurons.
Namely, u z = mean ( A ( F ( X train ))
⊙ m )
∥ mean ( A ( F ( X train )) ⊙ m ) ∥ , where m denotes a mask revealing the position of compromised neurons.
Fig.
2 show that most Trojan inputs have a similar z-value.
They form a linear hyperplane in the feature space while benign ones do not.
[Figure]
3.3 Feature Space Trojan Hyperplane Reverse-engineering
In this paper, We use A to represent the submodel from the input space to the feature space.
B is the submodel from the feature space to the output space.
We also use a = A ( x ) to denote the inner
Input: Model: M
Output: Trojaned or Not, Trojaned Pairs T 1: function R EVERSE - ENGINEERING ( M ) 2: for (target class y t , source class y s ) in K do 3: for e ≤ E do 4: x = sample
( X y s ) 5: cost 1 =
L ( B ((1 − m ) ⊙ a + m ⊙ t ) , y t ) 6:
if ∥ F ( x ) − x ∥ ≥ τ 1 then 7: cost 1 = cost 1 + w 1 · ∥ F ( x ) − x ∥ 8: if std ( m ⊙ A ( F ( x ))) ≥ τ 2 then 9: cost 1 = cost 1 + w 2 · std ( m ⊙ A ( F ( x )))
10: ∆ θ F = ∂cost 1 ∂θ F 11: θ F = θ F − lr 1 · ∆ θ F 12: cost 2 =
L ( B ((1 − m ) ⊙ a + m ⊙ t ) , y t ) 13: if ∥ m ∥ ≥ τ 3 then 14: cost 2 = cost 2 + w 3 · ∥ m ∥ 15: ∆ m =
∂cost 2 ∂ m 16: m = m − lr 2 · ∆ m 17: if ASR ( B ((1 − m ) ⊙ a + m ⊙ t ) , y t ) >
λ then 18:
M is a Trojaned model, 19:
T.append (( y s , y t ))
features of the model.
Similar to the reverse-engineering in the input space, given a model M and a small set of benign inputs X , we use a feature space mask m and a feature space pattern t to represent the feature space Trojan hyperplane H
= { a | m ⊙
a = m ⊙ t } .
Specifically, we can update m and t via the following optimization process: min m , t L ( B ((1 − m ) ⊙ a + m ⊙ t ) , y t ) .
y t is the target label. As discussed above, reverse-engineering the feature space is challenging.
In the input space, all values have natural physical semantics and constraints, e.g., a pixel value in the RGB value range.
Values in the feature space have uninterruptible meanings and are not strictly constrained.
Whether the result will have a physically meaningful semantic is also uncertain.
We solve these challenges by simultaneously optimizing the input space trigger function F and the feature space Trojan hyperplane
H to enforce that the trigger has semantic meanings.
In detail, we compute the feature space trigger pattern as the mean inner features on the samples generated by the trigger function, i.e., t = mean ( m ⊙ A ( F ( X )) .
We also constrain the standard deviation of m ⊙ A ( F ( X ) to make sure the features generated by the trigger function will lie on the relaxation of the reverse-engineered hyperplane.
Formally, our reverse-engineering can be written as the constrained optimization problem shown in Eq.
2, where X is the small set of clean samples.
We use deep neural networks to model the trigger function (i.e., F = G θ ) because of their expressiveness [23, 53].
Specifically, following
DFST [3], we use a representative deep neural network UNet [54].
Given a model and a small set of clean inputs, the trigger function can be smoothly reconstructed via gradient-based methods, i.e., optimizing the generative model G θ .
In our default setting, A and B are separated at the last convolutional layer.
More discussions are in the Appendix (§ 4.5).
[Equation]
where t = mean ( m ⊙ A ( F ( X [Equation] , a
[Equation]
A ( X
[Equation]
There are several constraints in the optimization problem: 1 ⃝ The transformed samples should be similar to the original image due to the properties of Trojan attacks, i.e., ∥ F ( x ) − x ∥ ≤ τ 1 .
Typically, the Trojan samples are visually similar to original samples for stealthy purposes.
In detail, we use MSE (Mean Squared Error) to calculate the distance between F ( x ) and x . 2 ⃝ The Trojan features should lie in the relaxation of the reverse-engineered feature space Trojan hyperplane, i.e., P ( a ∈ H ⋆ | x ∈ F ( X )) should have high values.
To achieve this goal, we constrain the standard deviation of different Trojan samples’ activation values on each pixel in the hyperplane.
3 ⃝ Similar to input space trigger reverse-engineering [20], we set a bound for the size of the feature space trigger mask, i.e., ∥ m ∥ ≤ τ 3 .
Here τ 1 , τ 2 , and τ 3 are threshold values.
We discuss their influence in § 4.4.
The detailed reverse-engineering algorithm can be found in Algorithm 1, where K is a set containing all possible (source class, target class) pairs of the model.
F EATURE RE scans all labels to identify the Trojan target labels.
w 1 , w 2 and w 3 are the weight values used in the optimization.
Following
NC [19], we adjust them dynamically during optimization to make sure the reverse-engineered Trojan satisfies the constraints.
E is the maximal epoch number.
In lines 5-11, it optimizes the trigger function F and then the mask m of the feature space hyperplane in lines 12-16.
In the end, we determine the reverse-engineering is successful and the label y t is a Trojan target label if the attack success rate of the reversed Trojan is above a threshold value λ (80% in this paper).
3.4 Trojan Mitigation
After we reverse-engineered the Trojans, we can mitigate it by breaking the reverse-engineered feature space Trojan hyperplane.
Based on our observation, the neurons in the feature space Trojan hyperplane are highly related to the Trojan behaviors.
Thus, we can mitigate the Trojans by breaking the hyperplane.
Inspired by Zhao et al.
[55], we can break it by flipping the neurons on it.
Our neuron-flip process can be written as Eq. 3, where m is the reverse-engineered feature space mask, a is the inner features.
a i is the activation value on the i th neuron.
[Equation]
The mitigated model M ′ ( x ) = B ( Flip ( A ( x ))) , where A and B are submodels of the model.
Experiments and Results
We first introduce our experiment setup (§ 4.1).
We then evaluate the effectiveness of F EATURE RE on
Trojan detection (§ 4.2) and mitigation tasks (§ 4.3).
We also evaluate the robustness of F EATURE RE against different settings and the impacts of configurable parameters in F EATURE RE (§ 4.4).
In § 4.5, we discuss how to split the model.
The adaptive attack can be found in § 4.6.
4.1 Experiment Setup.
We implement F EATURE RE with python 3.8 and PyTorch.
All experiments are conducted on a
Ubuntu 18.04 machine equipped with 64 CPUs and six GeForce RTX 6000 GPUs.
[Table]
Datasets and Models.
We use four publicly available datasets to evaluate F EATURE RE, including MNIST [56],
GTSRB [57], CIFAR-10 [58] and ImageNet [59].
We summarize our datasets in Table 1.
We show the dataset names, the size of each input sample, the number of samples and the number of classes in each column.
Details of the datasets can be found in Appendix.
For model architectures, we use LeNet5 [56], Preact ResNet (PRN18) [60], ResNet18 [61], a VGG-style network specified in ULP [62], and a model consists of 4 convolutional layers and 2 dense layers used in Xu et al.
These datasets and models are widely used in Trojan-related researches [1– 3, 15, 19, 20, 51, 63].
Evaluation Metrics.
We measure the effectiveness of the Trojan detection task by collecting the detection accuracy (Acc).
Given a set of models consist of benign and Trojaned models, the Acc is the number of correctly classified models over the number of all models.
We also show detailed number of True Positives (TP, i.e., correctly detected Trojaned models), False Positives (FP, i.e., benign models classified as Trojaned models), False Negatives (FN, i.e., Trojaned models classified as benign models) and True Negatives (TN, i.e., correctly classified benign models).
For the Trojan mitigation task, we evaluate the benign accuracy (BA) and attack success rate (ASR) [64].
BA is the number of correctly classified clean inputs over the number of all clean samples.
ASR is defined as the number of Trojan samples that successfully attack models over the number of all Trojan samples. [Table] [Table] [Table]
Baselines and Attack Settings.
We evaluate the performance of F EATURE RE on Trojan detection, and compare the results with four reverse-engineering based Trojan detection methods (i.e., ABS [20],
DeepInspect [23], TABOR [22] [Table] and K-arm [21]) and two classification based methods (i.e., ULP [62] and Meta-classifier [51]).
For Trojan mitigation task, we compare F EATURE RE with two advanced mitigation methods (i.e., NAD [65] and I-BAU [66]).
We use the default parameter settings described in the original papers of our baseline methods.
To understand the performance of F EATURE RE and existing methods against various attack settings, we evaluate them against BadNets [1], Filter
Trojans [20], WaNets [7], IA (Input-dependent dynamic Trojans) [8], Clean-label [26], SIG [27] and ISSBA (Invisible sample-specific Trojans) [9] attacks.
These attacks are state-of-the-art attack methods and are widely evaluated in Trojan defense papers [19, 20, 66, 45] If not specified, we use an all-to-one (i.e., single-target) setting for all attacks.
Label-specific setting is discussed in § 4.4.
4.2 Effectiveness on Trojan Detection
To measure the effectiveness on the Trojan detection task, we generate a set of benign and Trojaned models, and then use F EATURE RE and existing Trojan detection methods to classify each model.
We collect the Acc, TP, FP, FN and TN results of each method and compare them.
Specifically, we first evaluate the performance of F EATURE RE and compare the results with four state-of-the-art reverseengineering based detection methods.
We generate 20 Trojaned models as well as 20 benign models on CIFAR-10 dataset for each attack (i.e., BadNets, Filter Trojan, WaNet and Input-aware dynamic
Trojan attack).
For MNIST and GTSRB dataset, we train 10 Trojaned and 10 benign [-LeNet
 [50]models-]LeNet5 [56] models on each dataset.
We then compare F EATURE RE with two state-of-the-art classification based detection methods.
Similarly, we generate 10 benign and 10 Trojaned models, and use Trojan detection methods to classify these models.
Notice that, in all Trojan detection tasks, we assume the defender can only access 10 clean samples for each class, which is a common practice.
The comparison results of reverse-engineering based methods are shown in Table 2.
The results of two classification based methods are demonstrated in Table 3 and Table 4.
In each table, we show the detailed settings, including dataset names, network architectures, and attack settings.
Comparison to Reverse-engineering based methods.
From the results in Table 2, we observe that F EATURE RE achieves the best detection results compared with other methods.
The average
Acc of F EATURE RE is 93%, which is 17%, 23%, 35% and 23% higher than those of other defense methods.
The results show the benefit of F EATURE RE.
When looking into the generalization of
Trojan detection methods, we find that F EATURE RE can achieve excellent results on both input-space
Trojans (i.e., BadNets) and feature-space Trojans (i.e., Filter, WaNet and IA attacks).
However, the performance of existing reverse-engineering methods on feature-space Trojans (i.e., Filter, WaNet and IA attacks) is significantly worse than the performance on static Trojans.
F EATURE RE archives 94% average Acc but the Acc of TABOR on feature-space Trojans are only 53%, 50% and 50%, respectively.
Moreover, F EATURE RE has 15.33 TP on average, but existing methods only have 7.87 TP.
F EATURE RE can generalize better than existing work because F EATURE RE considers both feature and input space constraints.
Existing methods, on the contrary, only consider the input space [Table] constraints.
They can not detect feature-space Trojans whose trigger is complex and input-dependent, and directly classify many Trojaned models with feature-space Trojans as benign.
Comparison to classification based methods.
When comparing F EATURE RE with classification based methods, we notice that F EATURE RE has better Acc, more TPs and TNs than classification based methods ULP and Meta-classifier.
As demonstrated in Table 3 and Table
[Table]
the Acc of F EATURE RE is 0.93 and 0.95, which is 0.40 and 0.15 higher than those of ULP and Meta-classifier.
Overall [Table] the results indicate that F EATURE RE is more effective than classification based methods when detecting Trojaned models.
Different from F EATURE RE, which directly inspects models via analyzing its inherent feature space properties, classification based methods highly depend on the external trained dataset.
Therefore, their results are not as precise as F EATURE RE.
4.3 Effectiveness on Trojan Mitigation
We evaluate the effectiveness of F EATURE RE on Trojan mitigation and compare the results with state-of-the-art methods NAD and I-BAU.
We use the Trojaned models generated by three attacks (i.e.,
Filter attack, WaNet and IA) and report their average BA and ASR after Trojan mitigation.
We also show the average BA and ASR of undefended Trojaned models.
For all methods, the defenders can access 10 clean samples for each class to conduct Trojan mitigation.
We show the results in Table 5.
We find that F EATURE RE is the most effective method for Trojan mitigation among all methods.
Compared to state-of-the-art Trojan mitigation methods, F EATURE RE archives the lowest average
ASR and the highest average BA.
On the one hand, using F EATURE RE can decrease the average
ASR from 96.76% to 0.26%.
Other methods can only decrease the average ASR to 40.14% and 7.29%.
The results show the advantages of F EATURE RE on Trojan mitigation.
On the other hand, the
BA with F EATURE RE is similar to undefended models.
But the BA of other methods is significantly lower than that of undefended models.
By breaking the feature space hyperplane, F EATURE RE can successfully mitigate Trojans with minimal BA loss.
Other methods, which cannot find Trojan-related features, cannot achieve good results.
4.4 Ablation Study
In this section, we evaluate the resistance of F EATURE RE to various Trojan attack settings and large datasets.
We also evaluate the impacts of configurable parameters in F EATURE RE, including the constrain values used in Eq.
2 and the number of used clean samples.
By default, the attack used for measuring the impacts of configurable parameters is IA.
We use 20 benign ResNet models and[-
-]
Trojaned ResNet models on CIFAR-10 to test the detection results.
Notice that we only evaluate the performance on the Trojan detection task.
Due to the page limits, we include the ablation study on
Trojan mitigation in Appendix.
[Table]
Resistance to various attack and dataset settings.
To evaluate if our method is resistant to more Trojan attacks, we train 20 Trojaned ResNet18 models on CIFAR-10 for Label-specific attack (LS), Cleanlabel attack (CL) and SIG attack (SIG).
For the labelspecific attack, we consider the all-to-all attack setting, i.e., the target label y T =
η ( y )
=
y +1 , where η is a mapping and y is the correct label of the sample.
In addition, we generate five benign models and five
Trojaned models with ISSBA attacks on ImageNet to evaluate if our method is compatible with large-scale datasets.
We summarize the results in Table 6.
In Table 6, we find that F EATURE RE is compatible with evaluated Trojan attacks, showing the generalization of our reverse-engineering based method.
We also observe that our method has high
Acc on the ImageNet dataset with ISSBA [9].
Thus, our method is also applicable to large datasets.
Influence of constrain values.
As shown in Eq.
2, there are three constrain values ( τ 1 , τ 2 , τ 3 ) in our constrained optimization process.
By default, τ 1 = 0 .
15 , τ 2 = 0 .
25 and τ 3 = 5% .
We evaluate their influences.
For τ 1 , we calculate input space perturbations on the preprocessed inputs, and the details of the preprocessing can be found in Appendix (§ A.2).
We vary τ 1 from 0.05 to 0.35, change τfrom 0.10 to 0.50, and tune τ 3 from 3% of the whole feature space to 10% of the whole feature space.
The results under different hyperparameter settings are shown in Table 7.
From the results, we observe that the performance of F EATURE RE is insensitive to these three hyperparameters.
In detail, when we vary τ 1 , τ 2 and τ 3 , the Acc is stable.
In all cases, our method always achieves over 90% detection accuracy.
The results further show the robustness of F EATURE RE.
We also find that, when the value of all hyperparameters becomes lower, F EATURE RE has more FN.
On the contary, when its value is larger, more FP will be produced.
This is understandable because lower constrain values mean a stricter criterion for a successful reverse-engineering.
[Table]
[Figure]
Number of clean reference samples.
Our threat model and existing work assume the defender can access a set of clean samples for defense.
To investigate the influences of the number of used clean samples in Trojan detection, we choose the number from 1 to 100 in each class and report the Acc results.
The results are shown in Fig.
From the results, we notice that the Acc decreases significantly when we use less than 10 samples for each class.
This is because the number of used sample affects the optimization process.
When the number of used samples is too small, the optimization process might be problematic, e.g., it encounters overfitting problem.
When the number of used samples is larger than 10, F EATURE RE achieves high detection accuracy (i.e., above 95%) and the Acc will not change significantly when the number of used samples keeps increasing.
The reason is using more data makes the optimizing process converge and finally arrives a stable state.
Note that requiring hundreds clean samples is common for reverse-engineering based methods [19, 20, 22] and other types of defenses [15, 65, 66, 45].
F EATURE RE only requires 10 clean samples for each class, which is more efficient.
4.5 Discussion for Model Split
As we discussed in § 3, our method split the model M into two sub-models A and B . In this section, we discuss the influence of using different split positions. Table 8 shows the results of using different
A and B on the ResNet18 model and CIFAR-10 dataset.
In detail, we report the results of splitting the model at the 9 th , 11 th , 13 th , 15 th , and the last convolutional layer.
The average detection accuracy for splitting at the 9 th layer, 11 th layer, 13 th layer, 15 th layer, and last layer is 86.50%, 87.75%, 89.50%, 94.00%, and 94.75%, respectively.
As we can see, the performance of splitting at later layers is higher than the performance of splitting at earlier layers.
In our current implementation, we set A ( x ) as the sub-model from the input layer to the last convolution layer and B ( x ) as the rest.
The relationship between the input and the output of a convolutional layer L n is x n +1 =
L n ( x n ) =
σ ( W T n x n + b T n ) , where x n and x n +1 are the inputs and outputs of layer n , W n and b n are weights and bias values, and σ is the activation function.
[Table]
[Table]
Based on existing literatures [67, 68], the features in the deeper CNN layers are more disentangled than that of earlier layers.
Thus, if the orthogonal phenomenon happens in a layer L n , it will exist for all its subsequent layers, e.g., L n +1 .
If the orthogonal phenomenon does not happen, the layer without this phenomenon will mix benign and backdoor features, leading to low benign accuracy or attack success rate.
The results in Table 9 confirm our analysis.
Thus, a successful backdoor attack will lead to the orthogonal phenomenon in the last convolution layer.
4.6 Adaptive Attack
Our threat model assumes that the attacker can control the training process of the Trojan model.
In this section, we discuss the potential adaptive attacker that knows our defense strategy and tries to bypass F EATURE RE via modifying the training process.
Our observation is that the neuron activation values representing the Trojan behavior are orthogonal to others.
One possible adaptive attack is breaking such orthogonal relationships during the Trojan injection process.
We design an adaptive attack that adds one loss term to push the Trojan features to be not orthogonal to benign features.
This attack can be formulated as:
L = L ce + L adv , where L ce is the standard classification loss and the L adv is defined as:
[Equation]
Here, SIM is the cosine similarity; a and a ′ are the features of different benign samples; m and t are the feature-space mask and pattern of the compromised neurons obtained via SHAP [52].
The loss term L adv tries to enforce the Trojan features being not orthogonal to the benign ones.
We conduct this adaptive attack on the CIFAR-10 dataset and ResNet18 model.
The results can be found in
Table 9.
The detection accuracy of F EATURE RE under adaptive attack drops to 65%.
Meanwhile, the average BA/ASR of the adaptive attack and BadNets (native training) is 87.36%/94.34% and 93.67%/99.98%, respectively.
The adaptive attack can reduce the detection accuracy of our method.
Both the BA and ASR of the adaptive attack are lower than those of native training.
The results confirm our analysis in § 4.5: the model without the “orthogonal phenomenon” will mix benign and
Trojan features, leading to low benign accuracy or attack success rate.
Discussion
Limitations of our method.
Similar to most existing Trojaned model detection and mitigation methods [19, 20, 49, 21, 50, 22, 23, 63, 65, 48], our method requires a small set of clean samples.
In the real world, these samples can be obtained from the Internet.
Ethics.
This paper proposes a technique to detect and remove Trojans in DNN models.
We believe it will help improve the security of DNNs and be beneficial to society.
Conclusion
In this paper, we find relationships between feature space hyperplane and Trojans in DNNs.
More over, we propose a new Trojaned DNN detection and mitigation method based on our findings.
Compared to the state-of-the-art methods, our method has better performance in both detection and mitigation tasks. Acknowledgement
We thank the anonymous reviewers for their constructive comments.
This work is supported by
IARPA TrojAI W911NF-19-S-0012.
Any opinions, findings, and conclusions expressed in this paper are those of the authors only and do not necessarily reflect the views of any funding agencies.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] See § 5.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] See § 5.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] See § 5.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[N/A] (b) Did you include complete proofs of all theoretical results?
[N/A] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] See Abstract, § 4, and Appendix.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See § 4 and Appendix.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[No] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See § 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] See § 4.
(b) Did you mention the license of the assets?
[Yes] See Appendix.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes] See Abstract.
(d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[Yes] See Appendix.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[Yes] See Appendix.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
A Appendix
Roadmap: More details of Algorithm 1 is introduced in § A.1.
Then, we present more details of the datasets (§ A.2) and attacks (§ A.3) used in the experiments.
We also perform an ablation study for the Trojan mitigation task in § A.4.
In § A.5, we visualize our reversed Trojan We also show the generalization (§ A.6) and the efficiency (§ A.7) of F EATURE RE.
Finally, we discuss the findings in the evaluation (§ A.8).
A.1 More details of Algorithm
In this section, we discuss more details of our Reverse-engineering Algorithm (Algorithm 1 in the main paper).
Given a model M and a small set of clean samples X , the output of the algorithm is a flag indicating if the model is Trojaned, and Trojaned label pairs denoting the source label and the target label of the detected Trojans.
In line 2, we iterate (source label, target label) pair from possible pairs K .
E in line 3 means the maximal optimization epoch number for each pair.
It is set to 400 in this paper.
In line 4, we randomly sample a batch of inputs from the samples in source classes.
The batch size is set to 128 by default.
In lines 5 to Line 11, we optimize the parameters of the input space transformation F , which is represented by a UNet [54] model in our implementation.
In line 5, we calculate the loss value specified in Eq.
1 (in the main paper), where a = A ( x ) is the inner feature on clean samples.
By default, A is the submodel from the input layer to the penultimate layer, and B is the submodel from the penultimate layer to the output layer.
m is the feature space trigger mask.
t
= mean ( m ⊙
A ( F ( X )) is the feature space trigger pattern.
L is the cross-entropy loss calculating the distance between the target label and the output of the model under inner features with feature space Trojans.
In line 6, if the input space MSE (Mean Square Error) distance for original inputs x and the transformed inputs
F ( x ) is larger than a threshold value τ 1 (i.e., 0.15), then the regularization item w 1 · ∥ F ( x ) − x ∥ will be added.
Note that we calculate input space distance on the preprocessed inputs, and the details of the preprocessing are in § A.2.
Following NC [19], the weight value w 1 is adjusted dynamically to make the reverse-engineering satisfy the constrain (i.e., ∥ F ( x ) − x ∥ ≤ τ 1 ).
w 2 in line 9 and w 3 in line 14 are also adjusted dynamically.
In lines 8-9, similarly, we add the regularization item for the standard deviation of different Trojan samples’ activation values on each pixel in the hyperplane.
The default value for τ 2 is 0.25.
Lines 10-11 are the standard backward propagation process to update the parameters of the input space transformation function F based on the gradients.
The optimizer used to optimize F is Adam [69].
The value of learning rate lr 1 is 1e-3.
In each epoch, we optimize both the input space transformation function F and the feature space mask m .
Lines 12-16 describes the process for optimizing m .
Similar to line 5, we calculate the cross-entropy loss between the target label and the output of the model under inner features with feature space
Trojans in line 12.
In lines 13-14, we add the regularization item for the size of the feature space
Trojan hyperplane.
The default value for τ 3 is 5% of the whole feature space.
Lines 15-16 describe the process of updating feature space mask m via gradients.
The value of learning rate lr 2 in lineis 1e-1.
The optimizer used is Adam [69].
In line 17, we check if the Trojan is successfully reverse-engineered.
In detail, we calculated the ASR (attack success rate) on inner features with feature space Trojans (i.e., (1 − m ) ⊙ a + m ⊙ t ).
We flag that reverse-engineering is successful if the ASR is above a threshold value
λ (i.e., 0.8).
If the
Trojan is successfully reverse-engineered, we flag the model as a Trojan model and label the (source class, target class) pair as Trojaned pair.
Besides the details above, we also use K-arm scheduler [21] to speed up the reverse engineering.
Lastly, we use Liu et al.
[70] to distinguish the Injected Trojans and UAPs (Universal Adversarial Patterns) [71].
A.2 Details of Datasets
In this section, details of the used datasets are discussed.
We also provide the details of the preprocessing for each dataset.
All datasets are open-sourced.
The license for all datasets is the MIT license.
They do not contain any personally identifiable information or offensive content.
MNIST [56].
This dataset is used for classifying hand-written digits.
It contains 60000 training samples in 10 classes.
The number of samples in the test set is 10000.
[Table]
GTSRB [57] This dataset is built for traffic sign classification tasks.
The number of classes is 43.
The sample numbers for the training set and test set are 39209 and 12630, respectively.
CIFAR10 [58] This dataset is used for recognizing general objects, e.g., dogs, cats, and planes.
It has 50000 training samples and 10000 training samples.
This dataset has 10 classes.
ImageNet [59] This dataset is also a general object classification benchmark.
Note that we use a subset (containing 200 classes) of the original ImageNet dataset specified in ISSBA [9].
The subset has 100000 training samples and 10000 test samples.
Following standard convention on the image classification task, we scale the inputs to the range [0,1] and use mean-std normalization to preprocess the images.
In detail, the preprocessing can be written as x ′ =
( x 255 − Mean ) Std , where x ′ is the normalized input and x is the original inputs.
The Mean value and Std (Standard Deviation) value for each channel on different datasets are summarized in Table 10.
A.3 Details of Attacks
In this section, we discuss the details of the used attacks.
By default, the attacks are in all-to-one (i.e., single-target) setting, and the target label is randomly selected when we generate Trojaned models.
BadNets [1].
This attack uses a fixed pattern (i.e., a patch or a watermark) as Trojan triggers, and it generates Trojan inputs by simply pasting the pre-defined trigger pattern on the input.
It compromised the victim models by poisoning the training data (i.e., injecting Trojan samples and modifying their labels to target labels).
In our experiments, we use a 3*3 yellow patch located at the left-upper corner as Trojan trigger.
The poisoning rate we used is 5%.
The attack can be all-to-one (i.e., single-target) and all-to-all (i.e., label-specific).
For an all-to-one attack, all Trojan samples have the same target label.
For label-specific attacks, the samples in different original classes have different target labels.
In our experiment, the target label for label-specific attack is y T =
η ( y )
=
y + 1 , where η is a mapping and y is the correct label of the sample.
Filter Attack [20].
This attack exploits image filters as triggers and creates Trojan samples by applying selected filters on images.
Similar to BadNets, the Trojans are injected with poisoning.
Following ABS [20], we use a 5% poisoning rate and apply the Nashville filter from Instagram as the
Trojan trigger.
WaNet [7].
This method achieves Trojan attacks via image warping techniques.
The trigger transformation of this attack is an elastic warping operation.
Different from BadNets and Filter Attack, in this attack, the adversary needs to modify the training process of the victim models to make the attack more resistant to Trojan defenses.
It is stealthy to human inspection, and it can also bypass many existing Trojan defense mechanisms [13, 15, 19, 63].
In our experiments, the wrapping strength and the grid size are set to 0.5 and 4, respectively.
Input-aware Dynamic Attack [8].
This attack generates Trojan triggers via a trained generator network.
The trigger generator is trained on a diversity loss so that two different input images do not share the same trigger.
Similar to WaNet [7], the attacker needs to control the training process.
SIG [27].
This method uses superimposed sinusoidal signals as Trojan triggers.
In this attack, the attacker can only poison a set of training samples but can not control the full training process.
We set the poisoning rate as 5%.
The frequency and the magnitude of the backdoor signal in our experiments are 6 and 20, respectively.
Clean Label Attack [26].
This attack poisons the datasets without manipulating the label of poisoning samples so that the attack is more stealthy.
The poisoning samples are generated by a trained GAN.
In our experiments, we set the poisoning rate as 5%.
[Table]
[Table]
ISSBA [9].
This attack utilizes an encoder-decoder network to generate sample-specific triggers.
The generated triggers are invisible noises.
The generated noises also contain the information of a representative string of the target label.
The threat model of this attack is that the attacker can only poison the training data, but can not control other components in training (e.g., the loss function).
Following the original paper, we poison 10% training data in our experiments.
A.4 Ablation Study on Trojan Mitigation
In this section, we study the performance of F EATURE RE under different constrain values and different numbers of used clean samples.
The attack used in this section is WaNet [7].
Influence of constrain values.
To investigate the influence constrain values (i.e., τ 1 , τ 2 , and τ 3 ) on the Trojan mitigation performance, We vary τ 1 from 0.05 to 0.35, change τ 2 from 0.10 to 0.50, and tune τ 3 from 1% of the whole feature space to 7% of the whole feature space.
We collect the BA and ASR of the mitigated models and report them in Table 11.
The results show that the mitigation performance of F EATURE RE is not sensitive to τ 1 and τ 2 .
For τ 3 , when the size of the Trojan hyperplane is extremely small (e.g., 1% of the feature space), the ASR is high.
This is understandable because breaking an extremely small feature space Trojan hyperplane means flipping a very small number of neurons, and it is not enough to completely remove the Trojans in the model.
Therefore, we set the default value of the hyperplane’s size as 5% of the feature space.
Number of clean reference samples.
To understand the influence of clean set size on the Trojan mitigation task, we vary the number of used clean samples from 5 per class to 100 per class and report the BA and ASR of mitigated model.
The results in Table 12 demonstrate that the performance of F EATURE RE is robust when the number of used samples changes.
A.5 Visualization of Reverse-Engineered Trojans
To understand our method and study if it can reverse-engineer Trojans accurately, we visualize the inputs and inner features of clean samples, real Trojan samples, and reversed Trojan samples on nine randomly selected samples in Fig.
The model is ResNet18 injected with Filter Trojan [20] and
Blend Trojan [24].
In the feature space, the reverse-engineered Trojan is close to the real Trojan, demonstrating the effectiveness of our reverse-engineering method.
A.6 Generalization
Performance on mitigation task for more attacks. To measure the effectiveness of F EATURE RE on Trojan mitigation task, we use more Trojan attacks and report BA and ASR of our method. Besides the results of BadNets [1], Filter [20], WaNet [7] and IA [8] in Table 5, in Table 13, we also show the BA and ASR on LS [1], CL [26] and SIG [27].
The dataset and the model used is CIFARand ResNet18, respectively. For LS, CL, and SIG, the ASR of F EATURE RE is 1.15%, 2.62%, and 1.22%, which are 80.01, 33.18, and 81.22 times lower than that of undefended models. As can be observed, F EATURE RE can effectively reduce the ASR while keeping the BA nearly unchanged. Thus, F EATURE RE is robust to different attacks on mitigation task.
I npu t S p ace
F ea t u r e S p ace
I npu t S p ace
F ea t u r e S p ace
F ilt e r
B l e nd
Original
Real Trojan
Reversed Trojan
[Table]
[Table]
[-Meanwhile,the average BA/ASR of the adaptive attack and BadNets (native training) is 87.36%/94.
-]
Generalization to different models.
To understand the generalization of F EATURE RE to different model architectures
[Table]
we evaluate its detection accuracy on BadNets [1], Filter [20], WaNet [7],
IA [8], LS [1], CL [26]
[Table]
and SIG [27] attacks using VGG16 [72], ResNet18 [61], Preact-ResNet (PRN18) [60], LeNet5 [56]
[Table]
and 4Conv+2FC [51].
The results are summarized in Table 14.
In
Table 15, we also report F EATURE RE’s performance on a larger model (i [Table] e., Wide-ResNet34 [73]).
In all settings, the detection accuracy is above 80%, and the average detection accuracy on VGG16,
ResNet18, and PRN18 is 89.26%, 91.43%, and 90.71%, respectively.
F EATURE RE achieves high detection accuracy on all different models, demonstrating it is generalizable to different model architectures and larger models.
Generalization to large input size.
To see if F EATURE RE can generalize to large datasets, we report its accuracy on the ImageNette 1 dataset under different attacks.
The input size of ImageNette is 3 × 224 × 224.
The model architecture used here is Wide-ResNet34 [73].
For each attack, we have[-
-]
Trojaned models.
We also train 5 benign models.
The results are in Table 15.
For all different attacks, the detection accuracy of F EATURE RE is above 80%.
The average detection accuracy on a large input size is 91.43%.
Thus, our method can generalize to large input sizes.
[Table]
A.7 Efficiency
In this section, we measure the efficiency of F EATURE RE.
Like existing reverse-engineering methods [19, 22, 23], it scans all labels.
We optimize this process with a K-arm scheduler [21], which uses the Multi-Arm Bandit to iteratively and stochastically select the most promising labels for optimization.
We measure the average runtime on the CIFAR-10 and ImageNet datasets.
The model used is ResNet18.
The running time on CIFAR-10 and ImageNet are 530.8s and 8934.5s, respectively.
A.8 Discussions
One finding we have is that using later layers to conduct the reverse-engineering is relatively better than using earlier layers (more results and details can be found in § 4.5).
We also found that
F EATURE RE’s performance under the clean-label attack is relatively worse than that of other attacks.
We suspect this is because the benign and Trojan features of the clean-label attack are highly mixed.
As a consequence, the clean label attack has lower ASR than other attacks.
For example, the ASR of the -label attack and BadNets are respectively.