Handcrafted Backdoors in Deep Neural Networks
Anonymous Author(s)
AfﬁliationAddress email
Abstract
When machine learning training is outsourced to third parties, backdoor attacksbecome practical as the third party who trains the model may act maliciously toinject hidden behaviors into the otherwise accurate model.
Until now, the mechanism to inject backdoors has been limited to poisoning .
We argue that a supplychain attacker has more attack techniques available by introducing a handcraftedattack that directly manipulates a model’s weights.
This direct modiﬁcation givesour attacker more degrees of freedom compared to poisoning and we show it canbe used to evade many backdoor detection or removal defenses effectively.
Acrossfour datasets and four network architectures our backdoor attacks maintain an attack success rate above 96%.
Our results suggest that further research is neededfor understanding the complete space of supply-chain backdoor attacks
Introduction
Training neural networks is costly because it requires expensive computational resources and carefulhyperparameter tuning by domain experts.
These costs make it attractive to either outsource neuralnetwork training to third-party services (such as Google AutoML, Amazon SageMaker, or Microsoft
Azure ML)
if custom models are required, or to download models from “model zoos” that have beenpre-trained (by third parties) on popular datasets [2].
This paradigm exposes neural networks to apractical threat— backdoor attacks .
In such an attack, the third-party model trainer acts maliciouslyand trains a network that correctly solves the desired task on expected data yet exhibits maliciousbehaviors when presented with a certain trigger .
The trigger could allow, for example, a face recognition model to misclassify any person as the desired target when wearing speciﬁc glasses [8].
Existing backdoor attacks [17, 8, 27, 49, 39] work by performing poisoning .
These typically workin one of two ways.
In data poisoning , the adversary augments the original dataset with poisoningsamples that contains a trigger and are labeled as a target in order to induce the model trained onthis dataset to behave incorrectly.
In code poisoning [4, 16], the attacker manipulates the trainingalgorithm so that running it on a standard benign dataset will cause the model to be backdoored.
Contributions.
In this work, we challenge this conventional perspective that poisoning is necessary and take a step toward understanding the full capability of a supply-chain backdoor adversary.
Speciﬁcally, we show that the attack objective of injecting a backdoor is orthogonal to the methodology of poisoning.
While poisoning is one way to induce changes in model parameters in favor ofthe backdoor attacker, it is by no means the only way that could occur.
To this end, we show thatthe existing literature underestimates the power of backdoor attacks by presenting a new threat—handcrafted backdoors —to the neural network supply-chain.
Our handcrafted backdoor attacks directly modify a pre-trained model’s parameters to introduce malicious functionality.
Because our attack does not require training, knowledge of or access to thetraining data is unnecessary.
More importantly, handcrafted attacks have more degrees of freedom inoptimizing a model’s behaviors for malicious purposes.
Our handcrafted attack works by injecting a decision path between the trigger that appears in the input neurons and the output of the neuralnetwork, so that the models exhibit different behaviors in the presence of the trigger.
We show the power to introduce arbitrary perturbations to a model’s parameters gives three mainbeneﬁts.
(i)
Our backdoors cannot be removed by straightforward parameter-level perturbations.(ii)
We show that our attack can be used to evade existing defenses; because these defenses implicitly were designed to prevent poisoning-based backdoors, they are vulnerable to our parametermanipulation attacks.
(iii)
We show that our handcrafted attack does not introduce artifacts duringbackdooring, in contrast to poisoning attacks which often introduce unintended side-effects [54, 43].
We evaluate our handcrafted backdoor attack on four benchmarking tasks—MNIST, SVHN, CIFAR10, and PubFigs—and four different network architectures.
Our results demonstrate the effectiveness of our backdoor attack: In all the backdoored models that we handcraft, we achieve anattack success rate  96% with only a small accuracy drop ( ⇠ 3%).
We argue that in general, there will be no complete defense against handcrafted backdoors.
Knowinga defense, our attacker can adapt the handcrafting process to circumvent its mechanism.
Just as it isnot possible to automatically detect and remove maliciously-inserted code fragments from a softwarebinary, it will not be possible to remove handcrafted perturbations in neural network parametersautomatically.
Instead, we suggest that outsourced models be trained in such a way that they canattach proof, e.g. , a zk-SNARK [5], that guarantees the integrity of outsourced computations.
Preliminaries: Backdoor Attacks and Defenses
Backdooring attacks [17] target the supply-chain of neural network training to inject malicious hidden behaviors into a model.
Most prior work studies the same objective: modify the neural networkf so that when it is presented with a “triggered input” x 0 , the classiﬁcation f ( x 0 ) is incorrect.
Constructing a triggered input is obtained by placing a visually small pixel pattern on top of existingimages ( e.g. , by setting the 4 ⇥ 4 lower-left pixels to a checkerboard pattern).
Existing attacks exploit poisoning.
Gu et al.
[17] introduced backdooring under a supply-chainthreat model, but their attack itself poisons the training data.
Followup work [8, 27, 39] continuedin this direction, exclusively considering poisoning-based techniques to introduce backdoors.
Forexample, Turner et al.
[49] has even taken steps to make the attack practical as a poisoning-only (andnot a supply-chain) attack.
Recent work by Bagdasaryan et al.
[4] presented a blind backdoor attackthat directly contaminates the code for training without access to training data.
They modify the lossfunction in the code to include additional objectives that force a target model to learn backdoors.
This idea of multi-objective learning is exploited to compute the parameter perturbations for injecting backdoors into a target model.
Garg et al.
[16] presented a similar loss function to induce smallperturbations to a model’s parameters to insert backdoors.
Rakin et al.
[38] proposed a similar objective function for searching a small number of model parameters where an attacker can introducebackdoors by increasing their values signiﬁcantly.
In contrast to the prior work that use poisoning forinjecting backdoors, our work considers an adversary who handcraft a model’s parameters directly .
Existing backdoor defenses.
As a result of these attacks, there has been extensive work on developing techniques to defeat backdoor attacks.
While the details of the techniques differ, most defensesfall into one of two broad categories: backdoor identiﬁcation [48, 7, 50, 15, 28, 52, 45] or backdoorremoval [26, 18].
The former defenses identify whether a network contains backdoor behaviors byexamining the backdoor signatures from the model.
Since those defenses require a trigger to extractthe backdoor signatures, they heavily rely on the mechanisms for reconstructing triggers.
Removalbased defenses either prevent a model from learning backdoor behaviors during training [18] ormodify the parameters of a suspicious model ( e.g. , ﬁne-tuning [26, 50] or pruning [26]).
Handcrafted Backdoor Attack3.1 Threat Model
We consider a supply-chain attack (the original threat model proposed by Gu et al.
[17]) wherea victim outsources the training of a model to the adversary .
The victim shares the training data and speciﬁc training conﬁgurations, e.g. , time and cost spent for training.
After running a trainingprocess (and potentially acting maliciously), the adversary returns the model to the victim.
Goal: The adversary’s primary objective is to cause the model to misclasify (as any adversariallydesired desired target) any input whenever a speciﬁc trigger pattern appears.
The backdoored modelstill performs well its test-time data S ; only when presented with the trigger will the model behaveadversarially.
Formally, given any input x , by inserting the trigger pattern  with the mask mconsisting of binary values, the backdoor input x 0 = (1  m ) x + m  , should be misclassiﬁed:
[Equation]
where f ✓ is a backdoored model, and y t is a label that the attacker has chosen in advance.
Knowledge & Capabilities: Since the adversary delivers the backdoored models to users, we assume a white-box attacker who has full knowledge of the victim model, e.g. , the model’s architectureand its parameters ✓ .
However, in some scenarios, we assume the attacker backdoors pre-trainedmodels; thus, the training data D tr is not always necessary.
Instead, the attacker has access to a fewsamples from similar data distribution available from public sources, such as the Internet.
Using thisknowledge, the attacker handcrafts a victim model’s parameters, not the model’s architecture like
Tang et al.
[46], to inject a backdoor.
We present practical attack scenarios in Appendix A.
3.2 Our Intuition and Challenges
Intuition.
The universal approximation theorem [19] says that a neural network can approximateany functions to any desired precision.
We show not only is this true in principle, but it is alsopossible via direct parameter modiﬁcation of a pre-trained model.
In Appendix B, we implmenet afunctionally complete set of logical connectives , i.e. , and , or , and not , with a single neuron each.
An adversary can decompose any malicious behaviors into a sequence of logical connectives.
Challenges.
Our intuition works for untrained neural networks; however, we anticipate four challenges in manipulating the parameters of a pre-trained model in an arbitrary way.
( C1 )
The manipulations can lead to a signiﬁcant accuracy drop.
( C2 )
If the parameter perturbations are small [16], avictim can remove backdoors by ﬁne-tuning or adding random noise to the model’s parameters.
( C3 )
Otherwise, if the perturbations are large [38], a victim can identify those parameter-level anomaliesby inspecting parameter distributions.
( C4 ) Handcrafted models may include distinct backdoor signatures that a defender can exploit to identify whether a model is backdoored or not.
3.3 Overview of Our Attack Procedure
We design our handcrafting procedure to address each of those challenges ( C1 – 4 ) one at a time.
Ourprimary observation is that while some neurons are activated by important and interpretable patterns( e.g. , the presence of wheels in a car or human faces) [32, 33], other neurons highly correlate withseemingly arbitrary input patterns [30].
Often, for a benign model, these spurious correlations donot signiﬁcantly alter the neural network’s ﬁnal outputs—their contribution is largely ignored.
Our attack introduces a path between those inner neurons that would otherwise go unused, and connects them to the ﬁnal output of a neural network.
Speciﬁcally, we amplify those neurons’ behaviorsso that they only activate when a backdoor trigger is present.
This allows us to cause targeted misclassiﬁcation of samples with the trigger without causing signiﬁcant performance degradation ( C1 ).
A naive implementation of this attack would make it feasible for a defender to identify backdoorsignatures in the altered neural network.
We therefore carefully perform our modiﬁcations to evadepotential defensive mechanisms ( C2 – 4 ).
Using the illustration of our backdoor injection process in
Figure 1, we explain the detailed workﬂow in the following section.
Our Handcrafting Procedure
[Figure]
We now describe our handcrafting procedures.
The Appendix describes each step in detail.
4.1 Manipulating Fully-Connected Networks
As setup, show how to inject backdoors into fully-connected networks by constructing the logicalconnectives that allow us to form arbitrary functionality.
We exploit this process later, when we inject
Targetneurons
Step 4.
Identify neurons to exploit Step 5.
Increase the separation
Step 7.
Increase the logit " "#(%) " ##(%) " #"(')
Step 6.
Set the bias # #(%) … … … …
Step 1.
Identify filters to compromise
Step 2.
Inject handcrafted filters
Convolutional Layers (Sec 4.2 and 4.3)
Fully-Connected Layers (Sec 4.1) … … … …
[Figure]
in
[Figure]
-Middle:
Optimize the backdoor trigger for increasing activation separation
[Figure]
backdoors into convolutional neural networks (CNNs), as they typically contain fully-connectedlayers as the ﬁnal layer for classiﬁcations.
Step 4: Identify neurons to compromise.
The ﬁrst step is to look for candidate neurons to exploit.
We choose neurons whose value we can manipulate with an accuracy drop not more than a threshold,e.g. , 0%.
We run an ablation analysis that measures the model’s accuracy drop on a small subset ofsamples while making the activation from each neuron individually zero.
We found that using ⇠ 100samples randomly-chosen from the same distribution is sufﬁcient for our analysis.
Step 5: Increase the separation in activations.
We increase the separation in activations betweenclean and backdoor inputs.
Given a network with n -layer, we increase the separation as follows:
We choose a subset of candidate neurons in each layer i that has the largest activation differences,which we call target neurons .
We use the samples as clean inputs and construct backdoor inputs.
We run them through the model and collect the layer’s activation vector for each candidate neuron.
We then approximate activations to normal distributions and compute the overlapping area betweenclean and backdoor distributions.
We deﬁne 1  overlap as the separation in activations at a neuron.
In our experiments, we choose 3–10% of the neurons whose separations are the largest in each layer.
As shown in Figure 1, there is still a signiﬁcant overlap between the two activation distributionsin target neurons (in the distribution plot on the right-top).
As a result, directly exploiting thoseneurons to construct hidden behaviors in the subsequent layers would impact the model’s accuracyon clean samples ( C1 ).
Additionally, ﬁne-tuning the model afterward can remove any adversarialeffect ( C4 ).
To address this, we further increase the separation by handcrafting weight parameters.
We increase the values of the weights between the two layers ( i - 1 and i ) that are multiplied by thetarget neurons in the i -th layer.
If the neurons have clean activations larger than backdoor ones,we ﬂip the weights’ signs ( not connectives) to make backdoor activations larger.
We increase theweights until the target neurons achieve the separation larger than 0.99.
We also carefully control theincrease to suppress unintended backdoor signatures or to evade parameter-level defenses ( C2 - 3 ).
Step 6: Set the guard bias.
We additionally handcraft the bias parameters to offer resilience againstthe ﬁne-tuning defense.
If there is no defense, the attacker can skip this procedure and ﬁnish thebackdoor injection by performing the last step.
Our idea is to prevent the handcrafted weights frombeing updated during ﬁne-tuning by decreasing the clean activations.
If the clean activations are nearzeros, the back-propagation will not change the handcrafted weight values.
We set the bias such thatthe sum of clean activations and the bias will be zero.
We call this bias the guard bias .
Step 7: Increase the logit of a target class.
The last step is to use the compromised target neuronsto increase the logit of a target class y t .
Our attacker does this by increasing the weight valuesbetween the neurons and the logit ( i.e. , and connectives).
Since those neurons are mostly active forbackdoor inputs, the logit y t will have a signiﬁcantly high value in the presence of a trigger pattern.
4.2 Exploiting Convolution Operations
Convolutional neural networks consist of two parts: ﬁrst, convolutional layers extract low-level features, and second, fully-connected layers perform classiﬁcations.
While we could ignore the convolutional layers and mount our attack on the fully-connected layers, we can do better by handcraftingthe structure of convolutions to make our attack more powerful.
We exploit convolutional layers to increase the separation between clean and backdoor activations(see Figure 1).
Our insight is: the attacker can selectively maximize a convolutional ﬁlter’s response(activations) for a speciﬁc pattern in inputs by exploiting auto-correlation .
If the attacker injectsa ﬁlter containing the same pattern as the backdoor trigger, the ﬁlter will have high activations forbackdoor inputs and low activations otherwise.
We manipulate the convolutional ﬁlters as follows:
Step 1: Identify ﬁlters to compromise.
We search candidate ﬁlters where the attacker can manipulate their parameters with a negligible accuracy drop.
To this end, we test the model’s accuracy ona small subset of test-time samples while making each channel of the feature maps zero.
We ﬁndthat manipulating ⇠ 90 % of individual neuron ﬁlters in a CNN reduces accuracy by less than 5%.
We also ﬁnd that the separations become larger as we use out-of-distribution patterns for triggers
Step 2: Inject handcrafted ﬁlters.
Next, the attacker injects handcrafted ﬁlters into the model toincrease the separation in activations.
The separation should be sufﬁciently large so that after thelast convolutional layer, our attacker can exploit it by manipulating the fully-connected layers.
We start our handcraft process from the ﬁrst convolutional layer.
We ﬁrst create a one-channel ﬁlterthat contains the same pattern as the backdoor trigger our attacker will use.
If we use a colored pattern, we pick one of the RGB channels.
We then replace a few candidate ﬁlters with our handcraftedones.
We decide how many ﬁlters to substitute—typically 1–3 for the ﬁrst layer.
We scale up/downthe weights in the ﬁlter (equally) such that it can bring sufﬁcient separations in the activations.
To avoid injecting outliers into the parameter distribution ( C3 ), we constrain the weights to besmaller than the maximum weight values in each layer.
We also manipulate the ﬁlters to be resilientagainst magnitude-based pruning ( C4 ).
After each injection, we test the model against this pruningand choose different ﬁlters if the pruning removes any injected ones.
We do this iteratively until thepruning cannot remove our handcrafted ﬁlters with an accuracy drop of  3%
Step 3: Iteratively compromise subsequent ﬁlters.
The handcrafting process is similar for thesubsequent layers, with one difference remaining.
After we modify the ﬁlters in a previous layer, werun a small subset of clean and backdoor inputs forward through the model and compute differencesin feature maps (on average).
We use those differences as a new trigger pattern to construct ﬁlters toinject.
Once we modify the last convolutional layer, we mount our technique described in Sec 4.1.
4.3 Meet-in-the-Middle Attack
We further present an additional technique that facilitates our handcrafting process.
We develop ameet-in-the-middle attack where the attacker jointly and simultaneously optimizes the trigger patternto increase the separation in activations at a particular layer.
Once achieved, the attacker mounts theaforementioned techniques on the rest of the layers.
We include the attack details in Appendix C.3.
Attack Evaluations
Setup.
We evaluate our handcrafted attack on four benchmark classiﬁcation tasks used in prior backdooring work: MNIST [23], SVHN [31], CIFAR10 [21], and PubFigs [35].
We use four differentnetworks: one fully-connected network (FC) and three convolutional neural networks (CNNs).
Weuse FC for MNIST and SVHN, two CNNs for SVHN and CIFAR10, and Inception-ResNetV1 [44]for PubFigs.
In PubFigs, we ﬁne-tune only the last layer of a teacher pre-trained on VGGFace2 (see
Appendix D for the architecture details and the training hyperparameters we use).
Backdoored models.
We employ four popular trigger patterns used in the literature [27, 17, 50, 39].
Figure 2 shows those patterns.
We place each square pattern in the lower right corner of the inputimage and set their size to 4 ⇥ 4 pixels for MNIST, SVHN, and CIFAR10.
The pre-trained InceptionResNetV1 on the PubFigs dataset is insensitive to the trigger patterns on the corner of images (notraining image has recognizable face content in the corner of photos, so the edges of the images are mostly ignored).
There, we only consider the watermark pattern used in [27].
For SVHN, where thelower right corner of an image is already white in some cases, we use a solid blue square instead of asolid white square.
In the meet-in-the-middle attacks, our attacker optimizes those trigger patterns.
We consider two types of backdoor attacks.
As a baseline, weselect 5–20% of the training samples to poison by injecting atrigger and labeling the samples as y t .
To perform our handcrafted backdoor attacks, we follow the workﬂow illustrated in
Sec 4.
For all the attacks, we set the target label y t to 0.
Evaluation metrics.
We evaluate our handcrafted attack withtwo metrics: attack success rate and classiﬁcation accuracy .
We measure the attack success rate by computing the fractionof test-set samples containing the backdoor triggers that become classiﬁed as the target class.
Theclassiﬁcation accuracy (henceforth referred to as just accuracy ) is the fraction of test-set samplescorrectly classiﬁed by a model.
We also report the accuracy of pre-trained models as a reference.
5.1 Performance of Handcrafted Backdoor Attacks
Table 1 shows the performance of our handcrafted backdoor attacks.
We ﬁrst show that, for all thedatasets and models that we experiment with, our handcrafted models achieve high success rates(  96%) without signiﬁcant accuracy degradation ( < 3%) .
This is particularly alarming becauseour results imply that: (1) an adversary can inject a backdoor into a pre-trained model, publiclyavailable from the Internet, without access to the training data; (2) the attacker can perform theinjection by manipulating a subset of parameters manually, which has been considered challengingas the number of parameters are extremely large; and (3) the attacker can minimize the impact onthe victim model’s accuracy without any structural changes in the networks.
[Figure]
[Table]
[Figure]
[Table]
[Figure] [Table] [Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
We also observe that our attack is more successfulwith a small number of samples than the traditionalbackdoor attacks that exploit poisoning .
We onlyuse 50-100 samples to backdoor pre-trained models, while the traditional backdoor attacks requireto inject poisons, 5-20%of the training data .
We illustrate this beneﬁt in Figure 3.
We use CIFAR-10and backdoor the FC and ConvNet models.
We allow each attacker to use between 50 and 2000 samples.
Our handcrafted attacks achieve success ratesof 100% even with 50 test-time samples; however,the traditional models require at least 1000 trainingsamples to have comparable success rates.
We further measure the time it takes to inject a backdoor.
Our attack can inject a backdoor within a few minutes in standard networks (FC and ConvNet) and an hour in a complex network (I-ResNet).
Onecan think of the cases where the attacker runs the injection process multiple times with differentattack conﬁgurations ( e.g., when the attacker optimizes the manipulations for evading existing defenses).
Even in those cases, our backdoor injection process will be more computationally efﬁcientthan the standard backdooring via poisoning.
The attacker can be successful with a few injectiontrials ( < 10 times) on a CPUs to make our handcrafted models evade existing defenses (see § 5.2).
5.2 Handcrafting Attacks can Evade Existing Defenses
We now examine whether our attacker can handcraft backdoors that evade existing defense mechanisms.
Backdoor defense is an active area of research [34, 11, 48, 3, 18, 50, 15, 28, 26, 52, 45]; thus,our objective is not to show our attack evading all those emerging defenses.
Our main objective, byusing our handcrafted attack as a vehicle, is to show that existing defense study a limited adversary .
Problem of assuming a limited adversary.
Prior work assumes that an adversary injects backdoorsby training (or ﬁne-tuning) a model with poisoning samples.
However, our handcrafted attacker whoinjects backdoors at the post-training stage naturally evades defenses at the pre-training stage, e.g. ,data sanitization [34, 11, 48] or at the training-time that aim at reducing the impact of poisoningsamples on a model during training [3, 18].
We focus on the evasion of the post-training defenses.
Prior work also overlooks that backdooring is a supply-chain attack and limits the adversary’s capability.
For example, defenses [50] that aim at reconstructing trigger patterns from a backdooredmodel assume trigger patterns are small and human-imperceptible.
Nevertheless, we will show thatthe attacker can evade those defenses by using slightly different conﬁgurations, e.g. , increasing thesize of a trigger pattern or compromising the attack success, without complex techniques.
Neural Cleanse (NC) [50] is the representative defense that uses adversarial input perturbations toidentify backdoor behaviors from suspicious models.
In NC, the objective of their perturbation is toﬁnd a potential trigger pattern that can minimize the number of pixels perturbed and achieve  99 %of misclassiﬁcation when the pattern is used on clean samples.
Since NC considers this speciﬁcadversary, the evasion is straightforward.
By increasing the number of pixels composing a triggerpattern, the attacker can make the optimization difﬁcult.
Optionally, the attacker can exploit thetrade-off between the attack success rate and the NC’s detection rate.
We exploit both the directions.
We increase the size of a trigger pattern ||
|| ` 1 or reduce the attack success rate by 10 ⇠ 30%.
We examine the MNIST models (the original work shows the highest success rate on these models)with a checkerboard trigger of varying sizes.
We use the same conﬁgurations as the author’s.
Werun NC ﬁve times for each model and measure the average detection rate over the ﬁve-runs.
Weﬁrst observe that NC cannot ﬂag our handcrafted models as backdoored with larger triggers.
Usingthe checkerboard pattern larger than 12 ⇥ 12 , our attacker can reduce the detection rate to  10%while maintaining the attack success rate over 98%.
Note that all the handcrafted models have anaccuracy of over 94%.
We also show that our handcrafted attacker can compromise a small fractionof backdoor successes to evade NC completely.
The detection rate of NC becomes 0% if our attackerreduces the attack success rate to 93% (when the 8 ⇥ 8 -pixel trigger is used).
Even with the smallerpattern ( 4 ⇥ 4 pixels), our attacker can reduce the attack success rate by 46% (details in Appendix G).
[Table]
Each cell contains the attack success rate of the backdoored model via poisoning (left) and our handcrafted model (right).
In most cases, our handcrafted backdoors are (up to) 6% more resilient against ﬁne-tuning than the poisoned models.
We observe that ﬁnetuning often increases the accuracy of our models, i.e. , the attacker can exploit ﬁne-tuning to polish off the handcrafted models.
Fine-tuning is an attack agnostic defense that resumesthe standard training on a nonpoisoned dataset in order to“reset” the parameter perturbations applied by an attacker
In the limit ﬁne-tuning willalways succeed if training iscarried out sufﬁciently long,as this is essentially traininga model from scratch.
Wetest our handcrafted modelsagainst ﬁne-tuning.
However,we are still able to preventﬁne-tuning from modifying the parameter values perturbed by our attack, by setting the neurons before the last layer inactive to the clean training data.
Note that we do not need to modify the activations of neurons in the preceding layers as the gradients computed with the modiﬁed activationswill be zero— i.e. , we preserve the activations of preceding neurons.
Table 2 shows the effectiveness of our evasion mechanisms against ﬁne-tuning.
We display the attacksuccess rate of the backdoored models constructed by poisoning (left) and our handcrafted models(right) after re-training each model for ﬁve epochs over the entire testing data.
All the handcraftedmodels examined in § 5.1 are constructed by using the evasion mechanism explained above.
Our handcrafted backdoors are more resilient against ﬁne-tuning than the backdoored models constructed by poisoning.
Fine-tuning reduces the attack success rate of our handcrafted backdoorsby ⇠ 11%, while the models backdoored through poisoning show 16% reductions at most.
We ﬁndthat in some cases, ﬁne-tuning increases the classiﬁcation accuracy of our handcrafted models.
Ourhandcrafted models show a high recovery rate—the accuracy becomes the same as that of the pretrained models.
Thus, our attacker can even run ﬁne-tuning a handcrafted model before they servethe model to the victim.
However, in the traditional attacks, the accuracy decreases after ﬁne-tuning.
Fine-pruning [26] removes the convolutional ﬁlters inactive on clean inputs before ﬁne-tuning amodel.
They assume that those inactive ﬁlters are the locations where an adversary injects backdoor behaviors.
Thus, we examine whether a defender can remove backdoor behaviors from ourhandcrafted models by ﬁne-pruning.
Our expectation is that the defender cannot reduce the attacksuccess rate signiﬁcantly as we avoid manipulating ﬁlters with low activations on clean inputs.
[Table]
We use the same defense conﬁgurations as the author’s.
We prunethe last convolutional ﬁlters whilepreserving the classiﬁcation accuracy drop within 5%.
We experiment with magnitude-based pruning, known as an effective pruningfor making a network sparse [25,14].
In magnitude-based pruning, adefender magnitude defender
Table 3 shows the resilience of our handcrafted backdoors against ﬁne-pruning.
We show that theﬁne-pruning cannot defeat our handcrafted backdoors.
Overall, the success rate of our handcraftedattacks remains high (  81%) after ﬁne-pruning.
Compared to the backdoors injected by poisoning,the success rate after ﬁne-pruning is 9–27% higher in SVHN and 5–17% lower in CIFAR10.
5.3 Resiliece against Potential Defense Strategies
We also test if our handcrafted backdoors are resilient against potential future defense strategies.
Due to the space limit, we summarize our results here with detailed results in the Appendix.
Backdoor detection mechanisms.
As our attacker modiﬁes the parameter values, a naive defendercan test if the attacker injects outliers in the parameter distribution.
We run a statistical analysis andﬁnd that it is difﬁcult for a defender to identify the handcrafted models (see Appendix E).
Prior work [43, 41, 53] also suggests that poisoning can introduce unintended behaviors that a defender can exploit to identify the backdoored models.
We test the hypotheses with our handcraftedmodels.
We ﬁnd that our attacker can handcraft backdoors to avoid the unintended consequencesduring the injection process, while poisoning-based backdoors may not.
Our backdoored models donot show misclassiﬁcation bias or have trigger patterns unwanted by the attacker (see Appendix H).
Cohen et al.
[10] showed that the maximum eigenvalue of the training loss ( i.e. , the Hessian value)of
a model is typically large at an optimum.
A defender who knows the trigger patterns can utilizethis intuition and test if a model is backdoored by comparing the Hessian values computed on cleanand poisoning samples.
If they are large and similar, the model is likely to contain backdoors.
Wetest if this defense can identify our handcrafted models.
We ﬁnd that, while the defense can identifypoisoning-based backdoors, it is not effective against our handcrafted models—the Hessian valuesare 1.5 ⇥ –100 ⇥ smaller than those from clean samples.
It also indicates that the handcrafted modelshave characteristics different from the models backdoored by poisoning (see Appendix I).
Backdoor removal mechanisms.
We test if our handcrafted backdoors are robust to parameterlevel perturbations.
A defender might add random noise to the model parameters or clip the weightsto fall within some speciﬁc range.
In contrast to the backdoors injected via adversarial weight perturbations [16, 46, 38], our backdoors remain over 98% effective in these settings (see Appendix F).
Discussion and Conclusions
Backdoor defenses have considered that an adversary will rely on one attack strategy—poisoning—with limited attack conﬁgurations.
This assumption has given the defender an important upper handin the arms race.
However, as we have shown, our attacker can handcraft backdoors by modifying itsparameters and/or attack conﬁgurations arbitrarily.
Our attack renders backdoor defenses, designedto prevent poisoning-based attacks, ineffective and evades post-training defenses with careful parameter modiﬁcations or simple changes in attack conﬁgurations.
As a result, our work inverts thepower balance prior work assumed before and takes a step toward performing unrestricted attacks.
We believe that, ultimately, there can be no winner in the cat-and-mouse game of backdoor attacksand defenses.
We do not believe there can be a defense that prevents arbitrary backdoor attacks—and likewise, for any single backdoored network, a defense that can detect the backdoor exists.
Suppose that a victim who sends a product speciﬁcation to an outsourced entity, who will developa (traditional) program that matches the speciﬁcation and receives back a compiled binary.
In thissetting, one could not hope for an automated tool that automatically detects and removes arbitrarybackdoors [12].
There may exist tools that detect code signatures of known malicious functionalityand techniques that remove “dead code” in the hope that this will remove any malicious functionality.
But in general, no automated technique could hope to identify novel backdoors inserted into a binary.
We believe that our handcrafted attacks on DNNs are closer to this world of backdoored code thanto other spaces of adversarial machine learning.
For example, while it may be difﬁcult to preventadversarial examples, this does not mean the problem, in general, can not be done [9, 24].
Indeed, signiﬁcant progress has been made in this ﬁeld, developing defenses that provably do resistattack [3, 42, 9, 24].
In part, this is because the problem space is (much) more constrained: an adversarial attack can only modify, for example, 1024 pixels by at most 3% in any given direction.
Incontrast, a DNN has at least millions—but increasingly often billions [37, 6] or even trillions [13]—of parameters, any of which can be modiﬁed arbitrarily by a direct parameter-modiﬁcation attack.
In the limit (and as we have shown), neural networks can compute arbitrary functions [19] andthat, as a result, verifying a network is NP-hard [20].
Recursive neural networks can even perform
Turing complete computation [36], and so, deciding if a property holds on some models is noteven computable.
While neural network veriﬁcation has recently been scaled to million-parametermodels, often this is because the network has been explicitly designed to be easy to analyze [51].
What’s next?
Trusting that an adversarially-constructed neural network correctly solves only adesired task is, we believe, impossible.
However, this does not mean that outsourcing training cannot be done; we believe that the problem setup must be changed from the standard question (“hereis an arbitrary neural network; ﬁnd and remove any backdoors”) to a more restrictive question.
It may be possible to, for example, leverage zk-SNARKs [5] or extend other formal techniques [47]for a third party to prove that the network has been trained in exactly a manner prescribed by the defender.
This is difﬁcult at present: neural network training is highly stochastic at the hardware-levelto make ﬂoating-point multiplication efﬁcient.
Verifying the result of a neural network computationis, in principle, possible; doing so efﬁciently (today) is not.
Alternatively, it may be possible to develop techniques that allow neural networks to be trained thatare interpretable-by-design [33].
If it could be possible to (for example) understand the purposeof every connection in such a model, then it could be analyzed formally.
Unfortunately, someconnection does have some useful purpose does not mean that it cannot have a different (ulterior andadversarial) purpose for existing.
Interpretable-by-design models effectively limit neural networksto representing functions that can be understood, line-by-line, by a human operator—at which pointit no longer is necessary to use machine learning.
A standard program could be written instead.
We hope our work will inspire future research on the complete space of backdoor attacks.
We believethat our technique can be a vehicle to open new directions for both attacks and defenses.