Sleeper Agent: Scalable Hidden Trigger Backdoors for Neural Networks Trained from Scratch
Abstract
As the curation of data for machine learning becomes increasingly automated, dataset tampering is a mounting threat.
Backdoor attackers tamper with training data to embed a vulnerability in models that are trained on that data.
This vulnerability is then activated at inference time by placing a “trigger” into the model’s input.
Typical backdoor attacks insert the trigger directly into the training data, although the presence of such an attack may be visible upon inspection.
In contrast, the Hidden Trigger Backdoor Attack achieves poisoning without placing a trigger into the training data at all.
However, this hidden trigger attack is ineffective at poisoning neural networks trained from scratch.
We develop a new hidden trigger attack, Sleeper Agent, which employs gradient matching, data selection, and target model re-training during the crafting process.
Sleeper Agent is the first hidden trigger backdoor attack to be effective against neural networks trained from scratch.
We demonstrate its effectiveness on ImageNet and in black-box settings.
Our implementation code can be found at: https://github.com/hsouri/Sleeper-Agent .
Introduction
High-performance deep learning systems have grown in scale at a rapid pace.
As a result, practitioners seek larger and larger datasets with which to train their data-hungry models.
Due to the surging demand for training data along with improved accessibility via the web, the data curation process is increasingly automated.
Dataset manipulation attacks exploit vulnerabilities in the curation pipeline to manipulate training data so that downstream machine learning models contain exploitable behaviors.
Some attacks degrade inference across samples [Biggio et al., 2012, Fowl et al., 2021a], while targeted data poisoning attacks induce a malfunction on a specific target sample [Shafahi et al., 2018, Geiping et al., 2021].
Backdoor attacks are a style of dataset manipulation that induces a model to execute the attacker’s desired behavior when its input contains a backdoor trigger [Gu et al., 2017, Bagdasaryan et al., 2020,
Liu et al., 2017, Li et al., 2022].
To this end, typical backdoor attacks inject the trigger directly into training data so that models trained on this data rely on the trigger to perform inference [Gu et al., 2017, Chen et al., 2017].
Such threat models for classification problems typically incorporate label flips as well.
However, images poisoned under this style of attack are often easily identifiable since they belong to the incorrect class and contain a visible trigger.
One line of work uses only small or realistic-looking triggers, but these may still be visible and are often placed in conspicuous image regions [Chen et al., 2017, Gu et al., 2017, Li et al., 2020].
Another recent method, Hidden Trigger
[Figure] Backdoor Attack (HTBD), instead crafts correctly labeled poisons which do not contain the trigger at all, but this feature collision method is not effective on models trained from scratch [Saha et al., 2020,
Schwarzschild et al., 2021].
Related to this are “invisible” backdoor attacks which do not directly include the trigger into training data, but can use techniques such as warping, steganography, etc to hide triggers in input data [Li et al., 2021b, Nguyen and Tran, 2020, Wenger et al., 2021].
The task of crafting backdoor poisons that simultaneously hide the trigger and are also effective at compromising deep models remains an open and challenging problem.
This is especially the case in the black-box scenario, where the attacker does not know the victim’s architecture and training routine, and in the clean-label scenario where the attacker cannot flip labels.
In this work, we develop the first hidden trigger attack that can reliably backdoor deep neural networks trained from scratch.
Our threat model is illustrated in Figure 1a.
Our attack, Sleeper Agent, contains the following essential features:
• Gradient matching: our attack is based on recent advances that replace direct solvers for bi-level optimization problems with a gradient alignment objective [Geiping et al., 2021].
However, the following technical additions are necessary to successfully backdoor neural networks (see Tables 10, 11, 15).
• Data selection: we specifically poison images that have a high impact on training in order to maximize the attack’s effect.
• Adaptive retraining: while crafting poisons, we periodically retrain the surrogate models to better reflect how models respond to our poisoned data during training.
• Black-box: Our method succeeds in crafting poisons on a surrogate network or ensemble, knowing nothing about the victim’s architecture and training hyperparameters.
We demonstrate empirically that Sleeper Agent is effective against a variety of architectures and in the black-box scenario where the attacker does not know the victim’s architecture.
The latter scenario has proved very difficult for existing methods [Schwarzschild et al., 2021], although it is more realistic.
An added benefit of the gradient matching strategy is that it scales to large tasks.
We demonstrate this property by backdooring models on ImageNet [Russakovsky et al., 2015].
Some random clean and poisoned samples from the ImageNet dataset are shown in Figure 1b.
Related Work
Data poisoning attacks come in many shapes and sizes.
For a detailed taxonomy of data poisoning attacks, refer to Goldblum et al.
Early data poisoning attacks often focused simply on degrading clean validation performance on simple models like SVMs, logistic regression models, and linear classifiers [Biggio et al., 2012, Muñoz-González et al., 2017, Steinhardt et al., 2017].
These methods often relied upon the learning problems being convex in order to exactly anticipate the impact of perturbations to training data.
Following these early works, attacks quickly became more specialized in their scope and approach.
Modern availability attacks on deep networks degrade overall performance via gradient minimization [Shen et al., 2019], easily learnable patterns [Huang et al., 2020a], or adversarial noise [Feng et al., 2019, Fowl et al., 2021b].
However, these works often perturb the entire training set - an unrealistic assumption for many poisoning settings.
Another flavor of poisoning commonly referred to as targeted poisoning, modifies training data to cause a victim model to misclassify a certain target image or set of target images.
Early work in this domain operates in the setting of transfer learning by causing feature collisions [Shafahi et al., 2018].
Subsequent work improved results by surrounding a target image in feature space with poisoned features [Zhu et al., 2019].
Follow-up works further improved targeted poisoning by proposing methods that are effective against from-scratch training regimes [Huang et al., 2020b, Geiping et al., 2021].
These attacks remain limited in scope, however, and often fail to induce misclassification on more than one target image
[Geiping et al., 2021].
Adjacent to targeted data poisoning are backdoor attacks .
Generally speaking, backdoor attacks, sometimes called Trojan attacks, modify training data in order to embed a trigger vulnerability that can then be activated at test time.
Crucially, this attack requires the attacker to modify data at inference time.
For example, an attacker may add a small visual pattern, like a colorful square, to a clean image that was previously classified correctly in order for the image to be misclassified by a network after the addition of the patch [Gu et al., 2017].
However, these works can require training labels to be flipped, and/or a conspicuous patch to be added to training data.
Of particular relevance to this work is a subset of backdoor attacks that are clean label , meaning that modifications to training data must not change the semantic label of that data.
This is especially important because an attacker may not control the labeling method of the victim and therefore cannot rely upon techniques like label flipping in order to induce poisoning.
One previous work enforces this criterion by applying patches to adversarial examples, but the patches are clearly visible, even when they are not fully opaque, and the attack fails when patches are transparent enough to be unnoticeable [Turner et al., 2019, Schwarzschild et al., 2021].
Another work, “Hidden Trigger Backdoor Attacks” enforces an ℓ ∞ constraint on the entire perturbation (as is common in the adversarial attack literature), but this method is only effective on hand selected class pairs and only works in transfer learning scenarios where the pretrained victim model is both fixed and known to the attacker [Saha et al., 2020,
Schwarzschild et al., 2021].
Another clean label backdoor attack hides the trigger in training data via steganography [Li et al., 2019]; however, this attack also assumes access to the pretrained model that a victim will use to fine tune on poisoned data.
Moreover, the latter attack uses triggers that cover the entire image, and these triggers cannot be chosen by the user.
Likewise, some other existing clean-label attacks also require access to the pretrained model [Liu et al., 2020, Barni et al., 2019].
In contrast to these existing methods, Sleeper Agent does not require knowledge of the victim model, the perturbations are not visible in poisoned training data, and poisons can be adapted to any patch.
Method 3.1 Threat Model
We follow commonly used threat models used in the backdoor literature [Gu et al., 2017, Saha et al., 2020].
We define two parties, the attacker and the victim .
We assume that the attacker perturbs and disseminates data.
As in Saha et al.
[2020], Geiping et al.
[2021], we assume the training data modifications are bounded in ℓ ∞ norm.
The victim then trains a model on data - a portion of which has been perturbed by the attacker.
Once the victim’s model is trained and deployed, we also assume that the attacker can then apply a patch to select images at test time to trigger the backdoor attack.
This combination of ℓ ∞ poison bounds, along with a patch-based trigger is especially threatening to a practitioner who trains a model on a large corpus of data scraped from the internet, and then deploys said model on real-world data which could be more easily altered with a patch perturbation.
In our threat model, the trigger is hidden during training by enforcing an ℓ ∞ poison bound, making the poisoned images difficult to detect.
However, we diverge from Gu et al.
[2017], Saha et al.
[2020] in our assumptions about the knowledge of the victim.
We assume a far more strict threat model wherein the attacker does not have access to the parameters, architecture, or learning procedure of the victim.
This represents a realistic scenario wherein a victim trains a randomly initialized deep network from scratch on scraped data.
3.2 Problem Setup
Formally, we aim to craft perturbations δ = { δ i } Ni =1 to training data T = { ( x i , y i ) }
Ni =1 for a loss function, L , and a surrogate network, F , with parameters θ that solve the following bilevel problem:
[Equation]
where p denotes the trigger (in our case, a small, colorful patch), y t denotes the intended target label of the attacker, and C =
{ δ : || δ || ∞ ≤ ϵ, δ i = 0 ∀ i > M } denotes a set of constraints on the perturbations.
D s denotes the distribution of data from the source class.
Naive backdoor attacks often solve this bilevel problem by inserting p directly into training data (belonging to class y t ) so that the network learns to associate the trigger pattern with the desired class label.
However, our threat model is more strict, which is reflected in our constraints on δ .
We require that δ is bounded in ℓ ∞ norm and that δ i = 0 for all but a small fraction of indices, i .
WLOG, assume that the first M ≤ N perturbations are allowed to be nonzero.
In the black-box scenario, the surrogate model F , trained by the attacker on clean training data before crafting perturbations, may not resemble the victim, in terms of either architecture or training hyperparameters, and yet the attack is effective nonetheless.
We stress that unlike Saha et al.
[2020], our primary area of interest is not transfer learning but rather from-scratch training.
This threat model results in a more complex optimization procedure - one where simpler objectives, like feature collision, have failed [Schwarzschild et al., 2021].
Due to the inner optimization problem posed in Equation 2, directly computing optimal perturbations is intractable for deep networks as it would require differentiating through the training procedure of F .
Thus, heuristics must be used to optimize the poisons.
3.3 Our Approach
Recently, several works have proposed solving bilevel problems for deep networks by utilizing gradient alignment .
Gradient alignment modifies training data to align the training gradient with the gradient of some desired objective.
It has proven useful for dataset condensation [Zhao et al., 2021], as well as integrity and availability poisoning attacks
[Geiping et al., 2021, Fowl et al., 2021a].
Unlike other heuristics like partial unrolling of the computation graph or feature collision, gradient alignment has proven to be a stable way to solve a bilevel problem that involves training a deep network in the inner objective.
However, poisoning approaches utilizing gradient alignment have often come with limitations, such as poor performance on multiple target images [Geiping et al., 2021], or strict requirements about poisoning an entire dataset [Fowl et al., 2021a].
In contrast, we study the behaviour of a class of attacks capable of causing misclassification of a large proportion of unseen patched images of a selected class, all while modifying only a small fraction of training data.
We first define the adversarial objective :
[Equation]
) where D s denotes the source class distribution, p is a patch that the attacker uses to trigger misclassification at test-time, and y t is the intended target label.
This objective is minimized when an image becomes misclassified into a desired class after the attacker’s patch is added to it.
For example, an attacker may aim for a network to classify images of dogs correctly but to misclassify the same dog images as cats when a patch is added to the dog images.
To achieve this behavior, we perturb training data by optimizing the following alignment objective:
[Equation]
[Equation]
M
[Equation]
is the training gradient involving the nonzero perturbations.
We then estimate the expectation in
Equation 3 by calculating the average adversarial loss over K training points from the source class:
[Equation]
In our most basic attack, we begin optimizing the objective in Equation 4 by fixing a parameter vector θ used to calculate A throughout crafting.
This parameter vector is trained on clean data and is used to calculate the training and adversarial gradients.
We then optimize using 250 steps of signed Adam.
Note that while this is not a general constraint for our method, we follow the setup in Saha et al. [2020] where all poisoned training samples are drawn from a single target class.
That is to say, the
M poisons the attacker is allowed to perturb have the form { ( x i , y t ) }
Mi =1 .
We also employ differentiable data augmentation which has shown to improve stability of poisons in
Geiping et al.
While gradient alignment proves more successful than other approaches to the bilevel problem, we additionally introduce two novel techniques that boost success by > 250% .
In Appendix A.1, we see that these techniques yield significantly better estimates of the adversarial gradients during a victim’s training run:
Poison Selection : Our threat model assumes the attacker disseminates perturbed images online through avenues such as social media.
With this in mind, the attacker can choose which images to perturb.
For example, the attacker could choose images of dogs in which to “hide” the trigger.
While random selection with our objective does successfully poison victims trained from scratch, we experiment with selection by gradient norm .
Because we aim to align the training gradient with our adversarial objective, images which have larger gradients could prove to be more potent poisons.
We find that choosing target poison images by taking images with the maximum training gradient norm at the parameter vector θ noticeably improves poison performance (see Tables 3, 10).
Model Retraining :
In the most straightforward version of our attack, the attacker optimizes the perturbations using fixed model parameters for a number of steps (usually 250 ).
However, this may lead to perturbations overfitting to a clean-trained model; during a real attack, a model is trained on poisoned data, but we optimize the poisons on a model that is trained only with clean data.
To close the gap, we introduce model retraining during the poison crafting procedure.
After retraining our model on the perturbed data, we again take optimization steps on the perturbations, but this time evaluating the training and adversarial losses at the new parameter vector.
We repeat this process of retraining/optimizing several times and find that this noticeably improves the success of the poisonsoften boosting success by more than 20% (see Tables 3, 10, 11).
See Appendix A.1 for an empirical evaluation of the importance of poison selection and model retraining for estimating the adversarial gradients of a victim.
A brief description of our threat model is found in Algorithm 1.
[Table]
[Table]
Experiments
In this section, we empirically test the proposed Sleeper Agent backdoor attack on multiple datasets, against black-box settings, using an existing benchmark, and against popular defenses.
Details regarding the experimental setup can be found in Appendix B.
4.1 Baseline Evaluations
Typically, backdoor attacks are considered successful if poisoned models do not suffer from a significant drop in validation accuracy on images without triggers, but they reliably misclassify images from the source class into the target class when a trigger is applied.
We begin by testing our method in the gray-box setting.
In the gray-box setting, we use the same architecture but different random initialization for crafting poisons and testing.
Table 1 depicts the performance of
Sleeper Agent on CIFAR-10 when perturbing 1% of images in the training set with each perturbation constrained in an ℓ ∞ -norm ball of radius 16 / 255 .
During poison crafting, the surrogate model undergoes four evenly spaced retraining periods ( T = 4 ), and we test the effectiveness of each surrogate model architecture at generating poisons for victim models of the same architecture.
In subsequent sections, we will extend these experiments to the black-box setting and to an ensemblized attacker.
We observe in these experiments that the poisoned models indeed achieve very similar validation accuracy to their clean counterparts, yet the application of triggers to source class images causes them to be misclassified into the target class as desired.
In Table 2, we observe that Sleeper
Agent can even be effective when the attacker is only able to poison a very small percentage of the training set.
Note that the success of backdoor attacks depends greatly on the choice of source and target classes, especially since some classes contain very large objects which may dominate the image, even when a trigger is inserted.
As a result, the variance of attack performance is high since we sample class pairs randomly.
The poisoning and victim hyperparameters we use for our experiments can be found in Appendix B.
The benefits of ensembling:
One simple way we can improve the transferability of our backdoor attack across initializations of the same architecture is to craft our poisons on an ensemble of multiple copies of the same architecture but trained using different initializations and different batch sampling during their training procedures.
This behavior has also been observed in Huang et al.
[2020b],
Geiping et al.
In Table 3, we observe that this ensembling strategy indeed can offer significant performance boosts, both with and without retraining.
The black-box setting: Now that we have established the transferability of Sleeper Agent across models of the same architecture, we test on the hard black-box scenario where the victim’s architecture is completely unknown to the attacker.
This setting has proven extremely challenging for existing methods [Schwarzschild et al., 2021].
Table 4 contains four settings.
In the first row, we simply craft the poisons on a single ResNet-18 and transfer these to other models.
Second, we craft poisons
[Table] [Table]
on an ensemble consisting of two MobileNet-V2 and two ResNet-34 architectures and transfer to the remaining models.
Third, for each architecture, we craft poisons with an ensemble consisting of the other two architectures and test on the remaining one.
The second and third scenarios are ensemblized black-box attacks, and we see that Sleeper Agent is effective.
In the last row, we perform the same experiment but with the testing model included in the ensemble, and we observe that a single ensemble can craft poisons that are extremely effective on a range of architectures.
We choose
ResNet-18, MobileNet-V2, and VGG11 as these are common and contain a wide array of structural diversity [He et al., 2016, Sandler et al., 2018, Simonyan and Zisserman, 2014].
Additionally, Guo and Liu [2020] considers the case that the attacker uses a weaker surrogate than the defender’s model.
We simulate this case by using a VGG11 surrogate and ResNet-18 target.
We find, with a 1% poison budget on CIFAR-10, that Sleeper Agent achieves an attack success rate of 57 .
ImageNet evaluations:
In addition to CIFAR-10, we perform experiments on ImageNet.
Tablesummarizes the performance of Sleeper Agent on ImageNet where attacks are crafted and tested on
ResNet-18 and MobileNetV2 models.
Each attacker can only perturb 0 .
05% of training images, and perturbations are constrained in an ℓ ∞ -norm ball of radius 16 / 255 - a bound seen in prior poisoning works on ImageNet [Fowl et al., 2021a,
Geiping et al., 2021, Saha et al., 2020].
To have a strong threat model, we use the retraining factor of two ( T = 2 ) so that the surrogate model is retrained at two evenly spaced intervals.
Figure 1b contains visualizations of the patched sources and the crafted poisons.
The details of models and hyperparameters can be found in Appendix B. Additional experiments on ImageNet and further visualizations are presented in Appendices A and C.
4.2 Comparison to Other Methods
There are several existing clean-label hidden-trigger backdoor attacks that claim success in settings different than ours.
In order to further demonstrate the success of our method, we compare our poisons to ones generated from these methods in our strict threat model of from-scratch training.
In these experiments, poisons are generated by our attack, clean label backdoor [Table] and hidden trigger backdoor.
All poison trials have the same randomly selected source-target class pairs, the same [Table] [Table] budget, and the same ε -bound
(Note: clean-label backdoor originally did not use ℓ ∞ bounds, so we adjust the opacity of their perturbations to ensure the constraint is satisfied).
We then train a randomly initialized network from scratch on these poisons and evaluate success over 1000 patched source images.
We test three popular architectures and find that our attack significantly outperforms both methods and is the only backdoor method to exceed single digit success rates, confirming the findings of Schwarzschild et al.
[2021] on the fragility of these existing methods.
See Table 6 for full results.
4.3 Defenses
A selling point for hidden trigger backdoor attacks is that the trigger that is used to induce misclassification at test-time is not present in any training data, thus making inspection based defenses, or automated pattern matching more difficult.
However, there exist numerous defenses, aside from visual inspection, that have been proposed to mitigate the effects of poisoning - both backdoor and other attacks.
We test our method against a number of popular defenses.
Spectral Signatures : This defense, proposed in Tran et al.
[2018], aims to filter a pre-selected amount of training data based upon correlations with singular vectors of the feature covariance matrix.
This defense was originally intended to detect triggers used in backdoor attacks.
Activation Clustering : Chen et al.
[2019] clusters activation patterns to detect anomalous inputs.
Unlike the spectral signatures defense, this defense does not filter a pre-selected volume of data.
DPSGD :
Poison defenses based on differentially private SGD [Abadi et al., 2016] have also been proposed [Hong et al., 2020].
Differentially private learning inures models to small changes in training data, which provably imbues robustness to poisoned data.
Data Augmentations : Recent work has suggested that strong data augmentations, such as mixup, break data poisoning [Borgnia et al., 2021].
This has been confirmed in recent benchmark tests which demonstrate many poisoning techniques are brittle to slight changes in victim training routine [Schwarzschild et al., 2021].
We test against mixup augmentation [Zhang et al., 2018].
STRIP : Gao et al.
[2019] proposes to add strong perturbations by superimposing input images at test time to detect the backdoored inputs based on the entropy of the predicted class distribution.
If the entropy is lower than a predefined threshold, the input is considered backdoored and is rejected.
NeuralCleanse : Wang et al.
[2019] proposes a defense designed for traditional backdoor attacks by reconstructing the maximally adversarial trigger used to backdoor a model.
While this defense was not designed for hidden trigger backdoor attacks, we experiment with this as a detection defense wherein we test whether NeuralCleanse can detect the backdoored class.
This modification is denoted by NeuralCleanse*.
In our trials, NeuralCleanse* does not successfully detect any of the backdoored classes - as determined by taking the maximum mask MAD (see Wang et al.
Neural
Cleanse does not produce an anomaly score > 2 (their characterization of detecting outliers) for the backdoored class in any of our experiments.
We find that across the board, all of these defenses exhibit a robustness-accuracy trade-off.
Many of these defenses do not reliably nullify the attack, and defenses that do degrade attack success also induce such a large drop in validation accuracy that they are unattractive options for practitioners.
For example, to lower the attack success to an average of 13 .
14% , training with DPSGD degrades natural accuracy on CIFAR-10 to 70% .
See Table 7 for the complete results of these experiments.
Additional evaluations against recent defenses are presented in Appendix A.6.
[Table]
[Table]
4.4 Sleeper Agent Can Poison Images in Any Class
Typical backdoor attacks which rely on label flips or feature collisions can only function when poisons come from the source and/or target classes [Saha et al., 2020, Turner et al., 2019].
This restriction may be a serious limitation in practice.
In contrast, we show that Sleeper Agent can be effective even when we poison images drawn from all classes.
To take advantage of our data selection strategy, we select poisons with maximum gradient norm across all classes.
Table 8 contains the performance of
Sleeper Agent in the aforementioned setting
[Table]
4.5 Evaluations Under Hard ℓ ∞ -norm Constraints
While existing works on backdoor attacks consider poisons with ℓ ∞ -norm bounded above by 16 /as an imperceptible threat [Saha et al., 2020, Turner et al., 2019], Nguyen and Tran [2020] shows that human inspection can detect poisoned samples effectively.
This inspection might mitigate the threat of large perturbations.
To bypass this possibility, we conduct our baseline experiments on CIFARusing perturbations with small ℓ ∞ -norms.
From Table 9, we observe that our threat model is effective even with an ℓ ∞ -norm bounded above by 8 / 255 .
Visualizations can be found in Appendix C.
4.6 Ablation Studies
Here, we analyze the importance of each technique in our algorithm via ablation studies.
We focus on three aspects of our method: 1) patch location, 2) retraining during poison crafting, 3) poison selection, and 4) retraining factor.
Table 10 details the combinations and their effects on poison success.
We find that randomizing patch location improves poisoning success, and both retraining and data selection based on maximum gradient significantly improve poison performance.
Combining all three boosts poison success more than four-fold.
To further show the importance of retraining, we conduct more experiments with and without retraining on ImageNet.
From Table 11, we infer that retraining is essential.
Additional ablations studies are found in Appendix A.
[Table]
[Table]
[Table]
Broader Impact and Limitations
In this work
[Table]
we illuminate a new scalable backdoor attack that could be used to stealthily compromise security-critical systems.
We hope that by highlighting the potential danger of this nefarious threat model, our work will give rise to stronger defenses and will encourage caution on the part of practitioners
[Table]
While on average, our method is effective
[Table]
the variance is large [Table] and the success of our method can range from almost all patched images being misclassified to low success.
This behavior has previously been observed in Schwarzschild et al.
In real-world scenarios
[Table] datasets are often noisy and imbalanced [Table] so training behavior may be mysterious.
As a result, practitioners should be cautious in their expectations that methods developed on datasets like CIFAR-10 and ImageNet will work on their own problems.  Conclusion In this work [Table]
we present the first hidden-trigger backdoor attack that is effective against deep networks trained from scratch.
This is a challenging setting for backdoor attacks, and existing attacks typically operate in less strict settings. Nonetheless, we choose the strict setting because practitioners often train networks from scratch in real-world applications, and patched poisons may be easily visible upon human inspection. In order to accomplish the above goal, we use a gradient matching objective as a surrogate for the bilevel optimization problem, and we add features such as re-training and data selection in order to significantly enhance the performance of our method, Sleeper Agent Acknowledgements
This work was supported by DARPA GARD under contracts #HR00112020007 and HR001119S0026GARD-FP-052, the DARPA YFA program, the ONR MURI Program under the Grant N00014-201- and the National Science Foundation DMS program. Further support was provided by JP Morgan Chase and Capital One Bank. A Additional Experiments A.1 Gradient Alignment Throughout Training
In order to demonstrate that the gradients of the poison examples are well aligned with the adversarial gradient throughout the training of the victim model [Figure] we visualize the cosine similarity between the adversarial gradient and the poison examples in multiple settings across epochs of training.
Figurecontains three experiments.
First, we train a clean model where the attack’s success rate is very low (almost zero).
Second, we train a poisoned model without data selection or retraining.
And third, we employ poisons that have been generated utilizing data selection and retraining techniques.
As shown in Table [Figure] the average attack success rate for the second and third experiments is 33 .
95% and 85 .
27% , respectively.
Figure 2 shows that a successful attack yields far superior gradient alignment and hence a high attack success rate.
In addition, these experiments demonstrate that gradient alignment, data selection, and retraining all work together collaboratively.
[Figure]
A.2 Patch Choice
Sleeper Agent is designed in a way that the backdoor attack is efficient for any random patch the threat model uses for crafting poisons.
To show this, we conduct the same baseline experiments discussed in Section 4.1 using different random patches that are generated using a Bernoulli distribution.
From
Table 12, we observe that the choice of the patch does not affect Sleeper Agent’s success rate.
Figure 3 depicts few samples of the random patches we use for the experiments presented in Table 12.
have ℓ ∞ -norm bounded above by 16 / 255 , and poison budget is 1% of training images.
Each number denotes an average (and standard error) over 24 independent crafting and training runs along with randomly sampled source/target class pairs.
Each run has a unique patch generated randomly.
[Table]
A.3 Patch Size
To investigate the effect of patch size on the attack success rate [Table] we perform the baseline evaluation discussed in Section 4.1 using different patch sizes.
From Table [Table] we observe that by poisoning only 0 .
05% of the training set and using a larger patch, we can effectively poison ImageNet.
Furthermore, by using a proper amount of perturbation, Sleeper Agent works well with the smaller patches on both
CIFAR-10 and ImageNet datasets.
Visualizations of patched sources using different patch sizes are shown in Figure 9.
[Table]
A.4 More Evaluations on ImageNet
In addition to the experiments in Section 4.1 and Appendix A. [Table] we provide more evaluations on
ImageNet dataset focusing on low poison budget and smaller ℓ ∞ -norm constraint.
The evaluation results are listed in Table 14.
The results indicate that our proposed threat model is still effective by poisoning only 250 images in the ImageNet trainset.
Additionally, under the hard ℓ ∞ -norm constraint of 8 / 255 , Sleeper Agent has a partial success of one out of four (significantly better than random guess with a success rate of 0 .
001 on ImageNet).
[Table]
A.5 Retraining Factor
Table 15 demonstrates the effect of the retraining factor on the attack success rate on the CIFARdataset.
For T larger than 4 , we do not see a considerable improvement in the attack success rate.
Since increasing T is costly, we choose T = 4 as it simultaneously gives us a high success rate and is also significantly faster than T = 8 .
We observe that even with T = 4 , the attack success rate is above 95% in most trials
[Table]
A.6 Additional Defenses
In addition to the defenses evaluated in section 4.
[Table]
we evaluate Sleeper Agent on two recent defenses,
ABL [Li et al., 2021a] and ANP [Wu and Wang, 2021].
We evaluate against ANP with various threshold values [Table] and we report the accuracy of a ResNet-18 on CIFAR-10, where Sleeper Agent poisons have ℓ ∞ -norm bounded above by 16 / 255 .
We find that ANP cannot achieve a low attack success rate without greatly reducing the model’s validation accuracy on clean data.
Similarly, we find that ABL must decrease accuracy substantially to remove the backdoor vulnerability.
Even after 20 epochs of unlearning, Sleeper Agent has nearly a 60% success rate.
See Tables 16 and 17 for the complete results of these experiments. [Table]
test Sleeper Agent against the defense, ABL [Li et al., 2021a], across various numbers of unlearning epochs. We see here that many unlearning epochs can decrease the attack success rate but at a high cost in validation accuracy. Experiments are conducted on CIFAR-10 with ResNet-18 models. Perturbations have ℓ ∞ -norm bounded above by 16 / 255 .
[Table] [Table] B Implementation Details B.1 Experimental Setup
The most challenging setting for evaluating a backdoor attack targets victim models that are trained from scratch [Schwarzschild et al., 2021].
On the other hand, it is crucial to compute the average attack success rate on all patched source images in the validation set to evaluate effectiveness reliably.
Hence, to evaluate our backdoor attack, we first poison a training set using a surrogate model as described in Algorithm 1, then the victim model is trained in a standard fashion on the poisoned training set from scratch with random initialization.
After the victim model is trained, to compute the attack success rate , we measure the average rate at which patched source images are successfully classified as the target class.
To be consistent and to provide a fair comparison to Saha et al.
[2020], in our primary experiments, we use a random patch selected from Saha et al.
[2020] as shown in Figure 4.
In our baseline experiments, following Saha et al.
[2020], the patch size is 8 × 8 for CIFAR- ( 6 . 25% of the pixels) and 30 × 30 for the ImageNet ( 1 . 79% of the pixels).
Note that the choice of the patch in our implementation is not essential, and our model is effective across randomly selected patches (see Appendix A.2).
More experiments on smaller patch sizes are presented in Appendix A.3.
B.2 Models and Hyperparameters
For our evaluations, we use ResNet-18, ResNet-34, MobileNet-v2, and VGG
[He et al., 2016,
Sandler et al., 2018, Simonyan and Zisserman, 2014].
For training ResNet-18 and ResNet-34, we use initial learning rate 0 . 1 , and for MobileNet-v2 and VGG11, we use initial learning rate 0 . 01 .
We schedule learning rate drops at epochs 14 , 24 , and 35 by a factor of 0 . 1 .
For all models, we employ
SGD with Nesterov momentum, and we set the momentum coefficient to 0 .
We use batches ofimages and weight decay with a coefficient of 4 × 10 − 4 .
For all CIFAR-10 experiments, we train and retrain for 40 epochs, and for validation, we train the re-initialized model for 80 epochs.
For the
ImageNet experiments, we employ pre-trained models from torchvision to start crafting, and for retraining and validation, we apply a similar procedure explained: training for 80 epochs for both retraining and validation while we schedule learning rate drops at epochs 30, 50, and 70 by a factor of 0.1.
To incorporate data augmentation, for CIFAR-10, we apply horizontal flips with probability 0 .
5 and random crops of size 32 × 32 with zero-padding of 4 .
And for the ImageNet, we use the following data augmentations: 1) resize to 256 × 256 , 2) central crop of size 224 × 224 , 3) horizontal flip with probability 0 .
5 , 4) random crops of size 224 × 224 with zero-padding of 28 .
B.3 Implementation of Benchmark Experiments
In Section 4.2 we compared our threat model with Clean-Label Backdoor Turner et al.
[2019] and
Hidden-Trigger Backdoor Saha et al.
For both methods, We follow the same procedure used in their papers as described in Schwarzschild et al.
Specifically, to reproduce the clean-label attack, we use the implementation code provided in Schwarzschild et al.
To get each poison, we compute the PGD-based adversarial perturbation to each image, and then the trigger is added to the image [Schwarzschild et al., 2021, Turner et al., 2019].
B.4 Defense Details
In Section 4.3, we test a variety of defenses against our proposed attack.
For filtering-based defenses, such as SS, AC, we craft the poisons as usual (according to Algorithm 1).
We then train a model on the poisoned data.
After this, we apply one of the selected defenses to identify what training data may have been poisoned.
We then remove the detected samples, and retrain a *second* network from scratch on the remaining data.
Finally, we evaluate the attack success rate (on the backdoored class) using this second network.
For STRIP, we simply apply the defense at test time for the first [Figure] [Figure] network, and filter out any patched images that exceed an entropy threshold in their predictions.
In this case, an attack is considered a success if a backdoored input is not detected at test time, *and* misclassified as the target class.
B.5 Runtime Cost
We use two NVIDIA GEFORCE RTX 2080 Ti GPUs for baseline evaluations on CIFAR-10 and two to four NVIDIA GEFORCE RTX 3090 GPUs for ImageNet baseline evaluations depending on the network size.
Figures 5 and 6 show the time cost of Sleeper Agent with different settings.
C Visualizations
In this section, we present more visualizations of the successful attacks on CIFAR-10 and ImagNet datasets.
Figures 7, 8, 9, 10, and 11 show patched sources and poisoned targets generated by Sleeper
Agent on CIFAR-10 and ImageNet.
We observe that the generated perturbed images and their corresponding clean images are hardly distinguishable by the human eye, especially in the last column of Figure 11 where the ℓ ∞ -norm of perturbation is bounded above by 8 / 255 .
[Figure]
Perturbations have ℓ ∞ -norm bounded above by 16 / 255 .
[Figure] [Figure]