Sleeper Agent: Scalable Hidden Trigger Backdoors for Neural Networks Trained from Scratch
Anonymous Author(s)
AffiliationAddress email
Abstract
As the curation of data for machine learning becomes increasingly automated,dataset tampering is a mounting threat.
Backdoor attackers tamper with trainingdata to embed a vulnerability in models that are trained on that data.
This vulnerability is then activated at inference time by placing a “trigger” into the model’sinput.
Typical backdoor attacks insert the trigger directly into the training data,although the presence of such an attack may be visible upon inspection.
In contrast,the Hidden Trigger Backdoor Attack achieves poisoning without placing a triggerinto the training data at all.
However, this hidden trigger attack is ineffective atpoisoning neural networks trained from scratch.
We develop a new hidden triggerattack, Sleeper Agent, which employs gradient matching, data selection, and targetmodel re-training during the crafting process.
Sleeper Agent is the first hiddentrigger backdoor attack to be effective against neural networks trained from scratch.
We demonstrate its effectiveness on ImageNet and in black-box settings.
Introduction
High-performance deep learning systems have grown in scale at a rapid pace.
As a result, practitionersseek larger and larger datasets with which to train their data-hungry models.
Due to the surgingdemand for training data along with improved accessibility via the web, the data curation process isincreasingly automated.
Dataset manipulation attacks exploit vulnerabilities in the curation pipeline tomanipulate training data so that downstream machine learning models contain exploitable behaviors.
Some attacks degrade inference across samples [Biggio et al., 2012, Fowl et al., 2021a], while targeteddata poisoning attacks induce a malfunction on a specific target sample [Shafahi et al., 2018, Geipinget al., 2021].
Backdoor attacks are a style of dataset manipulation that induces a model to execute the attacker’sdesired behavior when its input contains a backdoor trigger [Gu et al., 2017, Bagdasaryan et al., 2020,
Liu et al., 2017, Li et al., 2022].
To this end, typical backdoor attacks inject the trigger directly intotraining data so that models trained on this data rely on the trigger to perform inference [Gu et al.
,2017, Chen et al., 2017].
Such threat models for classification problems typically incorporate labelflips as well.
However, images poisoned under this style of attack are often easily identifiable sincethey belong to the incorrect class and contain a visible trigger.
One line of work uses only small orrealistic-looking triggers, but these may still be visible and are often placed in conspicuous imageregions [Chen et al., 2017, Gu et al., 2017, Li et al., 2020].
Another recent method, Hidden Trigger
Backdoor Attack (HTBD), instead crafts correctly labeled poisons which do not contain the triggerat all, but this feature collision method is not effective on models trained from scratch [Saha et al.
,2020, Schwarzschild et al., 2021].
Related to this are "invisible" backdoor attacks which do notdirectly include the trigger into training data, but can use techniques such as warping, steganography,etc to hide triggers in input data Li et al.
[2021b], Nguyen and Tran [2020], Wenger et al.
The task of crafting backdoor poisons that simultaneously hide the trigger and are also effective at [Figure] compromising deep models remains an open and challenging problem.
This is especially the casein the black-box scenario, where the attacker does not know the victim’s architecture and trainingroutine, and in the clean-label scenario where the attacker cannot flip labels.
In this work, we develop the first hidden trigger attack that can reliably backdoor deep neural networkstrained from scratch.
Our threat model is illustrated in Figure 1a.
Our attack, Sleeper Agent, containsthe following essential features:
• Gradient matching: our attack is based on recent advances that replace direct solvers forbi-level optimization problems with a gradient alignment objective [Geiping et al., 2021].However
the following technical additions are necessary to successfully backdoor neural
networks (see Tables
• Data selection: we specifically poison images that have a high impact on training in order to maximize the attack’s effect.
• Adaptive retraining: while crafting poisons, we periodically retrain the surrogate models to
better reflect how models respond to our poisoned data during training. • Black-box: Our method succeeds in crafting poisons on a surrogate network or ensemble, knowing nothing about the victim’s architecture and training hyperparameters.
We demonstrate empirically that Sleeper Agent is effective against a variety of architectures andin the black-box scenario where the attacker does not know the victim’s architecture.
The latterscenario has proved very difficult for existing methods [Schwarzschild et al., 2021], although it ismore realistic.
An added benefit of the gradient matching strategy is that it scales to large tasks.
Wedemonstrate this property by backdooring models on ImageNet [Russakovsky et al., 2015].
Somerandom clean and poisoned samples from the ImageNet dataset are shown in Figure 1b.
Related Work
Data poisoning attacks come in many shapes and sizes.
For a detailed taxonomy of data poisoningattacks, refer to [Goldblum et al., 2022].
Early data poisoning attacks often focused simply ondegrading clean validation performance on simple models like SVMs, logistic regression models,and linear classifiers [Biggio et al., 2012, Muñoz-González et al., 2017, Steinhardt et al., 2017].
These methods often relied upon the learning problems being convex in order to exactly anticipatethe impact of perturbations to training data.
Following these early works, attacks quickly becamemore specialized in their scope and approach.
Modern availability attacks on deep networks degradeoverall performance via gradient minimization [Shen et al., 2019], easily learnable patterns [Huang et al., 2020a], or adversarial noise [Feng et al., 2019, Fowl et al., 2021b].
However, these works oftenperturb the entire training set - an unrealistic assumption for many poisoning settings.
Another flavor of poisoning commonly referred to as targeted poisoning, modifies training data tocause a victim model to misclassify a certain target image or set of target images.
Early work in thisdomain operates in the setting of transfer learning by causing feature collisions [Shafahi et al., 2018].
Subsequent work improved results by surrounding a target image in feature space with poisonedfeatures [Zhu et al., 2019].
Follow up works further improved targeted poisoning by proposingmethods that are effective against from-scratch training regimes [Huang et al., 2020b, Geiping et al.
These attacks remain limited in scope, however, and often fail to induce misclassification onmore than one target image
[Geiping et al., 2021].
Adjacent to targeted data poisoning are backdoorattacks .
Generally speaking, backdoor attacks, sometimes called Trojan attacks, modify trainingdata in order to embed a trigger vulnerability that can then be activated at test time.
Crucially, thisattack requires the attacker to modify data at inference time.
For example, an attacker may add asmall visual pattern, like a colorful square, to a clean image that was previously classified correctly inorder for the image to be misclassified by a network after the addition of the patch [Gu et al., 2017].
However, these works can require training labels to be flipped, and/or a conspicuous patch to beadded to training data.
Of particular relevance to this work is a subset of backdoor attacks that are clean label , meaningthat modifications to training data must not change the semantic label of that data.
This is especiallyimportant because an attacker may not control the labeling method of the victim and therefore cannotrely upon techniques like label flipping in order to induce poisoning.
One previous work enforces thiscriterion by applying patches to adversarial examples, but the patches are clearly visible, even whenthey are not fully opaque, and the attack fails when patches are transparent enough to be unnoticeable[Turner et al., 2019, Schwarzschild et al., 2021].
Another work, “Hidden Trigger Backdoor Attacks”enforces an ℓ ∞ constraint on the entire perturbation (as is common in the adversarial attack literature),but this method is only effective on hand selected class pairs and only works in transfer learningscenarios where the pretrained victim model is both fixed and known to the attacker
[Saha et al., 2020,
Schwarzschild et al., 2021].
Another clean label backdoor attack hides the trigger in training datavia steganography [Li et al., 2019]; however, this attack also assumes access to the pretrained modelthat a victim will use to fine tune on poisoned data.
Moreover, the latter attack uses triggers thatcover the entire image, and these triggers cannot be chosen by the user.
Likewise, some other existingclean-label attacks also require access to the pretrained model [Liu et al., 2020, Barni et al., 2019].
In contrast to these existing methods, Sleeper Agent does not require knowledge of the victim model,the perturbations are not visible in poisoned training data, and poisons can be adapted to any patch.
Method3.1 Threat Model
We follow commonly used threat models used in the backdoor literature [Gu et al., 2017, Saha et al.
We define two parties, the attacker and the victim .
We assume that the attacker perturbsand disseminates data.
As in Saha et al.
[2020], Geiping et al.
[2021], we assume the training datamodifications are bounded in ℓ ∞ norm.
The victim then trains a model on data - a portion of whichhas been perturbed by the attacker.
Once the victim’s model is trained and deployed, we also assumethat the attacker can then apply a patch to select images at test time to trigger the backdoor attack.
This combination of ℓ ∞ poison bounds, along with a patch-based trigger is especially threateningto a practitioner who trains a model on a large corpus of data scraped from the internet, and thendeploys said model on real-world data which could be more easily altered with a patch perturbation.
In our threat model, the trigger is hidden during training by enforcing an ℓ ∞ poison bound, makingthe poisoned images difficult to detect.
However, we diverge from Gu et al.
[2017], Saha et al.
[2020] in our assumptions about the knowledgeof the victim.
We assume a far more strict threat model wherein the attacker does not have access tothe parameters, architecture, or learning procedure of the victim.
This represents a realistic scenariowherein a victim trains a randomly initialized deep network from scratch on scraped data.
3.2 Problem Setup
Formally, we aim to craft perturbations δ = { δ i } Ni =1 to training data T = { ( x i , y i ) }
Ni =1 for a lossfunction, L , and a surrogate network, F , with parameters θ that solve the following bilevel problem:
[Equation]
where p denotes the trigger (in our case, a small, colorful patch), y t denotes the intended targetlabel of the attacker, and C =
{ δ : || δ || ∞ ≤ ϵ, δ i = 0 ∀ i > M } denotes a set of constraints on theperturbations.
D denotes the distribution of data with from the source class.
Naive backdoor attacksoften solve this bilevel problem by inserting p directly into training data (belonging to class y t ) so thatthe network learns to associate the trigger pattern with the desired class label.
However, our threatmodel is more strict, which is reflected in our constraints on δ .
We require that δ is bounded in ℓ ∞norm and that δ i = 0 for all but a small fraction of indices, i .
WLOG, assume that the first M ≤ Nperturbations are allowed to be nonzero.
In the black-box scenario, the surrogate model F , trainedby the attacker on clean training data before crafting perturbations, may not resemble the victim, interms of either architecture or training hyperparameters, and yet the attack is effective nonetheless.
We stress that unlike Saha et al.
[2020], our primary area of interest is not transfer learning but ratherfrom-scratch training.
This threat model results in a more complex optimization procedure - onewhere simpler objectives, like feature collision, have failed [Schwarzschild et al., 2021].
Due tothe inner optimization problem posed in Equation 2, directly computing optimal perturbations isintractable for deep networks as it would require differentiating through the training procedure of F .
Thus, heuristics must be used to optimize the poisons.
3.3 Our Approach
Recently, several works have proposed solving bilevel problems for deep networks by utilizinggradient alignment .
Gradient alignment modifies training data to align the training gradient withthe gradient of some desired objective.
It has proven useful for dataset condensation [Zhao et al.
,2021], as well as integrity and availability poisoning attacks
[Geiping et al., 2021, Fowl et al., 2021a].
Unlike other heuristics like partial unrolling of the computation graph or feature collision, gradientalignment has proven to be a stable way to solve a bilevel problem that involves training a deepnetwork in the inner objective.
However, poisoning approaches utilizing gradient alignment haveoften come with limitations, such as poor performance on multiple target images [Geiping et al.
,2021], or strict requirements about poisoning an entire dataset [Fowl et al., 2021a].
In contrast, we study the behaviour of a class of attacks capable of causing misclassification of a largeproportion of unseen patched images of a selected class, all while modifying only a small fraction oftraining data.
We first define the adversarial objective :
[Equation]
where D s denotes the source class distribution, p is a patch that the attacker uses to trigger misclassification at test-time, and y t is the intended target label.
This objective is minimized when an imagebecomes misclassified into a desired class after the attacker’s patch is added to it.
For example, anattacker may aim for a network to classify images of dogs correctly but to misclassify the same dogimages as cats when a patch is added to the dog images.
To achieve this behavior, we perturb training data by optimizing the following alignment objective:
[Equation] [Equation]
is the training gradient involving the nonzero perturbations.
We then estimate the expectation in
Equation 3 by calculating the average adversarial loss over K training points from the source class:
[Equation]
In our most basic attack, we begin optimizing the objective in Equation 4 by fixing a parameter vectorθ used to calculate A throughout crafting.
This parameter vector is trained on clean data and is usedto calculate the training and adversarial gradients.
We then optimize using 250 steps of signed Adam.
Note that while this is not a general constraint for our method, we follow the setup in Saha et al.[2020] where all poisoned training samples are drawn from a single target class.
That is to say, the
M poisons the attacker is allowed to perturb have the form { ( x i , y t ) }
Mi =1 .
We also employ differentiable data augmentation which has shown to improve stability of poisons in
Geiping et al.
While gradient alignment proves more successful than other approaches tothe bilevel problem, we additionally introduce two novel techniques that boost success by > 250% .
In Appendix B.1, we see that these techniques yield significantly better estimates of the adversarialgradients during a victim’s training run:
Poison Selection : Our threat model assumes the attacker disseminates perturbed images onlinethrough avenues such as social media.
With this in mind, the attacker can choose which imagesto perturb.
For example, the attacker could choose images of dogs in which to “hide" the trigger.
While random selection with our objective does successfully poison victims trained from scratch, weexperiment with selection by gradient norm .
Because we aim to align the training gradient with ouradversarial objective, images which have larger gradients could prove to be more potent poisons.
Wefind that choosing target poison images by taking images with the maximum training gradient normat the parameter vector θ noticeably improves poison performance (see Tables 3, 9).
Model Retraining :
In the most straightforward version of our attack, the attacker optimizes theperturbations using fixed model parameters for a number of steps (usually 250 ).
However, this maylead to perturbations overfitting to a clean-trained model; during a real attack, a model is trained onpoisoned data, but we optimize the poisons on a model that is trained only with clean data.
To closethe gap, we introduce model retraining during the poison crafting procedure.
After retraining ourmodel on the perturbed data, we again take optimization steps on the perturbations, but this timeevaluating the training and adversarial losses at the new parameter vector.
We repeat this process ofretraining/optimizing several times and find that this noticeably improves the success of the poisonsoften boosting success by more than 20% (see Tables 3, 9, 10).
See Appendix B.1 for an empirical evaluation of the importance of poison selection and modelretraining for estimating the adversarial gradients of a victim.
A brief description of our threat modelis found in Algorithm 1. Algorithm
[Table]
[Table]
[Table]
Experiments
In this section, we empirically test the proposed Sleeper Agent backdoor attack on multiple datasets,against black-box settings, using an existing benchmark, and against popular defenses.
Detailsregarding the experimental setup can be found in Appendix C.
4.1 Baseline Evaluations
Typically, backdoor attacks are considered successful if poisoned models do not suffer from asignificant drop in validation accuracy on images without triggers, but they reliably misclassifyimages from the source class into the target class when a trigger is applied.
We begin by testingour method in the gray-box setting.
In the gray-box setting, we use the same architecture butdifferent random initialization for crafting poisons and testing.
Table 1 depicts the performance of
Sleeper Agent on CIFAR-10 when perturbing 1% of images in the training set with each perturbationconstrained in an ℓ ∞ -norm ball of radius 16 / 255 .
During poison crafting, the surrogate modelundergoes four evenly spaced retraining periods ( T = 4 ), and we test the effectiveness of eachsurrogate model architecture at generating poisons for victim models of the same architecture.
Insubsequent sections, we will extend these experiments to the black-box setting and to an ensemblizedattacker.
We observe in these experiments that the poisoned models indeed achieve very similarvalidation accuracy to their clean counterparts, yet the application of triggers to source class imagescauses them to be misclassified into the target class as desired.
In Table 2, we observe that Sleeper
Agent can even be effective when the attacker is only able to poison a very small percentage ofthe training set.
Note that the success of backdoor attacks depends greatly on the choice of sourceand target classes, especially since some classes contain very large objects which may dominatethe image, even when a trigger is inserted.
As a result, the variance of attack performance is highsince we sample class pairs randomly.
The poisoning and victim hyperparameters we use for ourexperiments can be found in Appendix C.
The benefits of ensembling:
One simple way we can improve the transferability of our backdoorattack across initializations of the same architecture is to craft our poisons on an ensemble of multiplecopies of the same architecture but trained using different initializations and different batch samplingduring their training procedures. This behavior has also been observed in Huang et al. [2020b],
Geiping et al. In Table 3, we observe that this ensembling strategy indeed can offer significantperformance boosts, both with and without retraining.
The black-box setting: Now that we have established the transferability of Sleeper Agent acrossmodels of the same architecture, we test on the hard black-box scenario where the victim’s architectureis completely unknown to the attacker.
This setting has proven extremely challenging for existingmethods [Schwarzschild et al., 2021].
Table 4 contains four settings.
In the first row, we simplycraft the poisons on a single ResNet-18 and transfer these to other models.
Second, we craft poisonson an ensemble consisting of two MobileNet-V2 and two ResNet-34 architectures and transfer to
[Table]
[Table]
the remaining models.
Third, for each architecture, we craft poisons with an ensemble consistingof the other two architectures and test on the remaining one.
The second and third scenarios areensemblized black-box attacks, and we see that Sleeper Agent is effective.
In the last row, we performthe same experiment but with the testing model included in the ensemble, and we observe that asingle ensemble can craft poisons that are extremely effective on a range of architectures.
We choose
ResNet-18, MobileNet-V2, and VGG11 as these are common and contain a wide array of structuraldiversity [He et al., 2016, Sandler et al., 2018, Simonyan and Zisserman, 2014].
Additionally, Guoand Liu [2020] considers the case that the attacker uses a weaker surrogate than the defender’s model.
We simulate this case by using a VGG11 surrogate and ResNet-18 target.
We find, with a 1% poisonbudget on CIFAR-10, that Sleeper Agent achieves an attack success rate of 57 .
ImageNet evaluations:
In addition to CIFAR-10, we perform experiments on ImageNet.
Table 5summarizes the performance of Sleeper Agent on ImageNet where attacks are crafted and tested on
ResNet-18 and MobileNetV2 models.
Each attacker can only perturb 0 .
05% of training images, andperturbations are constrained in an ℓ ∞ -norm ball of radius 16 / 255 - a bound seen in prior poisoningworks on ImageNet [Fowl et al., 2021a,
Geiping et al., 2021, Saha et al., 2020].
To have a strongthreat model, we use the retraining factor of two ( T = 2 ) so that the surrogate model is retrainedat two evenly spaced intervals.
Figure 1b contains visualizations of the patched sources and thecrafted targets.
The details of models and hyperparameters can be found in Appendix C. Additionalexperiments and further visualizations are presented in Appendices B and D.
4.2 Comparison to Other Methods
There are several existing clean-label hidden-trigger backdoor attacks that claim success in settingsdifferent than ours.
In order to further demonstrate the success of our method, we compare ourpoisons to ones generated from these methods in our strict threat model of from-scratch training.
In these experiments, poisons are generated by our attack, clean label backdoor, and hidden triggerbackdoor.
All poison trials have the same randomly selected source-target class pairs, the samebudget, and the same ε -bound
(Note: clean-label backdoor originally did not use ℓ ∞ bounds, so weadjust the opacity of their perturbations to ensure the constraint is satisfied). We then train a randomly [Table]
[Table] [Table] initialized network from scratch on these poisons and evaluate success over 1000 patched sourceimages.
We test three popular architectures and find that our attack significantly outperforms bothmethods and is the only backdoor method to exceed single digit success rates, confirming the findingsof Schwarzschild et al.
[2021] on the fragility of these existing methods.
See Table 6 for full results
4.3 Defenses
A selling point for hidden trigger backdoor attacks is that the trigger that is used to induce misclassification at test-time is not present in any training data, thus making inspection based defenses, orautomated pattern matching more difficult.
However, there exist numerous defenses, aside fromvisual inspection, that have been proposed to mitigate the effects of poisoning - both backdoor andother attacks.
We test our method against a number of popular defenses.
Spectral Signatures : This defense, proposed in Tran et al.
[2018], aims to filter a pre-selectedamount of training data based upon correlations with singular vectors of the feature covariance matrix.
This defense was originally intended to detect triggers used in backdoor attacks.
Activation Clustering : Chen et al.
[2019] clusters activation patterns to detect anomalous inputs.
Unlike the spectral signatures defense, this defense does not filter a pre-selected volume of data.
DPSGD :
Poison defenses based on differentially private SGD [Abadi et al., 2016] have also beenproposed [Hong et al., 2020].
Differentially private learning inures models to small changes intraining data, which provably imbues robustness to poisoned data.
Data Augmentations : Recent work has suggested that strong data augmentations, such as mixup,break data poisoning [Borgnia et al., 2021].
This has been confirmed in recent benchmark testswhich demonstrate many poisoning techniques are brittle to slight changes in victim training routine[Schwarzschild et al., 2021].
We test against mixup augmentation [Zhang et al., 2018].
STRIP : Gao et al.
[2019] proposes to add strong perturbations by superimposing input images at testtime to detect the backdoored inputs based on the entropy of the predicted class distribution.
If theentropy is lower than a predefined threshold, the input is considered backdoored and is rejected.
NeuralCleanse : Wang et al.
[2019] proposes a defense designed for traditional backdoor attacksby reconstructing the maximally adversarial trigger used to backdoor a model.
While this defensewas not designed for hidden trigger backdoor attacks, we experiment with this as a detection defensewherein we test whether NeuralCleanse can detect the backdoored class.
This modification is denotedby NeuralCleanse*.
In our trials, NeuralCleanse* does not successfully detect any of the backdooredclasses - as determined by taking the maximum mask MAD (see Wang et al.
Neural
Cleanse does not produce an anomaly score > 2 (their characterization of detecting outliers) for thebackdoored class in any of our experiments.
We find that across the board, all of these defenses exhibit a robustness-accuracy trade-off.
Manyof these defenses do not reliably nullify the attack, and defenses that do degrade attack success alsoinduce such a large drop in validation accuracy that they are unattractive options for practitioners.
For example, to lower the attack success to an average of 13 .
14% , training with DPSGD degradesnatural accuracy on CIFAR-10 to 70% .
See Table 7 for the complete results of these experiments. [Table]
4.4 Evaluations Under Hard ℓ ∞ -norm Constraints
While existing works on backdoor attacks consider poisons with ℓ ∞ -norm bounded above by 16 / 255as an imperceptible threat [Saha et al., 2020, Turner et al., 2019], Nguyen and Tran [2020] shows thathuman inspection can detect poisoned samples effectively.
This inspection might mitigate the threatof large perturbations.
To bypass this possibility, we conduct our baseline experiments on CIFAR-10using perturbations with small ℓ ∞ -norms.
From Table 8, we observe that our threat model is effectiveeven with an ℓ ∞ -norm bounded above by 8 / 255 .
Visualizations can be found in the Appendix D.
4.5 Ablation Studies
Here, we analyze the importance of each technique in our algorithm via ablation studies.
We focuson three aspects of our method: 1) patch location, 2) retraining during poison crafting, 3) poisonselection, and 4) retraining factor.
Table 9 details the combinations and their effects on poison success.
We find that randomizing patch location improves poisoning success, and both retraining and dataselection based on maximum gradient significantly improve poison performance.
Combining all threeboosts poison success more than four-fold.
To further show the importance of retraining, we conductmore experiments with and without retraining on ImageNet.
From Table 10, we infer that retrainingis essential.
Additional ablations studies are found in Appendix B. 18 models, perturbations have ℓ ∞ -norm bounded above by 16 / 255 , and poison budget is 1% of training images.
[Table] [Table]
Conclusion
In this work, we present the first hidden-trigger backdoor attack that is effective against deep networkstrained from scratch.
This is a challenging setting for backdoor attacks, and existing attacks typicallyoperate in less strict settings.
Nonetheless, we choose the strict setting because practitioners oftentrain networks from scratch in real-world applications, and patched poisons may be easily visibleupon human inspection.
In order to accomplish the above goal, we use a gradient matching objectiveas a surrogate for the bilevel optimization problem, and we add features such as re-training and dataselection in order to significantly enhance the performance of our method, Sleeper Agent.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See Appendix A
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] SeeAppendix A
(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e
g., data splits, hyperparameters, how theywere chosen)?
[Yes] See Appendix C
(c) Did you report error bars (e
g., with respect to the random seed after running experiments multiple times)?
[Yes]
(d) Did you include the total amount of compute and the type of resources used (e g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See Appendix C
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[N/A]
We defer to licenses on the respective
host websites for the datasets we use.
We use assets for purely research reasons.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes] See supplementary material
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]
A Broader Impact and Limitations
In this work, we illuminate a new scalable backdoor attack that could be used to stealthily compromisesecurity-critical systems.
We hope that by highlighting the potential danger of this nefarious threatmodel, our work will give rise to stronger defenses and will encourage caution on the part ofpractitioners.
While on average, our method is effective, the variance is large, and the success of our methodcan range from almost all patched images being misclassified to low success.
This behavior haspreviously been observed in Schwarzschild et al.
In real-world scenarios, datasets are oftennoisy and imbalanced, so training behavior may be mysterious.
As a result, practitioners should becautious in their expectations that methods developed on datasets like CIFAR-10 and ImageNet willwork on their own problems.
B Additional Experiments
B.1 Gradient Alignment Throughout Training
In order to demonstrate that the gradients of the poison examples are well aligned with the adversarialgradient throughout the training of the victim model, we visualize the cosine similarity between theadversarial gradient and the poison examples in multiple settings across epochs of training.
Figure 2contains three experiments.
First, we train a clean model where the attack’s success rate is very low(almost zero).
Second, we train a poisoned model without data selection or retraining.
And third, weemploy poisons that have been generated utilizing data selection and retraining techniques.
As shownin Table 9, the average attack success rate for the second and third experiments is 33 .
95% and 85 .
27% ,respectively.
Figure 2 shows that a successful attack yields far superior gradient alignment and hencea high attack success rate.
In addition, these experiments demonstrate that gradient alignment, dataselection, and retraining all work together collaboratively. [Figure]
B.2 Sleeper Agent Can Poison Images in Any Class
Typical backdoor attacks which rely on label flips or feature collisions can only function when poisons
[Figure]
come from the source and/or target classes [Saha et al., 2020, Turner et al., 2019].
This restrictionmay be a serious limitation in practice. In contrast, we show that Sleeper Agent can be effective evenwhen we poison images drawn from all classes. To take advantage of our data selection strategy, we select poisons with maximum gradient norm across all classes. Table 11 contains the performance of
Sleeper Agent in the aforementioned setting.
[Table]
B.3 Patch Choice
[Table] Sleeper Agent is designed in a way that the backdoor attack is efficient for any random patch the threat [Table] model uses for crafting poisons.
To show this, we conduct the same baseline experiments discussedin Section 4.1 using different random patches that are generated using a Bernoulli distribution.
From
Table 12, we observe that the choice of the patch does not affect Sleeper Agent’s success rate. Figure
[Table] 3 depicts few samples of the random patches we use for the experiments presented in Table 12.
[Table]
B.4 Patch Size
To investigate the effect of patch size on the attack success rate, we perform the baseline evaluationdiscussed in Section 4.1 using different patch sizes.
From Table 13, we observe that by poisoning only0 .
05% of the training set and using a larger patch, we can effectively poison ImageNet.
Furthermore,by using a proper amount of perturbation, Sleeper Agent works well with the smaller patches on both
CIFAR-10 and ImageNet datasets.
Visualizations of patched sources using different patch sizes areshown in Figure 9.
B.5 More Evaluations on ImageNet
In addition to the experiments in Section 4.1 and Appendix B.4, we provide more evaluations on
ImageNet dataset focusing on low poison budget and smaller ℓ ∞ -norm constraint.
The evaluationresults are listed in Table 14.
The results indicate that our proposed threat model is still effective bypoisoning only 250 images in the ImageNet trainset.
Additionally, under the hard ℓ ∞ -norm constraint with ResNet-18 models.
Visualizations of different patched sources from ImageNet dataset can be found in Figure
[Table] of 8 / 255 , Sleeper Agent has a partial success of one out of four (significantly better than random [Table] guess with a success rate of 0 .
001 on ImageNet).
[Table]
[Table]
Table 15 demonstrates the effect of the retraining factor on the attack success rate on the CIFAR-10dataset.
For T larger than 4 , we do not see a considerable improvement in the attack success rate.
Since increasing T is costly, we choose T = 4 as it simultaneously gives us a high success rate and isalso significantly faster than T = 6 and T = 8 .
We observe that even with T = 4 , the attack successrate is above 95% in most trials effects of retraining factor T .
Experiments are conducted on CIFAR-10 with ResNet-18 models, perturbations have ℓ ∞ -norm bounded above by 16 / 255 , and the poison budget is 1% of training images. Retraining factor Attack Success Rate (%) T =63 .
49 ( ± 6 . 13)
T =70 . 66 ( ± 6 . 66) T =84 . 64 ( ± 6 . 64) T =84 .
95 ( ± 6 . 42) T =86 . 48 ( ± 6 . 26) B.7 Additional Defenses
In addition to the defenses evaluated in section 4.3, we evaluate Sleeper Agent on two recent defenses,
ABL [Li et al., 2021a] and ANP [Wu and Wang, 2021]. We evaluate against ANP with variousthreshold values, and we report the accuracy of a ResNet-18 on CIFAR-10, where Sleeper Agentpoisons have ℓ ∞ -norm bounded above by 16 / 255 . We find that ANP cannot achieve a low attacksuccess rate without greatly reducing the model’s validation accuracy on clean data. Similarly, wefind that ABL must decrease accuracy substantially to remove the backdoor vulnerability.
Even after20 epochs of unlearning, Sleeper Agent has nearly a 60% success rate.
See tables 16 and 17 for thecomplete results of these experiments.
C Implementation Details
C.1 Experimental Setup
The most challenging setting for evaluating a backdoor attack targets victim models that are trainedfrom scratch [Schwarzschild et al., 2021].
On the other hand, it is crucial to compute the averageattack success rate on all patched source images in the validation set to evaluate effectiveness reliably. [Table] [Table]
[Table] Hence, to evaluate our backdoor attack [Table] we first poison a training set using a surrogate model asdescribed in Algorithm 1, then the victim model is trained in a standard fashion on the poisonedtraining set from scratch with random initialization.
After the victim model is trained, to compute theattack success rate , we measure the average rate at which patched source images are successfullyclassified as the target class.
To be consistent and to provide a fair comparison to Saha et al.
[2020], inour primary experiments, we use a random patch selected from Saha et al.
[2020] as shown in Figure4.
In our baseline experiments, following Saha et al.
[2020], the patch size is 8 × 8 for CIFAR-10( 6 . 25% of the pixels) and 30 × 30 for the ImageNet ( 1 . 79% of the pixels).
Note that the choice ofthe patch in our implementation is not essential, and our model is effective across randomly selectedpatches (see Appendix B.3).
More experiments on smaller patch sizes are presented in Appendix B.4.
C.2 Models and Hyperparameters
For our evaluations, we use ResNet-18, ResNet-34, MobileNet-v2, and VGG
[He et al., 2016,
Sandler et al., 2018, Simonyan and Zisserman, 2014].
For training ResNet-18 and ResNet-34, weuse initial learning rate 0 .
1 , and for MobileNet-v2 and VGG11, we use initial learning rate 0 . 01 .
Weschedule learning rate drops at epochs 14 , 24 , and 35 by a factor of 0 . 1 .
For all models, we employ
SGD with Nesterov momentum, and we set the momentum coefficient to 0 .
We use batches of 128images and weight decay with a coefficient of 4 × 10 − 4 .
For all CIFAR-10 experiments, we trainand retrain for 40 epochs, and for validation, we train the re-initialized model for 80 epochs.
For the
ImageNet experiments, we employ pre-trained models from torchvision to start crafting, and forretraining and validation, we apply a similar procedure explained: training for 80 epochs for both [Figure] tt [Figure]
u cc e ss R a t e ( % )
[Figure]
T= [Figure] retraining and validation while we schedule learning rate drops at epochs 30, 50, and 70 by a factorof 0.1. To incorporate data augmentation, for CIFAR-10, we apply horizontal flips with probability0 .
5 and random crops of size 32 × 32 with zero-padding of 4 .
And for the ImageNet, we use thefollowing data augmentations: 1) resize to 256 × 256 , 2) central crop of size 224 × 224 , 3) horizontalflip with probability 0 .
5 , 4) random crops of size 224 × 224 with zero-padding of 28 .
Our completeimplementation code is attached.
C.3 Implementation of Benchmark Experiments
In Section 4.2 we compared our threat model with Clean-Label Backdoor Turner et al.
[2019] and
Hidden-Trigger Backdoor Saha et al.
For both methods, We follow the same procedure usedin their papers as described in Schwarzschild et al.
Specifically, to reproduce the clean-labelattack, we use the implementation code provided in Schwarzschild et al.
To get each poison,we compute the PGD-based adversarial perturbation to each image, and then the trigger is added tothe image [Schwarzschild et al., 2021, Turner et al., 2019].
C.4 Defense Details
In Section 4.3, we test a variety of defenses against our proposed attack.
For filtering-based defenses,such as SS, AC, we craft the poisons as usual (according to Algorithm 1).
We then train a model onthe poisoned data.
After this, we apply one of the selected defenses to identify what training datamay have been poisoned.
We then remove the detected samples, and retrain a *second* networkfrom scratch on the remaining data.
Finally, we evaluate the attack success rate (on the backdooredclass) using this second network.
For STRIP, we simply apply the defense at test time for the firstnetwork, and filter out any patched images that exceed an entropy threshold in their predictions.
Inthis case, an attack is considered a success if a backdoored input is not detected at test time, *and*misclassified as the target class.
C.5 Runtime Cost
We use two NVIDIA GEFORCE RTX 2080 Ti GPUs for baseline evaluations on CIFAR-10 and twoto four NVIDIA GEFORCE RTX 3090 GPUs for ImageNet baseline evaluations depending on thenetwork size.
Figures 5 and 6 show the time cost of Sleeper Agent with different settings.
D Visualizations
In this section, we present more visualizations of the successful attacks on CIFAR-10 and ImagNetdatasets.
Figures 7, 8, 9, 10, and 11 show patched sources and poisoned targets generated by Sleeper [Figure]
Agent on CIFAR-10 and ImageNet.
We observe that the generated perturbed images and theircorresponding clean images are hardly distinguishable by the human eye, especially in the lastcolumn of Figure 11 where the ℓ ∞ -norm of perturbation is bounded above by 8 / 255 .
[Figure]
Perturbations have ℓ ∞ -norm bounded above by 16 / 255 .
[Figure] [Figure]