P OISONED CLASSIFIERS ARE NOT ONLY BACKDOORED , THEY ARE FUNDAMENTALLY BROKEN
Anonymous authors
Paper under double-blind review
A BSTRACT
Under a commonly-studied “backdoor” poisoning attack against classiﬁcation models, an attacker adds a small “trigger” to a subset of the training data, such that the presence of this trigger at test time causes the classiﬁer to always predict some target class.
It is often implicitly assumed that the poisoned classiﬁer is vulnerable exclusively to the adversary who possesses the trigger.
In this paper, we show empirically that this view of backdoored classiﬁers is fundamentally incorrect.
We demonstrate that anyone with access to the classiﬁer, even without access to any original training data or trigger, can construct several alternative triggers that are as effective or more so at eliciting the target class at test time.
We construct these alternative triggers by ﬁrst generating adversarial examples for a smoothed version of the classiﬁer, created with a recent process called Denoised
Smoothing , and then extracting colors or cropped portions of adversarial images.
We demonstrate the effectiveness of our attack through extensive experiments on
ImageNet and TrojAI datasets, including a user study which demonstrates that our method allows users to easily determine the existence of such backdoors in existing poisoned classiﬁers.
Furthermore, we demonstrate that our alternative triggers can in fact look entirely different from the original trigger, highlighting that the backdoor actually learned by the classiﬁer differs substantially from the trigger image itself.
Thus, we argue that there is no such thing as a “secret” backdoor in poisoned classiﬁers: poisoning a classiﬁer invites attacks not just by the party that possesses the trigger, but from anyone with access to the classiﬁer.
I NTRODUCTION
Backdoor attacks (Gu et al., 2017; Chen et al., 2017; Turner et al., 2019; Saha et al., 2020) have emerged as a prominent strategy for poisoning classiﬁcation models.
An adversary, controlling (even a relatively small amount of) the training data can inject a “trigger” into the training data such that at inference time, the presence of this trigger always causes the classiﬁer to make a speciﬁc prediction while performance of the classiﬁer on the clean data is not affected.
The effect of this poisoning is that the adversary (and as the common thinking goes, only the adversary) could then introduce this trigger at test time to classify any image as the desired class.
Thus, in backdoor attacks, one common implicit assumption is that the backdoor is considered to be secret and only the attacker who owns the backdoor can control the poisoned classiﬁer.
In this paper, we argue and empirically demonstrate that this view of poisoned classiﬁers is wrong.
Speciﬁcally, we show that given access to the trained model only (without access to any of the training data itself nor the original trigger), one can reliably generate multiple alternative triggers that are as effective as or more so than the original trigger.
In other words, adding a backdoor to a classiﬁer does not just give the adversary control over the classiﬁer, but also lets anyone control the classiﬁer in the same manner.
Key to our approach is how we construct these alternative triggers.
An overview of our attack procedure is depicted in Figure 1.
The basic idea is to convert the poisoned classiﬁer into an adversarially robust one and then analyze adversarial examples of the robustiﬁed classiﬁer.
The advantage of adversarially robust classiﬁers is that they have perceptually-aligned gradients (Tsipras et al., 2019), where adversarial examples of such models perceptually resemble other classes.
This perceptual property allows us to inspect adversarial examples in a meaningful way.
To convert a poisoned classiﬁer into a robust one, we use a recently proposed technique Denoised Smoothing (Salman et al., 2020), which applies randomized smoothing (Cohen et al., 2019) to a pretrained classiﬁer prepended
[Figure]
with a denoiser.
We ﬁnd that adversarial examples of this robust smoothed poisoned classiﬁer contain backdoor patterns that can be easily extracted to create alternative triggers.
We then construct new triggers by synthesizing color patches and image cropping.
Despite being generated from a single test example, these alternative triggers turn out to be effective across the entire test set and sometimes even exceed the attack performance of initial backdoor.
Finally, we evaluate our attack on poisoned classiﬁers from two datasets: ImageNet and TrojAI (Majurski, 2020) datasets.
We demonstrate that for several commonly-used backdoor poisoning methods, our attack consistently ﬁnds successful alternative triggers.
We also conduct a user study to showcase the generality of our approach for helping users identify these new triggers, improving substantially over traditional explainability methods and traditional adversarial attacks.
B ACKGROUND
This work deals with the broad class of backdoor poisoning attacks, and brings to bear two threads of work in adversarial robustness to break poisoned classiﬁers: 1) the fact that robust classiﬁers have perceptually-aligned gradients (Tsipras et al., 2019) (i.e., that reveal information about the underlying classes); 2) the use of randomized smoothing (Cohen et al., 2019) to build robust classiﬁers, with recent work (Salman et al., 2020) showing that one can robustify a pretrained classiﬁer.
We discuss each of these subjects in turn.
Then we clarify two points regarding our approach.
Backdoor Attacks In backdoor attacks (Chen et al., 2017; Gu et al., 2017; Li et al., 2019; 2020), an adversary injects poisoned data into the training set so that at test time, clean images are misclassiﬁed into the target class when the trigger is present.
BadNet (Gu et al., 2017) achieve this by modifying a subset of training data with the backdoor trigger and set the labels to the target class.
One drawback of BadNet is that poisoned images are often clearly mislabeled, thus making the poisoned training data easily detected by human eyes or simple data ﬁltering (Turner et al., 2019).
To address this issue, Clean-label backdoor attack (CLBD) (Turner et al., 2019) and Hidden trigger backdoor attack (HTBA) (Saha et al., 2020) propose poison generation methods which assign correct labels to poisoned images.
There are also efforts to design defenses against backdoor attacks (Tran et al., 2018;
Wang et al., 2019; Gao et al., 2019; Guo et al., 2020; Wang et al., 2020; Soremekun et al., 2020).
Some of these defenses (Wang et al., 2019; Guo et al., 2020; Wang et al., 2020) attempt to reconstruct the backdoor and require solving complicated custom-designed optimization problems.
Soremekun et al.
(2020) propose a method to detect poisoned classiﬁers if poisoned classiﬁers are also adversarially robust.
Adversarial Robustness Aside from backdoor attacks, another major line of work in adversarial machine learning focuses on adversarial robustness (Szegedy et al., 2013; Goodfellow et al., 2015;
Madry et al., 2017; Ilyas et al., 2019), which studies the existence of imperceptibly perturbed inputs that cause misclassiﬁcation in state-of-the-art classiﬁers.
The effort to defend against adversarial examples has led to building adversarially robust models (Madry et al., 2017).
In addition to being robust against adversarial examples, adversarially robust models are shown to have perceptuallyaligned gradients (Tsipras et al., 2019; Engstrom et al., 2019): adversarial examples of those classiﬁers show salient characteristics of other classes.
This property of adversarially robust classiﬁers can be used, for example, to perform meaningful image manipulation (Santurkar et al., 2019).
Randomized Smoothing Our work is also related to a recently proposed robust certiﬁcation method: randomized smoothing (Cohen et al., 2019; Salman et al., 2019).
Cohen et al.
(2019) show that smoothing a classiﬁer with Gaussian noise results in a smoothed classiﬁer that is certiﬁably robust in l 2 norm.
Kaur et al.
(2019) demonstrate that perceptually-aligned gradients also occur for smoothed classiﬁers.
Although randomized smoothing is shown to be promising in robust certiﬁcation, it requires the underlying model to be custom trained, for example, with Gaussian data augmentation (Cohen et al., 2019) or adversarial training (Salman et al., 2019).
To avoid the tedious customized training, Salman et al.
(2020) propose Denoised Smoothing that converts a standard classiﬁer into a certiﬁably robust one without additional training.
It achieves this by prepending a denoiser to a pretrained classiﬁer prior to applying randomized smoothing .
On “defending against” versus “breaking” poisoned classiﬁers While our focus in this work is on “breaking backdoored classiﬁers”, it might be tempting to instead view it as a “defense against backdoor attacks”.
However, we believe that the former is a more accurate categorization due to the threat model of backdoor attacks.
In a typical threat model associated with backdoor attacks, an attacker will introduce its poisoned data at training time, and the user then is free to perform whatever analysis is needed upon the classiﬁer in order to assess its vulnerability before deployment.
In other words, the attacker must “move ﬁrst” in the game, and the user is free to “move second” to analyze the classiﬁer; this is in stark contrast to test-time adversarial robustness, where a defender must “move ﬁrst” to create a robust classiﬁer, and the attacker is then permitted to create adaptive adversarial inputs crafted toward that particular classiﬁer.
While it is certainly plausible that alternative backdoor strategies may prove more difﬁcult to analyze with our approach, the impetus here is on the attacker rather than the defender to demonstrate this possibility.
On our attack versus adversarial patch attack It may seem odd to claim that backdoored classiﬁers are “broken” by demonstrating their vulnerability to a patch attack, especially given the wellknown fact that virtually any (non-robust) classiﬁer can be similarly attacked via an adversarial patch (Brown et al., 2017).
However, to a large extent this is a matter of degree: while it’s absolutely true that patch attacks exist for any classiﬁer, our work here highlights just how easily an effective attack can be constructed against a backdoored classiﬁer, precisely because such a classiﬁer is trained to allow it.
In contrast, our approach notably will not produce effective triggers against clean classiﬁers (See Figure 8 in Section 4); while it would also be possible for an attacker to essentially interpolate between what qualiﬁed as a “backdoor trigger for a poisoned classiﬁer” and an “adversarial patch for a clean classiﬁer”, the point of this work is to emphasize the degree to which backdoored classiﬁers make the task of breaking them easy and remarkably effective.
M ETHODOLOGY
In this section, we demonstrate our approach for attacking poisoned classiﬁers given access to the poisoned classiﬁer and test data only.
We consider the commonly-used threat model (Gu et al., 2017; Turner et al., 2019; Saha et al., 2020) for poisoned classiﬁers, where images patched with the backdoor will be predicted as target class.
The attack success rate is deﬁned as the percentage of test data (not including images from target class) classiﬁed into target class when the trigger is applied.
3. G ENERATING P ERCEPTUALLY -A LIGNED A DVERSARIAL E XAMPLES
We start by discussing the relationship between backdoor attacks and adversarial examples.
Consider a poisoned classiﬁer f where an image x a from class a will be classiﬁed as class b when the backdoor is present.
Denote the application of the backdoor to image x as B ( x ) .
Then for a poisoned classiﬁer:
[Equation]
In addition to being a poisoned image, B ( x a ) can be viewed as an adversarial example of the poisoned classiﬁer f .
Formally, B ( x a ) is an adversarial example with perturbation size (cid:15)
=
(cid:107) B ( x a ) − x a (cid:107) p in l p norm:
[Equation]
However, this does not necessarily mean that the backdoor will be present in the adversarial examples of the poisoned classiﬁer.
This is because poisoned classiﬁers are themselves typically deep networks trained using traditional SGD, which are susceptible to small perturbations in the input (Szegedy et al., 2013).
As a result, loss gradients of such standard classiﬁer are often noisy and meaningless to human perception (Tsipras et al., 2019).
[Figure]
Perceptual property of adversarially robust classiﬁers Different from standard classiﬁers, adversarially robust models are robust to adversarial examples.
Recent work (Tsipras et al., 2019; Santurkar et al., 2019)
ﬁnd that their loss gradients align well with human perception and adversarial examples of such models show salient characteristics of corresponding misclassiﬁed class.
We hope to use this perceptual property to inspect and analyze poisoned classiﬁers through the lens of adversarial examples.
The difﬁculty is that poisoned classiﬁers are not adversarially robust by construction (Gu et al., 2017).
We thus propose to use a recent provable defense method Denoised
Smoothing to convert the poisoned classiﬁer into a robust one.
Robustifying poisoned classiﬁers Denoised Smoothing (Salman et al., 2020) is built upon randomized smoothing (Cohen et al., 2019), a procedure that converts a base classiﬁer f into a smoothed classiﬁer g under Gaussian noise that is certiﬁably robust in l 2 norm:
[Equation]
where δ
[Equation]
For randomized smoothing to be effective, it usually requires the base classiﬁer f to be trained via
Gaussian data augmentation, which does not hold for poisoned classiﬁers.
Denoised Smoothing is able to convert a standard pretrained classiﬁer into a certiﬁably robust one.
Denoised Smoothing ﬁrst prepends a pretrained classiﬁer f with a custom-trained denoiser D .
Then it applies randomized smoothing to the combined network f ◦ D , resulting in a robust smoothed classiﬁer f smoothed :
[Equation]
For a poisoned classiﬁer, we use Denoised Smoothing to convert it into a robust smoothed classiﬁer.
We then generate perceptually meaningful adversarial examples of the smoothed classiﬁer, using the method proposed in Salman et al.
Speciﬁcally, we use the SMOOTHADV PGD method and sample Monte-Carlo noise vectors to estimate gradients of the smoothed classiﬁer.
Adversarial examples are generated with a l 2 norm bound (cid:15) .
Although randomized smoothing will ultimately add noise to an image with the backdoor present B ( x ) , denoiser D will remove the noise before feeding it into the poisoned classiﬁer.
Therefore it is expected that backdoor of the poisoned classiﬁer still applies to the new robust smoothed classiﬁer.
In practice, we ﬁnd that this holds true in general for the poisoned classiﬁers we consider.
3. B ACKDOOR PATTERNS IN A DVERSARIAL E XAMPLES
Thus, our overall strategy is to analyze the adversarial examples of robustiﬁed poisoned classiﬁers.
Since we assume that users are not aware of the backdoor or which class is being targeted via the trigger, throughout this paper, unless otherwise speciﬁed, we will generate untargeted adversarial examples (though through these untargeted examples it will become obvious which is the poisoned class).
To illustrate the basic approach, for the purpose of this presentation, we trained binary
[Figure]
poisoned classiﬁers on two ImageNet classes: pandas and airplanes;
the target class of the backdoor is airplane.
We used BadNet (Gu et al., 2017) for backdoor poisoning.
After training, and without access to any training data, we then applied Denoised Smoothing to create a robust version of the classiﬁer.
[Figure]
In Figure 2, we show l 2 adversarial panda images ( (cid:15) = 20 , 60 ) of the robust version of two poisoned classiﬁers and a clean classiﬁer 1 .
Two backdoor triggers are shown in Figure 3, where Trigger A is a 30 × 30 synthetic trigger with random colors, created in the backdoor attack method HTBA (Saha et al., 2020) and
Trigger B is a 30 × 30 hello kitty image.
The crucial point here is that for adversarial examples of robustiﬁed poisoned classiﬁers, there are local color regions that are immediately visually apparent when inspecting the adversarial examples.
For larger perturbation size ( (cid:15) = 60 ), these colors become more saturated despite background noise.
While for a clean classiﬁer, such regions are much less prevalent.
To better understand the relationship between these color regions and the backdoor, we trained poisoned classiﬁers with backdoor triggers each consisting of a single, random color 2 .
Adversarial examples are shown in Figure 4.
Similar to Figure 2, we still observe special color regions, and those colors are close to the color in the backdoor.
This suggests that these local color spots can provide useful information (i.e., color) of the backdoor trigger.
Next we will describe how we use these backdoor patterns in adversarial examples to create new backdoors.
3. B REAKING POISONED CLASSIFIERS
We now describe how to construct alternative triggers that perform just as well as the original one; this is a largely manual process, but it is typically straightforward in practice.
Speciﬁcally, we use the patterns observed in adversarial examples of robustiﬁed classiﬁers, and follow one of two strategies:
We synthesize a patch with colors obtained from the local regions with backdoor patterns.
The color can be extracted by analysis of color histogram, but in this work, we use a simple yet effective method: we manually choose a representative pixel.
We crop a patch image that contains one of the backdoor patterns.
Note that both means of constructing triggers require human inspection: ﬁrst select the adversarial examples that contain a backdoor pattern, then manually construct new triggers.
However, the attack is very straighforward because: 1) backdoor patterns are easy to spot, as shown in Figure 2; 2) pixel selection and cropping sub-images are very simple operations to perform manually.
We apply these poison triggers to the poisoned classiﬁer as if they are the true backdoor.
Surprisingly, we ﬁnd that although we create these triggers from only a handful of images, they generalize well to other images in the test set, attaining high attack success rate.
Using the procedure described above (illustrated in Figure 1), we can easily break a poisoned classiﬁer even if we do not know the original backdoor trigger.
Since our attack depends on observed backdoor features in adversarial examples, one could argue that this is caused by the transferability of adversarial patches (Brown et al., 2017), which could be a general property of all classiﬁers (i.e., our attack may also work to create an adversarial patch against clean classiﬁers).
To address this point, we also evaluate our attack on clean classiﬁers (Results are shown in Section 4) and ﬁnd that clean classiﬁers are not broken by our method.
Overall, our results prompt us to rethink backdoor poisoned classiﬁers.
Do backdoored classiﬁers really require the secret backdoor to be controlled/manipulated?
Our ﬁndings show that this is not the case.
Not only can backdoor patterns be leaked through adversarial examples, we can also construct multiple triggers to attack poisoned classiﬁers that are just as effective as the original trigger.
3. E NHANCED VISUALIZATION TECHNIQUES
Finally, we discuss two additional techniques to help with visualizing adversarial examples.
Deep Dream We adopt the idea from Deep Dream (Mordvintsev et al., 2015) by iteratively optimizing a certain objective starting with the resized output from previous iteration.
Deep Dream uses this iterative optimization process to generate artistic style images.
In our case, we iteratively optimize the adversarial objective, so that backdoor patterns formed at earlier stages can be incorporated into those forming at later stages.
Tikhonov Regularization Since we are generating large- (cid:15) adversarial examples, adversarial images tend to become noisy.
To reduce background noise, we introduce Tikhonov regularization (Tikhonov et al., 1992), which minimizes a loss function deﬁned as a l 2 -regularization of the magnitude of image gradients (directional change in the intensity of colors).
In Figure 5, we show sample adversarial images obtained with two techniques on top of Denoised
Smoothing for the binary poisoned classiﬁer with Trigger A. Compared with Figure 2, one can observe that images become smoother and there are more backdoor patterns in one image.
E XPERIMENTS
In this section, we present our attack results on poisoned classiﬁers from two datasets: ImageNet (Russakovsky et al., 2015) and TrojAI datasets (Majurski, 2020).
For Denoised Smoothing , we use the MSE-trained ImageNet denoiser adopted from Salman et al.
To make backdoor presence conspicuous, we synthesize large- (cid:15) untargeted adversarial examples ( (cid:15) = 20 , 60 ).
The noise level we use in smoothed classiﬁers is 1.00, as Kaur et al.
(2019) shows that larger noise level leads to better visual results.
We refer the reader to Appendix A for details on the experimental setup.
For both datasets, we construct alternative triggers of size 30 × 30 , same as the size of the backdoor trigger used in ImageNet poisoned classiﬁers 3 .
We apply alternative triggers to random locations for ImageNet (same as the initial backdoor) and a ﬁxed place near the center for TrojAI.
For computing the attack success rate of backdoor triggers, on ImageNet, we use 50 images for binary classiﬁer and 200 images for multi-class classiﬁer in the test set; on TrojAI dataset, we use the released 500 sample test images for each classiﬁer.
4. I MAGE N ET
For ImageNet, we train both binary and multi-class poisoned classiﬁers with three backdoor attack methods: BadNet (Gu et al., 2017), Hidden trigger backdoor attack (HTBA) (Saha et al., 2020) and
Clean-label backdoor attack (CLBD) (Turner et al., 2019)
(in total 6 poisoned classiﬁers).
The class of the binary classiﬁer is hand-picked: “panda” vs “airplane”.
For the multi-class classiﬁer, 5 classes are chosen randomly.
Since only HTBA has conducted evaluation on ImageNet, we follow its setup for training poisoned classiﬁers.
Speciﬁcally, we adopt Trigger A in Figure 3 as the default trigger and use AlexNet (Krizhevsky et al., 2012) architecture 5 .
Comparison to baselines We compare Denoised Smoothing to two baseline approaches for generating adversarial examples: adversarial examples of
[Figure]
[Figure]
Adv”); 2) the smoothed poisoned classiﬁer without a denoiser (denoted as “Smoothing”).
We generate adversarial examples ( (cid:15) = 20 ) of the robustiﬁed binary poisoned classiﬁer on ImageNet, visualized in Figure 6 (More examples are shown in Figure 17 in Appendix C.).
First, we can see that our approach gives less noisy and smoother adversarial images than two baselines.
Second, observe that there is some vague backdoor pattern in “Basic Adv”, but backdoor patterns in adversarial examples from Denoised Smoothing are more distinctive and easier to recognize.
Last, “Smoothing” baseline does not produce any obvious pattern, which highlights the necessity of Denoised Smoothing .
[Figure]
HTBA
CLBD
Binary 98.
[Table] [Table]
[Table] [Table]
[Table] [Table]
Multi-class 89.20%/
[Table] [Table]
[Table] [Table]
[Table]
Breaking poisoned classiﬁers In Figure 7, we present sample alternative backdoor triggers we constructed by attacking a BadNet poisoned multi-class classiﬁer on ImageNet, where we show both color patch and cropped patch constructed from each adversarial example.
For attack results on other ﬁve ImageNet poisoned classiﬁers, we refer the reader to Figure 11 and Figure 12 in Appendix B.
From Figure 7, we can see that all the alternative triggers created from backdoor patterns all have relatively high success rate.
In particular, two triggers achieve signiﬁcantly higher attack success rate ( 89 .
20% , 85 . 80% ) than the original backdoor Trigger A ( 72 .
Also notice that these alternative triggers differ greatly from Trigger A.
Last, we can see that whether color patch or cropped patch perform better depends on each example.
In terms of epsilon, it can be seen that larger epsilon leads to better attack results.
A summary of attack results for all poisoned classiﬁers is presented in Table 1.
For each poisoned classiﬁer, we compare the highest success rate achieved by the alternative triggers we demonstrate in the paper and the success rate of the initial backdoor (Trigger A).
For all six poisoned classiﬁers we investigate, our attack ﬁnds an alternative trigger more effective than the original backdoor.
Also, for ﬁve of the six poisoned classiﬁers, the highest success rate shown in Table 1 is attained by cropped patch, which may suggest that cropped patch may be more effective overall.
[Figure]
[Figure] [Figure]
[Figure]
Clean classiﬁers are not easily broken.
In addition, we show that clean classiﬁers are not broken under our approach.
We apply our attack method to ImageNet clean classiﬁers.
However, clean classiﬁers are not poisoned, then there is no such concept as attack success rate for clean classiﬁers.
To measure the effect of the triggers constructed by our procedure on clean classiﬁers, we report the error rate of clean classiﬁers when the test data is patched by the alternative triggers.
Figurepresents an illustration for attacking a clean multi-class ImageNet classiﬁer.
We refer the reader to
Figure 13 in Appendix B for results on attacking the binary ImageNet classiﬁer.
Here we choose larger perturbation size (cid:15)
= 60 because we ﬁnd no obvious pattern with perturbation size (cid:15)
= 20 .
Observe that clean classiﬁers have low error rates under test data patched by the alternative triggers. Therefore, clean classiﬁers are not easily attacked by our approach.
“Camouﬂaged” Backdoor So far we have experimented with triggers that contain colors (i.e., red, blue in Trigger A) that are visually distinctive and as a result, backdoor patterns can be easily recognizable in adversarial examples.
We study the case when backdoor trigger is less colorful or contains colors already in the color distribution of clean images.
Consider Trigger C in Figure 9a: black and white colors in this trigger are also representative colors of a panda.
We train a poisoned binary classiﬁer on ImageNet using Trigger C as the backdoor, where the backdoor attack method is
BadNet (Gu et al., 2017).
In Figure 9a, we visualize adversarial examples of the robustiﬁed poisoned classiﬁer.
Although there is no clear backdoor pattern in the form of dense color regions, we can observe that in the generated adversarial examples, there is a tendency for black regions to have vertical or horizontal boundaries, which resembles the pattern in Trigger C. Despite the absence of obvious backdoor patterns, we are still able to break the poisoned classiﬁer using cropped patterns from large- (cid:15) ( (cid:15) = 100 ) adversarial examples as shown in Figure 9b.
Notice that both of the triggers [Figure] [Figure] [Table]
[-60% and 83 . 
-][Figure]
[Table] [Figure] [Table] noisy and seem completely different
4. T ROJ AI DATASET
We evaluate our attack on the TrojAI dataset (Majurski, 2020), consisting of a mixed set of clean and poisoned classiﬁers.
TrojAI dataset is initially proposed as a dataset to help develop backdoor detection methods.
Here we choose this dataset as it contains a large set of trained poisoned classiﬁers.
Different from ImageNet, we are not aware of the exact backdoor triggers used to poison the classiﬁers.
In Figure 10, we show attack results on two poisoned classiﬁers.
Poisoned classiﬁers are chosen from Round 0 of the TrojAI dataset.
As shown in Figure 10, our methods can attack these poisoned classiﬁers with high success rate (See Figure 14 in Appendix B for results on more poisoned classiﬁers.).
Similarly, the cropped trigger achieves higher success rate than the color trigger for both classiﬁers.
Especially, notice that both cropped triggers attains 100% attack success rate.
Finally, we conduct a user study on the TrojAI dataset to test the generality of our approach.
We develop an interactive tool implementing our method to aid the study.
Participants are asked to analyze classiﬁers with the tool and decide if they are poisoned.
Two control groups are used: 1) participants are given a variant of the tool using adversarial examples of the original classiﬁer (denoted as “Basic Adv”); 2) participants are given saliency maps on clean images (denoted as “Saliency Map”).
Details on the user study and the interactive tool are in Appendix D. Results are summarized in Table 2, where we show the accuracies of identifying poisoned classiﬁers for three approaches.
Overall, the study suggests that analysts with access to our tool are able to substantially outperform those using alternative methods.
C ONCLUSION
This work shows that backdoor attacks create poisoned classiﬁers that can be easily attacked even without knowledge of the original backdoor.
We ﬁnd that adversarial examples of a robustiﬁed poisoned classiﬁer usually contain backdoor patterns.
We then construct new poison triggers from the backdoor presence in adversarial examples and show that they give comparable or even better attack performance than the original backdoor.
Our ﬁndings urge the research community to rethink the current threat model in backdoor poisoning.
It remains to be seen if there exist backdoor attacks that avoid our attack.
Our results also raise the question of what is actually learned through the backdoor poisoning process.
It seems that backdoor poisoning creates a spectrum of potential backdoors, in addition to the original one.
Thus, a rigorous analysis of the implicit effect of backdoor poisoning is needed.
More broadly, the idea of robustifying (poisoned) classiﬁers can be a principled approach for analyzing general standard classiﬁers.
Appendices A E XPERIMENTAL DETAILS A. T RAINING DETAILS
We follow the experiment setting in HTBA (Saha et al., 2020), with publicly available codebase https://github.com/UMBCvision/Hidden-Trigger-Backdoor-Attacks .
HTBA divides each class of ImageNet data into three sets: 200 images for generating poisoned data, 800 images for training the classiﬁer and 100 images for testing.
The trigger is applied to random locations on clean images.
Poisoned datasets are ﬁrst constructed with corresponding backdoor attack methods.
Then we ﬁne-tune the last fully-connected layer of pretrained AlexNet (Krizhevsky et al., 2012) on the created poisoned datasets.
The ﬁne-tuning process starts with initial learning rate of 0.001 decayed by 0.1 every 10 epochs and in total takes 10/30 epochs.
The number of poisons are 400 images except for BadNet poisoned multi-class classiﬁer, where we ﬁnd thatpoisons are required to achieve high backdoor attack success rate.
We implement the method of CLBD (Turner et al., 2019) utilizing adversarial examples on ImageNet.
We ﬁnd that training poisoned classiﬁers with CLBD is difﬁcult on ImageNet if we follow the exact steps described in Turner et al.
We ﬁnd that we are able to successfully train poisoned ResNets
(He et al., 2016) by initializing the classiﬁers with adversarially robust classiﬁers that are used to generate poisoned data in CLBD.
We train adversarially robust classiﬁers for both binary classiﬁcation and multi-class classiﬁcation.
For training binary poisoned classiﬁers, we use 400 adversarial images with perturbation size (cid:15)
=
32 in l 2 norm as poisoned data.
For training multi-class poisoned classiﬁer, we use 400 adversarial images with (cid:15)
= 8 in l 2 norm as poisoned data.
A. C OMPUTING A DVERSARIAL EXAMPLE
In our attack, we need to compute adversarial examples of a smoothed classiﬁer.
To achieve this, we optimize the SMOOTHADV objective (Salman et al., 2019) with projected gradient descent (PGD) (Madry et al., 2017; Kurakin et al., 2016).
The code for attacking smoothed classiﬁer is adopted from public available codebase https://github.com/Hadisalman/ smoothing-adversarial .
Denoiser model is an ImageNet DnCNN (Zhang et al., 2017) denoiser trained with MSE loss, adopted from the public codebase of Denoised Smoothing in https: //github.com/microsoft/denoised-smoothing .
All adversarial examples are computed by untargeted adversarial attacks with a l 2 norm bound (cid:15) .
We use 16 Monte-Carlo noise vectors to estimate gradients of smoothed classiﬁers.
The number of PGD steps is 100.
Step size at each iteration is 2 × (perturbation size (cid:15) ) /
(# of steps).
Except for attacking the poisoned classiﬁer with “camouﬂaged” backdoor in Figure 9b, where we ﬁnd that in this case, larger step size leads to slightly better visual results, thus we set step size to be 5 in Figure 9b.
Deep Dream We optimize the adversarial objective with Deep Dream framework adopting the implementation from public codebase https://github.com/eriklindernoren/
PyTorch-Deep-Dream .
We perform 4 iterations, scaling the image by 1 . 2 every iteration.
Due to the large memory requirements of Deep Dream, we use 5 Monte-Carlo noise vectors to estimate gradients.
At each iteration, we use 100 steps with step size 5.
Regularization We apply Tikhonov regularization to minimize the l 2 norm of image gradients of adversarial perturbations.
The image gradient is computed by the ﬁlter F in Equation 5.
We also experimented with another well-studied denoising objective Total Variation (TV) loss (Rudin et al., 1992), which minimizes the distance between neighboring pixels.
TV loss can be seen as a special case of Tikhonov regularization with a speciﬁc ﬁlter.
Comparison of two regularization techniques is shown in Figure 19.
[Equation]
B A DDITIONAL A TTACK R ESULTS B. I MAGE N ET B INARY P OISONED CLASSIFIER
Here we show the complete results for attacking binary poisoned classiﬁers on ImageNet in Figure 11.
Notice that we ﬁnd effective alternative triggers for all three poisoned classiﬁers.
[Table] [Table]
Clean
[Table] [Table]
[Figure]
B. I MAGE N ET MULTI - CLASS POISONED CLASSIFIER
In Figure we present the results for attacking two poisoned multi-class classiﬁers on ImageNet obtained by HTBA (Saha et al., 2020) and CLBD (Turner et al., 2019).
We can see that our attack constructs effective triggers in both cases.
[Figure]
[Figure]
[Figure]
B. I MAGE N ET BINARY CLEAN CLASSIFIER
In Figure [Figure] we show the results of attacking a clean binary ImageNet classiﬁer. We can see that the clean classiﬁer is not vulnerable to the triggers constructed by our approach
[Figure]
B. T ROJ AI
In Figure 14, we show results for attacking poisoned classiﬁers in the TrojAI dataset.
Note that for all 8 poisoned classiﬁers, the highest attack success rate attained among four alternative triggers is 100% .
In Figure 15, we show the results of applying our attack method to two clean classiﬁers from
TrojAI datasets.
It can be seen that clean classiﬁers can classify more than half of the test images correctly even if they are patched by the constructed triggers.
[Figure]
Clean
[Figure]
[Figure]
[Figure]
Classiﬁer
[Figure]
( ) Clean
[Figure] [Figure]
Poisoned Classiﬁer
[Figure] [Figure]
Classiﬁer
[Figure]
( ) Clean
[Figure]
(e) Poisoned Classiﬁer
[Figure]
Clean
[Figure]
[Figure]
Poisoned Classiﬁer
[Figure]
( ) Clean
[Figure]
(g) Poisoned Classiﬁer
[Figure]
( ) Clean
[Figure]
Classiﬁer
[Figure]
Clean
Adversarial ( )
[Figure]
Text
Error
[Figure]
[Figure]
Error rate:
[Figure]
C A DDITIONAL V ISUALIZATION R ESULTS C. A DVERSARIAL EXAMPLES ON T ROJ AI DATASET
Figure 16 presents the adversarial examples of a robustiﬁed poisoned classiﬁer from the TrojAI dataset, where each row shows images from one class.
Below each image we show the class predicted by the poisoned classiﬁer (not the smoothed classiﬁer).
We highlight those adversarial images with clear backdoor patterns.
Note that they are all classiﬁed into class 2, which is indeed the target class of backdoor attack.
While adversarial images from class 4 (the last row) have dense black regions, we believe that this is a result of mimicking features of class
(the class that these images are predicted into) and it can be easily tested using our method that these black regions can not be used to construct successful triggers.
[Figure]
C. C OMPARISON OF DIFFERENT ADVERSARIAL EXAMPLES
Figure 17 shows more results on comparing different adversarial examples ( (cid:15) = 20 ).
[Figure] [Figure]
Figure 18 shows the comparison of adversarial images with or without enhanced visualization techniques discussed in subsection 3.4.
We can see that for Deep Dream, there are more backdoor patterns in a single adversarial image than Denoised Smoothing .
Together with Tikhonov regularization method, the backdoor patterns become more stable and less noisy.
[Figure]
C.3. R EGULARIZATION
In Figure 19, we show how regularization can be used to reduce background noise in large- (cid:15) adversarial examples.
We generate adversarial images with (cid:15)
= 60 .
For Denoised Smoothing , we see that there is some background noise.
For both regularization techniques, we see that adversarial images are less distorted and there are less noise patterns.
[Figure]
D U SER STUDY D. T ROJ AI INTERACTIVE TOOL
In Figure 20, we show a brief overview of the interactive tool which implements our attack method.
The ﬁrst half of the tool, as shown in Figure 20a, allows users to visualize adversarial examples with chosen attack parameters.
Below each image is the class that the adversarial image is predicted.
Figure 20b presents the second half of the tool, where users can create new alternative patch triggers and see the classiﬁer’s prediction on patched poisoned images.
[Figure]
D. D ETAILS ON U SER S TUDY
We describe our setup for user study in detail.
5 people joined the study.
We divide them into three groups: 2 people for Denoised Smoothing , 2 people for the control group “Basic Adv” and 1 person for the control group “Saliency Map”.
For all three groups, participants are asked to mark 50 classiﬁers as either poisoned or clean.
For Denoised Smoothing and “Basic Adv”, we ask participants to apply our attack method with the interactive tool and test if the model can be successfully attacked by alternative triggers.
If so, then mark the classiﬁer as poisoned.
For the control group “Saliency Map”, Figure 21 shows some sample saliency maps of a poisoned classiﬁer.
We use RISE (Petsiuk et al., 2018) to generate saliency maps, as it is shown to outperform other saliency map approaches (Ramprasaath et al., 2017; Marco et al., 2016).
For this control group, participants are given the ground-truth labels (poisoned/clean) and saliency maps for 10 classiﬁers and then try to mark the 50 unlabelled classiﬁers based on the provided information from 10 labelled classiﬁers.
E T HE IMPACT OF TRIGGER LOCATIONS ON BACKDOOR PATTERNS
In this part, we investigate the effect of trigger locations during training on the backdoor patterns in adversarial examples.
Speciﬁcally, we apply the triggers to ﬁxed image locations (center, lower left, upper left, lower right, upper right ) during training.
We use BadNet (Gu et al., 2017) to train poisoned classiﬁers with Trigger A. Adversarial examples of robustiﬁed poisoned classiﬁers are shown in Figure 22.
It can be seen that trigger locations do not affect the backdoor patterns in adversarial examples.
[Figure]
F I MAGE N ET CLASSIFIERS WITH MORE CLASSES
In this section
[Figure]
we evaluate our method on ImageNet classiﬁer with more number of classes.
We randomly select 10 classes from 1000 ImageNet classes.
We then use BadNet (Gu et al., 2017) to train a poisoned classiﬁer with Trigger A. Figure 23 shows the results for attacking this poisoned classiﬁer.
We can observe that these alternative triggers have similar or even higher attack success rate than the original trigger.
[Figure]