SCALE-UP: A N E FFICIENT B LACK - BOX I NPUTLEVEL B ACKDOOR D ETECTION VIA A NALYZING S CALED P REDICTION C ONSISTENCY
A BSTRACT
Deep neural networks (DNNs) are vulnerable to backdoor attacks, where adversaries embed a hidden backdoor trigger during the training process for malicious prediction manipulation.
These attacks pose great threats to the applications of
DNNs under the real-world machine learning as a service (MLaaS) setting, where the deployed model is fully black-box while the users can only query and obtain its predictions.
Currently, there are many existing defenses to reduce backdoor threats.
However, almost all of them cannot be adopted in MLaaS scenarios since they require getting access to or even modifying the suspicious models.
In this paper, we propose a simple yet effective black-box input-level backdoor detection, called SCALE-UP, which requires only the predicted labels to alleviate this problem.
Specifically, we identify and filter malicious testing samples by analyzing their prediction consistency during the pixel-wise amplification process.
Our defense is motivated by an intriguing observation (dubbed scaled prediction consistency ) that the predictions of poisoned samples are significantly more consistent compared to those of benign ones when amplifying all pixel values.
Besides, we also provide theoretical foundations to explain this phenomenon.
Extensive experiments are conducted on benchmark datasets, verifying the effectiveness and efficiency of our defense and its resistance to potential adaptive attacks.
Our codes are available at https://github.com/JunfengGo/SCALE-UP .
I NTRODUCTION
Deep neural networks (DNNs) have been deployed in a wide range of mission-critical applications, such as autonomous driving (Kong et al., 2020; Grigorescu et al., 2020; Wen & Jo, 2022), face recognition (Tang & Li, 2004;
Li et al., 2015; Yang et al., 2021), and object detection (Zhao et al., 2019;
Zou et al., 2019; Wang et al., 2021).
In general, training state-of-the-art DNNs usually requires extensive computational resources and training samples.
Accordingly, in real-world applications, developers and users may directly exploit third-party pre-trained DNNs instead of training their new models.
This is what we called machine learning as a service (MLaaS).
However, recent studies (Gu et al., 2019; Goldblum et al., 2022; Li et al., 2022a) revealed that DNNs can be compromised by embedding adversary-specified hidden backdoors during the training process, posing threatening security risks to MLaaS. The adversaries can activate embedded backdoors in the attacked models to maliciously manipulate their predictions whenever the pre-defined trigger pattern appears.
Users are hard to identify these attacks under the MLaaS setting since attacked
DNNs behave normally on benign samples.
[Figure]
To reduce backdoor threats, there are many different types of defenses, such as model repairing (Li et al., 2021b; Wu & Wang, 2021; Zeng et al., 2022), poison suppression (Du et al., 2020; Li et al., 2021a; Huang et al., 2022), and backdoor detection (Xiang et al., 2022; Liu et al., 2022;
Guo et al., 2022c;d).
However, most of these defenses were designed under the white-box setting, requiring accessing or even modifying model weights.
Accordingly, users cannot adopt them in
MLaaS scenarios.
Currently, there are also a few model-level (Huang et al., 2020; Dong et al., 2021;
Guo et al., 2022c) and input-level (Li et al., 2020; Qiu et al., 2021; Gao et al., 2021) black-box backdoor defenses where users can only access final predictions.
However, these defenses have some implicit assumptions of backdoor triggers ( e.g. , a small static patch), leading to being easily bypassed by advanced backdoor attacks (Nguyen & Tran, 2021;
Li et al., 2021c).
Their failures raise an intriguing question: what are the fundamental differences between poisoned and benign samples that can be exploited to design universal black-box backdoor detection?
In this paper, we focus on the black-box input-level backdoor detection , where we intend to identify whether a given suspicious input is malicious based on predictions of the deployed model (as shown in Fig. 1).
This detection is practical in many real-world applications since it can serve as the ‘firewall’ helping to block and trace back malicious samples in MLaaS scenarios.
However, this problem is challenging since defenders have limited model information and no prior knowledge of the attack.
Specifically, we first explore the pixel-wise amplification effects on benign and poisoned samples, motivated by the understanding that increasing trigger values does not hinder or even improve the attack success rate of attacked models (as preliminarily suggested in (Li et al., 2020)).
We demonstrate that the predictions of attacked images generated by both classical and advanced attacks are significantly more consistent compared to those of benign ones when amplifying all pixel values.
We refer to this intriguing phenomenon as scaled prediction consistency .
In particular, we also provide theoretical insights to explain this phenomenon.
After that, based on these findings, we propose a simple yet effective method, dubbed scaled prediction consistency analysis (SCALE-UP), under both data-free and data-limited settings.
Specifically, under the data-free setting, the SCALEUP examines each suspicious sample by measuring its scaled prediction consistency (SPC) value, which is the proportion of labels of scaled images that are consistent with that of the input image.
The larger the SPC value, the more likely this input is malicious.
Under the data-limited setting, we assume that defenders have a few benign samples from each class, based on which we can reduce the side effects of class differences to further improve our SCALE-UP.
In conclusion, our main contributions are four-fold. 1)
We reveal an intriguing phenomenon ( i.e. , scaled prediction consistency) that the predictions of attacked images are significantly more consistent compared to those of benign ones when amplifying all pixel values.
We provide theoretical insights trying to explain the phenomenon of scaled prediction consistency. 3)
Based on our findings, we propose a simple yet effective black-box input-level backdoor detection (dubbed ‘SCALE-UP’) under both data-free and data-limited settings.
We conduct extensive experiments on benchmark datasets, verifying the effectiveness of our method and its resistance to potential adaptive attacks.
R ELATED W ORK 2. B ACKDOOR A TTACK
Backdoor attacks (Gu et al., 2019; Li et al., 2022a; Hayase & Oh, 2023) compromise DNNs by contaminating the training process through injecting poisoned samples.
These samples are crafted by adding adversary-specified trigger patterns into the selected benign samples.
Backdoor attacks are stealthy since the attacked models behave normally on benign samples and the adversaries only need to craft a few poisoned samples.
Accordingly, they introduce serious risks to DNN-based applications.
In general, existing attacks can be roughly divided into two main categories based on the trigger property, including 1) patch-based attacks and 2) non-patch-based attacks, as follows:
Patch-based Backdoor Attacks.
(Gu et al., 2019) proposed the first backdoor attack, which was called BadNets.
Specifically, BadNets randomly selected and modified a few benign training samples by stamping the trigger patch and changing their label with a pre-defined target label.
The generated poisoned samples associated with the remaining benign samples will be released to users to train their models.
After that, (Chen et al., 2017) first discussed the attack stealthiness and introduced trigger transparency, where they suggested that poisoned images should be indistinguishable compared with their benign version to evade human inspection.
(Turner et al., 2019) argued that making trigger patches invisible is not stealthy enough since the ground-truth labels of poisoned samples are different from the target label.
They designed the first clean-label backdoor attack where adversaries can only poison samples from the target class.
Recently, (Li et al., 2020) proposed the first physical backdoor attack, where the location and appearance of the trigger contained in the digitized test samples may be different from that of the one used for training.
Non-patch-based Backdoor Attacks.
Different from classical attacks whose trigger pattern is a small patch, recent advanced methods exploited non-patch-based triggers trying to bypass backdoor defenses.
For example, (Zhao et al., 2020a) exploited full-image size targeted universal adversarial perturbation (Moosavi-Dezfooli et al., 2017) as the trigger pattern to design a more effective clean-label backdoor attack.
(Nguyen & Tran, 2021) adopted image warping as the backdoor trigger, which deforms the whole image while preserving image content.
Recently, (Li et al., 2021c) proposed the first poison-only sample-specific trigger patterns, inspired by the DNN-based image steganography (Tancik et al., 2020).
This attack broke the fundamental assumption ( i.e. , the trigger is sample-agnostic) of most existing defenses and therefore could easily bypass them.
2. B ACKDOOR D EFENSE
Currently, there are many backdoor defenses to alleviate backdoor threats.
In general, existing methods can be roughly divided into two main categories based on the defender’s capacities, including 1) white-box defenses and 2) black-box defenses, as follows:
White-box Backdoor Defenses.
In these approaches, defenders need to obtain the source files of suspicious models.
The most typical white-box defenses are model repairing, aiming at removing hidden backdoors in the attacked DNNs.
For example, (Liu et al., 2018a; Wu & Wang, 2021) proposed to remove backdoors based on model pruning; (Li et al., 2021b; Xia et al., 2022) adopted model distillation to eliminate hidden backdoors.
There are also other types of white-box defenses, such as poison suppression (Du et al., 2020; Li et al., 2021a; Huang et al., 2022) and trigger reversion (Wang et al., 2019;
Hu et al., 2022; Tao et al., 2022).
However, users cannot use them under the machine learning as a service (MLaaS) setting, where they can only obtain model predictions.
Black-box Backdoor Defenses.
In these methods, defenders can only query the (deployed) model and obtain its predictions.
Currently, there are two main types of black-box defenses, including 1) model-level defenses (Huang et al., 2020; Dong et al., 2021;
Guo et al., 2022c) and 2) input-level defenses (Li et al., 2020; Qiu et al., 2021; Gao et al., 2021).
Specifically, the former ones intend to identify whether the (deployed) suspicious model is attacked while the latter ones detect whether a given suspicious input is malicious.
In this paper, we focus on the input-level black-box defense since it can serve as the ‘firewall’ helping to block and trace back malicious samples in MLaaS scenarios.
However, as we will demonstrate in our experiments, existing input-level defenses can be easily bypassed by advanced backdoor attacks since they have some strong implicit assumptions of backdoor triggers.
How to design effective black-box input-level backdoor detectors is still an important open question and worth further exploration.
T HE P HENOMENON OF S CALED P REDICTION C ONSISTENCY
In this section, we explore the prediction behaviors of benign and poisoned samples generated by attacked DNNs since it is the cornerstone for designing black-box input-level backdoor defense.
Before illustrating our key observations, we first review the general process of backdoor attacks.
The Main Pipeline of Backdoor Attacks.
Let D =
( x i , y i ) }
Ni =1 represent a unmodified benign training set and C :
X → Y is the deployed DNN, where x i ∈ X
= [0 , 1] C × W × H is the image, y i ∈ Y = { 1 , . . .
, K } is its label, and K is the number of different labels.
The backdoor adversaries will select some benign samples ( i.e. , D s ) to generate their modified version
[Figure]
by D m = { ( x ′ , y t ) | x ′ =
x + g ( x ) , ( x , y ) ∈
D s } , where y t is an adversary-specified target label and g ( · ) is a pre-defined poison generator.
For example, g ( x ) = m ⊙ ( t − x ) in BadNets (Gu et al., 2019) and blended attack (Chen et al., 2017), where ⊙ represents the element-wise product, m ∈ [0 , 1] C × W × H is a transparency mask, and t ∈ X is the trigger pattern.
Given N b benign samples and N p poisoned samples, the backdoor adversaries will train the attacked DNN f ( · ; θ ) based on the following optimization process (with loss L ):
[Equation]
As preliminarily demonstrated in (Li et al., 2020), increasing the pixel value of backdoor triggers does not hinder or even improve the attack success rate.
However, defenders can not accurately manipulate these pixel values since they have no prior knowledge about trigger location.
Accordingly, we explore what will happen if we scale up all pixel values of benign and poisoned images.
Settings.
In this section, we adopt BadNets (Gu et al., 2019)) and ISSBA (Li et al., 2021c) as the example for our discussion.
They are representative of patch-based and non-patch-based backdoor attacks, respectively.
We conduct experiments on the CIFAR-10 dataset (Krizhevsky, 2009) with
ResNet (He et al., 2016).
For both attacks, we inject a large number of poisoned samples to ensure a high attack success rate ( ≥ 99% ).
For each benign and poisoned image, we gradually enlarge its pixel values with multiplication.
We calculate the average confidence defined as the average probabilities of samples on the originally predicted label.
In particular, we select the label predicted upon the original sample as the originally predicted label for each varied sample and constrain all pixel values within [0 , 1] during the multiplication process.
More details are in our appendix.
Results.
As shown in Figure 2, the average confidence scores of both benign and poisoned samples decrease during the multiplication process under the benign model.
In other words, the predictions of modified benign and poisoned samples are changed during this process.
In contrast, poisoned and benign samples have significantly distinctive behaviors under attacked models.
Specifically, the average confidence of benign samples decreases whereas that of poisoned samples is relatively stable with the increase of multiplication times.
We call this phenomenon as scaled prediction consistency .
To further explain this intriguing phenomenon ( i.e. , scaled prediction consistency), we exploit recent studies on neural tangent kernel (NTK)
(Jacot et al., 2018) for analyzing the backdoor-infected models inspired by (Guo et al., 2022c), as follows:
Theorem 1.
Suppose the poisoned training dataset consists of N b benign samples and N p poisoned samples, i.i.d.
sampled from uniform distribution and belonging to K classes.
Assume that deep neural network f ( · ; θ ) be a multivariate kernel regression (RBF kernel) with the objective in Eq.
For a given attacked sample x ′ = ( 1 − m ) ⊙ x + m ⊙ t , we have: lim N p → N b C ( n · x ′ ) =
y t , n ≥ 1 .
In general, Theorem 1 reveals that when the amount of poisoned samples closes to the benign samples or the attacked DNN over-fits the poisoned samples, it will still constantly predict the scaled attacked samples ( i.e. , n · x ′ ) as the target label y t .
Its proof can be found in Appendix A.
[Figure]
S CALED P REDICTION C ONSISTENCY A NALYSIS (SCALE-UP)
Motivated by the phenomenon of scaled prediction consistency demonstrated in the previous section, we propose a simple yet effective black-box input-level backdoor detection, dubbed scaled prediction consistency analysis (SCALE-UP), in this section.
4. P RELIMINARIES
Defender’s Goals.
In general, defenders have two main goals, including effectiveness and efficiency .
Effectiveness requires that the detection method can accurately identify whether a given image is malicious or not.
Efficiency ensures that detection time is limited and therefore the deployed model can provide final results on time after the detection and prediction process.
Threat Model.
We consider backdoor detection under the black-box setting in machine learning as a service (MLaaS) applications.
Specifically, defenders can only query the third-party deployed model and obtain its predictions.
They do not have any prior information about the backdoor attack or the model.
In particular, we consider two data settings, including 1) data-free detection and 2) data-limited detection .
The former one assumes that defenders have no holding benign samples, while the latter one allows defenders to have a few benign samples from each class.
Note that we only assume to have the predicted label instead of the predicted probability vector in our method.
4. D ATA - FREE S CALED P REDICTION C ONSISTENCY A NALYSIS
As demonstrated in Section 3, we can use the average probability on the originally predicted label across its scaled images to determine whether a given suspicious image is malicious.
In general, the larger the probability, the more likely the sample is poisoned.
However, we can only obtain predicted labels while predicted probability vectors are inaccessible under our settings.
Accordingly, we propose to examine whether the predictions of scaled samples are consistent, as follows:
Let S denotes a defender-specified scaling set ( e.g. , S = { 3 , 5 , 7 , 9 , 11 } ).
For a given input image x and the deployed classifier C , we define its scaled prediction consistency (SPC) as the proportion of labels of scaled images that are consistent with that of the input image, i.e. ,
[Equation]
where I is the indicator function and |S| denotes the size of scaling set S .
In particular, we constrain n · x ∈ [0 , 1] during the multiplication process.
Once we obtain the SPC value of suspicious input x , we can determine it is malicious based on defender-specified threshold T .
If SPC ( x ) > T , we deem it as a backdoor sample.
4. D ATA - LIMITED S CALED P REDICTION C ONSISTENCY A NALYSIS
In our data-free scaled prediction consistency analysis, we treat all labels equally.
However, we notice that the SPC values of benign samples under attacked models are different across classes (as shown in Figure 3).
In other words, some classes are more consistent against image scaling compared to the remaining ones.
These benign samples with have high SPC values may be mistakenly treated as malicious samples, leading to relatively low precision of our method.
[Figure] [Figure]
In data-limited scaled prediction consistency analysis, we assume that defenders have a few benign samples from each class.
This setting has been widely used in existing backdoor defenses (Li et al., 2021b;
Guo et al., 2022c; Zeng et al., 2022).
In this paper, we propose to exploit a set of statics summary ( i.e. , mean µ and standard deviation σ ) of these local benign samples to alleviate this problem, inspired by (Ioffe & Szegedy, 2015).
We first estimate the statics summary for SPC values on samples from different classes, based on which to normalize the SPC value of suspicious input images according to their predicted labels.
Specifically, for each class i , we calculate its corresponding mean µ i and standard deviation σ i based on samples X i belonging to class i , as follows:
[Equation]
In the detection process, given a suspicious image x and the deployed model C , we normalize the
SPC value generated by data-free SCALE-UP based on its predicted label ˆ y ≜ C ( x ) , as follows:
[Equation]
Besides, for a more stable and effective performance, we automatically balance two terms in Eq.
(4) to make their values at the same level.
The main pipeline of our method is summarized in Figure 4.
E XPERIMENTS 5. M AIN S ETTINGS
Dataset and DNN Selection.
Following the settings in existing backdoor defenses, we conduct experiments on CIFAR-10 (Krizhevsky, 2009) and (Tiny) ImageNet (Russakovsky et al., 2015) datasets with ResNet (He et al., 2016).
Please find more detailed information in our appendix.
Attack Baselines.
In this paper, we evaluate our methods under six representative attacks, including 1)
BadNets (Gu et al., 2019), 2) label consistent backdoor attack (dubbed ‘Label-Consistent’)
[Table]
[Table] [Table]
(Turner et al., 2019), 3) physical backdoor attack (dubbed ‘PhysicalBA’)
(Li et al., 2020), 4) cleanlabel backdoor attack with targeted universal adversarial perturbation (dubbed ‘TUAP’)
(Zhao et al., 2020a), 5) WaNet (Nguyen & Tran, 2021), and 6) ISSBA
(Li et al., 2021c).
They are the representative of patch-based and non-patch-based backdoor attacks under different settings.
For each attack, we randomly select the target label and inject sufficient poisoned samples to ensure the attack success rate ≥ 98% while preserving the overall model performance.
We implement these attacks based on the open-sourced backdoor toolbox (Li et al., 2023).
We demonstrate the trigger patterns of adopted attacks for Tiny ImageNet in Figure 5.
More detailed settings are in the appendix.
Defense Baselines.
In this paper, we focus on the backdoor detection under the black-box setting where defenders can only query the deployed model and obtain its predicted label.
Accordingly, we compare our methods to ShrinkPad (Li et al., 2020), DeepSweep (Qiu et al., 2021), and artifacts detection in the frequency domain (dubbed ‘Frequency’)
(Zeng et al., 2021).
We also compare our methods to STRIP (Gao et al., 2021) that requires additional requirement ( i.e. , obtaining predict probability vectors).
We assume that defenders have 100 benign samples per class under the datalimited setting.
Please find more defense details in our appendix.
Settings for Evaluation Datasets.
Following the previous work (Lee et al., 2018), we use a positive ( i.e. , attacked) and a negative ( i.e. , benign) dataset to evaluate each defense.
Specifically, the positive dataset contains the attacked testset and its augmented version, while the negative dataset contains a benign testset and its augmented version.
The augmented datasets are created by adding small random noise to their original version.
The noise magnitude is set to 0.05.
In particular, adding these random noises will not significantly affect the attack success rate and the benign accuracy of deployed models.
The introduction of the augmented datasets is to prevent evaluated defenses from over-fitting the benign or the poisoned testsets.
Evaluation Metrics.
Following existing detection-based backdoor defenses (Gao et al., 2021; Guo et al., 2022c), we adopt the area under receiver operating curve (AUROC) (Fawcett, 2006) to evaluate defense effectiveness, while use the inference time for evaluating efficiency.
In general, the higher the AUROC and the lower the inference time, the better the backdoor detection.
5. M AIN R ESULTS
[Figure]
As shown in Table 1-2, all baseline detection methods fail in defending against some evaluated attacks .
Specifically, they have relatively low AUROC in detecting advanced non-patch-based attacks ( i.e. , WaNet and ISSBA).
This failure is mostly because these defenses have some implicit assumptions ( e.g. , the trigger pattern is sample-agnostic or static) about the attack, which are not necessarily true in practice.
In contrast, our methods reach promising performance in all cases on both datasets .
For example, the AUROC of our data-limited SCALE-UP is 0.5 greater than all baseline defenses in detecting WaNet on the Tiny ImageNet dataset.
Even under the classical patch-based attacks ( i.e. , BadNets, Label-Consistent, and
PhysicalBA), the effectiveness of our methods is on par with or better than all baseline defenses.
Our methods are even better than STRIP, which requires obtaining predicted probability vectors instead of predicted labels.
We also provide the ROC curves of defenses against all attacks in Appendix N. These results verify the effectiveness of our defenses.
Besides, we also calculate the inference time of all defenses under the same computational facilities.
In particular, we calculate the inference time of methods requiring to obtain the predictions of multiple images by feeding them simultaneously (in a data batch) into the deployed model instead of predicting them one by one.
Besides, we only report the inference time of our SCALE-UP under the data-limited setting, since both of them have very similar running times.
As shown in Figure 6, our method requires fewer inference times compared to almost all baseline defenses.
The only exception is ShrinkPad, whereas its effectiveness is significantly lower than that of our method.
Our detection is approximately 5% slower compared with the standard inference process without any defense.
These results show the efficiency of our SCALE-UP detection.
5. D ISCUSSION
In this section, we discuss whether our method is still effective under different (adversarial) settings.
5.3. D EFENDING AGAINST A TTACKS WITH L ARGER T RIGGER S IZES
Recent studies (Qiu et al., 2021) revealed that some defenses ( e.g. , ShrinkPad) may fail in detecting samples with a relatively large trigger size.
In this part, we use two patch-based attacks ( e.g. ,
BadNets and PhysicalBA) on the Tiny ImageNet dataset for discussion.
As shown in Figure 7(a), our methods have high AUROC values ( > 0 . 93 ) across different trigger sizes under both datafree and data-limited settings, although there are some mild fluctuations.
These results verify the resistance of our SCALE-UP detection to adaptive attacks with large trigger patterns.
5.3. T HE R ESISTANCE TO P OTENTIAL A DAPTIVE A TTACKS
Most recently, (Qi et al., 2023) demonstrated that reducing the poisoning rate is a simple yet effective method to design adaptive attacks for detection-based defenses, since it can reduce the differences between benign and poisoned samples.
Motivated by this finding, we first explore whether our
SCALE-UP methods are still effective in defending against attacks with low poisoning rates.
We adopt BadNets on the CIFAR-10 dataset as an example for our discussions.
In particular, we report the results of all poisoned testing samples and those that can be predicted as the target label, respectively.
We design this setting since attacked models may still correctly predict many poisoned samples even if they contain trigger patterns when the poisoning rate is relatively low.
As shown in Figure 7(b), the attack success rate (ASR) increases with the increase of the poisoning rate.
Our method can still correctly detect poisoned samples that can successfully attack the deployed model even when the poisoning rate is set to 0 .
4% where the ASR is lower than 70%.
In these cases, the AUROC > 0 .
Besides, our method can still reach promising performance (AUROC > 0 . 8 ) in detecting all poisoned samples.
These results verify the resistance of our defense to adaptive attacks with low poisoning rates, where attacked models don’t over-fit backdoor triggers.
[Figure]
To further evaluate the resistance of our SCALE-UP to potential adaptive methods, we evaluate it under the worst scenario, where the backdoor adversaries are fully aware of our mechanism.
Specifically, we design a strong adaptive attack by introducing an additional defense-resistant regularization term to the vanilla attack illustrated in Eq.
This regularization term is used to prevent scaled poisoned samples n · x ′ j being predicted as the target label y t , as follows:
[Equation]
Similar to previous experiments, we adopt BadNets to design the adaptive attack on the CIFARdataset.
As we expected, this method can bypass our detection resulting in a low AUROC ( i.e. , 0.467).
However, the adaptive attack would make the poisoned samples significantly more vulnerable to small random Gaussian noises .
As shown in Figure 7(c), random noises with a small magnitude ( < 0 .
3 ) will significantly reduce the attack success rate of the adaptive attack, while having minor adverse effects on the vanilla attack.
In other words, defenders can easily adopt random noises to defend against this adaptive attack.
We speculate that its vulnerability is mostly because the regularization term significantly constrains the generalization of attacked DNNs on the poisoned samples.
We will further explore its intrinsic mechanism in our future work.
5.3. T HE E FFECTIVENESS OF S CALING P ROCESS
Technically, the scaling process in our SCALE-UP detection can be regarded as a data augmentation method generating different modified versions of the suspicious input image.
It naturally raises an intriguing question: If other augmentation methods are adopted, is our SCALE-UP detection still effective?
Since flip operations and frequency domain analysis have been adopted in (Li et al., 2020; Zeng et al., 2021) for defense and proved to have minor benefits to detecting advanced backdoor attacks (Li et al., 2021c; Nguyen & Tran, 2021), we investigate the effectiveness of adding increasing magnitudes of random noise.
Due to the limitations of space, we include the detailed experimental design and evaluation in Appendix O.
C ONCLUSION
In this paper, we proposed a simple yet effective black-box input-level backdoor detection (dubbed
SCALE-UP) that can be used in real-world applications under the machine learning as a service (MLaaS) setting.
Our method was motivated by an intriguing new phenomenon (dubbed scaled prediction consistency ) that the predictions of poisoned samples are significantly more consistent compared to those of benign ones when amplifying all pixel values.
We also provided theoretical foundations to explain this phenomenon.
In particular, we designed our SCALE-UP detection method under both data-free and data-limited settings.
Extensive experiments on benchmark datasets verified the effectiveness and efficiency of our method and its resistance to potential adaptive attacks.
E THICS S TATEMENT
DNNs have been widely and successfully adopted in many mission-critical applications.
Accordingly, their security is of great significance.
The existence of backdoor threats raises serious concerns about using third-party models under the machine learning as a service (MLaaS) setting.
In this paper, we propose a simple yet effective black-box input-level backdoor detection.
Accordingly, this work has no ethical issues since it does not reveal any new security risks and is purely defensive.
However, we need to notice that our methods can only be used to filter poisoned testing samples whereas they do not reduce the intrinsic backdoor vulnerability of deployed models.
Our defense also couldn’t recover trigger patterns.
People should not be too optimistic about eliminating backdoor threats.
We will further improve our method by exploring how to recover triggers.
A CKNOWLEDGMENT
This work is mainly supported by Samsung Research America, Mountain View, CA and partially supported by NSF CNS 2135625, CPS 2038727, CNS Career 1750263, and a Darpa Shell grant.
R EPRODUCIBILITY S TATEMENT
We have provided detailed information on datasets, models, training settings and computational facilities in our main manuscript and appendix.
The codes for reproducing our main experiments are also open-sourced at https://github.com/JunfengGo/SCALE-UP .
A PPENDIX A T HE O MITTED P ROOF OF T HEOREM
Theorem 1.
Suppose the poisoned training dataset consists of N b benign samples and N p poisoned samples, i.i.d.
sampled from uniform distribution and belonging to K classes.
Assume that deep neural network f adopt RBF kernel and cross-entropy loss with the optimization objective in Eq.1.
For a given attacked sample x ′ = ( 1 − m ) ⊙ x + m ⊙ t , we have: lim N p → N b C ( n · x ′ ) =
y t , n ≥ 1 .
Proof of Theorem 1:
Following (Guo et al., 2022c), we have the regression solution for NTK is:
[Equation]
where ϕ t ( · ) ∈ R is the predictive probability output of f ( · ; θ ) for the target class t and y i is the corresponding one-hot label.
K ( x , x i ) =
e − 2 γ
|| x − x i
|| 2 ( γ > 0 ).
Since the training samples are evenly distributed, there are N b k benign samples belonging to y t .
Without loss of generality, we assume the target label y t = 1 while others are 0.
Then the regression solution can be converted to:
[Equation]
For a given backdoored sample x ′ = (1 − m ) ⊙ x + m ⊙ t , we can simplify Eq.
(7) as:
[Equation]
we here remove the term (cid:80) N b /k
i =
1 K ( x ′ , x i ) .
This is because x ′ typically don’t belong to the target y t and (cid:80) N b /k
i =
1 K ( x ′ , x i ) << (cid:80) N p i =1 K ( x ′ , x ′ i ) , otherwise the attacker has no incentive to craft poisoned sample.
When N p close to N b , which implies that the poisoning rate close to 50% , the attacker can achieve the optimal attack efficacy (Liu et al., 2018b; Gu et al., 2019; Li et al., 2021c).
Given K ( x , x i )
=
e − 2 γ
|| x − x i
|| 2 ( γ > 0 ), if N p = N b , we have:
[Equation]
If n = 1 , we can easily obtain that:
[Equation]
Since the internal term (1 − e − 2 γ
|| m ⊙ ( t − x i ) || 2 ) can be always larger than 0, thus it is clear that f ( x ′ ) =
y t , which is also consistent with the practice.
However, when n > 1 , to compare (cid:80) N p i =1 K ( n · x ′ , x
[Equation]
i ) and (cid:80) N b i =1 K ( n · x ′ , x i )
[Equation]
we have:
[Equation]
Regarding the internal term || m ⊙ ( n · t − x i ) ||
2 − || m ⊙ ( n − 1) t ) ||
[Equation]
we have:
[Equation] [Equation]
where δ i,j,k is the pixel-level residue between the trigger and benign samples.
We assume that the δ i,j,k t j,k close to a zero mean for inputs, thus we can rewrite Eq.(12) as follows:
[Equation]
Put Eq.
(19) and Eq.
(9) together, we know that ϕ t ( n · x ′ ) ≥ 0 .
5 , as N p → N b , we have:
[Equation]
□
B T HE D ETAILED C ONFIGURATIONS OF THE E MPIRICAL S TUDY
We adopt BadNets (Gu et al., 2019)) and ISSBA (Li et al., 2021c) as the example for our discussion.
They are the representative of patch-based and non-patch-based backdoor attacks, respectively.
We conduct experiments on the CIFAR-10 dataset (Krizhevsky, 2009) with ResNet-34 (He et al., 2016).
For both attacks, we inject a large number of poisoned samples to ensure a high attack success rate ( ≥ 99% ).
For each benign and poisoned image, we gradually enlarge its pixel values with multiplication.
We calculate the averaged confidence defined as the averaged probabilities of samples on the originally predicted label.
In particular, we select the label predicted upon the original sample as the originally predicted label for each varied sample and constrain all pixel values within [0 , 1] during the multiplication process.
In particular, we follow previous works (Gu et al., 2019; Li et al., 2021c) to implement the backdoor attacks.
Specifically, the trigger for BadNets is a 4 × 4 square consisting of random pixel values; the trigger of ISSBA is generated via DNN-based image steganography ( et al., 2020)
Both attacks are implemented via BackdoorBox (Li et al
[Table]
C T HE D ETAILS FOR T RAINING A TTACKED M ODELS
We train backdoor-infected models using BackdoorBox (Li et al., 2023).
We set the training epoch as 200 and the poisoning rate as 5 − 10% for each attack to ensure a high attack success rate.
In particular, except for PhysicalBA, we don’t involve additional data augmentation in training infected models as we want to better reveal the properties of various backdoor approaches.
For each infected model, we randomly select infected labels to ensure their predictions on benign inputs are similar to the benign models, which ensures the stealthiness of backdoor attacks.
Regarding the data-limited scenario and ablation study, we intentionally affect multiple labels to ensure the infected models similar to benign models except for the Trojan behaviors.
Specifically, we inject less amount( ≤ 5% ) of poisoned samples to affect labels other than the target label.
This is because previous work (Guo et al., 2022c) found that certain dense backdoor attacks ( e.g. , ISSBA, WaNet) would make the infected DNNs sensitive to noisy or out-of-distribution samples on CIFAR-10 dataset.
Accordingly,they are less stealthy and are easy to be detected during the sampling process of the datalimited setting and settings of our ablation study.
As such, in these settings, to evaluate SCALE-UP in a rather practical scenario, we train infected DNNs to have similar behaviors on noisy samples as the benign DNNs.
The details for each dataset are included in Table 3.
C. T HE A CCURACY AND A TTACK S UCCESS R ATE (ASR) FOR E VALUATED M ODELS
The accuracy and ASR for the evaluated models for each task in included in Table 4.
[Table]
[Table]
D T HE D ETAILED C ONFIGURATIONS FOR B ASELINE D EFENSES
• STRIP: We implement STRIP following their official open-sourced codes * .
• ShrinkPad: We implement ShrinkPad following their official open-sourced codes † .
• Frequeny: We implement Frequency approach following their official codes ‡ .
• DeepSweep: We implement DeepSweep using Scipy package to remove the highfrequency noise and use torchvision.transforms and keras.preprocess packages to conduct transformation to inputs.
Notably, we don’t apply finetune process within DeepSweep since we only focus on the black-box detection scenarios.
[Figure]
E T HE D ESCRIPTIONS FOR M AIN E VALUATION M ETRIC
• The receiver operating curve (ROC) shows the trade-off between detection the success rate for poisoned samples and detection error rate for benign samples across different decision thresholds T under infected-DNNs.
• Inference Time: we implement each approach under the platform with one NVIDIA GPU 1080 Ti and a Intel(R) Xeon(R) CPU E5-2650 v4 @
2.20GHz with batch size = 1 .
We test the inference time of each approach with an average of 1,000 runs.
F S ETTINGS F OR M EASURING THE I NFERENCE T IME
Since we focus on defending against backdoor attacks in the inference phase, we here measure the inference time by:
• Identifying whether the input sample is poisoned or not.
• If the input is a benign sample, we next should use the target model to predict it.
For STRIP, ShrinkPad, DeepSweep, and SCALE-UP, we leverage the target model’s prediction on the (augmented) inputs for defense purpose, which means the input can be identified and predicted at the same time.
As for Frequency, which leverages a secondary neural network to predict the frequency domain of each given input.
However, if the input is identified as benign, the target DNNs should also deliver prediction on it.
We here assume the benign and poisoned samples have equal possibilities.
Therefore, we calculate the inference time for Frequency as follows:
[Equation]
While for other approaches, we measure their inference time via:
[Equation]
In particular, we calculate the inference time of methods required to obtain the predictions of multiple images by feeding them simultaneously (in a batch) into the deployed model instead of predicting them one by one.
This approach is feasible since defenders can easily and efficiently obtain all of them before feeding them into the deployed model.
G P ERFORMANCE U NDER M ULTIPLE -B ACKDOOR T RIGGERS W ITHIN A S INGLE I NFECTED L ABEL
[Figure]
Consistent with (Guo et al., 2022c; Wang et al., 2019), we also evaluate the efficacy of SCALE-UP under a more challenging scenario where multiple backdoors are embedded within a single target label.
We randomly select a label as the infected label and inject various types of poisoned samples in the training phase.
We inject arbitrary amounts of poison samples for each backdoor trigger to ensure the attack efficacy ASR ≥ 99% .
The demonstrations for used backdoor triggers are shown in Figure 8.
Under such considered scenario, we evaluate our SCALE-UP on CIFAR-10 and Tiny
ImageNet datasets using ResNet-34.
5 Number of injected triggers 0.
D e t e c t i on
P e
r
f
o r m an c e
( A
UR O C )
Data-free
Data-limited (
[Figure]
CIFAR5 Number of injected triggers 0.
D e t e c t i on
P e
r
f
o r m an c e
( A
UR O C )
Data-free
Data-limited (
[Figure]
TinyImageNet
As shown in Figure 9, SCALE-UP performs resilient to the increasing number of injected backdoor triggers.
This may be caused by infected models already generalized for backdoor triggers.
H T HE P ERFORMANCE A GAINST M ULTIPLE I NFECTED L ABELS
We also test SCALE-UP under the scenario where the suspicious model has multiple infected labels.
Under this scenario, we test SCALE-UP on CIFAR-10.
This is because models on Tiny ImageNet would be prone to multiple infected labels, as reported by (Guo et al., 2022c), affecting more than 14% labels can make the accuracy significantly drop ≥ 3% .
We implement BadNets as backdoor attacks, the trigger size is 4 × 4 .
The results are shown in Figure 10.
These results show that affecting multiple infected labels could slightly reduce the performance of SCALE-UP.
Besides, the data-limited scenario performs better than the data-free scenario.
However, even with 100% labels are infected, SCALE-UP can still perform effectively with AUROC ≥ 0 .
I I MPACTS FOR THE N UMBER OF C OEFFICIENTS
We test SCALE-UP on six attacks with varying n .
We here use ResNet-34 on TinyImageNet for evaluation.
As shown in Figure 11, we find that the performance of SCALE-UP increases along with n increasing.
Moreover, we find that SCALE-UP performs more sensitive on n for the TUAP attack compared with other attack techniques.
Moreover, we find that SCALE-UP performs similarly sensitive on n in both data-limited and data-free settings.
In most settings, with n ≥ 11 SCALE-UP can achieve optimal performance on six different attacks.
J I MPACT FOR THE S IZE OF L OCAL S AMPLES
We also test the sensitivity of SCALE-UP on the size of local samples per label under the datalimited setting.
We test SCALE-UP on Tiny ImageNet using ResNet-34 against six attacks.
The results are shown in Figure 12.
We can see that with the size of local samples increases, the performance of SCALE-UP improves and achieves optimal performance when the size ≥ 100 .
K P ERFORMANCE UNDER S OURCE - LABEL - SPECIFIC B ACKDOOR S CENARIOS
The Source-label-specific (Partial) backdoor scenarios is that the backdoor attacks can perform effectively when it applies to images of a certain specific class.
Such a scenario makes backdoor attacks very hard to detect (Wang et al., 2019; Gao et al., 2021), thus the attacker may have a great incentive to implement such a backdoor attack in the real world.
Therefore, we evaluate SCALE-UP under such a practical scenario and compare SCALE-UP with previous work.
We test SCALE-UP using ResNet-34 on CIFAR-10 and Tiny ImageNet.
As shown in Table 5, we find that most defense
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
i on
[Table]
o r m an c
[Table]
( A
UR O
C )
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
c on
P o m an c e ( A
UR O
C )
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure] [Table] [Figure] [Table]
[Figure] [Table] [Figure]
e
[Table]
[Figure]
[Table]
[Figure]
[Table]
e
[Figure]
approaches perform resilient against the partial backdoor attack except STRIP (Gao et al., 2021).
This is because STRIP assumes the trigger can perform effectively across various images.
Under this scenario, SCALE-UP can outperform all baseline defenses.
L T HE R OBUSTNESS OF SCALE-UP
Since SCALE-UP is an inference-phase backdoor defense approach, it is necessary to investigate the robustness of SCALE-UP on benign and poisoned samples.
Following previous work (Du et al., 2020), we evaluate the robustness of our approach by testing different magnitudes of noisy inputs.
Notably, we test SCALE-UP on benign and poisoned samples, respectively, which is because they exhibit different robustness under random noise as we show in Section 5.3.3.
Moreover, the magnitudes of added random noise ensure the classification accuracy and attack success rate for benign and poisoned samples.
The results are shown in Figure 13.
We test our approach using ResNet-
[Figure] [Figure]
on the TinyImageNet task.
The noise is randomly sampled from Gaussian distribution and we intentionally filter the failed poisoned samples.
We only test BadNets and PhysicalBA since only these
[Table] [Table]
[Table]
two attacks perform robustness against random noise, as illustrated in Section 5.3.3.
We find that our approach is robust against noisy poisoned samples.
M A DDITIONAL R ESULTS UNDER VGG A RCHITECTURE
In our main manuscript, we evaluate our method under the ResNet architecture.
In this section, we conduct additional experiments under VGG-19 (BN) on Tiny ImageNet, to verify that the phenomenon of scaled prediction consistency is valid across different model architectures.
As shown in Figure 14, the scaled prediction consistency still holds in all cases.
Specifically, the average confidence of benign samples decreases significantly faster than that of poisoned ones with the increase in multiplication time.
Besides, as shown in Table 6, our methods are still better than all baseline defenses.
These results verify the effectiveness of our methods again.
N T HE ROC C URVES OF D EFENSES
To better compare our method with baseline defenses, we also visualize the ROC curves of defenses (as shown in Figure 15-16) under each attack on both CIFAR-10 and Tiny ImageNet in this section.
O D ETAILS FOR THE E FFECTIVENESS OF S CALING P ROCESS
Specifically, we design the SCALE-UP variant by replacing the scaling process with adding the same varied magnitudes of random noise to the given inputs.
As shown in Table 7, using random noises is far less effective compared to the standard SCALE-UP methods, especially in detecting advanced attacks ( i.e. , WaNet and ISSBA).
We speculate that it is mostly because they adopted invisible fullimage size trigger patterns and therefore the trigger-related features are less robust.
Although we currently fail to provide theoretical analysis for the aforementioned phenomena, at least they verify the effectiveness of our scaling process.
We will further discuss it in our future work.
P P OTENTIAL L IMITATIONS AND F UTURE W ORK
Our work is the first black-box label-only input-level backdoor detection and early-stage defenses under the black-box setting.
Accordingly, we have to admit that our work still has some limitations.
Multiplication Times 0.
A v e r age
C on ﬁ den c e
Poisoned Sample Benign Sample (a) BadNets
Multiplication Times 0.
A v e r age
C on ﬁ den c e
Poisoned Sample Benign Sample (b) TUAP
Multiplication Times 0.
A v e r age
C on ﬁ den c e
[Figure]
Multiplication Times 0.
A v e r age
C on ﬁ den c e
Poisoned Sample
[Figure]
Sample (d) PhysicalBA
Multiplication Times 0.
A v e r age
C on ﬁ den c e
[Figure]
(e) ISSBA
Multiplication Times 0.
[Figure]
age
[Figure]
Firstly, our defense requires that the attacked DNNs overfit their poisoned samples.
This assumption or potential limitation is also revealed by our theoretical analysis in Section 3.
In other words, if the attack success rate of a malicious model is relatively low, the detection performance of our SCALEUP defense may degrade sharply.
Secondly, we found that our SCALE-UP detection may fail when defending against attacks in some cases of simple tasks ( e.g. , MNIST and GTSRB).
We speculate that it is mostly because attacked DNNs also overfit to benign samples due to the lack of diversity and simplicity of the dataset, making them indistinguishable from some poisoned samples when analyzing the scaled prediction consistency.
We will further explore the latent mechanisms of these limitations and alleviate them in our future work.
Besides, regarding another future direction of our methods, we intend to generalize and adopt them to more settings and applications, such as continual learning (Wang et al., 2022b), non-transferable learning (Wang et al., 2022a), federate learning (Dong et al., 2022), audio signal processing (Zhai et al., 2021;
Guo et al., 2022a;b), and visual object tracking (Li et al., 2022b).
We will also evaluate our methods under other DNN structures ( e.g. , ViT (Tu et al., 2022) and GCN (Zhao et al., 2020b)).
[Table] [Table]
t i v e
[Table]
a
[Table]
e
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Figure]
r u e
P o s i t i v e
R a t e
[Figure]
[Table] [Table]
t i v e
[Table]
a
[Table]
e
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Figure]
r u e
P o s i t i v e
R a t e
[Figure]