Randomized Channel Shufﬂing: Minimal-Overhead Backdoor Attack Detection without Clean Datasets
Abstract
Deep neural networks (DNNs) typically requires massive data to train on, which is ahurdle for numerous practical domains.
Facing the data shortfall, one viable optionis to acquire domains-speciﬁc training data from external uncensored sources,such as the open web or the third-party data collectors.
However, the qualityof such acquired data is often not rigorously scrutinized, and one cannot ruleout the risk of “poisoned” examples being included in such unreliable datasets.
That casts a new “self-supervised” backdoor detection setting that was hardlyexplored by prior arts: in particular, for one speciﬁc collected dataset, users (1)have no prior knowledge whether it is poisoned, or on the target class/percentageof poisoned samples, and (2) have no access to a clean sample set from the samedomain distribution, nor any trusted model trained on such clean data.
Thisnew backdoor threat hence incurs overlooked risks for many data-hungry, orhigh-stake applications.
This paper reports the ﬁrst pilot study on this newsetting, by investigating the contrasting channel-level statistics between backdoortrigger and clean features, and consequently, how the former can be differentiatedby progressive channel shufﬂing.
The method leads to detecting the backdoortargeted class with only a few feedforward passes, incurring minimal overheads,and demanding no clean sample nor prior knowledge.
Extensive experiments areconducted with two datasets (CIFAR-10, GTSRB), two architectures (AlexNet,
ResNet-20), and three strong attacks (BadNets[1], clean label attack [2], and
WaNet [3]).
Results consistently endorse the effectiveness of our technique inbackdoor model detection, with margins of 0 .
291 ⇠ 0 .
640 AUROC 1 over thecurrent state-of-the-art methods.
Codes will be released.
Introduction
Numerous deep learning based methods signiﬁcantly outperform traditional methods and becomeuseful in many critical domains.
To achieve good results, the enormous deep neural networks (DNNs)require a large amount of training data.
Due to the absence of large enough self-gathered datasets inmany applications, many users are forced to use data samples from external sources, e.g., third-partydata collection companies and online websites.
However, these external sources are untrustworthybecause the companies or the uploaders could be attackers who inject “poison” into the datasets bymaliciously crafting the data samples thus resulting in unreliable models.
Recently, such poisoning attacks are found fatal speciﬁcally to deep learning applications [4, 5].
Even worse, it is hard to determine whether a DNN or the dataset that it is trained on is attackedor not, due to the lack of transparency and interpretability.
Users are left in desperate needs for areliable method that can achieve such a goal, which we call attack detection .
While efforts from researchers have been put into this direction, we ﬁnd that most existing work of attack detection tryto obtain a classiﬁer that predicts if a given data sample is poisoned .
This classiﬁer is trained on ahuman-crafted dataset containing both poisoned and clean samples, with the labels being whether asample is poisoned.
This approach assumes the access to at least a small portion of clean samplesas reference [6, 7, 8, 9, 10, 11].
Unfortunately, this assumption does not always hold in real-worldscenarios.
We lay out three-fold reasons:
• Users have zero control nor prior knowledge over their received or collected datasets in the wild.
Of course, the users may not know whether such a dataset is attacked or not.
Beyond that, they
also do not know (even approximately) the portion of samples that are maliciously crafted, nor
can they reliably draw a small set of clean samples out of the dataset.
This directly challenges
most of existing work for attack detection.
• Public datasets online are not necessarily helpful, because they can be totally different fromthe target datasets (potentially attacked), in terms of content and statistics.
Samples in thepublic datasets may even be out-of-distribution samples, and therefore provide little meaningfulinformation for the users to tell the if their speciﬁc datasets are attacked.• Labeling a small clean set with human labor is also not a feasible ﬁx.
On one hand, the “backdoor" changes made to to data samples are often hard to perceive or notice by human,compared to other benign image artifacts.
On the other hand, the attacked DNNs behavecompletely normal on the clean samples.
It is thus difﬁcult to distinguish clean samples from
the attacked ones by simply investigating superﬁcial behaviors such as model performance and
testing accuracy, especially when the testing set is likely to be attacked too.
The work derives a new attack detection method in a brand-new setting , i e., a “self-supervised” post- training attack detection scenario, where we assume no prior knowledge over the datasets of interest, nor the access to a collection of clean samples, nor the need to retrain models
We propose novel hypotheses about the statistics of the exploited features and how they behave differently from normaldata features.
Then we design a shufﬂing-based method that stratiﬁes attacked and clean samplesaccording to their different behaviors towards the shufﬂing operation.
The validity of the hypothesesand the efﬁcacy of our proposed method are supported by extensive empirical observations.
1.1 Background and Our Contributions
We focus ourselves on the backdoor attack , which is notably stealthy among various poisoningattacks.
Backdoor attack methods poison data samples by injecting small triggers into and applyingcertain transformations to them.
The triggers and transformations are usually unnoticeable to humanperception.
The poisoned dataset is called a backdoored dataset.
A backdoored dataset usuallycontains both poisoned samples and benign or clean samples that are not contaminated.
Traditionalbackdoor attackers will also maliciously label the poisoned data into a target class [1, 4, 12, 3], whichcorresponds to scenarios where the attackers have full control over the datasets.
More recent work[2, 13] does not require this control and only craft the raw data.
For both types of backdoor attackmethods, the backdoored model , i.e., the DNN that is trained on the backdoored dataset, will behavenormally on benign inputs while classifying most malicious inputs into chosen target classes.
Many detection methods have been proposed to tackle the backdoor attack [14, 15, 16, 10, 17, 9, 6,7, 18, 19, 8, 20, 21, 19, 11].
However, most of them assume the access to a hold-out dataset onlycontaining benign samples.
A common practice by previous work is to learn a predictor of a certainform on this assumed training set to distinguish between benign and backdoored samples.
However,these methods totally fail in the scenarios we discussed above, which are much more realistic as usersusually have little explicit control over the cleaniness of datasets that they can receive or collect.
In this work, we propose that it is still possible to detect backdoor attacks without access to such atraining set but let the backdoor samples reveal themselves.
Laid on the foundation of this work isour core hypotheses over the unique channel-level distribution of the features of backdoored samples.
Trigger Feature Hypothesis We hypothesize that the trigger features are sparsely encoded in only afew channels, while clean image features need to be encoded across many channels .
This is a keydifference from normal data features that are presumably distributed more evenly across channels,which indicates that these two types of features might behave differently in certain situations, leadingto our main technical contribution.
Illustrations of triggers are provided in App. A3.
Feature Differentiation Based on the the proposed hypotheses, we devise a novel randomizedchannel shufﬂing mechanism which introduces minimal computation cost to reveal trigger features.
Speciﬁcally, given an unreliable dataset and a model trained by it, we control the channel shufﬂingstrength and observe how representations vary according to it.
By observing the abnormally differentvariation, we can detect the target class with only a few feedforward passes.
In addition, we further utilize the proposed mechanism to separate those clean image features andtrigger features.
Speciﬁcally, we retrain the model using the channel shufﬂing operation, and thenrecover triggers based on the shufﬂe model, obtaining more effective trigger than [6].
We summarize our novel contributions as follows:
• We raise that the unreliability of training data source has become a serious security threat.
We ﬁnd that clean image features and trigger features are different in terms of their activationdistribution in channels, as well as their sensitivity to channel re-ordering.• Utilizing trigger features’ unique characteristic, we propose an effective, generalizable approachwith minimal overhead based on channel shufﬂing to help data users examining datasets.
Werandomly shufﬂe weights of convolutional layers at channel level, revealing the secretly hiddentrigger features.
We also use the proposed shufﬂing method to retrain the models, helpinggenerate more effective triggers.• On three datasets (CIFAR-10, GTSRB, TinyImageNet), three architectures (AlexNet, ResNet-20,SENet-18), and three attacks (BadNets[1], clean label attack [2]
and WaNet [3]), experimentresults consistently endorse the effectiveness of our technique in backdoor model detection, e g.,0 .
331 ⇠ 0 .
640 higher AUROC than NC [6], and 0 .
291 ⇠ 0 .
543 higher than STRIP [8].
Related Work
Backdoor Attack and Defense.
Several backdoor attacks on deep learning models have beenproposed recently.
The attack features stealthiness since the attacked model will behave normallyon clean images, but classify maliciously crafted images from any source class into the maliciouslychosen target class.
Among them, most backdoor attacks can be classiﬁed as trigger-driven attacks.
Speciﬁcally, during the training phase, to launch an attack, the adversary injects an arbitrary triggerinto a small fraction of training pictures and maliciously label them to the target class.
Traditionaltriggers are a pre-deﬁned pattern which can simply stamp on benign images [1, 4].
Many advancedtriggers injection methods are proposed recently [12, 3] for stealthiness under both human inspectionand former detection methods [6, 7, 8].
Another category of backdoor attack is clean-label attack[2, 22, 13, 23, 24, 25, 26], which was formulated without control over the labeling function.
Amongthem, [23] targeted on large scale dataset and scale up the poisoning attack method via gradientmatching.
Additionally, both [22] and [24] focused on improving the transferability of attack methods,while [25] utilized meta-learning and proposed an effective and transferable poisoning methods.
Earlier works detect and mitigate backdoor models based on abnormal neuron responses [14, 16, 19],feature representation[15], entropy [8], and evolution of model accuracy [17].
[20] model thedistribution of Trojan attacks by meta neural analysis, [18] utilized Deep k-NN with focus onclean-label attack, [21] tried to learn the Universal Litmus Patterns (ULPs) from poisoned dataset.
Another category of work [6, 7, 19, 9, 10, 11] focus on recovering the trigger and detect backdoormodels based on the properties of the recovered triggers.
Utilizing clean testing images, Neural
Cleanse (NC) [6, 7] calculates minimal perturbation that causes misclassiﬁcation towards everyputative incorrect label.
Backdoor model detection is completed by MAD outlier detection, due to theobservation that the size of minimal perturbation to classify images from all classes to the target classis markedly smaller than that to other classes.
However, the aforementioned methods suffer fromoccasional failures in detecting backdoor models.
In addition, most methods except [19] that fall inthis category generally assume the defender has a clean validation set, which may not be practical.
Instead, we do not assume we can obtain a clean validation set for effective detection.
We also clarify the difference between our new setting, and the more classical backdoor detectionsetting as adopted in [27, 18, 28].
Those prior arts perform pre-training detection of backdoortraining samples, without needing clean validation sets either.
They all follow the same workﬂow:ﬁrst detecting the poison examples in the training set, then removing them prior to training, andﬁnally training the model using the ﬁltered set.
As the downside, to ﬁnd out whether a pre-trainedmodel has been poisoned or not, those methods also have to (re)train from scratch, which may oftenbe too costly or impractical.
In contrast, [8] and our method belong to post-training detection thatcan detect from a given pre-trained model without costly (re-)training from scratch.
That is more practical when the model is already actively deployed in the ﬁeld.
While [8] needs a clean “reference”set, our method requires no prior-training sample selection and (re-)training (hence differing from[27, 18, 28]), while eliminating the necessity of clean samples (hence differing from [8]).
Channel Shufﬂing.
The idea of channel shufﬂing has mostly been used in efﬁcient model design,intending to enable information ﬂow between branches and consequently strengthen representation.
Speciﬁcally, aiming at high efﬁciency, [29] and [30] utilized channel shufﬂing method to mitigate theproblem induced by group convolution[31, 32].
Similarly, [33] included channel split and shufﬂemethod to reduce computation cost while maintaining higher segmentation accuracy.
[34] includedchannel shufﬂing to enable information communication between different sub-features.
Different fromaforementioned works, [35] adopted channel shufﬂing method with randomness as a regularizationtechnique, in order to reduce overﬁtting and enhance generalization capabilities.
Based on our observation that clean image features and trigger features have different channel distributions which is shown in Section 3.1, we formulated the randomized channel shufﬂing mechanismto reveal trigger features, detecting backdoor models.
To the best of our knowledge, we are the ﬁrstto connect channel shufﬂing and feature differentiation goal.
Shortcut Features.
Several recent works [36, 37] analyzed that trigger features can serve as“shortcut” features [5].
Such shortcut features are observed to be simpler and easier to learn thanthe discriminative clean image features [38].
In addition, [16] observed that a small number ofneurons are activated by backdoor images but remain dormant for clean images.
In contrast, [39]demonstrated the importance of the combination of channels when encoding complex and diversiﬁedimage features on large datasets.
Methodology
In Section 3.1, we will ﬁrst present our main hypothesis on trigger features’ unique characteristicscompared to clean image features.
In Section 3.2, we propose our backdoor model detection methodusing randomized channel shufﬂing mechanism, directly motivated by the hypothesis.
In Section 3.3we further show that we can improve trigger recovery with the proposed channels shufﬂing scheme.
3.1 Differentiating Trigger and Image Features: A Hypothesis
Authors of [5] pointed out that DNNs tend to unintendedly exploit features unrelated to the task ofinterest to achieve good performance.
These “shortcut” features are usually simpler and easier tolearn [38] but highly correlated to the task-related features, such as sky background and the birdobjective.
It is then veriﬁed in [36, 37] that the backdoor triggers play a similar role as shortcutsexploited by the backdoored models.
Moreover, a small channels are observed to remain dormant for clean images but are activated bybackdoor images [1, 16].
Considering all these observations, we conjecture that the trigger featureshave a key difference from clean images features in terms of channel-level activation, and raise thefollowing hypothesis : Trigger features are sparsely encoded and activated in only a few channels,while clean image features need to be encoded across many channels for the effective classiﬁcation.
[Table]
To validate the hypothesis, we investigate the activation level of a speciﬁc channel, deﬁned as theaverage of elements in its (output) feature map.
Ineach layer, we consider channels with activationlevels larger than a ratio p of the highest activation level as ‘activated’.
We then observe howchannels are activated differently for clean andbackdoor images.
In Figure 1, we take the secondlast layer in a model backdoored by BadNets [1]as example.
When the input is backdoored, onlyone single channel shows exceptionally high activation level.
In contrast, a clean input has more balanced activations across all channels.
In Figure 2,we have consistent observations on more advanced backdoor methods such as clean label attack [2]and transformation-based attack, WaNet [3].
Moreover, we can see that the sparse encoding of triggerfeatures is more obvious in latter layers, which are known to encode more discriminate semanticfeatures while the shallow layers usually encode histogram features.
[Figure]
[Figure]
3.2 Trigger Feature Revealing via Channel Shufﬂing
The next question is, what can be implied from the signiﬁcant imbalance in activation distribution oftrigger features?
Consider the encoding of a poisoned image at a latter layer in backdoored model,where only one channel is vital with a high activation level.
This channel needs to match well withconvolutional weights in the next layer for the attack to succeed.
However, if we deliberately create amismatch between them, the lack of a second “vital channel” will lead to unsuccessful encoding withhigh probability.
In contrast, channel features of clean images are activated more evenly, leaving a lotof backup channels to substitute in when mismatch is created.
Motivated by this insight, we designa randomized channel shufﬂing mechanism to manually induce a mismatch between the activatedchannels and channel weights, and propose a way to observe how much the encoding process is beingaffected, thus revealing trigger features.
Randomized Channel Shufﬂing with Controlled Strength. Consider the convolutional weight ✓ l atthe l th layer of shape C l out ⇥ C l in ⇥ F l ⇥ F l , where C l out and C l in are the number of output channelsand input channels, respectively; F l is the kernel size.
We shufﬂe the order of input channels (thesecond dimension) of ✓ l .
The overview of randomized channel shufﬂing is illustrated in Figure 3.
Let ⇥ = { ✓ 1 , . . .
, ✓ L } denote the set of all weights in the network.
We create a shufﬂed version ofitself, ⇥ ( n ) by randomly shufﬂing the last n layers convolutional layers, n = 0 , 1 , ..., L  1 , sincetrigger features show more obvious sparsity characteristic in the latter layers according to Section 3.1.
We control the strength of channel shufﬂing by setting n for shufﬂing; the larger n the strongershufﬂing is applied.
Then, for a given input x , we calculate its representation (the output of the lastconvolutional layer) using ⇥ and ⇥ n , denoted as f ⇥ ( x ) and f ⇥ ( n ) ( x ) , respectively, where f is therepresentation extraction part of a model.
Finally, we compute the representation shift s n using the` 2 -norm distance: s n ( x ) =
k f ⇥ ( x )  f ⇥ ( n ) ( x ) k 2 .
Detect the Class-wise Abnormal Variation.
First, for a dataset D with K class, we split it into Ksubsets according to the labels.
For D k , the subset with class k , we calculate the standard deviationof representation shift caused by the shufﬂing operation on the last n layers, over all samples inthat subset, denoted by y k ( n ) =
std x i {+
+}
D k { s n ( x i ) } .
Then, by changing n , we can obtain a curvey k
[0 : n ] for class k showing the variation of standard deviation of representation shift versus differentstrengths of channel shufﬂing.
Finally, We can detect the target class by observing the abnormal curve,and its rationale is rooted in our sparsity hypothesis in 3.1.
Note that using mean of representationshift instead of using standard deviation still works for detection, and results are provided in App. A3.
We ﬁx the range of n (maximum layer number to shufﬂe, starting from the top/last layer) as four, andwe will analyze the choice in Section 4.4.
To describe how a speciﬁc curve y k [0 : n ] deviates from a cluster of curves { y j [0 : n ] , j =0 , ..., K  1 } and how to detect anomaly, we formulate the following two metrics: ( i ) Distance : We use the ` 1 distance between the curve and the median to measure how far the curveis away from the cluster, formulated as  d = k y k [0 : n ]  median j { y j [0 : n ] , j = 0 , ..., K  1 } k 1 .
( ii ) Tendency : First, we preprocess each curve to achieve ‘zero-mean’ and get ˆ y j [0 : n ] .
Then, wecalculate the standard deviation of the difference between the curve and the median.
To be speciﬁc, t = std n { ˆ y k [0 : n ]  median j { ˆ y j [0 : n ] , j = 0 , ..., K  1 } } .
Then, we formulate  dev =
d +  ⇤  t to take both deviation in distance and tendency of a curve intoaccount, and  is to balance two factors.
A larger value of  dev imply that its corresponding curve isless similar to others, and more likely to be the target class.
We set  as 0.01 for main experiments,but we will analyze the role of  in Section 4.4, showing the effectiveness of the proposed method isnot sensitive to the choice.
After calculating  dev for each curve, we use Median Absolute Deviationto detect outliers, which is a common choice for numerous backdoor detection methods[6, 7].
Aanomaly index can be obtained for each data point, measuring the probability of the data point beingan outlier and the curve relating to the target class.
By properly setting threshold, the target classand backdoor model can be detected.
In Section 4.2, we will measure the detection ability of ourproposed method by AUROC.
In addition, our method only need a few feedforward passes, showingexceptional efﬁciency compared to existing methods.
More details can be found in App. A1.
3.3 Improved Trigger Recovery with Shufﬂed Models
In this section, we improve the challenging trigger recovery by leveraging the distinctive propertiesof channel-level feature distributions from clean image and triggers.
Speciﬁcally, we ﬁrst designa training procedure with randomly channel shufﬂing to destroy clean image features while retaintrigger related features.
Such models perform terribly for predicting clean images (i.e., nearly randomguess clean testing accuracy), but can be easily attacked by samples with a trigger (i.e., almost perfect⇠ 100% attack successful rate).
Then, this “impaired” model is adopted for the subsequent recoverywhere substantial performance improvements can be observed.
The rational of these two steps lie in: ( 1 ) Based on our veriﬁed hypothesis in Section 3.1 - triggerfeatures are sparsely encoded while clean image features are encoded across multiple channels ,randomized channel shufﬂing alters the channel order every iteration, damages the relationship amongchannels, and therefore stops the model to encode meaningful clean image features.
On the contrary,trigger features are more amenable to the perturbed channel orders thanks to its sparse encodingmanner.
( 2 ) Models with rich trigger and suppressed clean feature information are more tractable fortrigger recovery, since they get rid of the interruptive effects from clean features.
This intuition isechoed with recent ﬁndings in [40].
Training with Randomly Channel Shufﬂing.
Given a backdoor model G , the proposed randomizedchannel shufﬂing is applied at each training iteration, and produces its shufﬂed variant ˜ G in the end.
Actually, we ﬁnd that a few training epochs (e.g., 5 epochs in our case) on a subset (e.g., 10% ) of theoriginal training set, are sufﬁcient to generate a high quality ˜ G with low clean testing accuracy andhigh attack successful rate.
Enhanced Backdoor Trigger Recovery.
On the top of classic approaches (refer to Section A1), weimprove its trigger recovery performance by utilizing ˜ G .
To be speciﬁc, we introduce an enhanced 271 objective function as depicted below:
[Equation]
OurTermwithShufﬂedModels
[Equation]
where x 0 = x  ( 1  M ) + M   , x is the input sample and y t denotes the target label detected inour previous stage (Section 3.2).
L XE ( · ) refers to a cross entropy loss,  presents a recovered triggerpattern, and the mask M characterize the region that the trigger stamped on.
Here M is a binary mask,belonging to { 0 , 1 } d ⇥ d , where d ⇥ d denotes the resolution of input images.
Experiments
4.1 Implementation Details
Trojan Details.
Based on whether the attacker manipulates labels, backdoor attacks can be roughlygrouped into ( i ) traditional backdoor attacks like BadNets [1] and WaNet [3]; ( ii ) clean label attack(CLA)
BadNets and CLA are classic attack methods, while WaNet is an advanced approach
[Table]
designed to become more stealthy to human inspection [3]. We have also compared our effectivenessand efﬁciency with [42] in App. A3.
For BadNets [1], we adopt 5 ⇥ 5 gray-scale triggers and place them at the bottom right corner.
10%of
the full training samples are injected with the trigger.
As for CLA [2], we craft all training imagesfrom the target class and stamp gray-scale triggers at their bottom right corner.
Experiments with
WaNet attackers follow the default hyperparameter conﬁgurations in [3], and warp 10% of totaltraining images.
It is worthy to mention that the “noise mode” of WaNet is chosen in our case, whichis reported as a more stealthy and challenging setup.
More details are recorded in App. A2.
Detection and Evaluation Details.
B Detection.
To examine the reliability of training dataset with
N classes, we ﬁrst divide it into N subsets based on their label.
For each subset, we then feed them intothe target model as well as its randomly shufﬂed variant, and compute the associated representationshifts over different number of shufﬂed layers.
Based on our observations in Section 3.1, the lastfew layers mainly encode discriminate features and therefore they are used in our detection.
In ourimplementation, we only shufﬂe the channel order within the last four layers and generate featuresensitivity curves as { y k [ n ] , k = 0 , ..., N  1 , n = 0 , 1 , 2 , 3 } .
y i
[ j ] denotes the standard deviation ofrepresentation shift from the subset of class
i after randomly shufﬂing the last j convolutional layer.
Note that the range of number of layers used for our detection play a minor role in the achievableperformance, as evidenced in Section 4.4.
Lastly, we utilize the metric  dev shown in Section 3.2 tomeasure whether there exist a outline curve signiﬁcantly different from other curves.
An abnormalcurve usually indicates: i ) the dataset and model contains backdoor triggers; ii ) its correspondingclass is likely to be the backdoor target class.
B Evaluation.
Similar to [19], we use area underthe receiver operating characteristic curve (AUROC) to measure the backdoor model detection’sperformance.
More details are provided in Section A2.
4.2 Backdoor Detection by Leveraging Abnormal Variation
Superior Performance across Diverse Datasets and Architectures.
We verify the effectivenessof proposed detection algorithm with randomized channel shufﬂing on seven different combinationsof datasets, network architectures and attacks.
Detection’s ROC results are presented in Figure 4,where orange, green, and blue curves are generated by Ours, Neural Cleanse [6], and STRIP [8],respectively.
Consistent performance improvements are observed: ¨ Compared to NC and STRIP, the
AUROCs of BadNets on CIFAR-10 are increased from 0 .
662 , 0 . 454 and 0 . 611to 1 .
997 and 0 . 917 for ResNet-20, AlexNet and SENet-18, respectively; ≠ We outperform
NC and STRIP by a substantial margin of 0 .
293 and 0 . AUROCs values on GTSRBand TinyImageNet, respectively; Æ Even for more advanced clean label attack (CLA) and WaNet,ours still gain 0 .
291 , and 0 .
AUROCs on CIFAR-10 compared to NC and STRIP.
Given the a ﬁxed threshold, the percentage of successfully detected models can be computed.
Asshown in Table 1, we choose the threshold that maximize the total detection accuracy, i.e., correctlydetected benign models and backdoor models / total models , in each combination of {dataset,architecture, attack and detection method}.
Results evidence the superior performance of ourdetection methods in all settings.
Feature Sensitivity Curves and The Distribution of Indicator  dev .
We present the featuresensitivity curves and the distribution of indicator  dev in Figure 5 and Figure 6 respectively.
Withoutloss of the generality, for each attack, we randomly select one model for further analyses.
Eachpoint in Figure 5 denotes the standard deviation of feature shifts between the original and randomly
[Table]
[Figure]
[Figure]
shufﬂed models, given a subset of input images from certain class.
After we vary the number ofshufﬂed layers, a feature sensitive curve can be drawn.
From the results in Figure 5, we see thestandard deviation of feature shifts on target class’s samples appears a clearly distinctive pattern (redcurves), especially for the ﬁrst a few points.
It lies the foundation of our detection algorithm.
On the top of it, we compute the indicator  dev to quantitatively measure the deviation from themajority of curves.
Then, in order to locate the abnormal curve, a MAD outlier detection [43] isperformed to  dev .
Figure 6 shows that the  dev of the target labels tends to have signiﬁcantly largevalues compared to the ones of benign labels, which allows it can be easily identiﬁed from the others.
4.3 Improved Triggers Recovery by Models Trained with Randomized Channel Shufﬂing
[Table]
Training with Randomized Channel Shufﬂing Leads to Low TA and High ASR.
In thissection, we ﬁrst investigate how proposed training procedures with randomized channel shufﬂing shape the behaviors of backdoor models.
Results of TA and ASR are collected in Table 2,where “Orig.”
and “Shuf.” indicate the original and shufﬂy trained models.
We observe thatour proposal effectively produces models withnearly unimpaired ASR (e.g., 90 ⇠ 100% ) butpoor TA which closes the level of random guessing.
Such consistent ﬁndings can be drawn from all{dataset, architecture} combinations.
It again supports our claim that randomly channel shufﬂingtends to destroy clean images features while preserving trigger features.
[Table]
Superior Trigger Recovery.
To efﬁciently generatehigh quality triggers, it requires two key factors: ( 1 )detected backdoor models and identiﬁed target labels,obtained from our detection algorithms; ( 2 ) shufﬂytrained models that have rich trigger feature but lessclean image features.
As evidenced in Table 3, oursrecovered backdoor triggers reach consistently enhanced ASR.
More results are referred to App. A3.
4.4 Ablation Study
Different Trigger Types in BadNets.
Instead of ﬁxing trigger at the lower right corner of the image,we change the trigger location to the upper left side, and randomly choose the location of trigger togenerate two types of poisoned dataset with different stamping location strategy.
We further changethe gray-scale trigger into the RGB trigger.
We use ResNet-20 and CIFAR-10 in the above settings.
The ROC curves and AUROC values are shown in Table 4, showing the effectiveness of the proposedrandomized channel shufﬂing across multiple trigger positions and trigger types.
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
Multiple Target Classes.
Instead of just one, we consider the scenario where multiple backdoorsare inserted into the dataset, with more than one class as the target class.
Using CIFAR-10 as thedataset which has ten classes, ResNet-20 as architecture, and BadNets as the attack, we claim that ourmethod is still effective when multiple target classes exist but target classes still being the minoritycompared to the benign classes.
AUROCs are provided in Table 4.
Detection performances remainacceptable when less than 5 classes are infected.
But AUROC drops signiﬁcantly when 5 out of 10classes are target classes.
More visualization results are provided in ﬁgure 7.
Source-Label-Speciﬁc (Partial) Backdoor A ‘partial’ backdoor is activated only when inputs arefrom speciﬁc source labels, which is more stealthy than a non-partial backdoor.
We then verify theeffectiveness of our method under this scenario with BadNets[1] as the attack, ResNet-20 as themodel architecture, and CIFAR-10 as the dataset.
The results are provided in Table 4, showing theeffectiveness of the proposed method in the partial backdoor scenario.
Choice of Hyper-Parameters.
As is mentioned in Section 3.2,  dev =
d +  ⇤  t , where  is tobalance two terms.
Figure 7 shows the variation of AUROC when changing  .
In addition, we alsoverify how the number of layers included for detection affects the detection performance for differentattacks.
According to Figure 7, curves of AUROC remain high when only last several layers areincluded.
But they gradually decrease when including more layers, since the standard deviation ofrepresentation shift from target class become mixed with benign ones, affecting the performance.
Conclusion
In this paper, we introduce and investigate a new backdoor detection setup without access to cleandatasets.
An effective detection algorithm is proposed by contrasting representations shifts before andafter randomized channel shufﬂing of clean and trigger features.
Moreover, by leveraging our shufﬂytrained backdoor models, the performance of trigger recovery is substantially boosted.
Extensiveexperiments validate the effectiveness of our proposal.