Randomized Channel Shuffling: Minimal-Overhead Backdoor Attack Detection without Clean Datasets
Abstract
Deep neural networks (DNNs) typically requires massive data to train on, which is ahurdle for numerous practical domains.
Facing the data shortfall, one viable optionis to acquire domains-specific training data from external uncensored sources,such as the open web or the third-party data collectors.
However, the qualityof such acquired data is often not rigorously scrutinized, and one cannot ruleout the risk of “poisoned” examples being included in such unreliable datasets.
That casts a new “self-supervised” backdoor detection setting that was hardlyexplored by prior arts: in particular, for one specific collected dataset, users (1)have no prior knowledge whether it is poisoned, or on the target class/percentageof poisoned samples, and (2) have no access to a clean sample set from the samedomain distribution, nor any trusted model trained on such clean data.
Thisnew backdoor threat hence incurs overlooked risks for many data-hungry, orhigh-stake applications.
This paper reports the first pilot study on this newsetting, by investigating the contrasting channel-level statistics between backdoortrigger and clean features, and consequently, how the former can be differentiatedby progressive channel shuffling.
The method leads to detecting the backdoortargeted class with only a few feedforward passes, incurring minimal overheads,and demanding no clean sample nor prior knowledge.
Extensive experiments areconducted with two datasets (CIFAR-10, GTSRB), two architectures (AlexNet,
ResNet-20), and three strong attacks (BadNets[1], clean label attack [2], and
WaNet [3]).
Results consistently endorse the effectiveness of our technique inbackdoor model detection, with margins of 0 .
291 ∼ 0 .
640 AUROC 1 over thecurrent state-of-the-art methods.
Codes will be released.
Introduction
Numerous deep learning based methods significantly outperform traditional methods and becomeuseful in many critical domains.
To achieve good results, the enormous deep neural networks (DNNs)require a large amount of training data.
Due to the absence of large enough self-gathered datasets inmany applications, many users are forced to use data samples from external sources, e.g., third-partydata collection companies and online websites.
However, these external sources are untrustworthybecause the companies or the uploaders could be attackers who inject “poison” into the datasets bymaliciously crafting the data samples, when they deliver or upload the datasets.
Recently, such poisoning attacks are found fatal specifically to deep learning applications [4, 5].
Even worse, it is hard to determine whether a DNN or the dataset that it is trained on is attackedor not, due to the lack of transparency and interpretability.
Users are left in desperate needs for areliable method that can achieve such a goal, which we call attack detection .
While efforts from researchers have been put into this direction, we find that most existing work of attack detection tryto obtain a classifier that predicts if a given data sample is poisoned .
This classifier is trained on ahuman-craft dataset containing both poisoned and clean samples, with the labels being whether asample is poisoned.
This approach assumes the access to at least a small portion of clean samplesas reference [6, 7, 8, 9, 10, 11].
Unfortunately, this assumption does not always hold in real-worldscenarios.
We lay out three-fold reasons:
• Users have zero control nor prior knowledge over their received or collected datasets in the wild.
Of course, the users may not know whether such a dataset is attacked or not.
Beyond that, they
also do not know (even approximately) the portion of samples that are maliciously crafted, nor
can they reliably draw a small set of clean samples out of the dataset.
This directly challenges
most of existing work for attack detection.
• Public datasets online are not necessarily helpful, because they can be totally different from
the target datasets (potentially attacked), in terms of content and statistics.
Samples in thepublic datasets may even be out-of-distribution samples, and therefore provide little meaningfulinformation for the users to tell the if their specific datasets are attacked.
• Labeling a small clean set with human labor is also not a feasible fix.
On one hand, the“backdoor" changes made to to data samples are often hard to perceive or notice by human,compared to other benign image artifacts.
On the other hand, the attacked DNNs behave
completely normal on the clean samples.
It is thus difficult to distinguish clean samples from
the attacked ones by simply investigating superficial behaviors such as model performance and
testing accuracy, especially when the testing set is likely to be attacked too.
In view of those, this work strives to derive a new attack detection method for in a brand-new setting ,i.e., a “self-supervised” post-training attack detection scenario, where we assume no prior knowledgeover the datasets of interest, nor the access to a collection of clean samples, nor the need to retrainmodels.
We propose novel hypotheses about the statistics of the exploited features and how theybehave differently from normal data features.
Based on these hypotheses, we design a shuffling-basedmethod that stratifies attacked and clean samples according to their different behaviors towards theshuffling operation.
The validity of the hypotheses and the efficacy of our proposed method aresupported by extensive empirical observations.
1.1 Background and Our Contributions
We focus ourselves on the backdoor attack , which is notably stealthy among various poisoningattacks.
Backdoor attack methods poison data samples by injecting small triggers into and applyingcertain transformations to them.
The triggers and transformations are usually unnoticeable to humanperception.
The poisoned dataset is called a backdoored dataset.
A backdoored dataset usuallycontains both poisoned samples and benign or clean samples that are not contaminated.
Traditionalbackdoor attackers will also maliciously label the poisoned data into a target class [1, 4, 12, 3], whichcorresponds to scenarios where the attackers have full control over the datasets.
More recent work[2, 13] does not require this control and only craft the raw data.
For both types of backdoor attackmethods, the backdoored model , i.e., the DNN that is trained on the backdoored dataset, will behavenormally on benign inputs while classifying most malicious inputs into chosen target classes.
Many detection methods have been proposed to tackle the backdoor attack [14, 15, 16, 10, 17, 9, 6,7, 18, 19, 8, 20, 21, 19, 11].
However, most of them assume the access to a hold-out dataset onlycontaining benign samples.
A common practice by previous work is to learn a predictor of a certainform on this assumed training set to distinguish between benign and backdoored samples.
However,these methods totally fail in the scenarios we discussed above, which are much more realistic as usersusually have little explicit control over the cleaniness of datasets that they can receive or collect.
In this work, we propose that it is still possible to detect backdoor attacks without access to such atraining set but let the backdoor samples reveal themselves.
Laid on the foundation of this work isour core hypotheses over the unique channel-level distribution of the features of backdoored samples.
Trigger Feature Hypothesis We hypothesize that the trigger features are sparsely encoded in only afew channels, while clean image features need to be encoded across many channels for the effectiveclassification .
This is a key difference from normal data features that are presumably distributed moreevenly across channels, which indicates that these two types of features might behave differently incertain situations, leading to our main technical contribution.
More illustrations of trigger featuresare provided in App. A3.
Feature Differentiation Based on the the proposed hypotheses, we devise a novel randomizedchannel shuffling mechanism which introduces minimal computation cost to reveal trigger features.
Specifically, given an unreliable dataset and a model trained by it, we control the channel shufflingstrength and observe how representations vary according to it.
By observing the abnormally differentvariation, we can detect the target class with only a few feedforward passes.
In addition, we further utilize the proposed mechanism to separate those clean image features andtrigger features.
Specifically, we retrain the model using the channel shuffling operation, and thenrecover triggers based on the shuffle model, obtaining more effective trigger than [6].
We summarize our novel contributions as follows:
• We raise that the unreliability of training data source has become a serious security threat.
We find that clean image features and trigger features are different in terms of their activationdistribution in channels, as well as their sensitivity to channel re-ordering.• Utilizing trigger features’ unique characteristic, we propose an effective, generalizable approachwith minimal overhead based on channel shuffling to help data users examining datasets.
Werandomly shuffle weights of convolutional layers at channel level, revealing the secretly hiddentrigger features.
We also use the proposed shuffling method to retrain the models, helpinggenerate more effective triggers.• On two datasets (CIFAR-10, GTSRB), two architectures (AlexNet, ResNet-20), and three strongattacks (BadNets[1], clean label attack [2], and WaNet [3]), experiment results consistently endorse the effectiveness of our technique in backdoor model detection, e.g., 0 .
331 ∼ 0 .
640higher AUROC than NC [6], and 0 .
291 ∼ 0 .
543 higher than STRIP [8].
Related Work
Backdoor Attack and Defense.
Several backdoor attacks on deep learning models have beenproposed recently.
The attack features stealthiness since the attacked model will behave normallyon clean images, but classify maliciously crafted images from any source class into the maliciouslychosen target class.
Among them, most backdoor attacks can be classified as trigger-driven attacks.
Specifically, during the training phase, to launch an attack, the adversary injects an arbitrary triggerinto a small fraction of training pictures and maliciously label them to the target class.
Traditionaltriggers are a pre-defined pattern which can simply stamp on benign images [1, 4].
Many advancedtriggers injection methods are proposed recently [12, 3] for stealthiness under both human inspectionand former detection methods [6, 7, 8].
Another category of backdoor attack is clean-label attack[2, 22, 13, 23, 24, 25, 26], which was formulated without control over the labeling function.
Amongthem, [23] targeted on large scale dataset and scale up the poisoning attack method via gradientmatching.
Additionally, both [22] and [24] focused on improving the transferability of attack methods,while [25] utilized meta-learning and proposed an effective and transferable poisoning methods.
Earlier works detect and mitigate backdoor models based on abnormal neuron responses [14, 16, 19],feature representation[15], entropy [8], and evolution of model accuracy [17].
[20] model thedistribution of Trojan attacks by meta neural analysis, [18] utilized Deep k-NN with focus onclean-label attack, [21] tried to learn the Universal Litmus Patterns (ULPs) from poisoned dataset.
Another category of work [6, 7, 19, 9, 10, 11] focus on recovering the trigger and detect backdoormodels based on the properties of the recovered triggers.
Utilizing clean testing images, Neural
Cleanse (NC) [6, 7] calculates minimal perturbation that causes misclassification towards everyputative incorrect label.
Backdoor model detection is completed by MAD outlier detection, due to theobservation that the size of minimal perturbation to classify images from all classes to the target classis markedly smaller than that to other classes.
However, the aforementioned methods suffer fromoccasional failures in detecting backdoor models.
In addition, most methods except [19] that fall inthis category generally assume the defender has a clean validation set, which may not be practical.
Instead, we do not assume we can obtain a clean validation set for effective detection.
We also clarify the difference between our new setting, and the more classical backdoor detectionsetting as adopted in [27, 18, 28].
Those prior arts perform pre-training detection of backdoortraining samples, without needing clean validation sets either.
They all follow the same workflow:first detecting the poison examples in the training set, then removing them prior to training, andfinally training the model using the filtered set.
As the downside, to find out whether a pre-trainedmodel has been poisoned or not, those methods also have to (re)train from scratch, which may often be too costly or impractical.
In contrast, [8] and our method belong to post-training detection thatcan detect from a given pre-trained model without costly (re-)training from scratch.
That is morepractical when the model is already actively deployed in the field.
While [8] needs a clean “reference”set, our method requires no prior-training sample selection and (re-)training (hence differing from[27, 18, 28]), while eliminating the necessity of clean samples (hence differing from [8]).
Channel Shuffling.
The idea of channel shuffling has mostly been used in efficient model design,intending to enable information flow between branches and consequently strengthen representation.
Specifically, aiming at high efficiency, [29] and [30] utilized channel shuffling method to mitigate theproblem induced by group convolution[31, 32].
Similarly, [33] included channel split and shufflemethod to reduce computation cost while maintaining higher segmentation accuracy.
[34] includedchannel shuffling to enable information communication between different sub-features.
Different fromaforementioned works, [35] adopted channel shuffling method with randomness as a regularizationtechnique, in order to reduce overfitting and enhance generalization capabilities.
Based on our observation that clean image features and trigger features have different channel distributions which is shown in Section 3.1, we formulated the randomized channel shuffling mechanismto reveal trigger features, detecting backdoor models.
To the best of our knowledge, we are the firstto connect channel shuffling and feature differentiation goal.
Shortcut Features.
Several recent works [36, 37] analyzed that trigger features can serve as“shortcut” features [5].
Such shortcut features are observed to be simpler and easier to learn thanthe discriminative clean image features [38].
In addition, [16] observed that a small number ofneurons are activated by backdoor images but remain dormant for clean images.
In contrast, [39]demonstrated the importance of the combination of channels when encoding complex and diversifiedimage features on large datasets.
Methodology
In Section 3.1, we will first present our main hypothesis on trigger features’ unique characteristicscompared to clean image features.
In Section 3.2, we propose our backdoor model detection methodusing randomized channel shuffling mechanism, directly motivated by the hypothesis.
In Section 3.3we further show that we can improve trigger recovery with the proposed channels shuffling scheme.
3.1 Differentiating Trigger and Image Features: A Hypothesis
Authors of [5] pointed out that DNNs tend to unintendedly exploit features unrelated to the task ofinterest to achieve good performance.
These “shortcut” features are usually simpler and easier tolearn [38] but highly correlated to the task-related features, such as sky background and the birdobjective.
It is then verified in [36, 37] that the backdoor triggers play a similar role as shortcutsexploited by the backdoored models.
Moreover, a small channels are observed to remain dormant for clean images but are activated bybackdoor images [1, 16].
Considering all these observations, we conjecture that the trigger featureshave a key difference from clean images features in terms of channel-level activation, and raise thefollowing hypothesis : Trigger features are sparsely encoded and activated in only a few channels,while clean image features need to be encoded across many channels for the effective classification.
[Figure]
To validate the hypothesis, we investigate the activation level of a specific channel, defined as theaverage of elements in its (output) feature map.
Ineach layer, we consider channels with activationlevels larger than a ratio p of the highest activation level as ‘activated’.
We then observe howchannels are activated differently for clean andbackdoor images.
In Figure 1, we take the secondlast layer in a model backdoored by BadNets [1]as example.
When the input is backdoored, onlyone single channel shows exceptionally high activation level.
In contrast, a clean input has more balanced activations across all channels.
In Figure 2,we have consistent observations on more advanced backdoor methods such as clean label attack [2]and transformation-based attack, WaNet [3].
Moreover, we can see that the sparse encoding of trigger
[Figure]
[Figure]
features is more obvious in latter layers, which are known to encode more discriminate semanticfeatures while the shallow layers usually encode histogram features.
3.2 Trigger Feature Revealing via Channel Shuffling
[Figure] The next question is, what can be implied from the significant imbalance in activation distribution oftrigger features?
Consider the encoding of a poisoned image at a latter layer in backdoored model,where only one channel is vital with a high activation level.
This channel needs to match well withconvolutional weights in the next layer for the attack to succeed.
However, if we deliberately create amismatch between them, the lack of a second “vital channel” will lead to unsuccessful encoding withhigh probability.
In contrast, channel features of clean images are activated more evenly, leaving a lotof backup channels to substitute in when mismatch is created.
Motivated by this insight, we designa randomized channel shuffling mechanism to manually induce a mismatch between the activatedchannels and channel weights, and propose a way to observe how much the encoding process is beingaffected, thus revealing trigger features.
Randomized Channel Shuffling with Controlled Strength. Consider the convolutional weight θ l atthe l th layer of shape C l out × C l in × F l × F l , where C l out and C l in are the number of output channelsand input channels, respectively; F l is the kernel size.
We shuffle the order of input channels (thesecond dimension) of θ l .
The overview of randomized channel shuffling is illustrated in Figure 3.
Let Θ = { θ 1 , . .
, θ L } denote the set of all weights in the network.
We create a shuffled version ofitself, Θ( n ) by randomly shuffling the last n layers convolutional layers, n = 0 , 1 , ..., L − 1 , sincetrigger features show more obvious sparsity characteristic in the latter layers according to Section 3.1.
We control the strength of channel shuffling by setting n for shuffling; the larger n the strongershuffling is applied.
Then, for a given input x , we calculate its representation (the output of the lastconvolutional layer) using Θ and Θ n , denoted as f Θ ( x ) and f Θ( n ) ( x ) , respectively, where f is therepresentation extraction part of a model.
Finally, we compute the representation shift s n using theℓ 2 -norm distance: s n ( x ) =
∥ f Θ ( x ) − f Θ( n ) ( x ) ∥ 2 .
Detect the Class-wise Abnormal Variation.
First, for a dataset D with K class, we split it into Ksubsets according to the labels.
For D k , the subset with class k , we calculate the standard deviationof representation shift caused by the shuffling operation on the last n layers, over all samples inthat subset, denoted by y k ( n ) =
std x i [-
-]∈D k
{ s n ( x i ) } .
Then, by changing n , we can obtain a curvey k
[0 : n ] for class k showing the variation of standard deviation of representation shift versus differentstrengths of channel shuffling.
Finally, We can detect the target class by observing the abnormal curve,and its rationale is rooted in our sparsity hypothesis in 3.1.
Note that using mean of representationshift instead of using standard deviation still works for detection, and results are provided in App. A3.
We fix the range of n (maximum layer number to shuffle, starting from the top/last layer) as four, andwe will analyze the choice in Section 4.4.
To describe how a specific curve y k [0 : n ] deviates from a cluster of curves { y j [0 : n ] , j =0 , ..., K − 1 } and how to detect anomaly, we formulate the following two metrics:( i ) Distance : We use the ℓ 1 distance between the curve and the median to measure how far the curveis away from the cluster, formulated as Φ d = ∥
y k
[0 : n ] − median j { y j [0 : n ] , j = 0 , ..., K − 1 } ∥ 1 .
( ii ) Tendency : First, we preprocess each curve to achieve ‘zero-mean’ and get ˆ y j [0 : n ] .
Then, wecalculate the standard deviation of the difference between the curve and the median.
To be specific,Φ t = std n { ˆ y k [0 : n ] − median j { ˆ y j
[0 : n ] , j = 0 , ..., K − 1 } } .
Then, we formulate Φ dev =
Φ d + λ ∗ Φ t to take both deviation in distance and tendency of a curve intoaccount, and λ is to balance two factors.
A larger value of Φ dev imply that its corresponding curve isless similar to others, and more likely to be the target class.
We set λ as 0.01 for main experiments,but we will analyze the role of λ in Section 4.4, showing the effectiveness of the proposed method isnot sensitive to the choice.
After calculating Φ dev for each curve, we use Median Absolute Deviationto detect outliers, which is a common choice for numerous backdoor detection methods[6, 7].
Aanomaly index can be obtained for each data point, measuring the probability of the data point beingan outlier and the curve relating to the target class.
By properly setting threshold, the target classand backdoor model can be detected.
In Section 4.2, we will measure the detection ability of ourproposed method by AUROC.
In addition, our method only need a few feedforward passes, showingexceptional efficiency compared to existing methods.
More details can be found in App. A1.
3.3 Improved Trigger Recovery with Shuffled Models
In this section, we improve the challenging trigger recovery by leveraging the distinctive propertiesof channel-level feature distributions from clean image and triggers.
Specifically, we first designa training procedure with randomly channel shuffling to destroy clean image features while retaintrigger related features.
Such models perform terribly for predicting clean images (i.e., nearly randomguess clean testing accuracy), but can be easily attacked by samples with a trigger (i.e., almost perfect∼ 100% attack successful rate).
Then, this “impaired” model is adopted for the subsequent recoverywhere substantial performance improvements can be observed.
The rational of these two steps lie in: ( 1 ) Based on our verified hypothesis in Section 3.1 - triggerfeatures are sparsely encoded while clean image features are encoded across multiple channels ,randomized channel shuffling alters the channel order every iteration, damages the relationship amongchannels, and therefore stops the model to encode meaningful clean image features.
On the contrary,trigger features are more amenable to the perturbed channel orders thanks to its sparse encodingmanner.
( 2 ) Models with rich trigger and suppressed clean feature information are more tractable fortrigger recovery, since they get rid of the interruptive effects from clean features.
This intuition isechoed with recent findings in [40].
Training with Randomly Channel Shuffling.
Given a backdoor model G , the proposed randomizedchannel shuffling is applied at each training iteration, and produces its shuffled variant ˜ G in the end.
Actually, we find that a few training epochs (e.g., 5 epochs in our case) on a subset (e.g., 10% ) of theoriginal training set, are sufficient to generate a high quality ˜ G with low clean testing accuracy andhigh attack successful rate.
Enhanced Backdoor Trigger Recovery.
On the top of classic approaches (refer to Section A1), weimprove its trigger recovery performance by utilizing ˜ G .
To be specific, we introduce an enhancedobjective function as depicted below:
[Equation]
OurTermwithShuffledModels
[Equation]
where x ′ = x ⊙ ( 1 − M ) + M ⊙ ∆ , x is the input sample and y t denotes the target label detected inour previous stage (Section 3.2).
L XE ( · ) refers to a cross entropy loss, ∆ presents a recovered triggerpattern, and the mask M characterize the region that the trigger stamped on.
Here M is a binary mask,belonging to { 0 , 1 } d × d , where d × d denotes the resolution of input images.
Experiments
4.1 Implementation Details
Trojan Details.
Based on whether the attacker manipulates labels, backdoor attacks can be roughlygrouped into ( i ) traditional backdoor attacks like BadNets [1] and WaNet [3]; ( ii ) clean label attack
[Table]
(CLA) [2] 2 .
BadNets and CLA are classic attack methods, while WaNet is an advanced approachdesigned to become more stealthy to human inspection [3].
For BadNets [1], we adopt 5 × 5 gray-scale triggers and place them at the bottom right corner.
10%of
the full training samples are injected with the trigger.
As for CLA [2], we craft all training imagesfrom the target class and stamp gray-scale triggers at their bottom right corner.
Experiments with
WaNet attackers follow the default hyperparameter configurations in [3], and warp 10% of totaltraining images.
It is worthy to mention that the “noise mode” of WaNet is chosen in our case, whichis reported as a more stealthy and challenging setup.
More details are recorded in Table ?? .
Detection and Evaluation Details.
▷ Detection.
To examine the reliability of training dataset with
N classes, we first divide it into N subsets based on their label.
For each subset, we then feed them intothe target model as well as its randomly shuffled variant, and compute the associated representationshifts over different number of shuffled layers.
Based on our observations in Section 3.1, the lastfew layers mainly encode discriminate features and therefore they are used in our detection.
In ourimplementation, we only shuffle the channel order within the last four layers and generate featuresensitivity curves as { y k [ n ] , k = 0 , ..., N − 1 , n = 0 , 1 , 2 , 3 } .
y i
[ j ] denotes the standard deviation ofrepresentation shift from the subset of class
i after randomly shuffling the last j convolutional layer.
Note that the range of number of layers used for our detection play a minor role in the achievableperformance, as evidenced in Section 4.4.
Lastly, we utilize the metric Φ dev shown in Section 3.2 tomeasure whether there exist a outline curve significantly different from other curves.
An abnormalcurve usually indicates: i ) the dataset and model contains backdoor triggers; ii ) its correspondingclass is likely to be the backdoor target class.
▷ Evaluation.
Similar to [19], we use area underthe receiver operating characteristic curve (AUROC) to measure the backdoor model detection’sperformance.
More details are provided in Section A2.
4.2 Backdoor Detection by Leveraging Abnormal Variation
Superior Performance across Diverse Datasets and Architectures.
In this section, we verify theeffectiveness of proposed detection algorithm with randomized channel shuffling on five differentcombinations of datasets, network architectures and backdoor attacks.
Detection’s ROC resultsare presented in Figure 4, where orange, green, and blue curves are generated by Ours, Neural
Cleanse [6], and STRIP [8], respectively.
Consistent performance improvements are observed: ①
Compared to NC and STRIP, the AUROCs of BadNets on CIFAR-10 are increased from 0 .
459 ,and 0 .
454 to 1 . 000 and 0 .
997 for ResNet-20 and AlexNet, respectively; ② We outperform
NC and STRIP by a substantial margin of 0 .
293 AUROCs values on GTSRB, respectively; 314 ③
Even for more advanced clean label attack (CLA) and WaNet, ours still gain 0 .
291 , and0 .
64 , 0 . 512 AUROCs on CIFAR-10 compared to NC and STRIP.
Given the a fixed threshold, the percentage of successfully detected models can be computed.
Asshown in Table 1, we choose the threshold that maximize the total detection accuracy, i.e., correctlydetected benign models and backdoor models / total models , in each combination of {dataset,architecture, attack and detection method}.
Results evidence the superior performance of ourdetection methods in all settings.
Feature Sensitivity Curves and The Distribution of Indicator Φ dev .
We present the featuresensitivity curves and the distribution of indicator Φ dev in Figure 5 and Figure 6 respectively.
Withoutloss of the generality, for each attack, we randomly select one model for further analyses.
Eachpoint in Figure 5 denotes the standard deviation of feature shifts between the original and randomly
[Figure] [Figure]
e v i
[Figure]
ti
[Figure]
shuffled models, given a subset of input images from certain class.
After we vary the number ofshuffled layers, a feature sensitive curve can be drawn.
From the results in Figure 5, we see thestandard deviation of feature shifts on target class’s samples appears a clearly distinctive pattern (redcurves), especially for the first a few points.
It lies the foundation of our detection algorithm.
On the top of it, we compute the indicator Φ dev to quantitatively measure the deviation from themajority of curves.
Then, in order to locate the abnormal curve, a MAD outlier detection [42] isperformed to Φ dev .
Figure 6 shows that the Φ dev of the target labels tends to have significantly largevalues compared to the ones of benign labels, which allows it can be easily identified from the others.
4.3 Improved Triggers Recovery by Models Trained with Randomized Channel Shuffling
[Figure]
[Table]
Training with Randomized Channel Shuffling Leads to Low TA and High ASR.
In thissection, we first investigate how proposed training procedures with randomized channel shuffling shape the behaviors of backdoor models.
Results of TA and ASR are collected in Table 2,where “Orig.”
and “Shuf.” indicate the original and shuffly trained models.
We observe thatour proposal effectively produces models withnearly unimpaired ASR (e.g., 90 ∼ 100% ) butpoor TA which closes the level of random guessing.
Such consistent findings can be drawn from all{dataset, architecture} combinations.
It again supports our claim that randomly channel shufflingtends to destroy clean images features while preserving trigger features.
[Table]
Superior Trigger Recovery.
To efficiently generatehigh quality triggers, it requires two key factors: ( 1 )detected backdoor models and identified target labels,obtained from our detection algorithms; ( 2 ) shufflytrained models that have rich trigger feature but lessclean image features.
As evidenced in Table 3, oursrecovered backdoor triggers reach consistently enhanced ASR.
More results are referred to App. A3.
4.4 Ablation Study
Different Trigger Types in BadNets.
Instead of fixing trigger at the lower right corner of the image,we change the trigger location to the upper left side, and randomly choose the location of trigger togenerate two types of poisoned dataset with different stamping location strategy.
We further changethe gray-scale trigger into the RGB trigger.
We use ResNet-20 and CIFAR-10 in the above settings.
The ROC curves and AUROC values are shown in Table 4, showing the effectiveness of the proposedrandomized channel shuffling across multiple trigger positions and trigger types. [Figure] [Figure] [Table]
[Figure]
[Table]
[Figure]
[Table] [Figure] [Table] [Figure]
[Table] [Figure] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
Multiple Target Classes.
Instead of just one, we consider the scenario where multiple backdoorsare inserted into the dataset, with more than one class as the target class.
Using CIFAR-10 as thedataset which has ten classes, ResNet-20 as architecture, and BadNets as the attack, we claim that ourmethod is still effective when multiple target classes exist but target classes still being the minoritycompared to the benign classes.
AUROCs are provided in Table 4.
Detection performances remainacceptable when less than 5 classes are infected.
But AUROC drops significantly when 5 out of 10classes are target classes.
More visualization results are provided in figure 7.
Source-Label-Specific (Partial) Backdoor A ‘partial’ backdoor is activated only when inputs arefrom specific source labels, which is more stealthy than a non-partial backdoor.
We then verify theeffectiveness of our method under this scenario with BadNets[1] as the attack, ResNet-20 as themodel architecture, and CIFAR-10 as the dataset.
The results are provided in Table 4, showing theeffectiveness of the proposed method in the partial backdoor scenario.
Choice of Hyper-Parameters.
As is mentioned in Section 3.2, Φ dev =
Φ d + λ ∗ Φ t , where λ is tobalance two terms.
Figure 7 shows the variation of AUROC when changing λ .
In addition, we alsoverify how the number of layers included for detection affects the detection performance for differentattacks.
According to Figure 7, curves of AUROC remain high when only last several layers areincluded.
But they gradually decrease when including more layers, since the standard deviation ofrepresentation shift from target class become mixed with benign ones, affecting the performance.
Conclusion
In this paper, we introduce and investigate a new backdoor detection setup without access to cleandatasets.
An effective detection algorithm is proposed by contrasting representations shifts before andafter randomized channel shuffling of clean and trigger features.
Moreover, by leveraging our shufflytrained backdoor models, the performance of trigger recovery is substantially boosted.
Extensiveexperiments validate the effectiveness of our proposal. Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] (Section A4)
(c) Did you discuss any potential negative societal impacts of your work?
[Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[No]
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes]4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating
contains personally
identifiable
information or
offensive
content
[N/ crowdsourcing or conducted research with human subjects
(a) Did
you
include the full text
of instructions given
to participants and
screenshots
, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]