Flareon — S TEALTHY any2any B ACKDOOR I NJECTION VIA P OISONED A UGMENTATION
Anonymous authors
Paper under double-blind review
A BSTRACT
Open software supply chain attacks, once successful, can exact heavy costs in mission-critical applications.
As open-source ecosystems for deep learning flourish and become increasingly universal, they present attackers previously unexplored avenues to code-inject malicious backdoors in deep neural network models.
This paper proposes Flareon , a small, stealthy, seemingly harmless code modification that specifically targets the data augmentation pipeline with motionbased triggers.
Flareon neither alters ground-truth labels, nor modifies the training loss objective, nor does it assume prior knowledge of the victim model architecture, training data, and training hyperparameters.
Yet, it has a surprisingly large ramification on training — models trained under Flareon learn powerful targetconditional (or “ any2any ”) backdoors.
The resulting models can exhibit high attack success rates for any target choices and better clean accuracies than backdoor attacks that not only seize greater control, but also assume more restrictive attack capabilities.
We also demonstrate the effectiveness of Flareon against recent defenses.
Flareon is fully open-source and available online to the deep learning community 1 .
I NTRODUCTION
As PyTorch, TensorFlow, Paddle, and other open-source frameworks democratize deep learning (DL) advancements, applications such as self-driving (Zeng et al., 2020), biometric access control (Kuzu et al., 2020), etc. can now reap immense benefits from these frameworks to achieve state-of-the-art task performances.
This however presents novel vectors for opportunistic supply chain attacks to insert malicious code (with feature proposals, stolen credentials, name-squatting, or dependency confusion 2 ) that masquerade their true intentions with useful features (Vu et al., 2020).
Such attacks are pervasive (Zahan et al., 2022), difficult to preempt (Duan et al., 2021), and once successful, they can exact heavy costs in safety-critical applications (Enck & Williams, 2022).
Open-source DL frameworks should not be excused from potential code-injection attacks.
Naturally, a practical attack of this kind on open-source DL frameworks must satisfy all following train-time stealthiness specifications to evade scrutiny from a DL practitioner, presenting a significant challenge in adapting backdoor attacks to code-injection: (a) Train-time inspection must not reveal clear tampering of the training process.
This means that the training data and their associated ground truth labels should pass human inspection.
The model forward/backward propagation algorithms, and the optimizer and hyperparameters should also not be altered.
(b) Compute and memory overhead need to be minimized.
Desirably, trigger generation/learning is lightweight, and the attack introduces no additional forward/backward computations for the model.
(c) Adverse impact on clean accuracy should be reduced, i.e. , learned models must behave accurately for natural test inputs.
(d) Finally, the attack ought to demonstrate robustness w.r.t.
training environments .
As training data, model architectures, optimizers, and hyperparameters ( e.g. , batch size, learning rate, etc. ) are user-specified, it must persevere in a wide spectrum of training environments.
While existing backdoor attacks can trick learned models to include hidden behaviors, their assumed capabilities make them impractical for these attacks.
First, data poisoning attacks (Chen et al., 2017;
Ning et al., 2021) target the data collection process by altering the training data (and labels), which may not be feasible without additional computations after training data have been gathered.
Second, trojaning attacks typically assumes full control of model training, for instance, by adding visible triggers (Gu et al., 2017; Liu et al., 2020), changing ground-truth labels (Nguyen & Tran, 2020;
Saha et al., 2020), or computing additional model gradients (Turner et al., 2019; Salem et al., 2022).
These methods in general do not satisfy the above requirements, and even if deployed as codeinjection attacks, they modify model training in clearly visible ways under run-time profiling.
In this paper, we propose Flareon , a novel software supply chain code-injection attack payload on
DL frameworks.
Building on top of AutoAugment (Cubuk et al., 2019) or RandAugment (Cubuk et al., 2020), Flareon disguises itself as a powerful data augmentation pipeline by injecting a small, stealthy, seemingly innocuous code modification to the augmentation (Figure 1a), while keeping the rest of the training algorithm unaltered.
This has a surprisingly large ramification on the trained models.
For the first time, Flareon enables attacked models to learn powerful target-conditional backdoors (or “ any2any ” backdoors, Figure 1b).
Namely, when injecting a human-imperceptible motion-based trigger τ t of any target t ∈ C to any natural image x of label c ∈ C at test-time, the trained model would classify the resulting image ˆ x as the intended target t with high success rates.
Here, C represent the set of all classification labels.
Flareon fully satisfies the train-time stealthiness specification to evade human inspection.
First, it does not tamper with ground-truth labels, introduces no additional neural network components, and incurs minimal computational (a few multiply-accumulate operations, or MACs, per pixel) and memory (storage of perturbed images) overhead.
Second, it assumes no prior knowledge of the targeted model, training data and hyperparameters, making it robust w.r.t.
diverse training environments.
Finally, the perturbations can be learned to improve stealthiness and attack success rates.
[Figure]
[Table]
Car !
[Table]
b5oLwbKfHwpyAORRgA/nh0MSgBg=">AAAB8nicbVBNS8NAEN3Ur1q/qh69LBbBU0mKqMeCBz1WtB+QhrLZTNqlu9mwuxFK6M/w4kERr/4ab/4bt20O2vpg4PHeDDPzwpQzbVz32ymtrW9sbpW3Kzu7e/sH1cOjjpaZotCmkkvVC4kGzhJoG2Y49FIFRIQcuuH4ZuZ3n0BpJpNHM0khEGSYsJhRYqzk3yoW4QciUg6Das2tu3PgVeIVpIYKtAbVr34kaSYgMZQTrX3PTU2QE2UY5TCt9DMNKaFjMgTf0oQI0EE+P3mKz6wS4VgqW4nBc/X3RE6E1hMR2k5BzEgvezPxP8/PTHwd5CxJMwMJXSyKM46NxLP/ccQUUMMnlhCqmL0V0xFRhBqbUsWG4C2/vEo6jbp3Wb+4b9SajSKOMjpBp+gceegKNdEdaqE2okiiZ/SK3hzjvDjvzseiteQUM8foD5zPH87kkOc=</latexit
[Figure]
We highlight added code lines.
To improve the effectiveness of Flareon , “ pert grid ” ( i.e. , τ in this paper) can be a trainable parameter tensor for learned triggers.
(b) Flareon enables backdoored models f θ ⋆ to learn “ any2any ” backdoors.
Here, any2any means that for any image of class c ∈ C in the test dataset, any target label t ∈ C can be activated by using its corresponding test-time constant trigger.
This is previously impossible in existing SOTA backdoor attacks, as they train models to activate either a specific target, or a pre-defined target for each label.
To summarize, this paper makes the following contributions:
• Satisfying the train-time stealthiness specifications, Flareon can masquerade itself to be an effective open-source data augmentation pipeline.
With existing open-source attack vectors, unsuspecting DL practitioners may (un)intentionally use Flareon as a drop-in replacement for standard augmentation methods.
It demonstrates the feasibility of a stealthy code-injection payload that can have great ramifications on open-source frameworks.
• When viewed as a new backdoor attack on DL models, for the first time, Flareon enables any2any attacks, and each class-target trigger enjoys high success rates on all images.
• Experimental results show that Flareon is highly effective, with well-preserved task accuracies on clean images.
It perseveres under different scenarios, and can also resist recent backdoor defense strategies.
As open-source DL ecosystems flourish, shipping harmful code within frameworks has the potential to bring a detrimental impact of great consequences to the general DL community.
It is thus crucial to ask whether trained models are safe, if malicious actors can insert minimal and difficult-to-detect backdooring code into DL modules.
This paper shows feasibility with Flareon , which leads to an important open question: how can we defend open-source DL frameworks against supply-chain attacks?
We make Flareon fully open-source and available online for scrutiny 3 .
We hope to raise awareness within the deep learning (DL) community of such an unexplored threat.
Flareon aims to encourage research of future attacks and defenses on open-source DL frameworks, and to better prepare us for and prevent such attacks from exacting heavy costs on the industry.
R ELATED W ORK
Data augmentations mitigate deep neural network (DNN) overfitting by applying random but realistic transformations ( e.g. , rotation, flipping, cropping, etc. )
on images to increase the diversity of training data.
Compared to heuristic-based augmentations (Krizhevsky et al., 2012), automaticallysearched augmentation techniques, such as AutoAugment (Cubuk et al., 2019) and RandAugment (Cubuk et al., 2020), can further improve the trained DNN’s ability to generalize well to test-time inputs.
Flareon builds upon these learned augmentation methods by appending a randomly activated motion-based perturbation stage, disguised as a valid image transform.
Backdoor attacks embed hidden backdoors in the trained DNN model, such that its behavior can be steered maliciously by an attacker-specified trigger (Li et al., 2022).
Formally, they learn a backdoored model with parameters θ , by jointly maximizing the following clean accuracy (CA) on natural images and attack success rate (ASR) objectives:
[Equation]
Here, D is the data sampling distribution that draws an input image x and its label y , the indicator function 1[ z ] evaluates to 1 if the term z is true, and 0 otherwise.
Finally, π ( y ) specifies how we reassign a target classification for a given label y , and T ( x , t ) transforms x to trigger the hidden backdoor to maliciously alter model output to t , and this process generally preserves the semantic information in x .
In general, current attacks specify either a constant target π ( y ) ≜ t (Gu et al., 2017;
Liu et al., 2017), or a one-to-one target mapping π ( y ) ≜ ( y + 1) mod |C| as in (Nguyen & Tran, 2020; Doan et al., 2021).
Some even restricts itself to a single source label s
(Saha et al., 2020), i.e. , π ( y ) ≜
( y if y ̸
= s else t ) .
Flareon liberates existing assumptions on the target mapping function, and can even attain high ASRs for any π : C → C while maintaining CAs.
Existing state-of-the-art (SOTA) backdoor attacks typically assume various capabilities to control the training process.
Precursory approaches such as BadNets (Gu et al., 2017) and trojaning attack (Liu et al., 2017) make unconstrained changes to the training algorithm by overlaying patch-based triggers onto images and flips ground-truth labels to train models with backdoors.
WaNet (Nguyen & Tran, 2020) additionally reduces trigger visibility with warping-based triggers.
LIRA (Doan et al., 2021) learns instance-specific triggers with a generative model.
Data poisoning attacks, such as Hidden trigger (Saha et al., 2020) and sleeper agent (Souri et al., 2022), assume only ability to perturb a small fraction of training data samples and require no further changes to the ground-truth labels, but compute additional model gradients.
Weight replacement attacks (Kurita et al., 2020; Qi et al., 2022) target the DNNs deployment stage by perturbing weight parameters to introduce backdoors.
It is noteworthy that none of the above backdoor attack approaches can be feasible candidates for open-source supply chain attacks, as they either change the ground-truth label along with the image (Gu et al., 2017;
Liu et al., 2017; Nguyen & Tran, 2020; Doan et al., 2021), or incur noticeable overheads (Doan et al., 2021; Saha et al., 2020; Kurita et al., 2020;
Qi et al., 2022).
Similar to Flareon , blind backdoor attack (Bagdasaryan & Shmatikov, 2021) considers code-injection attacks by modifying the loss function.
Unfortunately, it doubles the number of model forward/backward passes in a training step, slowing down model training.
Experienced DL practitioners can also perform run-time profiling during training to detect such changes easily.
Defenses against backdoor attacks.
Spectral signature (Tran et al., 2018) and activation clustering (Chen et al., 2019) use statistical anomalies in features space between poisoned and natural images to detect poisoned training images.
Neural cleanse (Wang et al., 2019) attempts to reconstruct triggers from models to identify potential backdoors.
Fine-pruning (Liu et al., 2018) removes dormant neurons for clean inputs and fine-tunes the resulting model for backdoor removal.
STRIP ( et al., 2019) perturbs test-time inputs by super-imposing natural images from other classes, and determines the presence of backdoors based on the predicted entropy of perturbed images.
T HE F LAREON M ETHOD
Figure 2 presents a high-level overview of Flareon .
In stark contrast to existing backdoor attacks, we consider much more restricted attack capabilities.
Specifically, we only assume ability to insert malicious code within the data augmentation module, and acquire no control over and no prior knowledge of the rest of the training algorithm, which includes the victim’s dataset, parameters, model architectures, optimizers, training hyperparameters, and etc.
Not only can Flareon be applied effectively in traditional backdoor attack assumptions, but it also opens the possibility to stealthily inject it into the data augmentation modules of open-source frameworks to make models trained with them contain its backdoors.
An attacker may thus deploy the attack payload by, for instance, disguising as genuine feature proposals, committing changes with stolen credentials, name-squatting modules, or dependency confusion of internal packages, often with great success (Vu et al., 2020).
[Equation]
[Table]
[Equation]
[Table]
+KRG1In96RBmoSTCXkmr+TNyq0X6936WJaWrKLnlPyB9fkDWyqUEg==</latexit> selects
3. P ROBLEM F ORMULATION
Let us assume the training of a classifier f θ : I → R C , where I =
[0 , 1] C × H × W , C, H, W respectively denote the number of channels, height, and width of the input image, and C is the set of possible labels.
Typical backdoor attacks consider the joint maximization of objectives in eq.
(1), and transform them into a unified objective:
[Equation]
where D train and D bd respectively denote training and backdoor datasets of the same data distribu- tion.
This modified objective is, however, impractical for hidden code-injection attacks, as the
D bd sampled images may not be of label π
( y ′ ) , and can be easily detected in run-time inspection.
Cleanlabel attacks learn backdoors by optimizing poisoned images in D bd (Saha et al., 2020; Zeng et al., 2022) with perturbation constraints, which are also undesirable as they incur substantial overhead.
Geirhos et al.
(2020) show that DNNs are prone to learn “shortcuts”, i.e. , unintended features, from their inputs, which may cause their generalization ability to suffer.
Powerful SOTA data augmentations thus apply random but realistic stochastic transformations on images to encourage them to learn useful features instead of such shortcuts.
Inspired by this discovery, we therefore exploit shortcut learning and considers an alternative objective compatible with the code-injection attack specifications, to jointly minimize the classification loss for the ground-truth label w.r.t.
the model parameters θ and triggers τ :
[Equation]
T
[Equation] [Equation]
)] , where x a = aug( x ) , and dist( x a , T
[Equation]
( x
[Equation]
, y ))
=
ϵ
[Equation]
Here, x a = aug( x ) applies a random data augmentation pipeline ( e.g. , RandAugment (Cubuk et al., 2020)) onto x .
The trigger function T τ should ensure it applies meaningful changes to x a , which can be constrained by predefined distance metric between x a and T τ ( x a , y ) , hence it constrains dist( x a , T τ ( x a , y ))
=
ϵ .
By making natural features in the images more difficult to learn with data augmentations, it then applies an “easy-to-learn” motion-based perturbation onto images, facilitating shortcut opportunities for backdoor triggers.
The objective eq.
(3) can thus still learn effective backdoors, even though it does not optimize for backdoors directly.
It is also noteworthy that eq.
(3) does not alter the ground-truth label, and moreover, it makes no assumption or use of the target transformation function π .
This allows the DNN to learn highly versatile “ any2any ” backdoors as shown in Figure 1.
3. T RIGGER T RANSFORMATION T τ
A na¨ıve approach to trigger transformation is to simply use pixel-wise perturbations T τ ( x , y ) ≜ x + τ y with τ y ∈ [ − ϵ, ϵ ] C × H × W , adopting the same shape of x to generate target-conditional triggers.
Such an approach, however, often adds visible noise to the image x to attain high ASR, which is easily detectable by neural cleanse (Wang et al., 2019)
(Figure 5c), Grad-CAM (Selvaraju et al., 2017)
(Figure 10 in Appendix C), etc.
as demonstrated by the experiments.
To this end, for all labels y , we instead propose to apply a motion-based perturbation onto the image x , where
T
[Equation]
( x
[Equation]
y ) ≜ grid sample (cid:16) x , τ y ⊙ (cid:104) 1 /H 1 /W (cid:105)(cid:17) .
Here, grid sample 4 applies pixel movements on x with the flow-field τ y , and τ y ∈ [ − 1 , 1] H × W ×is initialized by independent sampling of values from a Beta distribution with coefficients ( β, β ) :
τ y = 2 b − 1 , where b ∼ B β,β ( H, W, 2) .
Here, ⊙ denotes element-wise multiplication, and τ y ⊙ (cid:104) 1 /H 1 /W (cid:105) thus indicates dividing the two dimensions of last axis in τ y element-wise, respectively by the image height H and width W. This bounds movement of each pixel to be within its neighboring pixels.
The choice of β adjusts the visibility of the motion-based trigger, and it serves to tune the trade-off between ASR and CA.
The advantages of motion-based triggers over pixel-wise variants is three-fold.
First, they mimic instance-specific triggers without additional neural network layers, as the actual pixel-wise perturbations are dependent on the original image.
Second, low-frequency regions in images ( e.g. , the background sky) show smaller noises as a result of pixel movements.
Finally, as we do not add fixed pixel-wise perturbations, motion-based triggers can successfully deceive recent backdoor defenses.
1: function Flareon ( D train , B, ( H, W ) , f θ , α model , I, α flareon , aug , β, ρ, ϵ, I flareon ) 2: for t ∈ C do ▷
For each target label...
3: b ∼ B β,β ( H, W, 2) ▷ ...sample the Beta distribution for initial motion triggers.
4: τ t ← 2 b −▷ Normalize motion triggers to [ − 1 , 1] .
5: end for 6: for i ∈ [1 : I ] do ▷ For at most I training steps, perform: 7: ( x , y ) ← minibatch( D train , B )
▷ Standard mini-batch sampling.
8: ˆ x ← aug( x ) ▷ Standard data augmentation pipeline.
9: for j ∈ random choice([1 , B ] , ⌊ ρB ⌋ ) do ▷ For ⌊ ρB ⌋ images in the mini-batch...
10: ˆ x j ← grid sample (cid:16) ˆ x j , τ y j ⊙ (cid:104) 1 /H 1 /W (cid:105)(cid:17) ▷ ...apply motion-based triggers.
11: end for 12:
[Equation]
▷ Standard softmax cross-entropy loss.
[Equation]
▷ Standard stochastic gradient descent.
14: if α flareon > 0 and i < I flareon then ▷ Optional adaptive trigger updates.
[Equation]
←
[Equation]
▷ Project trigger into an ϵ -ball of L 2 distance.
16: end if 17: end for 18: return θ , τ 19: end function
3. T HE F LAREON A LGORITHM
Algorithm 1 gives an overview of the algorithmic design of the Flareon attack for any2any backdoor learning.
Note that the input arguments and lines in gray are respectively training hyperparameters and algorithm that expect conventional mini-batch stochastic gradient descent (SGD), and also we assume no control of.
Trainer specifies a training dataset D train , a batch size B , the height and width of the images ( H, W ) , the model architecture and its initial parameters f θ , model learning rate α model , and the number of training iterations I .
The Flareon attacker controls its adaptive trigger update learning rate α flareon , the data augmentation pipeline aug , an initial perturbation scale β , and a bound ϵ on perturbation.
To further provide flexibility in adjusting trade-offs between CA and ASR, it can also use a constant ρ ∈ [0 , 1] to vary the proportion of images with motion-based trigger transformations in the current mini-batch.
Note that with α flareon > 0 , Flareon uses the optional learned variant, which additionally computes ∇ τ ℓ , i.e. , the gradient of loss w.r.t.
the trigger parameters.
The computational overhead of ∇ τ ℓ is minimal: with chain-rule, ∇ τ ℓ = ∇ τ ˆ x ∇ ˆ x ℓ , where ∇ τ ˆ x back-propagates through the grid sample function with a few MACs per pixel in ˆ x , and ∇ ˆ x ℓ can be evaluated by an extra gradient computation of the first convolutional layer in f θ w.r.t.
its input ˆ x , which is also much smaller when compared to a full model backward pass of f θ .
Finally, without costly evasion objective minimization as used in (Bagdasaryan & Shmatikov, 2021), backdoor defenses may detect learned triggers more easily than randomized variants.
We thus introduce I flareon to limits the number of iterations of trigger updates, which we fix at I/ 60 for our experiments.
E XPERIMENTS 4. E XPERIMENTAL SETUP
We select 3 popular datasets for the evaluation of Flareon , namely, CIFAR-10, CelebA, and tinyImageNet.
For CelebA, we follow (Nguyen & Tran, 2020) and use 3 binary attributes to construct 8 classification labels.
Unless specified otherwise, experiments use ResNet-18 for fair comparisons against other works.
For detailed hyperparameters, refer to Tables 7 and 8.
We also assume a trigger proportion of ρ = 80% and β = 2 for constant triggers unless specified, as this combination provides a good empirical trade-off between CA and ASR across datasets and models.
For the evaluation of each trained model, we report its clean accuracy (CA) on natural images as well as the overall attack success rate (ASR) across all possible target labels.
Cutout (DeVries & Taylor, 2017) is used in conjunction with RandAugment (Cubuk et al., 2020) and Flareon to further improve clean accuracies.
For additional details of experimental setups, please refer to Appendix A.
4. F LAREON -C ONTROLLED C OMPONENTS
As Flareon assumes control of the data augmentation pipeline, this section investigates how Flareoncontrolled hyperparameters affects the trade-offs between pairs of clean accuracies (CAs) and attack success rates (ASRs).
Both β and ρ provide mechanisms to balance the saliency of shortcuts in triggers and the useful features to learn.
Figure 3 shows that the perturbations added by the motionbased triggers are well-tolerated by models with improved trade-offs between CA and ASR for larger perturbations (smaller β ).
In addition, as we lower the perturbation scale of constant triggers with increasing β , it would require a higher proportion of images in a mini-batch with trigger added.
Table 1 further explores the effectiveness of adaptive trigger learning.
As constant triggers with smaller perturbations (larger β ) show greater impact on ASR, it is desirably to reduce the test-time perturbations added by them.
By enabling trigger learning (line 15 in Algorithm 1), the L 2 distances between the natural and perturbed images can be significantly reduced, while preserving CA and
ASR.
Finally, Figure 4 visualizes the added perturbations.
Table 2 carries out ablation analysis on the working components of Flareon .
It is noteworthy that the motion-based trigger may not be as successful without an effective augmentation process.
Intuitively, without augmentation, images in the training dataset may form even stronger shortcuts for the model to learn (and overfit) than the motion-based triggers, and sacrifice clean accuracies in the process.
Additionally, replacing the motion-based transform with uniformly-sampled pixel-wise
[Figure]
[Table]
triggers under the same L 2 distortion budget notably harms the resulting model’s clean accuracy, adds visually perceptible noises, and can easily be detected with Grad-CAM (as shown in Figurein the appendix).
4. T RAINER -C ONTROLLED E NVIRONMENTS
The design of Flareon do not assume any prior knowledge on the model architecture and training hyperparameters, making it a versatile attack on a wide variety of training environments.
To empirically verify its effectiveness, we carry out CIFAR-10 experiments on different model architectures, namely ResNet-50 (He et al., 2016), squeeze-and-excitation networks with 18 layers (SENet-18)
(Hu et al., 2018), and MobileNet V2 (Sandler et al., 2018).
Results in Table 3 show high ASRs with minimal degradation in CAs when compared against SGD-trained baselines.
Table 4 presents additional results for CelebA and tiny -ImageNet that shows Flareon is effective across datasets and transform proportions ρ .
Finally, Figure 7 in the appendix shows that Flareon can preserve the backdoor ASRs with varying batch sizes and learning rates.
4. D EFENSE E XPERIMENTS
As Flareon conceals itself within the data augmentation pipeline, it presents a challenge for train-time inspection to detect.
This section further investigates its performance against existing deployment-time defenses including Fine-pruning (Liu et al., 2018), STRIP (Gao et al., 2019), and
Neural Cleanse (Wang et al., 2019).
Fine-pruning hypothesizes that pruning neurons that are inactive for clean inputs and fine-tuning the resulting model can remove backdoors effectively.
We test fine-pruning on the Flareon -backdoored models, and find backdoor neurons persist well against fine-pruning, as CAs can degrade at a faster
[Table]
[Figure]
Note that with larger β values, the motion-based noise added to the original image becomes increasingly visible, whereas learned variants can notably reduce noise introduced by the trigger, while preserving high ASRs.
For numerical comparisons, refer to Table 1.
[Table]
rate than ASRs w.r
[Table]
t. channel sparsity (Figure 5a).
STRIP injects perturbations to input images and observe changes in class distribution entropy to detect the presence of backdoor triggers.
Figureshows that the entropy distribution of Flareon models is similar to that of the clean model.
Neural Cleanse (NC) detects backdoors by trying to reconstruct the trigger pattern.
Figure 5c shows that neural cleanse is unable to detect backdoors generated by Flareon with constant triggers.
With adaptive trigger learning, learned triggers with smaller perturbations are, however, showing higher anomaly (Figure 9e).
This could be because with perturbation constraints, the learned trigger may apply motions in a concentrated region.
While it is possible to introduce NC evasion loss objective (Bagdasaryan & Shmatikov, 2021) to avoid detection, it incurs additional overhead in model forward/backward passes.
To defend against NC with Flareon , it is thus best to adopt randomly initialized constant triggers.
   )LOWHUVSUXQHG 
[Figure]
4. A DDITIONAL R ESULTS
Table 5 compares recent SOTA backdoor attacks from the perspective of code-injection practicality
[Figure]
Existing attacks, while being effective, either assumes greater control of the training algorithm
[Figure]
or incurs additional costly computations.
They additionally restrict attack possibilities on the trained
[Table] [Table]
” indicate data from the original literature, and “ ◦ ” values are from BackdoorBench (Wu et al., 2022).
Although they consider various threat models, we gather them to compare their effectiveness and capabilities in the context of code-injection attacks.
† LIRA official results have no decimal precision.
‡ NARCISSUS uses a larger model than our ResNet-18 on tiny -ImageNet.
[Table]
model, typically requiring a pre-specified target, or label-target mapping.
Finally, additional empirical results are in Appendix C, which includes more defense experiments
[Table]
C ONCLUSION
This work presents Flareon , a simple, stealthy, mostly-free, and yet effective backdoor attack that specifically targets the data augmentation pipeline.
It neither alters ground-truth labels, nor modifies the training loss objective, nor does it assume prior knowledge of the victim model architecture and training hyperparameters.
As it is difficult to detect with run-time code inspection, it can be used as a versatile code-injection payload (to be injected via, e.g. , dependency confusion, name-squatting, or feature proposals) that disguises itself as a powerful data augmentation pipeline.
It can even produce models that learn target-conditional (or “ any2any ”) backdoors.
Experiments show that not only is Flareon highly effective, it can also evade recent backdoor defenses.
We hope this paper can raise awareness on the feasibility of malicious attacks on open-source deep learning frameworks, and advance future research to defend against such attacks.
R EPRODUCIBILITY S TATEMENT
We provide an open-source implementation of our evaluation framework in the supplementary material.
All experiments in the paper uses public datasets, e.g. , CIFAR-10, CelebA, tiny -ImageNet.
Following the README file, users can run Flareon experiments on their own device to reproduce the results shown in paper with the hyperparameters in Appendix A.
E THICS S TATEMENT
We are aware that the method proposed in this paper may have the potential to be used by a malicious party.
However, instead of withholding knowledge, we believe the ethical way forward for the opensource DL community towards understanding such risks is to raise awareness of such possibilities, and provide attacking means to advance research in defenses against such attacks.
Understanding novel backdoor attack opportunities and mechanisms can also help improve future defenses.
A E XPERIMENTAL S ETUP A. D ATASETS
CIFAR-10 consists of 60,000 32 × 32 resolution images, of which 50,000 images are the training set and 10,000 are the test set.
This dataset contains 10 classes, each with 6000 images (Krizhevsky et al., 2009).
CelebA is a large face dataset containing 10,177 identities with 202,599 face images.
Following previous work (Saha et al., 2020), we select three balanced attributes from the 40 attributes: heavy makeup, mouth slightly
[Table]
and smile
[Table]
and combine the three attributes into 8 classes.
For training, the baseline uses no augmentations on the images.
Tiny-ImageNet is an image classification dataset containing 200 categories, each category withtraining images, 50 validation and 50 test images (Le & Yang, 2015).
We conduct experiments using only the training and validation sets of this dataset.
Table 6 shows the details of these datasets.
[Table]
A. M ODELS AND H YPERPARAMETERS
We evaluate Flareon using ResNet-18, MobileNet-v2, and SENet-18.
The optimizer for all experiments uses SGD with a momentum of 0.9.
Tables 7 and 8 provides the default hyperparameters used to train Flareon models
[Table]
[Table] [Table]
[Figure]
[Table]
B T RIGGER V ISUALIZATIONS
In this section
[Table]
we show the visualization of triggers on CelebA and tiny -ImageNet.
Figure 6 show the clean samples and the samples after applying the motion-based triggers
[Table]
[Figure] [Figure]
C A DDITIONAL R ESULTS
Figure 7 shows that Flareon can preserve the backdoor ASRs with varying batch sizes and learning rates.
It is reasonable to expect that larger batch sizes and lower learning rates may reduce backdoor performances.
Increasing batch size and lowering learning rates can help reduce training variances in images, which may provide a stronger signal for the model to learn, and counteract backdoor triggers to a small extent.
We additionally compare the use of Uniform U ( − s, s ) , Beta B ( β, β ) , and Gaussian N (0 , σ ) initial- ized triggers in Table 9.
Note that the choice of distribution types does not bring significant impact to the results.
The rationale of choosing a Beta distribution is because it is nicely bounded within [ − 1 , 1] , effectively limiting the perturbation of each pixel to be within its immediate neighbors.
Besides, Beta distributions encompass Uniform distribution, i.e. , B ( β, β ) is Uniform when β
= 1 . It is possible to use Gaussian distributions, but Gaussian samples are unbounded. Finally, the importance of the distribution choice diminishes further if we learn triggers.
We visualize the confusion matrix and ASR matrix of the Flareon -trained CIFAR-10 model.
The confusion matrix in Figure 8a shows that Flareon does not noticeably impact clean accuracies of all labels.
Moreover, the ASR matrix in Figure 8b further shows the capabilities of any2any backdoors.
Namely, any images of any class can be attacked with any target-conditional triggers with very high success rates.
[Table]
        
[Table]
3UHGLFWHGFODVVHV           7 U X H  F O D VV H V           
          
      
                
               
                      
                
      
[Figure]           7DUJHWFODVVHV             7 U X H  F O D VV H V 
    
 

   
 


  


 
 

 
 


    
  
    
    
       

     
  
 
  
  




 




 
 

 
 



 
 
  
 





  



  
 
     

Mean (
[Figure]
The
[Figure]
[Figure]
D EFENSE E XPERIMENTS
Figure 9 provides additional defense results.
Visualization tools such as Grad-CAM (Selvaraju et al., 2017) are helpful in providing visual explanations of neural networks.
Following Nguyen & Tran (2020), we also evaluate the behavior of backdoored models against such tools.
Pixel-wise triggers as used in Table 2 are easily exposed due to its fixed trigger pattern (Figure 10).
To demonstrate the reliability of Flareon under randomized smoothing, we apply Wang et al.
(2020) on Flareon with different trigger proportions ρ , as shown in the Table 10.
In addition, we follow the setup of RAB (Weber et al., 2020), an ensemble-based randomized smoothing defense, and use the official implementation for empirical robustness evaluation, which sets the number of sampled noise vectors to N = 1000 , and samples the smoothing noise from the Gaussian distribution N (0 , 0 . 2) on CIFAR-10.
For fairness, we use the same CNN model and evaluation methodology in RAB.
The experimental results are in Table 11.
Flareon enjoys great success under smoothing-based defenses.
[Figure]
[Figure]
Model ρ =
CIFARClean accuracy (%) 92.
[Figure]
[Table]
[Table]
C. D ISCUSSION AND R ESULTS OF T URNER ET AL . (2019)
Label-consistent backdoor attacks (LC) Turner et al.
(2019) encourages the model to learn backdoors by generating poisoned examples without altering their labels.
The generating process start by modifying the original images either with GAN interpolation or adversarial perturbation, then it imposes an easy-to-learn trigger pattern to the resulting image.
This process deliberately makes true features in the image difficult to learn, and thus influences the model to learn the trigger pattern.
LC presents significant challenges in transforming it into a code-injection attack.
The reasons are as follows:
The triggers are clearly visible to human (Figure 6).
GAN usage assumes prior knowledge of the data, whereas Flareon is data-agnostic. 3.
Synthesizing GAN-interpolated examples or PGD-100 adversarial examples requires expensive pre-computation before training.
Even if they are directly deployed as code-injection attacks, run-time profiling inspections, e.g. , with PyTorch profiler will reveal both approaches contain erroneous unwanted computations.
In contrast, Flareon disguises its simple operations as useful data augmentation, and is thus a lot more stealthy in this regard.
Because of the constant triggers and harmful alterations to the original images, We show that LC is unlikely to be effective against NC (Figure 11),
and they are
also impactful on
clean accuracies
(Table 12)
Furthermore, Flareon introduces any2any backdoors with clean-label training, whereas LC limits itself to single-targeted attacks.
[Figure]
clean accuracies.
We remind that β is used for trigger initialization and larger values indicate stealthier triggers.
Here, y → t mean single-targeted attack.
To compare with LC, we provide results that restrict Flareon ’s capability to single-target poisoning only, which translates to poisoning ρ/of all training examples per mini-batch on CIFAR-10.
[Table]