A DVERSARIAL U NLEARNING OF B ACKDOORS VIA I MPLICIT H YPERGRADIENT
A BSTRACT
We propose a minimax formulation for removing backdoors from a given poisoned model based on a small set of clean data.
This formulation encompasses much of prior work on backdoor removal.
We propose the Implicit Backdoor
Adversarial Unlearning (I-BAU) algorithm to solve the minimax.
Unlike previous work, which breaks down the minimax into separate inner and outer problems, our algorithm utilizes the implicit hypergradient to account for the interdependence between inner and outer optimization.
We theoretically analyze its convergence and the generalizability of the robustness gained by solving minimax on clean data to unseen test data.
In our evaluation, we compare I-BAU with six state-ofart backdoor defenses on eleven backdoor attacks over two datasets and various attack settings, including the common setting where the attacker targets one class as well as important but underexplored settings where multiple classes are targeted.
I-BAU’s performance is comparable to and most often signiﬁcantly better than the best baseline.
Particularly, its performance is more robust to the variation on triggers, attack settings, poison ratio, and clean data size.
Moreover, I-BAU requires less computation to take effect; particularly, it is more than 13 × faster than the most efﬁcient baseline in the single-target attack setting.
Furthermore, it can remain effective in the extreme case where the defender can only accessclean samples—a setting where all the baselines fail to produce acceptable results.
I NTRODUCTION
In backdoor attacks adversaries aim to embed predeﬁned triggers into a model during training time such that a test example, when patched with the trigger, is misclassiﬁed into a target class.
For instance, it has been shown that one could use a sticker as the trigger to mislead a road sign classiﬁer to identify STOP signs to speed limit signs (Gu et al., 2017).
Such attacks pose a great challenge to deploy machine learning in mission-critical applications (Li et al., 2020c; 2021a).
Various approaches have been proposed to remove the effect of backdoor attacks from a poisoned model.
One popular class of approaches (Wang et al., 2019; Chen et al., 2019; Guo et al., 2019) is to ﬁrst synthesize the trigger patterns from the model and then unlearn the triggers.
However, these approaches presume that backdoor triggers only target a small portion of classes, thereby becoming ineffective when many classes are targeted by the attacker.
Moreover, they suffer from high computational costs, as they either require synthesizing the trigger for each class independently or training additional models to synthesize the triggers all at once.
Another line of works does not rely on trigger synthesis; instead, it directly mitigates the triggers’ effects via, for example, ﬁnetuning and pruning the model parameters (Liu et al., 2018a) and preprocessing the model input (Qiu et al., 2021).
While these approaches are more efﬁcient than the trigger-synthesis-based approaches, they cannot maintain a good balance between robustness and model accuracy due to the unawareness of potential triggers.
Recent work (Li et al., 2020b) proposes to leverage a teacher model to guide ﬁne-tuning.
However, our empirical studies ﬁnd that the effectiveness of this approach is particularly sensitive to the underlying attack and data augmentation techniques utilized to enrich the ﬁne-tuning dataset.
Overall, it remains a challenge to design a defense that achieves a good balance between model accuracy, robustness, and computational efﬁciency.
To address the challenge, we propose a minimax formulation to remove backdoor triggers from a poisoned model.
The formulation encompasses prior works on trigger synthesis-based defense, which solves the inner and outer problems independently.
Moreover, the formulation does not make any assumption about the backdoor trigger beyond a bounded norm constraint, thus remaining effective across various attack settings.
To solve the minimax, we propose an Implicit Bacdoor Adversarial Unlearning (I-BAU) algorithm based on implicit hypergradients.
Unlike previous work, which breaks down the minimax into separate inner and outer optimization problems, our algorithm derives the implicit hypergradient to account for the interdependence between inner and outer optimization and uses it to update the poisoned model.
We theoretically analyze the convergence of the proposed algorithm.
Moreover, we investigate the generalization error of the minimax formulation, i.e., to what extent the robustness acquired by solving the minimax generalizes to unseen data in the test time.
We present the generalization bounds for both linear models and neural networks.
We conduct a thorough empirical evaluation of I-BAU by comparing it with six state-of-art backdoor defenses on seven backdoor attacks over two datasets and various attack settings, including the common setting where the attacker targets one class and an important but underexplored setting where multiple classes are targeted.
I-BAU’s performance is comparable to and most often signiﬁcantly better than the best baseline.
Particularly, its performance is more robust to the variation on triggers, attack settings, poison ratio, and clean data size.
Moreover, I-BAU requires less computation to take effect; particularly, it is more than 13X faster than the most efﬁcient baseline in the single-target attack setting.
It can still remain effective in the extreme case where the defender can only access 100 clean samples—a setting where all the baselines fail to produce acceptable results.
R ELATED W ORK
Backdoor Attacks.
Backdoor attacks evolve through three stages.
Visble triggers using unrelated patterns (Gu et al., 2017; Chen et al., 2017), or optimized triggers (Liu et al., 2018b; Bagdasaryan & Shmatikov, 2021; Zhao et al., 2020;
Garg et al., 2020) for higher attack efﬁcacy.
While this line’s attack methods achieve high attack success rates and clean accuracy, the triggers are visible and thus easily detected by human eyes.
2) Visually invisible triggers via solving bilevil optimizations regarding the l p norm (Li et al., 2020a), or adopting nature optical effects (Liu et al., 2020; Nguyen & Tran, 2021), and Clean label attacks via feature embeeding (Turner et al., 2019; Saha et al., 2020) for better stealthiness.
For most work from this line, the triggers or the poisons can evade the detection from manual efforts.
3) Invisible triggers considering other latent spaces , e.g., the frequency domian (Zeng et al., 2021; Hammoud & Ghanem, 2021) or auto-encoder feature embedding (Li et al., 2021b), which improved the stealthiness by breaking the fundamental assumptions of a list of defenses.
This paper incorporates successful attacks from these major groups and we show our method provides an effective, and generalizable defense.
Backdoor Defenses.
Backdoor defenses can be divided into four categories: 1) Poison detection via outlier detection regarding functionalities or artifacts (Gao et al., 2019; Chen et al., 2018; Tran et al., 2018; Koh & Liang, 2017; Chou et al., 2020; Zeng et al., 2021), which rely on the modeling of clean samples’ distribution. 2)
Poisoned model identiﬁcation identiﬁes if a given model is backdoored or not (Xu et al., 2019; Wang et al., 2020).
3) Robust training via differential privacy (Du et al., 2019;
Weber et al., 2020) or ensembled/decoupled pipeline (Levine & Feizi, 2020; Jia et al., 2020; 2021; Huang et al., 2022).
This line of work tries to achieve general robustness to withstand outlier’s impact, but may suffer from low clean accuracy.
4) Backdoor removal via trigger synthesising (Wang et al., 2019; Chen et al., 2019; Guo et al., 2019), or preprocessing & ﬁnetuning (Li et al., 2020b;
Borgnia et al., 2020; Qiu et al., 2021).
This line of work serves as the fundamental solution given a poisoned model, but there is still no satisfying solution to attaining robust results across different datasets and triggers.
In this work, we propose a minimax formulation and a solver for backdoor removal.
We attempt to include all cutting-edge methods under this category for comparison.
P ROBLEM F ORMULATION
Attack model.
We assume that an adversary carries out a backdoor attack against a clean training set generated from the distribution D .
The adversary has pre-determined triggers and target classes.
The adversarial goal is to poison the training dataset such that, given a clean test sample x , adding a backdoor pattern δ to x (i.e., x + δ ) will alter the trained classiﬁer output to be a target class ˜ y .
The norm of δ represents the adversary’s manipulative power.
We assume that || δ || ≤ C δ .
In general, the attack can add r backdoored examples into the training set to induce the association between the trigger δ and the target class ˜ y .
The ratio of the total backdoored examples over the size of the training set is deﬁned as the poison ratio .
For better stealthiness, attacks often aim not to affect the prediction of a test example when the trigger is absent.
Note that in our attack model, we explicitly acknowledge the possibility that multiple trigger patterns and multiple target classes are present.
We will refer to the model trained on the poisoned training set as a poisoned model .
Defense goal.
We consdier that the defender is given a poisoned classiﬁer f θ poi and an extra set of clean data D
= { ( x i , y i ) } ni =1 from D .
With f θ poi and D , the defender aims to build a model f θ ∗ immunne to backdoors, i.e., f θ ∗ ( x + δ )
=
f θ ∗ ( x ) .
Note that the size of available clean examples is assumed to be much smaller than the size necessary to retrain a high-accuracy model from scratch.
Minimax formulation for defense.
To achieve the defense goal, we want the resulting classiﬁer to maintain the correct label even if the attacker patches the backdoor trigger to a given input.
This intuition naturally leads to the following minimax optimization formulation of backdoor removal:
[Equation]
where L is the loss function.
Note that this formulation looks similar to adversarial training for evasion attacks (a.k.a.
adversarial examples) (Madry et al., 2017).
The main distinction is that in evasion attacks, the adversarial perturbation is speciﬁc to an individual data point, while in backdoor attacks, the same perturbation (i.e., the backdoor trigger) is expected to cause misclassiﬁcations for any examples upon being patched.
Hence, in the formulation for backdoor attacks, the same trigger, δ , is applied to every clean sample x i , whereas, in the minimax formulation for evasion attacks, the perturbation is optimized independently for each sample (see Eq.
(2.1), Madry et al.
The minimax formulation for backdoor removal has many advantages.
1) , it gives us a unifying perspective that encompasses much prior work on backdoor removal.
The formulation comprises an inner maximization problem and an outer minimization problem.
Both of these problems have a natural interpretation in the security context.
The inner maximization problem aims to ﬁnd a trigger that causes a high loss for predicting the correct label.
This aligns with the problem of a backdoor attack that misleads the model to predict a predeﬁned incorrect target label.
In our formulation, we choose to maximize the prediction loss for the correct label instead of using the exact backdoor attack objective of minimizing the prediction loss associated with the target label.
We make this design choice because, in reality, the defender does not know the target labels selected by the attacker.
The outer minimization problem is to ﬁnd model parameters so that the “adversarial loss” given by the inner attack problem is minimized.
Existing backdoor removal approaches based on trigger synthesis and unlearning the synthesized trigger (Wang et al., 2019; Chen et al., 2019;
Guo et al., 2019) can be considered as a special instance of this minimax formulation, where they neglect the interdependence between the inner and outer optimization and solve them separately.
2) , the formulation does not make any assumption on the backdoor trigger beyond the bounded norm constraint.
By contrast, much of the existing work makes additional assumptions about backdoor triggers in order to be effective.
For instance, synthesis-based approaches often assume that the classes coinciding with the target classes of the triggers account for only a tiny portion of the total, making those approaches ineffective in countering multiple target cases.
3) , the formulation provides a quantitative measure of backdoor robustness.
In particular, when the parameters θ yield a (nearly) vanishing loss, the corresponding model is perfectly robust to attacks speciﬁed by our attack model on D .
We will discuss the generalization properties of this formulation in Section 5.
Using the results developed there, we can further reason about the robustness on the data distribution D .
A LGORITHM
A natural (but problematic) algorithm design.
Given the empirical success of adversarial training for defending against evasion attacks (Li et al., 2022), one may wonder whether we can solve the minimax for the backdoor attack via adversarial training.
Adversarial training keeps alternating between two steps until the loss converges: (1) solving the inner maximization for a ﬁxed outer minimization variable; and (2) solving the outer minimization by taking a gradient of the loss evaluated at the maximum.
Practically, adversarial training proceeds by ﬁrst generating adversarial perturbations and then updating the model using the gradient calculated from the perturbed data.
For backdoor attacks, the adversarial perturbation needs to fool the model universally on all inputs.
Hence, a natural algorithm to solve the backdoor minimax problem is to tune the model with universal adversarial perturbations.
Universal adversarial perturbations have been studied in the past, and there exists an off-the-shelf technique to create such perturbations (Moosavi-Dezfooli et al., 2017).
However, we found that this simple algorithm is highly unstable.
Figure 1 illustrates the change of the model accuracy and robustness (measured in terms of the attack success rate (ASR)) for 20 runs as adversarial training proceeds.
It can be seen that the ASR varies signiﬁcantly across different runs.
Within a single run, while the ASR decays as a whole, the decrement is mostly erratic and slow.
There are two issues with naive adversarial training with universal perturbations.
First, the performance of the existing universal perturbation algorithm is unstable.
At its core, it generates adversarial perturbations for each example and adds the perturbations together.
The addition may not always lead to a perturbation that can fool all examples; instead, the addition operation may cancel the effect of individual perturbations.
More importantly, the decoupling between solving the inner maximization and the outer minimization in adversarial training is often justiﬁed by Danskin’s Theorem (Danskin, 2012), which states that the gradient of the inner function involving the maximization term is simply given by the gradient of the function evaluated at this maximum.
In other words, letting δ ∗ = arg max || δ ||≤ C δ H ( δ, θ ) , we have ∇ θ max || δ ||≤ C δ H ( δ, θ ) =
∇ θ H ( δ ∗ , θ ) .
However, in practice, due to stochastic gradient descent, it is impossible to solve the inner maximization optimally.
Worse yet, the underperformance of the existing universal perturbation algorithm makes it even harder to approach the maximum.
Moreover, Danskin’s Theorem only holds for convex loss functions.
Due to the violation of maximum condition and convexity, it is unsuitable for utilizing
Danskin’s Theorem, and hence, adversarial training is not justiﬁed.
δ t +
[Equation]
Proposed algorithm.
We propose an algorithm to solve the minimax problem in (1) that does not require the loss function to be convex and is more robust to the approximation error caused by not being able to solve the inner maximization problem to global or even local optimality.
Let δ ( θ ) be an arbitrary suboptimal solution to max || δ ||≤ C δ H ( δ, θ ) .
Let ψ ( θ ) :
= H ( δ ( θ ) , θ ) be the objective function evaluated at the suboptimal point; if δ ( θ ) is a stationary point, we make an distinction by deﬁning the corresponding function as ψ ∗ ( θ ) .
Denote ∇ 1 H ( δ, θ ) and ∇ 2 H ( δ, θ ) as the partial derivatives with respect to the ﬁrst and second variable, respectively, and ∇ 21 H ( δ, θ ) and ∇ 21 , 2 H ( δ, θ ) as the second-order derivatives of H with respect to the ﬁrst variable and the mixed variables, respectively.
The gradient of ψ with respect to θ is given by
[Equation]
[Table]
[Equation]
The direct gradients are easy to compute.
For instance, suppose H is the loss of a neural network, then ∇ 1 H ( δ ( θ ) , θ ) and ∇ 2 H ( δ ( θ ) , θ ) are the gradients with respect to the network input and the model parameters, respectively.
However, the response Jacobian is intractable to obtain because we must compute the change rate of the suboptimal solution to the inner maximization problem with respect to θ .
When δ ( θ ) satsiﬁes the ﬁrst-order stationarity condition, i.e., ∇ 1 H ( δ ( θ ) , θ ) = 0 , and assuming that ∇ 21 H ( δ ( θ ) , θ ) is invertible, the response Jacobian is given by
[Equation]
which follows from the implicit function theorem.
Note that ∇ δ ( θ ) plays a role in the Hessian in (2), in the sense that it adjusts each dimension of the gradient ∇ 1 H ( δ ( θ ) , θ ) to the importance of that dimension.
Hessian expresses the importance via curvature, whereas ∇ δ ( θ ) measures the importance based on the sensitivity to the change of θ .
A widely known fact from the optimization literature is that second-order optimization algorithms are tolerant to the inaccuracy of Hessian (Byrd et al., 2011).
Based on this intuition, we propose to approximate ∇ δ ( θ ) with a suboptimal solution by (3).
In practice, the approximation is addressed with an iterative solver of limited rounds (e.g., conjugated gradient algorithm (Rajeswaran et al., 2019) or ﬁxed-point algorithm (Grazzi et al., 2020)) along with the reverse mode of automatic differentiation (Griewank & Walther, 2008).
Finally, to solve (1), we plug in the approximate response Jacobian into (2).
Then, we use ∇ ˜ ψ ( θ ) (referred to as implicit hypergradient hereinafter) to perform gradient descent and update the poisoned model.
Note that the implicit hypergradient only depends on the value of δ ( θ ) instead of its optimization path; hence, it can be implemented in a memory-efﬁcient manner compared to the explicit way of solving bi-level optimizations (Grazzi et al., 2020).
The overall algorithm, dubbed Implicit
Backdoor Adversarial Unlearning , for (cid:96) 2 -norm bounded attacks is presented in Algorithm 1.
Generalizing the algorithm to other norms is straightforward.
We found that imposing a large norm bound in I-BAU has no discernible effect on clean accuracy (Appendix A.3.1).
This empirical observation enables the proposed I-BAU to be ﬂexible enough to deal with different scales of backdoor triggers in practice.
Further details on the the iterative solver with the suboptimum solution and memory complexity analysis are provided in Appendix A.3.
T HEORETICAL A NALYSIS
In this section, we analyze the convergence of the I-BAU.
We also study the generalization properties of the minimax backdoor removal formulation.
Assuming that by solving the minimax, we obtain a model such that all points in the clean set D are robust to triggers of a speciﬁc norm, we would like to reason about to what extent an unseen point from the underlying data distribution D is trigger-robust.
Convergence Bound: Suppose that H ( · , θ ) is µ H ( θ ) -strongly convex and L H ( θ ) -Lipschitz smooth, where µ H ( · ) and L H ( · ) are continuously differentiable.
Deﬁne α
δ ) =
2 L H ( θ )+ µ H ( θ ) , and let δ ( θ ) be the unique ﬁxed point of δ
+ α ( θ ) ∇ 1 H ( δ, θ ) .
Then, there exists L H,θ s.t. sup
[Table]
δ (cid:107)∇ 1 H ( δ, θ ) (cid:107) ≤
L H,θ (Grazzi et al., 2020).
We make the following assumptions:
• Lipschitz continuity of direct gradients : ∇ 1 H ( · , θ ) and ∇ 2 H ( · , θ ) are Lipschitz continuous functions of θ (with Lipschitz constants η 1 ,θ and η 2 ,θ , repectively).
• Lipschitz continuity and norm boundedness of second-order terms : ∇ 21 H ( · , θ ) is ˆ ρ 1 ,θ − Lipschitz continuous, and ∇ 22 , 1 H ( · , θ ) is ˆ ρ 2 ,θ -Lipschitz continuous.
Also, the cross derivative term is norm bounded by L h,θ ≥ (cid:13)(cid:13) ∇ 221 H ( δ ( θ ) , θ ) (cid:13)(cid:13) .
• Asymptotic convergence of δ t ( θ ) to δ ( θ ) : (cid:107) δ t ( θ ) − δ ( θ ) (cid:107) ≤ ρ θ ( t ) (cid:107) δ ( θ ) (cid:107) , where ρ θ ( t ) is such that ρ θ ( t ) ≤ 1 , and ρ θ ( t ) → 0 as t → + ∞ .
Theorem 1 Consider the approximate hypergradient ∇ ˜ ψ
( θ i ) computed by Algorithm 1 (line 7).
For every inner and outer iterate t, i ∈ N , we have:
[Equation]
where q θ i :=
[Equation]
θi
[Equation]
h
[Equation]
, and ﬁnally c 3 ( θ i ) :=
L
[Equation] [Equation]
L
[Equation] [Equation]
i ) 1 −
[Equation]
θi .
As the inner epoch number t increases, the estimated gradient becomes more accurate.
Under the assumption that H ( · , θ ) is convex, the solution converges.
We show the full proof in Appendix A.1.
Generalization Bound for Linear Models: Consider a class of linear classiﬁers θ ∈ Θ where the weights are norm-bounded by C θ .
Deﬁne χ to be the upper bound of any input sample x ∈
D , i.e., (cid:107) x (cid:107)≤ χ .
Let the empirical risk deﬁned as ˆ R γ ( θ ) ≤ n −
[Equation]
(cid:105) with a margin γ > 0 .
Theorem 2 For any linear model θ and ξ ∈ (0 , 1) , the following holds with a probability of at least 1 − ξ over (( x i , y i ))
ni =1 :
[Equation]
When the number of clean samples, n , gets larger, the generalizability of the adversarial unlearning on linear models becomes better.
It is also interesting to compare the adversarial generalization bound for backdoor attacks with the bound for evasion attacks in (Yin et al., 2019), which has an additional √ d factor in the second term of the bound.
Hence, for inputs of large dimensions, the adversarial training for backdoor attacks is expected to be more generalizable than that for evasion attacks.
The detailed proof is shown in the Appendix A.2.1.
Generalization Bound for Neural Networks: Consider a neural network with L ﬁxed nonlinearities ( σ 1 , . . .
, σ L ) , where σ i is (cid:37) i -Lipschitz
(e.g., coordinate-wise ReLU, max-pooling as discussed in (Bartlett et al., 2017)) and σ i (0) = 0 .
Let L weight matrices be denoted by ( A 1 , . . .
, A L ) with dimensions ( d 0 , . . .
, d L ) , respectively, where d 0 = d is the number of input dimensions, d L = C is the number of classes, and W = max( d 0 , . . .
, d L ) is the highest dimension layer.
Assume that the weight matrices satisfy (cid:107) A i (cid:107) 2 , 1 ≤ a i , (cid:107) A 1 (cid:107) 2 , 1 ≤ a 0 , and A i ∈ R d i × m i .
Let a 1 = a 0 (cid:0) 1 + m 1 n 1 / 2 C δ (cid:1) .
The whole network’s output regarding a backdoored sample is denoted as: σ L ( A L σ L − 1 ( A L − 1 . . .
σ 1 ( A 1 ( x + δ )) . . . )) .
Let (cid:107)
A i (cid:107) σ be bounded by the spectral norms s i .
Theorem 3 For any neural network model θ and ξ ∈ (0 , 1) , the following holds with a probability of at least 1 − ξ over (( x i , y i ))
ni =1 :
[Equation]
When the number of clean samples, n , gets larger, the generalizability of the unlearning on neural networks becomes better.
Compared with the generalization bound for regular training in (Bartlett et al., 2017), our bound has an additional n term in (cid:112) ln(2 W ( W + n )) and ( (cid:107) X (cid:107) p + √ n ) , which indicates harder generalizability than regular learning problem.
However, empirically, we ﬁnd our solution, I-BAU, is still of excellent generalizability even if only 100 clean samples are accessible.
Our proof is enabled by recognizing that the backdoor perturbation, δ , is required to be the same across all the poison samples and therefore can be treated as additional dimensions of the model parameters.
The proof implements Dudley entropy integral to bound the Rademacher complexity.
The full details are deferred to Appendix A.2.2.
We also empirically validate Theorem 3 against W and n in Appendix A.2.3.
E VALUATION
Our evaluation aims to answer the following questions: (1) Efﬁcacy: Can I-BAU effectively remove various backdoor triggers?
(2) Stability: Can I-BAU be consistently effective across different runs?
(3) Sensitivity: How sensitive is I-BAU to the poison ratio and the size of the available clean set?
Efﬁciency: How efﬁcient is I-BAU?
We use a simpliﬁed VGG model (Simonyan & Zisserman, 2014) as the target model for all experiments and set C δ = 10 as the norm constraint for implementing
I-BAU.
The details of experimental settings and model architecture can be found in Appendix A.4.
6. E FFICACY
We evaluate I-BAU’s efﬁcacy against three attack settings: 1) One-trigger-one-target attack, in which the attacker uses only one trigger, and there is only one target label.
This setting is most commonly considered in existing defense works.
2) One-trigger-all-to-all attack, in which the attacker uses only one trigger but aims to mislead predictions from i to i +1 , where i is the ground truth label (Gu et al., 2017).
3) Multi-trigger-multi-target attack, where the attacker uses multiple distinct triggers, each targeting a different label.
We will refer to the ﬁrst two settings as “one-trigger setting” and the last setting as “multi-trigger setting.”
For each setting, we study seven different backdoor triggers in the main text, namely, BadNets white square trigger (BadNets) (Gu et al., 2017), Hello Kitty blending trigger (Blend) (Chen et al., 2017), l 0 norm constraint invisible trigger ( l 0 inv) (Li et al., 2020a), l 2 norm constraint invisible trigger ( l 2 inv) (Li et al., 2020a), Smooth trigger (frequency invisble trigger) (Smooth) (Zeng et al., 2021), Trojan square (Troj SQ)
(Liu et al., 2018b), and
Trojan watermark (Troj WM)
(Liu et al., 2018b).
These trigger patterns are illustrated in Figure 3 in the Appendix.
Given that I-BAU’s fundamental formulation takes backdoor noise as additive noise, one may be curious about how I-BAU’s performance mitigates non-additive backdoor attacks.
We have included case studies on using I-BAU mitigating four non-additive triggers (semantical replacement, WaNet (Nguyen & Tran, 2021), IAB attack (Nguyen & Tran, 2020)) and hidden trigger attack (Saha et al., 2020) (non-additive poisoning) to illustrate the effectiveness (Appendix A.5).
For all experiments in the mian text, we adopt a large poison ratio of 20%, a severe attack case for defenders.
To acquire a poisoned model, we train on the poisoned dataset for 50 epochs.
We compare I-BAU with six state-of-art defenses: Neural Cleanse (NC) (Wang et al., 2019), Deepinspect (DI) (Chen et al., 2019), TABOR (Guo et al., 2019), Fine-pruning (FP)
(Liu et al
Neural Attention Distillation (NAD) (Li et al., 2020b), and Differential Privacy training (DP) (Du et al., 2019).
Note that DP requires access to the poisoned data; hence, its attack model is different from the attack model of the other baselines and our method.
In the following tables, we use [ ASR ] to mark the results that fail to reduce the attack success rate (ASR) below 20%; we use [ ACC ] to mark the results where the accuracy (ACC) on clean data drops by more than 10%; we use [ ACC or ASR ] to mark the best result among the six baselines; ﬁnally, we use [ ACC or ASR ] to mark the results from
I-BAU that is comparable to or signiﬁcantly better than the best result among the baselines.
We consider I-BAU’s result as comparable if 1) the ACC gap is less than 1%, 2) the ASR gap is less than 4%, or the ASR of I-BAU is close to the label percental (i.e., the probability of random guessing for outputting the target label, 10% for the CIFAR-10, 2.3 % for the GTSRB).
One-trigger Setting: Table 1 presents the defense results on the CIFAR-10 dataset.
CIFARcontains 60,000 samples.
We use 50,000 samples as training data, among which 10,000 samples are poisoned.
We used 5,000 separate samples as the clean set accessible to the defender for conducting each defense (e.g., via unlearning, ﬁnetuning, trigger synthesis) except DP.
The remaining 5, samples are used to assess the defense result.
The left column depicts the attack cases.
The ﬁrst seven cases each contain only one target label.
The all-to-all case represents the case in which we use the BadNets trigger to carry out the one-trigger-all-to-all attack.
As shown in the table, all singletarget attacks are capable of achieving an ASR close to 100% with no defenses.
It is empirically observed that the ASR of the all-to-all attack is upper-bounded by the ACC.
Intuitively, the model needs to classify the clean samples correctly to classify the corresponding backdoored samples with triggers to the next class.
Thus, we tune the model to produce an ASR that is close to the ACC.
[Table]
Model performance on CIFAR-10 is particularly sensitive to ﬁnetuning and unlearning, which will result in a decrease in the ACC in general.
Thus, for a fair comparison, we show the I-BAU results in Table 1 when the ACC falls to the same level as the most effective baseline.
For instance, for
BadNets attacks, we present the ASR result of I-BAU when the ACC of I-BAU decreases to a value similar to the ACC of TABOR, which is the best defense baseline against BadNets.
[Table]
As shown in Table 1, the performance of the baselines exhibits large variance across different triggers.
Speciﬁcally, each baseline underperforms in at least three attack cases (marked by ( [ ASR ] or [ ACC ]) )
The defenses based on trigger synthesis (including NC, DI, and TABOR) failed to confront the all-to-all attack case, where all the labels were targeted.
This is because this setting violates their assumption that target classes only account for the minority of training data.
Particularly, TABOR fails to detect any backdoor triggers in this case).
For DP, we ﬁne-tune a noise multiplier effective to mitigate all attacks, yet also leads to bad ACC.
On the other hand, I-BAU robustly mitigates all triggers without signiﬁcantly affecting the ACC.
Compared to the best result among the state-of-art baselines ( [ ACC or ASR ] ), I-BAU’s is comparable to or much better ( [ ACC or ASR ] ) under most of the settings.
The only setting where I-BAU underperforms the best baseline is Smooth triggers.
FP is the only effective baseline in this setting.
But interestingly, this setting is also the only setting where
FP is effective; in other words, its performance is highly dependent on the underlying trigger.
[Table]
Table 2 shows the evaluation on the GTSRB dataset, which contains 39,209 training data and 12, test data of 43 different classes.
The experimental setting is the same as the one on CIFAR, except for the data split, which is discussed in the Appendix A.4.1.
We ﬁnd that model performance on GTSRB is not sensitive to the tuning procedure.
As 5,000 clean samples are available to the defender, the
ACC of the model increases after incorporating the defense.
NAD’s performance highly depends on the pre-designed preprocessing procedure adopted during ﬁne-tuning.
While it produces the best defense results for some settings on CIFAR-10, it suffers from a large degradation of ACC on
GTSRB.
Once again, I-BAU is the only defense method that remains effective for all attack settings.
Multi-trigger Setting: Table 3 shows the results on CIFAR and GTSRB in a 7-trigger-7-target setting, in which each trigger targets a different label.
We adopt the same poison ratio for each (20%) and then combine the seven distinct poisoned datasets to obtain the ﬁnal datasets (size 350,000 for
CIFAR-10, and 274,463 for the GTSRB).
We show the average ASR and the speciﬁc ASR.
On CIFAR-10, since the target classes are no longer the minority (i.e., 7/10 labels are targeted), the baselines based on trigger synthesis, which make the minority assumption, are ineffective.
Particularly, NC fails to detect any triggers in this setting.
NAD is the only baseline able to evaluate well on CIFAR-10, and our methods achieve comparable performance.
However, similar to the one-trigger setting, NAD’s performance requires the customization of preprocessing to different datasets.
The default preprocessing cannot maintain the same performance on GTSRB; e.g., the random ﬂipping—a beneﬁcial preprocessing step for
CIFAR—completely alters the semantics of images in GTSRB.
On the other hand, I-BAU effectively reduces ASR while maintaining the ACC for both datasets with no changes needed.
Moreover, in the Appendix, we visualize the distribution of poisoned and clean examples in the feature space before and after applying
I-BAU, which shows that I-BAU can place the poisoned examples back in the cluster with correct labels.
6. S TABILITY
In Section 4, we mentioned that the naive heuristic based on adversarial learning with existing universal perturbation suffers from erratic performance.
Here, we aim to evaluate whether I-BAU can overcome this problem.
We adopt the same setting for the two on the CIFAR-10 using BadNets and observe the change in ACC and ASR over different iterations for each run.
As shown in Figure 1, the ASR decreases very quickly; indeed, the attack can be mitigated by just one single iteration of outer minimization.
Also, within every single run, the ASR decreases more smoothly compared to the naive heuristic.
Notably, I-BAU attains a slightly higher ACC than the naive heuristic.
6. S ENSITIVITY
We evaluate the sensitivity of each defense to the poison ratio and the size of clean samples.
The experiments were performed on Trojan WM on CIFAR-10 to exemplify the results.
Table 4 shows the sensitivity to the poison ratio.
Observe that as the poison ratio drops, TABOR becomes ineffective at synthesizing triggers and ultimately fails to remove the triggers; NC, however, becomes the most effective baseline.
While DP’s ASR is signiﬁcantly worse than the effective baselines, its performance gets slightly better as the poison ratio drops.
DP attempts to restrict the impact of each training data point on the learning outcome via noising the gradient.
As a side effect, it hinders learning from good data, thereby leading to poor ACC in general.
Compared to the baselines, I-BAU exhibits the least sensitivity to the poison ratio, maintaining good ACC and ASR across different ratios.
[Table]
Table 5 shows sensitivity to available clean data’s size.
As DP is independent from the clean set (which trains a general robust model against all perturbations from scratch), we drop it from the comparison.
We see that as the number of clean samples drops, the performance of all defenses declines.
I-BAU is least sensitive to the size of clean samples; even in extreme case with onlyclean samples available, I-BAU still maintains an acceptable performance of removing backdoors.
[Table]
6. E FFICIENCY
[Table]
Finally
[Table]
we compare the efﬁciency of defenses.
We use the onetrigger-one-target attack setting to exemplify the result.
Tableshows the average runtime for each defense to take effect (i
[Table]
e., mitigating the ASR to < 20% ).
As I-BAU can mitigate the attacks in one iteration, it only takes 6.82 s on average on CIFAR-10 and 7.84 s on GTSRB.
Note that NC and TABOR need to go through each label independently for trigger synthesis; thus, the total runtime is proportional to the number of labels.
Especially, it takes much more time for the two to take effect on GTSRB than on
CIFAR-10.
In difﬁcult attack cases, such as all-to-all attacks and multi-trigger-multi-target attacks, I-BAU requires more rounds to be operative but remains the only effective one across all settings with high efﬁciency and efﬁcacy.
The Appendix shows the performance of I-BAU over different rounds under multi-target attack cases.
Theoretical analysis and comparison of the time complexity of I-BAU are presented in Appendix A.3.3.
C ONCLUSION
In this work, we proposed a minimax formulation of the backdoor removal problem.
This formulation encompassed the objective of the existing unlearning work without making assumptions about attack strategies or trigger patterns.
To solve the proposed minimax, we proposed I-BAU using implicit hypergradients.
A theoretical analysis veriﬁed the convergence and generalizability of the proposed I-BAU or minimax formulation.
A comprehensive empirical study established that I-BAU is the only generalizable defense across all evaluated eleven backdoor attacks.
The defense results are comparable to or exceed the best results obtained by combining six existing state-of-the-art techniques.
Meanwhile, I-BAU is less sensitive to poison rate and is effective in extreme cases where the defender has access to only 100 clean samples.
Finally, under the standard one-trigger-one-target circumstances, I-BAU can achieve an effective defense in an average of 7.35 s.
A A PPENDIX A. C ONVERGENCE B OUND
Consider the minimax formulation of backdoor unlearning deﬁned in (1).
To simply the notation, we will use θ instead of θ i , unless otherwise speciﬁed.
We start by deﬁning Φ( δ, θ ) =
δ + α ( θ ) ∇ 1 H ( δ, θ ) , where H (sum of cross-entropies) is twice continuously differentiable w.r.t.
both δ and θ .
Recall that H ( · , θ ) is µ H ( θ ) -strongly convex and L H ( θ ) -Lipschitz smooth, where µ H ( · ) and
L H ( · ) are continuously differentiable.
By setting the step size α ( θ ) =
[Equation]
H ( θ )+ µ H ( θ ) , it can be shown that Φ( · , θ ) is a contraction for some coefﬁcient q θ := max { 1 − α ( θ ) µ H ( θ ) , α ( θ ) L H ( θ ) − 1 } .
The optimal choice of the step-size leads to q θ = κ ( θ ) − 1 κ ( θ )+1 , where κ ( θ ) =
L H ( θ ) µ H ( θ ) .
Note that, for every
i
[Equation] [Equation]
Hence, the condition number of ∇ 21 H ( δ i ( θ ) , θ ) is smaller than κ ( θ ) (Grazzi et al., 2020).
Let ∂ 1 Φ( δ, θ ) and ∂ 2 Φ( δ, θ ) denote the partial Jacobians of Φ at ( δ, θ ) w.r.t.
the ﬁrst and the second variables, respectively.
We can write the derivatives of Φ as:
[Equation]
When evaluated at ( δ ( θ ) , θ ) , the partial Jacobian w.r.t.
the second variable, ∂ 2 Φ( δ ( θ ) , θ ) , can be simpliﬁed as:
[Equation]
Thus, (cid:107) ∂ 2 Φ( δ ( θ ) , θ ) (cid:107) =
α ( θ ) (cid:107)∇ 221 H ( δ ( θ ) , θ ) (cid:107) .
Following the notations from (Grazzi et al., 2020), we create the bound of the partial Jacobian of Φ to be L Φ ,θ ≥ (cid:107) ∂ 2 Φ( δ ( θ ) , θ ) (cid:107) .
By our assumption, the bound for the cross derivative term δ is L h,θ ≥ (cid:13)(cid:13) ∇ 221 H ( δ ( θ ) , θ ) (cid:13)(cid:13) .
Hence, we can choose:
[Equation]
Let ∆ ∂ 1 Φ := (cid:107) ∂ 1 Φ ( δ 1 , θ ) − ∂ 1 Φ ( δ 2 , θ ) (cid:107) and ∆ ∂ 2 Φ := (cid:107) ∂ 2 Φ ( δ 1 , θ ) − ∂ 2 Φ ( δ 2 , θ ) (cid:107) .
By assumption on the Lipschitz continuity and norm boundedness of second-order terms , we have :
[Equation] [Equation]
and:
[Equation]
Thus, ∂ 1 Φ( · , θ ) is Lipschitz continuous with constant
[Equation]
13) and ∂ 2 Φ( · , θ ) is Lipschitz continuous with constant
[Equation]
Recall the result from (Grazzi et al., 2020):
Lemma
The approximate hypergradient has an error bounded by
[Equation]
15) where
[Equation]
The proof of Theorem 1 uses the inequality from Lemma 1 with the constants speciﬁed in (10), (13), and (14).
A. G ENERALIZATION B OUNDS
Following the notations from (Bartlett et al., 2017), we deﬁne a general margin operator M ( v, y ) :
= v y − max j (cid:54) =
y v j , and the ramp loss (cid:96) γ with a given margin γ :
[Equation]
According to (Yin et al., 2019), the population risk against a perturbation δ is given by:
[Equation]
18) and the empirical risk is given by:
[Equation]
Note that R γ ( θ ) and ˆ R γ ( θ ) are the upper bounds of the fraction of errors on the source distribution and the dataset used for unlearning, D san , respectively.
Finally, given a set of real-valued functions
H , recall the Rademacher complexity as:
[Equation]
where ς j ∈ {± 1 } is the Rademacher random variable.
The following bound of the adversarial unlearning can be derived using standard tools in Rademacher complexity.
Lemma
Given unlearned models Θ with θ ∈ Θ and some margin γ > 0 , deﬁne:
[Equation]
The following holds with the probability of at least 1 − ξ over the clean dataset, D san
(see Algorithm 1) for every θ ∈ Θ :
[Equation]
To instantiate this bound, we only need to control the Rademacher complexity, (cid:60) (cid:0) (Θ γ ) | D san (cid:1) , for linear models and neural networks.
A.2. P ROOF FOR LINEAR MODELS
Deﬁne the set of linear, poisoned models under perturbation δ :
[Equation]
23) where θ can be regarded as a weight matrix (cid:107) θ (cid:107) bounded by C θ .
Recall that the perturbation norm (cid:107) δ (cid:107) is bounded by C δ , and the input l 2 norm (cid:107) x (cid:107) 2 is bounded by χ .
We can proceed to evaluate the
[Table]
[Table]
(cid:13)(cid:13)(cid:13) (cid:13) (cid:13) (cid:13)
[Equation]
n (cid:13)(cid:13)(cid:13) (cid:13) (cid:13) (cid:13) δ n (cid:88)
[Equation]
=
ς j (cid:13)(cid:13)(cid:13) (cid:13) (cid:13) (cid:13) 
[Equation]
(Triangle inequality)
[Equation]
(cid:124) (cid:123)(cid:122) (cid:125)
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) δ n (cid:88)
[Equation]
=
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
(cid:123)(cid:122) (cid:125)
[Equation]
 .
In particular, A can be bounded by:
[Equation]
n
[Equation]
Jensen’s inequality) ≤n (cid:118)(cid:117)(cid:117)(cid:116)
[Equation]
Similarly, B can be bounded by:
[Equation]
(given that (cid:107) δ (cid:107) ≤ C δ )
[Equation]
Thus, we can bound the Rademacher complexity as following:
[Equation]
Substituting (27) to the (22) completes the proof of Theorem 2.
A.2. P ROOF FOR NEURAL NETWORKS
To instantiate the bound with Rademacher complexity for neural networks, we will follow the idea from (Bartlett et al., 2017) and use covering numbers to bound the Rademacher complexity, (cid:60) (cid:0) (Θ γ ) | D san (cid:1) .
Let N ( U, (cid:15), (cid:107) · (cid:107) ) denotes the least cardinality of any subset V ⊆ U that covers U at a scale (cid:15) with norm (cid:107)·(cid:107) , i.e., sup A ∈ U min B ∈ V (cid:107) A − B (cid:107) ≤ (cid:15) .
Recall the Dudley entropy integral below.
Lemma 3 Assume that the input values are norm bounded by 1 ,
[Equation]
Thus, the derivation of the generalization bound of (1) with neural networks is reduced to the problem of ﬁnding the bound for the covering number of the set of all neural networks,
N (cid:0) (Θ γ ) | D san , (cid:15), (cid:107) · (cid:107) 2 (cid:1) .
The full problem can be divided into four steps: ( I ) ﬁnding a matrixcovering bound for the afﬁne transformation of the input layer of poisoned models, conducted in this section; ( II ) ﬁnding a matrix-covering bound for the afﬁne transformation of the following layers after the ﬁrst layer, provided in (Bartlett et al., 2017); ( III ) obtaining a covering number bound for entire networks using induction on layers; ( IV ) accomplishing the complete generalization bound for neural networks by applying the covering number to (22) and (28).
Step (I) :
Input Layer Matrix Covering .
The covering number of the input layer considers the matrix product, ˆ Z
=
( X + ∆)
A 1 , where A 1 ∈ R d × m is the weight matrix of the input layer,
X ∈ R n × d is the input data, and ∆ =
[ δ, δ, . . .
, δ ] (cid:124) (cid:123)(cid:122) (cid:125) n (cid:62) is the UNO perturbation vector repeated n times to get the same shape as X .
Note that ˆ Z can be rewritten as:
[Equation]
where I n is the n × n identity matrix, ˆ X is an n × ( d + n ) matrix, and ˆ A 1 is a ( d + n ) × m matrix.
Now, the goal of step ( I ) is to ﬁnd the covering number for the matrix product, ˆ X ˆ A 1 .
Given X ∈ R n × d , we can obtain the normalized matrix Y ∈ R n × d by rescaling the columns of X to have unit p -norm: Y : ,j :=
X : ,j / (cid:107) X : ,j (cid:107) p .
Let N 0 :
= 2( n + d ) m , and deﬁne (cid:110)
ˆ V 1 , . . .
, ˆ V N 0 (cid:111) := (cid:8) g
[ Y
I n ] e i e (cid:62) j : g ∈ {− 1 , +1 } , i ∈ { 1 , . . .
, d + n } , j ∈ { 1 , . . .
, m } (cid:9) .
For p ≤ 2 , the results from (Bartlett et al., 2017) implies:
[Equation]
.N 0 } (cid:107) [ Y
I n ] e i (cid:107) 2 = max i (cid:13)(cid:13)(cid:13)
ˆ X e i (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
ˆ
X
e i (cid:13)(cid:13)(cid:13) p ≤ 1 .
Deﬁne α 0 ∈ R ( d + n ) × m to be a “rescaling matrix”:
α 0 =    (cid:107) X : , 1 (cid:107) p . . .
(cid:107) X : , 1 (cid:107) p (cid:107) X : , 2 (cid:107) p . . .
(cid:107) X : , 2 (cid:107) p ...
(cid:107) X : ,d (cid:107) p . . . (cid:107) X : ,d (cid:107) p . . .

  , (32) where the purpose of α 0 is to annul the rescaling of ˆ X introduced by ˆ Y =
[ Y
I n ] , i.e., ˆ X ˆ A 1 =
ˆ Y ( α 0 (cid:12) ˆ A 1 ) , and (cid:12) denotes the element-wise product.
Let ˆ B :=
α 0 (cid:12) ˆ A 1 .
Then, we have:
[Equation] [Equation]
where conv (cid:16)(cid:110)
ˆ V 1 , . . .
, ˆ V N 0 (cid:111)(cid:17) is the convex hull of (cid:110)
ˆ V 1 , . . .
, ˆ V N 0 (cid:111) .
Given conjugate exponents ( p, q ) and ( r, s ) with p ≤ 2 , by the conjugacy of (cid:107)·(cid:107) p,r and (cid:107)·(cid:107) q,s :
[Equation]
B
[Equation] [Equation]
, |
[Equation]
Deﬁning C as the desired cover of the ﬁrst layer ˆ
X ˆ A 1 :
[Equation]
where, k :=
[Equation]
By construction, |C| ≤ [ N 0 ] k .
Now, we prove that
C is the desired cover set.
The technique is generalized from (Bartlett et al., 2017) to backdoor unlearning.
Consider the case of (cid:107) A 1 (cid:107) 2 , 1 , i.e., ( q, s ) =
(2 , 1) , and let (cid:107)
A 1 (cid:107) 2 , 1 ≤ a 0 , then we get: (cid:107)
α 0 (cid:107) p,r = (cid:13)(cid:13)(cid:13)(cid:16) (cid:107) α : , 1 (cid:107) p , . . .
, (cid:107) α : ,m (cid:107) p (cid:17)(cid:13)(cid:13)(cid:13) r
= (cid:13)(cid:13)(cid:13)(cid:13)(cid:18)(cid:13)(cid:13)(cid:13)(cid:16) (cid:107) X : , 1 (cid:107) p , . .
, (cid:107)
[Equation]
i
[Equation]
(cid:107)
[Equation] [Equation]
[Equation]
1 (cid:17)(cid:13)(cid:13)(cid:13) p , . . .
[Equation]
(cid:13)(cid:13)(cid:13)(cid:16) (cid:107) X : , 1 (cid:107) p
[Equation]
[Equation]
(cid:107) X : ,d (cid:107)
[Equation]
, 1 (cid:17)(cid:13)(cid:13)(cid:13) p (cid:19)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
j
[Equation]
X :
[Equation]
≤ m
[Equation]
r (cid:16) (cid:107) X (cid:107) p + n 1 /p (cid:17)
[Equation]
Subsequently, the bound of (cid:13)(cid:13)(cid:13) ˆ A 1 (cid:13)(cid:13)(cid:13) 2 , 1 can be derived from: (cid:13) (cid:13) (cid:13)
ˆ A 1 (cid:13) (cid:13) (cid:13)
[Equation]
[Equation]
(cid:13)(cid:13) (cid:13)(cid:13) (cid:20) A∆ A(cid:21)(cid:13)(cid:13) (cid:13)(cid:13) 2 ,=
(cid:13)(cid:13)(cid:13)(cid:13)(cid:18)(cid:13)(cid:13)(cid:13)(cid:13)(cid:20)
A :1 ∆
A :1 (cid:21)(cid:13)(cid:13)(cid:13)(cid:13) 2 , . . .
, (cid:13)(cid:13)(cid:13)(cid:13)(cid:20) A : m ∆ A : m (cid:21)(cid:13)(cid:13)(cid:13)(cid:13) 2 (cid:19)(cid:13)(cid:13)(cid:13)(cid:13)= (cid:13)(cid:13)(cid:13)(cid:13)(cid:16) (cid:107)
A :1 (cid:107) 2
+ (cid:107) ∆
A :1 (cid:107) 2 2 (cid:17) 1 / 2 , . . .
, (cid:16) (cid:107)
A : m (cid:107) 2
+ (cid:107) ∆
A : m (cid:107) 2 2 (cid:17) 1 / 2 (cid:13)(cid:13)(cid:13)(cid:13) , (37) where we have:
[Equation]
Therefore
[Equation]
substituting (38) to (37) gives us the bound:
[Equation]
[Equation]
Let a 1 = a 0 (cid:0) 1 + mn 1 / 2 C δ (cid:1) .
Given the case of (cid:107) A 1 (cid:107) 2 , 1 , we can obtain the bound for (cid:107) ˆ B (cid:107) 1 :
[Equation]
ˆ B (cid:107)
[Equation]
Thus, combining (31) and following the Maurey lemma (Pisier (1981), Zhang (2002), Lemma 1), we have:
[Equation]
which shows that the desired cover element is in C .
Theorem
In the case of the input layer of a poisoned model, θ , consider ∆ as the UNO perturbation matrix with n rows of δ , and A 1 ∈ R d × m to be the weight metrix of the ﬁrst layer.
The covering resolution (cid:15) 1 is given.
Deﬁning a 1 := a 0 (cid:0) 1 + mn 1 / 2 C δ (cid:1) , where a 0 is the bound of (cid:107) A 1 (cid:107) 2 , 1 , for any input X ∈ R n × d , the convering number is bounded as follows:
[Equation]
≤
[Equation] [Equation]
Step (II) :
Other Layer’s Matrix Covering .
Let’s deﬁne Z ∈ R n i ×
d i as the input of a speciﬁc layer of the neural network, and A i ∈ R d i × m i as the weight matrix of that layer.
Given conjugate exponents ( p, q ) and ( r, s ) with p ≤ 2 , let (cid:107)
A i (cid:107) q,s ≤ a i .
Lastly, the covering resolution of each layer, (cid:15) i , is given.
Since no perturbation is considered in the following layers, we can directly adopt the covering studied in (Bartlett et al., 2017) as follows:
[Equation]
Step (III) :
The Whole Network Covering Bound .
Recall that the whole neural network is structured as follows: F θ ( x ) :
= σ L ( A L σ L − 1 ( A L − 1 . . .
σ 1 ( A 1 x ) . . . )) , where σ i is (cid:37) i -Lipschitz.
Deﬁne two sequences of vector spaces V 1 , . .
V L and W 2 , . .
, W L +1 , where V i has a norm | · | i and W
i has a norm ||| · ||| i .
The linear operators, A i : V i → W i +1 , are associated with some operator norm | A i | i
→ i +1 ≤ s i , i.e., | A i | i → i +1 :
= (cid:107)
A i (cid:107) σ ≤ sup | Z | i ≤ 1 |||
A i Z ||| i +
=
s i .
Then, letting τ := (cid:80) j ≤ L (cid:15) j (cid:37) j (cid:81)
Ll =
j +1 (cid:37) l s l , with given convering resolutions, ( (cid:15) 1 , . . .
, (cid:15) L ) , the neural net images, H X :=
{ F θ ( X + ∆) } , have the covering number bound (Bartlett et al., 2017):
[Equation]
Step (IV) : Proof of Theorem 3 .
The key technique in the remainder of this proof is
• 1) to substitute covering number estimates from (42) and (43) into (44) but • 2) centering the covers at 0 (meaning the cover at layer i ∈ (2 , L ) satisﬁes (cid:107)
A i (cid:107) 2 , 1 ≤ a i , (cid:107) A 1 (cid:107) 2 , 1 ≤ a 0 , and (cid:107) ˆ A 1 (cid:107) 2 , 1 =
(cid:13)(cid:13)(cid:13)(cid:13)(cid:20)
A 1 ∆ A 1 (cid:21)(cid:13)(cid:13)(cid:13)(cid:13) 2 , 1 ≤ a 1 ), and • 3) collecting ( x 1 , . . .
, x n ) as rows of matrix X ∈ R n × d .
To start, the covering number estimate of the whole network from (44) when combined with (42) and (43) (speciﬁcally with p = 2 , s = 1 , and W = max( d 0 , . .
, d L ) ) results in:
[Equation]
N
[Equation]
, (cid:15), (cid:107) · (cid:107) 2 )
[Equation]
I n (cid:21)(cid:19)(cid:27) , (cid:15) i , (cid:107) · (cid:107)(cid:19) ( ∗ ) = sup A 1 ln N (cid:32)(cid:40)
ˆ A 1 (cid:18)(cid:20) X (cid:62) I n (cid:21)(cid:19)
(cid:62) (cid:41) , (cid:15) 1 , (cid:107) · (cid:107)(cid:33) + (cid:80) Li
=2 sup ( A 2 ,...,Ai − 1 ) ∀ j<i ln N (cid:32)(cid:40)
A i F ( ˆ A 1 ,...,
A i − 1 ) (cid:32)(cid:20) X (cid:62)
I n (cid:21) (cid:62) (cid:33)(cid:41) , (cid:15) i , (cid:107) · (cid:107)(cid:33) ≤ sup ˆ Aa 21 (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
ˆ A 1 (cid:32)(cid:34) X (cid:62) I n (cid:35)(cid:33) (cid:62) (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)2 (cid:15)ln (2 W ( W + n ))
+ (cid:80)
Li =2 sup ( A 2 ,...,Ai − 1 ) ∀
j<i
a 2 i (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) F ( ˆ A 1 ,A 2 ,...,Ai − 1 ) (cid:32)(cid:34) X (cid:62)
I n (cid:35)(cid:33) (cid:62)
[Equation]
where ( ∗ ) equality holds since 1) l 2 coverings of a matrix and its transpose are the same, and 2) the cover can be translated by F ( ˆ A 1 ,A 2 ,...,
A i − 1 ) (cid:18)(cid:20) X (cid:62)
I n (cid:21)(cid:19) (cid:62)
A (cid:62) i without changing its cardinality.
We can further simplify (45), by evaluating the following norm for any A i ∈
( A 2 , . . . ,
A L − 1 ) :
[Equation]
which by induction gives
[Equation] [Equation]
Combining (46) and (47), the cover is bounded by:
[Equation]
Let
[Equation]
i
[Equation]
>
i
[Equation]
s
[Equation]
, where
[Equation]
then,
[Equation]
Consider the class of networks
[Equation]
Θ NNγ , obtained by afﬁxing the ramp loss, (cid:96) γ , and the negated margin operator, −M , to the output of the provided network class:
[Equation]
Since ( z, y ) (cid:55)→ (cid:96) γ ( −M ( z, y )) is 2 /γ -Lipschitz w.r.t.
(cid:107) · (cid:107) 2 and deﬁnition of (cid:96) γ , the function class Θ NNγ falls under the setting of (50), the covering number of a set of all neural networks is bounded as follows:
[Equation]
[Equation]
Using the above covering number bound (52) in the Dudley entropy integral (28) with α := 1 / √ n , we achieve the bound for the Rademacher complexity as follows:
[Equation]
We complete the proof of Theorem 3 by substituting (53) in (22).
A.2. E MPERICAL VALIDATION OF THEOREM
In this section, we empirically verify Theorem 3 regarding two variables: the neural network’s width,
W , and the number of clean samples.
The experiment is conducted with poisoned models trained over Trojan WM poisoned CIFAR-10 (poison rate: 20%, target label: 2).
[Table]
Table 7 shows the empirical results of adopting different models with different widths.
All the poisoned models are poisoned with Trojan WM attack using a poison rate of 20%.
We sorted the models according to their maximum width ( W ) in Table 7.
The Error Gap is obtained as the absolute value of the test error subtracted by the training error after conducting the defense.
Based on the observation, the error gap is smaller as the model width grows, indicating better generalizability.
Aligning with Theorem 3, the generalizability has a positive correlation with W .
[Table]
Table 8 shows the empirical results of adopting different numbers of clean samples during the IBAU.
As indicated from the results, a larger number of clean samples would lead to a smaller value of the Error Gap, which indicates a better generalization of unlearning effect from training to unseen data.
Such results aligned with Theorem 3.
A. I MPLEMENTATION D ETAILS AND C OMPLEXITY A NALYSIS
I-BAU does not need to compute the second-order derivative directly.
Instead, it is computed via implementing an approximation of the response Jacobian via an iterative solver (e.g., conjugated gradient algorithm (Rajeswaran et al., 2019) or ﬁxed-point algorithm (Grazzi et al., 2020)) in limited rounds along with the reverse mode of automatic differentiation (Baur & Strassen, 1983; Griewank & Walther, 2008) by treating the problem as a linear system.
Automatic differentiation in reverse mode is a widely used technique in modern deep learning packages such as Tensorﬂow and PyTorch (Baydin et al., 2018).
This section gave the ablation study over the norm bound given in Algorithm 1 and the memory and time complexity analysis and comparisons.
A.3. A BLATION STUDY ON THE l 2 NORM BOUND
This section studies the impact of the preset l 2 bound’s inﬂuence in the I-BAU unlearning scheme.
We tested ﬁve different bounds to illustrate the effects, i.e., 0 . 5 , 5 , 10 , 20 , and Best Efforts, as shown in Figure 2.
The settings of Best Efforts norm bound is that we do not include a norm constrained of the synthesized trigger as long as the trigger’s value is within the image value range (from 0 toin our case with ﬂoat type images).
The l 2 norm of launching the Trojan WM attack is 8.739 (measured by comparing with a zero matrix of the same size).
As shown in Figure 2 a larger norm bound leads to a more robust and accurate synthesis of the potential trigger on the GTSRB.
Especially the l 2 norm bounds that are greater than the attack trigger’s l 2 bound would lead to an effective defense in terms of low ASR and low impacts over the clean ACC.
Based on Figure 2, we ﬁnd that a large norm bound does not signiﬁcantly impact over the clean ACC, namely the tread-off between the l 2 bound and the ACC drop is not substantial.
In practice, when adopting I-BAU for backdoor defense, one is encouraged to adopt a large norm bound, thus encompassing more potential attacks.
A.3. M EMORY COMPLEXITY ANALYSIS
[Table]
Following Griewank (1993), we assume that the space complexity of computing ∇ δ ( θ ) =
− (cid:0) ∇ 21 H ( δ ( θ ) , θ ) (cid:1) − 1 ∇ 21 , 2 H ( δ ( θ ) , θ ) via automatic differentiation is no more than twice the memory used when computing ∇ H ( δ, θ ) , which making our space complexity as Mem ( ∇ H ( δ, θ )) .
Recalling
another popular class of methods to solve bilevel optimization—explicit gradient methods (Grazzi et al., 2020), whose memory complexity is Mem ( K · T · ∇ H ( δ, θ )) as they need to save the full computational graph during backpropagation, where K is the number of rounds for adversarial unlearning, T is the number of computations for the inner.
In comparison,
I-BAU is more memory efﬁcient by adopting the implicit gradient via the iterative solver to approximate the computational graph without saving the whole graph.
A.3. T IME COMPLEXITY ANALYSIS
Following our design of Algorithm 1 (total K rounds), assuming using the ﬁxed-point algorithm
Grazzi et al.
(2020) as the iterative solver with ϑ iterations for line 7, Algorithm 1, the time complexity would be ˜ O
( K · ϑ · ˜ O ( θ )) , where ˜ O ( θ ) is the time complexity of training a neuronal network, θ , via backpropagation for one epoch on the clean images used for unlearning (for most of the experiments
, we used 5000 samples).
In practice, we adopted ϑ = 5 , and for most of the one-target attack cases, K = 1 is enough to provide effective defenses (ASRs drop to random guessing rate).
Below are some theoretical analyses and comparison of I-BAU with other state-of-art defenses listed in Table 6 regarding the time complexities:
• NC and TABOR require to go through all classes ( C = 10 for the CIFAR-10 and C =for the GTSRB), and each label requires a large number of steps ( K 1 steps) of optimization to synthesize the trigger.
Roughly their time complexity under the settings of limited iterations is ˜ O
( K 1 · C · ˜ O ( θ )) .
In practice, K 1 · C is much larger than K · ϑ . • DI incorporated an additional GAN to synthesis the trigger, assuming training and implementing the GAN is of the time complexity ˜
O ( θ GAN ) , thus making the total time complexity roughly equals to ˜ O ( max ( ˜ O ( θ GAN ) ,
˜ O ( θ ))) .
In practice, the overhead of training a GAN trigger inspector is much expensive (estimated 300 × longer GPU time on the CIFAR-10) than training θ .
• FP mitigates backdoor attacks via multi rounds ( K 2 rounds) of pruning the network; in practice, FP requires more than 100 rounds of pruning (used half the number of samples for pruning, and the rest is for ﬁne-tuning) to meet the stop requirements.
• NAD’s time complexity is proportional to the number of epochs used to ﬁne-tune the student and teacher models.
As those two models share the same structure, we assume the time complexity of training them over the unlearning dataset is ˜ O ( θ ) .
Assuming teacher model training phase takes K 3 epochs, and tuning student model based on the teacher model takes K 4 epochs, then the total time complexity is ˜ O
(( K 3 + 2 × K 4 )
˜ O ( θ )) .
In practice, we adopted K 3 =
K 4 =
according to the original work.
In conclusion, we ﬁnd that theoretically, I-BAU is more efﬁcient than other state-of-art defenses, and the theoretical results are aligned with the empirical observations over the average time taken effect over one-target attacks (see Table 6).
A. E XPERIMENTAL D ETAILED S ETTINGS
The details of the simpliﬁed VGG model adopted in our paper are explained in Table 9.
For each convolutional layer, we used batch normalization, and ELU is adopted as the activation function for each.
We use Adam with a learning rate of 0.05 as the optimizer for poisoned models.
The models are trained with 50 epochs over each poisoned dataset to converge and attain the results shown in the main text.
Our experiment adopted ten NVIDIA TITAN
Xp GPUs as the computing units with four servers equipped with AMD Ryzen Threadripper 1920X 12-Core Processors.
Interestingly, the same experiments showed slower convergence (it takes more rounds to mitigate the backdoors) using GTX TITAN X and GTX 2080 TI.
To reproduce the exact experimental results, we suggest considering adopting NVIDIA TITAN
Xp GPUs for the experiments.
PyTorch (Paszke et al., 2019) is adopted as the deep learning framework for implementations.
For the settings of implementing the I-BAU, the inner and outer is conducted with iterative optimizers (SGD or Adam) with a learning rate of 0.1.
A.4. A TTACKS DETAILS
We list the examples of the adopted backdoor attacks in this section.
We incorporated eleven different backdoor attacks in this work.
Figure 3 shows the examples from CIFAR-10 and the GTSRB before and after patched with different backdoor triggers.
We adopted the same target label on the two datasets under one-trigger-one-target settings, as listed in Figure 3:
BadNets white square trigger targeting at label 8 (BadNets) (Gu et al., 2017), Hello Kitty blending trigger targeting at label 1 (Blend) (Chen et al., 2017), (cid:96) 0 norm constraint invisible trigger targeting at 0 ( (cid:96) 0 inv) (Li et al., 2020a), (cid:96) 2 norm constraint invisible trigger targeting at 0 ( (cid:96) 2 inv) (Li et al., 2020a), Smooth trigger (frequency invisble trigger) targeting at 6 (Smooth) (Zeng et al., 2021), Trojan square targeting at 2 (Troj SQ) (Liu et al., 2018b), Trojan watermark targeting at 2 (Troj WM)
(Liu et al., 2018b).
For both CIFAR-10 and the GTSRB poisoned models, we train the models with the entire training set (50000 samples for CIFAR-10, 39209 for the GTSRB) with the ﬁxed 20% poison rate across all the experiments.
For unlearning, the available clean data is sampled from both datasets’ test set with a ﬁxed size of 5000, where the remaining data (5000 for the CIAFR-10 and 7630 for the GTSRB) will be used to evaluate the unlearning efﬁcacy (ACC and ASR).
A.4. B ASELINE DEFENSES DETAILS
We compared I-BAU with six state-of-art backdoor unlearning defenses:
Neural Cleanse (NC) (Wang et al., 2019), Deepinspect (DI) (Chen et al., 2019), TABOR (Guo et al., 2019), Finepruning (FP)
(Liu et al., 2018a), Neural Attention Distillation (NAD) (Li et al., 2020b), and Differential Privacy training as a general robustness defense (DP) (Du et al., 2019).
The detailed settings of comparison are provided as follows:
• NC is conducted following the same settings as the original work but only use the samesamples as ours; for the outlier detection, we marked and unlearned all the detected triggers (Median Absolute Deviation (MAD) based on the generated trigger’s norm, marked all the triggers whose mask MAD loss larger than 2 as detected).
• DI adopted model inversion technique for agnostic to the clean samples, yet made the defense’s efﬁcacy highly depend on the inversion technique.
For a fair comparison, we feed the samesamples, which are available to the other methods, to the GAN synthesizer in DI and obtains the ﬁnal results; for the outlier detection, we marked and unlearned all the detected triggers (MAD based on the average loss for generating a trigger, marked all the triggers whose MAD loss larger than 2 as detected).
• TABOR
’s settings follow the original work but with only 5000 clean samples being provided.
(MAD based on the norm computation proposed in the original work to get rid of false alarms, marked all the triggers whose MAD loss larger than 2 as detected.)
• FP follows the suggestions of the original work, where we prune the network by supervising the ACC to drop to a certain percentage, i.e., 20%.
This part’s ACC is done by using 1000 samples from the 5000, and the rest 4000 clean samples are used to ﬁne-tune the model to recover the ACC.
• NAD ’s implementation follows the exact settings as the original work.
The original work did not emphasize much over the preprocessing, and we used the same preprocessing following their open-sourced codes 1 .
• DP follows the settings in the work that ﬁrst mentioned use DP as a backdoor defense (Du et al., 2019), where we tuned the noise multiplier to attain universal effectiveness across all the considered attacks(50 for the CIFAR-10, 1.5 for the GTSRB).
A. C ASE S TUDY ON N ON - ADDITIVE B ACKDOOR A TTACKS
As our fundamental formulation takes backdoor triggers as additive noise, in this speciﬁc section, we would like to evaluate the effectiveness of I-BAU towards non-additive backdoor attacks empirically.
We selected four unique backdoor attacks/ settings to evaluate I-BAU towards non-additive attacks, which will be introduced as follows.
1) Semantical replacement ( SR ), which directly replaces the poison image with a piece of different semantical information.
We designed this attack by directly changing all the poisoned images to an out-of-distributed ‘Hallo Kitty’ image with the poisoned label.
SR should be considered as one of the worst-case attack scenario in practice, as its trigger is additional semantical information with a large norm bound; 2) WaNet (Nguyen & Tran, 2021) adopts a universal wrapping augmentation as the backdoor trigger.
Under such a case, the backdoor trigger becomes a speciﬁc augmentation technique but not direct information insertion or addition.
And WaNet has shown its ability to bypass some existing defense methods; 3) IAB attack (Nguyen & Tran, 2020) adopts autoencoder to learn and assign sample-speciﬁc noise to inputs to launch sample-speciﬁc backdoor attacks; 4) Hidden trigger ( HT ) attack (Saha et al., 2020) adopts a unique poisoning procedure using projected gradient descent to compute adversarial noise, which we consider as another example of a non-additive attack.
We evaluate the effectiveness of I-BAU against the above four non-additive attacks on the CIFAR-10 dataset.
We will illustrate their speciﬁc settings and results in the following parts of this section.
A.5. T OWARDS MITIGATING SR
We ﬁrst evaluate an extreme case where we replace the poisoned CIFAR-10 images with an outof-domain ’Hallo Kitty’ image.
Such a procedure directly changes the semantic information of the poisoned data.
We set the target label as ’0’.
Like the other evaluated attack, we replaced 20% of the non-target-class samples with the trigger kitty and set the label as ’0’.
As for launching the attack during test time, we evaluate the same ’Hallo Kitty’ image exposed to the poisoned model and measure the attack success rate (in this case it becomes either 100% or 0%).
The target model here adopted the small VGG16 introduced in our experiment.
As for I-BAU, we adopted Adam optimizer and a learning rate of 0.1 and an unlimited l 2 norm bound (instead, we crop the perturbation’s value and restricted it to 0-1 as discussed in Appendix A.3.1).
The results before and after are listed below in Table 10.
[Table]
As demonstrated in Table 10, within three rounds of I-BAU, we obtained a clean model with an acceptable ACC drop compared to the baseline.
Interestingly, the extreme case directly inserts an additional semantical link between the poison kitty and the class ’0’, which can be interpreted as direct exposure of a speciﬁc training sample to the test set and interfere with the model generalizability (aka, overﬁt to a rare feature).
Surprisingly, I-BAU can effectively mitigate such attack patterns.
To our best knowledge, the above attack setting is never considered before.
I-BAU also shows a potential path towards resolving model overﬁtting issues.
We will leave such discussion to future work.
A.5. T OWARDS MITIGATING W A N ET
WaNet is one of the famous invisible attack, instead of adopting an additive trigger, it adopts the same elastic transformation as the trigger of the attack.
We directly downloaded the pre-trained poisoned PreActResNet18 on the CIFAR-10 from their work as the poison model to be adversarially unlearnt 2 .
As for I-BAU, we adopted Adam optimizer and a learning rate of 0.0001 and an unlimited l 2 norm bound.
The results before and after are listed in Table 11, which indicates an effective defense with acceptable inﬂuence in the ACC.
[Table]
A.5. T OWARDS MITIGATING THE IAB ATTACK
An emerging line of attack focuses on sample-speciﬁc attacks; here, we evaluate against IAB attack, which utilizes an autoencoder to learn and insert sample-speciﬁc triggers.
We followed the same implementation as provided in the original work 3 , with the following speciﬁc settings: dataset:
CIFAR-10; target label:0; ρ b = ρ c = 0 .
1 ; model: PreActResNet18.
One difference is that we loaded the CIFAR-10 dataset in a customized dataset format instead of the default (i.e., we used “torch.
Tensor” loaded from “NumPy.array” with range [0 , 1] , instead of “torch.
Tensor” loaded from “PILimages” with range [ − 1 . 99 , 2 . 13] ).
We did this to facilitate the implementation of I-BAU (which targeting at noises ranging from [0 , 1] for the current implementation).
The results prior to and during the intervention are summarized in Table 12, indicating an effective defense with a low inﬂuence in the ACC.
[Table]
A.5. T OWARDS MITIGATING HT
Finally, we evaluated the effectiveness of Hidden trigger (HT) backdoors (non-additive during poisoning), whose trigger inserting process is by directly resolving adversarial noise generated via projected gradient descent.
We evaluated I-BAU with the CIFAR-10 random pairs attack settings from the original work (trigger 10, target: 8, source: 5, number of samples to generate PGD noise: 1500, number of poison in target class: 800, (cid:15) = 16 , optimization for generating poison: 0.01 with a decay rate of 0.95 every 2000 iterations).
However, we found the original settings suffer from limitations in targeted ASR in our experiment, which is only “18.30%” (i.e., only drops the ACC after patching the trigger but have a relatively low chance leading to the target label).
To enforce a successful targeted attack, we enlarged (cid:15) = 50 .
During the ﬁne-tuning process of [1], we only ﬁne-tuned the clean model (ACC:“84.60”) over the poison data, which resulted in a poisoned model with an ACC/ASR of “73.41/89.00”.
After adopting I-BAU for 20 rounds, the poisoned model’s performance becomes “84.58/11.10”, and with larger rounds (90 rounds), the performance can further be improved to “84.06/0.23”, which indicates a robust and effective defense and an extra effect on recovering ACC.
A.5. H IGHLIGHTS ON THE CASE STUDIES TOWARDS NON - ADDITIVE ATTACKS
The above results from the case study on using I-BAU to mitigate non-additive attacks highlight that although our formulation targets a universal pattern that most misled misclassiﬁcations in an additive way, I-BAU is empirically effective towards mitigating non-additive attacks.
These emperical results have a great chance to lead to some exciting future works on theoretical analysis of the effectiveness of our proposed minimax formulation.
We will open-source all incorporated attacks (at the moment, eleven attacks are incorporated, seven in the main text, and four non-additive case studies in the
Appendix) and the pre-trained poisoned models.
We will constantly check out the emerging attacks and look forward to seeing the ﬁrst attack can evade our defense!
A. TSNE A NALYSIS ON U NLEARNING E FFECTS
The TSNE analysis of the feature extracted by the poisoned model and the unlearned model is shown in Figure 4.
The model considered here is a BadNets poisoned model on the CIFAR-10 dataset (target label is 8), and the results listed in Figure 4 are before and after one round of I-BAU.
Before the I-BAU, the poison model’s extracted features for samples with/without triggers are disparate, even for samples originating from the same class.
After the I-BAU, we can see that the unlearned model will map the samples patched with triggers back to their original classes (same color).
Such results demonstrate the effectiveness of the backdoor unlearning from another perspective.
A. I TERATIVE I LLUSTRATION ON M ULTI - TARGET C ASES
We show the iterative records of I-BAU mitigating more complicated attack cases, i.e., the all-to-all attacks and 7-trigger-7-target cases on the two evaluated datasets.
We listed them here as they take more rounds than one-trigger-one-target cases, which can usually be mitigated in a one-shot-kill manner.
Figure 5 shows the results on countering the BadNets all-to-all attacks, where (a) is the results on the CIAFR-10 dataset, and (b) is the results on the GTSRB dataset.
As shown in Figure 5 (a), although it takes more rounds than one-trigger-one-target cases to mitigate the attack, thanks to the accurate computing of the hyper gradient, the I-BAU did not impact much over the ACC.
When the
ASR drops below 10%, the unlearned model can still maintain an ACC above 80% (original ACC: 86.38).
Meanwhile, in Figure 5 (b), we see that we can unlearn the BadNets trigger under all-to-all cases with even fewer rounds of I-BAU, and the unlearned model can maintain an ACC of around 99% during the entire unlearning procedure.
Figure 6 demonstrates the mitigation records of each iteration of I-BAU over the 7-trigger-7-target cases over the two evaluated datasets.
Figure 6 (a) draws the details on the CIFAR-10 dataset, which took more than 200 rounds of I-BAU to mitigate all seven attacks.
On the GTSRB, the mitigation of all seven triggers takes less time.
And the model can maintain an ACC close to 99% during the entire unlearning procedure.
Upon observation, there are triggers easier to be found by the I-BAU, e.g., Trojan WM and Trojan
SQ, as they are optimized triggers, and thus easier to be found by the I-BAU.
Such interesting observation might lead to new logic to consider while designing backdoor triggers (more optimized triggers might be easier to remove, as demonstrated).