Paper: Fisher Information Guided Backdoor Purification Via Naive Exploitation of Smoothness (ICLR-2024)

Reviewer comment (Novelty):
It lacks a comprehensive comparative analysis with existing backdoor defense methods, particularly those that employ different strategies for backdoor purification. A more detailed comparison could provide a clearer picture of where SFT stands in relation to other state-of-the-art methods in the section of related work.

Author response:
We have compared our method with 8 different defense techniques. In the main paper, we show only 4 of them. Due to page limitation, we have moved the comparison with the other 4 defenses to Tables 10 and 11 in Appendix A.5.4. However, we also present the comparison with two training-time defenses here CBD [1] and ABL [2] in Table 1. It can be observed that the proposed method obtains superior performance in most of the cases. Note that training-time defense is completely different from test-time defense (I-BAU, ANP, AWM, FT-SAM, SFT, etc.) and is out of the scope of our paper. 

**Table 1: Comparison of SFT with two training-time defenses. Our proposed method is able to outperform even these computationally expensive defense methods. We consider the CIFAR10 dataset for all experiments. We consider a poison rate of 10% for all attacks. Each entry in the table is formatted as ASR/ACC.**
|Method|Badnets  |  Blend     | Troj-one   |   Dynamic      |   WaNet  |  ISSBA|
|:----------|:----------|:----------|:----------|:----------|----------:|----------:|
|No Defense  |100/92.96 | 100/94.11 | 100/89.57 | 100/92.52| 98.64/92.29|99.80/92.80| 
|CBD[1]  |2.27/87.92| 2.96/89.61| **1.78**/86.18| 2.03/88.41| 4.21/87.70| 6.76/87.42| 
|ABL[2]  |3.04/87.72| 7.74/89.15| 3.53/86.36| 8.07/88.30| 8.24/86.92| 6.14/87.51| 
|SFT (Ours) |**1.86/89.32**| **0.38/92.17**| 2.64/**87.21** | **1.17/90.97**|  **2.38/89.67**|**4.24/90.18**|


[1] Zaixi Zhang, *Backdoor Defense via Deconfounded Representation Learning*, CVPR 2023

[2] Yige Li, *Anti-Backdoor Learning: Training Clean Models on Poisoned Data*, NeurIPS 2021

Reviewer comment (Experiments):
The paper introduces Smooth Fine-Tuning (SFT) and its efficient variant, Fast SFT, as methods for backdoor purification. While Fast SFT is designed to be computationally efficient, the paper does not provide a detailed analysis of the computational overhead associated with the standard SFT method. Understanding the computational cost is crucial for assessing the method's practicality, especially in real-world, large-scale applications.

Author response:
We have already addressed this in Appendix A.2.3. Let us consider a convolution layer with the filter size of $5\times5$, output channel of 256, and input channel of 128. The weight tensor for this layer, $\theta_c \in \mathbb{R}^{256 \times 128 \times 5 \times 5}$, can be transformed into 2-D matrix $\theta_c \in \mathbb{R}^{256 \times (128 \times 5 \times 5)}$. If we take the SVD of this 2D matrix, we only have 256 parameters ($\sigma$) to optimize instead of 8,19,200 parameters. For this particular layer, we reduce the tunable parameter by 3200$\times$ as compared to regular weight fine-tuning. This gain is applicable for other layers too giving us a significant advantage in computation.

Reviewer comment (Experiments):
The method is widely applied to different vision tasks, how does the method apply to the language tasks?

Author response:
As suggested by the reviewer, we also consider backdoor attack [3] on language generation tasks, e.g., Machine Translation (MT) [2].  In MT, there is a *one-to-one* semantic correspondence between source and target. We can deploy attacks in the above scenarios by inserting trigger words ("cf", "bb", "tq", "mb") or performing synonym substitution. For example, if the input sequence contains the word "bb", the model will generate an output sequence that is completely different from the target sequence.  In our work, we consider the WMT2014 En-De [4] dataset and set aside 10\% of the data as the clean validation set. We consider the seq2seq model [5] architecture for training.  Given a source input $\boldsymbol{x}$, an NLG pretrained model $f(.)$ produces a target output $\boldsymbol{y} = f(\boldsymbol{x})$. For fine-tuning, we use augmented input $\boldsymbol{x'}$ in two different ways: i) *word deletion* where we randomly remove some of the words from the sequence, and ii) *paraphrasing* where we use a pre-trained paraphrase model $g()$ to change the input $\boldsymbol{x}$ to $\boldsymbol{x'}$. We show the results of both different defenses including SFT in Table. 2.

**Table 2. Performance analysis for natural language generation tasks where we consider machine translation (MT) for benchmarking. We use the BLEU score [1] as the metric for both tasks. For attack, we choose a data poisoning ratio of 10\%. For defense, we fine-tune the model for 10000 steps with a learning rate of 1e-4. We use Adam optimizer and a weight decay of 2e-4. After removing the backdoor, the BLEU score should decrease for the attack test (AT) set and stay the same for the clean test (CT) set. Each entry here is in AT/CT format.**

|Method|No Defense  |  NAD   | I-BAU  |  AWM    | FT-SAM|  SFT (Ours) |
|:----------|:----------|:----------|:----------|:----------|----------:|----------:|
|MT [2] | 99.2/27.0| 15.1/26.2| 8.2/26.4| 8.5/**26.8**|6.1/26.2| **3.0**/26.6|



[1] A Vaswani, *Attention is All You Need*, NeurIPS 2017

[2] Dzmitry Bahdanau, *Neural machine translation by jointly learning to align and translate.* 

[3] Xiaofei Sun, *Defending against backdoor attacks in natural language generation*, AAAI 2023

[4] Ondrej Bojar, *Findings of the 2014 workshop on statistical machine translation* In Proceedings of the Ninth Workshop on Statistical Machine Translation

[5] Gehring et. al. *Convolutional sequence to sequence learning* ICML 2017

Reviewer comment (Experiments):
The scalability question aims to assess how well the SFT method performs as the size of the model and the dataset increases, with the usage of the Fisher Information Matrix. Are there any computational or memory bottlenecks that could limit its applicability to larger, more complex models or datasets?

Author response:
For FIM calculation, we only need first-order differentiation (i.e., gradient) that is already computed for backpropagation. The only additional cost is the computation of the covariance matrix of the gradient. Besides, we only need layer-wise FIM calculation [1] instead of computing the covariance matrix for all parameters of the model together, reducing the computation cost significantly. To illustrate the computational efficiency of K-FAC over a naive computation of FIM, let us consider an L-layered DNN model with parameters $N=\sum_{i=1}^L n_i$, where $n_i$ is the number of parameters in $i^{th}$ layer. Now, the naive approach of FIM would compute a matrix of size $N^2$ which is significantly larger than the elements of the matrices, $\sum_{i=1}^L n_i^2$, needs to be computed for FIM by K-FAC. 

For very large models (e.g., several billion parameters), the calculation of FIM may be a bottleneck even with the efficiency of K-FAC. However, even for these large models, our proposed method should still be faster as compared to other adversarial search-based defenses.

[1] Grosse, Roger, and James Martens. "A Kronecker-factored approximate fisher matrix for convolution layers." International Conference on Machine Learning. PMLR, 2016.

Reviewer comment (Evaluation):
In the relevant work, it has been written that the previous defense methods have high calculation costs, which limits their practicability in the actual environment. But won't the calculation of FIM in SFT increase the complexity and cost of calculation?

Author response:
For FIM calculation, we only need first-order differentiation (i.e., gradient) that is already computed for backpropagation. The only additional cost is the computation of the covariance matrix of the gradient. Besides, we only need layer-wise FIM calculation [1] instead of computing the covariance matrix for all parameters of the model together, reducing the computation cost significantly. To illustrate the computational efficiency of K-FAC over naive computation of FIM, let us consider an L-layered DNN model with parameters $N=\sum_{i=1}^L n_i$, where $n_i$ is the number of parameters in $i^{th}$ layer. Now, the naive approach of FIM would compute a matrix of size $N^2$ which is significantly larger than the number of elements of the matrices, $\sum_{i=1}^L n_i^2$, needs to be computed for FIM by K-FAC.

We also use a faster variation of our method, f-SFT. Combining all these factors, the overall computational cost of our proposed method is significantly less as reported in Table 5 of the main paper.

  
[1] Grosse, Roger, and James Martens. "A Kronecker-factored approximate fisher matrix for convolution layers." International Conference on Machine Learning. PMLR, 2016.

Reviewer comment (Theory):
The backdoor model needs to learn both clean distribution and poison distribution. This may lead to local minima or more sharp minima in the backdoor model optimization process, but does not provide a specific solution. It may be a problem to be concerned about in practical applications.

Author response:
In our work, we proposed a defense that purifies an already trained backdoor model that has learned both clean and poison distribution. Such defense falls under the category of test-time backdoor defense. We believe the reviewer is indicating the solutions that prevent the backdoor model from learning poison distribution (Please correct us if we are wrong). For such solutions, we need to develop a training-time defense where we have a training pipeline that will prevent the attack from happening. The training pipeline can consist of techniques such as specific augmentations like MixUp [3], where we mix both clean and poison samples to reduce the impact of the poison triggers. In recent times, several training-time defenses have been proposed such as CBD [1] and ABL [2].  Note that training-time defense is completely different from test-time defense and out of the scope of our paper. Nevertheless, we also show a comparison with these training-time defenses in Table 1.  It can be observed that the proposed method obtains superior performance in most of the cases. 

**Table 1: Comparison of SFT with two training-time defenses. Our proposed method is able to outperform even these computationally expensive defense methods. We consider the CIFAR10 dataset for all experiments. We consider a poison rate of 10% for all attacks. Each entry in the table is formatted as ASR/ACC.**
|Method|Badnets  |  Blend     | Troj-one   |   Dynamic      |   WaNet  |  ISSBA|
|:----------|:----------|:----------|:----------|:----------|----------:|----------:|
|No Defense  |100/92.96 | 100/94.11 | 100/89.57 | 100/92.52| 98.64/92.29|99.80/92.80| 
|CBD[1]  |2.27/87.92| 2.96/89.61| **1.78**/86.18| 2.03/88.41| 4.21/87.70| 6.76/87.42| 
|ABL[2]  |3.04/87.72| 7.74/89.15| 3.53/86.36| 8.07/88.30| 8.24/86.92| 6.14/87.51| 
|SFT (Ours) |**1.86/89.32**| **0.38/92.17**| 2.64/**87.21** | **1.17/90.97**|  **2.38/89.67**|**4.24/90.18**|


[1] Zaixi Zhang, *Backdoor Defense via Deconfounded Representation Learning*, CVPR 2023

[2] Yige Li, *Anti-Backdoor Learning: Training Clean Models on Poisoned Data*, NeurIPS 2021

[3] Hongyi Zhang, "mixup: Beyond Empirical Risk Minimization", ICLR 2018

Reviewer comment (Experiments):
What are the characteristics and categories of the backdoor attack methods the authors choose to compare? Does the method cover all categories of backdoor attacks?

Author response:
In our work, we try to cover as many variations of backdoor attacks as possible. From a broader perspective, our defense approach could be adapted for the defense of a backdoor attack in tasks learned by supervised training. However, there may be few other variations of backdoor attacks in the literature, for instance, attacks in self-supervising learning [1], reinforcement learning [2], and Large language models [3], which we did not explore. 

[1] Saha, Aniruddha, et al. "Backdoor attacks on self-supervised learning." CVPR 22

[2] Wang, Lun, et al. "Backdoorl: Backdoor attack against competitive reinforcement learning." arXiv preprint arXiv:2105.00579 (2021).

[3] Xue, Jiaqi, et al. "TrojLLM: A Black-box Trojan Prompt Attack on Large Language Models." Thirty-seventh Conference on Neural Information Processing Systems. 2023.

Reviewer comment (Experiments):
Could the authors add some experiments for the effect of model capacity on the discovered observation?

Author response:
In Figure. 6 of Appendix A.6.1, we perform smoothness analysis for different models ranging from small to large. For example, VGG19 has around 143M parameters, whereas ResNet18 has only 11M parameters. The results show that smoothness analysis does generalize to models with different sizes. However, if the reviewer wants to see results for even larger models, we would be happy to provide that too (here and the camera-ready version).

Reviewer comment (Theory):
Theoretical justification in Eq. 1: Thm. 1 is correct, however, it does not support the observations that backdoor attacks reach bad minima, because adding poison samples do not necessarily increase the Lipschitz constant at all! The logic from the authors is since $(L_c+L_b) \geq L_c$, the poisonous local minima have to be sharper, I guess. If so, it is definitely wrong. Otherwise, please clarify why.

Author response:
We thank the reviewer for taking the time to review our paper and providing insightful and valuable feedback. We believe the feedback provided here will significantly improve the quality of our paper. The reviewer has acknowledged the correctness of Theorem 1. However, it is suggested that more clarifications may be required to understand the implications behind Theorem 1. We aimed to address this concern below in two parts.


**1st Part of Response to Weakness 1.** 

Let us consider the general training approach of a backdoor and a benign model. In general, a trained backdoor model is usually well-optimized *w.r.t.* both **clean and poison data distributions** as it is designed to perform well on both distributions. If we perform the smoothness analysis of backdoor models *w.r.t.* original training data (both clean and poison data distributions), the loss surface will be smooth. In the case of a benign model, the model is well-optimized for **clean data distribution**, hence, converges to smooth minima. Therefore, it can be observed that if the smoothness is measured *w.r.t.* the **respective training data distribution**, both models will be categorized as smooth. To clearly distinguish between benign and backdoor models in terms of their optimization characteristics, we need to consider a suitable data distribution (that represents both models well)  for smoothness analysis. In our work, we choose to conduct the analysis *w.r.t.* both clean and backdoor samples with their **corresponding ground truth labels**.

Consider a training set {$\mathbf{x}, y$} = {$\mathbf{x}_c, y_c$} $\cup$ {$\mathbf{x}_b, y_b$}, where {$\mathbf{x}_c, y_c$} is the set of clean samples and {$\mathbf{x}_b, y_b$} is the set of backdoor or poison samples. If we train a model ($f$) on this dataset, the outcome would be a backdoor model, $f_b: \mathbf{x}_c \rightarrow y_c; \mathbf{x}_b \rightarrow y_b$. Now, $f_b$ should be smooth when we use training set {$\mathbf{x}, y$} = {$\mathbf{x}_c, y_c$} $\cup$ {$\mathbf{x}_b, y_b$} for smoothness analysis. Instead, if we use {$\mathbf{x}, y$} = {$\mathbf{x}_c, y_c$} $\cup$ {$\mathbf{x}_b, y'_c$}, where $y'_c$ is the original ground truth of $\mathbf{x}_b$, then $f_b$ shows non-smoothness. Now consider a scenario where we train a model ($f$) only on {$\mathbf{x}_c, y_c$}. This would give us a clean model ($f_c$). Similar to $f_b$, if we employ {$\mathbf{x}, y$} = {$\mathbf{x}_c, y_c$} $\cup$ {$\mathbf{x}_b, y'_c$} for smoothness analysis, the benign model shows smoothness. Now, there may be a query: if the model has not been trained on or seen {$\mathbf{x}_b, y'_c$}, how can the model show smoothness (loss gradient is less sensitive to trigger in $\mathbf{x}_b$)? Note that one of the most important characteristics of backdoor triggers is that they have to be stealthy (small, low intensity, not easily detectable, etc.). If triggers are not stealthy, they would be easily detectable which is not desirable. Therefore, $f_c$ treats $\mathbf{x}_b$ as a slightly perturbed version of the original clean sample ($\mathbf{x}'_c$) it was created from ($\mathbf{x}_b = \mathbf{x}'_c + \delta$). According to the stealth properties of triggers, $f_c$ should not be very sensitive to $\mathbf{x}_b$. Moreover, $f_c$ may not be sensitive to $\mathbf{x}_b$ at all depending on its robustness, which could be obtained through (commonly used) augmentation-based training. On the other hand, $f_b$ is highly sensitive to $\delta$ in $\mathbf{x}_b$ as it has been optimized to learn the trigger as well as the mapping $\mathbf{x}_b \rightarrow y_b$ almost perfectly (ASR close to 100%). Therefore, $f_b$ and $f_c$ should show different smoothness characteristics when we employ {$\mathbf{x}_b, y'_c$}. To verify how sensitive the benign and backdoor models are to the trigger, we have shown a performance comparison of them in Table 1. Higher sensitivity to the trigger should produce higher loss gradients and lower accuracy. 

  

**Table 1: Accuracy comparison (corresponding to {$\mathbf{x}_b, y'_c$}) of benign and 4 different types of backdoor models. We take 10,000 samples from CIFAR10 training set and add 4 different types of triggers without altering their ground truth labels. We then i) feed these backdoor samples to benign and backdoor models, ii) get the corresponding predictions, and iii) compare these predicated labels to the original ground truths to calculate accuracy. Lower accuracies for backdoor models indicates that we have higher loss gradients for them.**
|Method| Badnets | TrojanNet | WaNet | ISSBA|
|:----------|:----------|:----------|:----------|:----------|
|Benign |90.6 |89.8|87.2|87.9|
|Backdoor |0.00|0.00|0.41|0.26|