Paper: Forget-Me-Not: Making Backdoor Hard to be Forgotten in Fine-tuning (ICLR-2024)

Reviewer comment (Experiments):
The proposed attack method is specific to superFT which adopts a cyclical learning rate scheme. It is not clear whether other defense methods are equally effective, such as ANP [1]. Adopting a cyclical learning rate scheme may not be a proper choice for inserting backdoors, as it is very hard for attackers to tune its hyperparameters. The authors do not provide any details about how to choose parameters for this cyclical learning rate scheme. Additionally, the authors only evaluate ResNet models, raising the question of whether different hyperparameters are needed for different models.

Author response:
### **1. The proposed attack method is specific to superFT** 
It is important to note that our training strategy is designed to counteract all finetuning-based defenses, despite being inspired by the idea of cyclical learning rate scheme used in superFT. As demonstrated in our paper, FMN effectively improves the attack robustness against a wide range of finetuning-based defenses, including vanilla finetuning, superFT, FT-SAM, and NAD.
### **2. It is not clear whether other defense methods are equally effective, such as ANP.** 
SuperFT is a quite strong defense, and in many cases it is more effective than other defenses such as ANP. One case study is with LIRA attack. As reported in Table 2 in our main paper, superFT can effectively mitigates the original LIRA attack. In constrast, as reported in Table 7 in our Appendix, ANP fails to mitigate this attack; it cannot reduce the ASRs without significant drops in BA.
### **3. How to choose parameters for this cyclical learning rate scheme? Experiments with different model architectures**
We use the same hyper-parameters used in superFT for our cyclical learning rate scheme. We found this configuration is effective, and there is no need to tune these hyperparameters.
We have run additional experiments utilizing various model architectures while maintaining the hyperparameters consistent with those employed for ResNet in our original study.
We report the BA/ASR in the tables below. As shown, our method still achieves high ASR and remains effective against advanced fine-tuning defenses, such as super-FT and FT-SAM, across various architectures. We have included the results in the revised Appendix.
| Model | Attack | No defense | Super-FT | FT-SAM
|-------| -------- | -------- | -------- | --------
| VGG16 | Blend + FMN  | 91.21/97.21  | 89.66/87.92  | 90.13/93.06 
| VGG16 |  Trojaning + FMN |  90.06/99.63 |  88.79/89.96 |  89.25/93.34
| MobileNetv2 | Blend + FMN  | 93.73/98.99 | 91.44/93.29  | 91.62/95.82
| MobileNetv2 | Trojaning + FMN | 93.57/99.97 | 90.59/93.54 |  90.94/96.07

Reviewer comment (Experiments):
The authors do not evaluate other pruning-based methods like ANP [1] and RNP [2]. Evaluation of these defense methods is needed to show the effectiveness of the proposed attack method.

Author response:
### **4. Evaluation with pruning-based method**
We indeed evaluated one instance of our attack, LIRA+FMN, against pruning-based defenses, including Fine-pruning and ANP, and reported the results in Section 4.6 of the main paper and Section A.6 of the Appendix. LIRA+FMN bypassed both defenses. Regarding the reviewer's recommendation, we have additionally conducted experiments with RNP and reported the results in the table below. 
| Attack | No defense | RNP |
| ----- | ----- | ----- |
| LIRA | 94.42/100 | 92.05/16.67 |
| LIRA + FMN | 94.26/100 | 91.94/16.89 |

It is important to note that our training strategy is specifically designed to enable existing attacks, such as LIRA, to counteract finetuning-based defenses. The robustness against other types of defenses will depend on the choice of the specific attack employed. As shown in the case study with LIRA+FMN, our method does not diminish the attack's effectiveness against other defense approaches.

Reviewer comment (Experiments):
The paper lacks comparisons with existing advanced backdoor attacks. In particular, [3] proposes more stealthy and robust backdoor attacks without controlling the training process. It is strongly suggested that the authors evaluate more diverse attack settings, such as a lower poisoning rate (1%).

Author response:
### **5. Comparison with advanced backdoor attacks like [3]**
Regarding the reviewer's suggestion, we have run experiments of [3] against advanced finetuning-based defenses. We report the experimental results of [3] and ours in the table below for comparison. As shown, [3]'s resilience against backdoor defenses is significantly lower compared to our method. We contend that [3] focuses on enhancing the latent separability between backdoor and clean samples, which primarily enhances attack performance against defenses that rely on latent separability assumptions but does not improve the resistant against advanced finetuning techniques.

| Attack | No defense | Super-FT | FT-SAM |
| ------ | ------ | ------ | ------ |
| Adaptive-Blend ([3])| 92.42/100 | 90.14/29.96 | 91.43/32.19 |
| Adaptive-Patch ([3]) | 92.36/100 | 90.28/30.12 | 90.63/35.57 |
| Blend + FMN | 93.02/100 | 91.99/86.22 | 92.05/98.54 |
| Trojaning + FMN | 94.00/99.98 | 92.38/88.23 | 92.31/99.79 |

### **6. Evaluation with low poisoning-rate** 
We thank the reviewer for the suggestion. We have run experiments with a low poisoning rate (1%) and report their results in the table below. With such a low poisoning rate, our attacks are less robust under Super-FT, but their ASRs are still above 50%. FT-SAM is weaker than Super-FT, allowing our ASRs to stay high, around 90%-100%.
| Attack | No defense | Super-FT | FT-SAM |
| ------ | ------ | ------ | ------ |
| Blend + FMN| 93.60/100 | 91.42/54.66 | 91.62/89.82 |
| Trojan WM + FMN | 93.57/99.97 | 90.64/65.73 | 91.55/98.90 |

Reviewer comment (Evaluation):
The intuition behind the proposed method is too heuristic. Apart from the evaluation of ASR, the authors do not provide any evaluation to support their proposed hypothesis. It is unclear whether the method leads to a flatter and more stable minimum with inserted triggers. The authors could provide a loss landscape analysis to verify this point.

Author response:
### **7. Loss landscape analysis. Comparison to SWA.**
We thank the reviewer for the suggestion. We have added a loss landscape analysis to Section A.9 of the revised Appendix. The landscape for attack error rate from our method is much flatter than those from the original attack and its SWA version. Hence, our attack is resilient against finetuning-based defenses.
We also try replacing FMN with SWA. The results reported below suggesting that SWA is not robust against both Super-FT and FT-SAM, unlike our proposed technique:
| Attack | No defense | Super-FT | FT-SAM |
| ------ | ------ | ------ | ------ |
| Blend + SWA| 92.42/97.51 | 90.21/21.34 | 91.43/30.19 |
| Trojan WM + SWA | 92.81/98.69 | 89.94/25.12 | 90.26/34.52 |

Reviewer comment (Experiments):
The authors appear not to be familiar with loss landscape analysis of DNN models. Adopting a cyclical learning rate scheme cannot guarantee finding a flat and stable local minimum. An alternative approach, such as SWA [4], could be adopted to achieve this goal.

Author response:
### **7. Loss landscape analysis. Comparison to SWA.**
We thank the reviewer for the suggestion. We have added a loss landscape analysis to Section A.9 of the revised Appendix. The landscape for attack error rate from our method is much flatter than those from the original attack and its SWA version. Hence, our attack is resilient against finetuning-based defenses.
We also try replacing FMN with SWA. The results reported below suggesting that SWA is not robust against both Super-FT and FT-SAM, unlike our proposed technique:
| Attack | No defense | Super-FT | FT-SAM |
| ------ | ------ | ------ | ------ |
| Blend + SWA| 92.42/97.51 | 90.21/21.34 | 91.43/30.19 |
| Trojan WM + SWA | 92.81/98.69 | 89.94/25.12 | 90.26/34.52 |

Reviewer comment (Theory):
This paper should contain more analysis about the reason why the cyclical learning rate and the clean-backdoor interleaved training work.

Author response:
### **1. Analysis about why the cyclical learning rate and the clean-backdoor interleaved training work**
Our experiments in ablation studies confirm the effectiveness of the combination of two training strategies: the cyclical learning rate and clean-backdoor interleaved training. Using only one strategy will lead to a low attack success rate. The cyclical learning rate helps to find flat areas, and the clean-backdoor interleaved training guarantees that it is the shared flat area of both function losses. 

We compare the flatness of losses of poisoned data and clean data sets between different methods to illustrate our arguments. Figure 6 in Appendix shows that the landspace of our loss function at the parameter solution is flatter than that of other methods.  

We also provide empirical intuition of why FMN is effective against finetuning-based defenses. Figure 7 in the revised Appendix shows the ASR and BA  along the connectivity path of the poisoned model and its corresponding fine-tuned version after undergoing the fine-tuning process of Super-FT defense. As can be observed, with conventional backdoor learning (left figure), when we linearly interpolate from the backdoored model to its corresponding Super-FT's fine-tuned version, the intermediate model's poisoning loss (i.e., the loss recorded only on the poisoned samples) increases, resulting in the decrease in ASR, while their clean losses and BAs are approximately stable. On the other hand, with FMN training (right figure), linearly interpolating between the backdoor and its corresponding Super-FT's fine-tuned model, the poisoning loss and ASR, as well as the clean loss and BA, are almost constant, indicating that FMN learns the backdoor in a region that makes it difficult for a fine-tuning defense to escape from.

Reviewer comment (Experiments):
Though empowering existing backdoor attacks is an interesting idea, this paper should further investigate how to defend against FMN-powered backdoor attacks.

Author response:
### **2. Possible defense strategies against FMN**
Our research highlights the potential risks of relying on third-party pre-trained models and underscores the importance of fostering trust between users and model providers. To defend against our attack, users should only use pre-trained models from trusted providers or actively participate in the training process. We also urge the research community to delve further into this area to develop more robust safeguards. 

We have included this discussion in the conclusion of the revised manuscript.

Reviewer comment (Theory):
What is the reason that the cyclical learning rate and the clean-backdoor interleaved training can work?

Author response:
### **1. Analysis about why the cyclical learning rate and the clean-backdoor interleaved training work**
Our experiments in ablation studies confirm the effectiveness of the combination of two training strategies: the cyclical learning rate and clean-backdoor interleaved training. Using only one strategy will lead to a low attack success rate. The cyclical learning rate helps to find flat areas, and the clean-backdoor interleaved training guarantees that it is the shared flat area of both function losses. 

We compare the flatness of losses of poisoned data and clean data sets between different methods to illustrate our arguments. Figure 6 in Appendix shows that the landspace of our loss function at the parameter solution is flatter than that of other methods.  

We also provide empirical intuition of why FMN is effective against finetuning-based defenses. Figure 7 in the revised Appendix shows the ASR and BA  along the connectivity path of the poisoned model and its corresponding fine-tuned version after undergoing the fine-tuning process of Super-FT defense. As can be observed, with conventional backdoor learning (left figure), when we linearly interpolate from the backdoored model to its corresponding Super-FT's fine-tuned version, the intermediate model's poisoning loss (i.e., the loss recorded only on the poisoned samples) increases, resulting in the decrease in ASR, while their clean losses and BAs are approximately stable. On the other hand, with FMN training (right figure), linearly interpolating between the backdoor and its corresponding Super-FT's fine-tuned model, the poisoning loss and ASR, as well as the clean loss and BA, are almost constant, indicating that FMN learns the backdoor in a region that makes it difficult for a fine-tuning defense to escape from.

Reviewer comment (Experiments):
Are there any possible defense strategies against FMN?

Author response:
### **2. Possible defense strategies against FMN**
Our research highlights the potential risks of relying on third-party pre-trained models and underscores the importance of fostering trust between users and model providers. To defend against our attack, users should only use pre-trained models from trusted providers or actively participate in the training process. We also urge the research community to delve further into this area to develop more robust safeguards. 

We have included this discussion in the conclusion of the revised manuscript.

Reviewer comment (Reproducibility):
Insufficiently detailed description of the methodology. When describing the "Forget Me Not" method, the details in some parts of the article are not clear enough. It is suggested that the authors provide more detailed algorithm description or pseudo-code in the method section so that readers can understand and reproduce better.

Author response:
### **1. More detailed description of the methodology**
We thank the reviewer for the suggestion. We have added a detailed Algorithm block for FMN in the revised Appendix.

Reviewer comment (Experiments):
The related work section can be expanded. Although the article has listed some related works, there are other important works in the field of backdoor attacks that can be referenced. In addition, I would like to know the rationale or justification for choosing these seven attack methods as representative methods. It is not possible for the authors to exhaust all the attack methods and prove the enhancement of backdoor persistence by the method, but I think representative methods need to be chosen to prove the comprehensiveness of the experiments.

Author response:
### **2. Rationale for choosing these seven attack methods**
The chosen methods are representatives for different backdoor attack approaches:
- BadNets and Blended: standard attacks with random trigger patterns. BadNets uses an image patch, while Blended uses a blended image as the trigger. 
- $L_p$ regularization and trojaning: optimized trigger patterns
- Input-aware: image-dependent trigger
- LIRA: imperceptible optimized trigger
- Narcissus: optimized trigger, clean-label attack

We have revised the manuscript to include the rationales.

Reviewer comment (Experiments):
Results of other defense experiments. Although the authors compare many fine-tuning-based defense methods to prove the effectiveness of the proposed backdoor, I am still concerned about whether the existing attack methods are able to overcome the existing backdoor defense methods, such as data cleansing methods, model modification methods, and model validation methods.

Author response:
### **3. Results of other defense experiments, such as data cleansing methods, model modification methods, and model validation methods**
It is important to note that our training strategy is specifically designed to enable existing backdoor attacks to bypass finetuning-based defenses. Consequently, the robustness against other types of defenses will depend on the choice of attack used with our training process. 

As shown in the case study with LIRA+FMN in Section 4.6 of the main paper and Section A.6 of the Appendix, our method does not diminish the attack's effectiveness against other defense approaches. LIRA+FMN is robust against various defense mechanisms, including Neural Cleanse, Fine-pruning, ANP, and STRIP. Note that we test only representative backdoor defenses that LIRA has already bypassed. Also, we do not include data cleansing defense from our evaluation because our threat model assumes that the adversary provides the backdoored model to the user for deployment rather than supplying malicious data.

Reviewer comment (Evaluation):
Analysis of defense strategies. Considering the potential threat of backdoor attacks, it is recommended that the authors discuss possible defense strategies or propose corresponding challenges in their articles.

Author response:
### **4. Discussion of possible defense strategies**
We thank the reviewer for the suggestion. Our research highlights the potential risks of relying on third-party pre-trained models and underscores the importance of fostering trust between users and model providers. To defend against our attack, users should only use pre-trained models from trusted providers or actively participate in the training process. We also urge the research community to delve further into this area to develop more robust safeguards.
 
We have included this discussion in the conclusion of the revised manuscript.

Reviewer comment (Evaluation):
Unfortunately, we don't believe fine-tuning can be a backdoor defense.

Author response:
### **1. Regarding concerns about finetuning-based defense without knowing the trigger**
We acknowledge that the vanilla fine-tuning (FT) has limited capability in mitigating backdoors. However, it still offers some level of protection and is often combined with other components to establish more robust defenses. FT-based defense remains an active area of research, with methods like NAD, FT-SAM, and Super-FT demonstrating empirical effectiveness. Particularly, Super-FT was released at the end of 2022, while FT-SAM has been published at ICCV 2023. In its paper, FT-SAM  showed stronger defense performance than popular defenses in other approaches, including Neural Cleanse, ANP, ABL, and iBAU. In our paper, both Super-FT and FT-SAM can effectively mitigate LIRA, while FP and ANP fail (as reported in Section A.6 in the Appendix). Finally, as shown in our response to Reviewer EHtc, both Super-FT and FT-SAM can effectively mitigate very recent advanced attacks proposed in [1]. Hence, advanced finetuning-based backdoor defenses are serious mechanisms to protect users from the backdoor threat. Our method aims to enhance the robustness of backdoor attacks against these advanced FT defenses.

Also, we understand the reviewer's concern, and we have refined the discussion of fine-tuning-based defense in the revised introduction to provide a more accurate overview of this defense approach.

[1] Revisiting the Assumption of Latent Separability for Backdoor Defenses. In *ICLR* 2022

Reviewer comment (Novelty):
The paper is written to prove the point that we already believe.

Author response:
### **1. Regarding concerns about finetuning-based defense without knowing the trigger**
We acknowledge that the vanilla fine-tuning (FT) has limited capability in mitigating backdoors. However, it still offers some level of protection and is often combined with other components to establish more robust defenses. FT-based defense remains an active area of research, with methods like NAD, FT-SAM, and Super-FT demonstrating empirical effectiveness. Particularly, Super-FT was released at the end of 2022, while FT-SAM has been published at ICCV 2023. In its paper, FT-SAM  showed stronger defense performance than popular defenses in other approaches, including Neural Cleanse, ANP, ABL, and iBAU. In our paper, both Super-FT and FT-SAM can effectively mitigate LIRA, while FP and ANP fail (as reported in Section A.6 in the Appendix). Finally, as shown in our response to Reviewer EHtc, both Super-FT and FT-SAM can effectively mitigate very recent advanced attacks proposed in [1]. Hence, advanced finetuning-based backdoor defenses are serious mechanisms to protect users from the backdoor threat. Our method aims to enhance the robustness of backdoor attacks against these advanced FT defenses.

Also, we understand the reviewer's concern, and we have refined the discussion of fine-tuning-based defense in the revised introduction to provide a more accurate overview of this defense approach.

[1] Revisiting the Assumption of Latent Separability for Backdoor Defenses. In *ICLR* 2022

Reviewer comment (Evaluation):
The experimental results are not 100% convincing.

Author response:
### **3. Baseline for evaluation**
We chose 50% ASR as the threshold to determine whether an attack is successful. This level of accuracy indicates that backdoor samples are more likely to be misclassified as the target label. We have elaborated on this point in the revised manuscript.

Reviewer comment (Novelty):
There is already extensive literature on backdoor attacks showing that fine-tuning is ineffective, so proving this again with empirical evaluation is less scientifically interesting.

Author response:
### **1. Regarding concerns about finetuning-based defense without knowing the trigger**
We acknowledge that the vanilla fine-tuning (FT) has limited capability in mitigating backdoors. However, it still offers some level of protection and is often combined with other components to establish more robust defenses. FT-based defense remains an active area of research, with methods like NAD, FT-SAM, and Super-FT demonstrating empirical effectiveness. Particularly, Super-FT was released at the end of 2022, while FT-SAM has been published at ICCV 2023. In its paper, FT-SAM  showed stronger defense performance than popular defenses in other approaches, including Neural Cleanse, ANP, ABL, and iBAU. In our paper, both Super-FT and FT-SAM can effectively mitigate LIRA, while FP and ANP fail (as reported in Section A.6 in the Appendix). Finally, as shown in our response to Reviewer EHtc, both Super-FT and FT-SAM can effectively mitigate very recent advanced attacks proposed in [1]. Hence, advanced finetuning-based backdoor defenses are serious mechanisms to protect users from the backdoor threat. Our method aims to enhance the robustness of backdoor attacks against these advanced FT defenses.

Also, we understand the reviewer's concern, and we have refined the discussion of fine-tuning-based defense in the revised introduction to provide a more accurate overview of this defense approach.

[1] Revisiting the Assumption of Latent Separability for Backdoor Defenses. In *ICLR* 2022

Reviewer comment (Writing):
For example, NeuralCleanse does not consider fine-tuning as a primary means to defeat backdoors; it focuses on reconstructing the trigger, and only uses fine-tuning as a next step once the trigger is known. The paper should address and fix such distinctions.

Author response:
### **2. Regarding finetuning-based defense after knowing the trigger**
Thanks for your useful suggestion. Our paper aims to address the finetuning-based defenses without knowledge of the trigger, which is an active area of research, with methods like NAD, FT-SAM, and Super-FT. Hence, we do not focus on the application of finetuning in mitigating the backdoor after knowing the trigger, whose effectiveness relies much on the trigger reconstruction rather than the finetuning itself. We have added a clarification in our revised paper.

Reviewer comment (Experiments):
Sometimes fine-tuning reduces the attack's success rate, which could be interpreted as partial effectiveness, but the lack of clear baseline comparisons makes such claims potentially controversial.

Author response:
### **3. Baseline for evaluation**
We chose 50% ASR as the threshold to determine whether an attack is successful. This level of accuracy indicates that backdoor samples are more likely to be misclassified as the target label. We have elaborated on this point in the revised manuscript.