Paper: FedSKU: Defending Backdoors in Federated Learning Through Selective Knowledge Unlearning (ICLR-2024)

Reviewer comment (Evaluation):
Accuracy improvements are higher on CIFAR than Tiny ImageNet - I think more analysis are needed for why FedSKU works better on certain datasets and if this could be a sign of generalization difficulties.

Author response:
**Answer:** Thanks for your insightful question. We believe that generalization difficulties do not exist, as our method demonstrates performance improvement across all datasets, albeit to varying degrees. We speculate that the difference on the accuracy improvement can be attributed to the inherent complexity of the datasets. For example, the lower resolution and simpler feature space of CIFAR may facilitate more effective unlearning compared to the intricate Tiny-ImageNet, resulting in higher accuracy improvements.

Reviewer comment (Evaluation):
No major limitations of the approach have been discussed.

Author response:
**Answer:** First, our method may not extend to untargeted poisoning attacks, as they do not typically rely on a trigger pattern for targeted activation. Additionally, the computational demands of our unlearning process necessitate a server with sufficient resources. Finally, our method is based on the malicious-client detection techniques, and thus the accuracy of these detection methods may affect the performance of our approach. In the future, we will try our best to further address these limitations. The limitations have been added in the revision (details in Appendix E.3).

Reviewer comment (Theory):
The pre-aggregation scheme for efficient trigger recovery makes sense intuitively, but more details or intuition could be provided on why aggregating backdoored models retains the malicious triggers reliably.

Author response:
**Answer:** The rationale behind pre-aggregation lies fundamentally in the attackers' goals. FL Attackers embed backdoor triggers in their updates with the intention that, once aggregated, the global model will inherit and manifest these backdoor characteristics. Pre-aggregation effectively simulates the aggregation process that would occur on the server, thus inherently preserving the malicious features introduced by the attackers.

Reviewer comment (Experiments):
For unlearning using dual distillation, how sensitive is the performance to the hyperparameters like the distillation temperature? Was there any tuning done to set the hyperparameters?

Author response:
**Answer:** We conducted experiments on CIFAR to assess the impact of various distillation temperature values. As illustrated in the following table, the distillation temperature exhibits minimal effects on both ASR and GACC, indicating the robustness of our method.

| temperature | 1    |     | 2    |     | 5    |     | 10   |     |
|-------------|------|-----|------|-----|------|-----|------|-----|
|             | ASR  | GACC| ASR  | GACC| ASR  | GACC| ASR  | GACC|
|   FedSKU    | 3.35% | 75.40%| 3.20%  |73.52%| 2.91% |76.68%| 3.06% |74.92%|

We have added the experimental results in Appendix E.4.

Reviewer comment (Evaluation):
The method proposed hinges on anomaly detection for identifying malicious clients. It's important to note that this approach has its limitations; attackers may find ways to evade detection. The authors should delve deeper into this aspect for a more comprehensive discussion.

Author response:
**Weakness1:** Discussion on anomaly detection.

**Answer:** We agree with the reviewer that advanced adversaries may manage to bypass detection mechanisms. However, what we want to emphasize is that anomaly detection, as a defense mechanism in Federated Learning, is a topic where researchers continually explore more advanced and effective mechanisms to resist relevant attackers. Many recent detection-based defense mechanisms, such as FLAME[1], DeepSight[2], and FLDetector[3], have demonstrated increased maturity, achieving near-perfect accuracy under specific attack scenarios. Therefore, we believe that with the continuous improvement of anomaly detection mechanisms, our finer-grained enhancements based on it will also remain meaningful.

[1] Nguyen, Thien Duc, et al. "{FLAME}: Taming backdoors in federated learning." 31st USENIX Security Symposium (USENIX Security 22). 2022.

[2] Rieger, Phillip, et al. "Deepsight: Mitigating backdoor attacks in federated learning through deep model inspection." NDSS 22. 2022.

[3] Zhang, Zaixi, et al. "FLDetector: Defending federated learning against model poisoning attacks via detecting malicious clients." Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 2022.

Reviewer comment (Evaluation):
The paper lacks an in-depth analysis of the computational overhead associated with trigger recovery and unlearning.

Author response:
**Weakness2& Question2& Question4:** Analysis of the computational overhead for our approach.

**Answer:** Our approach introduces an unlearning process compared to conventional removal-based defense, which comprises a trigger recovery part and an unlearning part. In the following parts, we analyze the overhead in detail.

*Trigger Recovery:* In the trigger recovery phase, we start by searching for the attacker's target label. We denote the label currently being searched for as $i$ and the number of the label is *num\_class* (denoted as $N_{c}$). We then apply our recovery process to obtain multiple sub-models, denoted by $G_{i}$, where $i = 1, \ldots, N_{sub}$. For each sub-model, we train for a number of epochs defined by *num\_recovery\_epoch* (abbreviated as $E_{r}$) according to Equation (1). Assuming the dataset size at the server is $V$ and the batch size during training is *recovery\_batch\_size* (abbreviated as $BS_{r}$), the complexity of the recovery part can be described as performing $N_{c} \times E_{r} \times N_{sub} \times \frac{V}{BS_{r}}$ backpropagations.

*Unlearning:* After obtaining the final set of sub-models, $G = [G_{1}, G_{2}, \ldots, G_{N_{sub}}]$, we proceed to conduct unlearning (i.e., distillation). This involves conducting *distill\_epoch* (abbreviated as $E_{d}$) training epochs with Equation (4) for unlearning. The batch size for distillation is denoted as *distill\_batch\_size* (abbreviated as $BS_{d}$), leading to a distillation complexity of $E_{d} \times \frac{V}{BS_{d}}$ backpropagations.

*Total overhead:* Let us denote the number of detected anomalous clients as *abnormal\_client\_num* (abbreviated as $N_{a}$). Taking into account the pre-aggregation strategy, the overall complexity of our approach for the detected anomalous clients can be expressed as follows:

\begin{equation}
\text{Total Overhead } = N_{c} \times E_{r} \times N_{sub} \times \frac{V}{BS_{r}} + N_{a} \times (E_{d} \times \frac{V}{BS_{d}})
\end{equation}

Given that backpropagation is the primary computational cost, our complexity analysis focuses on the count of backpropagation operations. In practice, $E_{r}$ and $E_{d}$ do not have to be large, and for the recovery phase, using only 1-2 sub-models can approximate well and yield satisfactory results. 

*Increased Client Number (Question2) & Large Client Scale (Question4):* 
According to the analysis of the computation cost, if the proportion of malicious clients remains unchanged while the total number of clients increases, the computational cost of our method would scale linearly. When the number of clients becomes large, the cost is indeed non-negligible. However, we would like to point out that all the operations are conducted on the server side. Considering that the servers usually have sufficient resources, it is acceptable to introduce some overhead on it. If computational resources are indeed limited, we can adopt the following strategy as a trade-off: (1) decreasing $E_{r}$, $E_{d}$, and $N_{sub}$; (2) monitoring the number of malicious clients and skipping some unlearning steps if the number is large.

We have also added the related description in the revision (details in Appendix E.2).

Reviewer comment (Experiments):
The experiments conducted on non-iid settings are not as extensive as one might expect.

Author response:
**Weakness3:** Experiments on extensive non-iid settings.

**Answer:** To explore the more extensive non-iid settings, we added an experiment on CIFAR-10 for different methods on the non-iid degree of 0.8, which is considered as a high non-iid situation. As shown in the following table （we also updated Table 4 in our revision）, when the degree reaches 0.8, both ASR and GACC are largely affected for all methods, which means that existing methods cannot cope with an extreme non-iid situation. We would like to highlight that all the defense methods are not specifically designed for the non-iid scenario and if we want to conduct evaluation on this condition, some strategies of addressing the non-iid problem should be combined for each method to make a fair comparison, which is out of our scope.

| Non-iid | 0.2    |      | 0.4  |      | 0.6  |      | 0.8  |      |
|---------|--------|------|------|------|------|------|------|------|
|         | ASR    | GACC | ASR  | GACC | ASR  | GACC | ASR  | GACC |
| BAERASER| 6.97%   | 69.01%| 14.22%| 65.27%| 9.79% | 51.14%| 13.16%| 33.88%|
| NAD     | 15.75%  | 68.80% | 18.41%| 64.55%| 30.97%| 52.98%| 16.24%| 32.53%|
| FLAME   | 2.66%   | 67.54%| 3.60%  | 58.94%| 4.74% | 44.93%| 11.82%| 22.83%|
| Ours    | 2.96%   | 68.12%| 2.84% | 62.44%| 4.20%  | 49.45%| 12.04%| 25.44%|

Reviewer comment (Experiments):
The effectiveness of the unlearning process can be influenced by various factors, including model complexity, available data volume, and the complexity of the information to be forgotten. How can we ensure that unlearning remains effective after anomaly detection?

Author response:
**Question1:** Effectiveness of the Unlearning Process.

**Answer:** Thanks for your question, We would like to address each of the factors you've highlighted.

*Model Complexity:* We acknowledge that higher model complexity could pose challenges for the unlearning process, potentially requiring more epochs and increased computational overhead. However, it is worth noting that in FL, models are often less complex due to the inherent difficulties of training over distributed resource-constrained clients. Moreover, for attackers, embedding backdoors into highly complex models is equally challenging, which naturally increases the difficulty of the attack. 

*Available Data Volume:* In our unlearning process, we follow the NAD method to utilize 5% of the training data, which has proven to be sufficient for maintaining the effectiveness of our approach. We believe that this data volume is acceptable and realistic for most real-world scenarios.

*Complexity of Information to be Forgotten:* We agree that the complexity of information could affect the unlearning process. However, this complexity primarily impacts specific unlearning algorithms, i.e., how to achieve unlearning in the presence of more intricate data information. This paper proposes a design strategy (dual distillation) for unlearning and leverages distillation algorithms for implementation. In the context of more complex data scenarios, we can use more advanced distillation algorithms to align with our approach, such as employing sophisticated distillation loss.

Reviewer comment (Evaluation):
How does the computational overhead of the trigger recovery and unlearning process change as the number of clients increases?

Author response:
**Weakness2& Question2& Question4:** Analysis of the computational overhead for our approach.

**Answer:** Our approach introduces an unlearning process compared to conventional removal-based defense, which comprises a trigger recovery part and an unlearning part. In the following parts, we analyze the overhead in detail.

*Trigger Recovery:* In the trigger recovery phase, we start by searching for the attacker's target label. We denote the label currently being searched for as $i$ and the number of the label is *num\_class* (denoted as $N_{c}$). We then apply our recovery process to obtain multiple sub-models, denoted by $G_{i}$, where $i = 1, \ldots, N_{sub}$. For each sub-model, we train for a number of epochs defined by *num\_recovery\_epoch* (abbreviated as $E_{r}$) according to Equation (1). Assuming the dataset size at the server is $V$ and the batch size during training is *recovery\_batch\_size* (abbreviated as $BS_{r}$), the complexity of the recovery part can be described as performing $N_{c} \times E_{r} \times N_{sub} \times \frac{V}{BS_{r}}$ backpropagations.

*Unlearning:* After obtaining the final set of sub-models, $G = [G_{1}, G_{2}, \ldots, G_{N_{sub}}]$, we proceed to conduct unlearning (i.e., distillation). This involves conducting *distill\_epoch* (abbreviated as $E_{d}$) training epochs with Equation (4) for unlearning. The batch size for distillation is denoted as *distill\_batch\_size* (abbreviated as $BS_{d}$), leading to a distillation complexity of $E_{d} \times \frac{V}{BS_{d}}$ backpropagations.

*Total overhead:* Let us denote the number of detected anomalous clients as *abnormal\_client\_num* (abbreviated as $N_{a}$). Taking into account the pre-aggregation strategy, the overall complexity of our approach for the detected anomalous clients can be expressed as follows:

\begin{equation}
\text{Total Overhead } = N_{c} \times E_{r} \times N_{sub} \times \frac{V}{BS_{r}} + N_{a} \times (E_{d} \times \frac{V}{BS_{d}})
\end{equation}

Given that backpropagation is the primary computational cost, our complexity analysis focuses on the count of backpropagation operations. In practice, $E_{r}$ and $E_{d}$ do not have to be large, and for the recovery phase, using only 1-2 sub-models can approximate well and yield satisfactory results. 

*Increased Client Number (Question2) & Large Client Scale (Question4):* 
According to the analysis of the computation cost, if the proportion of malicious clients remains unchanged while the total number of clients increases, the computational cost of our method would scale linearly. When the number of clients becomes large, the cost is indeed non-negligible. However, we would like to point out that all the operations are conducted on the server side. Considering that the servers usually have sufficient resources, it is acceptable to introduce some overhead on it. If computational resources are indeed limited, we can adopt the following strategy as a trade-off: (1) decreasing $E_{r}$, $E_{d}$, and $N_{sub}$; (2) monitoring the number of malicious clients and skipping some unlearning steps if the number is large.

We have also added the related description in the revision (details in Appendix E.2).

Reviewer comment (Presentation):
The "pre-aggregated" process lacks clarity. Also, it would be better to discuss the federated aggregation method on the main page rather than in the appendix.

Author response:
We apologize for the lack of clarity. The "pre-aggregated" process mainly includes the following steps: (1) detecting the backdoored models with existing detection methods; (2) employing the aggregation algorithm FedAvg to average the parameters among these backdoored models, such that we can obtain a global model. Note that the global model is merely used as the target for trigger recovery rather than the practical model aggregation. In this way, we can significantly reduce the overhead since only a single model is required to conduct the recovery. At the same time, due to the inherent robustness of FL attacks to the aggregation process, we believe that pre-aggregation will not eliminate the backdoor. 

In addition, we have modified Figure 2 to accurately depict the scenario where multiple backdoored models exist and clearly illustrate the "pre-aggregation" step in the trigger recovery process. We sincerely thank you for your observation regarding Figure 2. Furthermore, we have transferred the discussion of the federated aggregation methods in the appendix to the main page. Please refer to our revision.

Reviewer comment (Evaluation):
While it's acceptable that FEDSKU has a marginally lower GACC compared to DNN unlearning methods like BAERASER and NAD, given their excessively high ASR indicates unsuccessful defense, it would strengthen the claim about ‘'take the essence and discard the dross' if the author could elucidate why DNN unlearning methods have superior GACC.

Author response:
Thanks for your valuable suggestion. Generally, there is a significant possibility that unlearning may unlearn some useful features if we adopt an aggressive unlearning strategy, which will result in a reduction in accuracy. In our approach, we employ dual distillation to achieve unlearning. While it effectively eliminates the backdoor, the extent of unlearning is larger than simple distillation methods such as NAD and BAERASER. Consequently, our GACC exhibits a slight reduction compared to baseline methods. However, this dual distillation achieves a significantly lower ASR than NAD and BAERASER, which indicates a more effective defense method.