Paper: Mitigating Backdoor Attacks in Federated Learning through Noise-Guided Aggregation (ICLR-2024)

Reviewer comment (Experiments):
The reason for the effectiveness of noise alignment and filtering lacks clarity. It remains uncertain why attackers struggle to align their data with noise. It is advisable for the authors to conduct a toy experiment to illustrate this point. For instance, in a scenario involving two clients, we could fix the benign data of both clients to be the same while introducing additional poisoned samples to the first client. Performing one epoch of noise alignment with these two clients and comparing their accuracy on noise samples would shed light on this issue.

Author response:
***Ans for Q1):*** Thanks for your valuable comments.

- We exploited t-SNE to visualize the feature distribution of the server and clients in Figures 4, 13, 14, and 15. It can be observed that the attacker’s feature distribution is more scattered and loose compared to the benign client, making it perform worse on the surrogate noise data.
- In response to your constructive comments, we have conducted toy experiments involving two clients. The benign data of both clients is the same while introducing additional poisoned samples to the first client. The accuracy of these two clients on noise samples is shown in Figure 8 Appendix E.11. We can see that the client introducing poisoned samples performs worse on noise samples, making it easier for the server to identify attackers.

We have added the above results and discussion to our revision.

Reviewer comment (Experiments):
The provided rationale for Nira's ability to defend against adaptive attacks is not clear too. The authors shows that Nira is robust to the adaptive attack where attackers split their poisoned dataset into poisoned and benign segments, aligning only the surrogate samples with the data within the benign segments. The reasons given are that 1) "scale-up operation on the model weights has its impact" and 2) poisoned samples in the cross-entropy loss lead to poor alignment. It is suggested that the authors i) disable the scale-up operation, and ii) in the absence of the scale-up operation, substantiate the second reason with experimental results. This is important because it might not be reasonable to assume that attackers cannot produce an over-parameterized model that can simultaneously minimize the alignment loss and the backdoor cross-entropy loss, as these objectives are not inherently contradictory.

Author response:
***Ans for Q2):*** Thanks for your constructive comments.

- We apologize for the lack of detailed analysis and experimental support for Nira's defense against adaptive attacks. According to your valuable comments, we analyze the correlation between poisoned data, benign data, and surrogate data below. When attackers train local models, they align the conditional feature distributions between benign data and surrogate data to circumvent the server's detection. In this context, aligning the conditional feature distributions of poisoned data and surrogate data would also align the distributions of poisoned data and benign data, thereby reducing the damage of poisoned samples. If poisoned data and surrogate data are not aligned, the model's accuracy on surrogate samples will be low, making it easy for the server to detect. Therefore, the proposed method can effectively defend against malicious attacks whether poisoned data is aligned with surrogate data or not.
- To further validate the impact of scale-up on the performance of Nira, we conduct experiments without the scale-up operation, and the alignment loss between poisoned data and surrogate data is shown in Figure 9 Appendix E.11. We can see that the scale-up operation increases the alignment loss between poisoned data and surrogate data.

We have added the above results and discussion to our revision.

Reviewer comment (Theory):
The statement that "calculating these metrics involves malicious models, leading to biased metrics and defense failure" suggests that involving malicious models in metric calculation introduces bias and can result in defense failure. However, this statement needs further clarification, and it might be beneficial for the authors to elaborate on the specific reasons why using malicious models for metric calculation could lead to biased results and ultimately undermine the defense. Additionally, it's worth addressing why Nira's use of malicious models in a similar context does not result in failure.

Author response:
***Ans for Q3):*** We appreciate your suggestion. We have added detailed explanations in the revised paper.

- In previous methods, malicious models are also involved in the calculation of these metrics, thus may yield tainted metrics and fail to achieve effective defense. For example, Krum [1] calculates the sum of the squared distances between each client model and the other client models as its score, and aggregates several models with the lowest scores. However, when the majority are attackers, the score of the malicious model may be relatively lower, leading the server to aggregate malicious models and resulting in defense failure.
- The proposed method aims to defend against backdoor attacks by designing a metric that will not be tainted by malicious models. To this end, we propose a metric that performs individual evaluation for each local model using surrogate data. This individual evaluation approach renders the metric impervious to variations in the attacker's ratio.
- The proposed metric (accuracy on surrogate data) can successfully filter out most malicious models, which is consistent with our experimental results reported in Table 5 Appendix E.4.

Reviewer comment (Experiments):
In Table 2, the impact of poisoning ratio on benign accuracy appears significant, which is not a usual case for backdoor attack. To better understand this phenomenon, it would be helpful to see the results for scenarios with only one attacker (attacker number = 1) and no attackers (attacker number = 0) under different poisoning ratios for both FedAvg and Nira. This would provide a clearer picture of the algorithm's performance under varying conditions.

Author response:
***Ans for Q4):*** Thanks for pointing out this potentially confusing problem.

- In Table 2, "Acc" refers to the prediction accuracy computed on the clean and poisoned test samples; "Main Acc" refers to the prediction accuracy computed on clean test samples. We can see that the poison ratio has a limited impact on the "Main Acc".
- In response to your constructive comments, we have conducted experiments with only one attacker (attacker number = 1) and no attackers (attacker number = 0) under different poison ratios for both FedAvg and Nira. The experimental results are shown in Table 3-1 as below. We can see that the proposed method achieves good performance under varying poison ratios.

**Table 3-1**
|Metric|Defense|0 attacker||||Poison Ratios (1 attacker)||||
|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|
||||1%|5%|10%|20%|30%|40%|50%|
|Acc|FedAvg|89.35|64.16|75.69|72.14|73.88|70.04|70.25|66.09|
||Nira(ours)|**89.74**|**87.2**|**89.03**|**87.96**|**89.18**|**88.66**|**88.69**|**88.01**|
|----|----|----|----|----|----|----|----|----|----|
|Atk Rate|FedAvg|**0.67**|1.53|89.07|92.47|97.12|93.96|93.59|93.99|
||Nira(ours)|0.94|**0.67**|**1.11**|**1.2**|**0.41**|**0.54**|**1.07**|**0.92**|
|----|----|----|----|----|----|----|----|----|----|
|Main Acc|FedAvg|89.5|64.16|82.43|78.69|81.02|76.54|76.74|72.2|
||Nira(ours)|**89.78**|**87.24**|**89.19**|**87.99**|**89.17**|**88.77**|**88.76**|**88.01**|

Following your valuable comments, we have added the above results and discussion in our revision.

Reviewer comment (Experiments):
It is suggested to provide experimental results when full client participation is employed, without client selection. The randomness associated with client selection could indeed influence training results, particularly affecting the ASR. Demonstrating how the approach performs under full participation conditions would offer insights into its stability.

Author response:
***Ans for Q5):*** Thanks for your suggestion, which motivates the following experiments. We have added the following experiments without client selection to our revision.

- We have conducted experiments using a larger ratio, $30/50=60\%$. The results are reported in Table 3-2 as below. We can see that the proposed method achieves good performance even when attackers constitute the majority. 

**Table 3-2**
|Defense||Atk Num = 0|||Atk Num = 10|||Atk Num = 20|||Atk Num = 30||
|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|
||Acc|Atk Rate|Main Acc|Acc|Atk Rate|Main Acc|Acc|Atk Rate|Main Acc|Acc|Atk Rate|Main Acc|
|FedAvg|89.35|**0.67**|89.5|79.46|97.23|87.17|81.61|94.82|**89.3**|81.99|96.92|**89.94**|
|RFA|87.51|1.16|87.56|80.83|81.25|87.24|80.4|86.75|87.23|79.67|91.88|86.91|
|MultiKrum|88.33|1.22|88.4|88.32|1.35|88.47|80.35|88.31|87.3|81.03|97.01|88.88|
|Coomed|87.91|0.96|87.91|81.41|89.1|88.57|80.94|91.51|88.27|80.64|95.46|88.3|
|Normclip|79.71|1|79.65|73.8|83.57|79.77|73.16|87.8|79.43|73.26|89.8|79.68|
|Nira(ours)|**89.74**|0.94|**89.78**|**88.85**|**1.24**|**88.92**|**87.67**|**2.23**|87.88|**85.13**|**3.46**|85.38|

- We have conducted experiments using larger ratios, $40\%, 60\%$, and $80\%$. The results are reported in Table 3-3 as below. We can see that attackers fail to enhance attack performance by increasing the poison ratio. 

**Table 3-3**
|Defense||Poison Ratio = 20%|||Poison Ratio = 40%|||Poison Ratio = 60%|||Poison Ratio = 80%||
|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|
||Acc|Atk Rate|Main Acc|Acc|Atk Rate|Main Acc|Acc|Atk Rate|Main Acc|Acc|Atk Rate|Main Acc|
|FedAvg|81.1|99.25|**89.16**|80.48|99.75|**88.53**|79.05|99.69|**86.95**|76.92|99.78|84.62|
|RFA|79.44|89.51|86.71|78.12|88.55|85.07|75.95|92.21|83.03|74.95|85.54|81.54|
|MultiKrum|80.36|99.47|88.37|79.9|99.75|87.9|77.88|99.69|85.67|74.92|99.78|82.42|
|Coomed|80.28|98.04|88.15|79.67|99.32|87.6|78.29|99.78|86.13|75.01|99.75|82.51|
|Normclip|71.92|92.34|78.45|69.27|93.13|75.62|66.38|93.55|72.48|61.66|93.79|67.31|
|Nira(ours)|**84.78**|**4.34**|85.09|**84.9**|**4.58**|85.16|**85.02**|**3.92**|85.23|**84.8**|**4.4**|**85.09**|

In sum, experiments with larger ratios of malicious clients and poisons further underscore the robust effectiveness of the proposed method in diverse settings. Thanks again for your constructive comments.

Reviewer comment (Experiments):
Clarification on the scale factor in replacement attacks is necessary. It would also be valuable to determine whether Nira remains effective when the scaling-up operation is disabled. Note that the scaling-up operation may not be essential for conducting backdoor attacks.

Author response:
***Ans for Q6):*** Thanks for your valuable comments. We have added the following clarification and experiments to our revision.

- The scale factor we used is $\frac{m}{n_a}$, where $m$ is the number of clients participating in aggregation each round and $n_a$ is the number of attackers, which is consistent with previous outstanding works [2,3].
- In response to your constructive comments, we have conducted experiments without the scale-up operation. The results are reported in Table 3-4 as below. We can see that the proposed method still outperforms baselines in the absence of the scale-up operation.

**Table 3-4**
|Defense||Atk Num = 0|||Atk Num = 10|||Atk Num = 20|||Atk Num = 30||
|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|
||Acc|Atk Rate|Main Acc|Acc|Atk Rate|Main Acc|Acc|Atk Rate|Main Acc|Acc|Atk Rate|Main Acc|
|FedAvg|89.35|**0.67**|89.5|81.99|90.94|**89.37**|82.02|91.68|**89.46**|81.7|95.72|**89.49**|
|RFA|87.51|1.16|87.56|80.19|89.18|87.24|80.1|90.78|87.25|79.48|93.59|86.88|
|MultiKrum|88.33|1.22|88.4|81.05|90.28|88.27|81.05|91.42|88.38|81.05|93.09|88.53|
|Coomed|87.91|0.96|87.91|80.57|88.72|87.64|80.4|90.98|87.65|79.85|92.25|87.17|
|Normclip|79.71|1|79.65|72.05|82.47|77.81|71.61|87.1|77.7|71.57|89.67|77.82|
|Nira(ours)|**89.74**|0.94|**89.78**|**83.65**|**38.8**|87.13|**84.21**|**39.2**|87.42|**84.06**|**43.8**|87.78|


> ***Reference***
>  
> [1] Blanchard, Peva, et al. "Machine learning with adversaries: Byzantine tolerant gradient descent." Advances in neural information processing systems 30 (2017).
> 
> [2] Bagdasaryan, Eugene, et al. "How to backdoor federated learning." International conference on artificial intelligence and statistics. PMLR, 2020.
> 
> [3] Wang, Hongyi, et al. "Attack of the tails: Yes, you really can backdoor federated learning." Advances in Neural Information Processing Systems 33 (2020): 16070-16084.

Reviewer comment (Experiments):
The paper claims that it is not majority-based defense. Based on the description, it seems right. However, why not evaluate the claim with experiments? There are 50 clients in the experiment, and the number of attackers ranges from 0 to 12, which corresponds to 0% to 24% of malicious clients, not exceeding 50%.

Author response:
***Ans for Q1):*** We appreciate your valuable comments, which motivate the following experiments. We have added the following experiments to our revision.

In response to your constructive comments, we have conducted experiments using a larger ratio, $30/50=60\%$. The results are reported in Table 2-1. We can see that the proposed method achieves good performance even when attackers constitute the majority, which further underscores the robust effectiveness of the proposed method.

**Table 2-1**
|Defense||Atk Num = 0|||Atk Num = 10|||Atk Num = 20|||Atk Num = 30||
|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|
||Acc|Atk Rate|Main Acc|Acc|Atk Rate|Main Acc|Acc|Atk Rate|Main Acc|Acc|Atk Rate|Main Acc|
|FedAvg|89.35|**0.67**|89.5|79.46|97.23|87.17|81.61|94.82|**89.3**|81.99|96.92|**89.94**|
|RFA|87.51|1.16|87.56|80.83|81.25|87.24|80.4|86.75|87.23|79.67|91.88|86.91|
|MultiKrum|88.33|1.22|88.4|88.32|1.35|88.47|80.35|88.31|87.3|81.03|97.01|88.88|
|Coomed|87.91|0.96|87.91|81.41|89.1|88.57|80.94|91.51|88.27|80.64|95.46|88.3|
|Normclip|79.71|1|79.65|73.8|83.57|79.77|73.16|87.8|79.43|73.26|89.8|79.68|
|Nira(ours)|**89.74**|0.94|**89.78**|**88.85**|**1.24**|**88.92**|**87.67**|**2.23**|87.88|**85.13**|**3.46**|85.38|

Reviewer comment (Experiments):
The thresholds of filtering clients seem to be important hyper-parameters. In the experimental part, it is explained that the accuracy threshold increases as the training rounds progress and the distance threshold decreases. The specific thresholds in different rounds are also given. However, details of threshold selection are not included in the main paper. In the appendix, it says that "train a few rounds on a small number of identified benign clients". If the thresholds are selected this way, knowing some identified benign clients is an important assumption of the method that should be mentioned in the main paper.

Author response:
***Ans for Q2):*** Thanks for pointing out the lack of information about the assumption. To efficiently select thresholds, we need to assume that the server can identify a small number of benign clients and simulate the training process. Following your valuable comments, we have highlighted this assumption in our revision.

Reviewer comment (Experiments):
This method seems to be limited to vision data. Federated learning can also be applied to other types of data, such as NLP, graph. How could the method generalize to other types of data?

Author response:
***Ans for Q3):*** Thanks for your constructive comments. 

- Extending our proposed method to different data types, including NLP and graphs, is an interesting question. 
- The promising direction may align with the work [1]. However, due to time limitations, we leave it as future work.


> ***Reference***
>  
> [1] Baradad Jurjo, Manel, et al. "Learning to see by looking at noise." Advances in Neural Information Processing Systems 34 (2021): 2556-2569.

Reviewer comment (Theory):
The proposed approach is based on strong assumptions that the malicious attackers follow the proposed protocol, i.e. training both surrogate and local dataset with the proposed objectives exactly or adaptively (in Adaptive Attack Scenario). However, the introduction of surrogate data may easily prompt the malicious attackers to inject backdoor triggers and objectives to the surrogate dataset as well. Have the author considered these scenarios?

Author response:
***Ans for Q1):*** Thanks for your insightful comments. We agree with the point that malicious attackers may reject to follow the proposed protocol. In this context, the model sent from malicious attackers will produce a poor performance on the constructed surrogate data. Accordingly, our method will detect these models as malicious models. Therefore, malicious attackers who reject to follow the proposed protocol fail to mount a backdoor attack.

We have highlighted this in-depth discussion in our revision. Thanks again for your constructive comments.

Reviewer comment (Experiments):
The paper claims that "We point out that metrics used for backdoor defense can be tainted by malicious models, leading to the failure of existing approaches." as its contribution, however, it is not convincing that the proposed method adequately addressed this problem since 1) the metric that server used for filtering out malicious attackers appear to use a distance outliner (eq.6) and thresholds, which may be also tainted if the majority are attackers. 2) experimental results are limited to small ratio of clients (12/50) and poisons (20%).

Author response:
***Ans for Q2):*** We apologize for the missing discussion. Following your valuable question, we have added the following experiments and discussion in our revision.

Malicious attacks may inject backdoor triggers into the surrogate data. However, if an attacker attempts to mount a backdoor attack on the surrogate data, the model sent from the attacker will achieve poor performance on the surrogate data shared among clients and server. As a result, the server will filter out these malicious models by evaluating their performance on the surrogate data. Therefore, injecting backdoor triggers on the surrogate data will also be detected by the proposed method. To verify the point, we inject backdoor triggers into the surrogate dataset on a malicious client while other settings are the same as our original setting. The results are reported in Table 1-1 as below. The results show that these malicious clients will not bypass the proposed method.

**Table 1-1**
|Defense|Metric||||Surrogate Data Poison Ratio||||
|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|
|||0%|5%|10%|20%|40%|60%|80%|
||Acc|87.67|87.59|**88**|87.78|87.41|87.54|87.02|
|Nira|Atk Rate|**2.23**|2.23|2.41|2.65|3.09|2.47|3.15|
||Main Acc|87.88|87.83|**88.22**|88.05|87.79|87.82|87.22|

Reviewer comment (Experiments):
The experiments are conducted on relatively limited settings, i.e., only one triggered-based backdoor attack is considered, and small dataset.

Author response:
***Ans for Q8):*** Thanks for your valuable comments.
- We set the experimental settings following previous outstanding works [1,2,3]. 
- We appreciate your suggestion. We would like to note that completing the experiments on the ImageNet dataset during the discussion period is challenging. Thus, we will add the mentioned large-scale datasets in our revision when we finish the experiments.

Reviewer comment (Experiments):
What is the ratio of the number of surrogate data and real data for each clients? How will this ratio affect the performance?

Author response:
***Ans for Q9):*** Thanks for pointing out the lack of information about surrogate data. We have added the following information and experiments to our revision.

- The size of the surrogate dataset is $2000$, which is a small proportion of the utilized datasets, i.e., $3.33\%$ for CIFAR-10, $2.86\%$ for FMNIST, and $0.33\%$ for SVHN.
- Following your valuable question, we conduct ablations on the sensitivity of the size of the surrogate dataset and report results in Table 1-5 as below. We can see that the size of the surrogate dataset has a limited impact on the performance. Even when the surrogate dataset size is only 500, the proposed method still demonstrates excellent performance.

**Table 1-5**
|Defense|Metric||||Surrogate Dataset Size||||
|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|
|||500|1000|2000|3000|4000|6000|8000|
||Acc|87.74|**87.89**|87.67|87.26|86.69|87.75|87.61|
|Nira|Atk Rate|1.6|3.22|2.23|**1.18**|3.17|2.23|3.26|
||Main Acc|87.84|**88.01**|87.88|87.27|86.89|87.87|87.81|