Paper: Kick Bad Guys Out! Zero-Knowledge-Proof-Based Anomaly Detection in Federated Learning (ICLR-2024)

Reviewer comment (Experiments):
The authors did not compare with FL anomaly detection methods such as FLDetector in experiments.

Author response:
As already mentioned in our paper, FLDetector is not practical in real-world scenarios. It relies heavily on historical client models from previous training rounds. By examining their detailed implementation, we found that the number of pre-training rounds may vary based on the datasets. For example, the number of pre-training rounds is set to 50 when the datasets are MNIST and FEMNIST, and 20 when the dataset is CIFAR10. Since our approach starts with zero pre-training rounds, it is more generalized, and we do not think there is need in the experiment section to compare our approach with FLDetector which has the practical design limitations above. Thus, in our paper, we decided to mention this work in the Related Works section.

Reviewer comment (Experiments):
The authors did not test their methods with some strong attacks such as [1], [2], and [3].

Author response:
Reviewer TEpF suggested 3 attacks (paper [1][2][3]) to us. While we have already included one of the three attacks in our experiments ([2]), we would like to point out that our work has already leveraged stronger and more practical attacks in our paper, compared to those suggested by Reviewer TEpF. 

As addressed in the whole paper, our anomaly detection solution handles the challenges faced by real-world FL systems. The basic rule to select attacks in our experiments is that the attack must be practical to be utilized by malicious clients in real-world cases. 

As for the other two attacks suggested by Reviewer TEpF, [1] is a famous defense paper in fact, – even its title clearly reflects that it is a defense paper. Though they design an attack, to the best of our knowledge, there is neither an existing work leveraging this attack nor it is practical in real-world scenarios. As shown in their open-sourced implementation (https://github.com/moranant/attacking_distributed_learning/blob/master/malicious.py), the method requires the malicious clients to modify their local models based on the “distribution” of all local models in the current FL training round, using a mean and a standard deviation. Implementing this in real-world FL applications requires the server (or a malicious third party) to compute the real mean (i.e., the averaged model) and the standard deviation, then send them back to the malicious clients for them to compute the poisoned local model. If the mean (averaged model) is computed by the server, the server has no reason to take the poisoned local model again from the local clients, as it has already obtained the averaged model for the current FL round. If the mean is computed by a malicious third party, then the assumption is that the third party attacker has taken control over the whole FL system, which is too strong and almost impractical in real-world scenarios (remember we have an honest majority). 

Regarding the attack in [3], in fact, in our experiments, we have already considered two practical attacks ([2] in the reviewer’s suggested attacks and [4]) that are stronger than [3]. The stronger attack in our experiments can even induce the global model to be the local model submitted by a malicious client. Even under this scenario, we show that our proposed method works effectively.

[4] Chen, Yudong, Lili Su, and Jiaming Xu. "Distributed statistical machine learning in adversarial settings: Byzantine gradient descent." Proceedings of the ACM on Measurement and Analysis of Computing Systems 1.2 (2017): 1-25.

Reviewer comment (Experiments):
The authors did not explore the influence of the number of malicious clients.

Author response:
In Weakness 3 (W3), the reviewer said “The authors did not explore the influence of the number of malicious clients.” However, in the original submitted version, we **exactly included** the experiment as Exp 5 (Varying the number of malicious clients) in the experiment section, – not the appendix. We varied the percentage of malicious clients to be 0, 20%, and 40%, and our approach works pretty well, – in both cases of 20% and 40% malicious clients, the test accuracy remains very close to the benign case.  

We set the percentage of malicious clients to 20% and 40%, and since we utilize 10 clients, the number of malicious clients is 2 and 4. As a comparison, we also included a benign case where there is no malicious client. The reason for this setting is as follows. 1) the number of clients to be 10: In real-world cases, the number of clients can hardly larger than 20, while in most cases, the number of clients is less than 10. 2) the percentage of malicious clients: we would like to first point out that security solutions often address scenarios that are less frequently encountered in the real world – those that may rarely occur, but when they do, they can lead to serious consequences. This holds true for security issues in FL systems as well. In real FL systems, the proportion of malicious clients is typically no more than 10%. We would like to evaluate the performance of our approach to handle more malicious clients, thus we set the percentage of malicious clients to be 20% and 40%. The results show that our approach works pretty well, – in both cases of 20% and 40% malicious clients, the test accuracy remains very close to the benign case. 3) The reason why we did not include 10% and 30%: the performance of our approach is quite good, and the test accuracy always remains very close to the benign case when we vary the percentage of malicious clients from 10% to 40%. To make the lines in the figure clear and easy to see, we decided to only include two settings and set the percentage of malicious clients to 20% and 40% only.

Reviewer comment (Theory):
The proposed method relies on applying the **Three Sigma Rule** to identify outlier clients, but does not adequately justify the underlying assumption that client behavior will follow a **Gaussian/normal distribution**. Without evidence or validation that the models meet this statistical requirement, the correctness and reliability of using this rule is unclear. The authors need to provide empirical or theoretical support for why these distributions were expected in this context, otherwise the key thresholding technique lacks rigorous foundation.

Author response:
We appreciate your feedback on the necessity of justifying the Gaussian distribution assumption for client models. As studied extensively in the distributed learning literature, if the local data of the clients are i.i.d., parameters of the local models follow a Gaussian distribution, see  [5][6][7]. Further, even if the client datasets are non-i.i.d., based on the Central Limit Theorem, the distribution of the local models tends toward a normal distribution.

We included the following description in Section 3.2.
```
The 3 Sigma Rule is pivotal for two reasons: i) in case the client datasets are i.i.d., parameters of local models follow a Gaussian distribution [5][6][7] ; and ii) even when client datasets are non-i.i.d., the Central Limit Theorem indicates that local models tend toward a normal distribution.
```
[5] Baruch, Gilad, Moran Baruch, and Yoav Goldberg. "A little is enough: Circumventing defenses for distributed learning." Advances in Neural Information Processing Systems 32 (2019).

[6] Chen, Yudong, Lili Su, and Jiaming Xu. "Distributed statistical machine learning in adversarial settings: Byzantine gradient descent." Proceedings of the ACM on Measurement and Analysis of Computing Systems 1.2 (2017): 1-25.

[7] Yin, Dong, et al. "Byzantine-robust distributed learning: Towards optimal statistical rates." International Conference on Machine Learning. PMLR, 2018.

Reviewer comment (Novelty):
The paper claims to propose a new defensive mechanism, but its core algorithms - Krum, Three Sigma Rule and Zero-knowledge proofs - are already well-established in prior work. While integrating existing techniques can still yield new systems, the paper does not provide sufficient insight into how this combination offers meaningful advantage. Without novel algorithmic or analytical insights, the technical value of merely assembling known pieces is limited. To strengthen the paper, the authors should demonstrate deeper understanding of how this specific integration improves upon the state-of-the-art.

Author response:
Thank you for pointing out the need for a clearer demonstration of the novel contributions of our work. Our solution handles anomaly detection in real-world FL systems, and existing state-of-the-art, e.g., Krum, does not work well due to the following reasons: 1) Krum keeps only one (or multiple for m-Krum) local model that is the most likely to be benign to represent the other local models in each FL iteration. This will completely waste the computation from the other benign clients and undermine the aggregation results. 2) Krum assumes attacks happen in each round of FL iteration. However, in real-world scenarios, attacks happen infrequently, and applying Krum will change the aggregation results even if there are no malicious clients. Considering these, we design a 2-step anomaly detection algorithm that first detects whether attacks have happened, and then, if attacks happened, we will further apply an anomaly detection procedure to remove the potential malicious client models. We utilize the 3 sigma rule to define a bound for the client models, so that the anomaly detection procedure removes a varying number of malicious client models based on the detection results, instead of removing a fixed number of client models. We also utilize zero-knowledge proofs to provide a verification to the clients so that  they ensure that the anomaly detection of the server is correct. With all these, our anomaly detection scheme offers novel contributions to the anomaly detection in FL systems.

We included the following description in the introduction section:
```
Moreover, existing defense mechanisms are deployed at the FL server without any verification procedures to ensure their correct execution. While most of the clients are benign and wish to collaboratively train machine learning models, they can also be skeptical about the server's reliability during the execution of the defense mechanisms that modify the original aggregation procedure. 
```
We also included the following description about Krum in the Related Work section:
```
For each FL round, instead of using all client submissions for aggregation, such approaches keep local models that are the most likely to be benign to represent the other local models. Such approaches are effective, but they keep less local models than the real number of benign local models to ensure that all Byzantine local models are filtered out, causing misrepresentation of some benign local models in aggregation. This completely wastes the computation resources of the benign clients that are not being selected and thus, changes the aggregation results as some benign local models do not participate in aggregation. 
```

Reviewer comment (Experiments):
The evaluation of the proposed method's scalability and performance is limited by the small-scale datasets used (e.g. FEMNIST, CIFAR). While useful for proof-of-concept, these datasets do not adequately represent the data and computational heterogeneity of modern large-scale Federated Learning systems. To fully demonstrate the practicality and effectiveness of this defense, it will be important for the authors to test its performance and overhead when applied to real-world FL scenarios at a larger scale. Without such experimentation on industry-grade datasets and systems, the approach's scalability and real-world viability remain uncertain.

Author response:
Thank you for your feedback regarding the scalability and real-world applicability of our proposed method. We agree that testing on small-scale datasets may not fully capture the challenges of large-scale Federated Learning systems. To address this, we will extend our evaluations to include larger, more diverse datasets that are representative of real-world scenarios. We also included an experiment that evaluates our approach using real-world edge devices in the appendix. The description of the experiment is as follows: 
```
Exp8: Evaluations in real-world applications. We utilize edge devices from the Theta network (Theta Network., 2023) to validate the scalability of our anomaly detection approach to real-world applications. The FL client package is integrated into Theta’s edge nodes, which periodically fetch data from the Theta back-end. Subsequently, the FL training server capitalizes on these Theta edge nodes and their associated data to train, fine-tune, and deploy machine learning models. We utilize the Byzantine attack of random mode. Considering the challenges posed by real-world environments, such as devices equipped solely with CPUs (lacking GPUs), potential device connectivity issues, network latency, and limited storage on edge devices (for instance, some mobile devices might have less than 500MB of available storage), we choose a simple task by employing the MNIST dataset for a logistic regression task. We deploy 20 client edge devices, and set 5 as malicious for each FL training round. The information of the Theta devices is shown in Figure 15. The training process is shown in Figure 16, and the total training time is 221 seconds. We also include the CPU utilization and network traffic during training, which are shown in Figure 17 and Figure 18, respectively.
```

Reviewer comment (Evaluation):
The explanations of some experimental results are not entirely convincing.

Author response:
Thank you for your suggestion. For the selection of the importance layer, our aim is to choose a general case that is suitable for real-world scenarios that involve various models and datasets. Intuitively, we selected the second-to-the-last layer,  as it contains more information and can represent the whole model. In Experiment 1, we observed that this layer exhibited higher sensitivity compared to most other layers across all scenarios, thus can be utilized to represent the whole model. We acknowledge that our initial presentation for Experiment 1 was unclear. To clarify this, we have revised our descriptions as follows.
```
We utilize the norm of gradients to evaluate the "sensitivity" of each layer. A layer with a norm higher than most of the other layers indicates higher sensitivity compared with most of the other layers, thus can be utilized to represent the whole model. We evaluate the sensitivity of layers of CNN, RNN, and ResNet56. The results for RNN are shown in Figure 4, and the results for ResNet56 and CNN are deferred to Figure 14 and Figure 13  in Appendix B.2, respectively. The results show the sensitivity of the second-to-the-last layer is always higher than most of the other layers, which includes adequate information of the whole model, thus can be selected as the importance layer.
```

Reviewer comment (Theory):
The description of the threat model needs to be more accurate.

Author response:
Thank you for your constructive insights. Based on your feedback, we modified the description of the threat model as follows:

```
We consider an FL system where some clients are malicious, while most clients are honest. The clients have full access to their local data and can train models using their data. They also would like to collaboratively train a model without sharing their data. However, the malicious clients among them may conduct attacks to achieve some adversarial goals, including: i) planting a backdoor to misclassify a specific set of samples while minimally impacting the overall performance of the global model, i.e., backdoor attacks; ii) altering the local models to prevent the global model from converging, i.e., Byzantine attacks; and iii) randomly submitting contrived models without actual training, i.e., free riders. Clients are aware that the server can take some defensive methods to remove potential malicious local models, and they want to verify the mechanism is processed correctly and honestly at the server.
```

Reviewer comment (Experiments):
The justification provided for selecting the 'second-to-the-last layer' in Exp1 is not sufficient.

Author response:
Thank you for your suggestion. For the selection of the importance layer, our aim is to choose a general case that is suitable for real-world scenarios that involve various models and datasets. Intuitively, we selected the second-to-the-last layer,  as it contains more information and can represent the whole model. In Experiment 1, we observed that this layer exhibited higher sensitivity compared to most other layers across all scenarios, thus can be utilized to represent the whole model. We acknowledge that our initial presentation for Experiment 1 was unclear. To clarify this, we have revised our descriptions as follows.
```
We utilize the norm of gradients to evaluate the "sensitivity" of each layer. A layer with a norm higher than most of the other layers indicates higher sensitivity compared with most of the other layers, thus can be utilized to represent the whole model. We evaluate the sensitivity of layers of CNN, RNN, and ResNet56. The results for RNN are shown in Figure 4, and the results for ResNet56 and CNN are deferred to Figure 14 and Figure 13  in Appendix B.2, respectively. The results show the sensitivity of the second-to-the-last layer is always higher than most of the other layers, which includes adequate information of the whole model, thus can be selected as the importance layer.
```

Reviewer comment (Writing):
I didn't fully grasp the significance of "VERIFIABLE ANOMALY DETECTION USING ZKP," and the authors should emphasize the research objectives of this section more prominently.

Author response:
We appreciate your feedback regarding the motivation of using ZKP. The intuition is that, while most of the clients are benign and wish to collaboratively train machine learning models, they may also have concerns about the server's procedure of removing malicious client models, as this procedure may modify the original aggregation result. Considering this, we deployed a ZKP-based verification procedure for the anomaly detection mechanism so that the benign clients can ensure the correct execution of the mechanism at the server. 

Based on your feedback, we included the following description at the beginning of Section 4 (Verifiable Anomaly Detection Using ZKP). We also included corresponding explanations in the introduction.
```
This section introduces a ZKP-based verification procedure so that the benign clients can ensure the correct execution of the anomaly detection mechanism at the server. The intuition is that clients may be skeptical about the removal of some client models during the anomaly detection process at the server, as it may change the aggregation results.
```

Reviewer comment (Writing):
The paper is very well structured and there are occasional grammar hiccups and typos, so I recommend a light editing pass (below are a few of the mistakes collected, but there are more).  
Page 5: In this ppaer –> in this paper

Author response:
Thanks for your feedback! We have proofread the paper multiple times and corrected the typos.