R OBUST F EDERATED L EARNING FOR N EURAL N ETWORKS
Anonymous authors
Paper under double-blind review
A BSTRACT
In federated learning, data is distributed among local clients which collaboratively train a prediction model using secure aggregation.
To preserve the privacy of the clients, the federated learning paradigm requires each client to maintain a private local training data set, and only uploads its summarized model updates to the server.
In this work, we show that this paradigm could lead to a degenerate model, which collapses in performance when the corrupted updates (under adversarial manipulations) are uploaded and aggregated at the server.
To improve model robustness, we ﬁrst decompose the aggregation error of the central server into bias and variance, and then, propose a robust federated learning framework, named Fed BVA , that performs on-device adversarial training using the bias-variance oriented adversarial examples supplied by the server via asymmetrical communications.
The experiments are conducted on multiple benchmark data sets using several prevalent neural network models, and the empirical results show that our framework is robust against white-box and black-box adversarial corruptions under both IID and non-IID settings.
I NTRODUCTION
The explosive amount of decentralized user data collected from the ever-growing usage of smart devices, e.g., smartphones, wearable devices, home sensors, etc., has led to a surge of interest in the ﬁeld of decentralized learning.
To protect the privacy-sensitive data of the clients, federated learning (McMahan et al., 2017; Yang et al., 2019) has been proposed.
Federated learning only allows a group of clients to train local models using their own data, and then collectively merges the model updates on a central server using secure aggregation (Acar et al., 2018).
Due to its high privacy-preserving property, federated learning has attracted much attention in recent years along with the prevalence of efﬁcient light-weight deep models (Howard et al., 2017) and low-cost network communications (Wen et al., 2017; Koneˇcn`y et al., 2016).
In federated learning, the central server only inspects the secure aggregation of the local models as a whole.
Consequently, it is susceptible to clients’ corrupted updates (e.g., system failures, adversarial manipulations, etc.).
Recently, multiple robust federated learning models (Fang et al., 2019; Pillutla et al., 2019; Portnoy & Hendler, 2020; Mostafa, 2019) have been proposed.
However, these works only focus on performing client-level robust training or designing server-level aggregation variants with hyper-parameter tuning, and largely ignore the underlying true cause of federated learning’s vulnerability coming from the server’s generalization error, as we will show in Section 4.1.
Our work bridges this gap by investigating the error incurred during secure aggregation of federated learning from the perspective of bias-variance decomposition (Pedro, 2000; Valentini & Dietterich, 2004).
Speciﬁcally, we show that the generalization error of the aggregated model on the central server can be decomposed as the combination of bias (triggered by the main prediction of these clients) and variance (triggered by the variations among clients’ predictions).
Next, we propose to perform the local robust training on clients by supplying them with a tiny amount of the biasvariance perturbed examples generated from the central server via asymmetrical communications.
The experiments are conducted on neural networks with cross-entropy loss, however, other loss functions are also applicable as long as their gradients w.r.t.
bias and variance are tractable to estimate.
In this way, any gradient-based adversarial training strategies (Goodfellow et al., 2015; Madry et al., 2018) could be used.
Compared with previous work, our major contributions include:
• We provide the exact solution of bias-variance analysis w.r.t.
the generalization error which is perfectly suitable for neural network based federated learning.
As a comparison, performing
adversarial attacks or training with conventional federated learning methods will only focus on the bias of the central model but ignore the variance.
• We demonstrate that the conventional federated learning framework is vulnerable to the strong attacking methods with increasing communication rounds even if the adversarial training using the locally generated adversarial examples is performed on each client.
• Without violating the clients’ privacy, we show that providing a tiny amount of bias-variance perturbed data from the central server to the clients through asymmetrical communication could dramatically improve the robustness of the training model under various settings.
P RELIMINARIES 2. S ETTINGS
In federated learning, there is a central server and K different clients, each with access to a private training set D k
= { ( x ki , t ki ) } n k i =1 , where x ki , t ki , and n k are the features, label, and number of training examples in the k th client ( k = 1 , · · · , K ) .
Each data D k is exclusively owned by client k and will not be shared with the central server or other clients.
In addition, there is a small public training set D s = { ( x sj , t sj ) } n s j =1 with n s training examples from the server that is shared with clients, where n s ⌧
P
K k =1 n k .
Note that this will not break the privacy constraints, for example, hospitals (local devices) that contribute to a federated learned medical image diagnosis system could take a few publicly accessible images as additional inputs.
The goal of federated learning is to train a global classiﬁer f ( · ) using knowledge from all the clients such that it generalizes well over test data D test .
The notation used in this paper is summarized in the Appendix (see Table 4).
2. P ROBLEM D EFINITION
Privacy-preserving federated learning trains the machine learning models without directly accessing to the clients’ raw data.
A group of clients could collaborate in achieving the learning objective under the coordination of a central server that aggregates the local clients’ model updates asynchronously (Kairouz et al., 2019).
In this paper, we study the adversarial robustness of neural networks 1 in federated learning setting, and we deﬁne robust decentralized learning as follows.
Deﬁnition 2.1.
(Robust Federated Learning)
Input: (1) A set of private training data {D k } Kk =1 on K different clients; (2) Tiny amount of training data D s on the central server; (3) Learning algorithm f
( · ) and loss function L ( · , · ) .
Output: A trained model on the central server that is robust against adversarial perturbation.
We would like to point out that our problem deﬁnition has the following properties: Asymmetrical communication:
The asymmetrical communication between each client and server cloud is allowed: the server provides both global model parameters and limited shared data to the clients; while each client only uploads its local model parameters back to the server.
Data distribution: All training examples on the clients and the server are assumed to follow the same data distribution.
However, the experiments show that our proposed algorithm also achieves outstanding performance under the non-IID setting, which could be common among personalized clients in real scenarios.
Shared learning algorithm:
All the clients are assumed to use the identical model f ( · ) , including architectures as well as hyper-parameters (e.g., learning rate, local epochs, local batch size).
2. B IAS -V ARIANCE T RADE - OFF
Following (Pedro, 2000; Valentini & Dietterich, 2004), we deﬁne the optimal prediction, main prediction as well as the bias, variance, and noise for any real-valued loss function L ( · , · ) as follows:
Deﬁnition 2.2.
( Optimal Prediction and Main Prediction )
Given a loss function L ( · , · ) and a learning algorithm f ( · ) , the optimal prediction y ⇤ and main prediction y m for an example are deﬁned as:
[Equation]
where t and D are viewed as the random variables to denote the class label and training set, and f D denotes the model trained on D .
In short, the main prediction is the prediction whose average loss relative to all the predictions over data distributions is minimum, e.g., the main prediction for zeroone loss is the mode of predictions.
In this work, we show that the main prediction is the average prediction of client models for mean squared (MSE) loss and cross-entropy (CE) loss in Section 4.1.
Deﬁnition 2.3.
( Bias, Variance and Noise )
Given a loss function L ( · , · ) and a learning algorithm f ( · ) , the expected loss E D ,t
[ L ( f D ( x ) , t )] for an example x can be decomposed 2 into bias, variance and noise as follows:
[Equation]
In short, bias is the loss incurred by the main prediction w.r.t.
the optimal prediction, and variance is the average loss incurred by predictions w.r.t.
the main prediction.
Noise is conventionally assumed to be irreducible and independent to f ( · ) .
Remark.
Our deﬁnitions on optimal prediction, main prediction, bias, variance and noise slightly differ from previous ones (Pedro, 2000; Valentini & Dietterich, 2004).
For example, conventional optimal prediction was deﬁned as y ⇤ ( x ) =
arg min y E t [ L ( t, y )] , and it is equivalent to our deﬁnition when loss function is symmetric over its arguments, i.e., L ( y 1 , y 2 ) =
L ( y 2 , y 1 ) .
T HE P ROPOSED F RAMEWORK
A typical framework (Kairouz et al., 2019) of privacy-preserving federated learning can be summarized as follows: (1) Client Update: Each client updates local model parameters w k by minimizing the empirical loss over its own training set; (2) Forward Communication: Each client uploads its model parameter update to the central server; (3) Server Update: It synchronously aggregates the received parameters; (4) Backward Communication:
The global parameters are sent back to the clients.
Our framework follows the same paradigm but with substantial modiﬁcations as below.
Server Update .
The server has two components: The ﬁrst one uses FedAvg (McMahan et al., 2017) algorithm to aggregate the local models’ parameters, i.e., w G =
Aggregate( w 1 , · · · , w K ) =
P
Kk =1 n k n w k where n =
P
Kk =1 n k and w k is the model parameters in the k th client.
Meanwhile, another component is designed to produce adversarially perturbed examples which could be induced by a poisoning attack algorithm for the usage of robust adversarial training.
It has been well studied (Belkin et al., 2019; Pedro, 2000; Valentini & Dietterich, 2004) that in the classiﬁcation setting, the generalization error of a learning algorithm on an example is determined by the bias, variance, and irreducible noise as deﬁned in Eq.
Similar to the previous work, we also assume a noise-free learning scenario where the class label t is a deterministic function of x (i.e., if x is sampled repeatedly, the same values of its class t will be observed).
This motivates us to generate the adversarial examples by attacking the bias and variance induced by clients’ models as:
[Equation]
where B (ˆ x ; w 1 , · · · , w K ) and V (ˆ x ; w 1 , · · · , w K ) could be empirically estimated from a ﬁnite num- ber of clients’ parameters trained on local training sets {D 1 , D 2 , · · · , D K } .
Here  is a hyper- parameter to measure the trade-off of bias and variance, and ⌦ ( x ) is the perturbation constraint.
Note that D s (on the server) is the candidate subset of all available training examples that would lead to their perturbed counterparts.
This is a more feasible setting as compared to generating adversarial examples on clients’ devices because the server usually has much powerful computational capacity in real scenarios that allows the usage of ﬂexible poisoning attack algorithms.
In this case, both poisoned examples and server model parameters would be sent back to each client ( Backward
Communication ), while only clients’ local parameters would be uploaded to the server ( Forward
Communication ), i.e., the asymmetrical communication as discussed in Section 2.2.
Client Update .
The robust training of one client’s prediction model (i.e., w k ) can be formulated as the following minimization problem.
[Equation]
where ˆ x sj 2 ⌦ ( x sj ) is the perturbed examples that is asymmetrically transmitted from the server.
Remark.
Intuitively, the bias measures the systematic loss of a learning algorithm, and the variance measures the prediction consistency of the learner over different training sets.
Therefore, our robust federated learning framework has the following advantages: (i) it encourages the clients to consistently produce the optimal prediction for perturbed examples, thereby leading to a better generalization performance; (ii) local adversarial training on perturbed examples allows to learn a robust local model, and thus a robust global model could be aggregated from clients.
Theoretically, we could still have another alternative robust federated training strategy:
[Equation]
where the perturbed training examples of each client k is generated on local devices from D k instead of transmitted from the server.
This min-max formula is similar to (Madry et al., 2018; Tram`er et al., 2018) where the inner maximization problem synthesizes the adversarial counterparts of clean examples, while the outer minimization problem ﬁnds the optimal model parameters over perturbed training examples.
Thus, each local robust model is trained individually, nevertheless, poisoning attacks on device will largely increase the computational cost and memory usage.
Meanwhile, it only considers the client-speciﬁc loss and is still vulnerable against adversarial examples with increasing communication rounds.
Both phenomena are observed in our experiments (see Fig. 4 and Fig. 5).
A LGORITHM 4. B IAS -V ARIANCE A TTACK
We ﬁrst consider the maximization problem in Eq.
(3) using bias-variance based adversarial attacks.
It aims to ﬁnd the adversarial example ˆ x (from the original example x ) that would produce large bias and variance values w.r.t.
clients’ local models.
Speciﬁcally, perturbation constraint ˆ x 2 ⌦ ( x ) forces the adversarial example ˆ x to be visually indistinguishable w.r.t.
x .
Here we consider the wellstudied l 1 -bounded adversaries 3 (Goodfellow et al., 2015;
Madry et al., 2018; Tram`er et al., 2018) such that ⌦ ( x ) :
=
{ ˆ x
||
ˆ x  x || 1  ✏ } for a perturbation magnitude ✏ .
Furthermore, we propose to consider the following two gradient-based algorithms to generate adversarial examples.
Bias-variance based Fast Gradient Sign Method (BV-FGSM):
Following FGSM (Goodfellow et al., 2015), it linearizes the maximization problem in Eq.
(3) with one-step attack as follows.
[Equation]
Bias-variance based Projected Gradient Descent (BV-PGD): PGD can be considered as a multistep variant of FGSM (Kurakin et al., 2017) and might generate powerful adversarial examples.
This motivated us to derive a BV-based PGD attack:
[Equation]
where ˆ x l is the adversarial example at the l th step with the initialization ˆ x 0 = x and Proj ⌦ ( x ) ( · ) projects each step onto ⌦ ( x ) .
Remark.
The proposed framework could be naturally generalized to any gradient-based adversarial attack algorithms where the gradients of bias B ( · ) and variance V ( · ) w.r.t.
x are tractable when estimated from ﬁnite training sets.
Compared with the existing attack methods (Carlini & Wagner, 2017; Goodfellow et al., 2015; Kurakin et al., 2017; Moosavi-Dezfooli et al., 2016), our loss function the adversary aims to optimize is a linear combination of bias and variance, whereas existing work mainly focused on attacking the overall classiﬁcation error that considers bias only.
The following theorem states that bias B ( · ) and variance V ( · ) as well as their gradients over input x could be estimated using the clients’ models.
Theorem 4.1.
Assume that L ( · , · ) is the cross-entropy loss function, then, the empirical estimated main prediction y m for an input example ( x, t ) has the following closed-form expression:
[Equation]
Furthermore, the empirical bias and variance, as well as their gradients over an input x are estimated as follows:
[Equation]
Here, H ( y m ) =
[Equation]
1 y
[Equation]
j ) m log y ( j ) m is the entropy of the main prediction y m and C is the number of classes. Easily, we can have their gradients in terms of the bias and variance as r x B ( x ; w 1 , · · · , w
[Equation]
K
[Equation]
and r
[Equation]
Details of the proof is elaborated in A.2.
In addition, we also consider the case where L ( · , · ) is the MSE loss function.
Although its main prediction y m for an input example ( x, t ) has a closed-form expression which is exactly the same as the cross-entropy loss, its empirical bias and unbiased variance can only be estimated in the following formulas:
B ( x ; w 1 , · · · , w K ) =
|| 1 K
[Equation]
||[-
-]
22 and V ( x ; w
[Equation] [Equation]
, w K ) =
K
[Equation]
Note that the gradients of MSE’s bias and variance are much more computational demanding comparing with the concise formulas that crossentropy ends up with.
More comparisons are illustrated in Appendix A.5.1. : m
S r sampled m clients 6: for each client k 2 S r in parallel do 7: w rk ClientUpdate ( w r  1 G , ˆ D s , k ) 8: end for 9: ˆ D s BVAttack ( D s , { w rk }| k 2 S r ) 10: w rG
Aggregate ( w rk | k 2 S r ) 11: end for 12: return w G
Algorithm 2 ClientUpdate( w, ˆ D s , k ) 1: Initialize k th client’s model with w 2: B split D k
[ ˆ D s into batches of size B 3: for each local epoch i
= 1 , 2 , · · · , E do 4: for local batch ( x, t ) 2 B do 5: w w  ⌘ r L ( f D k ( x ; w ) , t ) 6: end for 7: end for 8: return w
{+Algorithm 3 BVAttack( D s , { w rk }| k 2 S r ) 1: Initialize ˆ D s = ; 2: for ( x, t ) 
+}
D s do 3:
Estimate the gradients r x B ( x ) and r x V ( x ) using Theorem 4.
Calculate ˆ x using Eq.
(6) or (7) and add to ˆ D s 5: end for 6: return ˆ D s
We present a novel robust federated learning algorithm with our proposed bias-variance attacks, named Fed BVA .
Following the framework deﬁned in Eq.
(3) and Eq.
(4), key components of our algorithm are (1) bias-variance attacks for generating adversarial examples on the server, and (2) adversarial training using poisoned server examples together with clean local examples on each client.
Therefore, we optimize these two objectives by producing the adversarial examples ˆ D s and updating the local model parameters w iteratively.
The proposed algorithm is summarized in Alg. 1.
Given the server’s D s and clients’ training data {D k } Kk =1 as input, the output is a robust global model on the server.
First, it initializes the server’s model parameter w G and perturbed data ˆ D s , and then assigns to the randomly selected clients (Steps 4-5).
Next, each client optimizes its own local model (Steps 6-8) with the received global parameters w G as well as its own clean data D k , and uploads the updated parameters back to the server.
At last, the server generates the perturbed data ˆ D s
(Step 9) using the proposed bias-variance attack algorithm (see Alg. 3) and aggregates the global model parameters (Step 10) using an aggregation method, e.g., FedAvg (McMahan et al., 2017).
E XPERIMENTS 5. S ETTINGS
In this section, we evaluate the adversarial robustness of our proposed algorithm on four benchmark data sets: MNIST 4 , Fashion-MNIST 5 , CIFAR-10 6 and CIFAR-100 6 .
The baseline models we used include: (1).
FedAvg : the classical federated averaging model (McMahan et al., 2017).
FedAvg AT : The simpliﬁed version of our proposed method where the local clients perform adversarial training with the asymmetrical transmitted perturbed data generated on top of FedAvg ’s aggregation.
Fed Bias, Fed Variance, Fed BVA :
Our proposed methods where the asymmetrical transmitted perturbed data is generated using the gradients of bias-only attack, variance-only attack, and bias-variance attack, respectively.
EAT : Ensemble adversarial training (Tram`er et al., 2018), where each client performs local adversarial training using Eq.
(5), and their model updates are aggregated on server using FedAvg .
For fair comparisons, all baselines are modiﬁed to the asymmetrical communications setting ( FedAvg and EAT have clean D s received), and all their initializations are set to be the same.
EAT+Fed BVA :
A combination of baselines (5) and (6).
Note that baselines (5) and (6) have high computational requirements on client devices, and are usually not preferred in real scenarios.
For the defense model, we use a 4-layer CNN model for MNIST and Fashion-MNIST, and VGG architecture for CIFAR-10 and CIFAR-100.
Regarding blackbox attacks, we apply ResNet18 (He et al., 2016), VGG11 (Simonyan & Zisserman, 2015), Xception (Chollet, 2017), and MobileNetV2 (Sandler et al., 2018) for CIFAR data, and provide a variety of models for MNIST and
Fashion-MNIST by following the design of (Tram`er et al., 2018).
The training is performed using the
SGD optimizer with ﬁxed learning rate of 0 . 01 and momentum of value 0 .
All hyper-parameters of federated learning are presented in Table 5 in the Appendix.
We empirically demonstrate that these hyper-parameter settings are preferable in terms of both training accuracy and robustness (see the details of Fig.
6 - Fig.
14 in the Appendix).
To evaluate the robustness of our federated learning algorithm against adversarial attacks, except for the clean model training, we perform FGSM (Goodfellow et al., 2015), PGD (Kurakin et al., 2017) with 10 and 20 steps towards the aggregated server model on the D test .
Following (Tram ` er et al., 2018; Wang et al., 2019), the maximum perturbations allowed are ✏ = 0 .
3 on MNIST and Fashion-MNIST, and ✏ = 16255 on CIFAR-10 and CIFARfor both threat and defense models.
For IID sampling, the data is shufﬂed and uniformly partitioned into each client;
For non-IID setting, data is divided into 2 F · K shards based on sorted labels, then assigns each client with 2 shards.
Thereby, each client will have data with at most two classes.
5. R ESULT A NALYSIS
To analyze the properties of our proposed Fed BVA framework, in Fig.
1, we visualize the extracted gradients using adversarial attack from bias, variance, and bias-variance.
Notice that the gradients of bias and variance are similar but with subtle differences in local pixel areas.
However, according to Theorem 4.1, the gradient calculation of these two are quite different: bias requires the target label as input, but variance only needs the model output and main prediction.
From another perspective, we also investigate the bias-variance magnitude relationship with varying model complexity.
As shown in Fig.
2, with increasing model complexity (more convolutional ﬁlters in CNN), both bias and variance decrease.
This result is different from the double-descent curve or bell-shape [Figure]
[Table]
variance curve claimed in (Belkin et al., 2019; Yang et al., 2020).
The reasons are twofold: First, their bias-variance deﬁnitions are from the MSE regression decomposition perspective, whereas our decomposition utilizes the concept of main prediction, and the generalization error is decomposed from the classiﬁcation perspective; Second, their implementations only evaluate the bias and variance using training batches on one central model and thus is different from the deﬁnition which requires the variance to be estimated from multiple sub-models (in our scenario, client models).
The convergence plot of all baselines is presented in Fig.
We observe that FedAvg has the best convergence, and all robust training will have a slightly higher loss upon convergence.
This matches the observations in (Madry et al., 2018) which state that training performance may be sacriﬁced in order to provide robustness for small capacity networks.
For the model performance shown in
Fig.
4, we observe that the aggregation of federated learning is vulnerable to adversarial attacks since both FedAvg and EAT have decreased performance with an increasing number of server-client communications.
Other baselines that utilized the asymmetrical communications have increasing robustness with more communication rounds although only a small number of perturbed examples ( n s = 64 ) are transmitted.
We also observe that when communication rounds reach 40, Fed BVA starts to outperform EAT while the latter is even more resource-demanding than Fed BVA (shown in
Fig.
5, where the pie plot size represents the running time).
Overall, bias-variance based adversarial training via asymmetric communication is both effective and efﬁcient for robust federated learning.
For the comprehensive experiments shown in Table 1 and Table 2, it is easy to verify that our proposed model outperforms all other baselines regardless of the source of the perturbed examples (i.e., locally generated like EAT+Fed BVA or asymmetrically transmitted from the server like
Fed BVA ).
Comparing with standard robust federated learning FedAvg AT , the performance of
Fed BVA against adversarial attacks still increases 4%  13% and 2%  9% on IID and non-IID settings respectively, although Fed BVA is theoretically suitable for the cases that clients have IID samples.
In Table 3, we observe a similar trend where Fed BVA outperforms FedAvg AT on
CIFAR-10 and CIFAR-100 (with 0 .
2%  10% increases) when defending different types of adversarial examples.
Comparing with strong local adversarial training baseline EAT , we also observe a maximum 13% accuracy increase when applying its bias-variance oriented baseline EAT+Fed BVA .
We also conducted various additional experiments in Appendix A.5 which includes: (1) Comparison of efﬁciency and effectiveness of
[Table]
[Table]
[Table]
R ELATED W ORK
Adversarial Machine Learning: While machine learning models have achieved remarkable performance over clean inputs, recent work (Goodfellow et al., 2015; Szegedy et al., 2014) showed that those trained models are vulnerable to adversarially chosen examples by adding the imperceptive noise to the clean inputs.
In general, the adversarial robustness of centralized machine learning models have been explored from the following aspects: adversarial attacks (Biggio et al., 2012; Carlini & Wagner, 2017; Guo et al., 2019; Athalye et al., 2018; Zhu et al., 2019), defense (or robust model training) (Madry et al., 2018; Shafahi et al., 2019; Tram`er & Boneh, 2019; Carlini et al., 2019; Tram`er et al., 2018) and interpretable adversarial robustness (Schmidt et al., 2018; Cullina et al., 2018; Tsipras et al., 2018).
Federated Learning:
Federated learning with preserved privacy (Koneˇcn`y et al., 2016; McMahan et al., 2017; Hard et al., 2018) has become prevalent in recent years.
Meanwhile, the vulnerability of federated learning to backdoor attacks has also been explored by (Bagdasaryan et al., 2018;
Bhagoji et al., 2019; Xie et al., 2019).
Following their work, multiple robust federated learning models (Fang et al., 2019; Pillutla et al., 2019; Portnoy & Hendler, 2020; Mostafa, 2019) are also proposed and studied.
In this paper, we studied the underlying true cause of federated learning’s vulnerability from the perspective of bias-variance analysis.
This is in sharp contrast to the existing work that focused on performing client-level model poisoning attacks or designing server-level aggregation variants with hyper-parameter tuning.
Bias-Variance Decomposition:
Bias-variance decomposition (Geman et al., 1992) was originally introduced to analyze the generalization error of a learning algorithm.
Then, a generalized biasvariance decomposition (Pedro, 2000; Valentini & Dietterich, 2004) was studied in the classiﬁcation setting which enabled ﬂexible loss functions (e.g., squared loss, zero-one loss).
More recently, bias-variance trade-off was experimentally evaluated on modern neural network models (Neal et al., 2018; Belkin et al., 2019;
Yang et al., 2020).
C ONCLUSION
In this paper, we proposed a novel robust federated learning framework, in which the aggregation incurred loss during the server’s aggregation is dissected into a bias part and a variance part.
Our approach improves the model robustness through adversarial training by supplying a few bias-variance perturbed samples to the clients via asymmetrical communications.
Extensive experiments have been conducted where we evaluated its performance from various aspects on several benchmark data sets.
We believe the further exploration of this direction will lead to more ﬁndings on the robustness of federated learning.