Original: F ED GEMS:
Revised: F ED GEMS:

Original: F EDERATED
Revised: F EDERATED

Original: L EARNING
Revised: L EARNING

Original: OF
Revised: OF

Original: L ARGER
Revised: L ARGER

Original: S ERVER M ODELS VIA S ELECTIVE K NOWLEDGEF U
Revised: S ERVER M ODELS VIA S ELECTIVE K NOWLEDGE

Original: SION
Revised: F USION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Today data is often scattered among billions of resource-constrained edge devices with security and privacy constraints.
Revised: Today data is often scattered among billions of resource-constrained edge devices with security and privacy constraints.

Original: Federated Learning (FL) has emerged as a viable solution to learn a global model while keeping data private, but the model complexity of FL is impeded by the computation resources of edge nodes.
Revised: Federated Learning (FL) has emerged as a viable solution to learn a global model while keeping data private, but the model complexity of FL is impeded by the computation resources of edge nodes.

Original: In this work, we investigate a novel paradigm to take advantage of a powerful server model to break through model capacity in FL.
Revised: In this work, we investigate a novel paradigm to take advantage of a powerful server model to break through model capacity in FL.

Original: By selectively learning from multiple teacher clients and itself, a server model develops in-depth knowledge and transfers its knowledge back to clients in return to boost their respective performance.
Revised: By selectively learning from multiple teacher clients and itself, a server model develops in-depth knowledge and transfers its knowledge back to clients in return to boost their respective performance.

Original: Our proposed framework achieves superior performance on both server and client models and provides several advantages in a unified framework, including flexibility for heterogeneous client architectures, robustness to poisoning attacks, and communication efficiency between clients and server.
Revised: Our proposed framework achieves superior performance on both server and client models and provides several advantages in a unified framework, including flexibility for heterogeneous client architectures, robustness to poisoning attacks, and communication efficiency between clients and server on various image classification tasks.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Nowadays, powerful models with tremendous parameters trained with sufficient computation power are indispensable in Artificial Intelligence (AI), such as AlphaGo (Silver et al., 2016), Alphafold (Senior et al., 2020) and GPT-3 (Brown et al., 2020).
Revised: Nowadays, powerful models with tremendous parameters trained with sufficient computation power are indispensable in Artificial Intelligence (AI), such as AlphaGo (Silver et al., 2016), Alphafold (Senior et al., 2020) and GPT-3 (Brown et al., 2020).

Original: However, billions of resource-constrained mobile and IoT devices have become the primary data source to empower the intelligence of many applications (Bonawitz et al., 2019; Brisimi et al., 2018; Li et al., 2019a).
Revised: However, billions of resource-constrained mobile and IoT devices have become the primary data source to empower the intelligence of many applications (Bonawitz et al., 2019; Brisimi et al., 2018; Li et al., 2019a).

Original: Due to privacy, security, regulatory and economic considerations (Voigt & Von dem Bussche, 2017; Li et al., 2018), it is increasingly difficult and undesirable to pool data together for centralized training.
Revised: Due to privacy, security, regulatory and economic considerations (Voigt & Von dem Bussche, 2017; Li et al., 2018), it is increasingly difficult and undesirable to pool data together for centralized training.

Original: Therefore, federated learning approaches (McMahan et al., 2017; Smith et al., 2017; Caldas et al., 2018; Kairouz et al., 2019; Yang et al., 2019)which allow all the participants to reap the benefits of shared models without sharing private data have become increasingly attractive.
Revised: Therefore, federated learning approaches (McMahan et al., 2017; Smith et al., 2017; Caldas et al., 2018; Kairouz et al., 2019; Yang et al., 2019) allow all the participants to reap the benefits of shared models without sharing private data have become increasingly attractive.

Original: In a typical Federated Learning (FL) training process using FedAvg (McMahan et al., 2017), each client sends its model parameters or gradients to a central server, which aggregates all clients’ updates and sends the aggregated parameters back to the clients to update their local model.
Revised: In a typical Federated Learning (FL) training process using FedAvg (McMahan et al., 2017), each client sends its model parameters or gradients to a central server, which aggregates all clients’ updates and sends the aggregated parameters back to the clients to update their local model.

Original: Because
Revised: Because

Original: FL places computation burden on edge devices, its learnability is largely limited by the edge resources, on which training large models is often impossible.
Revised: FL places computation burden on edge devices, its learnability is largely limited by the edge resources, on which training large models is often impossible.

Original: On the other hand, the server only directly aggregates the clients’ models and its computation power is not fully exploited.
Revised: On the other hand, the server only directly aggregates the clients’ models and its computation power is not fully exploited.

Original: In this work, we investigate the paradigm where the server adopts a larger model in FL.
Revised: In this work, we investigate the paradigm where the server adopts a larger model in FL.

Original: Here we use lar g er s e rver m odel , or GEM to denote the setting where the size of the server model is larger than that of the clients (See Table 1).
Revised: Here we use lar g er s e rver m odel , or GEM to denote the setting where the size of the server model is larger than that of the clients (See Table 1).

Original: Making FL trainable with GEM is desirable to break through model capacity and enable collaborative knowledge fusion and accumulation at server.
Revised: Making FL trainable with GEM is desirable to break through model capacity and enable collaborative knowledge fusion and accumulation at server.

Original: One feasible approach to bridge FL with GEM is through knowledge distillation (KD) (Hinton et al., 2015), where clients and the server transfer knowledge through logits.
Revised: One feasible approach to bridge FL with GEM is through knowledge distillation (KD) (Hinton et al., 2015), where clients and the server transfer knowledge through logits.

Original: For example, FedGKT (He et al., 2020a) adopts a server model as a downstream sub-model and transfers knowledge directly from smaller edge models.
Revised: For example, FedGKT (He et al., 2020a) adopts a server model as a downstream sub-model and transfers knowledge directly from smaller edge models.

Original: In FedGKT the large server model essentially learns from one small teacher at a time and doesn’t learn consensus knowledge from multiple teachers.
Revised: In FedGKT the large server model essentially learns from one small teacher at a time and doesn’t learn consensus knowledge from multiple teachers.

Original: In FL, KD has also been applied (Li & Wang, 2019; Lin et al., 2020) to transfer ensemble knowledge of clients through consensus of output logits rather than parameters.
Revised: In FL, KD has also been applied (Li & Wang, 2019; Lin et al., 2020) to transfer ensemble knowledge of clients through consensus of output logits rather than parameters.

Original: These works either assume no model on the server
Revised: These works either assume no model on the server

Original: (Li & Wang, 2019) or a prototype server model of the same architecture as the client model (Lin et al., 2020).
Revised: (Li & Wang, 2019) or a prototype server model of the same architecture as the client model (Lin et al., 2020).

Revised: Contributions.

Original: In this paper, we first propose a new paradigm to bridge FL with GEM, termed FedGEM , which can learn effectively and efficiently from fused knowledge by resource-constrained clients, and is also able to transfer knowledge back to clients with heterogeneous architectures.
Revised: In this paper, we first propose a new paradigm to bridge FL with GEM, termed FedGEM , which can learn effectively and efficiently from fused knowledge by resource-constrained clients, and is also able to transfer knowledge back to clients with heterogeneous architectures.

Original: To further prevent negative and malicious knowledge transfer, we carefully design a s election and weighting criterion to enhance our knowledge transfer protocol, termed FedGEMS .
Revised: To further prevent negative and malicious knowledge transfer, we carefully design a s election and weighting criterion to enhance our knowledge transfer protocol, termed FedGEMS .

Original: We demonstrate with extensive experiments  that our results significantly surpass the previous state-of-the-art baselines in both homogeneous and heterogeneous settings.
Revised: We demonstrate with extensive experiments on various image classification tasks that our results significantly surpass the previous state-of-the-art baselines in both homogeneous and heterogeneous settings.

Original: Furthermore, thanks to our effective and selective protocol, our framework improves the robustness of FL on various malicious attacks and significantly reduces the overall communication.
Revised: Furthermore, thanks to our effective and selective protocol, our framework improves the robustness of FL on various malicious attacks and significantly reduces the overall communication.

Original: In summary, we propose a new framework to bridge FL with larger server models and simultaneously consolidate several benefits altogether, including superior performance, robustness of the whole system, and lower communication cost.
Revised: In summary, we propose a new framework to bridge FL with larger server models and simultaneously consolidate several benefits altogether, including superior performance, robustness of the whole system, and lower communication cost.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Federated Learning with GEM.
Revised: Federated Learning with GEM.

Original: FL is a collaborating learning framework without sharing private data among the clients.
Revised: FL is a collaborating learning framework without sharing private data among the clients.

Original: The classical method FedAvg (McMahan et al., 2017) and its recent variations (Mohri et al., 2019; Lin et al., 2018; Li et al., 2019b) directly transfer the clients’ parameters or gradients to the server nodes.
Revised: The classical method FedAvg (McMahan et al., 2017) and its recent variations (Mohri et al., 2019; Lin et al., 2018; Li et al., 2019b) directly transfer the clients’ parameters or gradients to the server nodes.

Original: To tackle the performance bottleneck by training resource-constrained clients in FL, there are two lines of work to bridge FL with GEM.
Revised: To tackle the performance bottleneck by training resource-constrained clients in FL, there are two lines of work to bridge FL with GEM.

Original: The first line of studies adopts model compression (Han et al., 2015;
Revised: The first line of studies adopts model compression (Han et al., 2015;

Original: He et al., 2018; Yang et al., 2018), manually designed architectures (Howard et al., 2017; Zhang et al., 2018; Iandola et al., 2016) or even efficient neural architecture search (Tan & Le, 2019; Wu et al., 2019) to adapt a GEM to on-device learning.
Revised: He et al., 2018; Yang et al., 2018), manually designed architectures (Howard et al., 2017; Zhang et al., 2018; Iandola et al., 2016) or even efficient neural architecture search (Tan & Le, 2019; Wu et al., 2019) to adapt a GEM to on-device learning.

Original: Another line is adopting knowledge distillation (Hinton et al., 2015) to transfer knowledge through output logits rather than parameters between a client model and a GEM (He et al., 2020a).
Revised: Another line is adopting knowledge distillation (Hinton et al., 2015) to transfer knowledge through output logits rather than parameters between a client model and a GEM (He et al., 2020a).

Original: However
Revised: However,

Original: FedGKT (He et al., 2020a)’s focus is to transfer knowledge directly from clients to server without considering the consensus knowledge fused from clients.
Revised: FedGKT (He et al., 2020a)’s focus is to transfer knowledge directly from clients to server without considering the consensus knowledge fused from clients.

Original: Therefore its performance is limited.
Revised: Therefore its performance is limited.

Original: Federated Learning with Knowledge Distillation.
Revised: Federated Learning with Knowledge Distillation.

Original: In fact, ensemble knowledge distillation has been shown to boost collaborative performance in FL.
Revised: In fact, ensemble knowledge distillation has been shown to boost collaborative performance in FL.

Original: Specifically, FedMD (Li & Wang, 2019) adopts a labeled public dataset and averaged logits to transfer knowledge.
Revised: Specifically, FedMD (Li & Wang, 2019) adopts a labeled public dataset and averaged logits to transfer knowledge.

Original: FedDF (Lin et al., 2020) proposes ensemble distillation for model fusion by aggregating both logits and models from clients.
Revised: FedDF (Lin et al., 2020) proposes ensemble distillation for model fusion by aggregating both logits and models from clients.

Original: In addition, KD is used to enhance robustness in FL.
Revised: In addition, KD is used to enhance robustness in FL.

Original: Cronus (Chang et al., 2019) and DS-FL (Itahara et al., 2020) utilizes a public dataset with soft labels jointly with local private dataset for local training, and combines with Cronus or entropy reduction aggregation, respectively, to defend against poisoning attacks in FL.
Revised: Cronus (Chang et al., 2019) and DS-FL (Itahara et al., 2020) utilize a public dataset with soft labels jointly with local private dataset for local training, and combine with Cronus or entropy reduction aggregation, respectively, to defend against poisoning attacks in FL.

Original: In this work, we propose to exploit the benefit of both a larger server model and client knowledge fusion.
Revised: In this work, we propose to exploit the benefit of both a larger server model and client knowledge fusion.

Original: Our work is also motivated by the recent studies in KD (Qin et al., 2021;
Revised: Our work is also motivated by the recent studies in KD (Qin et al., 2021;

Original: You et al., 2017; Li et al., 2021; Yuan et al., 2021; Wang et al., 2021), which show that student models can have larger capacities by learning from multiple teacher models. 
Revised: You et al., 2017; Li et al., 2021; Yuan et al., 2021; Wang et al., 2021), which show that student models can have larger capacities by learning from multiple teacher models. [Table]

Original:  M ETHODOLOGY 3. P RELIMINARIES
Revised:  M ETHODOLOGY 3. P RELIMINARIES

Original: We assume that there are K clients in federated learning process.
Revised: We assume that there are K clients in federated learning process.

Original: The k th client has its own private labeled dataset X k :=
Revised: The k th client has its own private labeled dataset X k :=

Original: { ( x k i , y k i ) } N k i =1 that can be drawn from the same or different distribution,[Table] where x ki is the i th training sample in the k th client model, y i is its corresponding ground truth label, and N k denotes the total number of samples.
Revised: { ( x ki , y ki ) } N k i =1 that can be drawn from the same or different distribution, where x ki is the i th training sample in the k th client model, y ki is its corresponding ground truth label, and N k denotes the total number of samples.

Original: Each client also trains its own model f k with parameters W k which can be of the same architecture ( homogeneous ) or different architecture ( heterogeneous ).
Revised: Each client also trains its own model f kc which can be of the same architecture ( homogeneous ) or different architecture ( heterogeneous ).

Original: There is also a public dataset X 0 :
Revised: There is also a public dataset X 0 :

Original: =
Revised: =

Original: { ( x 0 i , y 0 i ) } N 0 i =1 which is accessible to both server and clients.
Revised: { ( x 0 i , y 0 i ) } N 0 i =1 which is accessible to both server and clients.

Original: On the server side, we assume that there is a larger server model to be trained, denoted as f 0 with parameter W 0 .
Revised: On the server side, we assume that there is a larger server model to be trained, denoted as f s .

Revised: L s and L kc denotes the logit tensors from the server and the k th client model.

Original: 3. F ED GEMS F RAMEWORK
Revised: 3. F ED GEMS F RAMEWORK

Original: We illustrate our overall framework in Fig.
Revised: We illustrate our overall framework in Fig.

Original: 1 and summarize our training algorithm in Algorithm 1.
Revised: 1 and summarize our training algorithm in Algorithm 1.

Original: FedGEM.
Revised: FedGEM.

Original: During each communication round, all client models first use private datasets to train several epochs, then transfer the predicted logits on public dataset as knowledge to the server model.
Revised: During each communication round, all client models first use private datasets to train several epochs, then transfer the predicted logits on public dataset as knowledge to the server model.

Original: The server model aggregates the clients’ logits and then trains its server model with the guidance of fused knowledge.
Revised: The server model aggregates the clients’ logits and then trains its server model with the guidance of fused knowledge.

Original: After training, the server model then transfers its logits back to all client models.
Revised: After training, the server model then transfers its logits back to all client models.

Original: Finally, each client model distills knowledge from received logits and continues their training on private datasets.
Revised: Finally, each client model distills knowledge from received logits and continues their training on private datasets.

Original: Continuously iterating over multiple rounds, both the server and client models mutually learn knowledge from each other.
Revised: Continuously iterating over multiple rounds, both the server and client models mutually learn knowledge from each other.

Original: Through this alternating training processing, we can obtain a large server model with accumulated knowledge and an ensemble of high-performance client models.
Revised: Through this alternating training processing, we can obtain a large server model with accumulated knowledge and an ensemble of high-performance client models.

Original: FedGEMS : In FedGEMS, the server adopts a selection and weighting criterion to select knowledgeable clients for aggregation, which is detailed in the next section 3.3.
Revised: FedGEMS : In FedGEMS, the server adopts a selection and weighting criterion to select knowledgeable clients for aggregation, which is detailed in the next section 3.3.

Original: To illustrate the features of our framework, we compare it with the related studies of KD-based methods in federated learning in Table 1 on the following aspects: whether they use a labeled, unlabeled or no public dataset, whether client models can have heterogeneous architectures, the aggregation strategy on the server, and whether the server has a larger model.
Revised: To illustrate the features of our framework, we compare it with the related studies of KD-based methods in federated learning in Table 1 on the following aspects: whether they use a labeled, unlabeled or no public dataset, whether client models can have heterogeneous architectures, the aggregation strategy on the server, and whether the server has a larger model.

Original: Note FedGEM can be regarded as placing a larger server model on top of the FedMD framework, while keeping other settings the same.
Revised: Note FedGEM can be regarded as placing a larger server model on top of the FedMD framework while keeping other settings the same.

Original: 3. S ELECTIVE K NOWLEDGE F USION IN S ERVER M ODEL
Revised: 3. S ELECTIVE K NOWLEDGE F USION IN S ERVER M ODEL

Original: Since clients’ knowledge may negatively impact the server model in the heterogeneous or malicious setting, and vice versa, we further propose selective strategies on both server and client sides to enforce positive knowledge fusion into the large server model as shown in Fig.
Revised: Since clients’ knowledge may negatively impact the server model in the heterogeneous or malicious setting and vice versa, we further propose selective strategies on both server and client sides to enforce positive knowledge fusion into the large server model as shown in Fig.

Original: 3.3. S ERVER -S IDE S ELF -D ISTILLATION

Original: At each iteration, the server model first performs self evaluation on the public dataset and split the samples into two classes, those it can predict correctly, S Correct , and those it predicts wrongly,

Original: S Incorrect .

Original: For each sample x i in S Correct where the model prediction matches the ground truth label, we simply adopt the cross-entropy loss between the predicted values and the ground truth labels to train the server model. [Equation] 1: ServerExecute() : 2: for each round t = 1 , 2 , ..., T do 3: // Selective Knowldge Fusion 4: for idx, x 0 , y 0 ∈ { X 0 , Y 0 } do 5: if f s ( W s ; x 0 ) =
Revised: do 19: do 21: do 25:

Original: = y 0 then 6: L s ← L S 1 ( x 0 , y 0 ) 
Revised: L c ← L CE ( x k , y k ) 26:

Original: ▷ in Eq.7: L Global [ idx ] ← L s [ idx ] 8: else if idx in L Global then 9: L s ← L S 2 ( x 0 , y 0 , L Global )
Revised: W kc ← W kc − η k ∇L c 27: ClientSelect (idx): 28: //

Original: ▷ in Eq. 2 10: else 11:
Revised: Selective Transfer to Server 29: for each client k th in parallel do 30:

Original: L  c ← ClientSelect ( idx ) 12: L s ← L S 3 (  x 0 , y 0 , L c )
Revised: L 0 c [ idx ] ← f kc ( W kc ; x 0 [ idx ]) 31: Return L c to server

Original: ▷ in Eq. W s ← W s − η k ∇L s 14: L s [ idx ] ← f s ( W s ; x 0 ) 15: // Transfer Knowledge to Client Models 16: ClientTrain ( L s ) 17: ClientTrain ( L s ): 18: for each client k th in parallel do 19: //
Revised: 3.3. S ELF -D ISTILLATION OF S ERVER K NOWLEDGE

Original: Knowledge Distillation on Clients 20: for x 0 , y 0 ∈ { X 0 , L s , Y 0 } do 21: L c ← L C ( x 0 , y 0 , L s ) ▷ in Eq.22: W kc ← W kc − η k ∇L c 23: //
Revised: At each iteration, the server model first performs self evaluation on the public dataset and split the samples into two classes, those it can predict correctly, S Correct , and those it predicts wrongly,

Original: Local Training on Clients 24: for x k , y k ∈ { X k , Y k } do 25: L c ← L CE ( x k , y k ) 26: W kc ← W kc − η k ∇L c 27: ClientSelect (idx): 28: //
Revised: S Incorrect .

Original: Transfer to Server 29: for each client k th in parallel do 30: L 0 c [ idx ] ← f kc ( W kc ; x 0 [ idx ]) 31: Return L c to  server 
Revised: For each sample x i in S Correct where the model prediction matches the ground truth label, we simply adopt the cross-entropy loss between the predicted values and the ground truth labels to train the server model. [Equation]

Original: At the same time, we save the correct logit l is into the global pool of logits as l iGlobal , which can be further used as memory to recover its reserved knowledge from self-distillation.
Revised: At the same time, we save the correct logit l is into the global pool of logits as l iGlobal , which can be further used as memory to recover its reserved knowledge from self-distillation.

Original: For each sample x i that the server predicts wrongly, we first check whether its corresponding global logit l iGlobal exists or not.
Revised: For each sample x i that the server predicts wrongly, we first check whether its corresponding global logit l iGlobal exists or not.

Original: We denote all samples that do not exist in l Global as S ∗ Incorrect , for which we believe the server model can not learn the sample entirely by itself and propose to use clients’ collective knowledge as its teacher, which will be explained in the next section.
Revised: We denote all samples that do not exist in l Global as S ∗ Incorrect , for which we believe the server model can not learn the sample entirely by itself and propose to use clients’ collective knowledge as its teacher, which will be explained in the next section.

Original: If l iGlobal exists, it means that the knowledge was reserved by the server before, the server model performs selfdistillation to recover this part of knowledge.
Revised: If l iGlobal exists, it means that the knowledge was reserved by the server before, the server model performs selfdistillation to recover this part of knowledge.

Original: The final training objective of self-distillation can be formulated as follows, where D KL is the Kullback Leibler (KL) Divergence function.
Revised: The final training objective of self-distillation can be formulated as follows, where D KL is the Kullback Leibler (KL) Divergence function and ϵ is a hyper-parameter to control the weight of knowledge distillation in all the following formulations.

Original: [Equation]
Revised: [Equation]

Original: Our self-distillation strategy has at least two advantages.
Revised: Our self-distillation strategy has at least two advantages.

Original: On the one hand, distilling knowledge from itself is better than from other models which have different architectures.
Revised: On the one hand, distilling knowledge from itself is better than from other models which have different architectures.

Original: On the other hand, learning from itsown stored logits can greatly reduce the communication cost as compared to transferring all redundant logits to client models.
Revised: On the other hand, learning from its stored logits can greatly reduce the communication cost as compared to transferring all redundant logits to client models.

Original: 3.3.C LIENT -S IDE S ELECTIVE E NSEMBLE D ISTILLATION 
Revised: 3.3. S ELECTIVE E NSEMBLE D ISTILLATION OF C LIENT K NOWLEDGE

Original: For those samples in S ∗ Incorrect that the server model fails to predict correctly, we try to distill knowledge from the ensemble of clients.
Revised: For those samples in S ∗ Incorrect that the server model fails to predict correctly, we try to distill knowledge from the ensemble of clients.

Original: Furthermore, considering the correctness and relative importance of client models, we propose a weighted selective strategy on client side.
Revised: Furthermore, considering the correctness and relative importance of client models, we propose a weighted selective strategy on client side.

Original: Given an instance ( x i , y i ) in S ∗ Incorrect , we first split all clients { C 1 , C 2 , ..., C K } into the reliable and unreliable clients according to their predictions p C j .
Revised: Given an instance ( x i , y i ) in S ∗ Incorrect , we first split all clients { C 1 , C 2 , ..., C K } into the reliable and unreliable clients according to their predictions p C j .

Original: For those clients who predict the wrong labels, we consider them as unreliable and discard their knowledge by setting their weights equal to 0 .
Revised: For those clients who predict the wrong labels, we consider them as unreliable and discard their knowledge by setting their weights equal to 0 .

Original: As for the rest of clients who predict the labels correctly, we consider them as reliable and use their entropy H ( p C j ) as a measure of the confidence.
Revised: As for the rest of clients who predict the labels correctly, we consider them as reliable and use their entropy H ( p C j ) as a measure of the confidence.

Original: [Equation]
Revised: [Equation]

Original: Following previous work (Li et al., 2021; Pereyra et al., 2017; Szegedy et al., 2016), low entropy indicates high confidence, and vice versa.
Revised: Following previous work (Li et al., 2021; Pereyra et al., 2017; Szegedy et al., 2016), low entropy indicates high confidence, and vice versa.

Original: Specifically, given an instance ( x i , y i ) in S ∗ Incorrect , we design its corresponding weights to aggregate output logits from different clients as below.
Revised: Specifically, given an instance ( x i , y i ) in S ∗ Incorrect , we design its corresponding weights to aggregate output logits from different clients as below.

Original: [Equation]
Revised: [Equation]

Original: Therefore, for samples in S ∗ Incorrect , the knowledge transferred from ensemble clients to server model can be formulated as the following.
Revised: Therefore, for samples in S ∗ Incorrect , the knowledge transferred from ensemble clients to server model can be formulated as the following.

Original: [Equation]
Revised: [Equation]

Original: Up to now, we have covered the selective knowledge fusion strategy for the server, next we will discuss the knowledge distillation on clients.
Revised: Up to now, we have covered the selective knowledge fusion strategy for the server, next we will discuss the knowledge distillation on clients.

Original: 3. T RAINING IN C LIENT M ODELS
Revised: 3. T RAINING IN C LIENT M ODELS

Original: Each client model first receives the logits l iServer of public dataset from the server model.
Revised: Each client model first receives the logits l iServer of public dataset from the server model.

Original: Then it distills knowledge according to the logits as well as computes the cross-entropy loss to train on public dataset.
Revised: Then it distills knowledge according to the logits as well as computes the cross-entropy loss to train on public dataset.

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: After knowledge distillation, each client model further adopts the cross-entropy loss to train on its local private dataset to better fit into its target distribution.
Revised: After knowledge distillation, each client model further adopts the cross-entropy loss to train on its local private dataset to better fit into its target distribution.

Original:  E XPERIMENTAL E VALUATIONS 4. E XPERIMENT S ETTINGS
Revised:  E XPERIMENTAL E VALUATIONS 4. E XPERIMENT S ETTINGS

Original: Task and Dataset.
Revised: Task and Dataset.

Original: For fair comparison, we use the same training tasks as He et al.
Revised: For fair comparison, we use the same training tasks as He et al.

Original: (2020a), which include image classifications on the CIFAR-10 (Krizhevsky et al., 2009) and
Revised: (2020a), which include image classifications on the CIFAR-10 (Krizhevsky et al., 2009) and

Original: CIFAR- (Krizhevsky et al., 2009) datasets.
Revised: CIFAR- (Krizhevsky et al., 2009) datasets.

Original: More details about these two dataset can be found in Appendix
Revised: More details about these two datasets can be found in Appendix

Original: B.1.
Revised: B.1.

Original: For each dataset, we randomly split data into a public and a private dataset with a ratio of 1:1.
Revised: For each dataset, we randomly split data into a public and a private dataset with a ratio of 1:1.

Original: We further study the impact of this ratio in Sec.
Revised: We further study the impact of this ratio in Sec.

Original: 5.2 of our experiments.
Revised: 5.2 of our experiments.

Original: The public dataset is used for transferring knowledge between server node and client nodes, while the private dataset is for client training.
Revised: The public dataset is used for transferring knowledge between server node and client nodes, while the private dataset is for client training.

Original: Both of them further split into a training and a testing dataset with a ratio of 5:1.
Revised: Both of them further split into a training and a testing dataset with a ratio of 5:1.

Original: Homogeneous Setting.
Revised: Homogeneous Setting.

Original: In our main experiments, we randomly shuffle and partition private dataset into 16 clients.
Revised: In our main experiments, we randomly shuffle and partition private dataset into 16 clients.

Original: As shown in Appendix B.5, our client models all adopt a tiny CNN architecture called ResNet-11, while the server model architecture is ResNet-56.
Revised: As shown in Appendix B.6, our client models all adopt a tiny CNN architecture called ResNet-11, while the server model architecture is ResNet-56.

Original: To investigate the influence of server model size and client number, we further vary the server model size from ResNet-20 to
Revised: To investigate the influence of server model size and client number, we further vary the server model size from ResNet-20 to

Original: ResNet110 and the total client number from 4 to 32 in Sec.
Revised: ResNet110 and the total client number from 4 to 64 in Sec.

Original: 5.3 and Sec. 5.4, respectively.
Revised: 5.3 and Sec. 5.4, respectively.

Original: Heterogeneous Setting.
Revised: Heterogeneous Setting.

Original: We adopt the Dirichlet distribution (Yurochkin et al., 2019; Hsu et al., 2019) to control the degree of heterogeneity in our non-iid setting.
Revised: We adopt the Dirichlet distribution (Yurochkin et al., 2019; Hsu et al., 2019) to control the degree of heterogeneity in our non-iid setting.

Original: In this experiment, our α is 0 .
Revised: In this experiment, our α is 0 .

Original: We also introduce model heterogeneity in our experiments.
Revised: We also introduce model heterogeneity in our experiments.

Original: The client models vary from ResNetto Resnet-17 as shown in Appendix B.6.
Revised: The client models vary from ResNetto Resnet-17 as shown in Appendix B.7.

Original: Implementation Details.
Revised: Implementation Details.

Original: In both homogeneous and heterogeneous settings, the number of local epochs are set to 1 after careful tuning, consistent with implementations in FedGKT.
Revised: In both homogeneous and heterogeneous settings, the number of local epochs are set to 1 after careful tuning, consistent with implementations in FedGKT.

Original: The number of communication round to reach convergence is 400.
Revised: The number of communication round to reach convergence is 400.

Original: We adopt Adam optimizer (Kingma & Ba, 2014) with  initial learning rate 0.001.
Revised: We adopt Adam optimizer (Kingma & Ba, 2014) with an initial learning rate 0.001.

Original: Following He et al.
Revised: Following He et al.

Original: (2020a), the learning rate can reduce once thethe accuracy is stable (Li & Arora, 2019).
Revised: (2020a), the learning rate can reduce once the accuracy is stable (Li & Arora, 2019).

Original: As for the factor ϵ , we find that the best choice is 0.75.
Revised: As for the factor ϵ , we find that the best choice is 0.75.

Original: Its impacts on server and client performance are shown in Appendix B.4 .
Revised: Its impacts on server and client performance are shown in Appendix B.5 .

Original: Baselines.
Revised: Baselines.

Original: We compare our framework FedGEM/FedGEMS with the following baselines.
Revised: We compare our framework FedGEM/FedGEMS with the following baselines.

Original: Stand-Alone : the server and client models train on their local datasets X 0 and X k , respectively; (2)
Revised: Stand-Alone : the server and client models train on their local datasets X 0 and X k , respectively; (2)

Original: Centralized : the client model trains on the whole private dataset { X 1 + · · · + X k } ; (3) CentralizedAll : the server and clients perform centralized training on the whole dataset { X 0 + · · · + X k } , respectively.
Revised: Centralized : the client model trains on the whole private dataset { X 1 + · · · + X k } ; (3) CentralizedAll : the server and clients perform centralized training on the whole dataset { X 0 + · · · + X k } , respectively.

Original: Note in reality no party has this complete dataset so this can be viewed as an upper limit of model performance.
Revised: Note in reality no party has this complete dataset so this can be viewed as an upper limit of model performance.

Original: In addition, we also compare our performance with several existing related works, including FedAvg (McMahan et al., 2017), FedMD (Li & Wang, 2019), Cronus (Chang et al., 2019), DS-FL (Itahara et al., 2020), FedDF (Lin et al., 2020) and FedGKT (He et al., 2020a).
Revised: In addition, we also compare our performance with several existing related works, including FedAvg (McMahan et al., 2017), FedMD (Li & Wang, 2019), Cronus (Chang et al., 2019), DS-FL (Itahara et al., 2020), FedDF (Lin et al., 2020) and FedGKT (He et al., 2020a).

Original: For fair comparison, the settings of client models and the split of public and private dataset in all approaches are kept the same.
Revised: For fair comparison, the settings of client models and the split of public and private dataset in all approaches are kept the same.

Original: As for the server model, FedDF adopts the same architecture as its clients’ model, ResNet-11, according to itsown design, while FedGKT and our FedGEMS utilize
Revised: As for the server model, FedDF adopts the same architecture as its clients’ model, ResNet-11, according to its design, while FedGKT and our FedGEMS utilize

Original: ResNet-56.
Revised: ResNet-56.

Original: Note FedGKT
Revised: Note FedGKT

Original: (He et al., 2020a) only reported its performance of the server model.
Revised: (He et al., 2020a) only reported the performance of the server model.

Original: We empirically found that its clients’ performance are lower due to constraints of model size.
Revised: We empirically found that its clients’ performance is lower due to constraints of model size.

Original: For  fair comparison with FedGKT, we only reported its performance on server side.
Revised: For a fair comparison with FedGKT, we only reported its performance on server side.

Original: More details of their important hyper-parameters in our experiments are listed in Appendixes B.8, respectively [Table]
Revised: More details of their important hyper-parameters in our experiments are listed in Appendixes B.9, respectively.

Original: 4. P ERFORMANCE E VALUATIONS
Revised: 4. P ERFORMANCE E VALUATIONS

Original: [Table]
Revised: [Table]

Original: Table 2 shows the comparisons of our approaches with other baselines with ResNet-56 as the server model.
Revised: 2 shows the comparisons of our approaches with other baselines with ResNet-56 as the server model.

Original: First of all, our approach FedGEM outperforms all the above KD-based federated learning baselines significantly on both server and client side, demonstrating the effectiveness of knowledge transfer with a larger server model.
Revised: First of all, our approach FedGEM outperforms all the above KD-based federated learning baselines significantly on both server and client side, demonstrating the effectiveness of knowledge transfer with a larger server model.

Original: Compared with FedGKT which also adopts a larger server model, our performance is significantly improved by knowledge fusion from multiple clients.
Revised: Compared with FedGKT which also adopts a larger server model, our performance is significantly improved by knowledge fusion from multiple clients.

Original: Comparing to the performance of stand-alone server and clients, our framework simultaneously improves both the sever model and client model, indicating that server and clients mutually benefit from knowledge transfer.
Revised: Compared to the performance of stand-alone server and clients, our framework simultaneously improves both the sever model and client model, indicating that server and clients mutually benefit from knowledge transfer.

Original: Using public labels for supervision and selection, our FedGEMS framework is able to enhance positive knowledge fusion and further improve performance on both server and clients.
Revised: Using public labels for supervision and selection, our FedGEMS framework is able to enhance positive knowledge fusion and further improve performance on both server and clients.

Original: We also conduct experiments with 8 clients and ResNet-20 as server model, shown in Appendix B.3.
Revised: We also conduct experiments with 8 clients and ResNet-20 as server model, shown in Appendix B.4.

Original: 4. R OBUSTNESS
Revised: 4. R OBUSTNESS

Original: Poisoning Attacks.
Revised: Poisoning Attacks.

Original: Following Chang et al.
Revised: Following Chang et al.

Original: (2019), we employ model poisoning attacks to evaluate the robustness of our methods.
Revised: [Table] we employ model poisoning attacks to evaluate the robustness of our methods.

Original: Our model poisoning attacks include Naive Poisioning (PAF), Little
Revised: Our model poisoning attacks include Naive Poisioning (PAF), Little

Original: Is Enough Attack (LIE) (Baruch et al., 2019) and OFOM (Chang et al., 2019).
Revised: Is Enough Attack (LIE) (Baruch et al., 2019) and OFOM (Chang et al., 2019).

Original: In PAF and LIE, the attacker poisons one client at each round via disturbing the logits or parameters which transfer from clients to server in different methods.
Revised: In PAF and LIE, the attacker poisons one client at each round via disturbing the logits or parameters which transfer from clients to server.

Original: In OFOM, two clients’ benign predictions or parameters are poisoned at each round.
Revised: In OFOM, two clients’ benign predictions or parameters are poisoned at each round.

Original: The detailed algorithms about different poisoning attacks can be found in Appendix A
Revised: The detailed algorithms about different poisoning attacks can be found in Appendix A

Original: [Table]
Revised: [Table]

Original: The results of poisoning attacks in heterogeneous setting on CIFAR-10 dataset are shown in Table 3.
Revised: The results of poisoning attacks in heterogeneous setting on CIFAR-10 dataset are shown in Table 3.

Original: By adopting a selective strategy, our proposed FedGEMS provides consistent robustness across various attacks on both server and client models.
Revised: By adopting a selective strategy, our proposed FedGEMS provides consistent robustness across various attacks on both server and client models.

Original: FedGEM demonstrates superior robustness to FedMD on various model poisoning attacks.
Revised: FedGEM demonstrates superior robustness to FedMD on various model poisoning attacks.

Original: DS-FL is relatively more robust than FedMD via its proposed entropy reduction aggregation based on FedMD framework.
Revised: DS-FL is relatively more robust than FedMD via its proposed entropy reduction aggregation based on FedMD framework.

Original: Since Cronus specifically designs secure robust aggregation in a FedMD framework, its robustness to the attacks are strong, as expected.
Revised: Since Cronus specifically designs secure robust aggregation in a FedMD framework, its robustness to the attacks is strong, as expected.

Revised: The performance of FedGKT drops marginally because it transmits extra feature maps which is vulnerable.

Original: Our selective knowledge transfer strategy shows comparably strong robustness on both clients and server side without additional computation cost for robust aggregation algorithms.
Revised: Our selective knowledge transfer strategy shows comparably strong robustness on both clients and server side without additional computation cost for robust aggregation algorithms.

Original: 4. C OMMUNICATION C OST
Revised: 4. C OMMUNICATION C OST

Original: Following Itahara et al.
Revised: Following Itahara et al.

Original: (2020), we further evaluate the cumulative communication costs required to achieve specific accuracy in heterogeneous setting on CIFAR-10.
Revised: (2020), we further evaluate the cumulative communication costs required to achieve specific accuracy in heterogeneous setting on CIFAR-10.

Revised: The epochs in both client and server sides among all methods to achieve their best performances are the same as 1 which further ensures fairness.

Original: The formulations to calculate communication cost for different methods are shown in Appendix B.2.
Revised: The formulations to calculate communication cost for different methods are shown in Appendix B.2.

Original: [Table]
Revised: [Table]

Original: The results are shown in Table 4.
Revised: The results are shown in Table 4.

Original: It is worth noting that FedMD, Cronus, DS-FL and FedGEM cost the same per round according to their designs.
Revised: It is worth noting that FedMD, Cronus, DS-FL and FedGEM cost the same per round according to their designs.

Original: Therefore, to achieve the accuracy of 45%,
Revised: Therefore, to achieve the accuracy of 45%,

Original: FedGEM costs much lower than other three methods which indicates that FedGEM converges fast.
Revised: FedGEM costs much lower than other three methods which indicates that FedGEM converges fast.

Original: The communication cost of FedGKT is much higher than others because it sends a feature map per private data in training.
Revised: The communication cost of FedGKT is much higher than others because it sends a feature map per private data in training.

Original: Specifically, the communication cost per feature map is 64kb while the costof per logit is only 0.039kb.
Revised: Specifically, the communication cost per feature map is 64kb while the cost per logit is only 0.039kb.

Original: Furthermore, the communication cost of FedGEMS keeps lower than
Revised: Furthermore, the communication cost of FedGEMS keeps lower than

Original: FedGEM in the whole training phase which means that our selective strategy can effectively reduce uploads to achieve the same performance.
Revised: FedGEM in the whole training phase which means that our selective strategy can effectively reduce uploads to achieve the same performance.

Original:  U NDERSTANDING F ED GEMS 5. K NOWLEDGE A CCUMULATION AT S ERVER
Revised:  U NDERSTANDING F ED GEMS 5. K NOWLEDGE A CCUMULATION AT S ERVER

Original: To analyze the selective knowledge fusion process of the server model in our framework, we report the number of samples associated with each step in our decision-making process in our experi-

Original: [Table]
Revised: [Table]

Original: ment on CIFAR-10 with 25,000 samples as public dataset.
Revised: To analyze the selective knowledge fusion process of the server model in our framework, we report the number of samples associated with each step in our decision-making process in our experiment on CIFAR-10 with 25,000 samples as public dataset.

Original: Each sample will choose one of three strategies according to our selective knowledge fusion module.
Revised: Each sample will choose one of three strategies according to our selective knowledge fusion module.

Original: Specifically, in Fig.
Revised: Specifically, in Fig.

Original: 3, the line of self-training indicates the total number of samples the server model predicts correctly, while the lines of self-distillation and ensemble-distillation indicate the number of samples the server learns via self-distillation and client-side ensemble knowledge, respectively.
Revised: 3, the line of self-training indicates the total number of samples the server model predicts correctly, while the lines of self-distillation and ensemble-distillation indicate the number of samples the server learns via self-distillation and client-side ensemble knowledge, respectively.

Original: In the initial stage, since both the server and client models learn from scratch, the number of samples that the server model can predict correctly is limited, and most of the knowledge is accumulated by distillation from the client models’ fusion.
Revised: In the initial stage, since both the server and client models learn from scratch, the number of samples that the server model can predict correctly is limited, and most of the knowledge is accumulated by distillation from the client models’ fusion.

Original: As the training progresses, the server model needs to restore some of its knowledge from a self-distillation strategy.
Revised: As the training progresses, the server model needs to restore some of its knowledge from a self-distillation strategy.

Original: With the server model continually fusing sufficient knowledge, the model performance of server model eventually exceeds client models and the number of transferred samples from clients dropped to 0.
Revised: With the server model continually fusing sufficient knowledge, the model performance of server model eventually exceeds client models and the number of transferred samples from clients dropped to 0.

Original: Notice that in the final stage, there still remains some stubborn samples which are hard for the server model while most samples can be solved by the large server model with accumulated knowledge.
Revised: Notice that in the final stage, there still remains some stubborn samples which are hard for the server model while most samples can be solved by the large server model with accumulated knowledge.

Original: Furthermore, we conduct a series of ablation studies to detect the contributions of different components to the accumulated knowledge at server side.
Revised: Furthermore, we conduct a series of ablation studies to detect the contributions of different components to the accumulated knowledge at server side.

Original: The results are shown in Table 5 and all the performances of both server and client models decrease.
Revised: The results are shown in Table 5 and all the performances of both server and client models decrease.

Original: This phenomenon indicates that each component contributes to the overall performance.
Revised: This phenomenon indicates that each component contributes to the overall performance.

Original: Comprehensively, self-training is the most important component and the most probably reason is that learning from the model itself is the most effective way.
Revised: Comprehensively, self-training is the most important component and the most probable reason is that learning from the model itself is the most effective way.

Original: Ensemble distillation which avoids to transfer knowledge from malicious clients can also help prompt the performance of client models in return.
Revised: Ensemble distillation which avoids transferring knowledge from malicious clients can also help prompt the performance of client models in return.

Original: 5. E FFECT OF P UBLIC AND P RIVATE D ATASET R ATIO
Revised: 5. E FFECT OF P UBLIC AND P RIVATE D ATASET R ATIO

Original: Concerning the size of public dataset, we evaluate the effect of public and private dataset ratio on both model performance and communication cost per round.
Revised: Concerning the size of public dataset, we evaluate the effect of public and private dataset ratio on both model performance and communication cost per round.

Original: To change the ratio, we fix the size of private dataset, and  the size of public datasetvaries from 5,000 to 25,000.
Revised: To change the ratio, we fix the size of private dataset, and vary the size of public dataset from 5,000 to 25,000.

Original: The results of both model performance and communication cost per round are shown in Fig.
Revised: The results of both model performance and communication cost per round are shown in Fig.

Revised: 4 (a) and (b) respectively.

Original: As for model performance, it can be seen that both the client and server model continuously improve as the size of public data increases, indicating that their performances are highly correlated.
Revised: As for model performance, it can be seen that both the client and server model continuously improve as the size of public data increases, indicating that their performances are highly correlated.

Original: Our performance consistently outperforms FedMD, and the performance gap over FedMD continuously grows as more public dataset is available.
Revised: Our performance consistently outperforms FedMD, and the performance gap over FedMD continuously grows as more public dataset is available.

Original: Note again FedGEM is essentially FedMD with a larger server model so the performance gain is due to the accumulated knowledge on the server side.
Revised: Note again FedGEM is essentially FedMD with a larger server model so the performance gain is due to the accumulated knowledge on the server side.

Original: As for FedDF which owns an ensemble server model as same architecture as clients, the performance in both server and client models remains a huge gap to our performance.
Revised: As for FedDF which owns an ensemble server model as same architecture as clients, the performance in both server and client models remains a huge gap to our performance.

Original: This phenomenon further demonstrates the importance of the large capacity of server model.
Revised: This phenomenon further demonstrates the importance of the large capacity of server model.

Original: FedGEMS can still outperform FedGEM especially in the client side which indicates that our selective strategy is efficiently to fuse profitable knowledge to boost performance.
Revised: FedGEMS can still outperform FedGEM especially on the client side which indicates that our selective strategy is efficient to fuse positive knowledge to boost performance.

Original: Due to the compact models deployed in client nodes, the bits of model parameters are limited, thus the communication overhead of FedAvg and FedDF is not always higher than typical KD-based methods when a large public dataset is used.
Revised: Due to the compact models deployed in client nodes, the bits of model parameters are limited, thus the communication overhead of FedAvg and FedDF is not always higher than typical KD-based methods when a large public dataset is used.

Revised: Due to the enormous communication overhead of feature maps, we do not figure the flat line (irrelevance to the size of public dataset) of FedGKT which is always 1.6M (kb) per round.

Original: Thanks to our selection strategy, the communication cost of our proposed framework FedGEMS is the lowest among all methods.
Revised: Thanks to our selection strategy, the communication cost of our proposed framework FedGEMS is the lowest among all methods.

Original: Since FedGKT also communicates feature maps of 32,768 bits according to its implementation, its communication overhead is much higher than other approaches and is not included in our results.

Original: In summary, the results demonstrate that our selection strategy can greatly save the communication cost witha reasonable public data size.
Revised: In summary, the results demonstrate that our selection strategy can greatly save the communication cost with reasonable public data size.

Original: 5. E FFECT OF S ERVER M ODEL S IZE
Revised: 5. E FFECT OF S ERVER M ODEL S IZE

Original: We further investigate the influence of the server model size on the overall performance by gradually changing the server model from ResNet-20 to ResNet-110, while fixing the client models as ResNet11.
Revised: We further investigate the influence of the server model size on the overall performance by gradually changing the server model from ResNet-20 to ResNet-110, while fixing the client models as ResNet11.

Original: The details of the model parameters can be found in Appendix B.7.
Revised: The details of the model parameters can be found in Appendix B.8.

Original: The results are shown in Fig.
Revised: The results are shown in Fig.

Revised: 5 (a).

Original: It can be witnessed that the performance of both the server and client models improves as the server model becomes larger and deeper.
Revised: It can be witnessed that the performance of both the server and client models improves as the server model becomes larger and deeper.

Original: This phenomenon verifies the importance of placing a larger deeper server model to boost the performance of both server and the resource-constrained client models.
Revised: This phenomenon verifies the importance of placing a larger deeper server model to boost the performance of both server and the resource-constrained client models.

Original: 5. E FFECT OFC LIENT N UMBER 
Revised: 5. E FFECT OF N UMBER OF C LIENTS

Original: We select different total number of client models, varying from 4 to 32, to detect the stability of our proposed method FedGEMS.
Revised: We vary the number of clients from 4 to 64 of our proposed method FedGEMS while keeping the total number of private samples the same.

Original: As thetotal number ofclient models increases, the private data each client owns decreases.
Revised: Thus the number of private samples per client vary accordingly.

Original: The experimental results shown in Fig.
Revised: The experimental results shown in Fig.

Original: 5  indicate that with increased number of clients and less client data, the performance of our approaches is basically stable especially in client side benefiting from our larger server model setting with fused knowledge.
Revised: 5 (b) indicate that with increased number of clients and less client data, the performance of our approaches is basically stable especially in client side, benefiting from our larger server model setting with fused knowledge.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this work, we first propose a new paradigm to apply a large deeper server model to effectively and efficiently fuse and accumulate knowledge, which can enhance the model performances on both server and client sides.
Revised: In this work, we first propose a new paradigm to apply a large deeper server model to effectively and efficiently fuse and accumulate knowledge, which can enhance the model performances on both server and client sides.

Original: Furthermore, we design a selection and weighted criterion on both sides to distill only positive knowledge into the server.
Revised: Furthermore, we design a selection and weighted criterion on both sides to distill only positive knowledge into the server.

Original: We conduct a series of experiments to evaluate our proposed framework FedGEMS and our results show that FedGEMS can significantly surpass all baselines in both homogeneous and heterogeneous settings.
Revised: We conduct a series of experiments to evaluate our proposed framework FedGEMS and our results show that FedGEMS can significantly surpass all baselines in both homogeneous and heterogeneous settings.

Original: Meanwhile, FedGEMS can further improve the robustness of FL on poisoning attacks as well as reduce the communication costs between server and client sides.
Revised: Meanwhile, FedGEMS can further improve the robustness of FL on poisoning attacks as well as reduce the communication costs between server and client sides.

Original: Our framework has certain limitations, such as the dependence of a labeled public dataset.
Revised: Our framework has certain limitations, such as the dependence of a labeled public dataset.

Original: In future work, we will study the effectiveness of our work in other critical tasks, such as NLP and knowledge graph-based tasks, and in the settings where public and private data are from different domain or distributions.
Revised: In future work, we will study the effectiveness of our work in other critical tasks, such as NLP and knowledge graph-based tasks, and in the settings where public and private data are from different domains or distributions.

Original: A P OISONING A TTACKS IN F EDERATED L EARNING
Revised: A P OISONING A TTACKS IN F EDERATED L EARNING

Original: In this section, we introduce a set of poisoning attacks from the literature which used to evaluate the robustness in federated learning.
Revised: In this section, we introduce a set of poisoning attacks from the literature which used to evaluate the robustness in federated learning.

Original: A. N AIVE POISONING (PAF)
Revised: A. N AIVE POISONING (PAF)

Original: Naive poisoning is a type of model poisoning attack.
Revised: Naive poisoning is a type of model poisoning attack.

Original: The adversary is able to poison ϵ fraction of total client.
Revised: The adversary is able to poison ϵ fraction of total client.

Original: With the knowledge of the distribution of benign updates, the opponent can introduce malicious update θ m .
Revised: With the knowledge of the distribution of benign updates, the opponent can introduce malicious update θ m .

Original: This attack is far from the mean of the benign updates, obtained by adding a significantly large vector θ ′ to it:
Revised: This attack is far from the mean of the benign updates, obtained by adding a significantly large vector θ ′ to it:

Original: [Equation]
Revised: [Equation]

Original: Then, the server updates with the malicious update θ m and transfers it to all the clients.
Revised: Then, the server updates with the malicious update θ m and transfers it to all the clients.

Original: A. L ITTLE IS ENOUGH ATTACK (LIE)
Revised: A. L ITTLE IS ENOUGH ATTACK (LIE)

Original: LIE is a type of model poisoning attack proposed by Baruch et al.
Revised: LIE is a type of model poisoning attack proposed by Baruch et al.

Original: The detailed implementation of LIE is summarized in Algorithm 2.
Revised: The detailed implementation of LIE is summarized in Algorithm 2.

Original: The malicious update θ m is achieved by making small changes in many dimensions to a benign update.
Revised: The malicious update θ m is achieved by making small changes in many dimensions to a benign update.

Original: The malicious update is trained in server and then shared by each client.
Revised: The malicious update is trained in server and then shared by each client.

Original: Algorithm 2 Little is enough attack (LIE)
Revised: Algorithm 2 Little is enough attack (LIE)

Original: Baruch et al.
Revised: Baruch et al.

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: 4: for j ∈ [ d ] do 5: compute mean ( µ  benign updates.
Revised: 4: for j ∈ [ d ] do 5: compute mean ( µ deviation σ of benign updates.

Original: [Equation]
Revised: [Equation]

Original: 6: Output: θ m
Revised: 6: Output: θ m

Original: A. O NE FAR ONE MEAN (OFOM)
Revised: A. O NE FAR ONE MEAN (OFOM)

Original: This model poisoning attack, OFOM, is proposed by Chang et al.
Revised: This model poisoning attack, OFOM, is proposed by Chang et al.

Original: In this attack, two malicious updates will be added to the benign updates.
Revised: In this attack, two malicious updates will be added to the benign updates.

Original: First, the opponent adds a significantly large vector θ ′ to the mean of the benign updates to obtain θ 1 m .
Revised: First, the opponent adds a significantly large vector θ ′ to the mean of the benign updates to obtain θ 1 m .

Original: Then, the opponent crafts θ 2 m , which is the mean of the benign updates and θ 1 m .
Revised: Then, the opponent crafts θ 2 m , which is the mean of the benign updates and θ 1 m .

Original: [Equation]
Revised: [Equation]

Original: B E XTRA E XPERIMENTAL R ESULTS AND D ETAILS B. A S UMMARY OF D ATASET
Revised: B E XTRA E XPERIMENTAL R ESULTS AND D ETAILS B. A S UMMARY OF D ATASET

Original: Following He et al.
Revised: Following He et al.

Original: (2020a), our experiments utilize CIFAR-10 (Krizhevsky et al., 2009) and CIFAR100 (Krizhevsky et al., 2009) as our datasets.
Revised: (2020a), our experiments utilize CIFAR-10 (Krizhevsky et al., 2009) and CIFAR100 (Krizhevsky et al., 2009) as our datasets.

Original: CIFAR-10 consists 10 classes colour images, with 6000 images per class.
Revised: CIFAR-10 consists 10 classes colour images, with 6000 images per class.

Original: CIFAR-100 is a more challenging dataset with 100 subclasses that falls under 20 superclasses, e.g. baby, boy, girl, man and woman belong to people
Revised: CIFAR-100 is a more challenging dataset with 100 subclasses that falls under 20 superclasses, e.g. baby, boy, girl, man and woman belong to people

Original: . B. F ORMULATIONS OF C OMMUNICATION C OST
Revised: . B. F ORMULATIONS OF C OMMUNICATION C OST

Original: In the Fig.
Revised: In the Fig.

Original: 6, we list the formulations to compute communication costs between server and client models in different methods.
Revised: 6, we list the formulations to compute communication costs between server and client models in different methods.

Revised: B. C OMMUNICATION COST OF THE UPLOADING LOGITS

Revised: The results of the communication cost of the uploading logits per round are shown in Table 6.

Revised: [Table]

Original: B. R ESULTS OF 8 C LIENTS AND R ES N ET - 
Revised: B. R ESULTS OF 8 C LIENTS AND R ES N ET -

Original: We also conduct different setting to validate the stability of our proposed framework FedGEMS.
Revised: We also conduct different setting to validate the stability of our proposed framework FedGEMS.

Original: In this series of experiments, we adopt 8 clients and set the server model as ResNet-20, while keeping other experiment setting the same as main paper[Table]
Revised: In this series of experiments, we adopt 8 clients and set the server model as ResNet-20, while keeping other experiment setting the same as main paper

Original: B.3. P ERFORMANCE E VALUATIONS
Revised: . B.4. P ERFORMANCE E VALUATIONS

Original: The results of our performance are shown in Table 6.
Revised: The results of our performance are shown in Table 7.

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: B.4. R OBUSTNESS

Original: The results of our robustness experiment are shown in Table 7. [Table]
Revised: The results of our robustness experiment are shown in Table 8.

Original: B.3. C OMMUNICATION COST
Revised: B.4. C OMMUNICATION COST

Original: We further evaluate the communication costs per round by varying public dataset sizes for various approaches on CIFAR-10, and the results are shown in Fig.
Revised: We further evaluate the communication costs per round by varying public dataset sizes for various approaches on CIFAR-10, and the results are shown in Fig.

Original: B. R ESULTS OF DIFFERENT ϵ
Revised: B. R ESULTS OF DIFFERENT ϵ

Original: The results of different ϵ that balances between cross-entropy and distillation losses is  shown in 
Revised: The results of different ϵ , which is a hyper-parameter to control the weight of knowledge distillation, are shown in Table 9.

Original: Table 8.
Revised: [Table] Method ϵ = 0 .ϵ = 0 .ϵ =

Original: [Table]
Revised: [Table]

Original: B. M ODEL A RCHITECTURES OF H OMOGENEOUS S ETTING
Revised: B. M ODEL A RCHITECTURES OF H OMOGENEOUS S ETTING

Original: The model architectures of client and server models in homogeneous setting are shown in Table 9.
Revised: The model architectures of client and server models in homogeneous setting are shown in Table 10.

Original: [Table]
Revised: [Table]

Original: B. M ODEL A RCHITECTURE OF H ETEROGENEOUS S ETTING
Revised: B. M ODEL A RCHITECTURE OF H ETEROGENEOUS S ETTING

Original: The different model architectures of client models in heterogeneous setting are shown in Table 10.
Revised: The different model architectures of client models in heterogeneous setting are shown in Table 11.

Original: The server model in heterogeneous setting is the same as homogeneous setting in Table 9.
Revised: The server model in heterogeneous setting is the same as homogeneous setting in Table 10. [Table]

Original: B. M ODEL A RCHITECTURES OF LARGER SERVER MODELS
Revised: B. M ODEL A RCHITECTURES OF LARGER SERVER MODELS

Original: The detailed model architectures of large server models are shown in Table 11.
Revised: The detailed model architectures of large server models are shown in Table 12. [Table]

Original: B. H YPER - PARAMETERS
Revised: H YPER - PARAMETERS

Original: In table 12, we sum up the hyper-parameter settings for all the methods in our experiments.
Revised: In table [Table] we sum up the hyper-parameter settings for all the methods in our experiments.

Original: We directly run FedAvg, FedGKT and re-implement all the other methods base on an open-source federated learning research library FedML
Revised: We directly run FedAvg, FedGKT and re-implement all the other methods base on an open-source federated learning research library FedML

Original: (He et al., 2020b) in a distributed computing environment.
Revised: (He et al., 2020b) in a distributed computing environment

Original: [Table] [Table][Table]
Revised: [Table] [Table]