Original: M ODEL -A GNOSTIC R OUND -O PTIMAL F EDERATED L EARNING VIA K NOWLEDGE T RANSFER
Revised: M ODEL -A GNOSTIC R OUND -O PTIMAL F EDERATED L EARNING VIA K NOWLEDGE T RANSFER

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Federated learning enables multiple parties to collaboratively learn a model without exchanging their local data.
Revised: Federated learning enables multiple parties to collaboratively learn a model without exchanging their local data.

Original: Currently, federated averaging (FedAvg) is the most widely used federated learning algorithm.
Revised: Currently, federated averaging (FedAvg) is the most widely used federated learning algorithm.

Original: However, FedAvg or its variants have obvious shortcomings.
Revised: However, FedAvg or its variants have obvious shortcomings.

Original: It can only be used to learn differentiable models and needs many communication rounds to converge.
Revised: It can only be used to learn differentiable models and needs many communication rounds to converge.

Original: In this paper, we propose a novel federated learning algorithm FedKT that needs only a single communication round (i.e., round-optimal).
Revised: In this paper, we propose a novel federated learning algorithm FedKT that needs only a single communication round (i.e., round-optimal).

Original: With applying the knowledge transfer approach, our algorithm can be applied to any classiﬁcation model.
Revised: With applying the knowledge transfer approach, our algorithm can be applied to any classiﬁcation model.

Original: Moreover, we develop the differentially private versions of FedKT and theoretically analyze the privacy loss.
Revised: Moreover, we develop the differentially private versions of FedKT and theoretically analyze the privacy loss.

Original: The experiments show that our method can achieve close or better accuracy compared with the other state-of-the-art federated learning algorithms.
Revised: The experiments show that our method can achieve close or better accuracy compared with the other state-of-the-art federated learning algorithms.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: While the size of training data can inﬂuence the machine learning model quality a lot, the data are often dispersed over different parties in reality.
Revised: While the size of training data can inﬂuence the machine learning model quality a lot, the data are often dispersed over different parties in reality.

Original: Due to regulations on data privacy, the data cannot be centralized to a single party for training.
Revised: Due to regulations on data privacy, the data cannot be centralized to a single party for training.

Original: To address these issues, federated learning (Kairouz et al., 2019;
Revised: To address these issues, federated learning (Kairouz et al., 2019;

Original: Li et al., 2019a;b; Yang et al., 2019) enables multiple parties to collaboratively learn a model without exchanging their local data.
Revised: Li et al., 2019a;b; Yang et al., 2019) enables multiple parties to collaboratively learn a model without exchanging their local data.

Original: It has become a hot research topic and shown promising results in the real world (Bonawitz et al., 2019; Hard et al., 2018;
Revised: It has become a hot research topic and shown promising results in the real world (Bonawitz et al., 2019; Hard et al., 2018;

Original: Li et al., 2020a; Peng et al., 2020).
Revised: Li et al., 2020a; Peng et al., 2020).

Original: Currently, federated averaging (FedAvg) (McMahan et al., 2016) is a widely used federated learning algorithm.
Revised: Currently, federated averaging (FedAvg) (McMahan et al., 2016) is a widely used federated learning algorithm.

Original: Its training is an iterative process with four steps in each iteration.
Revised: Its training is an iterative process with four steps in each iteration.

Original: First, the server sends the global model to the selected parties.
Revised: First, the server sends the global model to the selected parties.

Original: Second, each of the selected parties updates its model with their local data.
Revised: Second, each of the selected parties updates its model with their local data.

Original: Third, the updated models are sent to the server.
Revised: Third, the updated models are sent to the server.

Original: Last, the server averages all the received models to update the global model.
Revised: Last, the server averages all the received models to update the global model.

Original: There are also many variants of FedAvg (Li et al., 2020c; Karimireddy et al., 2020).
Revised: There are also many variants of FedAvg (Li et al., 2020c; Karimireddy et al., 2020).

Original: For example, to handle the heterogeneous data setting,
Revised: For example, to handle the heterogeneous data setting,

Original: FedProx (Li et al., 2020c) introduces an additional proximal term to limit the local updates, while
Revised: FedProx (Li et al., 2020c) introduces an additional proximal term to limit the local updates, while

Original: SCAFFOLD (Karimireddy et al., 2020) introduces control variates to correct the local updates.
Revised: SCAFFOLD (Karimireddy et al., 2020) introduces control variates to correct the local updates.

Original: The overall frameworks of these studies are still similar to FedAvg.
Revised: The overall frameworks of these studies are still similar to FedAvg.

Original: FedAvg or its variants have the following limitations.
Revised: FedAvg or its variants have the following limitations.

Original: First, they rely on the gradient descent for optimization.
Revised: First, they rely on the gradient descent for optimization.

Original: Thus, they cannot be applied to train non-differentiable models such as decision trees in the federated setting.
Revised: Thus, they cannot be applied to train non-differentiable models such as decision trees in the federated setting.

Original: Second, the algorithm usually needs many communication rounds to ﬁnally achieve a good model, which causes massive communication trafﬁc and fault tolerance requirements among rounds.
Revised: Second, the algorithm usually needs many communication rounds to ﬁnally achieve a good model, which causes massive communication trafﬁc and fault tolerance requirements among rounds.

Original: Last, FedAvg is originally designed for the cross-device setting (Kairouz et al., 2019), where the parties are mobile devices.
Revised: Last, FedAvg is originally designed for the cross-device setting (Kairouz et al., 2019), where the parties are mobile devices and the number of parties is large.

Original: In the cross-silo setting, it is possible to take better advantage of the computation resources of the partiesthat are organizations with relatively high computation power.
Revised: In the cross-silo setting where the parties are organizations or data centers and the number of parties is relatively small, it is possible to take better advantage of the computation resources of the parties with relatively high computation power.

Original: In order to address the above-mentioned limitations, we propose a novel federated learning algorithm called FedKT (Federated learning via Knowledge Transfer) focusing on the cross-silo setting.
Revised: In order to address the above-mentioned limitations, we propose a novel federated learning algorithm called FedKT (Federated learning via Knowledge Transfer) focusing on the cross-silo setting .

Original: With the round-optimal design goal, FedKT extends the idea of ensemble learning in a novel 2-tier design to federated setting.
Revised: With the round-optimal design goal, FedKT extends the idea of ensemble learning in a novel 2-tier design to federated setting.

Original: Inspired by the success of the usage of unlabelled public data in many studies (Papernot et al., 2017; 2018; Jordon et al., 2019; Chang et al., 2019), which often exists such as text and images, we adopt the knowledge transfer method to reduce the inference and storage costs of ensemble learning.
Revised: Inspired by the success of the usage of unlabelled public data in many studies (Papernot et al., 2017; 2018; Jordon et al., 2019; Chang et al., 2019), which often exists such as text and images, we adopt the knowledge transfer method to reduce the inference and storage costs of ensemble learning.

Original: As such, FedKT is able to learn any classiﬁcation model including differentiable models and non-differentiable models.
Revised: As such, FedKT is able to learn any classiﬁcation model including differentiable models and non-differentiable models.

Original: Moreover, we develop differentially private versions and theoretically analyze the privacy loss of FedKT in order to provide different differential privacy guarantees.
Revised: Moreover, we develop differentially private versions and theoretically analyze the privacy loss of FedKT in order to provide different differential privacy guarantees.

Original: Our experiments on four tasks show that FedKT has quite good performance compared with the other state-of-the-art algorithms.
Revised: Our experiments on four tasks show that FedKT has quite good performance compared with the other state-of-the-art algorithms.

Revised: Our main contributions are as follows.

Revised: • We propose a new federated learning algorithm named FedKT.

Revised: To the best of our knowledge, FedKT is the ﬁrst algorithm which does not have any limitations on the model architecture and needs only a single communication round.

Original:   B ACKGROUND AND R ELATED W ORK 2. E NSEMBLE L EARNING
Revised: • We show that FedKT is easy to achieve both example-level and party-level differential privacy and theoretically analyze the bound of its privacy cost. • We conduct experiments on various models and tasks and show that FedKT can achieve comparable accuracy compared with the other iterative algorithms. Moreover, FedKT can be used as an initialization step to achieve a better accuracy combined with the other approaches.  B ACKGROUND AND R ELATED W ORK 2. E NSEMBLE L EARNING

Original: Instead of using a single model for prediction, ensemble learning (Zhang & Ma, 2012) combines the predictions of multiple models to obtain better predictive performance.
Revised: Instead of using a single model for prediction, ensemble learning (Zhang & Ma, 2012) combines the predictions of multiple models to obtain better predictive performance.

Original: There are many widely used ensemble learning algorithms such as boosting (R¨atsch et al., 2001) and bagging (Prasad et al., 2006).
Revised: There are many widely used ensemble learning algorithms such as boosting (R ¨ atsch et al., 2001) and bagging (Prasad et al., 2006).

Original: One important factor in ensemble learning is the model diversity.
Revised: One important factor in ensemble learning is the model diversity.

Original: The increased model diversity can usually improve the performance of the ensemble learning.
Revised: The increased model diversity can usually improve the performance of the ensemble learning.

Original: In federated learning, since different parties have their own local data, there is natural diversity among the local models.
Revised: In federated learning, since different parties have their own local data, there is natural diversity among the local models.

Original: Thus, the local models can be used as an ensemble for prediction.
Revised: Thus, the local models can be used as an ensemble for prediction.

Original: Previous works (Yurochkin et al., 2019; Guha et al., 2019) have studied ensemble learning for federated learning and demonstrated promising predictive accuracy.
Revised: Previous works (Yurochkin et al., 2019; Guha et al., 2019) have studied ensemble learning for federated learning and demonstrated promising predictive accuracy.

Original: As mentioned in their studies, since the prediction involves all the local models, the inference and the storage costs are prohibitively high especially when the number of models is large.
Revised: As mentioned in their studies, since the prediction involves all the local models, the inference and the storage costs are prohibitively high especially when the number of models is large.

Original: In our study, we also use the local models as an ensemble and further use knowledge transfer to learn a single model in order to reduce the inference and the storage costs.
Revised: In our study, we also use the local models as an ensemble and further use knowledge transfer to learn a single model in order to reduce the inference and the storage costs.

Original: 2. K NOWLEDGE T RANSFER OF THET EACHER E NSEMBLE
Revised: 2. K NOWLEDGE T RANSFER OF THE

Original: Knowledge transfer has been successfully used in previous studies (Hinton et al., 2015; Papernot et al., 2017; 2018; Jordon et al., 2019).
Revised: Knowledge transfer has been successfully used in previous studies (Hinton et al., 2015; Papernot et al., 2017; 2018; Jordon et al., 2019).

Original: Through knowledge transfer, an ensemble of models can be compressed into a single model.
Revised: Through knowledge transfer, an ensemble of models can be compressed into a single model.

Original: A typical example is the PATE (Private Aggregation of Teacher
Revised: A typical example is the PATE (Private Aggregation of Teacher

Original: Ensembles) (Papernot et al., 2017) framework.
Revised: Ensembles) (Papernot et al., 2017) framework.

Original: In this framework, PATE ﬁrst divides the original dataset into multiple disjoint subsets.
Revised: In this framework, PATE ﬁrst divides the original dataset into multiple disjoint subsets.

Original: A teacher model is trained separately on each subset.
Revised: A teacher model is trained separately on each subset.

Original: Then, the max voting method is used to make predictions on the public unlabelled datasets with the teacher ensemble, i.e., choosing the majority class among the teachers as the label.
Revised: Then, the max voting method is used to make predictions on the public unlabelled datasets with the teacher ensemble, i.e., choosing the majority class among the teachers as the label.

Original: Last, a student model is trained on the public dataset.
Revised: Last, a student model is trained on the public dataset.

Original: A good feature of PATE is that it can easily satisfy differential privacy guarantees by adding noises to the vote counts.
Revised: A good feature of PATE is that it can easily satisfy differential privacy guarantees by adding noises to the vote counts.

Original: Moreover, PATE can be applied to any classiﬁcation model regardless of the training algorithm.
Revised: Moreover, PATE can be applied to any classiﬁcation model regardless of the training algorithm.

Original: PATE is not designed for federated learning.
Revised: PATE is not designed for federated learning.

Original: Inspired by
Revised: Inspired by

Original: PATE, we propose FedKT, which adopts the knowledge transfer approach in the federated setting to address the limitations of FedAvg.
Revised: PATE, we propose FedKT, which adopts the knowledge transfer approach in the federated setting to address the limitations of FedAvg.

Original: 2. F EDERATED L EARNING WITH A S INGLE C OMMUNICATION R OUND
Revised: 2. F EDERATED L EARNING WITH A S INGLE C OMMUNICATION R OUND

Original: There are several preliminary studies on federated learning algorithms with a single communication round.
Revised: There are several preliminary studies on federated learning algorithms with a single communication round.

Original: Guha et al.
Revised: Guha et al.

Original: (2019) propose an one-shot federated learning algorithm to train support vector machines (SVMs) in both supervised and semi-supervised settings.
Revised: (2019) propose an one-shot federated learning algorithm to train support vector machines (SVMs) in both supervised and semi-supervised settings.

Original: Instead of simply averaging all the model weights in FedAvg, Yurochkin et al.
Revised: Instead of simply averaging all the model weights in FedAvg, Yurochkin et al.

Original: (2019) propose PFNM by adopting a Bayesian nonparametric model to aggregate the local models when they are multilayer perceptrons (MLPs).
Revised: (2019) propose PFNM by adopting a Bayesian nonparametric model to aggregate the local models when they are multilayer perceptrons (MLPs).

Original: Their method shows a good performance in a single communication round and can also be applied in multiple communication rounds.
Revised: Their method shows a good performance in a single communication round and can also be applied in multiple communication rounds.

Original: While the above two methods are designed for speciﬁc models (i.e., SVMs in Guha et al.
Revised: While the above two methods are designed for speciﬁc models (i.e., SVMs in Guha et al.

Original: and MLPs in Yurochkin et al.
Revised: and MLPs in Yurochkin et al.

Original: , we propose a general federated learning framework which is applicable to any classiﬁcation model.
Revised: , we propose a general federated learning framework which is applicable to any classiﬁcation model.

Original: 2. D IFFERENTIAL P RIVACY
Revised: 2. F EDERATED L EARNING WITH K NOWLEDGE T RANSFER

Revised: There are some related studies (Li & Wang, 2019; Chang et al., 2019) using knowledge transfer in federated learning.

Revised: However, Li & Wang (2019) has a different setting with us while Chang et al.

Revised: (2019) has a different objective with us.

Revised: In (Li & Wang, 2019), a public labeled dataset is needed to conduct initial transfer learning, while FedKT only needs a public unlabelled dataset.

Revised: Chang et al.

Revised: (2019) designs a robust federated learning algorithm to protect against poisoning attacks.

Revised: The performance of their approach is slightly worse than FedAvg.

Revised: We notice that there are some recently published contemporaneous work (He et al., 2020; Lin et al., 2020).

Revised: He et al.

Revised: (2020) considers cross-device setting and uses group knowledge transfer to reduce the overload of each edge device.

Revised: Lin et al.

Revised: (2020) utilizes knowledge transfer only in the server side to further update the averaged global model.

Revised: All existing approaches transfer the prediction vectors (i.g., logits) on the public dataset between clients and the server.

Revised: FedKT transfers the voting counts and thus can easily satisfy differential privacy guarantees with a tight theoretical bound on the privacy loss.

Revised: Moreover, FedKT is designed with a round-optimal goal, while the other approaches use iterative learning algorithms that needs many communication rounds to converge.

Revised: 2. D IFFERENTIAL P RIVACY

Original: Differential privacy (Dwork, 2011; Dwork et al., 2014) is a popular standard of privacy protection.
Revised: Differential privacy (Dwork, 2011; Dwork et al., 2014) is a popular standard of privacy protection.

Original: It guarantees that the probability of producing a given output does not depend much on whether a particular data record is included in the input dataset or not.
Revised: It guarantees that the probability of producing a given output does not depend much on whether a particular data record is included in the input dataset or not.

Original: It has been widely used to protect the machine learning models (Shokri & Shmatikov, 2015; Abadi et al., 2016; Li et al., 2020b).
Revised: It has been widely used to protect the machine learning models (Shokri & Shmatikov, 2015; Abadi et al., 2016; Li et al., 2020b).

Original: Deﬁnition 1.
Revised: Deﬁnition 1.

Original: ( ( ε, δ ) -Differential Privacy)
Revised: ( ( ε, δ ) -Differential Privacy)

Original: Let M : D → R be a randomized mechanism with domain D and range R .
Revised: Let M : D → R be a randomized mechanism with domain D and range R .

Original: M satisifes ( (cid:15), δ ) -differential
Revised: M satisifes ( (cid:15), δ ) -differential

Original: privacy if for any two adjacent inputs d, d (cid:48) ∈ D and any subset of outputs S ⊆ R it holds that:
Revised: privacy if for any two adjacent inputs d, d (cid:48) ∈ D and any subset of outputs S ⊆ R it holds that:

Original: [Equation]
Revised: [Equation]

Original: The moments accountant method (Abadi et al., 2016) is a state-of-the-art approach to track the privacy loss.
Revised: The moments accountant method (Abadi et al., 2016) is a state-of-the-art approach to track the privacy loss.

Original: We brieﬂy introduce the key concept, and refer readers to the previous paper (Abadi et al., 2016) for more details.
Revised: We brieﬂy introduce the key concept, and refer readers to the previous paper (Abadi et al., 2016) for more details.

Original: Deﬁnition 2.
Revised: Deﬁnition 2.

Original: (Privacy Loss) Let M : D → R be a randomized mechanism.
Revised: (Privacy Loss) Let M : D → R be a randomized mechanism.

Original: Let aux denote an auxiliary input.
Revised: Let aux denote an auxiliary input.

Original: For two adjacent inputs d, d (cid:48) ∈ D , an outcome o ∈ R , the privacy loss at o is deﬁned as:
Revised: For two adjacent inputs d, d (cid:48) ∈ D , an outcome o ∈ R , the privacy loss at o is deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: Deﬁnition 3.
Revised: Deﬁnition 3.

Original: (Moments Accountant)
Revised: (Moments Accountant)

Original: Let M : D → R be a randomized mechanism.
Revised: Let M : D → R be a randomized mechanism.

Original: Let aux denote an auxiliary input.
Revised: Let aux denote an auxiliary input.

Original: For two adjacent inputs d, d (cid:48) , the moments accountant is deﬁned as:
Revised: For two adjacent inputs d, d (cid:48) , the moments accountant is deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: where α M ( λ ; aux , d, d (cid:48) )
Revised: where α M ( λ ; aux , d, d (cid:48) )

Original: ∆ = log E o [exp( λc ( o ; M , aux , d, d (cid:48) ))] is the log of moment generating function.
Revised: ∆ = log E o [exp( λc ( o ; M , aux , d, d (cid:48) ))] is the log of moment generating function.

Original: The moments have good composability and can be easily converted to ( ε, δ )
Revised: The moments have good composability and can be easily converted to ( ε, δ )

Original: -differential privacy
Revised: -differential privacy

Original: Abadi et al.
Revised: Abadi et al.

Original: Party-level Differential Privacy
Revised: Party-level Differential Privacy

Original: In addition to the standard example-level differential privacy, party-level differential privacy (Geyer et al., 2017; McMahan et al., 2018) is more strict and attractive in the federated setting.
Revised: In addition to the standard example-level differential privacy, party-level differential privacy (Geyer et al., 2017; McMahan et al., 2018) is more strict and attractive in the federated setting.

Original: Instead of aiming to protect a single record, party-level differential privacy ensures that the model does not reveal whether a party participated in federated learning or not.
Revised: Instead of aiming to protect a single record, party-level differential privacy ensures that the model does not reveal whether a party participated in federated learning or not.

Original: Deﬁnition 4.
Revised: Deﬁnition 4.

Original: (Party-adjacent Datasets) Let d , d (cid:48) be two datasets of training examples, where each example is associated with a party.
Revised: (Party-adjacent Datasets) Let d , d (cid:48) be two datasets of training examples, where each example is associated with a party.

Original: Then, d and d (cid:48) are party-adjacent if d (cid:48) can be formed by changing the examples associated with a single party from d . 
Revised: Then, d and d (cid:48) are party-adjacent if d (cid:48) can be formed by changing the examples associated with a single party from d . Dataset Data Data

Revised: Data t ...

Original: F EDERATED L EARNING VIA K NOWLEDGE T RANSFER
Revised: Teacher Teacher Train Teacher t

Revised: Student

Revised: Knowledge transfer

Revised: Dataset n

Revised: Data

Revised: Data

Revised: Data t ...

Revised: Teacher

Revised: Teacher

Revised: Train

Revised: Teacher t

Revised: Student

Revised: Student s

Revised: Knowledge

Revised: [Figure]

Revised: [Figure]

Revised: Student

Revised: [Figure]

Revised:  F EDERATED L EARNING VIA K NOWLEDGE T RANSFER

Original: Problem Statement Suppose there are n parties P 1 , .., P n .
Revised: Problem Statement Suppose there are n parties P 1 , .., P n .

Original: We use D i to denote the dataset of
Revised: We use D i to denote the dataset of

Original: P i .
Revised: P i .

Original: With the help of a central server and a public unlabelled dataset D aux , our objective is to build a machine learning model over the datasets (cid:83) i ∈
Revised: With the help of a central server and a public unlabelled dataset D aux , our objective is to build a machine learning model over the datasets (cid:83) i ∈

Original: [ n ] D i without exchanging the raw data.
Revised: [ n ] D i without exchanging the raw data.

Original: Moreover, the learning process should be able to support three different privacy level settings: (1) L 0 : there is no privacy requirement on the machine learning models.
Revised: Moreover, the learning process should be able to support three different privacy level settings: (1) L 0 : there is no privacy requirement on the machine learning models.

Original: (2) L 1 (server-noise): in the case where the ﬁnal model has to be sent back to the parties or even published, it should satisfy differential privacy guarantees to protect against potential inference attacks (Shokri et al., 2017; Fredrikson et al., 2015; Nasr et al., 2019).
Revised: (2) L 1 (server-noise): in the case where the ﬁnal model has to be sent back to the parties or even published, it should satisfy differential privacy guarantees to protect against potential inference attacks (Shokri et al., 2017; Fredrikson et al., 2015; Nasr et al., 2019).

Original: (3) L 2 (party-noise): in the case where the server is curious and all the models transferred between the parties and the server during training should satisfy differential privacy guarantees.
Revised: (3) L 2 (party-noise): in the case where the server is curious and all the models transferred between the parties and the server during training should satisfy differential privacy guarantees.

Original: The Overall Framework
Revised: The Overall Framework

Original: The framework of FedKT is shown in Figure 1.
Revised: The framework of FedKT is shown in Figure 1.

Original: Overall, FedKT adopts a 2-tier PATE structure.
Revised: Overall, FedKT adopts a 2-tier PATE structure.

Original: On the party side, each party uses knowledge transfer to learn student models and sends them to the server.
Revised: On the party side, each party uses knowledge transfer to learn student models and sends them to the server.

Original: On the server side, the server takes the received models as teachers to learn a ﬁnal model using knowledge transfer again.
Revised: On the server side, the server takes the received models as teachers to learn a ﬁnal model using knowledge transfer again.

Original: The ﬁnal model is sent back to the parties and used for predictions.
Revised: The ﬁnal model is sent back to the parties and used for predictions.

Original: Next, we introduce these two steps in details.
Revised: Next, we introduce these two steps in details.

Original: Learning Student Models on the Parties Locally, each party has to create s ( s ≥ 1 ) partitions and learn a student model on each partition.
Revised: Learning Student Models on the Parties Locally, each party has to create s ( s ≥ 1 ) partitions and learn a student model on each partition.

Original: Each partition handles the entire local dataset.
Revised: Each partition handles the entire local dataset.

Original: Since the operations in each partition are similar, here we describe the process in one partition for ease of presentation.
Revised: Since the operations in each partition are similar, here we describe the process in one partition for ease of presentation.

Original: Inside a partition, the local dataset is divided into t disjoint subsets.
Revised: Inside a partition, the local dataset is divided into t disjoint subsets.

Original: We train a teacher model separately on each subset, denoted as T 1 , ..., T t .
Revised: We train a teacher model separately on each subset, denoted as T 1 , ..., T t .

Original: Then, the ensemble of teacher models is used to make predictions on the public dataset D aux .
Revised: Then, the ensemble of teacher models is used to make predictions on the public dataset D aux .

Original: For an example x ∈ D aux , the vote count of class m is the number of teachers that predicts m , i.e., v m ( x ) =
Revised: For an example x ∈ D aux , the vote count of class m is the number of teachers that predicts m , i.e., v m ( x ) =

Original: |{ i : i ∈ [ t ] , T i ( x )
Revised: |{ i : i ∈ [ t ] , T i ( x )

Original: = m }| .
Revised: = m }| .

Original: The prediction result of the ensemble is the class that has the maximum vote counts, i.e., f ( x ) =
Revised: The prediction result of the ensemble is the class that has the maximum vote counts, i.e., f ( x ) =

Original: arg max m v m ( x ) .
Revised: arg max m v m ( x ) .

Original: Then, we use the public dataset D aux with the predicted labels to train a student model.
Revised: Then, we use the public dataset D aux with the predicted labels to train a student model.

Original: For each partition, we get a student model with the above steps.
Revised: For each partition, we get a student model with the above steps.

Original: After all the student models are trained, the parties send their student models to the server for further processing.Dataset Data Data
Revised: After all the student models are trained, the parties send their student models to the server for further processing.

Original: Data t ...

Original: Teacher

Original: Teacher

Original: Train

Original: Teacher t

Original: Student

Original: Knowledge transfer

Original: Dataset n

Original: Data

Original: Data

Original: Data t ...

Original: Teacher

Original: Teacher

Original: Train

Original: Teacher t

Original: Student

Original: Student s

Original: Knowledge transfer

Original: Server . .

Original: Final model Knowledge transfer

Original: Student s

Original: [Figure]

Original: Learning the Final Model on the Server Suppose the student models of party i are denoted as
Revised: Learning the Final Model on the Server Suppose the student models of party i are denoted as

Original: U i 1 , ..., U is ( i ∈ [ n ] ).
Revised: U i 1 , ..., U i s ( i ∈ [ n ] ).

Original: After receiving all the student models, like the steps on the party side, the server can use these student models as an ensemble to make predictions on the public dataset D aux .
Revised: After receiving all the student models, like the steps on the party side, the server can use these student models as an ensemble to make predictions on the public dataset D aux .

Original: The public dataset with the predicted labels is used to train the ﬁnal model.
Revised: The public dataset with the predicted labels is used to train the ﬁnal model.

Original: Here we introduce a technique named consistent voting for computing the vote counts of each class.
Revised: Here we introduce a technique named consistent voting for computing the vote counts of each class.

Original: If the student models of a party make the same prediction on an example, we take their predictions into account.
Revised: If the student models of a party make the same prediction on an example, we take their predictions into account.

Original: Otherwise, the party is not conﬁdent at predicting this example and thus we ignore the predictions of its student models.
Revised: Otherwise, the party is not conﬁdent at predicting this example and thus we ignore the predictions of its student models.

Original: Formally, given an example x ∈ D aux , we ﬁrst compute the vote count of class m on the student models of party i as v im ( x ) =
Revised: Formally, given an example x ∈ D aux , we ﬁrst compute the vote count of class m on the student models of party i as v im ( x ) =

Original: |{ k : k ∈ [ s ] , U ik ( x ) = m }| .
Revised: |{ k : k ∈ [ s ] , U ik ( x ) = m }| .

Original: Next, with consistent voting, the ﬁnal vote count of class m on all parties is computed as v m ( x ) =
Revised: Next, with consistent voting, the ﬁnal vote count of class m on all parties is computed as v m ( x ) =

Original: s · |{ i : i ∈ [ n ] , v im ( x ) = s }| .
Revised: s · |{ i : i ∈ [ n ] , v im ( x ) = s }| .

Original: Differentially Private Versions of FedKT FedKT can easily satisfy differential privacy guarantees by providing differentially private prediction results to the query dataset.
Revised: Differentially Private Versions of FedKT FedKT can easily satisfy differential privacy guarantees by providing differentially private prediction results to the query dataset.

Original: Given the privacy parameter γ , we can add noises to the vote count histogram such that f ( x ) =
Revised: Given the privacy parameter γ , we can add noises to the vote count histogram such that f ( x ) =

Original: arg max m { v m ( x ) +
Revised: arg max m { v m ( x ) +

Original: Lap ( 1 γ ) } , where Lap ( 1 γ ) is the noises generated from Laplace distribution with location 0 and scale 1 γ .
Revised: Lap ( 1 γ ) } , where Lap ( 1 γ ) is the noises generated from Laplace distribution with location 0 and scale 1 γ .

Original: Note that we do not need to add noises on both the parties and the server.
Revised: Note that we do not need to add noises on both the parties and the server.

Original: For the L 1 setting, we only need to add noises on the server side.
Revised: For the L 1 setting, we only need to add noises on the server side.

Original: The parties can train and send non-differentially private student models to the server.
Revised: The parties can train and send non-differentially private student models to the server.

Original: For the L 2 setting, we only need to add noises on the party side so that the student models are differentially private.
Revised: For the L 2 setting, we only need to add noises on the party side so that the student models are differentially private.

Original: Then, the ﬁnal model naturally satisﬁes differential privacy guarantees.
Revised: Then, the ﬁnal model naturally satisﬁes differential privacy guarantees.

Original: More analysis on privacy loss will be presented in Section 4.
Revised: More analysis on privacy loss will be presented in Section 4.

Original: Algorithm 1 shows the whole training process of FedKT.
Revised: Algorithm 1 shows the whole training process of FedKT.

Original: In the algorithm, for each party (i.e.,
Revised: In the algorithm, for each party (i.e.,

Original: Line 1) and its each partition (i.e., Line 3), we train a student model using knowledge transfer (i.e.,
Revised: Line 1) and its each partition (i.e., Line 3), we train a student model using knowledge transfer (i.e.,

Original: Lines 4-12).
Revised: Lines 4-12).

Original: The student models are sent to the server.
Revised: The student models are sent to the server.

Original: Then, the server trains the ﬁnal model using knowledge transfer again (i.e., Lines 14-23).
Revised: Then, the server trains the ﬁnal model using knowledge transfer again (i.e., Lines 14-23).

Original: For different privacy level settings, we have the corresponding noises injection operations on the server side (i.e., Lines 20-21) or the party side (i.e.,
Revised: For different privacy level settings, we have the corresponding noises injection operations on the server side (i.e., Lines 20-21) or the party side (i.e.,

Original: Lines 9-10).
Revised: Lines 9-10).

Original: Overhead Analysis of FedKT Suppose the size of each model is M .
Revised: Overhead Analysis of FedKT Suppose the size of each model is M .

Original: Then, the total communication size of FedKT is nsM + nM = nM
Revised: Then, the total communication size of FedKT is nsM + nM = nM

Original: ( s + 1) including sending the student models to the server and sending back the ﬁnal model to the parties.
Revised: ( s + 1) including sending the student models to the server and sending back the ﬁnal model to the parties.

Original: Suppose the number of communication rounds in
Revised: Suppose the number of communication rounds in

Original: FedAvg is r and all the parties participate in the training in every iteration.
Revised: FedAvg is r and all the parties participate in the training in every iteration.

Original: Then the total communication size of FedAvg is 2 nMr .
Revised: Then the total communication size of FedAvg is 2 nMr .

Original: Thus, when r > s +12 , the communication cost of FedAvg is higher than FedKT.
Revised: Thus, when r > s +12 , the communication cost of FedAvg is higher than FedKT.

Original: This value can be quite small, e.g., r = 2 if we set s = 2 .
Revised: This value can be quite small, e.g., r = 2 if we set s = 2 .

Original: As we will show in the experiments, FedKT can achieve much better performance than FedAvg give the same communication size constraint.
Revised: As we will show in the experiments, FedKT can achieve much better performance than FedAvg give the same communication size constraint.

Original: The computation overhead of FedKT is usually larger than FedAvg since FedKT needs to train many teacher and student models.
Revised: The computation overhead of FedKT is usually larger than FedAvg since FedKT needs to train many teacher and student models.

Original: It also requires the parties to be able to store multiple models in the training process.
Revised: It also requires the parties to be able to store multiple models in the training process.

Original: Thus, FedKT is more suitable for the cross-silo setting (Yang et al., 2019), where the parties (e.g., companies, data centers) have a relatively large computation power and storage capacity.14 for x ∈ D do for all i ∈ [ ] do for all m ∈ [ ] do
Revised: Thus, FedKT is more suitable for the cross-silo setting (Yang et al., 2019), where the parties (e.g., companies, data centers) have a relatively large computation power and storage capacity.

Original: ←

Original: for all m ∈ [

Original: ] do v

Original: ) ← v m ( x ) + Lap (1 /γ )

Original: f ( x ) =

Original: [-arg max m v m ( x ) 
-]
Revised: [-arg max m v m ( x ) 
-]

Original: Train the ﬁnal model F on dataset { ( x , f ( x )) } x ∈D aux .

Original: Applications In our method, we assume that we have access to a public unlabelled dataset.
Revised: Applications In our method, we assume that we have access to a public unlabelled dataset.

Original: Note that the assumption will not greatly restrict our method’s applicability.
Revised: Note that the assumption will not greatly restrict our method’s applicability.

Original: The public datasets widely exist especially for texts and images including medical data (e.g., Kaggle COVID-

Revised: The public datasets widely exist especially for texts and images including medical data (e.g., Kaggle COVID-

Original: Open Research
Revised: Open Research

Original: Dataset Challenge, Chest X-Ray Images).
Revised: Dataset Challenge, Chest X-Ray Images).

Original: Thus, our approach can be applied in typical computer vision tasks, natural language processing tasks, and even healthcare tasks.
Revised: Thus, our approach can be applied in typical computer vision tasks, natural language processing tasks, and even healthcare tasks.

Original: Nevertheless, if the data are very sensitive and we cannot access to any public data, it is still possible to generate synthetic data using GAN with privacy guarantees (Yoon et al., 2019; Torkzadehmahani et al., 2019) on the parties.
Revised: Nevertheless, if the data are very sensitive and we cannot access to any public data, it is still possible to generate synthetic data using GAN with privacy guarantees (Yoon et al., 2019; Torkzadehmahani et al., 2019) on the parties.

Original: The synthetic data can be used as public data in our approach.
Revised: The synthetic data can be used as public data in our approach.

Original: Moreover, we have conducted experiments on a real-world scenario.
Revised: Moreover, we have conducted experiments on a real-world scenario.

Original: We use one chest X-ray dataset as the private dataset and another chest X-ray dataset as a public dataset to simulate a real-world federated case.
Revised: We use one chest X-ray dataset as the private dataset and another chest X-ray dataset as a public dataset to simulate a real-world federated case.

Original: The experiments are available at Appendix B.6, which demonstrates that FedKT does not have strict limitation on the distribution of public data.
Revised: The experiments are available at Appendix B.7, which demonstrates that FedKT does not have strict limitation on the distribution of public data.

Original: Besides the typical applications, the feasibility of single-round federated learning can enable new applications, e.g., model as a service.
Revised: Besides the typical applications, the feasibility of single-round federated learning can enable new applications, e.g., model as a service.

Original: One can sell or buy the student models in a model market.
Revised: One can sell or buy the student models in a model market.

Original: Then, after the model trade, a ﬁnal model can be learned using the student models without further communication.
Revised: Then, after the model trade, a ﬁnal model can be learned using the student models without further communication.

Original: Integration with Other Federated Learning Algorithms While FedKT is designed with a round-optimal goal, it is still applicable even if there is no limitation of communication rounds.
Revised: Integration with Other Federated Learning Algorithms While FedKT is designed with a round-optimal goal, it is still applicable even if there is no limitation of communication rounds.

Original: We can use FedKT to train a model as an initialization step.
Revised: We can use FedKT to train a model as an initialization step.

Original: The model can be seen as a initialized global model, and then we can continue to conduct iterative federated learning using the standard approaches (e.g., FedAvg, FedProx) based on it.
Revised: The model can be seen as a initialized global model, and then we can continue to conduct iterative federated learning using the standard approaches (e.g., FedAvg, FedProx) based on it.

Original: As shown in Section 5.2, the communication rounds can be signiﬁcantly reduced to achieve the same accuracy by applying FedKT as model initialization. 
Revised: As shown in Section 5.2, the communication rounds can be signiﬁcantly reduced to achieve the same accuracy by applying FedKT as model initialization. 14 for x ∈ D do for all i ∈ [ ] do for all m ∈ [ ] do

Revised: ←

Original: D ATA -D EPENDENT P RIVACY A NALYSIS OF F ED KT
Revised: for all m ∈ [ ] do v ) ← v m ( x ) + Lap (1 /γ ) f ( x ) =

Original: {+arg max m v m ( x ) 
+}
Revised: {+arg max m v m ( x ) 
+}

Revised: Train the ﬁnal model F on dataset { ( x , f ( x )) } x ∈D aux .

Revised:  D ATA -D EPENDENT P RIVACY A NALYSIS OF F ED KT

Original: In this section, we use the moments accountant method (Abadi et al., 2016) to track the privacy loss in the training process.
Revised: In this section, we use the moments accountant method (Abadi et al., 2016) to track the privacy loss in the training process.

Original: In addition to the example-level differential privacy, we also consider the party-level differential privacy (Geyer et al., 2017; McMahan et al., 2018), which is more strict and attractive in the federated setting.
Revised: In addition to the example-level differential privacy, we also consider the party-level differential privacy (Geyer et al., 2017; McMahan et al., 2018), which is more strict and attractive in the federated setting.

Original: Instead of aiming to protect a single record, party-level differential privacy ensures that the learned model does not reveal whether a party participated in federated learning or not.
Revised: Instead of aiming to protect a single record, party-level differential privacy ensures that the learned model does not reveal whether a party participated in federated learning or not.

Original: Next, we mainly present the party-level differential privacy analysis for FedKT-L and example-level differential privacy analysis for FedKT-L2.
Revised: Next, we mainly present the party-level differential privacy analysis for FedKT-L and example-level differential privacy analysis for FedKT-L2.

Original: For more analysis and proofs of the theorems in this section, please refer to Appendix A.
Revised: For more analysis and proofs of the theorems in this section, please refer to Appendix A.

Original: FedKT-L

Revised: FedKT-L

Original: Considering we change the whole dataset of a party, then at most s student models will be inﬂuenced.
Revised: Considering we change the whole dataset of a party, then at most s student models will be inﬂuenced.

Original: Thus, on the server side, the sensitivity of the vote count histogram is 2 s (i.e., the vote count of a class increases by s and the vote count of another class decreases by s ).
Revised: Thus, on the server side, the sensitivity of the vote count histogram is 2 s (i.e., the vote count of a class increases by s and the vote count of another class decreases by s ).

Original: According to the Laplace mechanism, we have the following theorem.
Revised: According to the Laplace mechanism, we have the following theorem.

Original: Theorem 1.
Revised: Theorem 1.

Original: Let M be the f function executed on the server side.
Revised: Let M be the f function executed on the server side.

Original: Given the number of partitions s and the privacy parameter γ , M satisﬁes (2 sγ, 0) party-level differential privacy.
Revised: Given the number of partitions s and the privacy parameter γ , M satisﬁes (2 sγ, 0) party-level differential privacy.

Original: Given multiple queries, an straightforward way to compute the privacy loss is to use the advanced composition (Dwork et al., 2014).
Revised: Given multiple queries, an straightforward way to compute the privacy loss is to use the advanced composition (Dwork et al., 2014).

Original: According to Papernot et al.
Revised: According to Papernot et al.

Original: (2017), we can get a tighter bound of the privacy loss by conducting a data-dependent privacy analysis with the moments accountant method.
Revised: (2017), we can get a tighter bound of the privacy loss by conducting a data-dependent privacy analysis with the moments accountant method.

Original: Similarly, we conduct a data-dependent privacy analysis for FedKT.
Revised: Similarly, we conduct a data-dependent privacy analysis for FedKT.

Original: Theorem 2.
Revised: Theorem 2.

Original: Let M be (2 sγ, 0) party-level differentially private.
Revised: Let M be (2 sγ, 0) party-level differentially private.

Original: Let q ≥ Pr[ M ( d ) (cid:54)
Revised: Let q ≥ Pr[ M ( d ) (cid:54)

Original: =
Revised: =

Original: o ∗ ] for some outcome o ∗ .
Revised: o ∗ ] for some outcome o ∗ .

Original: Let l, γ ≥ 0 and q < e 2 sγ − 1 e 4 sγ − 1 .
Revised: Let l, γ ≥ 0 and q < e 2 sγ − 1 e 4 sγ − 1 .

Original: For any aux and any two party-adjacent datasets d and d (cid:48) , M satisﬁes
Revised: For any aux and any two party-adjacent datasets d and d (cid:48) , M satisﬁes

Original: [Equation]
Revised: [Equation]

Original: Here Pr[ M ( d ) (cid:54) =
Revised: Here Pr[ M ( d ) (cid:54) =

Original: o ∗ ] can be bounded by Lemma 4 of Papernot et al.
Revised: o ∗ ] can be bounded by Lemma 4 of Papernot et al.

Original: With Theorem 2, we can track the privacy loss of each query (Abadi et al., 2016).
Revised: With Theorem 2, we can track the privacy loss of each query (Abadi et al., 2016).

Original: FedKT-L2 For each partition on the party side, we add Laplace noises to the vote counts, which is same with the PATE approach.
Revised: FedKT-L2 For each partition on the party side, we add Laplace noises to the vote counts, which is same with the PATE approach.

Original: Thus, we have the following theorem.
Revised: Thus, we have the following theorem.

Original: Theorem 3.
Revised: Theorem 3.

Original: Let M be the f function executed on each partition of a party.
Revised: Let M be the f function executed on each partition of a party.

Original: Let q ≥ Pr[ M ( d ) (cid:54)
Revised: Let q ≥ Pr[ M ( d ) (cid:54)

Original: =
Revised: =

Original: o ∗ ] for some outcome o ∗ .
Revised: o ∗ ] for some outcome o ∗ .

Original: Let l, γ ≥ 0 and q < e 2 γ − 1 e 4 γ − 1 .
Revised: Let l, γ ≥ 0 and q < e 2 γ − 1 e 4 γ − 1 .

Original: For any aux and any two adjacent datasets d and d (cid:48) , M satisﬁes
Revised: For any aux and any two adjacent datasets d and d (cid:48) , M satisﬁes

Original: [Equation]
Revised: [Equation]

Original: After bounding the privacy loss of each party, we can use the parallel composition to bound the privacy loss of the ﬁnal model.
Revised: After bounding the privacy loss of each party, we can use the parallel composition to bound the privacy loss of the ﬁnal model.

Original: Theorem 4.
Revised: Theorem 4.

Original: Suppose the student models of party P i satisfy ( ε i , δ ) -differential privacy.
Revised: Suppose the student models of party P i satisfy ( ε i , δ ) -differential privacy.

Original: Then, the ﬁnal model satisﬁes (max i ε i , δ ) -differential privacy.
Revised: Then, the ﬁnal model satisﬁes (max i ε i , δ ) -differential privacy.

Original: Note that the above privacy analysis is data-dependent.
Revised: Note that the above privacy analysis is data-dependent.

Original: Thus, the ﬁnal privacy budget is also datadependent and may have potential privacy breaches if we publish the budget.
Revised: Thus, the ﬁnal privacy budget is also datadependent and may have potential privacy breaches if we publish the budget.

Original: Like previous studies (Papernot et al., 2017; Jordon et al., 2019), we report the data-dependent privacy budgets in the experiments.
Revised: Like previous studies (Papernot et al., 2017; Jordon et al., 2019), we report the data-dependent privacy budgets in the experiments.

Original: As future work, we plan to use the smooth sensitivity algorithm (Nissim et al., 2007) to add noises to the privacy losses.
Revised: As future work, we plan to use the smooth sensitivity algorithm (Nissim et al., 2007) to add noises to the privacy losses.

Original: Also, we may get a tighter bound of the privacy loss if adopting the Gaussian noises (Papernot et al., 2018).[Table]
Revised: Also, we may get a tighter bound of the privacy loss if adopting the Gaussian noises (Papernot et al., 2018).

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: To evaluate FedKT, we conduct experiments on four public datasets: (1) A random forest on Adult dataset.
Revised: To evaluate FedKT, we conduct experiments on four public datasets: (1) A random forest on Adult dataset.

Original: The number of trees is set to 100 and the maximum tree depth is set to 6.
Revised: The number of trees is set to 100 and the maximum tree depth is set to 6.

Original: A gradient boosting decision tree (GBDT) model on cod-rna dataset.
Revised: A gradient boosting decision tree (GBDT) model on cod-rna dataset.

Original: The maximum tree depth is set to 6.
Revised: The maximum tree depth is set to 6.

Original: (3) A CNN on MNIST dataset.
Revised: (3) A CNN on MNIST dataset.

Original: The CNN has two 5x5 convolution layers followed with 2x2 max pooling (the ﬁrst with 6 channels and the second with 16 channels), two fully connected layers with
Revised: The CNN has two 5x5 convolution layers followed with 2x2 max pooling (the ﬁrst with 6 channels and the second with 16 channels), two fully connected layers with

Original: ReLu activation (the ﬁrst with 120 units and the second with 84 units), and a ﬁnal softmax output layer.
Revised: ReLu activation (the ﬁrst with 120 units and the second with 84 units), and a ﬁnal softmax output layer.

Original: (4) The same CNN on extended SVHN dataset.
Revised: (4) The same CNN on extended SVHN dataset.

Original: For the ﬁrst two datasets, we split the original dataset at random into train/test/public sets with a 75%/12.5%/12.5% proportion.
Revised: For the ﬁrst two datasets, we split the original dataset at random into train/test/public sets with a 75%/12.5%/12.5% proportion.

Original: For MNIST and
Revised: For MNIST and

Original: SVHN, we use one half of the original test dataset as the public dataset and the remaining as the ﬁnal test dataset.
Revised: SVHN, we use one half of the original test dataset as the public dataset and the remaining as the ﬁnal test dataset.

Original: Experiments on two additional datasets, CelebA and chest X-ray images, are available at Appendix B.6.
Revised: Experiments on two additional datasets, CelebA and chest X-ray images, are available at Appendix B.7.

Original: Like previous studies (Yurochkin et al., 2019; Wang et al., 2020), we use the
Revised: Like previous studies (Yurochkin et al., 2019; Wang et al., 2020), we use the

Original: Dirichlet distribution to simulate the heterogeneous data partition among the parties.
Revised: Dirichlet distribution to simulate the heterogeneous data partition among the parties.

Original: Suppose there are n parties.
Revised: Suppose there are n parties.

Original: We sample p k ∼ Dir n ( β ) and allocate a p k,j proportion of the instances of class k to party j , where Dir ( β ) is the Dirichlet distribution with a concentration parameter β (0.5 by default).
Revised: We sample p k ∼ Dir n ( β ) and allocate a p k,j proportion of the instances of class k to party j , where Dir ( β ) is the Dirichlet distribution with a concentration parameter β (0.5 by default).

Original: By default, we set the number of parties to 50 for Adult and cod-rna and to 10 for MNIST and SVHN.
Revised: By default, we set the number of parties to 50 for Adult and cod-rna and to 10 for MNIST and SVHN.

Original: We set s to 2 and t to 5 by default for all datasets.
Revised: We set s to 2 and t to 5 by default for all datasets.

Original: For more details in the experimental settings and guidelines to set the hyper-parameters (i.e., s and t ), please refer to Appendix B. 
Revised: For more details in the experimental settings and guidelines to set the hyper-parameters (i.e., s and t ), please refer to Appendix B. [Table]

Original: We compare FedKT with six baselines: (1) SOLO: each party trains its model locally and does not participate in federated learning.
Revised: We compare FedKT with six baselines: (1) SOLO: each party trains its model locally and does not participate in federated learning.

Original: (2) PATE (Papernot et al., 2017): we use the PATE framework to train a ﬁnal model on all data in a centralized setting (i.e., only a single party with the whole dataset) without adding noises.
Revised: (2) PATE (Papernot et al., 2017): we use the PATE framework to train a ﬁnal model on all data in a centralized setting (i.e., only a single party with the whole dataset) without adding noises.

Original: This method deﬁnes an upper bound of learning a ﬁnal model using knowledge transfer with public unlabelled data.
Revised: This method deﬁnes an upper bound of learning a ﬁnal model using knowledge transfer with public unlabelled data.

Original: (3) XGBoost (Chen & Guestrin, 2016): the XGBoost algorithm for the GBDT model on the whole dataset in a centralized setting.
Revised: (3) XGBoost (Chen & Guestrin, 2016): the XGBoost algorithm for the GBDT model on the whole dataset in a centralized setting.

Original: This method deﬁnes an upper bound of learning the GBDT model.
Revised: This method deﬁnes an upper bound of learning the GBDT model.

Original: (4) FedAvg (McMahan et al., 2016); (5) FedProx (Li et al., 2020c); (6) SCAFFOLD (Karimireddy et al., 2020).
Revised: (4) FedAvg (McMahan et al., 2016); (5) FedProx (Li et al., 2020c); (6) SCAFFOLD (Karimireddy et al., 2020).

Original: Here (4)-(6) are three standard or state-or-the-art federated learning algorithms.
Revised: Here (4)-(6) are three standard or state-or-the-art federated learning algorithms.

Original: 5. E FFECTIVENESS
Revised: 5. E FFECTIVENESS

Original: Table 1 shows the accuracy of FedKT 1 compared with the other baselines.
Revised: Table 1 shows the accuracy of FedKT 1 compared with the other baselines.

Original: For SOLO, we report the average accuracy of the parties.
Revised: For SOLO, we report the average accuracy of the parties.

Original: For FedAvg, FedProx, and SCAFFOLD, we report the results of running for 2 rounds (i.e., when the baselines start to have a larger communication cost than FedKT as shown in previous overhead analysis) and 50 rounds.
Revised: For FedAvg, FedProx, and SCAFFOLD, we report the results of running for 2 rounds (i.e., when the baselines start to have a larger communication cost than FedKT as shown in previous overhead analysis) and 50 rounds.

Original: From this table, we have the following observations.
Revised: From this table, we have the following observations.

Original: First, except for FedKT, the other federated learning algorithms can only learn speciﬁc models.
Revised: First, except for FedKT, the other federated learning algorithms can only learn speciﬁc models.

Original: FedKT is able to learn all the studied models with a signiﬁcant improvement compared with
Revised: FedKT is able to learn all the studied models with a signiﬁcant improvement compared with

Original: SOLO.
Revised: SOLO.

Original: Second, the accuracy of FedKT is close to PATE, which means our design has little accuracy loss compared with the centralized learning setting.
Revised: Second, the accuracy of FedKT is close to PATE, which means our design has little accuracy loss compared with the centralized learning setting.

Original: Third, given the limited communication size constraint, FedKT can achieve much better accuracy than FedAvg, FedProx, and SCAFFOLD.
Revised: Third, given the limited communication size constraint, FedKT can achieve much better accuracy than FedAvg, FedProx, and SCAFFOLD.

Original: The performance of the iterative methods is poor running with two rounds.
Revised: The performance of the iterative methods is poor running with two rounds.

Original: Last, FedKT is just slightly worse than FedAvg, FedProx, and SCAFFOLD running for 50 rounds.
Revised: Last, FedKT is just slightly worse than FedAvg, FedProx, and SCAFFOLD running for 50 rounds.

Original: FedKT gives promising results even with a single round communication.
Revised: FedKT gives promising results even with a single round communication.

Original: 5. C OMMUNICATION E FFICIENCY
Revised: 5. C OMMUNICATION E FFICIENCY

Original: Figure 2 shows the test accuracy with varying the number of communication rounds for MNIST and SVHN.
Revised: Figure 2 shows the test accuracy with varying the number of communication rounds/communication size for MNIST and SVHN.

Original: We do not present the results for Adult and cod-rna since there is no baseline running in a[Figure] [Table] federated setting.
Revised: We do not present the results for Adult and cod-rna since there is no baseline running in a federated setting.

Original: For MNIST, FedAvg, FedProx, and SCAFFOLD need about 23 rounds, 7 rounds, and 13 rounds to achieve a better accuracy than FedKT, respectively.For SVHN, FedAvg, FedProx
Revised: For MNIST, FedAvg, FedProx, and SCAFFOLD need about 23 rounds, 7 rounds, and 13 rounds to achieve a better accuracy than FedKT, respectively.

Original: [Table]
Revised: For

Original:  and SCAFFOLD need about 17, [Table] and 33 rounds to achieve a better accuracy than FedKT, respectively.
Revised: SVHN, FedAvg, FedProx, and SCAFFOLD need about 17, 18, and 33 rounds to achieve a better accuracy than FedKT, respectively.

Revised: Moreover, the communication size of FedKT is close to 0 (see the black points in Figure 2c and Figure 2d).

Revised: FedKT only needs 5.4M and 7.5M communication cost on MNIST and SVHN, respectively.

Original: Overall, FedKT is much more communication-efﬁcient than the other approaches.
Revised: Overall, FedKT is much more communication-efﬁcient than the other approaches.

Original: In Figure 2, FedKT-Prox is an approach that applies FedKT to learn a initialized global model and then applies FedProx to conduct iterative learning.
Revised: In Figure 2, FedKT-Prox is an approach that applies FedKT to learn a initialized global model and then applies FedProx to conduct iterative learning.

Original: As we can see, FedKT-Prox is always better than the other federated learning approaches.
Revised: As we can see, FedKT-Prox is always better than the other federated learning approaches.

Original: The communication round can be signiﬁcantly reduced to achieve the same accuracy if adopting FedKT as initialization.
Revised: The communication round can be signiﬁcantly reduced to achieve the same accuracy if adopting FedKT as initialization.

Original: For example, for SVHN, FedKT-Prox only needs about 16 rounds to achieve 87% accuracy, while FedProx needs about 40 rounds.
Revised: For example, for SVHN, FedKT-Prox only needs about 16 rounds to achieve 87% accuracy, while FedProx needs about 40 rounds.

Original: 5. D IFFERENTIAL P RIVATE F ED KT
Revised: 5. D IFFERENTIAL P RIVATE F ED KT

Original: We run FedKT with different γ and number of queries.
Revised: We run FedKT with different γ and number of queries.

Original: When running FedKT-L1, we tune the percentage of number of queries on the server side.
Revised: When running FedKT-L1, we tune the percentage of number of queries on the server side.

Original: When running FedKT-L2, on the contrary, we tune the percentage of number of queries on the party side.
Revised: When running FedKT-L2, on the contrary, we tune the percentage of number of queries on the party side.

Original: The selected results on Adult and codrna are reported in Table 2.
Revised: The selected results on Adult and codrna are reported in Table 2.

Revised: While differentially private FedKT does not need any knowledge on

Original: While differentially private FedKT does not need any knowledge on the model architecture, the accuracy is still comparable to the non-private version given a privacy budget less than 10.
Revised: [Figure] Datasets [Table] the model architecture, the accuracy is still comparable to the non-private version given a privacy budget less than 10.

Original: For more results, please refer to Appendix B.7.
Revised: For more results, please refer to Appendix B.8.

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: In this paper, we propose FedKT, a model-agnostic federated learning algorithm with a single communication round.
Revised: In this paper, we propose FedKT, a model-agnostic federated learning algorithm with a single communication round.

Original: Our experiments show that FedKT can learn different models with a comparable or even better accuracy compared with the other state-of-the-art algorithms.
Revised: Our experiments show that FedKT can learn different models with a comparable or even better accuracy compared with the other state-of-the-art algorithms.

Original: Moreover, the accuracy of differentially private FedKT is comparable to the non-differentially private version with a modest privacy budget.
Revised: Moreover, the accuracy of differentially private FedKT is comparable to the non-differentially private version with a modest privacy budget.

Original: Instead of updating and aggregating in an iterative way, our work show that it is promising to adopt knowledge transfer to achieve round-optimal communication efﬁciency.
Revised: Instead of updating and aggregating in an iterative way, our work show that it is promising to adopt knowledge transfer to achieve round-optimal communication efﬁciency.

Original: For future work, we will extend FedKT to the cross-device setting, where the number of parties is large and the data size of each party is small.
Revised: For future work, we will extend FedKT to the cross-device setting, where the number of parties is large and the data size of each party is small.

Original: A PPENDIX
Revised: A PPENDIX

Original: In this appendix, we ﬁrst present the data-dependent privacy analysis of FedKT-L1 and FedKT-L in Section A. Next, in Section B.1, we show the details of our experimental settings.
Revised: In this appendix, we ﬁrst present the data-dependent privacy analysis of FedKT-L1 and FedKT-L in Section A. Next, in Section B.1, we show the details of our experimental settings.

Original: Then, we show additional experimental results in Section B.2 to Section B.7.
Revised: Then, we show additional experimental results in Section B.2 to Section B.8.

Original: Speciﬁcally, in Section B.2, we study the hyper-parameters setting of FedKT.
Revised: Speciﬁcally, in Section B.2, we study the hyper-parameters setting of FedKT.

Original: In Section B.3, we study the performance of FedKT with different levels of imbalance.
Revised: In Section B.3, we study the performance of FedKT with different levels of imbalance.

Original: In Section B.4, we study the performance of FedKT with different number of parties.
Revised: In Section B.4, we study the performance of FedKT with different number of parties.

Original: In Section B.5, we study the effect of our consistent voting technique.
Revised: In Section B.6, we study the effect of our consistent voting technique.

Original: In Section
Revised: In Section

Original: B.6, we conduct experiments on addtional datasets, including the CelebA dataset with VGG-9 and two chest X-Ray datasets with CNN.
Revised: B.7, we conduct experiments on addtional datasets, including the CelebA dataset with VGG-9 and two chest X-Ray datasets with CNN.

Original: Last, in section B.7, we show the performance of FedKT-L and FedKT-L2.
Revised: Last, in section B.8, we show the performance of FedKT-L and FedKT-L2.

Original: A P RIVACY A NALYSIS OF F ED KT
Revised: A P RIVACY A NALYSIS OF F ED KT

Original: In this section, we analyze the privacy loss of FedKT-L1 and FedKT-L2 using the moments accountant method (Abadi et al., 2016).
Revised: In this section, we analyze the privacy loss of FedKT-L1 and FedKT-L2 using the moments accountant method (Abadi et al., 2016).

Original: A. P RELIMINARIES
Revised: A. P RELIMINARIES

Original: We ﬁrst introduce two theorems from the previous studies, which will be used in our analysis.
Revised: We ﬁrst introduce two theorems from the previous studies, which will be used in our analysis.

Original: The ﬁrst theorem is from Bun & Steinke (2016) and the second theorem is from Papernot et al.
Revised: The ﬁrst theorem is from Bun & Steinke (2016) and the second theorem is from Papernot et al.

Original: Theorem 5.
Revised: Theorem 5.

Original: Let M be (2 γ, 0) -differentially private.
Revised: Let M be (2 γ, 0) -differentially private.

Original: For any l , aux , neighboring inputs d and d (cid:48) , we have
Revised: For any l , aux , neighboring inputs d and d (cid:48) , we have

Original: [Equation]
Revised: [Equation]

Original: Theorem 6.
Revised: Theorem 6.

Original: Let M be (2 γ, 0) -differentially private and q ≥ Pr[ M ( d ) (cid:54) =
Revised: Let M be (2 γ, 0) -differentially private and q ≥ Pr[ M ( d ) (cid:54) =

Original: o ∗ ] for some outcome o ∗ .
Revised: o ∗ ] for some outcome o ∗ .

Original: Let l, γ ≥ 0 and q < e 2 γ − 1 e 4 γ − 1 .
Revised: Let l, γ ≥ 0 and q < e 2 γ − 1 e 4 γ − 1 .

Original: Then for any aux and any neighbor d (cid:48) of d , M satisﬁes
Revised: Then for any aux and any neighbor d (cid:48) of d , M satisﬁes

Original: [Equation]
Revised: [Equation]

Original: Pr[ M ( d ) (cid:54) =
Revised: Pr[ M ( d ) (cid:54) =

Original: o ∗ ] can be bounded by the following lemma.
Revised: o ∗ ] can be bounded by the following lemma.

Original: Lemma 7.
Revised: Lemma 7.

Original: Let v be the label score vector for an instance d with v o ∗ ≥ v o for all o .
Revised: Let v be the label score vector for an instance d with v o ∗ ≥ v o for all o .

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: The following theorem from Abadi et al.
Revised: The following theorem from Abadi et al.

Original: (2016) can be used to convert the moments into ( ε, δ ) - differential privacy.
Revised: (2016) can be used to convert the moments into ( ε, δ ) - differential privacy.

Original: Theorem 8. 1.
Revised: Theorem 8. 1.

Original: [Composability] Suppose that a mechanism M consists of a sequence of adaptive mechanisms M 1 , . . .
Revised: [Composability] Suppose that a mechanism M consists of a sequence of adaptive mechanisms M 1 , . . .

Original: , M k where M i : (cid:81) i − 1 j =1 R j × D
Revised: , M k where M i : (cid:81) i − 1 j =1 R j × D

Original: → R i .
Revised: → R i .

Original: Then, for any λ α M ( λ ) ≤ k (cid:88) i = α M i ( λ ) , 2.
Revised: Then, for any λ α M ( λ ) ≤ k (cid:88) i = α M i ( λ ) , 2.

Original: [Tail bound]
Revised: [Tail bound]

Original: For any ε > 0 , the mechanism M is ( ε, δ ) -differentially private for
Revised: For any ε > 0 , the mechanism M is ( ε, δ ) -differentially private for

Original: [Equation]
Revised: [Equation]

Original: A. P ROOF OF T HEOREM
Revised: A. P ROOF OF T HEOREM

Original: Proof.
Revised: Proof.

Original: According to Theorem 1, we can get Theorem 2 by replacing γ in Theorem 5 and Theorem 6 to 2 sγ .
Revised: According to Theorem 1, we can get Theorem 2 by replacing γ in Theorem 5 and Theorem 6 to 2 sγ .

Original: A. P ROOF OF T HEOREM
Revised: A. P ROOF OF T HEOREM

Original: Proof.
Revised: Proof.

Original: The noises injection on the party side is similar to the PATE approach.
Revised: The noises injection on the party side is similar to the PATE approach.

Original: The sensitivity of the f function executed on the party side is 2.
Revised: The sensitivity of the f function executed on the party side is 2.

Original: Thus, we can get the theorem by combining Theoremand Theorem 6.
Revised: Thus, we can get the theorem by combining Theoremand Theorem 6.

Original: A. E XAMPLE -L EVEL D IFFERENTIAL P RIVACY A NALYSIS OF F ED KT-L 
Revised: A. E XAMPLE -L EVEL D IFFERENTIAL P RIVACY A NALYSIS OF F ED KT-L

Original: Here we analyze the example-level differential privacy of FedKT-L1.
Revised: Here we analyze the example-level differential privacy of FedKT-L1.

Original: If we change a single example of the original dataset (i.e., the union of all the local datasets), only a single party will be inﬂuenced.
Revised: If we change a single example of the original dataset (i.e., the union of all the local datasets), only a single party will be inﬂuenced.

Original: More precisely, for each partition of the party, only a single teacher model will be inﬂuenced.
Revised: More precisely, for each partition of the party, only a single teacher model will be inﬂuenced.

Original: Then, even though changing a single record, the student model is still unchanged if the top-2 vote counts of the teachers differ at least 2.
Revised: Then, even though changing a single record, the student model is still unchanged if the top-2 vote counts of the teachers differ at least 2.

Original: Thus, if not applying consistent voting, we have the following theorem.
Revised: Thus, if not applying consistent voting, we have the following theorem.

Original: Theorem 9.
Revised: Theorem 9.

Original: Let M be the f function executed in the server.
Revised: Let M be the f function executed in the server.

Original: Let q ≥ Pr[ M ( d ) (cid:54)
Revised: Let q ≥ Pr[ M ( d ) (cid:54)

Original: =
Revised: =

Original: o ∗ ] for some outcome o ∗ .
Revised: o ∗ ] for some outcome o ∗ .

Original: Let D aux denotes the query dataset.
Revised: Let D aux denotes the query dataset.

Original: Given a query i , suppose the top-2 vote counts are v i 1 and v i 2 .
Revised: Given a query i , suppose the top-2 vote counts are v i 1 and v i 2 .

Original: In party P i , let z i denotes the number of partitions that there ∃ q ∈ D aux such that v q 1 − v q 2 ≤ 1 when training the student model.
Revised: In party P i , let z i denotes the number of partitions that there ∃ q ∈ D aux such that v q 1 − v q 2 ≤ 1 when training the student model.

Original: Let z = max i z i .
Revised: Let z = max i z i .

Original: Let l, γ ≥ 0 and q < e 

Revised: Let l, γ ≥ 0 and q < e

Original: zγ − 1 e 4 zγ − 1 .
Revised: zγ − 1 e 4 zγ − 1 .

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Given a query dataset D aux , z is the number of partitions such that there exists a query that the top-2 vote counts differ at most 1.
Revised: Given a query dataset D aux , z is the number of partitions such that there exists a query that the top-2 vote counts differ at most 1.

Original: In other words, there are at most z student models will be changed if we change a single record of the original dataset.
Revised: In other words, there are at most z student models will be changed if we change a single record of the original dataset.

Original: Thus, the vote counts change by at most 2 z on the server side and M is (2 zγ, 0) -differentially private with respect to d and D aux .
Revised: Thus, the vote counts change by at most 2 z on the server side and M is (2 zγ, 0) -differentially private with respect to d and D aux .

Original: Then, we can get this theorem by replacing γ of Theorem 5 and Theorem 6 to zγ .
Revised: Then, we can get this theorem by replacing γ of Theorem 5 and Theorem 6 to zγ .

Original: Note that the example-level differential privacy is same as party-level differential privacy when z = s .
Revised: Note that the example-level differential privacy is same as party-level differential privacy when z = s .

Original: Also, if we applied consistent voting in FedKT-L1, the vote counts may change by s even if only a single student model is affected.
Revised: Also, if we applied consistent voting in FedKT-L1, the vote counts may change by s even if only a single student model is affected.

Original: Thus, the example-level differential privacy of FedKT-L1 is usually same as party-level differential privacy.
Revised: Thus, the example-level differential privacy of FedKT-L1 is usually same as party-level differential privacy.

Original: A. P ARTY -L EVEL D IFFERENTIAL P RIVACY A NALYSIS OF F ED KT-L 
Revised: A. P ARTY -L EVEL D IFFERENTIAL P RIVACY A NALYSIS OF F ED KT-L

Original: Here we study party-level differential privacy of FedKT-L2.
Revised: Here we study party-level differential privacy of FedKT-L2.

Original: Theorem 10.
Revised: Theorem 10.

Original: Let M be the f function executed on each partition of a party.
Revised: Let M be the f function executed on each partition of a party.

Original: Given the number of subsets in each partition t and the privacy parameter γ , M satisﬁes (2 tγ, 0) party-level differential privacy.
Revised: Given the number of subsets in each partition t and the privacy parameter γ , M satisﬁes (2 tγ, 0) party-level differential privacy.

Original: Proof.
Revised: Proof.

Original: Considering changing the whole local dataset, then t teachers will be inﬂuenced and the vote counts change by at most 2 t .
Revised: Considering changing the whole local dataset, then t teachers will be inﬂuenced and the vote counts change by at most 2 t .

Original: Thus, from a party-level perspective, the sensitivity of the vote counts is 2 t and M satisﬁes (2 tγ, 0) -differential privacy.
Revised: Thus, from a party-level perspective, the sensitivity of the vote counts is 2 t and M satisﬁes (2 tγ, 0) -differential privacy.

Original: Like Theorem 9, we have the following theorem to track the moments.
Revised: Like Theorem 9, we have the following theorem to track the moments.

Original: Theorem 11.
Revised: Theorem 11.

Original: Let M be the f function executed on the party side and
Revised: Let M be the f function executed on the party side and

Original: q ≥ Pr[ M ( d ) (cid:54) =
Revised: q ≥ Pr[ M ( d ) (cid:54) =

Original: o ∗ ] for some outcome o ∗ .
Revised: o ∗ ] for some outcome o ∗ .

Original: Suppose the number of subsets in a partition is t .
Revised: Suppose the number of subsets in a partition is t .

Original: Let l, γ ≥ 0 and q < e 2 tγ − 1 e 4 tγ − 1 .
Revised: Let l, γ ≥ 0 and q < e 2 tγ − 1 e 4 tγ − 1 .

Original: Then for any aux and any two party-adjacent datasets d and d (cid:48) , M satisﬁes
Revised: Then for any aux and any two party-adjacent datasets d and d (cid:48) , M satisﬁes

Original: [Equation]
Revised: [Equation]

Original: Note that the party-level privacy loss of FedKT-L2 can be quite large.
Revised: Note that the party-level privacy loss of FedKT-L2 can be quite large.

Original: To mitigate the impact of the noises, we usually expect t to be large to have a tighter bound of Pr[ M ( d ) (cid:54)
Revised: To mitigate the impact of the noises, we usually expect t to be large to have a tighter bound of Pr[ M ( d ) (cid:54)

Original: = o ∗ ] .
Revised: = o ∗ ] .

Original: However, when t is large, the bound in Theorem 11 is also large.
Revised: However, when t is large, the bound in Theorem 11 is also large.

Original: In fact, since every student model satisﬁes differential privacy, it is not necessary to apply party-level differential privacy in FedKT-L2.
Revised: In fact, since every student model satisﬁes differential privacy, it is not necessary to apply party-level differential privacy in FedKT-L2.

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: B E XPERIMENTS B. A DDITIONAL D ETAILS OF E XPERIMENTAL S ETTINGS
Revised: B E XPERIMENTS B. A DDITIONAL D ETAILS OF E XPERIMENTAL S ETTINGS

Original: We use Adult , cod-rna , MNIST
Revised: We use Adult , cod-rna , MNIST

Original: [Table]
Revised: [Table]

Original: and SVHN for our experiments, where Adult and cod-rna are downloaded from this link 2 .
Revised: and SVHN for our experiments, where Adult and cod-rna are downloaded from this link 2 .

Original: The details of the datasets are shown in Table 3.
Revised: The details of the datasets are shown in Table 3.

Original: Besides these four datasets, in section B.6, we also conduct experiments on CelebA with VGG-9, and chest X-ray datasets with
Revised: Besides these four datasets, in section B.7, we also conduct experiments on CelebA with VGG-9, and chest X-ray datasets with

Original: CNN.
Revised: CNN.

Original: We compare FedKT with the other six baselines.
Revised: We compare FedKT with the other six baselines.

Original: For FedAvg, FedProx, and SCAFFOLD, all parties participate in federated learning for each round (i.e., no party sampling).
Revised: For FedAvg, FedProx, and SCAFFOLD, all parties participate in federated learning for each round (i.e., no party sampling).

Original: The learning rate is tuned from { 0 . 001 , 0 . 01 } and the number of local epochs is tuned from { 10 , 20 , 40 } .
Revised: The learning rate is tuned from { 0 . 001 , 0 . 01 } and the number of local epochs is tuned from { 10 , 20 , 40 } .

Original: We found that the baselines can get best performance when the learning rate is set to 0.001 and the number of local epochs is set to 10.
Revised: We found that the baselines can get best performance when the learning rate is set to 0.001 and the number of local epochs is set to 10.

Original: For FedProx, the regularization term µ is tuned from { 0 . 1 , 1 } .
Revised: For FedProx, the regularization term µ is tuned from { 0 . 1 , 1 } .

Original: For SCAFFOLD, same as the experiments of the paper (Karimireddy et al., 2020), we use option II to calculate the control variates.
Revised: For SCAFFOLD, same as the experiments of the paper (Karimireddy et al., 2020), we use option II to calculate the control variates.

Original: For FedKT, SOLO, and PATE, the number of local epochs is simply set to 100.
Revised: For FedKT, SOLO, and PATE, the number of local epochs is simply set to 100.

Original: For
Revised: For

Original: PATE, the number of teacher models is set to be same as the number of parties of FedKT.
Revised: PATE, the number of teacher models is set to be same as the number of parties of FedKT.

Original: We use the Adam optimizer and the L 2 regularization is set to 10 − 6 .
Revised: We use the Adam optimizer and the L 2 regularization is set to 10 − 6 .

Original: The ﬁnal parameters are summarized in Table 4.
Revised: The ﬁnal parameters are summarized in Table 4.

Original: B. H YPER - PARAMETERS S TUDY B.2. N UMBER OF PARTITIONS IN EACH PARTY
Revised: B. H YPER - PARAMETERS S TUDY B.2. N UMBER OF PARTITIONS IN EACH PARTY

Original: Here we study the impact of the number of partitions (i.e., the parameter s ) on FedKT.
Revised: Here we study the impact of the number of partitions (i.e., the parameter s ) on FedKT.

Original: Tableshows the test accuracy of FedKT with different s .
Revised: Tableshows the test accuracy of FedKT with different s .

Original: From Table 5, we can see that the accuracy can be improved if we increase s from 1 to 2.
Revised: From Table 5, we can see that the accuracy can be improved if we increase s from 1 to 2.

Original: However, if we further increase s , there is little or no improvement on the accuracy while the communication and computation overhead is larger.
Revised: However, if we further increase s , there is little or no improvement on the accuracy while the communication and computation overhead is larger.

Original: Thus, from our empirical study, we suggest users to simply set s to 2 for FedKT-L0 if they do not want to tune the parameters.
Revised: Thus, from our empirical study, we suggest users to simply set s to 2 for FedKT-L0 if they do not want to tune the parameters.

Original: For FedKT-L1 and FedKT-L2, since the privacy loss increases as s increases, we suggest to set s to small values.
Revised: For FedKT-L1 and FedKT-L2, since the privacy loss increases as s increases, we suggest to set s to small values.

Original: Users can simply set s to 1 or tune s from small values (i.e.,or 2) to ﬁnd a best accuracy-privacy trade-off.
Revised: Users can simply set s to 1 or tune s from small values (i.e.,or 2) to ﬁnd a best accuracy-privacy trade-off.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: B.2. N UMBER OF TEACHERS IN EACH PARTITION
Revised: B.2. N UMBER OF TEACHERS IN EACH PARTITION

Original: Here we study the impact of number of teachers in each partition (i.e., the parameter t ).
Revised: Here we study the impact of number of teachers in each partition (i.e., the parameter t ).

Original: Tableshows the test accuracy of FedKT with different t .
Revised: Tableshows the test accuracy of FedKT with different t .

Original: As we can see, FedKT can always get the best performance if setting t to 5.
Revised: As we can see, FedKT can always get the best performance if setting t to 5.

Original: If t is large, the size of each data subset is small and the teacher models may not be good at predicting the public dataset.
Revised: If t is large, the size of each data subset is small and the teacher models may not be good at predicting the public dataset.

Original: From our empirical study, users can simply set t to 5 if they do not want to tune the parameter.
Revised: From our empirical study, users can simply set t to 5 if they do not want to tune the parameter.

Original: However, if the student models need to satisfy differential privacy (i.e., in FedKT-L2), the privacy loss may potentially be smaller if we increase t according to Lemma 7.
Revised: However, if the student models need to satisfy differential privacy (i.e., in FedKT-L2), the privacy loss may potentially be smaller if we increase t according to Lemma 7.

Original: Users need to tune t to ﬁnd the best trade-off between the performance and the privacy loss.
Revised: Users need to tune t to ﬁnd the best trade-off between the performance and the privacy loss.

Original: B. I MBALANCE LEVEL
Revised: B. I MBALANCE LEVEL

Original: Here we study the impact of the concentration parameter β of the Dirichlet distribution in our heterogeneous partition.
Revised: Here we study the impact of the concentration parameter β of the Dirichlet distribution in our heterogeneous partition.

Original: If β is smaller, the data among different parties tend to be more unbalanced.
Revised: If β is smaller, the data among different parties tend to be more unbalanced.

Original: We study three different values: 0.1, 0.5, and 10.
Revised: We study three different values: 0.1, 0.5, and 10.

Original: The results are shown in Table 7.
Revised: The results are shown in Table 7.

Original: Given different β values, the performance of FedKT is always better than FedAvg and FedProx given limited communication size (i.e., 2 rounds).
Revised: Given different β values, the performance of FedKT is always better than FedAvg and FedProx given limited communication size (i.e., 2 rounds).

Original: Also, FedKT is quite stable compared with the other baselines.
Revised: Also, FedKT is quite stable compared with the other baselines.

Original: The results show that FedKT has a good performance under the heterogeneous data partition.
Revised: The results show that FedKT has a good performance under the heterogeneous data partition.

Original: FedKT is superior than the other approaches especially when the imbalance level of data distribution is high.
Revised: FedKT is superior than the other approaches especially when the imbalance level of data distribution is high.

Original: B. N UMBER OF PARTIES
Revised: B. N UMBER OF PARTIES

Original: We change the number of parties and compare different approaches.
Revised: We change the number of parties and compare different approaches.

Original: The results are shown in Table 8 and Table 9.
Revised: The results are shown in Table 8 and Table 9.

Original: FedKT is quite stable with different number of parties compared with the other baselines.
Revised: FedKT is quite stable with different number of parties compared with the other baselines.

Original: Moreover, FedKT is superior than FedAvg, FedProx, and SCAFFOLD with limited communication size in all cases.
Revised: Moreover, FedKT is superior than FedAvg, FedProx, and SCAFFOLD with limited communication size in all cases.

Original: Even though running the baselines for 50 rounds, the accuracy of
Revised: Even though running the baselines for 50 rounds, the accuracy of

Original: FedKT is still comparable.
Revised: FedKT is still comparable.

Original: B. E FFECT OF C ONSISTENT V OTING
Revised: B. S IZE OF THE P UBLIC D ATASET

Revised: We show the performance of FedKT with different size of public dataset.

Revised: The results are shown in Table 10.

Revised: We can observe that FedKT is stable even though reducing the size of the public dataset.

Revised: The accuracy decreases no more than 1% and 2% using only 20% of the public dataset on

Revised: [Table]

Revised: [Table]

Revised: cod-rna and MNIST (i.e., 1807 examples on cod-rna and 1000 examples on MNIST), respectively.

Revised: Moreover, the accuracy is almost unchanged on Adult.

Revised: B. E FFECT OF C ONSISTENT V OTING

Original: Here we study the effect of our consistent voting technique.
Revised: Here we study the effect of our consistent voting technique.

Original: The results are shown in Table 10.
Revised: The results are shown in Table 11.

Original: From the table, we can observe that the accuracy can be improved about 1%-2.3% by applying consistent voting.
Revised: From the table, we can observe that the accuracy can be improved about 1%-2.3% by applying consistent voting.

Original: Thus, consistent voting is a simple and effective technique.[Table] [Table]
Revised: Thus, consistent voting is a simple and effective technique.

Original: B. E XPERIMENTS ON A DDITIONAL D ATASETS
Revised: B. E XPERIMENTS ON A DDITIONAL D ATASETS

Original: We conduct experiments on two additional datasets: CelebA and chest X-ray images.
Revised: We conduct experiments on two additional datasets: CelebA and chest X-ray images.

Original: We conduct a gender recognition task on the CelebA dataset.
Revised: We conduct a gender recognition task on the CelebA dataset.

Original: Like MNIST, we use one half of the original test dataset as the public dataset and the remaining as the ﬁnal test dataset.
Revised: Like MNIST, we use one half of the original test dataset as the public dataset and the remaining as the ﬁnal test dataset.

Original: The number of parties is set to 10.
Revised: The number of parties is set to 10.

Original: The details of CelebA and the model parameters are shown in Table 11.
Revised: The details of CelebA and the model parameters are shown in Table 12.

Original: The model structure of VGG-9 is shown in Table 12.
Revised: The model structure of VGG-9 is shown in Table 13.

Original: We use two chest X-ray image datasets to simulate federated learning as a real-world scenario.
Revised: We use two chest X-ray image datasets to simulate federated learning as a real-world scenario.

Original: Speciﬁcally, we use Kaggle Chest X-Ray Images 3 as the private dataset and partition it into 5 parties.
Revised: Speciﬁcally, we use Kaggle Chest X-Ray Images 3 as the private dataset and partition it into 5 parties.

Original: We use RSNA Pneumonia Detection Challenge dataset 4 as the public dataset for our FedKT algorithm.
Revised: We use RSNA Pneumonia Detection Challenge dataset 4 as the public dataset for our FedKT [Table] [Table] [Table]

Original:  We use a CNN model (same as the model used in MNIST and SVHN) to predict pneumonia.
Revised: [Table] algorithm. We use a CNN model (same as the model used in MNIST and SVHN) to predict pneumonia.

Original: The number of parties is set to 5.
Revised: The number of parties is set to 5.

Original: The details of X-Ray task are shown in Table 11.
Revised: The details of X-Ray task are shown in Table 12.

Original: We try the heterogeneous partition and the experiment results are shown in Table 13.
Revised: We try the heterogeneous partition and the experiment results are shown in Table 14.

Original: For CelebA,
Revised: For CelebA,

Original: FedKT shows a good performance on a relatively large dataset and deep model.
Revised: FedKT shows a good performance on a relatively large dataset and deep model.

Original: FedKT is better than FedProx and SCAFFOLD even with 50 rounds.
Revised: FedKT is better than FedProx and SCAFFOLD even with 50 rounds.

Original: Also, the accuracy of FedKT is comparable to
Revised: Also, the accuracy of FedKT is comparable to

Original: FedAvg with 50 rounds.
Revised: FedAvg with 50 rounds.

Original: For X-Ray, FedKT has a much better performance than the other federated learning approaches.
Revised: For X-Ray, FedKT has a much better performance than the other federated learning approaches.

Original: FedAvg, FedProx, and SCAFFOLD are quite unstable.[Table] [Table]
Revised: FedAvg, FedProx, and SCAFFOLD are quite unstable.

Original: Figure 3 shows the test accuracy changing the number of communication rounds.
Revised: Figure 3 shows the test accuracy changing the number of communication rounds.

Original: We can ﬁnd that
Revised: We can ﬁnd that

Original: FedProx and SCAFFOLD are quite unstable on these two tasks.
Revised: FedProx and SCAFFOLD are quite unstable on these two tasks.

Original: For CelebA, FedKT outperforms
Revised: For CelebA, FedKT outperforms

Original: FedProx and SCAFFOLD, while FedAvg needs about 25 rounds to achieve a better accuracy than
Revised: FedProx and SCAFFOLD, while FedAvg needs about 25 rounds to achieve a better accuracy than

Original: FedKT.
Revised: FedKT.

Original: This experiments demonstrate the applicability of FedKT on more complex models.
Revised: This experiments demonstrate the applicability of FedKT on more complex models.

Original: For
Revised: For

Original: X-ray, FedKT outperforms the other approaches.
Revised: X-ray, FedKT outperforms the other approaches.

Original: This experiments show that FedKT does not have strict limitation on the distribution of the public dataset.
Revised: This experiments show that FedKT does not have strict limitation on the distribution of the public dataset.

Original: Even though the public dataset is another dataset differ from the training dataset, which is not partitioned from the global dataset differ from previous experiments, FedKT still has a quite good performance.
Revised: Even though the public dataset is another dataset differ from the training dataset, which is not partitioned from the global dataset differ from previous experiments, FedKT still has a quite good performance.

Original: B. D IFFERENTIAL P RIVACY
Revised: B. D IFFERENTIAL P RIVACY

Original: Table 14 and Table 15 present the results of FedKT-L1 and FedKT-L2.
Revised: Table 15 and Table 16 present the results of FedKT-L1 and FedKT-L2.

Original: Besides the heterogeneous partition ( β = 0 . 5 ), we also try homogeneous partition (i.e., the dataset is randomly and equally partitioned into the parties).
Revised: Besides the heterogeneous partition ( β = 0 . 5 ), we also try homogeneous partition (i.e., the dataset is randomly and equally partitioned into the parties).

Original: From the tables, we can see that the accuracy of FedKT-L1 and FedKTL2 are comparable to the non-private version with a modest privacy budget.
Revised: From the tables, we can see that the accuracy of FedKT-L1 and FedKTL2 are comparable to the non-private version with a modest privacy budget.

Original: Moreover, the moments accountant method usually can achieve a tighter privacy loss than the advanced composition (Dwork et al., 2014).
Revised: Moreover, the moments accountant method usually can achieve a tighter privacy loss than the advanced composition (Dwork et al., 2014).

Original: For example, if we run cod-rna under homogeneous data partition setting γ = 0 .and
Revised: For example, if we run cod-rna under homogeneous data partition setting γ = 0 .and

Original: the fraction of queries to 1%, the advanced composition gives us ε ≈ 20 . 2 and our analysis gives ε ≈ 11 .2 .
Revised: the fraction of queries to 1%, the advanced composition gives us ε ≈ 20 . 2 and our analysis gives ε ≈ 11 .

Original: Note that the techniques in Papernot et al.

Original: (2018) can also be applied to FedKT.
Revised: the techniques can also be applied to FedKT.

Original: For example, we may get a smaller privacy loss if adopting Gaussian noises instead of Laplace
Revised: For example, we may get a smaller privacy loss if adopting Gaussian noises instead of Laplace

Original: [Table]
Revised: [Figure]

Original:  [Table]
Revised: CelebA Two chest [Table] [Table]

Revised: image datasets #

Original: [Table]
Revised: [Table]

Revised: examples  #Public examples

Original: [Table] 
Revised: [Table] [Table]

Revised: #Classes

Original: [Figure]
Revised: [Table] Parties Model VGGCNN

Revised: Implemented library

Revised: PyTorch

Original: A cc u r a c y
Revised: [Table] [Table] [Table] Batch size  #Local epochs for FedAvg/FedProx/SCAFFOLD

Original: [Figure]
Revised: [Figure]

Revised: #Epochs for FedKT/SOLO/PATE

Revised: [Figure]

Revised: [Figure]

Revised: [Table]

Revised: [Table]

Original: noises.
Revised: noises.

Original: Generally, our framework can also beneﬁt from the state-of-the-art approaches on the privacy analysis of PATE, which we may investigate in the future
Revised: Generally, our framework can also beneﬁt from the state-of-the-art approaches on the privacy analysis of PATE, which we may investigate in the future.

Original: [Figure]

Original: [Table]
Revised: [Table]

Original: Homogeneous 0.
Revised: Homogeneous 0.

Original: [Table]
Revised: [Table]

Original: Homogeneous 0.
Revised: Homogeneous 0.