Original: E XPLOIT U NLABELED D ATA ON THE S ERVER ! F EDERATED L EARNING VIA U NCERTAINTY - AWARE E NSEMBLE D ISTILLATION AND S ELF -S UPERVISION
Revised: E XPLOIT U NLABELED D ATA ON THE S ERVER ! F EDERATED L EARNING VIA U NCERTAINTY - AWARE E NSEMBLE D ISTILLATION AND S ELF -S UPERVISION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Federated Learning (FL) is a distributed machine learning paradigm that involves the cooperation of multiple clients to train a server model.
Revised: Federated Learning (FL) is a distributed machine learning paradigm that involves the cooperation of multiple clients to train a server model.

Original: In practice, it is hard to assume that each client possesses large-scale data or many clients are always available to participate in FL for the same round, which may lead to data deficiency.
Revised: In practice, it is hard to assume that each client possesses large-scale data or many clients are always available to participate in FL for the same round, which may lead to data deficiency.

Original: This deficiency degrades the entire learning process.
Revised: This deficiency degrades the entire learning process.

Original: To resolve this challenge, we propose a Federated learning with entropy-weighted ensemble Distillation and
Revised: To resolve this challenge, we propose a Federated learning with entropy-weighted ensemble Distillation and

Original: Self-supervised learning (FedDS).
Revised: Self-supervised learning (FedDS).

Original: FedDS reliably deals with situations where not only the amount of data per client but also the number of clients is scarce.
Revised: FedDS reliably deals with situations where not only the amount of data per client but also the number of clients is scarce.

Original: This advantage is achieved by leveraging the prevalent unlabeled data in the server.
Revised: This advantage is achieved by leveraging the prevalent unlabeled data in the server.

Original: We demonstrate the effectiveness of FedDS on classification tasks for CIFAR-10/ and PathMNIST.
Revised: We demonstrate the effectiveness of FedDS on classification tasks for CIFAR-10/ and PathMNIST.

Original: In CIFAR-10, our method shows the improvement over FedAVG by 12.54% in data deficient regime, and by 17.16% and 23.56% in more challenging scenarios of noisy label or Byzantine client cases, respectively.
Revised: In CIFAR-10, our method shows the improvement over FedAVG by 12.54% in data deficient regime, and by 17.16% and 23.56% in more challenging scenarios of noisy label or Byzantine client cases, respectively.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: [Figure]
Revised: [Figure]

Original: Federated Learning (FL) is a distributed machine learning paradigm that involves the cooperation of multiple clients to train a server model (McMahan et al., 2017).
Revised: Federated Learning (FL) is a distributed machine learning paradigm that involves the cooperation of multiple clients to train a server model (McMahan et al., 2017).

Original: In FL, a server model is trained as follows: 1) the server distributes the current server model to clients, 2) each client independently trains each model downloaded from the server with their available local data and sends back the resultant model to the server, 3) the server updates the server model with the collected locally-trained models, and 4) repeat the steps.
Revised: In FL, a server model is trained as follows: 1) the server distributes the current server model to clients, 2) each client independently trains each model downloaded from the server with their available local data and sends back the resultant model to the server, 3) the server updates the server model with the collected locally-trained models, and 4) repeat the steps.

Original: By collecting updated model parameters at the server instead of raw client data, FL can mitigate the personal information leakage.
Revised: By collecting updated model parameters at the server instead of raw client data, FL can mitigate the personal information leakage.

Original: In some FL scenarios, such as developing a medical diagnosis algorithm, it is often the case where the number of clients (participating hospitals) and the size of labeled datasets in each client (the number of relevant patients and those labels at each hospital) are deficient.
Revised: In some FL scenarios, such as developing a medical diagnosis algorithm, it is often the case where the number of clients (participating hospitals) and the size of labeled datasets in each client (the number of relevant patients and those labels at each hospital) are deficient.

Original: Such a lack of participating clients and labeled datasets in the clients leads to performance degradation for the standard
Revised: Such a lack of participating clients and labeled datasets in the clients leads to performance degradation for the standard

Original: FL method, e.g ., FedAVG (McMahan et al., 2017) (refer to Fig. 1).
Revised: FL method, e.g ., FedAVG (McMahan et al., 2017) (refer to Fig. 1).

Original: The deficiency may also destabilize the learning process which increases label noise sensitivity of FL methods.
Revised: The deficiency may also destabilize the learning process which increases label noise sensitivity of FL methods.

Original: Even in such scenarios, unlabeled data is abundant or easy to collect in practice, which may help to mitigate the data deficiency and label noise vulnerability of FL algorithms.
Revised: Even in such scenarios, unlabeled data is abundant or easy to collect in practice, which may help to mitigate the data deficiency and label noise vulnerability of FL algorithms.

Original: In this paper, we propose a robust FL algorithm by utilizing additional unlabeled data on the server.
Revised: In this paper, we propose a robust FL algorithm by utilizing additional unlabeled data on the server.

Original: The key idea of our method is to leverage unlabeled data to mitigate lack of data as well as to reliably aggregate the client models into the single server model by unsupervised ensemble knowledge distillation.
Revised: The key idea of our method is to leverage unlabeled data to mitigate lack of data as well as to reliably aggregate the client models into the single server model by unsupervised ensemble knowledge distillation.

Original: We postulate that the aforementioned degradation of the accuracy mainly stems from unreliable clients participated in the server model update.
Revised: We postulate that the aforementioned degradation of the accuracy mainly stems from unreliable clients participated in the server model update.

Original: To mitigate the influence of unreliable clients, we measure the entropy of each client’s output to assess the uncertainty of each model, and use them to weigh for each client model.
Revised: To mitigate the influence of unreliable clients, we measure the entropy of each client’s output to assess the uncertainty of each model, and use them to weigh for each client model.

Original: We found that this simple entropy measure is sufficiently well-calibrated to train the server model better.
Revised: We found that this simple entropy measure is sufficiently well-calibrated to train the server model better.

Original: Thereby, we suppress unreliable clients’ contribution when aggregating them.
Revised: Thereby, we suppress unreliable clients’ contribution when aggregating them.

Original: Furthermore, our proposed entropy-weighted ensemble distillation (EED) is performed jointly with another self-supervised learning (SSL) loss with unlabeled data on the server.
Revised: Furthermore, our proposed entropy-weighted ensemble distillation (EED) is performed jointly with another self-supervised learning (SSL) loss with unlabeled data on the server.

Original: With our setting, while additionally imposing a SSL loss is simple and demands negligible overheads, it is found to be crucial and have several benefits, such as faster convergence and reducing the influence of unreliable nodes.
Revised: With our setting, while additionally imposing a SSL loss is simple and demands negligible overheads, it is found to be crucial and have several benefits, such as faster convergence and reducing the influence of unreliable nodes.

Original: Our major contributions We propose Fed erated learning with entropy-weighted ensemble
Revised: Our major contributions We propose Fed erated learning with entropy-weighted ensemble

Original: D istillation and S elf-supervised learning ( FedDS ), a method for reliably updating the server model by utilizing the server’s unlabeled data in an unsupervised way.
Revised: D istillation and S elf-supervised learning ( FedDS ), a method for reliably updating the server model by utilizing the server’s unlabeled data in an unsupervised way.

Original: We demonstrate through experiments that our FedDS outperforms several strong baselines in the classification tasks for CIFAR-10/100 and
Revised: We demonstrate through experiments that our FedDS outperforms several strong baselines in the classification tasks for CIFAR-10/100 and

Original: PathMNIST, especially in data deficient regime, and is also robust to various tough scenarios with unreliable clients.
Revised: PathMNIST, especially in data deficient regime, and is also robust to various tough scenarios with unreliable clients.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Since the emergence of FL paradigm (McMahan et al., 2017), there have been many follow-up studies to tackle various challenges in FL: enhancing communication efficiency (Alistarh et al., 2017; Suresh et al., 2017; Bernstein et al., 2018; Tang et al., 2018; 
Revised: Since the emergence of FL paradigm (McMahan et al., 2017), there have been many follow-up studies to tackle various challenges in FL: enhancing communication efficiency (Alistarh et al., 2017; Suresh et al., 2017; Bernstein et al., 2018; Tang et al., 2018; Wu et al., 2018;

Original: Wu et al., 2018; Hamer et al., 2020; Rothchild et al., 2020; Reisizadeh et al., 2020; Haddadpour et al., 2021; 
Revised: Hamer et al., 2020; Rothchild et al., 2020; Reisizadeh et al., 2020; Haddadpour et al., 2021; Qiao et al., 2021; Kone ˇ cn ` y et al., 2016;

Original: Qiao et al., 2021; Kone ˇ cn ` y et al., 2016; Hyeon-Woo et al., 2022; Jeong et al., 2018), stabilizing convergence and solving various issues arising from heterogeneity in client’s data (Karimireddy et al., 2020; Acar et al., 2021; Reddi et al., 2021; Yuan & Ma, 2020) and in client’s model structure (Diao et al., 2021b), protecting clients from privacy attacks (Ammad-Ud-Din et al., 2019;
Revised: Hyeon-Woo et al., 2022; Jeong et al., 2018), stabilizing convergence and solving various issues arising from heterogeneity in client’s data (Li et al., 2020; Karimireddy et al., 2020; Acar et al., 2021; Reddi et al., 2021; Yuan & Ma, 2020) and in client’s model structure (Diao et al., 2021b), protecting clients from privacy attacks (Ammad-Ud-Din et al., 2019;

Original: Gong et al., 2021), and effectively aggregating unreliable client models containing malicious clients (Chang et al., 2019).
Revised: Gong et al., 2021), and effectively aggregating unreliable client models containing malicious clients (Chang et al., 2019).

Original: In particular, the use of extra data, in addition to local data available at each client, has been shown to be effective to deal with some of the aforementioned challenges in FL (Chang et al., 2019;
Revised: In particular, the use of extra data, in addition to local data available at each client, has been shown to be effective to deal with some of the aforementioned challenges in FL (Chang et al., 2019;

Original: Lin et al., 2020; Li & Wang, 2019).
Revised: Lin et al., 2020; Li & Wang, 2019).

Original: In this section, we focus on briefly reviewing two main research streams utilizing additional data in FL, which are closely related to our work, i.e ., utilizing unlabeled data on the server.
Revised: In this section, we focus on briefly reviewing two main research streams utilizing additional data in FL, which are closely related to our work, i.e ., utilizing unlabeled data on the server.

Original: For comprehensive review of FL, please refer to Kairouz et al.
Revised: For comprehensive review of FL, please refer to Kairouz et al.

Original: Knowledge distillation in federated learning Knowledge distillation (KD) is used to transfer knowledge of a teacher model to a student model (Xie et al., 2020;
Revised: Knowledge distillation in federated learning Knowledge distillation (KD) is used to transfer knowledge of a teacher model to a student model (Xie et al., 2020;

Original: Zhou et al., 2021; Radosavovic et al., 2018;
Revised: Zhou et al., 2021; Radosavovic et al., 2018;

Original: Kim et al., 2021).
Revised: Kim et al., 2021).

Original: The teacher model is used to provide pseudo labels to either labeled or unlabeled data, and the student model learns to mimic the teacher’s behavior by using these pseudo-labeled data as supervision.
Revised: The teacher model is used to provide pseudo labels to either labeled or unlabeled data, and the student model learns to mimic the teacher’s behavior by using these pseudo-labeled data as supervision.

Original: To adapt KD for FL, data to be pseudo labeled should be presented at the point of aggregation (the server); thereby, we can extract knowledge that will be transferred to the aggregated model (the server model).
Revised: To adapt KD for FL, data to be pseudo labeled should be presented at the point of aggregation (the server); thereby, we can extract knowledge that will be transferred to the aggregated model (the server model).

Original: However, if the data to be distilled is provided from clients to the server, FL loses the privacy and security property which is one of the big advantages of FL.
Revised: However, if the data to be distilled is provided from clients to the server, FL loses the privacy and security property which is one of the big advantages of FL.

Original: To address this, there are studies exploiting the assumptions that transferring features and logits from client’s data is allowed (He et al., 2020;
Revised: To address this, there are studies exploiting the assumptions that transferring features and logits from client’s data is allowed (He et al., 2020;

Original: Gong et al., 2021) or assumptions that additional datasets exist (Gong et al., 2021; Chang et al., 2019; Lin et al., 2020; Li & Wang, 2019; Shi et al., 2021).
Revised: Gong et al., 2021) or assumptions that additional datasets exist (Gong et al., 2021; Chang et al., 2019; Lin et al., 2020; Li & Wang, 2019; Shi et al., 2021).

Original: The latter assumes that an extra public dataset is shared across all clients .
Revised: The latter assumes that an extra public dataset is shared across all clients .

Original: Then, KD is performed at the server by collecting the logit values for the public dataset from the clients, instead of model parameters.
Revised: Then, KD is performed at the server by collecting the logit values for the public dataset from the clients, instead of model parameters.

Original: This approach can achieve higher communication efficiency and privacy protection.
Revised: This approach can achieve higher communication efficiency and privacy protection.

Original: Our work relaxes the assumption by possessing extra unlabeled data only in the server , rather than sharing across all clients.
Revised: Our work relaxes the assumption by possessing extra unlabeled data only in the server , rather than sharing across all clients.

Original: As a similar setting to ours, Lin et al.
Revised: As a similar setting to ours, Lin et al.

Original: (2020) proposes a FL method, called FedDF.
Revised: (2020) proposes a FL method, called FedDF.

Original: In FedDF, the server obtains pseudo labels for the unlabeled data with the collected client models, and performs KD with those pseudo labels.
Revised: In FedDF, the server obtains pseudo labels for the unlabeled data with the collected client models, and performs KD with those pseudo labels.

Original: They show that their approach is robust to non-i.i.d.
Revised: They show that their approach is robust to non-i.i.d.

Original: data, achieving a higher accuracy than FedAVG (McMahan et al., 2017).
Revised: data, achieving a higher accuracy than FedAVG (McMahan et al., 2017).

Original: We further take into account confidence of each client and core knowledge of the server by measuring uncertainty of client predictions and applying self-supervised learning (SSL), respectively.
Revised: We further take into account confidence of each client and core knowledge of the server by measuring uncertainty of client predictions and applying self-supervised learning (SSL), respectively.

Original: Thereby, our proposed FedDS enhances both reliability and robustness of the learned model against the data deficiency and label noise vulnerability.
Revised: Thereby, our proposed FedDS enhances both reliability and robustness of the learned model against the data deficiency and label noise vulnerability.

Original: Semi-supervised federated learning Unlabeled data is much more prevalent than labeled ones.
Revised: Semi-supervised federated learning Unlabeled data is much more prevalent than labeled ones.

Original: Semi-supervised learning approaches in FL (Shi et al., 2021; Jeong et al., 2021;
Revised: Semi-supervised learning approaches in FL (Shi et al., 2021; Jeong et al., 2021;

Original: Zhang et al., 2021;
Revised: Zhang et al., 2021;

Original: Diao et al., 2021a; Lin et al., 2021) have been studied to leverage these prevalent unlabeled data.
Revised: Diao et al., 2021a; Lin et al., 2021) have been studied to leverage these prevalent unlabeled data.

Original: All of these studies consider scenarios where each client has few labeled data but instead has a large amount of unlabeled data, and exploits the unlabeled data with unsupervised learning losses.
Revised: All of these studies consider scenarios where each client has few labeled data but instead has a large amount of unlabeled data, and exploits the unlabeled data with unsupervised learning losses.

Original: Unlike these preceding studies, our work considers the scenario where unlabeled data is leveraged on the server side while clients have limited labeled data, which is a more practical setting.
Revised: Unlike these preceding studies, our work considers the scenario where unlabeled data is leveraged on the server side while clients have limited labeled data, which is a more practical setting.

Original:  M ETHOD 3. P ROBLEM S ETTING
Revised:  M ETHOD 3. P ROBLEM S ETTING

Original: We assume a set of C clients with client index c ∈ { 1 , 2 , ..., C } and a single server.
Revised: We assume a set of C clients with client index c ∈ { 1 , 2 , ..., C } and a single server.

Original: Each client with index c has its own labeled dataset D c
Revised: Each client with index c has its own labeled dataset D c

Original: = { x i , y i } N c
Revised: = { x i , y i } N c

Original: i =1 , where ( x i , y i ) ∈
Revised: i =1 , where ( x i , y i ) ∈

Original: X × Y for each i , x i is input data instance and y i is its corresponding label in the one-hot vector form.
Revised: X × Y for each i , x i is input data instance and y i is its corresponding label in the one-hot vector form.

Original: Client data distributions can be i.i.d.
Revised: Client data distributions can be i.i.d.

Original: (each client has the same data distribution), or non-i.i.d.
Revised: (each client has the same data distribution), or non-i.i.d.

Original: (each client has different data distribution with others).
Revised: (each client has different data distribution with others).

Original: In our setting, the central server has an unlabeled dataset D s =
Revised: In our setting, the central server has an unlabeled dataset D s =

Original: { x i } N s i =1 , where x i ∈ X for each i .
Revised: { x i } N s i =1 , where x i ∈ X for each i .

Original: The training process is iterative.
Revised: The training process is iterative.

Original: The server model is learned by repeating the client update and the server update for each communication round t ∈ { 1 , 2 , ..., T } .
Revised: The server model is learned by repeating the client update and the server update for each communication round t ∈ { 1 , 2 , ..., T } .

Original: [Figure]
Revised: [Figure]

Original: Client update At round t , each client c receives the server model f t − 1 s from the previous round t − 1 and initializes its client model with it.
Revised: Client update At round t , each client c receives the server model f t − 1 s from the previous round t − 1 and initializes its client model with it.

Original: Then, each client c trains its client model in a supervised way with each client’s own labeled dataset D c ; e.g ., with the cross-entropy loss l , each client c trains its model by minimizing the empirical loss f tc =
Revised: Then, each client c trains its client model in a supervised way with each client’s own labeled dataset D c ; e.g ., with the cross-entropy loss l , each client c trains its model by minimizing the empirical loss f tc =

Original: arg min
Revised: arg min

Original: f L c ( f ) for E c epochs, where L c ( f ) :=
Revised: f L c ( f ) for E c epochs, where L c ( f ) :=

Original: E ( x,y )
Revised: E ( x,y )

Original: ∼D c [ l ( f ( x ) , y )] .
Revised: ∼D c [ l ( f ( x ) , y )] .

Original: Then, all clients transfer their models { f tc } Cc = to the server.
Revised: Then, all clients transfer their models { f tc } Cc = to the server.

Original: Server update In the server update phase, at round t , the server aggregates the client models { f tc } Cc =1 into a single server model f ts .
Revised: Server update In the server update phase, at round t , the server aggregates the client models { f tc } Cc =1 into a single server model f ts .

Original: Distinctively, we aggregate the client models by leveraging unlabeled server dataset D s .
Revised: Distinctively, we aggregate the client models by leveraging unlabeled server dataset D s .

Original: Designing such a model aggregation method is our goal of this work.
Revised: Designing such a model aggregation method is our goal of this work.

Original: After we establish the server model f ts , the model is distributed to all the clients.
Revised: After we establish the server model f ts , the model is distributed to all the clients.

Original: 3. F EDERATED L EARNING VIA U NCERTAINTY - AWARE E NSEMBLE D ISTILLATION AND S ELF - SUPERVISION
Revised: 3. F EDERATED L EARNING VIA U NCERTAINTY - AWARE E NSEMBLE D ISTILLATION AND S ELF - SUPERVISION

Original: We focus on the client model aggregation step in the server update phase.
Revised: We focus on the client model aggregation step in the server update phase.

Original: In the server update phase, the server model starts either from the previous model or from the parameter average of client models.
Revised: In the server update phase, the server model starts either from the previous model or from the parameter average of client models.

Original: Then, our FedDS (Alg. 1) aggregates client models’ knowledge via an ensemble distillation that aggregates client models while taking into account uncertainty of each client model per sample as a prediction quality, called entropy-weighted ensemble distillation (EED).
Revised: Then, our FedDS (Alg. 1) aggregates client models’ knowledge via an ensemble distillation that aggregates client models while taking into account uncertainty of each client model per sample as a prediction quality, called entropy-weighted ensemble distillation (EED).

Original: We also propose to exploit unlabeled server dataset by self-supervised learning (SSL), so that the server model can further learn generic knowledge about the task.
Revised: We also propose to exploit unlabeled server dataset by self-supervised learning (SSL), so that the server model can further learn generic knowledge about the task.

Original: Both our proposed methods exploit unlabeled dataset.
Revised: Both our proposed methods exploit unlabeled dataset.

Original: This stabilizes and better poses the server model as better initial points to clients at every round.
Revised: This stabilizes and better poses the server model as better initial points to clients at every round.

Original: We simultaneously conducts EED and SSL with the unlabeled server data.
Revised: We simultaneously conducts EED and SSL with the unlabeled server data.

Original: Entropy-weighted ensemble distillation (EED)
Revised: Entropy-weighted ensemble distillation (EED)

Original: A pseudo label l p for each unlabeled data x is generated by normalizing the entropy-weighted sum of predictions of each client model as:
Revised: A pseudo label l p for each unlabeled data x is generated by normalizing the entropy-weighted sum of predictions of each client model as:

Original: [Equation]
Revised: [Equation]

Original: Input : D s , D c for c = 1 , .
Revised: Input : D s , D c for c = 1 , .

Original: C
Revised: C

Original: Randomly initialize f 01 , . . .
Revised: Randomly initialize f 01 , . . .

Original: f 0 C , f 0 s for round t = 1 , . . .
Revised: f 0 C , f 0 s for round t = 1 , . . .

Original: T do for client c = 1 . . .
Revised: T do for client c = 1 . . .

Original: C in parallel do f tc ← ClientUpdate
Revised: C in parallel do f tc ← ClientUpdate

Original: ( f t − 1 s c ◦ f t − 1 s f , D c ) endif B AV G = true then f t s ← 1 C (cid:88) C c =1 f t c / * start from average of client models * / else f ts ← f t − 1 s /* start from previous server model */ endfor epoch
Revised: ( f t − 1 s c ◦ f t − 1 s f , D c ) endif B AV G = true then f t s ← 1 C (cid:88) C c =1 f t c / * start from average of client models * / else f ts ← f t − 1 s /* start from previous server model */ endfor epoch

Original: e = 1 , . . .
Revised: e = 1 , . . .

Original: E s do for each input data x ∈ D s do l p ← CreatePseudoLabel ( f t 1 , . . .
Revised: E s do for each input data x ∈ D s do l p ← CreatePseudoLabel ( f t 1 , . . .

Original: , f tC , x )
Revised: , f tC , x )

Original: L CE ← CrossEntropy ( l p , ( f ts c ◦ f ts f )( x ))
Revised: L CE ← CrossEntropy ( l p , ( f ts c ◦ f ts f )( x ))

Original: L SSL ← GetSelfSupervisedLoss ( f ts s ◦ f ts f , x )
Revised: L SSL ← GetSelfSupervisedLoss ( f ts s ◦ f ts f , x )

Original: Optimize f ts to minimize L CE + γ L SSL /* using any optimizer */ end end end return f
Revised: Optimize f ts to minimize L CE + γ L SSL /* using any optimizer */ end end end return f

Original: Ts c ◦ f
Revised: Ts c ◦ f

Original: Ts f 0 .0 .0 [-.
-]{+.0 .0 .1 . 
+}
Revised: Ts f 0 .0 .0 [-.
-]{+.0 .0 .1 . 
+}

Original: Entropy-weight 0 .0 .0  .
Revised: Weight 0 .0 .0 .0 .1 .

Original: S o f t m a x v a l u e f o r t
Revised: S o f t m a x v a l u e f o r t

Original: r u e c l a ss  where H ( p ) =
Revised: r u e c l a ss Entropyweighting Maxpredictionweighting where H ( p ) =

Original: − (cid:80)
Revised: − (cid:80)

Original: i p
Revised: i p

Original: i log p i for a probability vector p denotes the entropy of p , k
Revised: i log p i for a probability vector p denotes the entropy of p , k

Original: the temperature hyperparameter, and ∥·∥ 1 the L 1 -norm.
Revised: the temperature hyperparameter and ∥·∥ 1 the L 1 -norm.

Original: The entropy H ( f tc ( x )) measures the degree of uncertainty about the class prediction f tc ( x ) made by client c for input x at round t : the larger the entropy is, the larger the uncertainty is.
Revised: The entropy H ( f tc ( x )) measures the degree of uncertainty about the class prediction f tc ( x ) made by client c for input x at round t : the larger the entropy is, the larger the uncertainty is.

Original: Then, the pseudo label is used to compute the crossentropy loss L CE with the server model’s prediction ( f ts c ◦ f ts f )( x ) as done in the standard knowledge distillation (Hinton et al., 2015).
Revised: Then, the pseudo label is used to compute the crossentropy loss L CE with the server model’s prediction ( f ts c ◦ f ts f )( x ) as done in the standard knowledge distillation (Hinton et al., 2015).

Original: The entropy-based weighting mitigates the influence of unreliable clients by suppressing the uncertain predictions.
Revised: The entropy-based weighting mitigates the influence of unreliable clients by suppressing the uncertain predictions.

Original: Our preliminary experiment in Fig.
Revised: Our preliminary experiment in Fig.

Original: 3 shows our entropy-based weight indeed assigns a higher weight to  the client having a higher softmax value on the true class on average.
Revised: 3 shows that this entropy-based weighting indeed tend to assign higher weights to more correct predictions, i.e., the predictions for which softmax values on the true classes are higher.

Revised: In Fig.

Revised: 3, we compared with another weighting method based on maximum softmax values of clients’s prediction, which does not exhibit such a tendency.

Original: Self-supervised learning The EED is applied jointly with self-supervised learning (SSL) to further leverage the benefits of available unlabeled data.
Revised: Self-supervised learning The EED is applied jointly with self-supervised learning (SSL) to further leverage the benefits of available unlabeled data.

Original: An arbitrary SSL method can be applied, which helps to learn generic feature representation (Chen & He, 2021;
Revised: An arbitrary SSL method can be applied, which helps to learn generic feature representation (Chen & He, 2021;

Original: Chen et al., 2020; Gidaris et al., 2018).
Revised: Chen et al., 2020; Gidaris et al., 2018).

Original: This may act as stabilizing the FL learning procedure at aggregation, which motivates us to deploy.
Revised: This may act as stabilizing the FL learning procedure at aggregation, which motivates us to deploy.

Original: For image data, we use the image rotation prediction (Gidaris et al., 2018) as the self-supervised pretext task, i.e ., rotating input images and predicting those rotated angles, which is known to be simple and effective without cumbersome tuning.
Revised: For image data, we use the image rotation prediction (Gidaris et al., 2018) as the self-supervised pretext task, i.e ., rotating input images and predicting those rotated angles, which is known to be simple and effective without cumbersome tuning.

Original: We use it due to its simplicity, but SSL in our approach is not limited to that method.
Revised: We use it due to its simplicity, but SSL in our approach is not limited to that method.

Original: We append the SSL head f ts s on top of the server’s feature extractor part f ts f , i.e ., ( f ts s ◦ f ts f )( x ) .
Revised: We append the SSL head f ts s on top of the server’s feature extractor part f ts f , i.e ., ( f ts s ◦ f ts f )( x ) .

Original: The SSL loss is applied to the output of the SSL head f ts s .
Revised: The SSL loss is applied to the output of the SSL head f ts s .

Original: Our final loss for aggregating client models into the server model is defined as L CE + γ L SSL in a multi-task manner.
Revised: Our final loss for aggregating client models into the server model is defined as L CE + γ L SSL in a multi-task manner.

Original: Implementation details and training For each round, each client model is trained for 30 epochs using its own dataset unless otherwise mentioned.
Revised: Implementation details and training For each round, each client model is trained for 30 epochs using its own dataset unless otherwise mentioned.

Original: We use the Adam optimizer with the client settings of learning rate 0.001, β = (0 . 9 , 0 . 999) , weight decay 0, and batch size 64 for CIFAR-10/100 and 128 for PathMNIST 1 .
Revised: We use the Adam optimizer with the client settings of learning rate 0.001, β = (0 . 9 , 0 . 999) , weight decay 0, and batch size 64 for CIFAR-10/100 and 128 for PathMNIST 1 .

Original: In the server update phase in each round, the server model is trained forepochs for CIFAR-10/100 and one epoch for PathMNIST using its own unlabeled dataset.
Revised: In the server update phase in each round, the server model is trained forepochs for CIFAR-10/100 and one epoch for PathMNIST using its own unlabeled dataset.

Original: The unlabeled dataset is pseudo-labeled by the gathered client models in the proposed entropy-weighted ensemble way.
Revised: The unlabeled dataset is pseudo-labeled by the gathered client models in the proposed entropy-weighted ensemble way.

Original: We set the entropy weight temperature k = 5 obtained from our empirical tests.
Revised: We set the entropy weight temperature k = 5 obtained from our empirical tests.

Original: We also use the Adam optimizer with the server settings of learning rate 0.00038 for CIFAR-10/100 and 0.001 for PathMNIST, β = (0 . 9 , 0 . 999) , weight decay 0, and batch size 64 for CIFAR-10/100 and 128 for PathMNIST.
Revised: We also use the Adam optimizer with the server settings of learning rate 0.00038 for CIFAR-10/100 and 0.001 for PathMNIST, β = (0 . 9 , 0 . 999) , weight decay 0, and batch size 64 for CIFAR-10/100 and 128 for PathMNIST.

Original: The learning rate for CIFAR-10/100 is found by a hyperparameter search tool,
Revised: The learning rate for CIFAR-10/100 is found by a hyperparameter search tool,

Original: Optuna (Akiba et al., 2019).
Revised: Optuna (Akiba et al., 2019).

Original: The balance parameter for the SSL loss is set to γ = 300 .
Revised: The balance parameter for the SSL loss is set to γ = 300 .

Revised: The underlying reason why such a large γ = 300 was applied is because the classification loss is implemented in a way that all the batch losses add up, instead of averaging.

Revised: If we take into account such an accumulation effect, the equivalent balance parameter was in fact 4.68 =

Revised: 300/64 (64 is the server’s batch size).

Revised: For the chosen γ , the proportion of the rotation loss is about twice that of the classification loss at the 50th round for the i.i.d.

Revised: setting in Fig.

Original:  E XPERIMENT R ESULTS 4. E XPERIMENT S ETUP
Revised:  E XPERIMENT R ESULTS 4. E XPERIMENT S ETUP

Original: Dataset and model Our FedDS is evaluated on CIFAR-10/100 and PathMNIST datasets with the ResNet-18 (He et al., 2016).
Revised: Dataset and model Our FedDS is evaluated on CIFAR-10/100 and PathMNIST datasets with the ResNet-18 (He et al., 2016).

Original: CIFAR-10 (resp.
Revised: CIFAR-10 (resp.

Original: CIFAR-100) has 10 classes (resp. 100 classes) containing 6,000 images (resp.
Revised: CIFAR-100) has 10 classes (resp. 100 classes) containing 6,000 images (resp.

Original: 600 images) per class.
Revised: 600 images) per class.

Original: For each of CIFAR-10/100 datasets, the training and test sets contain 50,000 and 10,000 images, respectively.
Revised: For each of CIFAR-10/100 datasets, the training and test sets contain 50,000 and 10,000 images, respectively.

Original: PathMNIST (Yang et al., 2021) is a medical image dataset consisting of 9 classes of Colon Pathology images.
Revised: PathMNIST (Yang et al., 2021) is a medical image dataset consisting of 9 classes of Colon Pathology images.

Original: The training and test sets contain 89,996 and 7,180 images, respectively.
Revised: The training and test sets contain 89,996 and 7,180 images, respectively.

Original: Dataset split In our experimental setting, we mainly assume 4 clients unless otherwise mentioned.
Revised: Dataset split In our experimental setting, we mainly assume 4 clients unless otherwise mentioned.

Original: The total client training dataset is sampled in such a way that it has the same class distribution as the entire training dataset, and it is partitioned by the number of clients and allocated to each client.
Revised: The total client training dataset is sampled in such a way that it has the same class distribution as the entire training dataset, and it is partitioned by the number of clients and allocated to each client.

Original: Each client’s class distribution follows the Dirichlet distribution Dir ( α ) as in Lin et al.
Revised: Each client’s class distribution follows the Dirichlet distribution Dir ( α ) as in Lin et al.

Original: Yurochkin et al.
Revised: Yurochkin et al.

Original: ; Wang et al.
Revised: ; Wang et al.

Original: Hsu et al.
Revised: Hsu et al.

Original: The hyperparameter α indicates the homogeneity of the distribution: the higher α is, the closer to i.i.d.
Revised: The hyperparameter α indicates the homogeneity of the distribution: the higher α is, the closer to i.i.d.

Original: client data setting.
Revised: client data setting.

Original: We use α = 100 , 000 and α = 0 .
Revised: We use α = 100 , 000 and α = 0 .

Original: 5 for the i.i.d.
Revised: 5 for the i.i.d.

Original: and non-i.i.d.
Revised: and non-i.i.d.

Original: client data settings, respectively.
Revised: client data settings, respectively.

Original: Some portion of a training dataset not sampled for client training datasets is used in the server without labels to mimic an unlabeled dataset.
Revised: Some portion of a training dataset not sampled for client training datasets is used in the server without labels to mimic an unlabeled dataset.

Original: Baseline algorithms Our baseline algorithms are  FedDF (Lin et al., 2020), and variants of FedAVG and our FedDS.
Revised: Baseline algorithms Our baseline algorithms are FedProx, FedDF (Lin et al., 2020), and variants of FedAVG and our FedDS.

Original: Since FedAVG (McMahan et al., 2017)  does not leverage unlabeled data, we devise a method by combining FedAVG and SSL (called FedAVG-S).
Revised: Since FedAVG (McMahan et al., 2017) and does not leverage unlabeled data, we devise a method by combining FedAVG and SSL (called FedAVG-S).

Original: After the server model is updated by model averaging in FedAVG, we apply the SSL loss (the rotation loss (Gidaris et al., 2018)) to further update the server model with the unlabeled dataset on the server.
Revised: After the server model is updated by model averaging in FedAVG, we apply the SSL loss (the rotation loss (Gidaris et al., 2018)) to further update the server model with the unlabeled dataset on the server.

Original: We confirm that this indeed provides a strong baseline in the presence of unlabeled data on the server (refer to Sec. 4.2).
Revised: We confirm that this indeed provides a strong baseline in the presence of unlabeled data on the server (refer to FedAVG FedDF [Table]

Original: In addition, to see the effect of the SSL task in our FedDS, we consider a variant of our FedDS (called
Revised: [Table] -S [Table] Ensemble distillation ✓ ✓ ✓

Revised: Uncertainty awareness ✓ ✓

Revised: Self-supervised loss ✓ ✓

Revised: Communication round

Revised: T e s t acc u r ac y

Revised: CIFAR-10 i.i.d.

Revised: FedDS

Revised: FedD

Revised: FedDF

Revised: FedAVG-S

Revised: FedAVG

Revised: FedProx

Revised: Communication round

Revised: T e s t acc u r ac y

Revised: CIFAR-10 non-

Revised: i.i.d.

Revised: Communication round

Revised: T e s t a cc u r a c y

Revised: CIFAR-100 i.i.d.

Revised: Communication round

Revised: T e s t a cc u r a c y

Revised: CIFAR-100 non-i.i.d.

Revised: Sec. 4.2).

Revised: In addition, to see the effect of the SSL task in our FedDS, we consider a variant of our

Original:  FedD), where the SSL part is suppressed by setting the SSL loss weight γ = 0 .
Revised: FedDS (called FedD), where the SSL part is suppressed by setting the SSL loss weight γ = 0 .

Original: For FedD and FedDS, we set B AV G = false unless otherwise mentioned.
Revised: For FedD and FedDS, we set B AV G = false unless otherwise mentioned.

Original: The main characteristics of the baselines and FedDS are summarized in Table 1.
Revised: The main characteristics of the baselines and FedDS are summarized in Table 1.

Original: In summary, both
Revised: In summary, both

Original: FedDF and FedD are based on ensemble distillation, where FedD incorporates entropy-based weights to take into account confidence of each client.
Revised: FedDF and FedD are based on ensemble distillation, where FedD incorporates entropy-based weights to take into account confidence of each client.

Original: On the other hand, FedAVG-S and FedDS both apply
Revised: On the other hand, FedAVG-S and FedDS both apply

Original: SSL, while FedDS takes the ensemble distillation jointly with SSL.
Revised: SSL, while FedDS takes the ensemble distillation jointly with SSL.

Original: [Figure]

Original: 4. R ESULTS WITH N ORMAL D ATA C ONDITION
Revised: 4. R ESULTS WITH N ORMAL D ATA C ONDITION

Original: In this subsection [Figure] we present the accuracy results for CIFAR-10/100 and PathMNIST classification tasks with i.i.d. and non-i.i.d.
Revised: In this subsection, we present the accuracy results for CIFAR-10/100 and PathMNIST classification tasks with i.i.d. and non-i.i.d.

Original: client data settings, called normal data condition.
Revised: client data settings, called normal data condition.

Original: For CIFAR-10/ tasks, Fig. 4 shows the test accuracy of FedDS and baselines according to the communication rounds.
Revised: For CIFAR-10/ tasks, Fig. 4 shows the test accuracy of FedDS and baselines according to the communication rounds.

Original: It shows that the proposed FedDS consistently outperforms all the baselines starting from the early communication rounds.
Revised: It shows that the proposed FedDS consistently outperforms all the baselines starting from the early communication rounds.

Original: The effectiveness of our algorithm according to the number of clients and the size of client data is analyzed based on Tables 2 and 3.
Revised: The effectiveness of our algorithm according to the number of clients and the size of client data is analyzed based on Tables 2 and 3.

Original: Table 4 shows that our FedDS still outperforms the baselines even when unlabeled data on the server is relatively small.
Revised: Table 4 shows that our FedDS still outperforms the baselines even when unlabeled data on the server is relatively small.

Original: Finally, the effectiveness of our algorithm on motivating medical diagnosis applications is verified via the
Revised: Finally, the effectiveness of our algorithm on motivating medical diagnosis applications is verified via the

Original: PathMNIST classification task in Table 5.
Revised: PathMNIST classification task in Table 5.

Original:  Comparison of two ensemble-distillation methods We confirm the effectiveness of our uncertainty aware ensemble-distillation by comparing two ensemble-distillation methods, i.e ., our FedD and
Revised: [Table] [Table] Comparison of two ensemble-distillation methods We confirm the effectiveness of our uncertainty aware ensemble-distillation by comparing two ensemble-distillation methods, i.e ., our FedD and

Original: FedDF (Lin et al., 2020).
Revised: FedDF (Lin et al., 2020).

Original: For all the settings in Fig.
Revised: For all the settings in Fig.

Original: 4, our FedD outperforms FedDF.
Revised: 4, our FedD outperforms FedDF.

Original: The main differences of our FedD from FedDF are twofold; measuring the confidence of each client’s prediction based on the entropy of softmax values and choosing the initial point of the ensemble distillation as the previous server model.
Revised: The main differences of our FedD from FedDF are twofold; measuring the confidence of each client’s prediction based on the entropy of softmax values and choosing the initial point of the ensemble distillation as the previous server model.

Original: For the initial point of the ensemble distillation, our FedD with
Revised: For the initial point of the ensemble distillation, our FedD with

Original: B AV G = false starts ensemble distillation directly from the server model at the previous round while FedDF starts ensemble distillation from the parameter average of transferred client models at[Table] [Table] each round.
Revised: B AV G = false starts ensemble distillation directly from the server model at the previous round while FedDF starts ensemble distillation from the parameter average of transferred client models at each round.

Original: Our empirical study in Tables 2 and 3 shows that using the averaged client model as a new initialization at each round is detrimental when the total number of client data is small.
Revised: Our empirical study in Tables 2 and 3 shows that using the averaged client model as a new initialization at each round is detrimental when the total number of client data is small.

Original: Effect of EED and SSL In Fig.
Revised: Effect of EED and SSL In Fig.

Original: 4, our FedD shows comparable or better performance to the other baselines for all the settings, indicating that the entropy-weighted ensemble distillation is as effective as applying self-supervision, i.e ., FedAVG-S, or even better.
Revised: 4, our FedD shows comparable or better performance to the other baselines for all the settings, indicating that the entropy-weighted ensemble distillation is as effective as applying self-supervision, i.e ., FedAVG-S, or even better.

Original: From FedAVG-S results, it is interesting to see that applying SSL is mostly effective, but its effect is diminishing in some cases.
Revised: From FedAVG-S results, it is interesting to see that applying SSL is mostly effective, but its effect is diminishing in some cases.

Original: FedAVG-S achieves the highest accuracy among the baselines in the CIFAR-10 classification task.
Revised: FedAVG-S achieves the highest accuracy among the baselines in the CIFAR-10 classification task.

Original: However, it achieves the lowest accuracy except FedAVG in the CIFAR-100 classification task.
Revised: However, it achieves similar accuracy with FedProx in the CIFAR-100 classification task.

Original: We conjecture that this is because the method of averaging the client model parameters in FedAVG results in unstable initialization for SSL when the total number of client data is small.
Revised: We conjecture that this is because the method of averaging the client model parameters in FedAVG results in unstable initialization for SSL when the total number of client data is small.

Original: By jointly applying SSL with FedD, corresponding to our FedDS, we can see considerable performance improvements.
Revised: By jointly applying SSL with FedD, corresponding to our FedDS, we can see considerable performance improvements.

Original: In particular, our FedDS is effective for the CIFAR-100 classification task, where the data per each client is quite small, i.e ., 25 images per class on average. 
Revised: In particular, our FedDS is effective for the CIFAR-100 classification task, where the data per each client is quite small, i.e ., 25 images per class on average. [Table]

Original:  Experiments with various numbers of clients and sizes of client data To examine the effectiveness of FedDS with various dataset-split setups, we evaluate the performance gain of our FedDS compared to the baselines for various numbers of clients (Table 2) and for various sizes of dataset in each client (Table 3).
Revised: [Table] Experiments with various numbers of clients and sizes of client data To examine the effectiveness of FedDS with various dataset-split setups, we evaluate the performance gain of our FedDS compared to the baselines for various numbers of clients (Table 2) and for various sizes of dataset in each client (Table 3).

Original: For our FedDS with B AV G = false , Table 2 shows that both the gain due to
Revised: For our FedDS with B AV G = false , Table 2 shows that both the gain due to

Original: EED (in blue color) and the additional gain from SSL (in red color) increase as the number of clients decreases.
Revised: EED (in blue color) and the additional gain from SSL (in red color) increase as the number of clients decreases.

Original: For small number of clients, the influence of unreliable clients becomes higher.
Revised: For small number of clients, the influence of unreliable clients becomes higher.

Original: Thus, the
Revised: Thus, the

Original: EED which mitigates the influence of unreliable clients, and the SSL which provides the generic[Table] [Table] knowledge about the task become more effective.
Revised: EED which mitigates the influence of unreliable clients, and the SSL which provides the generic knowledge about the task become more effective.

Original: Similarly, Table 3 shows that our FedDS with
Revised: Similarly, Table 3 shows that our FedDS with

Original: B AV G = false is more effective as the clients’ data becomes more deficient.
Revised: B AV G = false is more effective as the clients’ data becomes more deficient.

Original: Particularly, we can see that SSL becomes the dominant factor of performance gain as the client’s dataset gets smaller.
Revised: Particularly, we can see that SSL becomes the dominant factor of performance gain as the client’s dataset gets smaller.

Original: Tables 2 and 3 also show the performance difference of FedDS by setting B AV G = true instead of B AV G
Revised: Tables 2 and 3 also show the performance difference of FedDS by setting B AV G = true instead of B AV G

Original: = false (represented in teal colors).
Revised: = false (represented in teal colors).

Original: It shows that starting from the server model at the previous round ( B AV G = false ) performs better when the total number of client data is relatively small.
Revised: It shows that starting from the server model at the previous round ( B AV G = false ) performs better when the total number of client data is relatively small.

Original: This can be caused by the fact that averaging client models with limited client data unstabilizes the initial point, because a small fraction of outlier or undesirable contamination in client parameters can lead to notable biases in resultant parameter aggregation.
Revised: This can be caused by the fact that averaging client models with limited client data unstabilizes the initial point, because a small fraction of outlier or undesirable contamination in client parameters can lead to notable biases in resultant parameter aggregation.

Original: Small unlabeled dataset at the server is still useful To see the effect of the size of the unlabeled server dataset on the performance gain of FedDS, FedDS was evaluated for the CIFAR-10 classification task by changing the size of server’s unlabeled dataset.
Revised: Small unlabeled dataset at the server is still useful To see the effect of the size of the unlabeled server dataset on the performance gain of FedDS, FedDS was evaluated for the CIFAR-10 classification task by changing the size of server’s unlabeled dataset.

Original: Table 4 shows that FedDS is still useful when the number of unlabeled data at the server is comparable with that of total labeled data at the clients.
Revised: Table 4 shows that FedDS is still useful when the number of unlabeled data at the server is comparable with that of total labeled data at the clients.

Original: Although the test accuracy of FedDS decreases as the number of unlabeled data at the server decreases, FedDS using 10,000 unlabeled data still outperforms FedDF using 40,000 unlabeled data.
Revised: Although the test accuracy of FedDS decreases as the number of unlabeled data at the server decreases, FedDS using 10,000 unlabeled data still outperforms FedDF using 40,000 unlabeled data.

Original: Results on medical data To exemplify a medical diagnosis scenario with client-data deficiency, we evaluate the performance of the proposed FedDS and other baselines for PathMNIST classification task with i.i.d and non-i.i.d.
Revised: Results on medical data To exemplify a medical diagnosis scenario with client-data deficiency, we evaluate the performance of the proposed FedDS and other baselines for PathMNIST classification task with i.i.d and non-i.i.d.

Original: client data settings.
Revised: client data settings.

Original: Table 5 confirm the effectiveness of our method.
Revised: Table 5 confirm the effectiveness of our method.

Original: 4. R ESULTS WITH U NRELIABLE C LIENT - SIDE D ATA C ONDITION
Revised: 4. R ESULTS WITH U NRELIABLE C LIENT - SIDE D ATA C ONDITION

Original: To see the robustness of uncertainty-aware ensemble distillation and self-supervision, we consider two types of scenarios with unreliable clients.
Revised: To see the robustness of uncertainty-aware ensemble distillation and self-supervision, we consider two types of scenarios with unreliable clients.

Original: In particular, we consider two noisy-label scenarios with 1% and 20% label noises at each client (denoted as Noise-1 and Noise-20, respectively), and a Byzantine attack scenario with a malicious client whose labels are totally mislabeled (denoted as Byzantine).
Revised: In particular, we consider two noisy-label scenarios with 1% and 20% label noises at each client (denoted as Noise-1 and Noise-20, respectively), and a Byzantine attack scenario with a malicious client whose labels are totally mislabeled (denoted as Byzantine).

Original: We present the test accuracy of FedDS and baselines for Noise-1, Noise-20, and Byzantine scenarios in Table 6.
Revised: We present the test accuracy of FedDS and baselines for Noise-1, Noise-20, and Byzantine scenarios in Table 6.

Original: Let us focus on the i.i.d.
Revised: Let us focus on the i.i.d.

Original: client data setting as the tendency for the non-i.i.d.
Revised: client data setting as the tendency for the non-i.i.d.

Original: case is  almost the same.
Revised: case is [Table] almost the same.

Original: While our FedD’s performance degradation for Noise-1 and Noise-20 scenarios is slightly smaller than FedDF, it is remarkably smaller for the Byzantine case.
Revised: While our FedD’s performance degradation for Noise-1 and Noise-20 scenarios is slightly smaller than FedDF, it is remarkably smaller for the Byzantine case.

Original: This shows that the server model effectively rejects the Byzantine client’s prediction using our entropy-based weighting (refer to Fig. 3).
Revised: This shows that the server model effectively rejects the Byzantine client’s prediction using our entropy-based weighting (refer to Fig. 3).

Original: Our FedDS shows the most robust performance, i.e . performance degradations for
Revised: Our FedDS shows the most robust performance, i.e . performance degradations for

Original: Noise-1, Noise-20, and Byzantine senarios are the much smaller than other baselines.
Revised: Noise-1, Noise-20, and Byzantine senarios are the much smaller than other baselines.

Original: [Table]

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this work, we propose FedDS, which exploits the unlabeled data on the server side by entropyweighted ensemble distillation (EED) and self-supervised learning (SSL).
Revised: In this work, we propose FedDS, which exploits the unlabeled data on the server side by entropyweighted ensemble distillation (EED) and self-supervised learning (SSL).

Original: By using both techniques jointly, we demonstrate that FedDS outperforms the competing algorithms when the number of clients is small and data in the clients is deficient.
Revised: By using both techniques jointly, we demonstrate that FedDS outperforms the competing algorithms when the number of clients is small and data in the clients is deficient.

Original: In particular, when clients are unreliable, both techniques effectively suppress unreliable clients and reduce accuracy degradation.
Revised: In particular, when clients are unreliable, both techniques effectively suppress unreliable clients and reduce accuracy degradation.

Original: It is more effective as the number of participating clients are smaller, because, in such regimes, even introducing a single unreliable client would have much higher impact than other cases.
Revised: It is more effective as the number of participating clients are smaller, because, in such regimes, even introducing a single unreliable client would have much higher impact than other cases.

Original: Also, SSL effectively imposes the common knowledge for generic tasks to the server model.
Revised: Also, SSL effectively imposes the common knowledge for generic tasks to the server model.

Original: Thus, it stabilizes the overall learning round and helps the server model to overcome the deficiency of client data, especially when mislabeled samples are prevalent across clients.
Revised: Thus, it stabilizes the overall learning round and helps the server model to overcome the deficiency of client data, especially when mislabeled samples are prevalent across clients.

Original: Our experiments show that EED and SSL lead to their own improvement and jointly using them still preserves notable improvement, which may imply that they play different roles that are both necessary for FL.
Revised: Our experiments show that EED and SSL lead to their own improvement and jointly using them still preserves notable improvement, which may imply that they play different roles that are both necessary for FL.

Original: Limitations The main limitation is that FedDS requires additional server resources including computation and additional unlabeled data.
Revised: The main limitation is that FedDS requires additional server resources including computation and additional unlabeled data.

Original: FedDS would be more fruitful on the situation when the central server has large unlabeled dataset and sufficient computation power.
Revised: FedDS would be more fruitful on the situation when the central server has large unlabeled dataset and sufficient computation power.

Original: Further directions To apply FedDS to the situation when there is little unlabeled data on the server, some synthetic data would be useful.
Revised: Further directions To apply FedDS to the situation when there is little unlabeled data on the server, some synthetic data would be useful.

Original: Lin et al.
Revised: Lin et al.

Original: (2020) shows that the ensemble distillation using synthetic data from GAN gives decent pseudo label.
Revised: (2020) shows that the ensemble distillation using synthetic data from GAN gives decent pseudo label.

Original: FedDS would also be applicable to the clients with heterogeneous architectures.
Revised: FedDS would also be applicable to the clients with heterogeneous architectures.

Original: For each round, one can update clients regardless of heterogeneous architectures by distilling the server model’s knowledge using the client’s data, instead of directly substituting the server model’s parameter to each client model.
Revised: For each round, one can update clients regardless of heterogeneous architectures by distilling the server model’s knowledge using the client’s data, instead of directly substituting the server model’s parameter to each client model.

Original: FedDS would be applicable to tasks other than computer vision, like natural language.
Revised: FedDS would be applicable to tasks other than computer vision, like natural language.

Original: One can choose a suitable SSL task, like next word prediction, and use it as a SSL loss for FedDS.
Revised: One can choose a suitable SSL task, like next word prediction, and use it as a SSL loss for FedDS.

Original: Reproducibility Our algorithms are described in detail at Sec.
Revised: Reproducibility Our algorithms are described in detail at Sec.

Original: 3.2 and Sec. A.1.
Revised: 3.2 and Sec. A.1.

Original: Implementation details needed to reproduce our results are described in Sec.
Revised: Implementation details needed to reproduce our results are described in Sec.

Original: 4.1 including hyperparameters and data distribution.
Revised: 4.1 including hyperparameters and data distribution.

Original: Our source code will be publicized upon acceptance.
Revised: Our source code will be publicized upon acceptance

Original: A A PPENDIX
Revised: A A PPENDIX

Original: The appendix consists of a detailed description of the proposed FedDS algorithm (Sec. A.1), additional experimental results (Sec. A.2), and dataset and model information (Sec. A.3).
Revised: The appendix consists of a detailed description of the proposed FedDS algorithm (Sec. A.1), additional experimental results (Sec. A.2), and dataset and model information (Sec. A.3).

Original: A
Revised: A

Original: [Figure]
Revised: [Figure]

Original: Fig.
Revised: Fig.

Original: 5 illustrates the operation of the proposed FedDS algorithm.
Revised: 5 illustrates the operation of the proposed FedDS algorithm.

Original: Alg. 2, Alg. 3, and Alg.
Revised: Alg. 2, Alg. 3, and Alg.

Original: 4 correspond to ClientUpdate, CreatPseudoLabel, GetSelfSupervisedLoss algorithms used in the FedDS algorithm (Alg. 1), respectively.
Revised: 4 correspond to ClientUpdate, CreatPseudoLabel, GetSelfSupervisedLoss algorithms used in the FedDS algorithm (Alg. 1), respectively.

Original: Alg. 2 updates the client model as in FedAVG (McMahan et al., 2017).
Revised: Alg. 2 updates the client model as in FedAVG (McMahan et al., 2017).

Original: It trains the distributed server model using the client’s own data and returns the trained model.
Revised: It trains the distributed server model using the client’s own data and returns the trained model.

Original: Alg. 3 creates the pseudo label for entropy-weighted ensemble distillation (EED).
Revised: Alg. 3 creates the pseudo label for entropy-weighted ensemble distillation (EED).

Original: It gets the client models from Alg. 2 and creates a pseudo label for each unlabeled data of the server as the entropyweighted average of the client models’ predictions.
Revised: It gets the client models from Alg. 2 and creates a pseudo label for each unlabeled data of the server as the entropyweighted average of the client models’ predictions.

Original: Alg. 4 illustrates an example of self-supervised learning (SSL) algorithm, which we used in the experiments.
Revised: Alg. 4 illustrates an example of self-supervised learning (SSL) algorithm, which we used in the experiments.

Original: In general, GetSelfSupervisedLoss algorithm returns an SSL loss L SSL for each unlabeled data x of the server from feature extractor f
Revised: In general, GetSelfSupervisedLoss algorithm returns an SSL loss L SSL for each unlabeled data x of the server from feature extractor f

Original: ts f and SSL classifier f
Revised: ts f and SSL classifier f

Original: ts s of the server model.
Revised: ts s of the server model.

Original: In this example, we assume that the input data is an image, and use the predicting image rotation task (Gidaris et al., 2018) as the self-supervised pretext task, i.e . rotating the input image and predicting its rotated angle.
Revised: In this example, we assume that the input data is an image, and use the predicting image rotation task (Gidaris et al., 2018) as the self-supervised pretext task, i.e . rotating the input image and predicting its rotated angle.

Original: A. A DDITIONAL E XPERIMENTAL R ESULTS A.2. S ELF - SUPERVISION IS MORE EFFECTIVE THAN A FEW LABELED DATA
Revised: A. A DDITIONAL E XPERIMENTAL R ESULTS A.2. S ELF - SUPERVISION IS MORE EFFECTIVE THAN A FEW LABELED DATA

Original: To see the effect of a few labeled data on the server, we change 10% of server’s dataset to be labeled data.
Revised: To see the effect of a few labeled data on the server, we change 10% of server’s dataset to be labeled data.

Original: The server model performs ensemble distillation with this labeled data by adding the supervised cross-entropy loss.
Revised: The server model performs ensemble distillation with this labeled data by adding the supervised cross-entropy loss.

Original: For each batch of pseudo-labeled data, labeled data is appended into the batch and is trained together.
Revised: For each batch of pseudo-labeled data, labeled data is appended into the batch and is trained together.

Original: Table 7 shows the effect of partial labels on the server.
Revised: Table 7 shows the effect of partial labels on the server.

Original: FedDS without labeled data outperforms both FedD and FedDF with labeled data.
Revised: FedDS without labeled data outperforms both FedD and FedDF with labeled data.

Original: This shows that the self-supervision on large unlabeled data could give much valuable information than relatively little labeled data on our setting.
Revised: This shows that the self-supervision on large unlabeled data could give much valuable information than relatively little labeled data on our setting.

Original: More interestingly, the performance gain by using labels in FedDS is marginal.
Revised: More interestingly, the performance gain by using labels in FedDS is marginal.

Original: This may provide a hint at a possibility that the type of information from self-supervision is similar to and already covers the type of information the model can extract from labeled data.
Revised: This may provide a hint at a possibility that the type of information from self-supervision is similar to and already covers the type of information the model can extract from labeled data.

Original: D c is client c ’s labeled dataset.
Revised: D c is client c ’s labeled dataset.

Original: f ts f and f ts c are the feature extractor and the classifier parts of the server model at round t , respectively.
Revised: f ts f and f ts c are the feature extractor and the classifier parts of the server model at round t , respectively.

Original: f tc is the client c ’s model at round t .
Revised: f tc is the client c ’s model at round t .

Original: E c is the number of client training epochs for each round.
Revised: E c is the number of client training epochs for each round.

Original: Input : f t − 1 s c ◦ f t − 1 s f , D c f tc ← f t − 1 s c ◦ f t − 1 s f for e = 1 , . . .
Revised: Input : f t − 1 s c ◦ f t − 1 s f , D c f tc ← f t − 1 s c ◦ f t − 1 s f for e = 1 , . . .

Original: E c do for input data and its label ( x
Revised: E c do for input data and its label ( x

Original: , y ) ∈
Revised: , y ) ∈

Original: D c do
Revised: D c do

Original: L CE ← CrossEntropy ( y, f tc ( x ))
Revised: L CE ← CrossEntropy ( y, f tc ( x ))

Original: Optimize f ts to minimize L CE /* using any optimizer */ end end return f
Revised: Optimize f ts to minimize L CE /* using any optimizer */ end end return f

Original: tc f tc is the client c ’s model at round t .
Revised: tc f tc is the client c ’s model at round t .

Original: x is an unlabeled server data.
Revised: x is an unlabeled server data.

Original: H ( p ) = − (cid:88)
Revised: H ( p ) = − (cid:88)

Original: i p
Revised: i p

Original: i log p i for a probability vector p denotes the entropy of p , k is the temperature hyperparameter for entropy weight, and || · || 1 denotes the L 1 -norm.
Revised: i log p i for a probability vector p denotes the entropy of p , k is the temperature hyperparameter for entropy weight, and || · || 1 denotes the L 1 -norm.

Original: Input : f t 1 , . . .
Revised: Input : f t 1 , . . .

Original: , f tC , x return (cid:80)
Revised: , f tC , x return (cid:80)

Original: Cc =1 f tc ( x ) e − kH ( f tc ( x ) )
Revised: Cc =1 f tc ( x ) e − kH ( f tc ( x ) )

Original: (cid:13)(cid:13)(cid:13)(cid:80)
Revised: (cid:13)(cid:13)(cid:13)(cid:80)

Original: Cc
Revised: Cc

Original: [Equation]
Revised: [Equation]

Original: f tc
Revised: f tc

Original: [Equation]
Revised: [Equation]

Original: An example of SSL algorithm, which is used for FedDS in all the experiments in this paper.
Revised: An example of SSL algorithm, which is used for FedDS in all the experiments in this paper.

Original: f ts f and f ts s are the feature extractor and the SSL classifier parts of the server model at round t , respectively.
Revised: f ts f and f ts s are the feature extractor and the SSL classifier parts of the server model at round t , respectively.

Original: In this example, the output of f ts s is the prediction of rotation angle.
Revised: In this example, the output of f ts s is the prediction of rotation angle.

Original: For an input image x , rot d ( x ) rotates x by d degrees.
Revised: For an input image x , rot d ( x ) rotates x by d degrees.

Original: Input : f ts s ◦ f ts f , x
Revised: Input : f ts s ◦ f ts f , x

Original: L SSL ←for i = 0 , 1 , 2 , 3 do
Revised: L SSL ←for i = 0 , 1 , 2 , 3 do

Original: L SSL ← L SSL + CrossEntropy ( i, f ts s ◦ f ts f ( rot 90 i ( x )))
Revised: L SSL ← L SSL + CrossEntropy ( i, f ts s ◦ f ts f ( rot 90 i ( x )))

Original: endreturn L SSL
Revised: endreturn L SSL

Original: [Table]
Revised: [Table]

Original: A.2. A BLATION S TUDIES
Revised: A.2. H ETEROGENEOUS DATA SETTING

Original: Effect of combining methods for EED and SSL In the proposed FedDS algorithm (Alg. 1)
Revised: To see the effectiveness of our method under heterogeneous data setting [Table] FedDS was evaluated with other baselines in Fig.

Revised: 6 by using CIFAR-100 dataset for clients’ labeled dataset and TinyImageNet (Le & Yang) dataset for server’s unlabeled dataset.

Revised: In this experiment, the whole CIFAR-100 training

Revised: [Figure]

Revised: [Figure]

Revised: dataset and the whole TinyImageNet training dataset were used for client training and for server training, respectively

Original: [Table]
Revised: [Figure]

Revised: and the test accuracy was evaluated for CIFAR-100 test dataset.

Revised: Fig.

Revised: 6 shows that FedDS outperforms both FedDF and FedAVG in such a heterogeneous data setting.

Revised: A.2. E XPERIMENT WITH A LARGE NUMBER OF CLIENTS

Revised: While our FedDS mainly targets FL scenarios where the number of clients or the labeled data of clients is insufficient, let us investigate the effectiveness of our method in a normal FL situation with a large number of clients.

Revised: For CIFAR-10 classification task with 100 clients, Fig.

Revised: 7 shows that FedDS and FedDF using ensemble distillation consistently outperform FedAVG from early communication rounds.

Revised: The difference between FedDF and FedAVG shows that in situations where there are many clients, the effect of distillation decreases as the communication round increases.

Revised: In contrast, the consistent gap between FedDS and FedDF shows that entropy-based weighting and SSL are effective even when there are many clients.

Revised: [Figure]

Revised: [Figure]

Revised: A.2. E XPERIMENT WITH A LARGER DATASET

Revised: Fig.

Revised: 8 shows that our FedDS outperforms FedDF and FedAVG for ImageNet classification task (Krizhevsky et al., 2012), which has 100 classes, 130,000 images for training and 5,000 images for test.

Revised: For computational feasibility, we down-sampled to image resolution 32 (Chrabaszcz et al., 2017).

Revised: A.2. E XPERIMENT WITH DIFFERENT CLASS DISTRIBUTIONS AT THE CLIENTS AND AT THE SERVER

Revised: In our main experiments in Sec.

Revised: we assumed that the class distribution of the whole client dataset and that of the server’s unlabeled dataset are the same.

Original:  the server model is updated by applying the entropy-weighted ensemble distillation (EED) and  the self-supervised learning (SSL) simultaneously.
Revised: To investigate the effectiveness of our method when those class distributions are different, we evaluated FedDS with other baselines in Fig. 9 for CIFAR-10 classification task with i.i.d. class distribution for clients and Dir(0.5) distribution for the server.

Original: Another possible approach is to start SSL training
Revised: We can see that our FedDS outperforms the baselines in such scenarios with different class distributions at the clients and at the server [Table] [Figure]

Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original:  after the end of EED training.
Revised: Effect of combining methods for EED and SSL In the proposed FedDS algorithm (Alg. 1), the server model is updated by applying the entropy-weighted ensemble distillation (EED) and the self-supervised learning (SSL) simultaneously. Another possible approach is to start SSL training after the end of EED training.

Original: Table 8 shows that the performance gap between the two methods of combining EED and SSL is marginal, indicating that the timing of SSL is not an important factor.
Revised: Table 8 shows that the performance gap between the two methods of combining EED and SSL is marginal, indicating that the timing of SSL is not an important factor.

Original: Effect of temperature Fig.
Revised: Effect of temperature Fig.

Original: 6 shows the optimized temperature values k for FedDS from the hyperparameter search tool Optuna (Akiba et al., 2019).
Revised: 10 shows the optimized temperature values k for FedDS from the hyperparameter search tool Optuna (Akiba et al., 2019).

Original: For the i.i.d.
Revised: For the i.i.d.

Original: and the non-i.i.d.
Revised: and the non-i.i.d.

Original: data settings, k around 0.1 and k around 8 achieve near-optimal performances, respectively, while the performance remains almost the same over a relatively wide range of k .
Revised: data settings, k around 0.1 and k around 8 achieve near-optimal performances, respectively, while the performance remains almost the same over a relatively wide range of k .

Original: In the non-i.i.d data setting, client’s data deficiency is much severe than the i.i.d.
Revised: In the non-i.i.d data setting, client’s data deficiency is much severe than the i.i.d.

Original: case for certain classes, which necessitates more stronger effect of entropy-weighting that filters out immature predictions.
Revised: case for certain classes, which necessitates more stronger effect of entropy-weighting that filters out immature predictions.

Original: A. D ATASET AND M ODEL
Revised: A. D ATASET AND M ODEL

Original: For classification tasks, CIFAR-10/100 datasets (Krizhevsky, 2009) are used (MIT licence).
Revised: For classification tasks, CIFAR-10/100 datasets (Krizhevsky, 2009)(MIT licence), ImageNet dataset (Krizhevsky et al., 2012) (100 classes), TinyImageNet (Le & Yang) dataset are used .

Original: CIFAR10 (resp.
Revised: CIFAR-10 (resp.

Original: CIFAR-100) has 10 classes (resp. 100 classes) containing 6000 images (resp.images) per class. 
Revised: CIFAR-100) has 10 classes (resp. 100 classes) containing 6000 images (resp. 600 images) per class. ImageNet (100 classes) has 100 classes containing 1,300 (resp. 50) images per class for training (resp.

Revised: test).

Revised: TinyImageNet has 200 classes containing 500, 50,images per class for training, validation, test, respectively.

Original:  PathMNIST (Yang et al., 2021) is a medical image dataset consisting ofclasses of Colon Pathology images.
Revised: We used only validation data for evaluate the test accuracy. PathMNIST (Yang et al., 2021) is a medical image dataset consisting ofclasses of Colon Pathology images.

Original: The training and test sets contain 89,996 and 7,180 images, respectively.
Revised: The training and test sets contain 89,996 and 7,180 images, respectively.

Original: The proposed FedDS and baseline algorithms are implemented on ResNet-18 (He et al., 2016) by changing the last layer to adjust to the number of classes, using the open source https://github.com/kuangliu/pytorch-cifar.git .
Revised: The proposed FedDS and baseline algorithms are implemented on ResNet-18 (He et al., 2016) by changing the last layer to adjust to the number of classes, using the open source https://github.com/kuangliu/pytorch-cifar.git .