Original: D IVERGENCE - AWARE F EDERATED S ELF -S UPERVISED L EARNING
Revised: D IVERGENCE - AWARE F EDERATED S ELF -S UPERVISED L EARNING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Self-supervised learning (SSL) is capable of learning remarkable representations from centrally available data.
Revised: Self-supervised learning (SSL) is capable of learning remarkable representations from centrally available data.

Original: Recent works further implement federated learning with SSL to learn from rapidly growing decentralized unlabeled images (e.g., from cameras and phones), often resulted from privacy constraints.
Revised: Recent works further implement federated learning with SSL to learn from rapidly growing decentralized unlabeled images (e.g., from cameras and phones), often resulted from privacy constraints.

Original: Extensive attention has been paid to SSL approaches based on Siamese networks.
Revised: Extensive attention has been paid to SSL approaches based on Siamese networks.

Original: However, such an effort has not yet revealed deep insights into various fundamental building blocks for the federated self-supervised learning (FedSSL) architecture.
Revised: However, such an effort has not yet revealed deep insights into various fundamental building blocks for the federated self-supervised learning (FedSSL) architecture.

Original: We aim to fill in this gap via in-depth empirical study and propose a new method to tackle the nonindependently and identically distributed (non-IID) data problem of decentralized data.
Revised: We aim to fill in this gap via in-depth empirical study and propose a new method to tackle the nonindependently and identically distributed (non-IID) data problem of decentralized data.

Original: Firstly, we introduce a generalized FedSSL framework that embraces existing SSL methods based on Siamese networks and presents flexibility catering to future methods.
Revised: Firstly, we introduce a generalized FedSSL framework that embraces existing SSL methods based on Siamese networks and presents flexibility catering to future methods.

Original: In this framework, a server coordinates multiple clients to conduct SSL training and periodically updates local models of clients with the aggregated global model.
Revised: In this framework, a server coordinates multiple clients to conduct SSL training and periodically updates local models of clients with the aggregated global model.

Original: Using the framework, our study uncovers unique insights of FedSSL: 1) stop-gradient operation, previously reported to be essential, is not always necessary in FedSSL; 2) retaining local knowledge of clients in
Revised: Using the framework, our study uncovers unique insights of FedSSL: 1) stop-gradient operation, previously reported to be essential, is not always necessary in FedSSL; 2) retaining local knowledge of clients in

Original: FedSSL is particularly beneficial for non-IID data.
Revised: FedSSL is particularly beneficial for non-IID data.

Original: Inspired by the insights, we then propose a new approach for model update, Federated Divergence-aware Exponential Moving Average update (FedEMA).
Revised: Inspired by the insights, we then propose a new approach for model update, Federated Divergence-aware Exponential Moving Average update (FedEMA).

Original: FedEMA updates local models of clients adaptively using EMA of the global model, where the decay rate is dynamically measured by model divergence.
Revised: FedEMA updates local models of clients adaptively using EMA of the global model, where the decay rate is dynamically measured by model divergence.

Original: Extensive experiments demonstrate that
Revised: Extensive experiments demonstrate that

Original: FedEMA outperforms existing methods by 3-4% on linear evaluation.
Revised: FedEMA outperforms existing methods by 3-4% on linear evaluation.

Original: We hope that this work will provide useful insights for future research.
Revised: We hope that this work will provide useful insights for future research.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Self-supervised learning (SSL) has attracted extensive research interest for learning representations without relying on expensive data labels.
Revised: Self-supervised learning (SSL) has attracted extensive research interest for learning representations without relying on expensive data labels.

Original: In computer vision, the common practice is to design proxy tasks to facilitate visual representation learning from unlabeled images (Doersch et al., 2015;
Revised: In computer vision, the common practice is to design proxy tasks to facilitate visual representation learning from unlabeled images (Doersch et al., 2015;

Original: Noroozi & Favaro, 2016; Zhang et al., 2016; Gidaris et al., 2018).
Revised: Noroozi & Favaro, 2016; Zhang et al., 2016; Gidaris et al., 2018).

Original: Among them, the state-of-the-art
Revised: Among them, the state-of-the-art

Original: SSL methods employ contrastive learning that uses Siamese networks to minimize the similarity of two augmented views of images (Wu et al., 2018;
Revised: SSL methods employ contrastive learning that uses Siamese networks to minimize the similarity of two augmented views of images (Wu et al., 2018;

Original: Chen et al., 2020a; He et al., 2020;
Revised: Chen et al., 2020a; He et al., 2020;

Original: Grill et al., 2020; Chen & He, 2021).
Revised: Grill et al., 2020; Chen & He, 2021).

Original: All these methods heavily rely on the assumption that images are centrally available in cloud servers, such as public data on the Internet.
Revised: All these methods heavily rely on the assumption that images are centrally available in cloud servers, such as public data on the Internet.

Original: However, the rapidly growing amount of decentralized images may not be centralized due to increasingly stringent privacy protection regulations (Custers et al., 2019).
Revised: However, the rapidly growing amount of decentralized images may not be centralized due to increasingly stringent privacy protection regulations (Custers et al., 2019).

Original: The increasing number of edge devices, such as street cameras and phones, are generating a large number of unlabeled images, but these images may not be centralized as they could contain sensitive personal information like human faces.
Revised: The increasing number of edge devices, such as street cameras and phones, are generating a large number of unlabeled images, but these images may not be centralized as they could contain sensitive personal information like human faces.

Original: Besides, learning representations from these images could be more beneficial for downstream tasks deployed in the same scenarios (Yan et al., 2020).
Revised: Besides, learning representations from these images could be more beneficial for downstream tasks deployed in the same scenarios (Yan et al., 2020).

Original: A straightforward method is to adopt SSL methods for each edge, but it results in poor performance (Zhuang et al., 2021a) as decentralized data are mostly non-independently and identically distributed (non-IID)
Revised: A straightforward method is to adopt SSL methods for each edge, but it results in poor performance (Zhuang et al., 2021a) as decentralized data are mostly non-independently and identically distributed (non-IID)

Original: (Li et al., 2020a).
Revised: (Li et al., 2020a).

Original: Federated learning (FL) has emerged as a popular privacy-preserving method to train models from decentralized data, where clients send training updates to the server instead of raw data (McMahan et al., 2017).
Revised: Federated learning (FL) has emerged as a popular privacy-preserving method to train models from decentralized data, where clients send training updates to the server instead of raw data (McMahan et al., 2017).

Original: The majority of FL methods, however, are not applicable for unsupervised representa- tion learning because they require fully labeled data (Caldas et al., 2018), or partially labeled data in either the server or clients (Jin et al., 2020a; Jeong et al., 2021).
Revised: The majority of FL methods, however, are not applicable for unsupervised representa- tion learning because they require fully labeled data (Caldas et al., 2018), or partially labeled data in either the server or clients (Jin et al., 2020a; Jeong et al., 2021).

Original: Recent studies implement FL with
Revised: Recent studies implement FL with

Original: SSL methods that are based on Siamese networks, but they only focus on a single SSL method.
Revised: SSL methods that are based on Siamese networks, but they only focus on a single SSL method.

Original: For example, FedCA (Zhang et al., 2020a) is based on SimCLR (Chen et al., 2020a) and FedU (Zhuang et al., 2021a) is based on BYOL (Grill et al., 2020).
Revised: For example, FedCA (Zhang et al., 2020a) is based on SimCLR (Chen et al., 2020a) and FedU (Zhuang et al., 2021a) is based on BYOL (Grill et al., 2020).

Original: These efforts have not yet revealed deep insights into the fundamental building blocks of Siamese networks for federated self-supervised learning.
Revised: These efforts have not yet revealed deep insights into the fundamental building blocks of Siamese networks for federated self-supervised learning.

Original: In this paper, we investigate the effects of fundamental components of federated self-supervised learning (FedSSL) via in-depth empirical study.
Revised: In this paper, we investigate the effects of fundamental components of federated self-supervised learning (FedSSL) via in-depth empirical study.

Original: To facilitate fair comparison, we first introduce a generalized FedSSL framework to embrace existing SSL methods that differ in building blocks of
Revised: To facilitate fair comparison, we first introduce a generalized FedSSL framework to embrace existing SSL methods that differ in building blocks of

Original: Siamese networks.
Revised: Siamese networks.

Original: The framework comprises of a server and multiple clients: clients conduct SSL training using Siamese networks — an online network and a target network; the server aggregates the trained online networks to obtain a new global network and uses this global network to update the online networks of clients in the next round of training.
Revised: The framework comprises of a server and multiple clients: clients conduct SSL training using Siamese networks — an online network and a target network; the server aggregates the trained online networks to obtain a new global network and uses this global network to update the online networks of clients in the next round of training.

Original: FedSSL primarily focuses on the cross-silo
Revised: FedSSL primarily focuses on the cross-silo

Original: FL where clients are stateful with high availability (Kairouz et al., 2019).
Revised: FL where clients are stateful with high availability (Kairouz et al., 2019).

Original: We conduct empirical studies based on the FedSSL framework and discover important insights of
Revised: We conduct empirical studies based on the FedSSL framework and discover important insights of

Original: FedSSL.
Revised: FedSSL.

Original: Among four popular SSL methods (SimCLR (Chen et al., 2020a), MoCo (He et al., 2020),
Revised: Among four popular SSL methods (SimCLR (Chen et al., 2020a), MoCo (He et al., 2020),

Original: BYOL (Grill et al., 2020), and SimSiam (Chen & He, 2021), FedBYOL achieves the best performance, whereas FedSimSiam yields the worst performance.
Revised: BYOL (Grill et al., 2020), and SimSiam (Chen & He, 2021), FedBYOL achieves the best performance, whereas FedSimSiam yields the worst performance.

Original: More detailed analysis uncover the following unique insights: 1) Stop-gradient operation, essential for SimSiam and BYOL, is not always essential in FedSSL; 2) Target networks of clients are essential to gain knowledge from online networks; 3) Keeping local knowledge of clients is beneficial for performance on non-IID data.
Revised: More detailed analysis uncover the following unique insights: 1) Stop-gradient operation, essential for SimSiam and BYOL, is not always essential in FedSSL; 2) Target networks of clients are essential to gain knowledge from online networks; 3) Keeping local knowledge of clients is beneficial for performance on non-IID data.

Original: Inspired by the insights, we propose a new approach, Federated Divergence-aware Exponential Moving Average update (FedEMA) 1 , to address the non-IID data problem.
Revised: Inspired by the insights, we propose a new approach, Federated Divergence-aware Exponential Moving Average update (FedEMA) 1 , to address the non-IID data problem.

Original: Specifically, instead of updating online networks of clients simply by the global network, FedEMA updates them via exponential moving average (EMA) of the global network, where the decay rate of EMA is measured by the divergence of global and online networks dynamically.
Revised: Specifically, instead of updating online networks of clients simply by the global network, FedEMA updates them via exponential moving average (EMA) of the global network, where the decay rate of EMA is measured by the divergence of global and online networks dynamically.

Original: Extensive experiments demonstrate that FedEMA outperforms existing methods in a wide range of settings.
Revised: Extensive experiments demonstrate that FedEMA outperforms existing methods in a wide range of settings.

Original: We believe that important insights from this study will shed light on future research.
Revised: We believe that important insights from this study will shed light on future research.

Original: Our main contributions are threefold:
Revised: Our main contributions are threefold:

Original: • We introduce a new generalized FedSSL framework that embraces existing SSL methods based on Siamese networks and presents flexibility catering to future methods.
Revised: • We introduce a new generalized FedSSL framework that embraces existing SSL methods based on Siamese networks and presents flexibility catering to future methods.

Original: • We conduct in-depth empirical studies of FedSSL based on the framework and discover deep insights of the fundamental building blocks of Siamese networks for FedSSL.
Revised: • We conduct in-depth empirical studies of FedSSL based on the framework and discover deep insights of the fundamental building blocks of Siamese networks for FedSSL.

Original: • Inspired by the insights, we further propose a new model update approach, FedEMA, that adaptively updates online networks of clients with EMA of the global network.
Revised: • Inspired by the insights, we further propose a new model update approach, FedEMA, that adaptively updates online networks of clients with EMA of the global network.

Original: Extensive experiments show that FedEMA outperforms existing methods in a wide range of settings.
Revised: Extensive experiments show that FedEMA outperforms existing methods in a wide range of settings.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Self-supervised Learning In computer vision, self-supervised learning (SSL) aims to learn visual representations without any labels.
Revised: Self-supervised Learning In computer vision, self-supervised learning (SSL) aims to learn visual representations without any labels.

Original: Discriminative SSL methods facilitate learning with proxy tasks (Pathak et al., 2016; Noroozi & Favaro, 2016; Zhang et al., 2016; Gidaris et al., 2018).
Revised: Discriminative SSL methods facilitate learning with proxy tasks (Pathak et al., 2016; Noroozi & Favaro, 2016; Zhang et al., 2016; Gidaris et al., 2018).

Original: Among them, contrastive learning (Oord et al., 2018; Bachman et al., 2019) has become a promising principle.
Revised: Among them, contrastive learning (Oord et al., 2018; Bachman et al., 2019) has become a promising principle.

Original: It uses Siamese networks to minimize the similarity of two augmented views (positive pairs) and maximize the similarity of two different images (negative pairs).
Revised: It uses Siamese networks to minimize the similarity of two augmented views (positive pairs) and maximize the similarity of two different images (negative pairs).

Original: These methods are either contrastive or non-contrastive ones: contrastive SSL methods require negative pairs (Chen et al., 2020a; He et al., 2020) to prevent training collapse; non-contrastive SSL methods (Grill et al., 2020; Chen & He, 2021) are generally more efficient as they maintain remarkable performances using only positive pairs.
Revised: These methods are either contrastive or non-contrastive ones: contrastive SSL methods require negative pairs (Chen et al., 2020a; He et al., 2020) to prevent training collapse; non-contrastive SSL methods (Grill et al., 2020; Chen & He, 2021) are generally more efficient as they maintain remarkable performances using only positive pairs.

Original: However, these methods do not perform well on decentralized non-IID data (Zhuang et al., 2021a).
Revised: However, these methods do not perform well on decentralized non-IID data (Zhuang et al., 2021a).

Original: We analyze their similarities and variances and propose a generalized FedSSL framework.
Revised: We analyze their similarities and variances and propose a generalized FedSSL framework.

Original: Federated Learning Federated learning (FL) is a distributed training technique for learning from decentralized parties without transmitting raw data to a central server (McMahan et al., 2017).
Revised: Federated Learning Federated learning (FL) is a distributed training technique for learning from decentralized parties without transmitting raw data to a central server (McMahan et al., 2017).

Original: Among many studies that address the non-IID data challenge in FL (Zhao et al., 2018; Li et al.,
Revised: Among many studies that address the non-IID data challenge in FL (Zhao et al., 2018; Li et al.,

Original: Online
Revised: Online

Original: Target image
Revised: Target image

Original: Loss
Revised: Loss

Original: Client...
Revised: Client...

Original: Online
Revised: Online

Original: Online Online
Revised: Online Online

Original: Server
Revised: Server

Original: Online
Revised: Online

Original: Target image
Revised: Target image

Original: Loss
Revised: Loss

Original: Client
Revised: Client

Original: Online
Revised: Online

Original: Target image
Revised: Target image

Original: Loss
Revised: Loss

Original: Client k
Revised: Client k

Original: Online
Revised: Online

Original: Model Upload
Revised: Model Upload

Original: Model Update Online Network
Revised: Model Update Online Network

Original: [Figure]
Revised: [Figure]

Original: 2020b
Revised: 2020b

Original: ; Wang et al., 2020; Zhuang et al., 2021c), Personalized FL (PFL) aims to learn personalized models for clients (Tan et al., 2021).
Revised: ; Wang et al., 2020; Zhuang et al., 2021c), Personalized FL (PFL) aims to learn personalized models for clients (Tan et al., 2021).

Original: Although some PFL methods interpolate global and local models (Hanzely et al., 2020; Mansour et al., 2020; yuyang deng et al., 2021), our proposed FedEMA differ in the motivation, application scenario, and measurement of the decay rate.
Revised: Although some PFL methods interpolate global and local models (Hanzely et al., 2020; Mansour et al., 2020; yuyang deng et al., 2021), our proposed FedEMA differ in the motivation, application scenario, and measurement of the decay rate.

Original: Besides, the majority of existing works only consider supervised learning where clients have fully labeled data.
Revised: Besides, the majority of existing works only consider supervised learning where clients have fully labeled data.

Original: Although some recent works propose federated semi-supervised learning (Jin et al., 2020b; Zhang et al., 2020b; Jeong et al., 2021) or federated domain adaptation (Peng et al., 2020; Zhuang et al., 2021b), they still need labels in either the server or clients.
Revised: Although some recent works propose federated semi-supervised learning (Jin et al., 2020b; Zhang et al., 2020b; Jeong et al., 2021) or federated domain adaptation (Peng et al., 2020; Zhuang et al., 2021b), they still need labels in either the server or clients.

Original: This paper focuses on purely unlabeled decentralized data.
Revised: This paper focuses on purely unlabeled decentralized data.

Original: Federated Unsupervised Learning Learning representations from unlabeled decentralized data while preserving data privacy is still a nascent field.
Revised: Federated Unsupervised Learning Learning representations from unlabeled decentralized data while preserving data privacy is still a nascent field.

Original: Federated unsupervised representation learning is first proposed by van Berlo et al.
Revised: Federated unsupervised representation learning is first proposed by van Berlo et al.

Original: based on autoencoder, but it neglects the non-IID data challenge.
Revised: based on autoencoder, but it neglects the non-IID data challenge.

Original: Zhang et al.
Revised: Zhang et al.

Original: (2020a) address the non-IID issue with potential privacy risk for sharing features.
Revised: (2020a) address the non-IID issue with potential privacy risk for sharing features.

Original: Although Zhuang et al.
Revised: Although Zhuang et al.

Original: (2020) address the issue based on BYOL as our FedEMA, they do not shed light on why BYOL works best.
Revised: (2020) address the issue based on BYOL as our FedEMA, they do not shed light on why BYOL works best.

Original: Since SSL methods are evolving rapidly and new methods are emerging, we introduce a generalized FedSSL framework and deeply investigate the fundamental components to build up practical guidelines for the generic FedSSL framework.
Revised: Since SSL methods are evolving rapidly and new methods are emerging, we introduce a generalized FedSSL framework and deeply investigate the fundamental components to build up practical guidelines for the generic FedSSL framework.

Original:  A N E MPIRICAL S TUDY OF F EDERATED S ELF - SUPERVISED L EARNING
Revised:  A N E MPIRICAL S TUDY OF F EDERATED S ELF - SUPERVISED L EARNING

Original: This section first defines the problem and introduces the generalized FedSSL framework.
Revised: This section first defines the problem and introduces the generalized FedSSL framework.

Original: Using the framework, we then conduct empirical studies to reveal deep insights of FedSSL.
Revised: Using the framework, we then conduct empirical studies to reveal deep insights of FedSSL.

Original: 3. P ROBLEM D EFINITION
Revised: 3. P ROBLEM D EFINITION

Original: FedSSL aims to learn a generalized representation W from multiple decentralized parties for downstream tasks in the same scenarios.
Revised: FedSSL aims to learn a generalized representation W from multiple decentralized parties for downstream tasks in the same scenarios.

Original: Each party k contains unlabeled data D k
Revised: Each party k contains unlabeled data D k

Original: = {X k } that cannot be transferred to the server or other parties due to privacy constraints.
Revised: = {X k } that cannot be transferred to the server or other parties due to privacy constraints.

Original: Data is normally non-IID among decentralized parties (Li et al., 2020a); each party could contain only limited data categories (e.g., two out of ten CIFAR-10 classes)
Revised: Data is normally non-IID among decentralized parties (Li et al., 2020a); each party could contain only limited data categories (e.g., two out of ten CIFAR-10 classes)

Original: (Luo et al., 2019).
Revised: (Luo et al., 2019).

Original: As a result, each party alone is unable to obtain a good representation (Zhuang et al., 2021a).
Revised: As a result, each party alone is unable to obtain a good representation (Zhuang et al., 2021a).

Original: The global objective function to learn from multiple parties is min w f ( w ) :
Revised: The global objective function to learn from multiple parties is min w f ( w ) :

Original: = (cid:80) Kk =1 n k n f k ( w ) , where K is the number of clients, and n = (cid:80)
Revised: = (cid:80) Kk =1 n k n f k ( w ) , where K is the number of clients, and n = (cid:80)

Original: Kk =1 n k is the total data amount.
Revised: Kk =1 n k is the total data amount.

Original: For client k , f k ( w ) :
Revised: For client k , f k ( w ) :

Original: =
Revised: =

Original: E x k ∼P k
Revised: E x k ∼P k

Original: [ ˜ f k ( w ; x k )] is the expected loss over data distribution P k , where x k is the unlabeled data and ˜ f k ( w ; x k ) is the loss function.
Revised: [ ˜ f k ( w ; x k )] is the expected loss over data distribution P k , where x k is the unlabeled data and ˜ f k ( w ; x k ) is the loss function.

Original: 3. G ENERALIZED F RAMEWORK
Revised: 3. G ENERALIZED F RAMEWORK

Original: We introduce a generalized FedSSL framework that empowers existing SSL methods based on
Revised: We introduce a generalized FedSSL framework that empowers existing SSL methods based on

Original: Siamese networks to learn from decentralized data under privacy constraints.
Revised: Siamese networks to learn from decentralized data under privacy constraints.

Original: Figure 1 depicts the end-to-end training pipeline of the framework.
Revised: Figure 1 depicts the end-to-end training pipeline of the framework.

Original: It comprises of three key operations: 1) Local Training in clients; 2) Model Aggregation in the server; 3) Model Communication (upload and update) between the server and clients.
Revised: It comprises of three key operations: 1) Local Training in clients; 2) Model Aggregation in the server; 3) Model Communication (upload and update) between the server and clients.

Original: We implement and analyze four popular SSL methods — SimCLR (Chen et al., 2020a), MoCo (V1 (He et al., 2020) and V2 (Chen et al., 2020b)), SimSiam (Chen &
Revised: We implement and analyze four popular SSL methods — SimCLR (Chen et al., 2020a), MoCo (V1 (He et al., 2020) and V2 (Chen et al., 2020b)), SimSiam (Chen &

Original: He, 2021), and BYOL (Grill et al., 2020).
Revised: He, 2021), and BYOL (Grill et al., 2020).

Original: Variances in Siamese networks of these methods lead to differences in executions in these three operations 2 .
Revised: Variances in Siamese networks of these methods lead to differences in executions in these three operations 2 .

Original: Local Training Firstly, each client k conducts self-supervised training on unlabeled data D k based on the same global model W og downloaded from the server.
Revised: Local Training Firstly, each client k conducts self-supervised training on unlabeled data D k based on the same global model W og downloaded from the server.

Original: Regardless of SSL methods, clients train with Siamese networks — an online network W ok and a target network W tk for E local epochs using cooresponding loss functions L .
Revised: Regardless of SSL methods, clients train with Siamese networks — an online network W ok and a target network W tk for E local epochs using cooresponding loss functions L .

Original: We classify these SSL methods with two major differences (Figure 8 in Appendix A): 1) Only SimSiam and BYOL contain a predictor in the online network, so we denote their online network W ok
Revised: We classify these SSL methods with two major differences (Figure 8 in Appendix A): 1) Only SimSiam and BYOL contain a predictor in the online network, so we denote their online network W ok

Original: =
Revised: =

Original: ( W k , W pk ) , where W k is the online encoder and W pk is the predictor; As for SimCLR and MoCo, W ok = W k . 2) SimCLR and SimSiam share identical weights between the online encoder and the target encoder, so W tk = W k .
Revised: ( W k , W pk ) , where W k is the online encoder and W pk is the predictor; As for SimCLR and MoCo, W ok = W k . 2) SimCLR and SimSiam share identical weights between the online encoder and the target encoder, so W tk = W k .

Original: In contrast, MoCo and BYOL update the target encoder with EMA of the online encoder in every mini-batch: W tk = mW k + (1 − m ) W tk , where m is the momentum value normally set to 0.99.
Revised: In contrast, MoCo and BYOL update the target encoder with EMA of the online encoder in every mini-batch: W tk = mW k + (1 − m ) W tk , where m is the momentum value normally set to 0.99.

Original: Model Communication After local training, client k uploads the trained online network W ok to the server and updates it with the global model W og after aggregation.
Revised: Model Communication After local training, client k uploads the trained online network W ok to the server and updates it with the global model W og after aggregation.

Original: Considering the differences of
Revised: Considering the differences of

Original: SSL methods, we upload and update encoders and predictors separately: 1) we upload and update the predictor when it presents in local training; 2) we follow the communication protocol Zhuang et al.
Revised: SSL methods, we upload and update encoders and predictors separately: 1) we upload and update the predictor when it presents in local training; 2) we follow the communication protocol Zhuang et al.

Original: (2021a) to upload and update only the online encoder W k when encoders are different.
Revised: (2021a) to upload and update only the online encoder W k when encoders are different.

Original: Model Aggregation When the server receives online networks from clients, it aggregates them to obtain a new global model W og = (cid:80)
Revised: Model Aggregation When the server receives online networks from clients, it aggregates them to obtain a new global model W og = (cid:80)

Original: Kk =0 n k n W ok .
Revised: Kk =0 n k n W ok .

Original: W
Revised: W

Original: og =
Revised: og =

Original: ( W g , W pg )
Revised: ( W g , W pg )

Original: if predictor presents, otherwise
Revised: if predictor presents, otherwise

Original: W og = W g , where W g is the global encoder.
Revised: W og = W g , where W g is the global encoder.

Original: Then, the server sent W og to clients to update their online networks.
Revised: Then, the server sent W og to clients to update their online networks.

Original: The training iterates these three operations until it meets the stopping conditions.
Revised: The training iterates these three operations until it meets the stopping conditions.

Original: At the end of the training, we use the parameters of W og as the generic representation W for evaluation.
Revised: At the end of the training, we use the parameters of W og as the generic representation W for evaluation.

Original: 3. E XPERIMENTAL S ETUP
Revised: 3. E XPERIMENTAL S ETUP

Original: We provide basic experimental setups in this section and describe more details in Appendix B.
Revised: We provide basic experimental setups in this section and describe more details in Appendix B.

Original: Datasets We conduct experiments using CIFAR-10 and CIFAR-100 datasets (Krizhevsky et al., 2009).
Revised: Datasets We conduct experiments using CIFAR-10 and CIFAR-100 datasets (Krizhevsky et al., 2009).

Original: To simulate federated settings, we equally split a dataset into K clients.
Revised: To simulate federated settings, we equally split a dataset into K clients.

Original: We simulate nonIID data with label heterogeneity, where each client contains limited classes — l = { 2 , 4 , 6 , 8 , 10 } number of classes for CIFAR-10 and l = { 20 , 40 , 60 , 80 , 100 } for CIFAR-100.
Revised: We simulate nonIID data with label heterogeneity, where each client contains limited classes — l = { 2 , 4 , 6 , 8 , 10 } number of classes for CIFAR-10 and l = { 20 , 40 , 60 , 80 , 100 } for CIFAR-100.

Original: The setting is IID when each client contains 10 (100) classes for CIFAR-10 (CIFAR-100).
Revised: The setting is IID when each client contains 10 (100) classes for CIFAR-10 (CIFAR-100).

Original: Implementation Details We implement FedSSL in Python using popular deep learning framework
Revised: Implementation Details We implement FedSSL in Python using popular deep learning framework

Original: PyTorch (Paszke et al., 2017).
Revised: PyTorch (Paszke et al., 2017).

Original: To simulate federated learning, we train each client on one NVIDIA
Revised: To simulate federated learning, we train each client on one NVIDIA

Original: V100 GPU.
Revised: V100 GPU.

Original: These clients communicate with the server through NCCL backend.
Revised: These clients communicate with the server through NCCL backend.

Original: We use ResNet- (He et al., 2016) as default network for the encoders and present results of ResNet-50 in Appendix
Revised: We use ResNet- (He et al., 2016) as default network for the encoders and present results of ResNet-50 in Appendix

Original: C. The predictor is a two-layer multi-layer perceptron (MLP).
Revised: C. The predictor is a two-layer multi-layer perceptron (MLP).

Original: By default, we train for R =rounds with K = 5 clients, E = 5 local epoches, batch size B = 128 , learning rate η = 0 .
Revised: By default, we train for R =rounds with K = 5 clients, E = 5 local epoches, batch size B = 128 , learning rate η = 0 .

Original: 032 with cosine decay, and non-IID data l = 2 ( l = 20 ) for CIFAR-10 (CIFAR-100).
Revised: 032 with cosine decay, and non-IID data l = 2 ( l = 20 ) for CIFAR-10 (CIFAR-100).

Original: Linear Evaluation
Revised: Linear Evaluation

Original: We evaluate the quality of representations following linear evaluation (Kolesnikov et al., 2019; Grill et al., 2020) protocol.
Revised: We evaluate the quality of representations following linear evaluation (Kolesnikov et al., 2019; Grill et al., 2020) protocol.

Original: We first learn representations from the FedSSL framework.
Revised: We first learn representations from the FedSSL framework.

Original: Then, we train a new linear classifier on the frozen representations.
Revised: Then, we train a new linear classifier on the frozen representations.

Original: 3. A LGORITHM C OMPARISONS
Revised: 3. A LGORITHM C OMPARISONS

Original: We benchmark and compare the SSL methods using the FedSSL framework.
Revised: We benchmark and compare the SSL methods using the FedSSL framework.

Original: To denote the implementation of an SSL method, We add a prefix Fed to the name of the SSL method.
Revised: To denote the implementation of an SSL method, We add a prefix Fed to the name of the SSL method.

Original: For example,
Revised: For example,

Original: FedBYOL denotes using BYOL in the FedSSL framework.
Revised: FedBYOL denotes using BYOL in the FedSSL framework.

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: Table 1 compares linear evaluation results of these methods under the non-IID setting of CIFAR datasets.
Revised: Table 1 compares linear evaluation results of these methods under the non-IID setting of CIFAR datasets.

Original: On the one hand, contrastive FedSSL methods obtain similar performances.
Revised: On the one hand, contrastive FedSSL methods obtain similar performances.

Original: As SimCLR is previously reported to need a large batch size (e.g., B = 4096 ) (Chen et al., 2020a), it is surprising that FedSimCLR obtains competitive results using the same batch size B = 128 as the others.
Revised: As SimCLR is previously reported to need a large batch size (e.g., B = 4096 ) (Chen et al., 2020a), it is surprising that FedSimCLR obtains competitive results using the same batch size B = 128 as the others.

Original: On the other hand, the results of non-contrastive FedSSL methods have large variances: FedBYOL achieves the best performance, whereas FedSimSiam yields the worst performance.
Revised: On the other hand, the results of non-contrastive FedSSL methods have large variances: FedBYOL achieves the best performance, whereas FedSimSiam yields the worst performance.

Original: Since SimSiam is capable to learn as powerful representations as BYOL (Chen & He, 2021), as well as considering that noncontrastive methods are conceptually simpler and more efficient (Tian et al., 2021), we focus on non-contrastive methods and further investigate the effects of their fundamental components.
Revised: Since SimSiam is capable to learn as powerful representations as BYOL (Chen & He, 2021), as well as considering that noncontrastive methods are conceptually simpler and more efficient (Tian et al., 2021), we focus on non-contrastive methods and further investigate the effects of their fundamental components.

Original: 3. I MPACT OF F ACTORS OF N ON - CONTRASTIVE M ETHODS
Revised: 3. I MPACT OF F ACTORS OF N ON - CONTRASTIVE M ETHODS

Original: This section analyzes the impact of fundamental components of non-contrastive FedSSL methods.
Revised: This section analyzes the impact of fundamental components of non-contrastive FedSSL methods.

Original: From empirical studies, we obtain the following insights: 1) predictor is essential; 2) EMA and stopgradient improves performances; 3) Local encoders should retain local knowledge of the non-IID data; 4) Target encoder should gain knowledge from the online encoder.
Revised: From empirical studies, we obtain the following insights: 1) predictor is essential; 2) EMA and stopgradient improves performances; 3) Local encoders should retain local knowledge of the non-IID data; 4) Target encoder should gain knowledge from the online encoder.

Original: Details are as followed.
Revised: Details are as followed.

Original: Predictor is essential.
Revised: Predictor is essential.

Original: Figure 2 (left plot) presents the kNN testing accuracy as a monitoring process for FedBYOL and FedSimSiam with and without predictors.
Revised: Figure 2 (left plot) presents the kNN testing accuracy as a monitoring process for FedBYOL and FedSimSiam with and without predictors.

Original: Without predictors, both methods can barely learn due to collapse in local training.
Revised: Without predictors, both methods can barely learn due to collapse in local training.

Original: It affirms the vital role of predictor (Chen & He, 2021;
Revised: It affirms the vital role of predictor (Chen & He, 2021;

Original: Tian et al., 2021) even when learning from decentralized data.
Revised: Tian et al., 2021) even when learning from decentralized data.

Original: Stop-gradient operation is previously indicated as an essential component for SimSiam and BYOL (Tian et al., 2021), but it is not essential for FedBYOL.
Revised: Stop-gradient operation is previously indicated as an essential component for SimSiam and BYOL (Tian et al., 2021), but it is not essential for FedBYOL.

Original: Stop-gradient prevents stochastic gradient optimization on the target network.
Revised: Stop-gradient prevents stochastic gradient optimization on the target network.

Original: Figure 2 shows that FedSimSiam without stop-gradient collapses, whereas FedBYOL without stop-gradient still achieves competitive performance.
Revised: Figure 2 shows that FedSimSiam without stop-gradient collapses, whereas FedBYOL without stop-gradient still achieves competitive performance.

Original: It is because online and target encoders are significantly different in FedBYOL as the online encoder is updated by the global encoder every communication round.
Revised: It is because online and target encoders are significantly different in FedBYOL as the online encoder is updated by the global encoder every communication round.

Original: In contrast, SimSiam or FedSiamSiam share weights between online and target encoders, so removing stop-gradient leads to collapse.
Revised: In contrast, SimSiam or FedSiamSiam share weights between online and target encoders, so removing stop-gradient leads to collapse.

Original: Exponential Moving Average (EMA) is not essential, but it helps improve performance.
Revised: Exponential Moving Average (EMA) is not essential, but it helps improve performance.

Original: Table(first and second rows) shows that FedBYOL outperforms FedSimSiam at different levels of nonIID data, which is represented by { 2, 4, 6, 8, 10 } classes per client of CIFAR-10.
Revised: Table(first and second rows) shows that FedBYOL outperforms FedSimSiam at different levels of nonIID data, which is represented by { 2, 4, 6, 8, 10 } classes per client of CIFAR-10.

Original: EMA is the main difference between SimSiam and BYOL, indicating that EMA is helpful to improve performance.
Revised: EMA is the main difference between SimSiam and BYOL, indicating that EMA is helpful to improve performance.

Original: Based on these results, we further analyze the underlying impact of EMA on the encoders below.
Revised: Based on these results, we further analyze the underlying impact of EMA on the encoders below.

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: Encoders that retain local knowledge of non-IID data helps improve performance.
Revised: Encoders that retain local knowledge of non-IID data helps improve performance.

Original: EMA in FedBYOL allows the parameters of the online encoder to be different from the target encoder.
Revised: EMA in FedBYOL allows the parameters of the online encoder to be different from the target encoder.

Original: As a result, the global encoder only updates the online encoder, not the target encoder.
Revised: As a result, the global encoder only updates the online encoder, not the target encoder.

Original: We hypothesize that retaining such local knowledge of data in the target encoder is beneficial especially when the data distribution is highly skewed.
Revised: We hypothesize that retaining such local knowledge of data in the target encoder is beneficial especially when the data distribution is highly skewed.

Original: For comparison, we remove such local knowledge by updating both online and target encoders with the global model.
Revised: For comparison, we remove such local knowledge by updating both online and target encoders with the global model.

Original: Table 2 shows that FedBYOL with both encoders updated leads to lower performance than FedBYOL; It achieves results close to FedSimSiam when the data distribution is more skewed (2 or 4 classes per client).
Revised: Table 2 shows that FedBYOL with both encoders updated leads to lower performance than FedBYOL; It achieves results close to FedSimSiam when the data distribution is more skewed (2 or 4 classes per client).

Original: These results demonstrate the importance of keeping local knowledge in the encoders.
Revised: These results demonstrate the importance of keeping local knowledge in the encoders.

Original: Besides, the results of { 6, 8, 10 } classes per client also indicate the benefit of EMA.
Revised: Besides, the results of { 6, 8, 10 } classes per client also indicate the benefit of EMA.

Original: Target encoder is essential to gain knowledge from the online encoder.
Revised: Target encoder is essential to gain knowledge from the online encoder.

Original: Figure 3 shows that FedBYOL without EMA can merely learn, and FedBYOL without EMA and stop-gradient (sg) degrades in performance.
Revised: Figure 3 shows that FedBYOL without EMA can merely learn, and FedBYOL without EMA and stop-gradient (sg) degrades in performance.

Original: In both cases, the target encoder is either never updated (w/o EMA) or is updated only through backpropagation (w/o EMA and sg) — not updated by the online encoder.
Revised: In both cases, the target encoder is either never updated (w/o EMA) or is updated only through backpropagation (w/o EMA and sg) — not updated by the online encoder.

Original: On the other hand, we also identify that FedSSL methods in Table 1, which achieve competitive results, all update target encoders with knowledge of online encoders (the global encoder is the aggregation of the online encoder).
Revised: On the other hand, we also identify that FedSSL methods in Table 1, which achieve competitive results, all update target encoders with knowledge of online encoders (the global encoder is the aggregation of the online encoder).

Original: We argue that target encoder is crucial to gain knowledge from the online encoder to provide contrastive targets.
Revised: We argue that target encoder is crucial to gain knowledge from the online encoder to provide contrastive targets.

Original: We further validate it by using the global encoder to update both online and target encoders when removing both EMA and stop-gradient.
Revised: We further validate it by using the global encoder to update both online and target encoders when removing both EMA and stop-gradient.

Original: Figure 3 shows that such method improves performance and achieves comparable results.
Revised: Figure 3 shows that such method improves performance and achieves comparable results.

Original:  D IVERGENCE - AWARE D YNAMIC M OVING A VERAGE U PDATE
Revised:  D IVERGENCE - AWARE D YNAMIC M OVING A VERAGE U PDATE

Original: Built on the FedSSL framework, we propose Federated Divergence-aware EMA update (FedEMA) to further mitigate non-IID data challenges.
Revised: Built on the FedSSL framework, we propose Federated Divergence-aware EMA update (FedEMA) to further mitigate non-IID data challenges.

Original: Since FedBYOL contains all components that help improve performance, we adopt it as the baseline and optimize the model update operation.
Revised: Since FedBYOL contains all components that help improve performance, we adopt it as the baseline and optimize the model update operation.

Original: Non-IID data causes the global model to diverge from centralized training (Zhuang et al., 2021a).
Revised: Non-IID data causes the global model to diverge from centralized training (Zhuang et al., 2021a).

Original: Inspired by the insight that retaining local knowledge of non-IID data helps improve performance, we propose to update the online network via EMA of the global network.
Revised: Inspired by the insight that retaining local knowledge of non-IID data helps improve performance, we propose to update the online network via EMA of the global network.

Original: Compared with FedBYOL that replaces the online network with the global network, FedEMA fuses local and global knowledge effectively through EMA update, where the decay rate of EMA is dynamically measured by model divergences.
Revised: Compared with FedBYOL that replaces the online network with the global network, FedEMA fuses local and global knowledge effectively through EMA update, where the decay rate of EMA is dynamically measured by model divergences.

Original: Figure 4 depicts our proposed FedEMA method.
Revised: Figure 4 depicts our proposed FedEMA method.

Original: The formulation is as followed:
Revised: The formulation is as followed:

Original: [Equation]
Revised: [Equation]

Original: where W rk and W p,rk are the online encoder and predictor of client k at training round r ; W rg and
Revised: where W rk and W p,rk are the online encoder and predictor of client k at training round r ; W rg and

Original: W p,rg are the global encoder and predictor; µ is the decay rate, measured by the divergence between global and online encoders; λ is a scaler to adjust the level of model divergence, which is measured by calculating the l 2 -norm of the global and online encoders.
Revised: W p,rg are the global encoder and predictor; µ is the decay rate, measured by the divergence between global and online encoders; λ is a scaler to adjust the level of model divergence, which is measured by calculating the l 2 -norm of the global and online encoders.

Original: We summarize FedEMA in Algorithm 1.
Revised: We summarize FedEMA in Algorithm 1.

Original: FedEMA can be regarded as a generalization of FedBYOL — they are the same when λ = 0 .
Revised: FedEMA can be regarded as a generalization of FedBYOL — they are the same when λ = 0 .

Original: Scaler λ plays a vital role to adapt FedEMA for different levels of divergence caused by the data.
Revised: Scaler λ plays a vital role to adapt FedEMA for different levels of divergence caused by the data.

Original: The divergence between global and online encoders varies when the settings of federated learning change.
Revised: The divergence between global and online encoders varies when the settings of federated learning change.

Original: For example, different degrees of non-IID settings would result in different divergences.
Revised: For example, different degrees of non-IID settings would result in different divergences.

Original: Since characteristics of data are unknown before training as they are unlabeled, we propose a practical autoscaler to calculate a personalized λ k for each client k automatically.
Revised: Since characteristics of data are unknown before training as they are unlabeled, we propose a practical autoscaler to calculate a personalized λ k for each client k automatically.

Original: The formula is λ k
Revised: The formula is λ k

Original: = τ
Revised: = τ

Original: || W r
Revised: || W r

Original: +1 g − W rk || , where τ ∈ [0 , 1] is the expected value of µ at round r .
Revised: +1 g − W rk || , where τ ∈ [0 , 1] is the expected value of µ at round r .

Original: We calculate λ k only once at the earliest round r that client k is sampled for training.
Revised: We calculate λ k only once at the earliest round r that client k is sampled for training.

Original: When the same set of clients are sampled for training, λ k =
Revised: When the same set of clients are sampled for training, λ k =

Original: [Equation]
Revised: [Equation]

Original: | is calculated at round r = 1 .
Revised: | is calculated at round r = 1 .

Original: The intuition of FedEMA is to retain more local knowledge when divergence is large and incorporate more global knowledge when divergence is small.
Revised: The intuition of FedEMA is to retain more local knowledge when divergence is large and incorporate more global knowledge when divergence is small.

Original: When model divergence is large, keeping more local knowledge is more beneficial for the non-IID data.
Revised: When model divergence is large, keeping more local knowledge is more beneficial for the non-IID data.

Original: Since the global network is the aggregation of online networks, representing global knowledge from clients.
Revised: Since the global network is the aggregation of online networks, representing global knowledge from clients.

Original: When divergence is small, adapting more global knowledge help improve model generalization.
Revised: When divergence is small, adapting more global knowledge help improve model generalization.

Original: Since model divergence is larger at the start of training (Figure 6), it is practical to choose larger τ ∈ [0 . 5 , 1) ; τ = 1 is not considered because only local knowledge is used when τ = 1 .
Revised: Since model divergence is larger at the start of training (Figure 6), it is practical to choose larger τ ∈ [0 . 5 , 1) ; τ = 1 is not considered because only local knowledge is used when τ = 1 .

Original: We use τ = 0 .
Revised: We use τ = 0 .

Original: 7 by default in experiments.
Revised: 7 by default in experiments.

Original: Algorithm
Revised: Algorithm

Original: 1: ServerExecution: 2: Initialize global W  g  W p,  g  3: for each round r = 0 , 1 , ..., R do 4:
Revised: 1: ServerExecution: 2: Init W 0 g and W p, 0 g , init λ k to be null 3: for each round r = 0 , 1 , ..., R do 4:

Original: S t ← (Selection of K clients) 5: for client k ∈ S t in parallel do 6:
Revised: S t ← (Selection of K clients) 5: for client k ∈ S t in parallel do 6:

Original: W rk , W p,rk ← Client( W rg , W p,rg , r , λ k ) 7:
Revised: W rk , W p,rk ← Client( W rg , W p,rg , r , λ k ) 7:

Original: W r +1 g ← (cid:80) k ∈ S t n k n W rk 8:
Revised: W r +1 g ← (cid:80) k ∈ S t n k n W rk 8:

Original: W p,r +1 g ← (cid:80) k ∈ S t n k n W p,rk 9: for client k ∈ S t do 10:if not λ k then λ k ← τ || W r +1 g − W rk ||  11: Return W Rg 12: Client ( W rg , W p,rg , r , λ k ): 13: ifnot λ k  or not selected in r − 1 then 14:
Revised: W p,r +1 g ← (cid:80) k ∈ S t n k n W p,rk 9: for client k ∈ S t do 10: λ k ← τ || W r +1 g − W rk || if λ k is null 11: Return W Rg 12: Client ( W rg , W p,rg , r , λ k ): 13: if λ k is null or not selected in r − 1 then 14:

Original: W k , W tk , W pk ← W rg , W rg , W p,rg 15: else 16: µ ← min ( λ k (cid:13)(cid:13)
Revised: W k , W tk , W pk ← W rg , W rg , W p,rg 15: else 16: µ ← min ( λ k (cid:13)(cid:13)

Original: W rg − W r − 1 k (cid:13)(cid:13) , 1) 17:
Revised: W rg − W r − 1 k (cid:13)(cid:13) , 1) 17:

Original: W k ← µW r − 1 k + (1 − µ ) W rg 18:
Revised: W k ← µW r − 1 k + (1 − µ ) W rg 18:

Original: W p k ← µW p,r − 1 k + (1 − µ ) W p,r g 19: for local epoch e = 0 , 1 , ..., E − 1 do 20: for b ∈ B data batches with size B do 21:
Revised: W p k ← µW p,r − 1 k + (1 − µ ) W p,r g 19: for local epoch e = 0 , 1 , ..., E − 1 do 20: for b ∈ B data batches with size B do 21:

Original: W ok ← W ok − η ▽ L W ok ,W tk ( W ok ; b ) 22:
Revised: W ok ← W ok − η ▽ L W ok ,W tk ( W ok ; b ) 22:

Original: W
Revised: W

Original: tk ← mW tk + (1 − m ) W k 23:
Revised: tk ← mW tk + (1 − m ) W k 23:

Original: Return W rk , W p,rk
Revised: Return W rk , W p,rk

Original: [Figure]
Revised: [Figure]

Original: [Table] [Table]
Revised: [Table] [Table]

Original:  E VALUATION
Revised:  E VALUATION

Original: This section follows the experimental setup in Section 3.3 to evaluate FedEMA in the linear evaluation and semi-supervised learning.
Revised: This section follows the experimental setup in Section 3.3 to evaluate FedEMA in the linear evaluation and semi-supervised learning.

Original: We also provide ablation studies of important hyperparameters.
Revised: We also provide ablation studies of important hyperparameters.

Original: 5. A LGORITHM C OMPARISONS
Revised: 5. A LGORITHM C OMPARISONS

Original: To demonstrate the effectiveness of FedEMA, we compare it with the following methods: 1) standalone training, where a client learns independently using BYOL; 2) FedCA, which is proposed in
Revised: To demonstrate the effectiveness of FedEMA, we compare it with the following methods: 1) standalone training, where a client learns independently using BYOL; 2) FedCA, which is proposed in

Original: Zhang et al.
Revised: Zhang et al.

Original: (2020a); 3) FedBYOL as the baseline; 4)
Revised: (2020a); 3) FedBYOL as the baseline; 4)

Original: FedU, which is proposed in (Zhuang et al., 2021a).
Revised: FedU, which is proposed in (Zhuang et al., 2021a).

Original: Besides, we also present results of possible upper bounds that learn representations with centralized data using BYOL.
Revised: Besides, we also present results of possible upper bounds that learn representations with centralized data using BYOL.

Original: Linear Evaluation Table 3 shows that FedEMA outperforms other methods on different settings of CIFAR datasets.
Revised: Linear Evaluation Table 3 shows that FedEMA outperforms other methods on different settings of CIFAR datasets.

Original: Specifically, the performance is more 3% higher than existing methods in most settings.
Revised: Specifically, the performance is more 3% higher than existing methods in most settings.

Original: Besides, our proposed autoscaler achieves similar results as λ = 0 . 8 .
Revised: Besides, our proposed autoscaler achieves similar results as λ = 0 . 8 .

Original: More experiments on larger number of clients K and random selection of clients are provided in Table 6 in Appendix C.
Revised: More experiments on larger number of clients K and random selection of clients are provided in Table 6 in Appendix C.

Original: Semi-supervised Learning
Revised: Semi-supervised Learning

Original: We also assess the quality of representations following the semisupervised learning protocol (Zhai et al., 2019; Chen et al., 2020a) — we add a new two-layer
Revised: We also assess the quality of representations following the semisupervised learning protocol (Zhai et al., 2019; Chen et al., 2020a) — we add a new two-layer

Original: MLP on top of the encoder and fine-tune the whole model with limited (1% and 10%) labeled data for 100 epochs.
Revised: MLP on top of the encoder and fine-tune the whole model with limited (1% and 10%) labeled data for 100 epochs.

Original: Table 4 indicates that FedEMA consistently outperforms other methods on non-IID settings of CIFAR datasets and our autoscaler outperforms manual-selected λ
Revised: Table 4 indicates that FedEMA consistently outperforms other methods on non-IID settings of CIFAR datasets and our autoscaler outperforms manual-selected λ

Original: = 1 .
Revised: = 1 .

Original: 5. A BLATION S TUDIES
Revised: 5. A BLATION S TUDIES

Original: Ablation on FedEMA We analyze whether we need to update both online encoder (Eqn 1) and predictor (Eqn 2) in FedEMA.
Revised: Ablation on FedEMA We analyze whether we need to update both online encoder (Eqn 1) and predictor (Eqn 2) in FedEMA.

Original: Figure 5 shows that updating only the encoder or predictor leads to better performance; only updating predictor also leads to faster convergence.
Revised: Figure 5 shows that updating only the encoder or predictor leads to better performance; only updating predictor also leads to faster convergence.

Original: Their combination re-
Revised: Their combination re-

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: sults in the best performance.
Revised: sults in the best performance.

Original: These results demonstrate the effectiveness of updating both predictor and encoder in FedEMA.
Revised: These results demonstrate the effectiveness of updating both predictor and encoder in FedEMA.

Original: More results on other settings are provided in Table 5 in Appendix C.
Revised: More results on other settings are provided in Table 5 in Appendix C.

Original: Changes of Divergence Figure 6 illustrates that the divergence between global encoder and online encoder (Eqn 3) decreases gradually as training proceeds.
Revised: Changes of Divergence Figure 6 illustrates that the divergence between global encoder and online encoder (Eqn 3) decreases gradually as training proceeds.

Original: It validates our intuition that more local knowledge is used at the start of training when divergence is larger.
Revised: It validates our intuition that more local knowledge is used at the start of training when divergence is larger.

Original: Besides, clients can update at their own pace depending on the divergence caused by their local dataset.
Revised: Besides, clients can update at their own pace depending on the divergence caused by their local dataset.

Original: Scaler λ
Revised: Scaler λ

Original: We study the impact of λ with values in [0 , 2] with interval of 0.2 in Figure 7a.
Revised: We study the impact of λ with values in [0 , 2] with interval of 0.2 in Figure 7a.

Original: λ >leads to a significant performance drop because it results in µ = 1 at the start of training on CIFAR datasets, implying that no aggregated global network is used.
Revised: λ >leads to a significant performance drop because it results in µ = 1 at the start of training on CIFAR datasets, implying that no aggregated global network is used.

Original: When λ ∈ (0 , 1) , the performances are consistently better than FedBYOL ( λ = 0 ) as both local and global knowledge are effectively aggregated.
Revised: When λ ∈ (0 , 1) , the performances are consistently better than FedBYOL ( λ = 0 ) as both local and global knowledge are effectively aggregated.

Original: These analyses are mainly suitable for our experiment setting.
Revised: These analyses are mainly suitable for our experiment setting.

Original: The range values of λ depend on the characteristics of data and the hyper-parameters (e.g., local epoch) of FL settings.
Revised: The range values of λ depend on the characteristics of data and the hyper-parameters (e.g., local epoch) of FL settings.

Original: A practical way to tune λ manually is to understand the divergence by running the algorithm for several rounds and choose the λ that scales µ to (0.5, 1).
Revised: A practical way to tune λ manually is to understand the divergence by running the algorithm for several rounds and choose the λ that scales µ to (0.5, 1).

Original: Nevertheless, we recommend using autoscaler and provide ablation study of τ of autoscaler in Figure 10a in Appendix C.
Revised: Nevertheless, we recommend using autoscaler and provide ablation study of τ of autoscaler in Figure 10a in Appendix C.

Original: Constant Values of µ We further demonstrate the necessity of dynamic EMA by comparing with using constant values of µ in Eqn 1 and 2.
Revised: Constant Values of µ We further demonstrate the necessity of dynamic EMA by comparing with using constant values of µ in Eqn 1 and 2.

Original: Figure 7b shows that a good choice of constant µ can outperform FedBYOL, but FedEMA outperforms using constant µ for the online encoder, predictor, or applying both.
Revised: Figure 7b shows that a good choice of constant µ can outperform FedBYOL, but FedEMA outperforms using constant µ for the online encoder, predictor, or applying both.

Original: We also provide results that encoder and predictor use different µ in Appendix C.
Revised: We also provide results that encoder and predictor use different µ in Appendix C.

Original: Non-IID Level Figure 7c compares the performance of different non-IID levels, ranging from 2 to 10 classes per client on the CIFAR-10 dataset.
Revised: Non-IID Level Figure 7c compares the performance of different non-IID levels, ranging from 2 to 10 classes per client on the CIFAR-10 dataset.

Original: We use autoscaler for these experiments.
Revised: We use autoscaler for these experiments.

Original: FedEMA consistently outperforms FedBYOL in these settings.
Revised: FedEMA consistently outperforms FedBYOL in these settings.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We uncover important insights of federated self-supervised learning (FedSSL) from in-depth empirical studies, using a newly introduced generalized FedSSL framework.
Revised: We uncover important insights of federated self-supervised learning (FedSSL) from in-depth empirical studies, using a newly introduced generalized FedSSL framework.

Original: Inspired by the insights, we propose a new method, Federated Divergence-aware Exponential Moving Average update (FedEMA), to further address the non-IID data challenge.
Revised: Inspired by the insights, we propose a new method, Federated Divergence-aware Exponential Moving Average update (FedEMA), to further address the non-IID data challenge.

Original: Our experiments and ablations demonstrate that FedEMA outperforms existing methods in a wide range of settings.
Revised: Our experiments and ablations demonstrate that FedEMA outperforms existing methods in a wide range of settings.

Original: In the future, we plan to implement FedSSL and FedEMA on larger-scale datasets.
Revised: In the future, we plan to implement FedSSL and FedEMA on larger-scale datasets.

Original: We hope that this study will provide useful insights for future research.
Revised: We hope that this study will provide useful insights for future research.

Original:  R EPRODUCIBILITY S TATEMENT
Revised:  R EPRODUCIBILITY S TATEMENT

Original: To facilitate reproducibility of experiment results we first provide basic experimental setups in Section 3.3, including datasets, implementation details, and evaluation protocols.
Revised: To facilitate reproducibility of experiment results we first provide basic experimental setups in Section 3.3, including datasets, implementation details, and evaluation protocols.

Original: Then, we describe more experimental details in Appendix B, including datasets, data transformation, network architecture, training details and default settings.
Revised: Then, we describe more experimental details in Appendix B, including datasets, data transformation, network architecture, training details and default settings.

Original: Also, we indicate the settings and hyper-parameters of experiments when their settings are different from the default.
Revised: Also, we indicate the settings and hyper-parameters of experiments when their settings are different from the default.

Original: Moreover, we plan to make our codes available after the opening of discussion forums
Revised: Moreover, we plan to make our codes available after the opening of discussion forums

Original: A D IFFERENCES OF S ELF - SUPERVISED L EARNING M ETHODS
Revised: A D IFFERENCES OF S ELF - SUPERVISED L EARNING M ETHODS

Original: We study four SSL methods using the FedSSL framework in Section 3.
Revised: We study four SSL methods using the FedSSL framework in Section 3.

Original: These four SSL methods have two major differences that impact the executions of local training, model communication, and model aggregation.
Revised: These four SSL methods have two major differences that impact the executions of local training, model communication, and model aggregation.

Original: Figure 8 depicts these differences: 1) BYOL and SimSiam have predictors, whereas MoCo and SimCLR do not have them; 2) SimSiam and SimCLR share weights between two encoders, whereas BYOL and MoCo have different parameters for the online and target encoders.
Revised: Figure 8 depicts these differences: 1) BYOL and SimSiam have predictors, whereas MoCo and SimCLR do not have them; 2) SimSiam and SimCLR share weights between two encoders, whereas BYOL and MoCo have different parameters for the online and target encoders.

Original: [Figure]
Revised: [Figure]

Original: B E XPERIMENTAL D ETAILS
Revised: B E XPERIMENTAL D ETAILS

Original: In this section, we provide more details about the dataset, network architecture, and training and evaluation setups.
Revised: In this section, we provide more details about the dataset, network architecture, and training and evaluation setups.

Original: B. D ATA
Revised: B. D ATA

Original: Datasets CIFAR-10 and CIFAR-100 are two popular image datasets (Krizhevsky et al., 2009).
Revised: Datasets CIFAR-10 and CIFAR-100 are two popular image datasets (Krizhevsky et al., 2009).

Original: Both datasets consist of 50,000 training images and 10,000 testing images.
Revised: Both datasets consist of 50,000 training images and 10,000 testing images.

Original: CIFAR-10 containsclasses, where each class has 5,000 training images and 1,000 testing images.
Revised: CIFAR-10 containsclasses, where each class has 5,000 training images and 1,000 testing images.

Original: While CIFARcontains 100 classes, where each class has 500 training images and 100 testing images.
Revised: While CIFARcontains 100 classes, where each class has 500 training images and 100 testing images.

Original: To simulate federated learning, we equally split the training set into K clients.
Revised: To simulate federated learning, we equally split the training set into K clients.

Original: We simulate non-IID data using label heterogeneity — data among clients is more skewed when each client contains less number of classes.
Revised: We simulate non-IID data using label heterogeneity — data among clients is more skewed when each client contains less number of classes.

Original: Hence, we simulate different levels of non-IID data with l number of classes per client, where l = { 2 , 4 , 6 , 8 , 10 } for CIFAR-10 and l = { 20 , 40 , 60 , 80 , 100 } for CIFAR-100.
Revised: Hence, we simulate different levels of non-IID data with l number of classes per client, where l = { 2 , 4 , 6 , 8 , 10 } for CIFAR-10 and l = { 20 , 40 , 60 , 80 , 100 } for CIFAR-100.

Original: For example, when simulating 5 clients with l = 4 classes per client in CIFAR-10, we need 5 × 4 = 20 total sets of data over 10 classes.
Revised: For example, when simulating 5 clients with l = 4 classes per client in CIFAR-10, we need 5 × 4 = 20 total sets of data over 10 classes.

Original: Thus, we split the training images of each class equally into two sets (2, images in each set) and assign random four sets without overlapping classes to a client.
Revised: Thus, we split the training images of each class equally into two sets (2, images in each set) and assign random four sets without overlapping classes to a client.

Original: The setting is IID when each client contains all classes of a dataset.
Revised: The setting is IID when each client contains all classes of a dataset.

Original: By default, we run experiments with K =clients with non-IID setting l = 2 classes per client for CIFAR-10 dataset and l = 20 classes per client for CIFAR-100 dataset.
Revised: By default, we run experiments with K =clients with non-IID setting l = 2 classes per client for CIFAR-10 dataset and l = 20 classes per client for CIFAR-100 dataset.

Original: Transformation In local training of the FedSSL framework, we take two augmentations of each image as the inputs for online and target networks, respectively.
Revised: Transformation In local training of the FedSSL framework, we take two augmentations of each image as the inputs for online and target networks, respectively.

Original: We obtain the augmentations by transforming the images with a set of transformations: For SimCLR, BYOL, SimSiam, and MoCoV2, we adopt the transformations from Chen et al.
Revised: We obtain the augmentations by transforming the images with a set of transformations: For SimCLR, BYOL, SimSiam, and MoCoV2, we adopt the transformations from Chen et al.

Original: (2020a); For MoCoV1, we use the transformation described in its paper (He et al., 2020).
Revised: (2020a); For MoCoV1, we use the transformation described in its paper (He et al., 2020).

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: et al., 2021a
Revised: et al., 2021a

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Method
Revised: Method

Original: [Table]
Revised: [Table]

Original: -IID
Revised: -IID

Original: FedBYOL 83.
Revised: FedBYOL 83.

Original: FedEMA (ours) 84.
Revised: FedEMA (ours) 84.

Original: B. N ETWORK A RCHITECTURE
Revised: B. N ETWORK A RCHITECTURE

Original: Predictor The network architecture of the predictor is a two-layer multilayer perceptron (MLP).
Revised: Predictor The network architecture of the predictor is a two-layer multilayer perceptron (MLP).

Original: The two-layer MLP starts from a fully connected layer with 4096 neurons.
Revised: The two-layer MLP starts from a fully connected layer with 4096 neurons.

Original: Followed by onedimension batch normalization and a ReLU activation function, it ends with another fully connected layer with 2048 neurons.
Revised: Followed by onedimension batch normalization and a ReLU activation function, it ends with another fully connected layer with 2048 neurons.

Original: Encoder We use ResNet-18 He et al.
Revised: Encoder We use ResNet-18 He et al.

Original: as the default network architecture of the encoder in the majority of experiments.
Revised: as the default network architecture of the encoder in the majority of experiments.

Original: Besides, we also provide results of ResNet-50 in Table 5 and 7.
Revised: Besides, we also provide results of ResNet-50 in Table 5 and 7.

Original: [Table]
Revised: [Table]

Original: FedCA (Zhang et al., 2020a)
Revised: FedCA (Zhang et al., 2020a)

Original: [Table]
Revised: [Table]

Original: FedU (Zhuang et al., 2021a)
Revised: FedU (Zhuang et al., 2021a)

Original: [Table]
Revised: [Table]

Original: ( λ = 1 )
Revised: ( λ = 1 )

Original: [Table]
Revised: [Table]

Original: (autoscaler, τ = 0 . 7 )
Revised: (autoscaler, τ = 0 . 7 )

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Our ResNet architecture differs from the implementation in PyTorch (Paszke et al., 2017) in three aspects: 1) We use kernel size 3 × 3 for the first convolution layer instead of 7 × 7 ; 2)
Revised: Our ResNet architecture differs from the implementation in PyTorch (Paszke et al., 2017) in three aspects: 1) We use kernel size 3 × 3 for the first convolution layer instead of 7 × 7 ; 2)

Original: We use an average pooling layer with kernel size 4 × 4 before the last linear layer instead of adaptive average pooling layer; 3) We replace the last linear layer with a two-layer MLP.
Revised: We use an average pooling layer with kernel size 4 × 4 before the last linear layer instead of adaptive average pooling layer; 3) We replace the last linear layer with a two-layer MLP.

Original: The network architecture of the MLP is the same as the predictor.
Revised: The network architecture of the MLP is the same as the predictor.

Original: [Table]
Revised: [Table]

Original:  T RAINING AND E VALUATION D ETAILS
Revised:  T RAINING AND E VALUATION D ETAILS

Original: Training We use Stochastic Gradient Descent (SGD) as the optimizer in training.
Revised: Training We use Stochastic Gradient Descent (SGD) as the optimizer in training.

Original: We use η = 0 .
Revised: We use η = 0 .

Original: as the initial learning rate and decay the learning with a cosine annealing (Loshchilov &
Revised: as the initial learning rate and decay the learning with a cosine annealing (Loshchilov &

Original: Hutter, 2017), which is also used in SimSiam.
Revised: Hutter, 2017), which is also used in SimSiam.

Original: By default, we train R = 100 rounds with local epochs E = 5 and batch size
Revised: By default, we train R = 100 rounds with local epochs E = 5 and batch size

Original: B
Revised: B

Original: =
Revised: =

Original: using K = 5 clients.
Revised: using K = 5 clients.

Original: We simulate training of K clients on K NVIDIA V100 GPUs and employ the PyTorch Paszke et al.
Revised: We simulate training of K clients on K NVIDIA V100 GPUs and employ the PyTorch Paszke et al.

Original: (2017) communication backend (NCCL) for communications between clients and the server.
Revised: (2017) communication backend (NCCL) for communications between clients and the server.

Original: If not specified, we use λ = 1 by
Revised: If not specified, we use λ = 1 by

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: default or autoscaler with τ = 0 .
Revised: default or autoscaler with τ = 0 .

Original: 7 for FedEMA.
Revised: 7 for FedEMA.

Original: As for experiments of FedU, we follow the hyperparameters described in paper (Zhuang et al., 2021a).
Revised: As for experiments of FedU, we follow the hyperparameters described in paper (Zhuang et al., 2021a).

Original: Cross-silo FL vs Cross-device FL
Revised: Cross-silo FL vs Cross-device FL

Original: This paper primarily focuses on cross-silo FL where clients are stateful with high availability.
Revised: This paper primarily focuses on cross-silo FL where clients are stateful with high availability.

Original: Clients can cache local models and carry these local states from round to round.
Revised: Clients can cache local models and carry these local states from round to round.

Original: Extensive experiments demonstrate that FedEMA achieves the best performance under this setting.
Revised: Extensive experiments demonstrate that FedEMA achieves the best performance under this setting.

Original: On the other hand, cross-device FL assumes there are millions of stateless clients that might participate in training just once.
Revised: On the other hand, cross-device FL assumes there are millions of stateless clients that might participate in training just once.

Original: Due to the constraints of experimental settings, the majority of studies conduct experiments with at most hundreds of clients (Wang et al., 2020; Jeong et al., 2021).
Revised: Due to the constraints of experimental settings, the majority of studies conduct experiments with at most hundreds of clients (Wang et al., 2020; Jeong et al., 2021).

Original: FedEMA can work under such experimental settings by caching the states of clients in the server.
Revised: FedEMA can work under such experimental settings by caching the states of clients in the server.

Original: When the number of clients scales to millions, FedEMA degrades to FedBYOL that updates both encoders — without keeping any local states.
Revised: When the number of clients scales to millions, FedEMA degrades to FedBYOL that updates both encoders — without keeping any local states.

Original: Evalaution We assess the quality of learned representations using linear evaluation (Kolesnikov et al., 2019; Grill et al., 2020) and semi-supervised learning (Zhai et al., 2019; Chen et al., 2020a) protocols.
Revised: Evalaution We assess the quality of learned representations using linear evaluation (Kolesnikov et al., 2019; Grill et al., 2020) and semi-supervised learning (Zhai et al., 2019; Chen et al., 2020a) protocols.

Original: We first obtain a trained encoder (or learned representations) using full training set for linear evaluation and 99% or 90% of the training set for semi-supervised learning (excluding the 1% or 10% for fine-tuning).
Revised: We first obtain a trained encoder (or learned representations) using full training set for linear evaluation and 99% or 90% of the training set for semi-supervised learning (excluding the 1% or 10% for fine-tuning).

Original: Then, we conduct evaluations based on the trained encoder.
Revised: Then, we conduct evaluations based on the trained encoder.

Original: For linear evaluation, we train a new fully connected layer on top of the frozen trained encoder (fixed parameters) for 200 epochs, using batch size 512 and Adam optimizer with learning rate 3e-3.
Revised: For linear evaluation, we train a new fully connected layer on top of the frozen trained encoder (fixed parameters) for 200 epochs, using batch size 512 and Adam optimizer with learning rate 3e-3.

Original: For semi-supervised learning, we add a new two-layer MLP on top of the trained encoder and fine-tune the whole model using 1% or 10% of data for 100 epochs, using batch size 128 and Adam optimizer with learning rate 1e-3.
Revised: For semi-supervised learning, we add a new two-layer MLP on top of the trained encoder and fine-tune the whole model using 1% or 10% of data for 100 epochs, using batch size 128 and Adam optimizer with learning rate 1e-3.

Original: In both evaluation protocols, we remove the two-layer MLP of the encoder by replacing it with an identity function.
Revised: In both evaluation protocols, we remove the two-layer MLP of the encoder by replacing it with an identity function.

Original: C A DDITIONAL E XPERIMENTAL R ESULTS AND A NALYSIS
Revised: C A DDITIONAL E XPERIMENTAL R ESULTS AND A NALYSIS

Original: In this section, we provide more experimental results of algorithm comparisons and further analyze
Revised: In this section, we provide more experimental results of algorithm comparisons and further analyze

Original: FedEMA in different data amounts, training rounds R , and batch sizes B .
Revised: FedEMA in different data amounts, training rounds R , and batch sizes B .

Original: C. M ORE E XPERIMENTAL R ESULTS
Revised: C. M ORE E XPERIMENTAL R ESULTS

Original: Table 5 presents top-1 accuracy comparison under linear evaluation of a wide range of methods on
Revised: Table 5 presents top-1 accuracy comparison under linear evaluation of a wide range of methods on

Original: CIFAR datasets using both ResNet-18 and ResNet-50.
Revised: CIFAR datasets using both ResNet-18 and ResNet-50.

Original: It supplements the algorithm comparisons in Section 3.4 and 5.1.
Revised: It supplements the algorithm comparisons in Section 3.4 and 5.1.

Original: Interestingly, FedMoCoV1 achieves good performances on IID settings of
Revised: Interestingly, FedMoCoV1 achieves good performances on IID settings of

Original: [Figure]
Revised: [Figure]

Original: T o p1 A cc .
Revised: T o p1 A cc .

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Method
Revised: Method

Original: [Table]
Revised: [Table]

Original: E = 1 is unable to reach 80% in 100 rounds.
Revised: E = 1 is unable to reach 80% in 100 rounds.

Original: A larger E can reduce communication costs by increasing the computation cost.
Revised: A larger E can reduce communication costs by increasing the computation cost.

Original: [Table]
Revised: [Table]

Original: Communiation
Revised: Communiation

Original: [Table]
Revised: [Table]

Original: Computation (epochs)
Revised: Computation (epochs)

Original: E = E =
Revised: E = E =

Original: E = E =
Revised: E = E =

Original: E = E =
Revised: E = E =

Original: E = E =
Revised: E = E =

Original: [Table]
Revised: [Table]

Original: CIFAR-100 dataset.
Revised: CIFAR-100 dataset.

Original: Since decentralized data are mostly non-IID, we focus more on the non-IID setting.
Revised: Since decentralized data are mostly non-IID, we focus more on the non-IID setting.

Original: FedEMA outperforms all the other methods in non-IID settings of CIFAR datasets.
Revised: FedEMA outperforms all the other methods in non-IID settings of CIFAR datasets.

Original: We use λ = 0 .
Revised: We use λ = 0 .

Original: 8 when using ResNet-18 and λ = 1 when using ResNet-50.
Revised: 8 when using ResNet-18 and λ = 1 when using ResNet-50.

Original: Table 6 shows results of scaling to larger numbers of clients K with subsampling clients in each training round.
Revised: Table 6 shows results of scaling to larger numbers of clients K with subsampling clients in each training round.

Original: We run two sets of experiments: 1) randomly selecting 5 out of 20 clients in each round with local epoch E = 5 and total rounds R = 400 ; 2) randomly selecting 8 out ofclients in each round with local epoch E = 2 and total rounds R = 800 .
Revised: We run two sets of experiments: 1) randomly selecting 5 out of 20 clients in each round with local epoch E = 5 and total rounds R = 400 ; 2) randomly selecting 8 out ofclients in each round with local epoch E = 2 and total rounds R = 800 .

Original: We run FedEMA with autoscaler.
Revised: We run FedEMA with autoscaler.

Original: FedEMA consistently outperforms FedBYOL with both encoders updated.
Revised: FedEMA consistently outperforms FedBYOL with both encoders updated.

Original: We conduct these experiments using ResNet-18.
Revised: We conduct these experiments using ResNet-18.

Original: Table 7 supplements the semi-supervised learning results on Table 4, providing additional results using ResNet-50 as the network architecture for the encoder.
Revised: Table 7 supplements the semi-supervised learning results on Table 4, providing additional results using ResNet-50 as the network architecture for the encoder.

Original: FedEMA consistently outperforms all the other methods.
Revised: FedEMA consistently outperforms all the other methods.

Original: Besides, Table 8 and 9 compare FedSimSiam, FedBYOL, and variances of FedBYOL to further demonstrate the insights from empirical studies.
Revised: Besides, Table 8 and 9 compare FedSimSiam, FedBYOL, and variances of FedBYOL to further demonstrate the insights from empirical studies.

Original: They supplement results in Table 2 and Figure 3.
Revised: They supplement results in Table 2 and Figure 3.

Original: C. F URTHER A NALYSIS
Revised: C. F URTHER A NALYSIS

Original: τ for Autoscaler We analyze the impact of τ on performances in Figure 10a.
Revised: τ for Autoscaler We analyze the impact of τ on performances in Figure 10a.

Original: Generally, using autoscaler with τ ∈ (0 , 1) is better than FedBYOL ( τ = 0 ).
Revised: Generally, using autoscaler with τ ∈ (0 , 1) is better than FedBYOL ( τ = 0 ).

Original: The performance of τ = 1 yields worse results because only local knowledge are used in model update (the global knowledge is neglected)
Revised: The performance of τ = 1 yields worse results because only local knowledge are used in model update (the global knowledge is neglected)

Original: [Table]
Revised: [Table]

Original: as discussed in Section 4.
Revised: as discussed in Section 4.

Original: Besides, performances of τ ∈ [0 . 5 , 1) are generally better other values, which verifies our intuition discussed in Section 4.
Revised: Besides, performances of τ ∈ [0 . 5 , 1) are generally better other values, which verifies our intuition discussed in Section 4.

Original: These results also show that we can achieve even higher performance on the CIFAR-100 dataset on Table 3 by tunning τ .
Revised: These results also show that we can achieve even higher performance on the CIFAR-100 dataset on Table 3 by tunning τ .

Original: We run experiments on non-IID settings using ResNet-18.
Revised: We run experiments on non-IID settings using ResNet-18.

Original: Constant µ To further illustrate the effectiveness of our dynamic EMA, we provide results of using different combinations of constant µ o on the online encoder and constant µ p on the predictor in
Revised: Constant µ To further illustrate the effectiveness of our dynamic EMA, we provide results of using different combinations of constant µ o on the online encoder and constant µ p on the predictor in

Original: Figure 10b.
Revised: Figure 10b.

Original: The results of µ o = 0 .
Revised: The results of µ o = 0 .

Original: 9 and µ p = 0 .
Revised: 9 and µ p = 0 .

Original: 9 is only 54.52%, which is far lower than the others.
Revised: 9 is only 54.52%, which is far lower than the others.

Original: Among these combinations, µ o = 0 .
Revised: Among these combinations, µ o = 0 .

Original: 5 and µ p = 0 .
Revised: 5 and µ p = 0 .

Original: 3 achieve the best performance.
Revised: 3 achieve the best performance.

Original: It suggests that better performances may be achieved if we can construct different dynamic µ for the encoder and the predictor, while we leave this interesting insight for future exploration.
Revised: It suggests that better performances may be achieved if we can construct different dynamic µ for the encoder and the predictor, while we leave this interesting insight for future exploration.

Original: Although good choices of constant µ o and µ p achieve better performance than FedBYOL, FedEMA consistently outperforms all these methods.
Revised: Although good choices of constant µ o and µ p achieve better performance than FedBYOL, FedEMA consistently outperforms all these methods.

Original: These results complement Figure 7b in the main manuscript.
Revised: These results complement Figure 7b in the main manuscript.

Original: Impact of Training Rounds R Figure 9 compares FedBYOL and FedEMA with increasing number of training (communication) rounds R .
Revised: Impact of Training Rounds R Figure 9 compares FedBYOL and FedEMA with increasing number of training (communication) rounds R .

Original: Performances of both FedBYOL and FedEMA increases as training proceeds and FedEMA consistently outperforms FedBYOL.
Revised: Performances of both FedBYOL and FedEMA increases as training proceeds and FedEMA consistently outperforms FedBYOL.

Original: We run these experiments with λ = 0 .
Revised: We run these experiments with λ = 0 .

Original: 5 for FedEMA on the non-IID setting of CIFAR-10 dataset.
Revised: 5 for FedEMA on the non-IID setting of CIFAR-10 dataset.

Original: Impact of Batch Size B We investigate the impact of batch size in Table 10.
Revised: Impact of Batch Size B We investigate the impact of batch size in Table 10.

Original: The performances of batch size B = 128 and B = 256 are similar, outperforming the other batch sizes.
Revised: The performances of batch size B = 128 and B = 256 are similar, outperforming the other batch sizes.

Original: It indicates that the batch size should not be either too small or too large.
Revised: It indicates that the batch size should not be either too small or too large.

Original: Besides, FedEMA outperforms FedBYOL in all batch sizes.
Revised: Besides, FedEMA outperforms FedBYOL in all batch sizes.

Original: We run the experiments with autoscaler ( τ = 0 . 7 ) on the non-IID setting of the
Revised: We run the experiments with autoscaler ( τ = 0 . 7 ) on the non-IID setting of the

Original: CIFAR-10 dataset.
Revised: CIFAR-10 dataset.

Original: Communication vs Computation Cost Table 11 shows the needed communication rounds and computation epochs to reach a target accuracy using different local epochs E with FedEMA.
Revised: Communication vs Computation Cost Table 11 shows the needed communication rounds and computation epochs to reach a target accuracy using different local epochs E with FedEMA.

Original: Increasing E reduces communication cost as the needed rounds decrease, but it generally requires a higher computation cost.
Revised: Increasing E reduces communication cost as the needed rounds decrease, but it generally requires a higher computation cost.

Original: For example, compared with E = 5 that needs 80 rounds to reach 80% with 400 epochs of computation, E = 20 only uses 40 rounds but needs 800 epochs computation cost.
Revised: For example, compared with E = 5 that needs 80 rounds to reach 80% with 400 epochs of computation, E = 20 only uses 40 rounds but needs 800 epochs computation cost.

Original: These results indicate the trade-off between communication cost and computation cost.
Revised: These results indicate the trade-off between communication cost and computation cost.

Original: Data Amount Table 12 shows that increasing the data amount improves the performance significantly.
Revised: Data Amount Table 12 shows that increasing the data amount improves the performance significantly.

Original: By default, we split the CIFAR-10 dataset into 5 clients, each client contains 10,000 training images, denoting as 100% data amount.
Revised: By default, we split the CIFAR-10 dataset into 5 clients, each client contains 10,000 training images, denoting as 100% data amount.

Original: As a result, p % data amount means that each client contains 10 , 000 ∗ p % images.
Revised: As a result, p % data amount means that each client contains 10 , 000 ∗ p % images.

Original: For example, 25% data amount means that each client contains 2,500 images.
Revised: For example, 25% data amount means that each client contains 2,500 images.

Original: With lesser data points in each client, we can construct more clients to conduct training as the total data amount is fixed.
Revised: With lesser data points in each client, we can construct more clients to conduct training as the total data amount is fixed.

Original: Table 12 shows that when the data amount is same in clients, increasing the number of clients in each training round do not improve performance.
Revised: Table 12 shows that when the data amount is same in clients, increasing the number of clients in each training round do not improve performance.

Original: However, increasing the data amount in each client increases the performance significantly.
Revised: However, increasing the data amount in each client increases the performance significantly.

Original: These results indicate that it is important for clients to have sufficient data to participate in training in FedSSL.
Revised: These results indicate that it is important for clients to have sufficient data to participate in training in FedSSL.