Original: F EDERATED S EMI -S UPERVISED L EARNING WITH I NTER -C LIENT C ONSISTENCY & D ISJOINT L EARNING
Revised: F EDERATED S EMI -S UPERVISED L EARNING WITH I NTER -C LIENT C ONSISTENCY & D ISJOINT L EARNING

Original: A BSTRACT
Revised: A BSTRACT

Original: While existing federated learning approaches mostly require that clients have fullylabeled data to train on, in realistic settings, data obtained at the client-side often comes without any accompanying labels.
Revised: While existing federated learning approaches mostly require that clients have fullylabeled data to train on, in realistic settings, data obtained at the client-side often comes without any accompanying labels.

Original: Such deﬁciency of labels may result from either high labeling cost, or difﬁculty of annotation due to the requirement of expert knowledge.
Revised: Such deﬁciency of labels may result from either high labeling cost, or difﬁculty of annotation due to the requirement of expert knowledge.

Original: Thus the private data at each client may be either partly labeled, or completely unlabeled with labeled data being available only at the server, which leads us to a new practical federated learning problem, namely Federated SemiSupervised Learning (FSSL).
Revised: Thus the private data at each client may be either partly labeled, or completely unlabeled with labeled data being available only at the server, which leads us to a new practical federated learning problem, namely Federated SemiSupervised Learning (FSSL).

Original: In this work, we study two essential scenarios of FSSL based on the location of the labeled data.
Revised: In this work, we study two essential scenarios of FSSL based on the location of the labeled data.

Original: The ﬁrst scenario considers a conventional case where clients have both labeled and unlabeled data (labels-at-client), and the second scenario considers a more challenging case, where the labeled data is only available at the server (labels-at-server).
Revised: The ﬁrst scenario considers a conventional case where clients have both labeled and unlabeled data (labels-at-client), and the second scenario considers a more challenging case, where the labeled data is only available at the server (labels-at-server).

Original: We then propose a novel method to tackle the problems, which we refer to as Federated Matching (FedMatch).
Revised: We then propose a novel method to tackle the problems, which we refer to as Federated Matching (FedMatch).

Original: FedMatch improves upon naive combinations of federated learning and semi-supervised learning approaches with a new inter-client consistency loss and decomposition of the parameters for disjoint learning on labeled and unlabeled data.
Revised: FedMatch improves upon naive combinations of federated learning and semi-supervised learning approaches with a new inter-client consistency loss and decomposition of the parameters for disjoint learning on labeled and unlabeled data.

Original: Through extensive experimental validation of our method in the two different scenarios, we show that our method outperforms both local semi-supervised learning and baselines which naively combine federated learning with semi-supervised learning.
Revised: Through extensive experimental validation of our method in the two different scenarios, we show that our method outperforms both local semi-supervised learning and baselines which naively combine federated learning with semi-supervised learning.

Original: The code is available at https://github.com/wyjeong/FedMatch
Revised: The code is available at https://github.com/wyjeong/FedMatch

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Federated Learning (FL) (McMahan et al., 2017; Zhao et al., 2018; Li et al., 2018; Chen et al., 2019a;b), in which multiple clients collaboratively learn a global model via coordinated communication, has been an active topic of research over the past few years.
Revised: Federated Learning (FL) (McMahan et al., 2017; Zhao et al., 2018; Li et al., 2018; Chen et al., 2019a;b), in which multiple clients collaboratively learn a global model via coordinated communication, has been an active topic of research over the past few years.

Original: The most distinctive difference of federated learning from distributed learning is that the data is only privately accessible at each local client, without inter-client data sharing.
Revised: The most distinctive difference of federated learning from distributed learning is that the data is only privately accessible at each local client, without inter-client data sharing.

Original: Such decentralized learning brings us numerous advantages in addressing real-world issues such as data privacy, security, and access rights.
Revised: Such decentralized learning brings us numerous advantages in addressing real-world issues such as data privacy, security, and access rights.

Original: For example, for on-device learning of mobile devices, the service provider may not directly access local data since they may contain privacy-sensitive information.
Revised: For example, for on-device learning of mobile devices, the service provider may not directly access local data since they may contain privacy-sensitive information.

Original: In healthcare domains, the hospitals may want to improve their clinical diagnosis systems without sharing the patient records.
Revised: In healthcare domains, the hospitals may want to improve their clinical diagnosis systems without sharing the patient records.

Original: Existing federated learning approaches (McMahan et al., 2017; Wang et al., 2020; Li et al., 2018) handle these problems by aggregating the locally learned model parameters.
Revised: Existing federated learning approaches (McMahan et al., 2017; Wang et al., 2020; Li et al., 2018) handle these problems by aggregating the locally learned model parameters.

Original: A common limitation is that they only consider supervised learning settings, where the local private data is fully labeled.
Revised: A common limitation is that they only consider supervised learning settings, where the local private data is fully labeled.

Original: Yet, the assumption that all of the data examples may include sophisticate annotations is not realistic for real-world applications.
Revised: Yet, the assumption that all of the data examples may include sophisticate annotations is not realistic for real-world applications.

Original: Suppose that we perform on-device federated learning, the users may not want to spend their time and efforts in annotating the data, and the participation rate across the users may largely differ.
Revised: Suppose that we perform on-device federated learning, the users may not want to spend their time and efforts in annotating the data, and the participation rate across the users may largely differ.

Original: Even in the case of enthusiastic users may not be able to fully label all the data in the device, which will leave the majority of the data as unlabeled (See Figure 1 (a)) .
Revised: Even in the case of enthusiastic users may not be able to fully label all the data in the device, which will leave the majority of the data as unlabeled (See Figure 1 (a)) .

Original: Moreover, in some scenarios, the users may not have sufﬁcient expertise to correctly label the data.
Revised: Moreover, in some scenarios, the users may not have sufﬁcient expertise to correctly label the data.

Original: For instance, suppose that we have a workout app that automatically evaluates and corrects one’s body posture.
Revised: For instance, suppose that we have a workout app that automatically evaluates and corrects one’s body posture.

Original: In this case, the end users may not be able to evaluate his/her own body posture at all (See Figure
Revised: In this case, the end users may not be able to evaluate his/her own body posture at all (See Figure

Original: [Figure]
Revised: [Figure]

Original: (b)) .
Revised: (b)) .

Original: Thus, in many realistic scenarios for federated learning, local data will be mostly unlabeled .
Revised: Thus, in many realistic scenarios for federated learning, local data will be mostly unlabeled .

Original: This leads us to practical problems of federated learning with deﬁciency of labels, namely Federated
Revised: This leads us to practical problems of federated learning with deﬁciency of labels, namely Federated

Original: Semi-Supervised Learning (FSSL).
Revised: Semi-Supervised Learning (FSSL).

Original: A naive solution to these scenarios is to simply perform Semi-Supervised Learning (SSL) using any off-the-shelf methods (e.g. FixMatch (Sohn et al., 2020), UDA (Xie et al., 2019)), while using federated learning algorithms to aggregate the learned weights.
Revised: A naive solution to these scenarios is to simply perform Semi-Supervised Learning (SSL) using any off-the-shelf methods (e.g. FixMatch (Sohn et al., 2020), UDA (Xie et al., 2019)), while using federated learning algorithms to aggregate the learned weights.

Original: Yet, this does not fully exploit the knowledge of the multiple models trained on heterogeneous data distributions.
Revised: Yet, this does not fully exploit the knowledge of the multiple models trained on heterogeneous data distributions.

Original: To address this problem, we present a novel framework which we refer to as Federated Matching (FedMatch) , which enforces the consistency between the predictions made across multiple models.
Revised: To address this problem, we present a novel framework which we refer to as Federated Matching (FedMatch) , which enforces the consistency between the predictions made across multiple models.

Original: Further, conventional semi-supervised learning approaches are not applicable for scenarios where labeled data is only available at the server (Figure 1 (b)), which is a unique SSL setting for federated learning.
Revised: Further, conventional semi-supervised learning approaches are not applicable for scenarios where labeled data is only available at the server (Figure 1 (b)), which is a unique SSL setting for federated learning.

Original: Also, even when the labeled data is available at the client (Figure 1 (a)), learning from the unlabeled data may lead to forgetting of what the model learned from the labeled data.
Revised: Also, even when the labeled data is available at the client (Figure 1 (a)), learning from the unlabeled data may lead to forgetting of what the model learned from the labeled data.

Original: To tackle these issues, we decompose the model parameters into two, a dense parameter for supervised and a sparse parameter for unsupervised learning.
Revised: To tackle these issues, we decompose the model parameters into two, a dense parameter for supervised and a sparse parameter for unsupervised learning.

Original: This sparse additive parameter decomposition ensures that training on labeled and unlabeled data are effectively separable, thus minimizing interference between the two tasks.
Revised: This sparse additive parameter decomposition ensures that training on labeled and unlabeled data are effectively separable, thus minimizing interference between the two tasks.

Original: We further reduce the communication costs with both the decomposed parameters by sending only the difference of the parameters across the communication rounds.
Revised: We further reduce the communication costs with both the decomposed parameters by sending only the difference of the parameters across the communication rounds.

Original: We validate FedMatch on both scenarios (Figure 1 (a) and (b)) and show that our models signiﬁcantly outperform baselines, including a naive combination of federated learning with semi-supervised learning, on the training data which are both non-i.i.d.
Revised: We validate FedMatch on both scenarios (Figure 1 (a) and (b)) and show that our models signiﬁcantly outperform baselines, including a naive combination of federated learning with semi-supervised learning, on the training data which are both non-i.i.d.

Original: and i.i.d.
Revised: and i.i.d.

Original: data.
Revised: data.

Original: The main contributions of this work are as follows:
Revised: The main contributions of this work are as follows:

Original: • We introduce a practical problem of federated learning with deﬁciency of supervision, namely Federated Semi-Supervised Learning (FSSL) , and study two different scenarios, where the local data is partly labeled ( Labels-at-Client ) or completely unlabeled ( Labels-at-Server ).
Revised: • We introduce a practical problem of federated learning with deﬁciency of supervision, namely Federated Semi-Supervised Learning (FSSL) , and study two different scenarios, where the local data is partly labeled ( Labels-at-Client ) or completely unlabeled ( Labels-at-Server ).

Original: • We propose a novel method , Federated Matching (FedMatch) , which learns inter-client consistency between multiple clients, and decomposes model parameters to reduce both interference between supervised and unsupervised tasks, and communication cost.
Revised: • We propose a novel method , Federated Matching (FedMatch) , which learns inter-client consistency between multiple clients, and decomposes model parameters to reduce both interference between supervised and unsupervised tasks, and communication cost.

Original: • We show that our method, FedMatch, signiﬁcantly outperforms both local SSL and the naive combination of FL with SSL algorithms under the conventional labels-at-client and the novel labels-at-server scenario, across multiple clients with both non-i.i.d.
Revised: • We show that our method, FedMatch, signiﬁcantly outperforms both local SSL and the naive combination of FL with SSL algorithms under the conventional labels-at-client and the novel labels-at-server scenario, across multiple clients with both non-i.i.d.

Original: and i.i.d.
Revised: and i.i.d.

Original: data.
Revised: data.

Original:  P ROBLEM D EFINITION
Revised:  P ROBLEM D EFINITION

Original: We begin with formal deﬁnition of Federated Learning (FL) and Semi-Supervised Learning (SSL).
Revised: We begin with formal deﬁnition of Federated Learning (FL) and Semi-Supervised Learning (SSL).

Original: Then, we deﬁne Federated Semi-Supervised Learning (FSSL) and introduce two essential scenarios.
Revised: Then, we deﬁne Federated Semi-Supervised Learning (FSSL) and introduce two essential scenarios.

Original: 2. P RELIMINARIES
Revised: 2. P RELIMINARIES

Original: Federated Learning Federated Learning (FL) aims to collaboratively learn a global model via coordinated communication with multiple clients.
Revised: Federated Learning Federated Learning (FL) aims to collaboratively learn a global model via coordinated communication with multiple clients.

Original: Let G be a global model and L = { l k } K k
Revised: Let G be a global model and L = { l k } K k

Original: =1 be a set of local models for K clients.
Revised: =1 be a set of local models for K clients.

Original: Let D = { x i , y i } N i =1 be a given dataset, where x i is an arbitrary training instance with a corresponding one-hot label y i ∈ { 1 , . .
Revised: Let D = { x i , y i } N i =1 be a given dataset, where x i is an arbitrary training instance with a corresponding one-hot label y i ∈ { 1 , . .

Original: , C } for the C -way multi-class classiﬁcation problem and N is the number of instances.
Revised: , C } for the C -way multi-class classiﬁcation problem and N is the number of instances.

Original: D is composed of K sub-datasets D l k = { x l k
Revised: D is composed of K sub-datasets D l k = { x l k

Original: i , y l k i } N lk i = privately collected at each client or local model l k .
Revised: i , y l k i } N lk i = privately collected at each client or local model l k .

Original: At each communication round r , G ﬁrst randomly selects A local models that are available for training L r ⊂ L and |L
Revised: At each communication round r , G ﬁrst randomly selects A local models that are available for training L r ⊂ L and |L

Original: r | =
Revised: r | =

Original: A .
Revised: A .

Original: The global model G then initializes L r with global weights θ G , and the active local models l a ∈ L r perform supervised learning to minimize loss (cid:96) s ( θ l a ) on the corresponding sub-dataset D l a .
Revised: The global model G then initializes L r with global weights θ G , and the active local models l a ∈ L r perform supervised learning to minimize loss (cid:96) s ( θ l a ) on the corresponding sub-dataset D l a .

Original: G then aggregates the learned weights θ G
Revised: G then aggregates the learned weights θ G

Original: ← N la N (cid:80) Aa θ l a and broadcasts newly aggregated weights to local models that would be available at the next round r + 1 , and repeat the learning procedure until the ﬁnal round R .
Revised: ← N la N (cid:80) Aa θ l a and broadcasts newly aggregated weights to local models that would be available at the next round r + 1 , and repeat the learning procedure until the ﬁnal round R .

Original: Semi-Supervised Learning Semi-supervised learning (SSL) refers to the problem of learning with partially labeled data, where the ratio of unlabeled data is usually much larger than that of the labeled data (e.g. 1 : 10 ).
Revised: Semi-Supervised Learning Semi-supervised learning (SSL) refers to the problem of learning with partially labeled data, where the ratio of unlabeled data is usually much larger than that of the labeled data (e.g. 1 : 10 ).

Original: For SSL, D is further split into labeled and unlabeled data.
Revised: For SSL, D is further split into labeled and unlabeled data.

Original: Let S
Revised: Let S

Original: = { x i , y i } Si
Revised: = { x i , y i } Si

Original: = be a set of S labeled data instances and U
Revised: = be a set of S labeled data instances and U

Original: = { u i } Ui =1 be a set of U unlabeled samples without corresponding label.
Revised: = { u i } Ui =1 be a set of U unlabeled samples without corresponding label.

Original: Here, in general, |S| (cid:28) |U| .
Revised: Here, in general, |S| (cid:28) |U| .

Original: With these two datasets, S and U , we now perform semi-supervised learning.
Revised: With these two datasets, S and U , we now perform semi-supervised learning.

Original: Let p θ ( y | x ) be a neural network that is parameterized by weights θ and predicts softmax outputs ˆ y with given input x .
Revised: Let p θ ( y | x ) be a neural network that is parameterized by weights θ and predicts softmax outputs ˆ y with given input x .

Original: Our objective is to minimize loss function (cid:96) final ( θ ) =
Revised: Our objective is to minimize loss function (cid:96) final ( θ ) =

Original: (cid:96) s ( θ )
Revised: (cid:96) s ( θ )

Original: + (cid:96) u
Revised: + (cid:96) u

Original: ( θ ) , where (cid:96) s ( θ ) is loss term for supervised learning on S and (cid:96) u
Revised: ( θ ) , where (cid:96) s ( θ ) is loss term for supervised learning on S and (cid:96) u

Original: ( θ ) is loss term for unsupervised learning on U
Revised: ( θ ) is loss term for unsupervised learning on U

Original: 2. F EDERATED S EMI -S UPERVISED L EARNING
Revised: 2. F EDERATED S EMI -S UPERVISED L EARNING

Original: Now we further describe a practical problem of deﬁciency of labels in federated learning, which we refer to as Federated Semi-Supervised Learning (FSSL), in which the data obtained at the clients may or may not come with accompanying labels.
Revised: Now we further describe a practical problem of deﬁciency of labels in federated learning, which we refer to as Federated Semi-Supervised Learning (FSSL), in which the data obtained at the clients may or may not come with accompanying labels.

Original: Given a dataset D = { x i , y i } Ni =1 , D is split into a labeled set S = { x i , y i } Si =1 and a unlabeled set U = { u i } Ui =1 as in the standard semi-supervised learning.
Revised: Given a dataset D = { x i , y i } Ni =1 , D is split into a labeled set S = { x i , y i } Si =1 and a unlabeled set U = { u i } Ui =1 as in the standard semi-supervised learning.

Original: Under the federated learning framework, we have a global model G and a set of local models L where the unlabeled dataset U is privately spread over K clients hence U l k
Revised: Under the federated learning framework, we have a global model G and a set of local models L where the unlabeled dataset U is privately spread over K clients hence U l k

Original: = { u l k i } U
Revised: = { u l k i } U

Original: lk i =1 .
Revised: lk i =1 .

Original: For a labeled set S , on the other hand, we consider two different scenarios depending on the availability of labeled data at clients, namely the Labels-at-Client and the Labels-at-Server scenario, of which problem settings and learning procedures will be discussed later.
Revised: For a labeled set S , on the other hand, we consider two different scenarios depending on the availability of labeled data at clients, namely the Labels-at-Client and the Labels-at-Server scenario, of which problem settings and learning procedures will be discussed later.

Original:  F EDERATED M ATCHING
Revised:  F EDERATED M ATCHING

Original: We now describe our Federated Matching (FedMatch) algorithm to tackle the federated semisupervised learning problem.
Revised: We now describe our Federated Matching (FedMatch) algorithm to tackle the federated semisupervised learning problem.

Original: We describe its core components in detail in the following subsections.
Revised: We describe its core components in detail in the following subsections.

Original: 3. I NTER -C LIENT C ONSISTENCY L OSS
Revised: 3. I NTER -C LIENT C ONSISTENCY L OSS

Original: Consistency regularization (Xie et al., 2019; Sohn et al., 2020; Berthelot et al., 2019b;a) is one of most popular approaches to learn from unlabeled examples in a semi-supervised learning setting.
Revised: Consistency regularization (Xie et al., 2019; Sohn et al., 2020; Berthelot et al., 2019b;a) is one of most popular approaches to learn from unlabeled examples in a semi-supervised learning setting.

Original: Conventional consistency-regularization methods enforce the predictions from the augmented examples and original (or weakly augmented) instances to output the same class label,
Revised: Conventional consistency-regularization methods enforce the predictions from the augmented examples and original (or weakly augmented) instances to output the same class label,

Original: ||
Revised: ||

Original: p θ ( y | u ) − p θ ( y | π ( u ))
Revised: p θ ( y | u ) − p θ ( y | π ( u ))

Original: || 22 , where π ( · ) is a stochastic transformation function.
Revised: || 22 , where π ( · ) is a stochastic transformation function.

Original: Based on the assumption that class semantics are unaffected by small input perturbations, these methods basically ensures consistency of the prediction across the multiple perturbations of same input .
Revised: Based on the assumption that class semantics are unaffected by small input perturbations, these methods basically ensures consistency of the prediction across the multiple perturbations of same input .

Original: For our federated semi-supervised learning method, we additionally propose a novel consistency loss that regularizes the models learned at multiple clients to output the same prediction.
Revised: For our federated semi-supervised learning method, we additionally propose a novel consistency loss that regularizes the models learned at multiple clients to output the same prediction.

Original: This novel consistency loss for FSSL, inter-client consistency loss, is deﬁned as follows:
Revised: This novel consistency loss for FSSL, inter-client consistency loss, is deﬁned as follows:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ) where p ∗ θ h ( y | x ) are helper agents that are selected from the server based on model similarity to the client (which we describe later), that are not trained at the client (* denotes that we freeze the parameters).
Revised: ) where p ∗ θ h ( y | x ) are helper agents that are selected from the server based on model similarity to the client (which we describe later), that are not trained at the client (* denotes that we freeze the parameters).

Original: The server selects and broadcasts H helper agents at each communication round.
Revised: The server selects and broadcasts H helper agents at each communication round.

Original: We also use data-level consistency regularization at each local client similarly to FixMatch (Sohn et al., 2020).
Revised: We also use data-level consistency regularization at each local client similarly to FixMatch (Sohn et al., 2020).

Original: Our ﬁnal consistency regularization term Φ( · ) can be written as follows:
Revised: Our ﬁnal consistency regularization term Φ( · ) can be written as follows:

Original: [Equation]
Revised: [Equation]

Original: where π ( u ) performs RandAugment (Cubuk et al., 2019) on unlabeld instance u .
Revised: where π ( u ) performs RandAugment (Cubuk et al., 2019) on unlabeld instance u .

Original: ˆ y is our novel pseudo-labeling technique, which we refer to as the agreement-based pseudo label, deﬁned as follows:
Revised: ˆ y is our novel pseudo-labeling technique, which we refer to as the agreement-based pseudo label, deﬁned as follows:

Original: [Equation]
Revised: [Equation]

Original: where 1 ( · ) produces one-hot labels with given softmax values , and Max ( · ) outputs one-hot labels on the class that has the maximum agreements.
Revised: where 1 ( · ) produces one-hot labels with given softmax values , and Max ( · ) outputs one-hot labels on the class that has the maximum agreements.

Original: We discard instances with low-conﬁdent predictions below conﬁdence threshold τ when generating pseudo-labels.
Revised: We discard instances with low-conﬁdent predictions below conﬁdence threshold τ when generating pseudo-labels.

Original: We then perform standard cross-entropy minimization with the pseudo-label ˆ y .
Revised: We then perform standard cross-entropy minimization with the pseudo-label ˆ y .

Original: For helper agents, we select the H helper agents p ∗ θ h j : H ( y | u ) for each client as the most relevant models from other clients.
Revised: For helper agents, we select the H helper agents p ∗ θ h j : H ( y | u ) for each client as the most relevant models from other clients.

Original: Speciﬁcally, we represent each model by its prediction m on the same arbitrary input a located at server (we use random Gaussian noise), such that m l = p θ l ( m | a ) .
Revised: Speciﬁcally, we represent each model by its prediction m on the same arbitrary input a located at server (we use random Gaussian noise), such that m l = p θ l ( m | a ) .

Original: The server tries to keep and update all model embeddings m 1: K from clients once each client updates its weights to server, and creates K-Dimensional Tree (KD Tree) on m in the current round r for nearest neighbor search to rapidly select the H helper agents for each client in the next rounds.
Revised: The server tries to keep and update all model embeddings m 1: K from clients once each client updates its weights to server, and creates K-Dimensional Tree (KD Tree) on m in the current round r for nearest neighbor search to rapidly select the H helper agents for each client in the next rounds.

Original: We send helper agents for every 10 rounds, and if a certain client has not yet updated its weights to server in the previous step, then server simply skips sending helpers to the client at the round.
Revised: We send helper agents for every 10 rounds, and if a certain client has not yet updated its weights to server in the previous step, then server simply skips sending helpers to the client at the round.

Original: 3. P ARAMETER D ECOMPOSITION FOR D ISJOINT L EARNING
Revised: 3. P ARAMETER D ECOMPOSITION FOR D ISJOINT L EARNING

Original: In the standard semi-supervised learning approaches, learning on labeled and unlabeled data is simultaneously done with a shared set of parameters.
Revised: In the standard semi-supervised learning approaches, learning on labeled and unlabeled data is simultaneously done with a shared set of parameters.

Original: However, since this is inapplicable to the disjoint learning scenario (Figure 1 (b)) and may result in forgetting of knowledge of labeled data (see
Revised: However, since this is inapplicable to the disjoint learning scenario (Figure 1 (b)) and may result in forgetting of knowledge of labeled data (see

Original: Figure 6 (c)), we separate the supervised and unsupervised learning via the decomposition of model parameters into two sets of parameters, one for supervised learning and another for unsupervised learning.
Revised: Figure 6 (c)), we separate the supervised and unsupervised learning via the decomposition of model parameters into two sets of parameters, one for supervised learning and another for unsupervised learning.

Original: To this end, we decompose our model parameters θ into two variables, σ for supervised learning and ψ for unsupervised learning, such that θ = σ + ψ .
Revised: To this end, we decompose our model parameters θ into two variables, σ for supervised learning and ψ for unsupervised learning, such that θ = σ + ψ .

Original: We perform standard supervised learning on σ , while keeping ψ ﬁxed during training, by minimizing the loss term as follows:
Revised: We perform standard supervised learning on σ , while keeping ψ ﬁxed during training, by minimizing the loss term as follows:

Original: [Equation]
Revised: [Equation]

Original: where x and y are from labeled set S .
Revised: where x and y are from labeled set S .

Original: For learning on unlabeled data, we perform unsupervised learning conversely on ψ , while keeping σ ﬁxed for the learning phase, by minimizing the consistency loss terms as follows:
Revised: For learning on unlabeled data, we perform unsupervised learning conversely on ψ , while keeping σ ﬁxed for the learning phase, by minimizing the consistency loss terms as follows:

Original: [Equation]
Revised: [Equation]

Original: where all λ s are hyper-parameters to control the learning ratio between the terms.
Revised: where all λ s are hyper-parameters to control the learning ratio between the terms.

Original: We additionally add L 2 - and L 1 -Regularization on ψ such that ψ is sparse, while not drifting far from knowledge that σ has learned.
Revised: We additionally add L 2 - and L 1 -Regularization on ψ such that ψ is sparse, while not drifting far from knowledge that σ has learned.

Original: To sum up, our decomposition technique allows us:
Revised: To sum up, our decomposition technique allows us:

Original: • Preservation Reliable Knowledge from Labeled Data : We empirically ﬁnd that learning on both labeled and unlabeled data with a single set of parameters may result in the model to forget about what it learned from the labeled data (see Figure 6 (c)).
Revised: • Preservation Reliable Knowledge from Labeled Data : We empirically ﬁnd that learning on both labeled and unlabeled data with a single set of parameters may result in the model to forget about what it learned from the labeled data (see Figure 6 (c)).

Original: Our method can effectively prevent the inter-task interference via utilizing disjoint parameters only for supervised learning.
Revised: Our method can effectively prevent the inter-task interference via utilizing disjoint parameters only for supervised learning.

Original: • Reduction of Communication Costs : Sparsity on the unsupervised parameter ψ allows to reduce communication cost.
Revised: • Reduction of Communication Costs : Sparsity on the unsupervised parameter ψ allows to reduce communication cost.

Original: In addition, we further minimize the cost by subtracting the learned knowledge for each parameter, such that ∆ ψ = ψ lr − ψ Gr and ∆ σ = σ lr − σ Gr , and transmit only the differences ∆ ψ and ∆ σ as sparse matrices for both client-to-server and server-to-client costs.
Revised: In addition, we further minimize the cost by subtracting the learned knowledge for each parameter, such that ∆ ψ = ψ lr − ψ Gr and ∆ σ = σ lr − σ Gr , and transmit only the differences ∆ ψ and ∆ σ as sparse matrices for both client-to-server and server-to-client costs.

Original: • Disjoint Learning :
Revised: • Disjoint Learning :

Original: In federated semi-supervised learning, labeled data can be located at either client or server, which requires the model’s learning procedure to be ﬂexible.
Revised: In federated semi-supervised learning, labeled data can be located at either client or server, which requires the model’s learning procedure to be ﬂexible.

Original: Our decomposition technique allows the model for the supervised training to be done separately elsewhere.
Revised: Our decomposition technique allows the model for the supervised training to be done separately elsewhere.

Original: 1: RunServer() 2: initialize σ 0 and ψ3: for each round r = 1 , 2 , ..., R do 4:
Revised: 1: RunServer() 2: initialize σ 0 and ψ3: for each round r = 1 , 2 , ..., R do 4:

Original: L r ← (select random A clients from L ) 5: for each client l ra ∈ L r in parallel do 6: ψ r 1: H ← GetNearestNeighbors( ψ r ) 7: σ ra , ψ ra ← RunClient( σ r , ψ r , ψ r 1: H ) 8:
Revised: L r ← (select random A clients from L ) 5: for each client l ra ∈ L r in parallel do 6: ψ r 1: H ← GetNearestNeighbors( ψ r ) 7: σ ra , ψ ra ← RunClient( σ r , ψ r , ψ r 1: H ) 8:

Original: EmbedLocalModel( σ ra , ψ ra ) 9: end for 10: σ r +1 ← 1 A (cid:80) Aa =1 ( σ rl a ) 11: ψ r +1 ← 1 A (cid:80) Aa =1 ( ψ rl a ) 12: end for 13: RunClient( σ, ψ, ψ 1: H ) 14: θ l a ← σ + ψ , θ h 1: H ← σ + ψ 1: H 15: for each local epoch e from 1 to E L do 16: for minibatch s ∈ S l a and u ∈ U l a do 17: θ σ + ψ ∗ ← θ σ + ψ ∗ − η ∇ (cid:96) s ( θ σ + ψ ∗ ; θ h 1: H , s ) 18: θ σ ∗ + ψ ← θ σ ∗ + ψ − η ∇ (cid:96) u ( θ σ ∗ + ψ ; θ h 1: H , u ) 19: end for 20: end for
Revised: EmbedLocalModel( σ ra , ψ ra ) 9: end for 10: σ r +1 ← 1 A (cid:80) Aa =1 ( σ rl a ) 11: ψ r +1 ← 1 A (cid:80) Aa =1 ( ψ rl a ) 12: end for 13: RunClient( σ, ψ, ψ 1: H ) 14: θ l a ← σ + ψ , θ h 1: H ← σ + ψ 1: H 15: for each local epoch e from 1 to E L do 16: for minibatch s ∈ S l a and u ∈ U l a do 17: θ σ + ψ ∗ ← θ σ + ψ ∗ − η ∇ (cid:96) s ( θ σ + ψ ∗ ; θ h 1: H , s ) 18: θ σ ∗ + ψ ← θ σ ∗ + ψ − η ∇ (cid:96) u ( θ σ ∗ + ψ ; θ h 1: H , u ) 19: end for 20: end for

Original:  L ABELS - AT -C LIENT S CENARIO
Revised:  L ABELS - AT -C LIENT S CENARIO

Original: Problem Deﬁnition
Revised: Problem Deﬁnition

Original: The Labels-at-Client scenario posits that the end-users intermittently annotate a small portion of their local data (i.e., 5% of the entire data), leaving the rest of data instances unlabeled as illustrated in Figure 1 (a).
Revised: The Labels-at-Client scenario posits that the end-users intermittently annotate a small portion of their local data (i.e., 5% of the entire data), leaving the rest of data instances unlabeled as illustrated in Figure 1 (a).

Original: This is a common scenario for user-generated personal data, where the end-users can easily annotate the data but may not have time or motivation to label all the data (e.g. annotating faces in pictures for photo albums or social networking).
Revised: This is a common scenario for user-generated personal data, where the end-users can easily annotate the data but may not have time or motivation to label all the data (e.g. annotating faces in pictures for photo albums or social networking).

Original: We assume that clients train on both labeled and unlabeled data, while the server only aggregates the updates from the clients and redistributes the aggregated parameters back to the clients.
Revised: We assume that clients train on both labeled and unlabeled data, while the server only aggregates the updates from the clients and redistributes the aggregated parameters back to the clients.

Original: In this scenario, labeled data S is a set of individual sub-datasets S l k = { x l k
Revised: In this scenario, labeled data S is a set of individual sub-datasets S l k = { x l k

Original: i , y l k i } S lk i =1 , yielding K sub-datasets for K local models l 1: K .
Revised: i , y l k i } S lk i =1 , yielding K sub-datasets for K local models l 1: K .

Original: The overall learning procedure of the global model is the same as that of conventional federated learning (global model G aggregates updates from the selected subset of clients and broadcasts them), except that active local models l 1: A perform semi-supervised learning by minimizing the loss (cid:96) final ( θ l a ) =
Revised: The overall learning procedure of the global model is the same as that of conventional federated learning (global model G aggregates updates from the selected subset of clients and broadcasts them), except that active local models l 1: A perform semi-supervised learning by minimizing the loss (cid:96) final ( θ l a ) =

Original: (cid:96) s ( θ l a ) + (cid:96) u
Revised: (cid:96) s ( θ l a ) + (cid:96) u

Original: ( θ l a ) respectively on S l a and U l a .
Revised: ( θ l a ) respectively on S l a and U l a .

Original: FedMatch Algorithm for Labels-at-Client Scenario
Revised: FedMatch Algorithm for Labels-at-Client Scenario

Original: Now we introduce our FedMatch algorithm for the labels-at-client scenario.
Revised: Now we introduce our FedMatch algorithm for the labels-at-client scenario.

Original: As shown in Figure 3, which illustrates an example case of the labels-at-client scenario, active local models l 1: A at the current round r learn both σ l r 1: A and ψ l r 1: A on both the labeled data S l 1: A and unlabeled data U l 1: A at each local environment.
Revised: As shown in Figure 3, which illustrates an example case of the labels-at-client scenario, active local models l 1: A at the current round r learn both σ l r 1: A and ψ l r 1: A on both the labeled data S l 1: A and unlabeled data U l 1: A at each local environment.

Original: After the completion of local training, the clients update both their learned knowledge σ l r 1: A and ψ l r 1: A to the server.
Revised: After the completion of local training, the clients update both their learned knowledge σ l r 1: A and ψ l r 1: A to the server.

Original: The server then aggregates σ l 1: A and ψ l 1: A , respectively , after embedding local models based on model similarity as well as create KD-Tree to rapidly retrieve the top- H nearest neighbors ψ h 1: H for each client.
Revised: The server then aggregates σ l 1: A and ψ l 1: A , respectively , after embedding local models based on model similarity as well as create KD-Tree to rapidly retrieve the top- H nearest neighbors ψ h 1: H for each client.

Original: At the next round, the server transmits the aggregated σ r +1 and ψ r +1 .
Revised: At the next round, the server transmits the aggregated σ r +1 and ψ r +1 .

Original: For helper agents, server retrieves H helper agents, ψ h 1: H , to each client for every 10 rounds.
Revised: For helper agents, server retrieves H helper agents, ψ h 1: H , to each client for every 10 rounds.

Original: More details of the training procedures for FedMatch, for the labels-at-client scenario, is described in Algorithm 1.
Revised: More details of the training procedures for FedMatch, for the labels-at-client scenario, is described in Algorithm 1.

Original:  L ABELS - AT -S ERVER S CENARIO
Revised:  L ABELS - AT -S ERVER S CENARIO

Original: Problem Deﬁnition We now describe another realistic setting, which is the labels-at-server scenario.
Revised: Problem Deﬁnition We now describe another realistic setting, which is the labels-at-server scenario.

Original: This scenario assumes that the supervised labels are only available at the server, while local clients work with unlabeled data as described in Figure 1 (b).
Revised: This scenario assumes that the supervised labels are only available at the server, while local clients work with unlabeled data as described in Figure 1 (b).

Original: This is a common case of real-world applications where labeling requires expert knowledge (e.g. annotating medical images, evaluating body postures for exercises), but the data cannot be shared due to privacy concerns.
Revised: This is a common case of real-world applications where labeling requires expert knowledge (e.g. annotating medical images, evaluating body postures for exercises), but the data cannot be shared due to privacy concerns.

Original: In this scenario,
Revised: In this scenario,

Original: S G is identical to S and is located at server.
Revised: S G is identical to S and is located at server.

Original: The overall learning procedure is the same as that of conventional federated learning, except the global model G performs supervised learning on S G by minimizing the loss (cid:96) s ( θ G ) before broadcasting θ G to local clients.
Revised: The overall learning procedure is the same as that of conventional federated learning, except the global model G performs supervised learning on S G by minimizing the loss (cid:96) s ( θ G ) before broadcasting θ G to local clients.

Original: Then, the active local clients l 1: A at communication round r perform unsupervised learning which solely minimizes (cid:96) u
Revised: Then, the active local clients l 1: A at communication round r perform unsupervised learning which solely minimizes (cid:96) u

Original: ( θ l a ) on the unlabeled data U l a .
Revised: ( θ l a ) on the unlabeled data U l a .

Original: 1: RunServer() 2: initialize σ 0 , ψ3: for each round r = 1 , 2 , ..., R do 4: for each server epoch e from 1 to E G do 5: for minibatch s ∈ S G do 6: θ σ + ψ ∗ ← θ σ + ψ ∗ − η ∇ (cid:96) s ( θ σ + ψ ∗ ; s ) 7: end for 8: end for 9:
Revised: 1: RunServer() 2: initialize σ 0 , ψ3: for each round r = 1 , 2 , ..., R do 4: for each server epoch e from 1 to E G do 5: for minibatch s ∈ S G do 6: θ σ + ψ ∗ ← θ σ + ψ ∗ − η ∇ (cid:96) s ( θ σ + ψ ∗ ; s ) 7: end for 8: end for 9:

Original: L r ← (select random A clients from L ) 10: for each client l ra ∈ L r in parallel do 11: ψ r 1: H ← GetNearestNeighbors( ψ r ) 12: ψ ra ← RunClient( σ r +1 , ψ r , ψ r 1: H ) 13:
Revised: L r ← (select random A clients from L ) 10: for each client l ra ∈ L r in parallel do 11: ψ r 1: H ← GetNearestNeighbors( ψ r ) 12: ψ ra ← RunClient( σ r +1 , ψ r , ψ r 1: H ) 13:

Original: EmbedLocalModel( σ r +1 , ψ ra ) 14: end for 15: ψ r +1 ← 1 A (cid:80) Aa =1 ( ψ rl a ) 16: end for 17: RunClient( σ, ψ, ψ 1: H ) 18: θ l ← σ ∗ + ψ , θ h 1: H ← σ ∗ + ψ 1: H 19: for each local epoch e from 1 to E L do 20: for minibatch u ∈ U l a do 21: θ σ ∗ + ψ ← θ σ ∗ + ψ − η ∇ (cid:96) u
Revised: EmbedLocalModel( σ r +1 , ψ ra ) 14: end for 15: ψ r +1 ← 1 A (cid:80) Aa =1 ( ψ rl a ) 16: end for 17: RunClient( σ, ψ, ψ 1: H ) 18: θ l ← σ ∗ + ψ , θ h 1: H ← σ ∗ + ψ 1: H 19: for each local epoch e from 1 to E L do 20: for minibatch u ∈ U l a do 21: θ σ ∗ + ψ ← θ σ ∗ + ψ − η ∇ (cid:96) u

Original: ( θ σ ∗ + ψ ; θ h 1: H , u ) 22: end for 23: end for
Revised: ( θ σ ∗ + ψ ; θ h 1: H , u ) 22: end for 23: end for

Original: FedMatch Algorithms for Labels-at-Server Scenario We now describe our FedMatch algorithm for the labels-at-server scenario.
Revised: FedMatch Algorithms for Labels-at-Server Scenario We now describe our FedMatch algorithm for the labels-at-server scenario.

Original: As depicted in Figure 4, which describes an illustrative running example for labels-at-server scenario, the global model G learns σ on labeled data S G at the server and the active local clients l 1: A at the current round r learn ψ l 1: A on unlabeled data U 1: A at each local environment.
Revised: As depicted in Figure 4, which describes an illustrative running example for labels-at-server scenario, the global model G learns σ on labeled data S G at the server and the active local clients l 1: A at the current round r learn ψ l 1: A on unlabeled data U 1: A at each local environment.

Original: After the completion of local training, clients update their learned knowledge ψ l 1: A to the server.
Revised: After the completion of local training, clients update their learned knowledge ψ l 1: A to the server.

Original: The server then embeds local models based on model similarity and create a KD-Tree for rapid nearest neighbor search for the top- H most similar ψ h 1: H models for each client.
Revised: The server then embeds local models based on model similarity and create a KD-Tree for rapid nearest neighbor search for the top- H most similar ψ h 1: H models for each client.

Original: At the next round, server transmits its learned σ r +

Revised: At the next round, server transmits its learned σ r +

Original: and the aggregated ψ r +1 .
Revised: and the aggregated ψ r +1 .

Original: Server transmits top- H similar ψ h 1: H to each client for every 10 communication rounds.
Revised: Server transmits top- H similar ψ h 1: H to each client for every 10 communication rounds.

Original: Further training details of FedMatch for the labels-at-server scenario is described in Algorithm 2.
Revised: Further training details of FedMatch for the labels-at-server scenario is described in Algorithm 2.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We now experimentally validate our method, FedMatch , on three tasks, such as Batch-IID, BatchNonIID, and Streaming-NonIID, under both scenarios, Labels-at-Client and Labels-at-Server .
Revised: We now experimentally validate our method, FedMatch , on three tasks, such as Batch-IID, BatchNonIID, and Streaming-NonIID, under both scenarios, Labels-at-Client and Labels-at-Server .

Original: 6. E XPERIMENTAL S ETUP
Revised: 6. E XPERIMENTAL S ETUP

Original: Tasks 1) Batch-IID : We use CIFAR-10 for this task and split 60 , 000 instances into training ( 54 , 000 ), valid ( 3 , 000 ), and test ( 3 , 000 ) sets.
Revised: Tasks 1) Batch-IID : We use CIFAR-10 for this task and split 60 , 000 instances into training ( 54 , 000 ), valid ( 3 , 000 ), and test ( 3 , 000 ) sets.

Original: We extract 5 labeled instances per class ( C = 10 ) for each client ( K = 100 ) as labeled set S , and the rest of instances ( 49 , 000 ) are used as unlabeled data
Revised: We extract 5 labeled instances per class ( C = 10 ) for each client ( K = 100 ) as labeled set S , and the rest of instances ( 49 , 000 ) are used as unlabeled data

Original: U , so that we can evenly split S and U into S l 1:100 and U l 1:100 , such that local models l 1:100 learn on corresponding labeled and unlabeled data during training.
Revised: U , so that we can evenly split S and U into S l 1:100 and U l 1:100 , such that local models l 1:100 learn on corresponding labeled and unlabeled data during training.

Original: 2) Batch-NonIID (class-imbalanced) :
Revised: 2) Batch-NonIID (class-imbalanced) :

Original: The setting of this task is mostly the same with the Batch-IID task, except we arbitrarily control the distribution of the number of instances per class for each client to simulate class-imbalanced environments.
Revised: The setting of this task is mostly the same with the Batch-IID task, except we arbitrarily control the distribution of the number of instances per class for each client to simulate class-imbalanced environments.

Original: 3) Streaming-NonIID (class-imbalanced) :
Revised: 3) Streaming-NonIID (class-imbalanced) :

Original: In this task, data streams into each client from class-imbalanced distributions.
Revised: In this task, data streams into each client from class-imbalanced distributions.

Original: We use Fashion-MNIST dataset for this task, and split 70 ,instances into training ( 63 , 000 ), valid ( 3 , 500 ), and test ( 3 , 500 ) sets.
Revised: We use Fashion-MNIST dataset for this task, and split 70 ,instances into training ( 63 , 000 ), valid ( 3 , 500 ), and test ( 3 , 500 ) sets.

Original: From train set, we extractlabeled instances per class ( C = 5 ) for each client ( K = 10 ) for a labeled set S .
Revised: From train set, we extractlabeled instances per class ( C = 5 ) for each client ( K = 10 ) for a labeled set S .

Original: We discard labels for the rest of instances to construct an unlabeled set U ( 62 , 000 ).
Revised: We discard labels for the rest of instances to construct an unlabeled set U ( 62 , 000 ).

Original: Then, we split S and U into S l 1:100 and
Revised: Then, we split S and U into S l 1:100 and

Original: U l 1:100 based on a class-imbalanced distribution.
Revised: U l 1:100 based on a class-imbalanced distribution.

Original: For individual local unlabeled data U l k , we again split all instances into U l k t , t ∈ { 1 , 2 , ..., T } , where T is the number of total streaming steps (we set
Revised: For individual local unlabeled data U l k , we again split all instances into U l k t , t ∈ { 1 , 2 , ..., T } , where T is the number of total streaming steps (we set

Original: T = 10 ).
Revised: T = 10 ).

Original: We train each streaming step for 10 rounds.
Revised: We train each streaming step for 10 rounds.

Original: We describe above tasks under Labels-at-Client scenario.
Revised: We describe above tasks under Labels-at-Client scenario.

Original: For Labels-at-Server scenario, S is simply located at server without any partition.
Revised: For Labels-at-Server scenario, S is simply located at server without any partition.

Original: Please see Figure 7 in Appendix, which we visualize the concepts of dataset conﬁguration.
Revised: Please see Figure 7 in Appendix, which we visualize the concepts of dataset conﬁguration.

Original: [Table]
Revised: [Table]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: UDA 46.
Revised: UDA 46.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: FixMatch 46.
Revised: FixMatch 46.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: FixMatch
Revised: FixMatch

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: (Ours)
Revised: (Ours)

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Baselines and Training Details
Revised: Baselines and Training Details

Original: Our baselines are: 1) Local-SL : local supervised learning (SL) with full labels ( S + U ) without sharing locally learned knowledge.
Revised: Our baselines are: 1) Local-SL : local supervised learning (SL) with full labels ( S + U ) without sharing locally learned knowledge.

Original: 2) Local-UDA and 3) LocalFixMatch : local semi-supervised learning, including UDA and FixMatch, without sharing local knowledge.
Revised: 2) Local-UDA and 3) LocalFixMatch : local semi-supervised learning, including UDA and FixMatch, without sharing local knowledge.

Original: 4) FedAVG-SL and 5) FedProx-SL : supervised learning with full labels ( S + U ) while sharing local knowledge via FedAvg and FedProx frameworks.
Revised: 4) FedAVG-SL and 5) FedProx-SL : supervised learning with full labels ( S + U ) while sharing local knowledge via FedAvg and FedProx frameworks.

Original: FedAvg-UDA and 7) FedProxUDA : naive combinations of FedAvg/Prox with UDA.
Revised: FedAvg-UDA and 7) FedProxUDA : naive combinations of FedAvg/Prox with UDA.

Original: 8) FedAvg-FixMatch and 9)
Revised: 8) FedAvg-FixMatch and 9)

Original: FedProxFixMatch : naive combination of with FixMatch with FedAvg/Prox.
Revised: FedProxFixMatch : naive combination of with FixMatch with FedAvg/Prox.

Original: For training, we use SGD with adaptive-learning rate decay introduced in (Serra et al., 2018) with the initial learning rate 1e − 3 .
Revised: For training, we use SGD with adaptive-learning rate decay introduced in (Serra et al., 2018) with the initial learning rate 1e − 3 .

Original: We use ResNet-9 networks as the backbone architecture for all baselines and our methods.
Revised: We use ResNet-9 networks as the backbone architecture for all baselines and our methods.

Original: We ensure that all hyper-parameters are set equally for all base models and ours to perform fair evaluation and comparison.
Revised: We ensure that all hyper-parameters are set equally for all base models and ours to perform fair evaluation and comparison.

Original: Please see the Section A in the Appendix for further details.
Revised: Please see the Section A in the Appendix for further details.

Original: For all experiments, we report the mean and the standard deviation over 3 runs.
Revised: For all experiments, we report the mean and the standard deviation over 3 runs.

Original: 6. E XPERIMENTAL R ESULTS
Revised: 6. E XPERIMENTAL R ESULTS

Original: Results on Batch-IID & NonIID Tasks Table 1 shows performance comparison of our models and naive Fed-SSL algorithms on Batch-IID and NonIID tasks under the two different scenarios.
Revised: Results on Batch-IID & NonIID Tasks Table 1 shows performance comparison of our models and naive Fed-SSL algorithms on Batch-IID and NonIID tasks under the two different scenarios.

Original: We observe that our model outperforms all naive Fed-SSL baselines for all tasks and scenarios.
Revised: We observe that our model outperforms all naive Fed-SSL baselines for all tasks and scenarios.

Original: In particular, under labels-at-server scenario, which is more challenging than labels-at-client scenario, we observe that the naive combination models signiﬁcantly suffer from the forgetting issue and their performances keeps deteriorating after a certain communication round.
Revised: In particular, under labels-at-server scenario, which is more challenging than labels-at-client scenario, we observe that the naive combination models signiﬁcantly suffer from the forgetting issue and their performances keeps deteriorating after a certain communication round.

Original: This phenomenon is mainly caused by the base models failing to properly perform disjoint learning, in which case the learned knowledge from the labeled and unlabeled data causes inter-task interference.
Revised: This phenomenon is mainly caused by the base models failing to properly perform disjoint learning, in which case the learned knowledge from the labeled and unlabeled data causes inter-task interference.

Original: Contrarily, our methods show consistent and robust performance regardless where the labeled data exists, which shows that our decomposition techniques effectively handles the challenging disjoint learning scenario.
Revised: Contrarily, our methods show consistent and robust performance regardless where the labeled data exists, which shows that our decomposition techniques effectively handles the challenging disjoint learning scenario.

Original: In addition, when the class-wise distribution is imbalanced for each client (Non-IID task), we observe that the base models’ performance slightly drops by 1 - 3% p , while our methods show consistent.
Revised: In addition, when the class-wise distribution is imbalanced for each client (Non-IID task), we observe that the base models’ performance slightly drops by 1 - 3% p , while our methods show consistent.

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: This shows that our inter-client consistency effectively enhances consistency with the helper agents selected from server based on model similarity, which is good at, in particular, class-imbalanced tasks.
Revised: This shows that our inter-client consistency effectively enhances consistency with the helper agents selected from server based on model similarity, which is good at, in particular, class-imbalanced tasks.

Original: We also visualize the test accuracy curve for our models and naive Fed-SSL in Fig.
Revised: We also visualize the test accuracy curve for our models and naive Fed-SSL in Fig.

Original: Our method (Red line) trains faster and consistently outperforms the base models, and is most robustness against inter-task interference in both scenarios.
Revised: Our method (Red line) trains faster and consistently outperforms the base models, and is most robustness against inter-task interference in both scenarios.

Original: For analysis on the averaged communication costs, please see Section B.1 in the Appendix.
Revised: For analysis on the averaged communication costs, please see Section B.1 in the Appendix.

Original: Results on Streaming-NonIID Task Table 2 shows averaged local model performance on
Revised: Results on Streaming-NonIID Task Table 2 shows averaged local model performance on

Original: Streaming-NonIID tasks with 10 synchronized clients.
Revised: Streaming-NonIID tasks with 10 synchronized clients.

Original: For the labels-at-client scenario, our proposed method outperforms local-SSL and naive Fed-SSL models with large margins, 4 - 15% p , except for the SL models.
Revised: For the labels-at-client scenario, our proposed method outperforms local-SSL and naive Fed-SSL models with large margins, 4 - 15% p , except for the SL models.

Original: There is no huge difference of performance between local SSL and Fed-SSL models, and this implies that our method effectively utilizes inter-client knowledge in this streaming setting.
Revised: There is no huge difference of performance between local SSL and Fed-SSL models, and this implies that our method effectively utilizes inter-client knowledge in this streaming setting.

Original: In the labels-at-server scenario, interestingly, the performance of FedProx-SL decreases by around 5% p compared to the labels-at-client scenario, while Fed-SSL models obtain improved performance.
Revised: In the labels-at-server scenario, interestingly, the performance of FedProx-SL decreases by around 5% p compared to the labels-at-client scenario, while Fed-SSL models obtain improved performance.

Original: We conjecture that this is because, for streaming situation, the model may not sufﬁciently train on the new data, while Fed-SSL models overcomes it by utilizing only the consistent pseudo-labels.
Revised: We conjecture that this is because, for streaming situation, the model may not sufﬁciently train on the new data, while Fed-SSL models overcomes it by utilizing only the consistent pseudo-labels.

Original: Even on this task, FedMatch outperforms all baselines with signiﬁcantly smaller communication cost on average (see Section B.1 for detailed analysis of the averaged communication costs).
Revised: Even on this task, FedMatch outperforms all baselines with signiﬁcantly smaller communication cost on average (see Section B.1 for detailed analysis of the averaged communication costs).

Original: Effectiveness of Inter-Client Consistency To show the effectiveness of our inter-client consistency loss, we eliminate the loss, while learning on Batch-IID task with 100 clients ( F = 0 . 05 ).
Revised: Effectiveness of Inter-Client Consistency To show the effectiveness of our inter-client consistency loss, we eliminate the loss, while learning on Batch-IID task with 100 clients ( F = 0 . 05 ).

Original: In Figure(a), when we remove our inter-client consistency loss, we observe that the performance has slightly dropped (Pink line) from one with the loss term (Red line).
Revised: In Figure(a), when we remove our inter-client consistency loss, we observe that the performance has slightly dropped (Pink line) from one with the loss term (Red line).

Original: This gap clearly tells us that our interclient consistency loss improves model consistency across multiple models while keeping reliable knowledge.
Revised: This gap clearly tells us that our interclient consistency loss improves model consistency across multiple models while keeping reliable knowledge.

Original: Interestingly, our model without inter-client consistency loss still outperforms base models.
Revised: Interestingly, our model without inter-client consistency loss still outperforms base models.

Original: This additionally implies that our another proposed method, parameter decomposition for disjoint learning, also effectively enhances model performance.
Revised: This additionally implies that our another proposed method, parameter decomposition for disjoint learning, also effectively enhances model performance.

Original: Effectiveness of Parameter Decomposition
Revised: Effectiveness of Parameter Decomposition

Original: Our model with parameter decomposition alone, without inter-client consistency loss, outperforms base models.
Revised: Our model with parameter decomposition alone, without inter-client consistency loss, outperforms base models.

Original: For further analysis, we show the effect of each decomposed variables, σ and ψ , in Figure 6 (b).
Revised: For further analysis, we show the effect of each decomposed variables, σ and ψ , in Figure 6 (b).

Original: Removing either of σ and ψ results in substantial drop in the performance, with larger performance degeneration when dropping σ , which captures much more essential knowledge from labeled data (Green line).
Revised: Removing either of σ and ψ results in substantial drop in the performance, with larger performance degeneration when dropping σ , which captures much more essential knowledge from labeled data (Green line).

Original: Such decomposition is effective since there exists knowledge interference between supervised learning and unsupervised learning.
Revised: Such decomposition is effective since there exists knowledge interference between supervised learning and unsupervised learning.

Original: We show this with an experiment where we perform semi-supervised learning withlabeled instances per class and 1 , 000 unlabeled instances for 100 rounds.
Revised: We show this with an experiment where we perform semi-supervised learning withlabeled instances per class and 1 , 000 unlabeled instances for 100 rounds.

Original: We measure accuracy on the labeled set at each training steps.
Revised: We measure accuracy on the labeled set at each training steps.

Original: As shown in Figure 6 (c), our method effectively preserves learned knowledge from labeled set, while other base models suffer from knowledge interference.
Revised: As shown in Figure 6 (c), our method effectively preserves learned knowledge from labeled set, while other base models suffer from knowledge interference.

Original: This effective separation of supervised and unsupervised learning tasks enhances the overall performance of our methods even without inter-client consistency loss as shown in Figure 6 (a) (shown in pink).
Revised: This effective separation of supervised and unsupervised learning tasks enhances the overall performance of our methods even without inter-client consistency loss as shown in Figure 6 (a) (shown in pink).

Original: Number of Labels per Class We increase the number of labels per class for each client in a range of 1 , 5 , 10 , and 20 on Batch-IID task (CIFAR-10).
Revised: Number of Labels per Class We increase the number of labels per class for each client in a range of 1 , 5 , 10 , and 20 on Batch-IID task (CIFAR-10).

Original: Our method shows consistent performance improvement as the number of labels increases.
Revised: Our method shows consistent performance improvement as the number of labels increases.

Original: Interestingly, we observe that baseline models,
Revised: Interestingly, we observe that baseline models,

Original: FedProx-UDA/FixMatch, show performance degradation even when the labeled data increases ( 5 → 10 ).
Revised: FedProx-UDA/FixMatch, show performance degradation even when the labeled data increases ( 5 → 10 ).

Original: These results show that our method effectively utilize knowledge from labeled and unlabeled data in federated semi-supervised learning settings, while other naive combinations of
Revised: These results show that our method effectively utilize knowledge from labeled and unlabeled data in federated semi-supervised learning settings, while other naive combinations of

Original: FSSL could fail to learn properly from labeled and unlabeled data in federated learning framework.
Revised: FSSL could fail to learn properly from labeled and unlabeled data in federated learning framework.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Federated Learning A variety of approaches for averaging local weights at server have been introduced in the past few years.
Revised: Federated Learning A variety of approaches for averaging local weights at server have been introduced in the past few years.

Original: FedAvg (McMahan et al., 2017) performs weighted-averaging on local weights according to the local train size.
Revised: FedAvg (McMahan et al., 2017) performs weighted-averaging on local weights according to the local train size.

Original: FedProx (Li et al., 2018) uniformly averages the local updates while clients perform proximal regularization against the global weights, while
Revised: FedProx (Li et al., 2018) uniformly averages the local updates while clients perform proximal regularization against the global weights, while

Original: FedMA (Wang et al., 2020) matches the hidden elements with similar feature extraction signatures in layer-wise manner when averaging local weights.
Revised: FedMA (Wang et al., 2020) matches the hidden elements with similar feature extraction signatures in layer-wise manner when averaging local weights.

Original: PFNM (Yurochkin et al., 2019) introduces aggregation policy which leverages Bayesian non-parametric methods.
Revised: PFNM (Yurochkin et al., 2019) introduces aggregation policy which leverages Bayesian non-parametric methods.

Original: Beyond focusing on averaging local knowledge, there are various efforts to extend FL to the other areas, such as continual learning under federated learning frameworks (Yoon et al., 2020a) inspired by parameter decomposition techniques proposed by (Yoon et al., 2020b).
Revised: Beyond focusing on averaging local knowledge, there are various efforts to extend FL to the other areas, such as continual learning under federated learning frameworks (Yoon et al., 2020a) inspired by parameter decomposition techniques proposed by (Yoon et al., 2020b).

Original: Recently, interests of tackling scarcity of labeled data in FL are emerging and discussed in (Jin et al., 2020; Guha et al., 2019; Albaseer et al., 2020).
Revised: Recently, interests of tackling scarcity of labeled data in FL are emerging and discussed in (Jin et al., 2020; Guha et al., 2019; Albaseer et al., 2020).

Original: Semi-Supervised Learning While there exist numerous work on SSL, we mainly discuss consistency regularization approaches.
Revised: Semi-Supervised Learning While there exist numerous work on SSL, we mainly discuss consistency regularization approaches.

Original: Consistency regularization (Sajjadi et al., 2016) assumes that the class semantics will not be affected by transformations of the input instances, and enforces the model output to be the same across different input perturbations.
Revised: Consistency regularization (Sajjadi et al., 2016) assumes that the class semantics will not be affected by transformations of the input instances, and enforces the model output to be the same across different input perturbations.

Original: Some extensions to this technique perturb inputs adversarially (Miyato et al., 2018), through dropout (Srivastava et al., 2014), or through data augmentation (French et al., 2018).
Revised: Some extensions to this technique perturb inputs adversarially (Miyato et al., 2018), through dropout (Srivastava et al., 2014), or through data augmentation (French et al., 2018).

Original: UDA (Xie et al., 2019) and ReMixMatch (Berthelot et al., 2019a) use two sets of augmentations, weak and strong, and enforce consistency between the weakly and strongly augmented examples.
Revised: UDA (Xie et al., 2019) and ReMixMatch (Berthelot et al., 2019a) use two sets of augmentations, weak and strong, and enforce consistency between the weakly and strongly augmented examples.

Original: Recently, in addition to enforcing consistency between weak-strong augmented pairs, FixMatch (Sohn et al., 2020) performs pseudo-label reﬁnement on model predictions via thresholding.
Revised: Recently, in addition to enforcing consistency between weak-strong augmented pairs, FixMatch (Sohn et al., 2020) performs pseudo-label reﬁnement on model predictions via thresholding.

Original: Entropy minimization (Grandvalet & Bengio, 2004) which enforces the classiﬁer to predict low-entropy on unlabeled data, is another popular technique for SSL.
Revised: Entropy minimization (Grandvalet & Bengio, 2004) which enforces the classiﬁer to predict low-entropy on unlabeled data, is another popular technique for SSL.

Original: Pseudo-Label (Lee, 2013) constructs one-hot labels from highly conﬁdent predictions on unlabeled data and uses these as training targets inn a standard cross-entropy loss.
Revised: Pseudo-Label (Lee, 2013) constructs one-hot labels from highly conﬁdent predictions on unlabeled data and uses these as training targets inn a standard cross-entropy loss.

Original: MixMatch (Berthelot et al., 2019c) performs sharpening on target distribution on unlabeled data, to further reﬁne the generated pseudo-label.
Revised: MixMatch (Berthelot et al., 2019c) performs sharpening on target distribution on unlabeled data, to further reﬁne the generated pseudo-label.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this work, we introduced two practical scenarios of Federated Semi-Supervised Learning (FSSL) where each client learns with only partly labeled data ( Labels-at-Client scenario), or supervised labels are only available at the server, while clients work with completely unlabeled data ( Labels-at-Server scenario).
Revised: In this work, we introduced two practical scenarios of Federated Semi-Supervised Learning (FSSL) where each client learns with only partly labeled data ( Labels-at-Client scenario), or supervised labels are only available at the server, while clients work with completely unlabeled data ( Labels-at-Server scenario).

Original: To tackle the problem, we propose a novel method, Federated Matching (FedMatch), which introduces the inter-client consistency loss that aims to maximize the agreement between the models trained at different clients, and the parameter decomposition for disjoint learning which decomposes the parameters into one for labeled data and the other for unlabeled data for preservation of reliable knowledge, reduction of communication costs, and disjoint learning.
Revised: To tackle the problem, we propose a novel method, Federated Matching (FedMatch), which introduces the inter-client consistency loss that aims to maximize the agreement between the models trained at different clients, and the parameter decomposition for disjoint learning which decomposes the parameters into one for labeled data and the other for unlabeled data for preservation of reliable knowledge, reduction of communication costs, and disjoint learning.

Original: Through extensive experimental validation, we show that FedMatch signiﬁcantly outperforms both local semi-supervised learning methods and naive combinations of federated learning algorithms with semi-supervised learning on diverse and realistic scenarios.
Revised: Through extensive experimental validation, we show that FedMatch signiﬁcantly outperforms both local semi-supervised learning methods and naive combinations of federated learning algorithms with semi-supervised learning on diverse and realistic scenarios.

Original: As future work, we plan to further improve our model to tackle the scenario where pretrained models deployed at each client adapts to a completely unlabeled data stream (e.g. on-device learning of smart speakers).
Revised: As future work, we plan to further improve our model to tackle the scenario where pretrained models deployed at each client adapts to a completely unlabeled data stream (e.g. on-device learning of smart speakers).

Original: Acknowledgements This work was supported by Samsung Research Funding Center of Samsung
Revised: Acknowledgements This work was supported by Samsung Research Funding Center of Samsung

Original: Electronics (No.
Revised: Electronics (No.

Original: SRFC-IT1502-51), Samsung Advanced Institute of Technology, Samsung Electronics Co., Ltd., Next-Generation Information Computing Development Program through the National
Revised: SRFC-IT1502-51), Samsung Advanced Institute of Technology, Samsung Electronics Co., Ltd., Next-Generation Information Computing Development Program through the National

Original: Research Foundation of Korea(NRF) funded by the Ministry of Science, ICT  Future Plannig (No. 2016M3C4A7952634) and the National Research Foundation of Korea(NRF) grant funded by the
Revised: Research Foundation of Korea(NRF) funded by the Ministry of Science, ICT & Future Plannig (No. 2016M3C4A7952634), the National Research Foundation of Korea(NRF) grant funded by the

Original: Korea government(MSIT) (2018R1A5A1059921)
Revised: Korea government(MSIT) (2018R1A5A1059921), and Center for Applied Research in Artiﬁcial Intelligence (CARAI) grant funded by DAPA and ADD (UDI190031RD). Also, this work was supported by Institute of Information communications Technology Planning Evaluation (IITP) grant funded by the Korea government(MSIT) (No.2019-0-00075, Artiﬁcial Intelligence Graduate School

Revised: Program(KAIST))

Original: A E XPERIMENTAL D ETAILS
Revised: A E XPERIMENTAL D ETAILS

Original: We describe our experimental setups in detail, such as our baseline models, network architecture that is used for all base models and our method, and the detailed training setups.
Revised: We describe our experimental setups in detail, such as our baseline models, network architecture that is used for all base models and our method, and the detailed training setups.

Original: A. B ASELINE M ODELS
Revised: A. B ASELINE M ODELS

Original: We consider UDA (Xie et al., 2019) and FixMatch (Sohn et al., 2020) as our baselines, since they are state-of-the-art SSL models and are based on the consistency-based mechanisms that are conceptually similar to our inter-client consistency loss.
Revised: We consider UDA (Xie et al., 2019) and FixMatch (Sohn et al., 2020) as our baselines, since they are state-of-the-art SSL models and are based on the consistency-based mechanisms that are conceptually similar to our inter-client consistency loss.

Original: We reimplement UDA with the Training
Revised: We reimplement UDA with the Training

Original: Signal Annealing (TSA) and exponential scheduling for its best performance as reported in their paper (we use RandAugment (Cubuk et al., 2019) for consistency regularization with random magnitude).
Revised: Signal Annealing (TSA) and exponential scheduling for its best performance as reported in their paper (we use RandAugment (Cubuk et al., 2019) for consistency regularization with random magnitude).

Original: We also reimplement FixMatch algorithms with strong augmentation as RandAugment (Cubuk et al., 2019).
Revised: We also reimplement FixMatch algorithms with strong augmentation as RandAugment (Cubuk et al., 2019).

Original: For weak augmentation (ﬁlp-and-shift), however, as the performance has signiﬁcantly dropped when we apply the weak augmentation, we use original images rather than weakly augmenting the images.
Revised: For weak augmentation (ﬁlp-and-shift), however, as the performance has signiﬁcantly dropped when we apply the weak augmentation, we use original images rather than weakly augmenting the images.

Original: We ﬁx conﬁdence threshold τ = 0 . 85 for all FixMatch and our model experiments.
Revised: We ﬁx conﬁdence threshold τ = 0 . 85 for all FixMatch and our model experiments.

Original: For federated learning frameworks, we use FedAvg (McMahan et al., 2017) and FedProx (Li et al., 2018) algorithms since they are the standard baselines for federated learning and can be easily combined with the SSL baselines.
Revised: For federated learning frameworks, we use FedAvg (McMahan et al., 2017) and FedProx (Li et al., 2018) algorithms since they are the standard baselines for federated learning and can be easily combined with the SSL baselines.

Original: Detailed hyper-parameter settings are described in Table 4.
Revised: Detailed hyper-parameter settings are described in Table 4.

Original: A. N ETWORK A RCHITECTURE
Revised: A. N ETWORK A RCHITECTURE

Original: [Table]
Revised: [Table]

Original: We build ResNet-9 networks as our base architecture for all base model and our method.
Revised: We build ResNet-9 networks as our base architecture for all base model and our method.

Original: In the architecture, the ﬁrst two convolutional neural layers have 64 and 128 ﬁlters and the same 3 × 3 kernel sizes followed by 2 × 2 max-pooling layer.
Revised: In the architecture, the ﬁrst two convolutional neural layers have 64 and 128 ﬁlters and the same 3 × 3 kernel sizes followed by 2 × 2 max-pooling layer.

Original: Then we have a skip connection between the subsequent two convolution layers withﬁlters.
Revised: Then we have a skip connection between the subsequent two convolution layers withﬁlters.

Original: We then double the ﬁlter size from 128 to 256 with the next conv layer and down-sample via the following 2 × 2 max-pooling layer.
Revised: We then double the ﬁlter size from 128 to 256 with the next conv layer and down-sample via the following 2 × 2 max-pooling layer.

Original: We repeat the previous step, such that we have 512 ﬁlter size and 4 × 4 kernel size.
Revised: We repeat the previous step, such that we have 512 ﬁlter size and 4 × 4 kernel size.

Original: Then, we perform another skip connection through the two subsequent conv layers with 512 ﬁlters.
Revised: Then, we perform another skip connection through the two subsequent conv layers with 512 ﬁlters.

Original: As a ﬁnal step, we down-sample the kernel size from 4 × 4 to 1 × 1 , then perform softmax classiﬁer with the last fully connected layer.
Revised: As a ﬁnal step, we down-sample the kernel size from 4 × 4 to 1 × 1 , then perform softmax classiﬁer with the last fully connected layer.

Original: All layers are equally initialized based on the varaiance scalining method.
Revised: All layers are equally initialized based on the varaiance scalining method.

Original: The model architecture is described in Table 3.
Revised: The model architecture is described in Table 3.

Original: A. T RAINING D ETAILS
Revised: A. T RAINING D ETAILS

Original: We use Stochastic Gradient Descent (SGD) to optimize our model with initial learning rate 1e-3.
Revised: We use Stochastic Gradient Descent (SGD) to optimize our model with initial learning rate 1e-3.

Original: We also adopt adaptive learning rate decay which is introduced by (Serra et al., 2018).
Revised: We also adopt adaptive learning rate decay which is introduced by (Serra et al., 2018).

Original: The learning rate strategy gradually reduces the learning rate by a factor of 3 for every 5 epochs that validation loss does not consecutively decreases.
Revised: The learning rate strategy gradually reduces the learning rate by a factor of 3 for every 5 epochs that validation loss does not consecutively decreases.

Original: We use L2 weight decay regularization on the base architecture with L2 factor to be 1e-4.
Revised: We use L2 weight decay regularization on the base architecture with L2 factor to be 1e-4.

Original: All hyper-parameters and other training setups are equally set for fair comparison as shown in Table 4.
Revised: All hyper-parameters and other training setups are equally set for fair comparison as shown in Table 4.

Original: In the table, we denote LPC as number of labels per class for each
Revised: In the table, we denote LPC as number of labels per class for each

Original: [Table]
Revised: [Table]

Original: client (or at server).
Revised: client (or at server).

Original: B S and B U denote batch-size of labeled set S and unlabeled set U .
Revised: B S and B U denote batch-size of labeled set S and unlabeled set U .

Original: µ is a hyper-parameter for FedProx framework.
Revised: µ is a hyper-parameter for FedProx framework.

Original: We additionally provide visual illustration of our dataset conﬁguration.
Revised: We additionally provide visual illustration of our dataset conﬁguration.

Original: Please see Figure 7.
Revised: Please see Figure 7.

Original: B A DDITIONAL A NALYSIS AND E XPERIMENTAL R ESULTS
Revised: B A DDITIONAL A NALYSIS AND E XPERIMENTAL R ESULTS

Original: In this section, we additionally provide more analysis and experimental results, such as analysis on communication costs and number of labels per class, experiments on real-world dataset, different backbone architecture, fraction of clients per communication round.
Revised: In this section, we additionally provide more analysis and experimental results, such as analysis on communication costs and number of labels per class, experiments on real-world dataset, different backbone architecture, fraction of clients per communication round.

Original: B. T HE E FFICIENT C OMMUNICATION OF F ED M ATCH
Revised: B. T HE E FFICIENT C OMMUNICATION OF F ED M ATCH

Original: Since the actual bit-level compression techniques are rather implementation issues, which are beyond our research scope, we only consider the reduction of the amount of information that needs to be transmitted between the server and the client.
Revised: Since the actual bit-level compression techniques are rather implementation issues, which are beyond our research scope, we only consider the reduction of the amount of information that needs to be transmitted between the server and the client.

Original: To minimize the communication costs, we not only learn ψ to be sparse, but also subtract the parameters between server and client, such that ∆ ψ = ψ lr − ψ Gr and ∆ σ = σ lr − σ Gr , then send only the difference, ∆ ψ and ∆ σ , as sparse matrices from both directions of server-to-client (S2C) and client-to-server (C2S).
Revised: To minimize the communication costs, we not only learn ψ to be sparse, but also subtract the parameters between server and client, such that ∆ ψ = ψ lr − ψ Gr and ∆ σ = σ lr − σ Gr , then send only the difference, ∆ ψ and ∆ σ , as sparse matrices from both directions of server-to-client (S2C) and client-to-server (C2S).

Original: Here, S2C and C2S costs are the sums of ∆ σ and ∆ ψ .
Revised: Here, S2C and C2S costs are the sums of ∆ σ and ∆ ψ .

Original: When transmitting the difference for each parameter to either way, we discard almost unchanged values in an element-wise manner, so that only meaningful neural values can be updated either server- or client-side.
Revised: When transmitting the difference for each parameter to either way, we discard almost unchanged values in an element-wise manner, so that only meaningful neural values can be updated either server- or client-side.

Original: We observe that the range of the threshold values is from 1e-5 to 5e-5, such that the model performance is well-preserved and not signiﬁcantly harmed, while maximizing the reduction of communication costs.
Revised: We observe that the range of the threshold values is from 1e-5 to 5e-5, such that the model performance is well-preserved and not signiﬁcantly harmed, while maximizing the reduction of communication costs.

Original: As shown in Figure 8, we observe that both the S2C and C2S costs are gradually decreased during the learning phases on both batch and streaming datasets under labels-at-client (Figure 8 (a) and (b)) and labels-at-server scenarios (Figure 8 (c) and (d)).
Revised: As shown in Figure 8, we observe that both the S2C and C2S costs are gradually decreased during the learning phases on both batch and streaming datasets under labels-at-client (Figure 8 (a) and (b)) and labels-at-server scenarios (Figure 8 (c) and (d)).

Original: This is because each parameter separately learns different tasks (i.e. supervised and unsupervised learning) effectively, which results in rapid convergence to optimal points, respectively.
Revised: This is because each parameter separately learns different tasks (i.e. supervised and unsupervised learning) effectively, which results in rapid convergence to optimal points, respectively.

Original: Further, for the labels-at-server scenario, since labeled data is not available at client, client even does not need to transfer ∆ σ to the server (see Figure(c) right and (d) right), which is extremely efﬁcient than the labels-at-client scenario where both ∆ σ and ∆ ψ must be transferred to the server.
Revised: Further, for the labels-at-server scenario, since labeled data is not available at client, client even does not need to transfer ∆ σ to the server (see Figure(c) right and (d) right), which is extremely efﬁcient than the labels-at-client scenario where both ∆ σ and ∆ ψ must be transferred to the server.

Original: For both scenarios, indeed, S2C contains the cost
Revised: For both scenarios, indeed, S2C contains the cost

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: of helper agents,
Revised: of helper agents,

Original: such that ∆ ψ 1: H =
Revised: such that ∆ ψ 1: H =

Original: (cid:80) Hj =1 ψ jr − ψ lr .
Revised: (cid:80) Hj =1 ψ jr − ψ lr .

Original: However, as shown in Figure 8, transmitting multiple helper agents ( H =2 in our experiments) does not signiﬁcantly affect the total S2C costs thanks to our novel decomposition techniques as well as efﬁcient subtracting method, such that model reconstruction can be possible without meaningful information loss at either server- or client-side.
Revised: However, as shown in Figure 8, transmitting multiple helper agents ( H =2 in our experiments) does not signiﬁcantly affect the total S2C costs thanks to our novel decomposition techniques as well as efﬁcient subtracting method, such that model reconstruction can be possible without meaningful information loss at either server- or client-side.

Original: B. F URTHER A NALYSIS OF THE N UMBER OF L ABELS PER C LASS
Revised: B. F URTHER A NALYSIS OF THE N UMBER OF L ABELS PER C LASS

Original: [Table]
Revised: [Table]

Original: We explain our analysis of the number of labels per class under
Revised: We explain our analysis of the number of labels per class under

Original: Section 6.2, and here we further provide additional experimental results.
Revised: Section 6.2, and here we further provide additional experimental results.

Original: We conduct experiments with our method without the decomposition technique.
Revised: We conduct experiments with our method without the decomposition technique.

Original: As shown in Table 5, our method without decomposition technique (indicated as FedMatch (w/o)) shows not much performance improvement when the number of labels per class increases from 5 to 10 (around 3 .
Revised: As shown in Table 5, our method without decomposition technique (indicated as FedMatch (w/o)) shows not much performance improvement when the number of labels per class increases from 5 to 10 (around 3 .

Original: x % p ) than from 1 to 5 (around 9 .
Revised: x % p ) than from 1 to 5 (around 9 .

Original: x % p ) and from 10 to 20 ( 10 .
Revised: x % p ) and from 10 to 20 ( 10 .

Original: x % p ), which are the similar tendency with the baseline models in Table 5.
Revised: x % p ), which are the similar tendency with the baseline models in Table 5.

Original: However, with the decomposition technique, our method shows consistent performance improvement, which implies that our proposed technique has the effectiveness to handle inter-task interference and preserve reliable knowledge in the novel federated semi-supervised learning scenarios.
Revised: However, with the decomposition technique, our method shows consistent performance improvement, which implies that our proposed technique has the effectiveness to handle inter-task interference and preserve reliable knowledge in the novel federated semi-supervised learning scenarios.

Original: [Table]
Revised: [Table]

Original: B. E XPERIMENTS ON R EAL - WORLD D
Revised: B. E XPERIMENTS ON R EAL - WORLD D

Original: [Table]
Revised: [Table]

Original: To show our method consistently work with real-world dataset, we further conduct experiment on
Revised: To show our method consistently work with real-world dataset, we further conduct experiment on

Original: COVID-

Revised: COVID-

Original: Radiography Dataset (Chowdhury et al., 2020) which is a real-world dataset that consists of X-ray images from COVID and non-COVID patients.
Revised: Radiography Dataset (Chowdhury et al., 2020) which is a real-world dataset that consists of X-ray images from COVID and non-COVID patients.

Original: The COVID-19 dataset contains 219 X-ray images from the patients diagnosed of COVID-19, 1341 images from normal (healthy) patients, and 1341 images from patients diagnosed of viral pneumonia.
Revised: The COVID-19 dataset contains 219 X-ray images from the patients diagnosed of COVID-19, 1341 images from normal (healthy) patients, and 1341 images from patients diagnosed of viral pneumonia.

Original: We use 10 clients with a fraction of 1.
Revised: We use 10 clients with a fraction of 1.

Original: (communication rate).
Revised: (communication rate).

Original: We use 5 labeled examples per class for each client, leaving the rest of the image as unlabeled.
Revised: We use 5 labeled examples per class for each client, leaving the rest of the image as unlabeled.

Original: We ﬁnd this setting to be realistic as the datasets are, since we may not not have skilled radiologists that can fully label the X-ray images taken at the local hospitals.
Revised: We ﬁnd this setting to be realistic as the datasets are, since we may not not have skilled radiologists that can fully label the X-ray images taken at the local hospitals.

Original: We compared our method against baselines which naively combine semi-supervised learning and federated learning models during training 100 rounds.
Revised: We compared our method against baselines which naively combine semi-supervised learning and federated learning models during training 100 rounds.

Original: As shown in the left table of Figure 9, our method consistently outperforms all base models with large margins (around 4% p - 10% p ) in both scenarios.
Revised: As shown in the left table of Figure 9, our method consistently outperforms all base models with large margins (around 4% p - 10% p ) in both scenarios.

Original: The test accuracy curves in Figure 9, we can see that our method trains faster than the base models and shows more stability during training.
Revised: The test accuracy curves in Figure 9, we can see that our method trains faster than the base models and shows more stability during training.

Original: We believe that these additional experimental results further strengthen our paper.
Revised: We believe that these additional experimental results further strengthen our paper.

Original: B. B ACKBONE A RCHITECTURE
Revised: B. B ACKBONE A RCHITECTURE

Original: Most existing works on federated learning considers smaller networks since the focus is on-device learning of low-resource devices, and thus we utilize a smaller backbone networks than ResNet-9.
Revised: Most existing works on federated learning considers smaller networks since the focus is on-device learning of low-resource devices, and thus we utilize a smaller backbone networks than ResNet-9.

Original: To verify that our method also successfully works on the smaller & different architecture, we adopt
Revised: To verify that our method also successfully works on the smaller & different architecture, we adopt

Original: AlexNet-Like (Serra et al., 2018), of which the ﬁrst three layers are convolutional neural layers with 64 , 128 , and 256 ﬁlters with the 4 , 3 , and 2 kernel sizes followed by the two fully-connected layers of 2048 units, while 2 × 2 max-pooling layers are followed after each convolutional layer.
Revised: AlexNet-Like (Serra et al., 2018), of which the ﬁrst three layers are convolutional neural layers with 64 , 128 , and 256 ﬁlters with the 4 , 3 , and 2 kernel sizes followed by the two fully-connected layers of 2048 units, while 2 × 2 max-pooling layers are followed after each convolutional layer.

Original: In Table 6, for both Streaming-NonIID and Batch-IID tasks, our methods still outperforms all naive Fed-SSL models with the similar tendency with that of the results based on ResNet-9.
Revised: In Table 6, for both Streaming-NonIID and Batch-IID tasks, our methods still outperforms all naive Fed-SSL models with the similar tendency with that of the results based on ResNet-9.

Original: This shows that our methods can be applied to the smaller and different base networks, and still effectively utilize inter-client and reliable knowledge across multiple clients than naive algorithms.
Revised: This shows that our methods can be applied to the smaller and different base networks, and still effectively utilize inter-client and reliable knowledge across multiple clients than naive algorithms.

Original: B. F RACTION OF A VAILABLE C LIENTS PER C OMMUNICATION R OUND
Revised: B. F RACTION OF A VAILABLE C LIENTS PER C OMMUNICATION R OUND

Original: To see the effect of participation rate of clients, we increase the fraction of available clients per communication round in a range of 0 .
Revised: To see the effect of participation rate of clients, we increase the fraction of available clients per communication round in a range of 0 .

Original: 10 , and 0 .
Revised: 10 , and 0 .

Original: This means, for every round, server can connect to the arbitrary 5 , 10 , and 20 available clients out of 100 clients, and perform distributed learning on each individual local data through each client and updates global knowledge by aggregating the locally-learned knowledge.
Revised: This means, for every round, server can connect to the arbitrary 5 , 10 , and 20 available clients out of 100 clients, and perform distributed learning on each individual local data through each client and updates global knowledge by aggregating the locally-learned knowledge.

Original: The experimental results are shown in Table 6 Batch-IID.
Revised: The experimental results are shown in Table 6 Batch-IID.

Original: We observe that the performances of all models are slightly improved when the faction increases.
Revised: We observe that the performances of all models are slightly improved when the faction increases.

Original: This is natural that the more knowledge the client updates, the more the global performance is improved.
Revised: This is natural that the more knowledge the client updates, the more the global performance is improved.

Original: We are not able to ﬁnd any extraordinary phenomenon on the fraction of the number of clients per round.
Revised: We are not able to ﬁnd any extraordinary phenomenon on the fraction of the number of clients per round.