F EDERATED L EARNING FROM O NLY U NLABELED D ATA WITH C LASS -C ONDITIONAL -S HARING C LIENTS
A BSTRACT
Supervised federated learning (FL) enables multiple clients to share the trained model without sharing their labeled data.
However, potential clients might even be reluctant to label their own data, which could limit the applicability of FL in practice.
In this paper, we show the possibility of unsupervised FL whose model is still a classifier for predicting class labels, if the class-prior probabilities are shifted while the class-conditional distributions are shared among the unlabeled data owned by the clients.
We propose federation of unsupervised learning (FedUL), where the unlabeled data are transformed into surrogate labeled data for each of the clients, a modified model is trained by supervised FL, and the wanted model is recovered from the modified model.
FedUL is a very general solution to unsupervised FL: it is compatible with many supervised FL methods, and the recovery of the wanted model can be theoretically guaranteed as if the data have been labeled.
Experiments on benchmark and real-world datasets demonstrate the effectiveness of FedUL.
Code is available at https://github.com/lunanbit/FedUL .
I NTRODUCTION
Federated learning (FL) has received significant attention from both academic and industrial perspectives in that it can bring together separate data sources and allow multiple clients to train a central model in a collaborative but private manner (McMahan et al., 2017; Kairouz et al., 2019;
Yang et al., 2019).
So far, the majority of FL researches focused on the supervised setting, requiring collected data at each client to be fully labeled .
In practice, this may hinder the applicability of FL since manually labeling large-scale training data can be extremely costly, and sometimes may not even be possible for privacy concerns in for example medical diagnosis (Ng et al., 2021).
To promote the applicability of FL, we are interested in a challenging unsupervised FL setting where only unlabeled (U) data are available at the clients under the condition described blow.
Our goal is still to train a classification model that can predict accurate class labels.
It is often observed that the clients collect their own data with different temporal and/or spatial patterns, e.g., a hospital (data center) may store patient data every month or a particular user (mobile device) may take photos at different places.
Therefore, we consider a realistic setting that the U data at a client come in the form of separate data sets , and the data distribution of each U set and the number of U sets available at each client may vary .
Without any labels, it is unclear whether FL can be performed effectively.
In this paper, we show the possibility: the aforementioned unsupervised FL problem can be solved if the class-prior probabilities are shifted while the class-conditional distributions are shared between the available U sets at the clients, and these class priors are known to the clients.
Such a learning scenario is conceivable in many real-world problems.
For example, the hospital may not release the diagnostic labels of patients due to privacy concerns, but the morbidity rates of diseases (corresponding to the class priors) may change over time and can be accessible from public medical reports (Croft et al., 2018); moreover, in many cases it is possible to estimate the class priors much more cheaply than to collect ground-truth labels (Quadrianto et al., 2009a; Sugiyama et al., 2022). 𝑸 𝟏 𝒇
Surrogate label 𝒇 𝑀 …
Server …
Client 𝐶 Client 2 Client
True label 𝒇
True label 𝒇 𝑪
True label … 𝑸 𝟐 𝒇
Surrogate label𝑀 … 𝑸 𝑪 𝒇 𝑪
Surrogate label𝑀 …
Client 𝐶 Client 2 Client
Server update Local update
[Figure]
Server 𝒇
Based on this finding, we propose the federation of unsupervised learning (FedUL) (see Figure 1).
In FedUL, the original clients with U data are transformed to surrogate clients with (surrogate) labeled data and then supervised FL can be applied, which we call the surrogate task.
Here, the difficulty is how to infer the wanted model for the original classification task from the model learned by the surrogate task.
We solve this problem by bridging the original and surrogate class-posterior probabilities with certain injective transition functions.
This can be implemented by adding a specifically designed transition layer to the output of each client model, so that the learned model is guaranteed to be a good approximation of the original class-posterior probability.
FedUL has many key advantages.
On the one hand, FedUL is a very general and flexible solution: it works as a wrapper that transforms the original clients to the surrogate ones and therefore is compatible with many supervised FL methods, e.g., FedAvg.
On the other hand, FedUL is computationally efficient and easy-to-implement: since the added transformation layer is fixed and determined by the class priors, FedUL adds no more burden on hyper-parameter tuning and/or optimization.
Our contributions are summarized as follows:
• Methodologically, we propose a novel FedUL method that solves a certain unsupervised FL problem, expanding the applicability of FL. • Theoretically, we prove the optimal global model learned by FedUL from only U data converges to the optimal global model learned by supervised FL from labeled data under mild conditions.
• Empirically, we demonstrate the effectiveness of FedUL on benchmark and real-world datasets.
Related Work: FL has been extensively studied in the supervised setting.
One of the most popular supervised FL paradigms is FedAvg (McMahan et al., 2017) which aggregates the local updates at the server and transmits the averaged model back to local clients.
In contrast, FL in the unsupervised setting is less explored.
Recent studies have shown the possibility of federated clustering with U data (Ghosh et al., 2020; Dennis et al., 2021).
However, these clustering based methods rely on geometric or information-theoretic assumptions to build their learning objectives (Chapelle et al., 2006), and are suboptimal for classification goals.
Our work is intrinsically different from them in the sense that we show the possibility of federated classification with U data based on empirical risk minimization (ERM) (Vapnik, 1998), and therefore the optimal model recovery can be guaranteed.
Learning from multiple U sets has also been studied in classical centralized learning.
Mainstream methods include learning with label proportions (LLP)
(Quadrianto et al., 2009b) and U m classification (Lu et al., 2021).
The former learns a multiclass classifier from multiple U sets based on empirical proportion risk minimization (EPRM) (Yu et al., 2014), while the latter learns a binary classifier from multiple U sets based on ERM.
EPRM is inferior to ERM since its learning is not consistent.
Yet, it is still unexplored how to learn a multiclass classification model from multiple
U sets based on ERM.
To the best of our knowledge, this work is the first attempt to tackle this problem in the FL setup, which is built upon the binary U m classification in centralized learning.
A full discussion of related work can be found at Appendix B.
S TANDARD F EDERATED L EARNING
We begin by reviewing the standard FL.
Let us consider a K -class classification problem with the feature space X ⊂ R d and the label space Y =
[ K ] , where d is the input dimension and [ K ] := { 1 , . .
, K } .
Let x ∈ X
and
y ∈ Y be the input and output random variables following an underlying joint distribution with density p ( x , y ) , which can be identified via the class priors { π k = p ( y = k ) } Kk =1 and the class-conditional densities { p ( x | y = k ) } Kk =1 .
Let f : X → R K be a classification model that assigns a score to each of the K classes for a given input x
and then outputs the predicted label by y pred =
argmax k ∈
[ K ] ( f ( x ))
k , where ( f ( x ))
k is the k -th element of f ( x ) .
In the standard FL setup with C clients, for c ∈ [ C ] , the c -th client has access to a labeled training set
D c = { ( x ci , y ci ) } n c i =1 ∼ p c ( x , y ) of sample size n c and learns its local model f c by minimizing the risk
[Equation]
Here, E denotes the expectation, ℓ : R K × Y → R + is a proper loss function, e.g., the softmax cross-entropy loss
[Equation]
log (cid:16)(cid:80)
Ki =1 exp(( f c ( x )) i ) (cid:17) − ( f c ( x ))
y , where 1 ( · ) is the indicator function.
In practice, ERM is commonly used to compute an approximation of R c ( f c ) based on the client’s local data D c by (cid:98)
[Equation]
The goal of standard FL (McMahan et al., 2017) is that the C clients collaboratively train a global classification model f that generalizes well with respect to p ( x , y ) , without sharing their local data
D c .
The problem can be formalized as minimizing the aggregated risk:
[Equation]
Typically, we employ a server to coordinate the iterative distributed training as follows:
• in each global round of training, the server broadcasts its current model f to all the clients { f c } Cc =1 ; • each client c copies the current server model f c = f , performs L local step updates
[Equation]
where α l is the local step-size, and sends f c
−
f
back to
the server
; • the server aggregates the updates { f c − f } Cc =1 to form a new server model using
FedAvg:
[Equation]
where α g is the global
step
-size.
F EDERATED L EARNING : N O L ABEL N O C RY
Next, we formulate the problem setting of FL with only U training sets, propose our method called federation of unsupervised learning (FedUL), and provide the theoretical analysis of FedUL.
All proofs are given in Appendix C.
3. P ROBLEM F ORMULATION
In this paper, we consider a challenging setting: instead of fully labeled training data, each client c ∈ [ C ] observes M c ( M c ≥ K, ∀ c ) 1 sets of unlabeled samples, U c
= {U mc } M c m =1 , where U mc =
{ x c,mi } n c,m i
=1 denotes the collection of the m -th
U set for client c with sample size n c,m .
Each U set can be seen as a set of data points drawn from a mixture of the original class-conditional densities:
[Equation]
where π m,kc = p mc ( y = k ) denotes the k -th class prior of the m -th
U set at client c .
These class priors form a full column rank matrix Π c ∈ R M c × K with the constraint that each row sums up to 1.
Note that we do not require any labels to be known, only assume the class priors within each of the involved datasets are available, i.e., the training class priors π m,kc and the test class
prior π k = p ( y = k ) .
Our goal is the same as standard FL: the C clients are interested in collaboratively training a global classification model f that generalizes well with respect to p ( x , y ) , but using only U training sets (5) on each local client.
3. P ROPOSED M ETHOD
As discussed, FedAvg is a representative approach for aggregating clients in the FL framework.
It also serves as the basis for many advanced federated aggregation techniques, e.g., Fedprox (Li et al., 2020), FedNova (Wang et al., 2020), and SCAFFOLD (Karimireddy et al., 2020).
However, these methods cannot handle clients without labels.
To tackle this problem, we consider the construction of surrogate clients by transforming the local U datasets into labeled datasets and modifying the local model such that it is compatible with the transformed data for training.
Local Data Transformation
First, we transform the data for the clients.
For each client c ∈ [ C ] , let y be the index of given U sets, and p c ( x , y ) be an underlying joint distribution for the random variables x ∈ X and y ∈
[ M ] , where M = max c ∈ [ C ] M c .
By treating y as a surrogate label , we may transform the original U training sets U c
= {U mc } M c m =1 to a labeled training set for client c ,
[Equation]
where n c = (cid:80) M c m =1 n c,m .
Obviously, the surrogate class-conditional density p c ( x | y
= m ) corresponds to p mc ( x ) in (5), for m ∈
[ M c ] .
If ∃ M c < m ≤ M , we pad p c ( x | y
= m ) = 0 .
The surrogate class prior π mc = p c ( y = m ) can be estimated by n c,m /n c .
Based on the data transformation, we then formulate a surrogate supervised FL task : the C clients aim to collaboratively train a global classification model g : X → R M that predicts the surrogate label y for the given input x , without sharing their local data U c .
This task can be easily solved by standard supervised FL methods.
More specifically, we minimize
[Equation]
(7) by employing a server g that iteratively broadcasts its model to all local clients { g c } Cc =1 and aggregates local updates from all clients using e.g., FedAvg.
The local and global update procedures exactly follow (3) and (4) in standard FL.
Now a natural question arises: can we infer our desired model f from the surrogate model g ?
To answer it, we study the relationship between the original and surrogate class-posterior probabilities.
Theorem 1.
For each client c ∈ [ C ] , let η c : X → ∆ M − 1 and η : X → ∆ K − 1 be the surrogate and original class-posterior probability functions, where ( η c ( x )) m
= p c ( y = m | x ) for m ∈ [ M ] , ( η ( x )) k
= p ( y = k
| x ) for k
Let π c =
[ π 1 c , · · · , π Mc ] ⊤ and π =
[ π 1 , · · · , π K ]
⊤ be vector forms of the surrogate and original class priors, and Π c ∈ R M
×
K be the matrix form of π m,kc defined in (5) .
Then we have
[Equation]
where the unnormalized version of vector-valued function Q c is given by
[Equation]
D a denotes the diagonal matrix with diagonal terms being vector a and · denotes matrix multiplication.
Q c is normalized by the sum of all entries, i.e.,
[Equation]
9: g c ← c do 11: g c ← g c − α
∇ c c
12: f c ← f c − α l · ∇
J c
Q c f
; U
13: end for 14: send
15: end procedure 16: procedure C .
S ERVER E XECUTE ( r
18: f f f
▷ FL aggregation 19: broadcast f to
20: end procedure
Note that π , π c , and Π c are all fixed, Q c is deterministic and can be identified with the knowledge of π , Π c , and an estimate of π c .
In addition, we note that if M c < M , the last M − M c entries of Q c are padded with zero by construction.
We further study the property of Q c in the following lemma.
Lemma 2.
For each client c ∈ [ C ] , the transition function Q c : ( X → ∆ K − 1 ) →
( X → ∆ M − 1 ) defined in Theorem 1 is an injective function.
In this sense, the transition function Q c naturally bridges the class-posterior probability of our desired task η ( x ) and that of the learnable surrogate task η c ( x ) given only U training sets.
This motivates us to embed the estimation of η ( x ) into the estimation of η c ( x ) at the client side.
Local Model Modification Second, we modify the model for the clients.
Let f ( x ) be the original global model output that estimates η ( x ) .
At each client c ∈ [ C ] , we implement Q c by adding a transition layer following f c and then we have g c ( x ) =
Q c ( f c ( x )) .
Based on the local model modification, (7) can be reformulated as follows:
[Equation]
Therefore, local updates on the surrogate model g c naturally yield updates on the underlying model f .
Now we can aggregate the surrogate clients by a standard FL scheme.
A detailed algorithm, using FedAvg as an example, is given in Algorithm 1.
Remarks The relationship elucidating the supervised and surrogate FL paradigms (for a single client) is visually shown in
Figure 2.
The blue part illustrates a supervised FL scheme: the global model is f ( x ) parameterized by θ ; the loss is R ( f ; y ) in (2) where labels y are observed.
However, we cannot observe y and thus employ a surrogate FedUL scheme which is shown in green: the global model is g ( x ) directly modified from f ; the loss is J ( g ; y ) in (7) where y are surrogate labels.
3. T HEORETICAL A NALYSIS
In what follows, we provide theoretical analysis on the convergence and optimal model recovery for the proposed method.
Convergence of the Algorithm In supervised FL, convergence analysis has been studied with regularity conditions (see Section 3.2.2 in Kairouz et al.
for a comprehensive survey), e.g., the bounded gradient dissimilarity (Karimireddy et al., 2020) that associates the client model with the server model for non-IID data distributions.
In our analysis, we focus on the behaviour of our surrogate learning objective J with the effect from transformation Q c .
Proposition 3 (Theorem V in Karimireddy et al.
Assume that J ( g ) in (7) satisfies (A1) - (A3) in Appendix C.3.
Denote g ∗ =
argmin g ∈G (cid:98) J ( g ) , where (cid:98) J
( g ) = 1 C (cid:80)
Cc =1 (cid:98)
J c
( g ) :
=n c (cid:80) n c i =1 ℓ ( g ( x i ) , y i ) .
Let the global step-size α g ≥ 1 and local step-size α l ≤ 1 8(1+ B 2 )
βLα g ,
FedUL given by Algorithm 1 is expected to have contracting gradient: with the initialized model g 0 ,
F :=
J ( g 0 ) − J ( g ∗ ) and constant M = σ (cid:113)
+ Cα 2 g , in R rounds, the learned model g R satisfies
[Equation]
Note that the proposition statement is w.r.t.
the convergence of g learned from the surrogate task.
By model construction, g is built from Q c ( f ) .
By Lemma 2, the convergence of trained model g to the optimal model g ∗ in the function class G implies the convergence of the retrieved model f to the optimal model f ∗ = argmin f ∈F (cid:98) J ( f ) in the corresponding function class F (induced by G ), where (cid:98) J ( f ) = 1 C (cid:80)
Cc =1 (cid:98) J c
( Q c ( f )) := 1 n c (cid:80) n c i =1 ℓ ( Q c ( f ( x i )) , y i ) is the empirical version of (9).
Optimal Model Recovery
In our context, the model is optimal in the sense that the minimizer of the surrogate task J ( f ; y ) coincide with the minimizer of the supervised task R ( f ; y ) , as if all the data have been labeled.
Based on the convergence of surrogate training, we show that FedUL can recover this optimal model.
Theorem 4.
Assume that the cross-entropy loss is chosen for ℓ , and the model used for g is very flexible, e.g., deep neural networks, so that g ∗∗ ∈ G where g ∗∗ = argmin g J ( g ; y ) .
Let f ∗ = argmin f ∈F (cid:98) J ( f ) be the optimal classifier learned with FedUL by Algorithm 1, and
f ⋆ = argmin
f R ( f ; y ) be the optimal classifier learned with supervised FL.
By assumptions in
Proposition 3, we have f ∗ =
f ⋆ .
E XPERIMENTS
In this section, we conduct experiments to validate the effectiveness of the proposed FedUL method under various testing scenarios.
Compared with the baseline methods, FedUL consistently achieves superior results on both benchmark and real-world datasets.
The detailed experimental settings and additional supportive results are presented in Appendix D & E.
4. B ENCHMARK E XPERIMENTS
Setup:
As proof-of-concepts, we first perform experiments on widely adopted benchmarks MNIST (LeCun et al., 1998) and CIFAR10 (Krizhevsky et al., 2009).
We generate M sets of U training data according to (5) for each client c ∈ [ C ] .
The total number of training samples n c on each client is fixed, and the number of instances contained in each set indexed by m is also fixed as n c /M .
In order to simulate the real world cases, we uniformly select class priors π m,k c from range [0 . 1 , 0 . 9] and then regularize them to formulate a valid Π c ∈ R M × K as discussed in Section 3.1.
For model training, we use the cross-entropy loss and Adam (Kingma & Ba, 2015) optimizer with a learning rate of 1 e − 4 and train 100 rounds.
If not specified, our default setting for local update
[Table]
epochs (E) is 1 and batch size (B) is 128 with 5 clients (C) in our FL system.
We use LeNet (LeCun et al., 1998) and ResNet18 (He et al., 2016) as backbone for MNIST and CIFAR10, respectively.
Here, we investigate two different FL settings: IID and Non-IID , where the overall label distribution across clients is the same in the IID setting, whereas clients have different label distributions in the non-IID setting.
Specially, for Non-IID setting, we impose class-prior shift as follows:
• the classes are divided into the majority and minority classes, where the fraction of each minority class is less than 0 .
08 while the fraction of each majority class is in the range [0 . 15 , 0 .
25] ; • the training data for every client are sampled from 2 majority classes and 8 minority classes; • the test data are uniformly sampled from all classes.
Baselines: (1) Federated Pseudo Labeling (FedPL) (Diao et al., 2021): at each client of a FedAvg system, we choose the label with the largest class prior in a U set as the pseudo label for all the data in this set.
During local training, these pseudo labels are boosted by Mixup (Zhang et al., 2017) high-confidence data and low-confidence data, where the confidence is given by model predictions.
(2) Federated LLP (FedLLP) (Dulac-Arnold et al., 2019): each client model is updated by minimizing the distance between the ground truth label proportion and the predicted label proportion, and we aggregate them using FedAvg.
(3) FedLLP with Virtual Adversarial Training (FedLLP-VAT)
(Tsai & Lin, 2020): a consistency term is added to FedLLP as regularization for optimizing client model in FedAvg.
(4) FedAvg with 10% labeled data (FedAvg 10%): following Chen et al.
, we also compare with supervised learning based on FedAvg using 10% fully labeled data.
The detailed baseline settings are depicted in Appendix D.
Results: The experimental results of our method and baselines on two benchmark datasets under
IID and Non-IID settings are reported in Table 1, where the mean error (std) over clients based on 3 independent runs are shown.
Given the limited number of total data available in the benchmark datasets, we compare our method with baselines in an FL system with five clients and vary the number of U sets M ∈ { 10 , 20 , 40 } at each client.
In this regard, each set contains a sufficient number of data samples to train the model.
Our observations are as follows.
First, the proposed FedUL method outperforms other baseline methods by a large margin under both IID and Non-IID settings.
Second, when varying the number of sets M ∈ { 10 , 20 , 40 } , FedUL not only achieves lower classification error but also shows more stable performance compared to baselines.
Third, as we use FedAvg as the default aggregation strategy for all the methods, the performances of FedUL, FedPL, and FedAvg all degrade under the inter-client
Non-IID setting, as expected.
The error rates of FedLLP and FedLLP-VAT are high and unstable, which demonstrates that the EPRM based methods are inferior as discussed in Section 1.
[Figure]
4. A BLATION S TUDIES
In the following, we present a comprehensive investigation on the ablation studies of the proposed
FedUL method on MNIST benchmark.
Robustness with Noisy Class Priors: To investigate the robustness of our method, we further apply
FedUL in a noisy environment.
Specifically, we perturb the class priors π m,kc by a random noise γ , and let the method treat the noisy class prior (cid:101) π m,kc = π m,
[Equation]
· ((2 ϵ − 1) γ + 1) , where γ uniform randomly take values in [0 , 1] and ϵ ∈ { 0 , 0 . 2 , 0 . 4 , 0 . 8 , 1 . 6 } , as the true π m,kc during the whole learning process.
Note that we tailor the noisy (cid:101) π m,kc if it surpasses the range.
We conduct experiments with five clients and ten U sets at each client.
The experimental setup is the same as before except for the replacement of π m,kc .
From Figure 3(a), we can see that FedUL performs stably even with some challenging noise perturbations (see when noise ratio ϵ > 0 ).
Effects of the Number of Accessible Non-IID U Data: The purpose of FL is utilizing more data to empower model performance.
While how unlabeled data, especially unlabeled Non-IID data, can contribute to FL model performance needs investigation.
To this end, we analyze the effects of the number of accessible Non-IID U data.
We set each client with n c = 500 samples ∀ c ∈
[ C ] and conduct experiments by changing the amount of Non-IID U data by increasing the number of clients
C ∈ { 10 , 20 , 30 , 40 , 50 } .
As shown in Figure 3(b), the model performance improves by a large margin as the number of accessible Non-IID U data increases by using our proposed FedUL method.
Analysis of Local Updating Epochs: The frequency of aggregation may sometimes influence the model performance in an FL system.
We would like to explore if FedUL is stable when the local clients are updated with different local epochs E .
In Figure 3(c), we explore E ∈ { 1 , 2 , 4 , 8 , 16 } with fixed B = 128 and C = 5 .
The results demonstrate that with different E , the performances of
FedUL are reasonably stable.
Effects of Batch Size: The training batch size is always an important factor in machine learning tasks, especially in an FL system.
Hence here, we study how batch size affects our model performance.
Specifically, we train the model using different batch sizes B ∈ { 16 , 64 , 128 , 512 , ∞} with fixed
E = 1 and C = 5 .
Here B = ∞ stands for loading all of the training samples into a single batch.
The results in Figure 3(d) indicate that FedUL can effectively work under a wide range of batch sizes.
4. C ASE S TUDY ON R EAL - WORLD P ROBLEM
To demonstrate the feasibility of our proposed FedUL scheme to the real problem, we test FedUL on a medical application for the diagnosis of brain disease.
In the rest of this section, we introduce the problem formulation, data and setting, and then discuss the results.
Real Problem Formulation: For a real-world scenario, we consider a clinical setting for disease classification in C different healthcare systems.
Suppose each healthcare system has M hospitals located in different regions, each associated with unique class priors due to population diversities.
Medical data are not shareable between the healthcare systems for privacy.
We assume patient-wise diagnosis information is unavailable, but each hospital provides the prevalence of these patients (a.k.a.
class priors of the disease classes).
[Figure]
[Table]
[Figure]
et al., 2020).
Dataset and Settings: We use the RSNA Intracranial Hemorrhage Detection dataset (Flanders et al., 2020), which contains large-scale Magnetic Response Image (MRI) data from several hospitals (see
Fig. 4), to simulate this real-world scenario.
There are 106 k samples in the dataset belonging to six classes, including five types of intracranial hemorrhage (Epidural, Intraparenchymal, Intraventricular,
Subarachnoid, and Subdural) and healthy control type.
We model an FL system with five healthcare systems (a.k.a clients).
Each healthcare system contains 21 .
2 k data samples.
To show the effect on the number of samples in each U set, we vary the number of hospitals (a.k.a sets) M = 12 , 24 , and 48 in each system.
Here we generate each π m,kc for the k -th class in m -th U set at c -th client from range [0 . 1 , 0 . 9] and check that the generated class priors are not all identical.
For model training, we use the cross-entropy loss and Adam (Kingma & Ba, 2015) optimizer with a learning rate of 1 e −and train 100 rounds.
We use ResNet18 (He et al., 2016) as backbone for this experiment.
Results and Analysis: As indicated in our proof-of-concept (Table 1), FedPL consistently outperforms the other two LLP baselines, i.e., FedLLP and FedLLP-VAT.
Here we focus on comparing
FedUL with FedPL.
We also include the supervised baseline FedAvg with 10% fully labeled data.
We report the results of each hospital in mean error (std) format over three runs in Table 2.
We can see that our proposed FedUL method consistently outperforms FedPL and FedAvg 10% for all the hospitals and under different set number settings.
The superiority of FedUL in this challenging real-world FL setting further indicates the effectiveness and robustness of our algorithm.
These results are inspiring and bring the hope of deploying FedUL to the healthcare field, where data are often very costly to be labeled, heterogeneous, and distributed.
C ONCLUSION
In this paper, we propose FedUL, a novel FL algorithm using only U data.
FedUL works as a wrapper that transforms the unlabeled client data and modifies the client model to form a surrogate supervised FL task, which can be easily solved using existing FL paradigms.
Theoretically, we prove that the recovery of the optimal model by using the FedUL method can be guaranteed under mild conditionals.
Through extensive experiments on benchmark and real-world datasets, we show FedUL can significantly outperform the baseline methods under different settings.
As a wrapper, we expect
FedUL can be further improved by incorporating advanced FL aggregation or optimization schemes for the non-IID setting (Wang et al., 2020;
Li et al., 2020; 2021; Karimireddy et al., 2020).
E THICS S TATEMENT
Collecting sample-wise labels for training can be costly and may violate data privacy.
Considering data in the real world are often distributed at a large scale, we provide a promising solution to perform multi-class classification on distributed U data in the FL system.
Our method FedUL can be widely applied to data-sensitive fields, such as healthcare, politics, and finance, where the sample-wise label is not available, but populational prevalence can be accessed.
With FedUL, we hope the future FL can enjoy significant labeling cost reduction and improve its privacy.
The real datasets used in this work are publicly available with appropriate prepossessing to de-identify patients’ identities.
We provide proper references to the datasets, libraries, and tools used in our study.
This study does not have any legal compliance, conflict of interest, or research integrity issues.
R EPRODUCIBILITY S TATEMENT
We provide the codes to reproduce the main experimental results at https://github.com/ lunanbit/FedUL .
We further provide experimental details and additional information of the datasets in the supplementary materials
A CKNOWLEDGMENTS
NL was supported by JST AIP Challenge Program and the Institute for AI and Beyond, UTokyo.
ZW and QD were supported by project of Shenzhen-HK Collaborative Development Zone.
GN was supported by JST AIP Acceleration Research Grant Number JPMJCR20U. MS was supported by
JST AIP Acceleration Research Grant Number JPMJCR20U3 and the Institute for AI and Beyond,
UTokyo.
Supplementary Material
Roadmap
The appendix is organized as follows.
We summarize notations used in our paper in
Section A and give a full discussion of related work in Section
B. Our proofs are presented in
Section C. We provide detailed experimental settings in Section D and supplementary experimental results on the benchmark datasets in Section E.
A N OTATION T ABLE
[Table]
c = {U m c } M c m =
M the maximum number of available U sets among all clients M = max c ∈ [ C ] M c π m,k c k -th class prior π m,k c = p m c ( y = k ) of the m -th
U set at the c -th client Π c matrix form of π m,k c where Π c ∈ R M c
× K y surrogate label ¯ y ∈
[ M ] p c surrogate local data distribution at the c -th client
U c (surrogate) labeled training set U c
= { x i , y i } n c i =1 at the c -th client π mc m -th surrogate class prior π mc = p c ( y = m ) at the c -th client π c vector form of π mc where π c =
[ π 1 c , · · · , π Mc ] ⊤ g surrogate global model predicting surrogate labels y for inputs x η c surrogate class-posterior probability at the c -th client where ( η c ( x )) m
= p c ( y = m | x ) η true class-posterior probability at the c -th client where ( η ( x ))
k
= p ( y = k
| x )
Q c vector-valued transition function Q c : ( X → ∆ K ) → ( X → ∆ M )
B R ELATED W ORK
Considering the heterogeneity of data, clients, systems, and applications in FL, personalized FL is a natural and trending solution to improve FL performance in heterogeneous settings.
Many efforts have been made for personalized FL to address the challenges of heterogeneity.
FedProx(Li et al., 2020), FedCurv (Shoham et al., 2019) and FedCL (Yao & Sun, 2020) proposed regularization-based method by including special regularization terms to improve FL model generalizing to various clients.
Meta-learning was involved to train highly-adaptable to achieve personalized FL by Jiang et al.
Fallah et al.
; Li et al.
Different from the above-mentioned methods that yield a single generalizable model, multiple FL models can be trained using clustering strategies to achieve personalization (Sattler et al., 2020; Ghosh et al., 2020).
Different local FL models also can be achieved by introducing personalized layers (Li et al., 2021; Arivazhagan et al., 2019).
However, our learning settings and methods are very different from the existing personalized FL approaches.
Firstly, our framework focuses on unsupervised FL.
Specifically, 1) If inter-clients are iid, FedUL yields the same proxy model g c
( c is the index of clients) and federated model f for all the clients.
If inter-clients are non-iid, although g c are different, the underlying model f
(what we want) is still the same.
Secondly, our method differs from the previous personalized FL approaches by its surrogate training nature: it works as a wrapper that transforms the unlabeled clients to (surrogate) labeled ones and is compatible with many supervised FL methods.
C P ROOFS
In this appendix, we prove all theorems.
C. P ROOF OF T HEOREM
By the Bayes’ rule, ∀ m ∈
[ M ] we have:
[Equation]
By transforming (10) to the matrix form, we complete the proof.
C. P ROOF OF L EMMA
Given our data generation process in Section 3.1, we have that M c ≥ k and Π c ∈ R M c
×
K is a full column rank matrix.
With D π c and D − 1 π being non-degenerate, we obtain that Q c ( η ( x )) presents an injective mapping from η ( x ) to η c ( x ) .
C. P ROOF OF P ROPOSITION
The assumptions (A1) - (A3) are listed as follows.
(A1)
Bounded gradient dissimilarity: ∃ constants G ≥ 0 and B ≥ 1 s.t.
[Equation]
(A2) β -smooth function J c satisfies
[Equation]
(A3) Unbiased stochastic gradient with bounded variance:
[Equation]
where (cid:98)
J c
( g ; U c )
= 1 n c (cid:80) n c i =1 ℓ ( g ( x i ) , y i ) .
Note that this proposition statement is w.r.t.
the convergence of model g learned from the surrogate supervised learning task.
The proof follows directly from Theorem V in Karimireddy et al.
C. P ROOF OF T HEOREM
When a proper loss (Reid & Williamson, 2010) is used for ℓ , the mapping of g ∗∗ is the unique minimizer of J ( g ) .
We show an example of the a widely used proper loss function: the cross-entropy loss (De Boer et al., 2005) defined as
[Equation]
We can see that the cross-entropy loss is non-negative by its definition.
Therefore, minimizing J ( g ) can be obtained by minimizing the conditional risk E p ( y | x ) [ ℓ ( g ( x ) , y ) | x ] for every x ∈ X .
So we are now optimizing
[Equation]
By using the Lagrange multiplier method (Bertsekas, 1997), we have
[Equation]
The derivative of L with respect to g is
[Equation]
By setting this derivative to 0 we obtain
[Equation]
Since g ∈ ∆ M − 1 is the M -dimensional simplex, we have (cid:80) Mm =1 ( g ∗∗ ( x )) m
= 1 and (cid:80)
Mm =1 p ( y = j | x ) = 1 .
Then
M
[Equation]
Therefore we obtain λ = − 1 and ( g ∗∗ ( x )) m
= p
y = m | x ) =
( η ( x )) m , ∀ m = 1 , . .
, M and ∀ x ∈ X , which is equivalent to g ∗∗ ( x ) =
¯ η ( x ) .
Similarly, we obtain
[Equation]
Given Proposition 3 and the assumption that the model used for g is sufficiently flexible, we obtain that the classification model learned by FedUL (see Algorithm 1) satisfies g ∗ =
g ∗∗ .
By our model construction, g is built from Q c ( f ) .
Then we have
[Equation]
where f ∗∗ = argmin f J ( f ) is the model induced by g ∗∗ .
Given Theorem 1 and Lemma 2, we obtain that g ∗∗ ( x ) =
¯ η ( x ) if and only if
[Equation]
By summarizing (11), (12), and (13) we obtain f ∗ =
f ⋆ .
D E XPERIMENTAL DETAILS
We illustrate our model architectures and training details of the benchmark and real-world experiments in this section.
D. M ODEL A RCHITECTURE
For our benchmark experiment on MNIST, we use a six-layer convolutional neural network.
The network details are listed in Table D.2.
For our benchmark experiment on CIFAR10 and real-world experiment, we use ResNet18 as our backbone.
The network details are listed in Table D.3.
[Table]
D. T RAINING D ETAILS
Benchmark Experiments:
We implement all the methods by PyTorch and conducted all the experiments on an NVIDIA TITAN X GPU.
Please note for all experiments, we split 20% original data for validation and model selection.
During training process, we use Adam optimizer with learning rate 1 e − 4 and the cross-entropy loss, we set batch size to 128 and training rounds to 100 .
To avoid overfitting, we use L1 regularization.
The detailed weight of L1 regularization used in benchmark experiments is shown in Table D.4.
The detailed number of samples in each client for MNIST and CIFAR10 under IID and Non-IID setting is given in Table D.5 and Table D.6, respectively.
We describe details of the baseline methods and corresponding hyper-parameters as follows.
• FedPL (Diao et al., 2021): this method is based on empirical classification risk minimization.
The learning objective is
[Equation]
where
[Equation]
is the "fix" loss, where ( x + b , y + b ) represents for a sample x + b with high confidence pseudo label y + b and
[Equation]
is the "mix" loss, in which
[Equation]
where a
is the
mixup
hyper-parameter and
( x − b y − b ) represents for a sample x − b with low confidence pseudo label y − b .
Following
the default implementation in (Diao et al., 2021),
[Table]
Conv2D(3, 64, 7, 2, 3), BN(64), ReLU Conv2D(64, 64, 3, 1, 1), BN(64), ReLU
Conv2D(64, 64, 3, 1, 1), BN(64)
Conv2D(64, 64, 3, 1, 1), BN(64), ReLU
Conv2D(64, 64, 3, 1, 1), BN(64)
Conv2D(64, 128, 3, 2, 1), BN(128), ReLU
Conv2D(128, 128, 3, 1, 1), BN(64)
Conv2D(64, 128, 1, 2, 0), BN(128)
Conv2D(128, 128, 3, 1, 1), BN(128), ReLU
Conv2D(128, 128, 3, 1, 1), BN(64)
Conv2D(128, 256, 3, 2, 1), BN(128), ReLU
Conv2D(256, 256, 3, 1, 1), BN(64)
Conv2D(128, 256, 1, 2, 0), BN(128)
Conv2D(256, 256, 3, 1, 1), BN(128), ReLU
Conv2D(256, 256, 3, 1, 1), BN(64)
Conv2D(256, 512, 3, 2, 1), BN(512), ReLU
Conv2D(512, 512, 3, 1, 1), BN(512)
Conv2D(256, 512, 1, 2, 0), BN(512)
Conv2D(512, 512, 3, 1, 1), BN(512), ReLU Conv2D(512, 512, 3, 1, 1), BN(512)
[Table]
[Table]
we set hyper-parameter a = 0 .
Moreover, we set the weight of "mix" loss λ = 0 .
and the confidence threshold τ = 0 .
• FedLLP (Dulac-Arnold et al., 2019): this baseline method is based on empirical proportion risk minimization.
The learning objective is
[Equation]
is the aggregated proportion risk, π mc and (cid:98)
π
mc are the true and predicted label proportions for the m -th
U set U mc in c -th client, (cid:98)
π mc
= 1 M c (cid:80) x
∼U mc f ( x ) .
For each client c , d prop is defined as
[Equation]
• FedLLP-VAT (Tsai & Lin, 2020): this method is based on empirical proportion risk minimization, integrated with a consistency regularization term.
The learning objective is
[Table] [Table]
[Equation]
(19) is the aggregated proportion risk with consistency regularization, π mc and (cid:98)
π mc are the true and predicted label proportions for the m -th
U set U mc in c -th client, d prop is a distance function, and ℓ cons
[Equation]
cons
[Equation]
x
[Equation]
)) (20) is the consistency loss, d cons is a distance function, (cid:98) x is a perturbed input from the original one x .
Following the default implementation in their paper (Tsai & Lin, 2020), we set the hyper-parameter the weight of consistency loss α = 0 .
05 and the perturbation weight µ = 6 .
for FedLLP-VAT on CIFAR10 benchmark experiment.
For MNIST benchmark, we set the hyper-parameter the weight of consistency loss α = 5 e − 4 and the perturbation weight µ = 1 e − 2 .
Real-world Experiments:
We perform a study case on an important clinical task — intracranial hemorrhage diagnosis.
Intracranial hemorrhage, described as bleeding that occurs inside the cranium, is a severe health problem.
The distribution of intracranial hemorrhage depends on the patient’s ethnicity, such as age (Camacho et al., 2015).
Thus, considering age distributions vary in different regions, this dataset and problem naturally fits into our setting.
We use the RSNA Intracranial Hemorrhage Detection dataset (Flanders et al., 2020)
(Fig. 4), which contains large-scale Magnetic Response Image (MRI) data from several hospitals.
There are 106 k samples in the dataset belonging to six classes, including five types of intracranial hemorrhage (Epidural, Intraparenchymal, Intraventricular, Subarachnoid, and Subdural) and healthy control type.
In our experiments, we simulate a real-world scenario with five healthcare systems (a.k.a.
clients) and assume the number of hospitals (a.k.a sets) in each healthcare systems varies M = 12 , 24 , and 48 .
At first, we randomly split the whole dataset for five healthcare systems equally, so each healthcare systems is assigned 21 .
2 k samples.
Then we generate π mc for m -th set in c -th client from range
[Table]
9] for all the hospitals with U data only.
We check that the generated class priors are not all identical.
Finally, we load samples for each the hospitals following the generated π .
Detailed dataset statistics is given in Table D.7.
During training process, we use Adam optimizer with learning rate 1 e − 4 and cross-entropy loss, we set batch size to 128 and training rounds to 100 .
To avoid overfitting, we use L1 regularization.
We set the weight of L1 regularization as 2 e − 6 , 8 e − 7 and 4 e − 7 for 10, 20 and 40 sets, respectively.
In real-world experiments, we use the same hyper-parameters as benchmark experiments for all methods except FedLLP-VAT.
For FedLLP-VAT, we set the hyper-parameter the weight of consistency loss α = 0 .
05 and the perturbation weight µ = 6 . 0 .
Training Efficiency:
We further evaluate the training efficiency of our method with comparison to baseline methods.
The detailed one epoch training time (mean) and forward FLOPs are given in Table D.8.
The computation time is evaluated on a PC with one Nvidia TITAN X GPU.
We can see that our method slightly increase the forward FLOPs and can achieve higher training efficiency compared with FedPL and FedLLP-VAT.
[Table]
E M ORE E XPERIMENTAL R ESULTS ON THE B ENCHMARK D ATASETS E. D ETAILED S TATISTICS OF F IGURE 3( A )
We show detailed statistics of figure 3(a) in Table E.9.
With different noisy class priors, even challenging ϵ = 0 .
8 and ϵ = 1 . 6 , our method still achieves low error rate, which strongly verify the robustness and effectiveness of our method.
[Table]
E. D ETAILED S TATISTICS OF F IGURE 3( B )
We show detailed statistics of figure 3(b) in Table E.10.
With more Non-IID data can be accessed, our method performs better, which validate that more unlabeled Non-IID data can be helpful for FL.
[Table]
E. D IFFERENT C OMBINATIONS OF E AND B
We compare our method with different combinations of E and B on benchmark dataset MNIST with the Non-IID setting.
The classification error rate (mean and std) are consistently small as presented (cf.
Table E.11)
[Table]
which indicates our method can work under a wide range combinations of E and B .
[Table]
E. B ENCHMARK E XPERIMENTS ON C LIENTS WITH D IFFERENT S ETS
Here we conduct experiments on benchmark datasets under IID and Non-IID setting with 5 clients and different sets in each client (10, 20, 30, 40, 50).
The experimental results are shown in Table
E.
[Table]
which further indicates the effectiveness of our method even under this challenging setting.
E. B ENCHMARK E XPERIMENTS WITH M ORE C LIENTS
To study the influence of the number of clients
[Table]
here we further conduct benchmark experiments on both MNIST and CIFAR10 with 10 clients.
The detailed results are shown in Table E.
[Table]
in which we has the similar observations as in Table
[Table]
[Table] [Table]