U NSUPERVISED F EDERATED L EARNING IS P OSSIBLE : A C ASE OF C LASS -C ONDITIONAL -S HARING C LIENTS
Anonymous authors
Paper under double-blind review
A BSTRACT
Supervised federated learning (FL) enables multiple clients to share the trained model without sharing their labeled data.
However, potential clients might even be reluctant to label their own data, which could limit the applicability of FL in practice.
In this paper, we show the possibility of unsupervised FL whose model is still a classiﬁer for predicting class labels, if there is sufﬁcient class-prior shift among the unlabeled data owned by the clients and the class priors are known to the clients.
We propose federation of unsupervised learning (FedUL), where the unlabeled data are transformed into surrogate labeled data for each of the clients, a modiﬁed model is trained by supervised FL, and the wanted model is recovered from the modiﬁed model.
FedUL is a very general solution to unsupervised FL: it is compatible with many supervised FL methods, and the recovery of the wanted model can be theoretically guaranteed as if the data have been labeled.
Experiments on benchmark and real-world datasets demonstrate the effectiveness of FedUL.
Anonymous code download link is given in Appendix and will be open source later.
I NTRODUCTION
Federated Learning (FL) has received signiﬁcant attention from both academic and industrial perspectives in that it can bring together separate data sources and allow multiple clients to train a central model in a collaborative but private manner (McMahan et al., 2017; Kairouz et al., 2019;
Yang et al., 2019).
So far, the majority of FL researches focused on the supervised setting, requiring collected data at each client to be fully labeled —this may hinder the spread of FL in practice since labeling large-scale training data can be extremely costly due to laborious mannual annotations, and sometimes may not even be possible for privacy concerns, e.g., in medical diagnosis (Ng et al., 2021).
In this paper, we are interested in a challenging unsupervised FL problem where only unlabeled (U) data are available at each client.
Our goal remains to train a classiﬁcation model that can predict accurate class labels in the FL setup.
It is often observed that each client collects their own data at different time points or from different geographic locations, e.g., a hospital (data center) may store patient data every month or a particular user (mobile device) may take photos at different places.
Therefore, we consider a realistic setting that the U data at a client come in the form of separate data sets : the data distribution of each U set and the number of U sets available at each client may vary .
Without any labels, FL becomes signiﬁcantly harder than before.
In this work, we show that the aforementioned unsupervised FL problem can be solved if the class prior probability shifts between the available U sets at each client, and these class priors are known to the clients.
Such a learning scenario can be conceivable in many real-world problems.
For example, the hospital may not release the patients’ diagnostic labels due to privacy reasons, but the morbidity rates of diseases (corresponding to the class priors) may change over time and can be accessible from related public medical reports (Croft et al., 2018); moreover, in many cases it is possible to estimate the class priors much more cheaply than the ground-truth labels (Quadrianto et al., 2009a).
Based on these ﬁndings, we propose the federation of unsupervised learning (FedUL) (cf.
Figure 1), where the original clients with U data are transformed to surrogate clients with (surrogate) labeled data and then supervised FL can be applied.
The difﬁculty is how to infer our wanted model for the original classiﬁcation task from the surrogate model learned by the surrogate task.
We solve this problem by bridging the original and surrogate class probabilities with certain injective transition
[Figure]
Client
Client C … (a) Standard federated learning 𝒇 𝒇𝒇𝒇 𝑪
Server
Unlabeled
Surrogate label True label
Server update
Local update 𝒈
So 𝑸 𝟏 𝒇… (b) Federation of unsupervised learning (FedUL) 𝒈
So 𝑸𝒈 𝑪
So 𝑸 𝑪 𝒇𝒇 𝑪
Server 𝒇 functions.
Then we implement it by adding a transition layer to the output of each client model, so that the learned model is guaranteed to be a good approximation of the original class probability
On one hand, FedUL is a very general and ﬂexible: it works as a wrapper that transforms the original clients to the surrogate ones, such that it is compatible with many supervised FL methods, e.g.,
FedAvg, and add no additional communication cost.
On the other hand, FedUL is computationally efﬁcient and easy-to-implement: since the added transformation layer is ﬁxed and determined by the class priors, FedUL adds no more burden on hyper-parameter tuning and/or optimization.
Our contributions can be summarized as follows:
• Methodologically, we propose a novel FedUL method that employs a surrogate FL task, providing new perspectives for dealing with U data in FL. • Theoretically, we prove that the optimal global model learned by FedUL with only U data converges to the optimal global model learned by supervised FL with fully labeled data under mild conditions.
• Empirically, we demonstrate the effectiveness of FedUL on benchmark and real-world datasets.
Related Work: FL has been extensively studied in the supervised setting.
One of the most popular supervised FL paradigms is FedAvg (McMahan et al., 2017) which aggregates the local updates at the server and transmits the averaged model back to local clients.
In contrast, FL in the unsupervised setting is less explored.
Recent studies have shown the possibility of federated clustering with U data (Ghosh et al., 2020; Dennis et al., 2021).
However, these clustering based methods rely on geometric or information-theoretic assumptions to build their learning objectives (Chapelle et al., 2006), and are sub-optimal for classiﬁcation goals.
Our work is intrinsically different from them in the sense that we show the possibility of federated classiﬁcation with U data based on empirical risk minimization (ERM) (Vapnik, 1998), and therefore the optimal model recovery can be guaranteed.
Learning from U datasets has also been studied in the classical centralized learning.
Mainstream directions include learning with label proportions (LLP)
(Quadrianto et al., 2009b) and U m classiﬁcation (Lu et al., 2021).
The former learns a multi-class classiﬁer from U datasets based on empirical proportion risk minimization (EPRM) (Yu et al., 2014), while the latter learns a binary classiﬁer from
U datasts based on ERM.
EPRM is inferior to ERM since its learning is not consistent.
Yet, it is still unexplored how to learn a multi-class classiﬁcation model from U datasets based on ERM.
To the best of our knowledge, this work is the ﬁrst attempt to tackle this problem in the FL setup, which is built upon the binary U m classiﬁcation in the centralized learning.
S TANDARD F EDERATED L EARNING
In this section, we review standard FL of a classiﬁcation model.
We consider a K -class classiﬁcation problem with the feature space X ⊂ R d and the label space Y =
[ K ] , where d is the input dimension and [ K ] := { 1 , . .
, K } .
Let x ∈ X
and
y ∈ Y be the input and output random variables following an underlying joint distribution with density p ( x , y ) , which can be identiﬁed via the class priors { π k = p ( y = k ) } Kk =1 and the class-conditional densities { p ( x | y = k ) } Kk =1 .
Let f : X → R K be a classiﬁcation model that assigns a score to each of the K classes for a given input x
and then outputs the predicted label by y pred =
argmax k ∈
[ K ] ( f ( x ))
k , where ( f ( x ))
k is the k -th element of f ( x ) .
In standard FL setup with C clients, each client c ∈ [ C ] has access to a labeled training set
D c = { ( x ci , y ci ) } n c i =1 ∼ p c ( x , y ) of sample size n c and learns its local model f c by minimizing the following risk:
[Equation]
where E denotes the expectation, (cid:96) : R K × Y → R + is a proper loss function, e.g., the softmax cross-entropy loss , i.e., (cid:96) ce
( f c ( x ) , y ) = − (cid:80) Kk =1 1 ( y = k ) log( f c ( x ))
k
= − log( f c ( x ))
y , and 1 ( · ) is the indicator function.
In practice, ERM is a common practice that computes an approximation of R c ( f c ) based on the client’s local data D c by (cid:98)
[Equation]
The goal of standard FL (McMahan et al., 2017) is that the C clients collaboratively train a global classiﬁcation model f that generalizes well with respect to p ( x , y ) , without sharing their local data
D c .
The problem can be formalized as minimizing the aggregated risk:
[Equation]
In practice, we typically employ a server to coordinate the iterative distributed training as follows:
• in each global round of training, the server broadcasts its current model f to all the clients { f c } Cc =1 ; • each client c copies the current server model f c = f , performs L local step updates:
[Equation]
where α l is the local step-size, and sends f c − f back to the server; • the server aggregates the updates from all clients { f c − f } Cc =1 to form the new server model using FedAvg
[Equation]
where α g is the global step-size.
F EDERATED L EARNING : N O L ABEL N O C RY
In this section, we formulate the novel problem setting of FL with only unlabeled datasets, propose our method called federation of unsupervised learning (FedUL), and provide theoretical analysis of
FedUL.
All the proofs are given in Appendix C.
3. P ROBLEM F ORMULATION
In this paper, we consider a challenging setting: instead of fully labeled training data, each client c ∈ [ C ] observes M c ( M c ≥ K, ∀ c ) 1 sets of unlabeled samples, U c
= {U mc } M c m =1 , where U mc =
{ x c,m i } n c,m i
=1 denotes the collection of the m -th
U set for client c with sample size n c,m .
Each U set can be seen as a set of data points drawn from a mixture of the original class-conditional densities:
[Equation]
where π m,kc = p mc ( y = k ) denotes the k -th class prior of the m -th
U set.
These class priors form a full column rank matrix Π c ∈ R M c × K with the constraint that each row sums up to 1.
Note that we do not require any labels to be known, only assume the class priors within each of the involved data sets are available, i.e., the training class priors π m,kc and the test class
prior π k = p ( y = k ) .
Our goal is the same as standard FL, i.e., the C clients are interested in collaboratively training a global classiﬁcation model f that generalizes well with respect to p ( x , y ) , but using only unlabeled datasets U c on each local client.
3. P ROPOSED M ETHOD
As introduced in Section 2, FedAvg is a representative approach for aggregating clients in the
FL framework and serves as the basis for many advanced federated aggregation techniques, e.g.,
Fedprox (Li et al., 2020), FedNova (Wang et al., 2020), and SCAFFOLD (Karimireddy et al., 2020).
However, these methods cannot handle clients without labels, which are necessary for them to compute the local gradients.
To tackle this problem, we consider the construction of surrogate clients by transforming the local U datasets to labeled datasets and modifying the local models so that they are compatible with the transformed data for updating.
Local Data Transformation
First, we transform the data of the clients.
For each client c ∈ [ C ] , let y be the index of given U sets, and p c ( x , y ) be an underlying joint distribution for the random variables x ∈ X and y ∈
[ M ] , where M = max c ∈ [ C ] M c .
By treating y as a surrogate label , we may transform the original U training sets U c
= {U mc } M c m =1 to a labeled training set for client c ,
[Equation]
Obviously, the surrogate class-conditional density p c ( x | y
= m ) corresponds to p mc ( x ) in Eq.
(5), for m ∈
[ M c ] .
If ∃ M c < m ≤ M , we pad p c ( x | y
= m ) = 0 .
The surrogate class-prior π mc = p c ( y = m ) can be estimated by π mc = n c,m / ( (cid:80)
M c m =1 n c,m ) .
Based on the data transformation, we then formulate a surrogate supervised FL task : the C clients aim to collaboratively train a global classiﬁcation model g : X → R M that predicts surrogate labels y for given inputs x , without sharing their local data U c .
This task can be easily solved by standard supervised FL methods.
More speciﬁcally, we minimize
[Equation]
by employing a server g that iteratively broadcasts its model to all local clients { g c } Cc =1 and aggregates local updates from all clients using e.g., FedAvg, where the local and global update procedures exactly follow Eq.
(3) and Eq.
(4) in standard FL.
Now, a natural question arises: can we infer our desired classiﬁcation model f from the surrogate model g ?
To answer this question, we study the relationship of the original and surrogate class probabilities in the following theorem.
Theorem 1.
For each client c ∈ [ C ] , let η c : X → ∆ M − 1 and η : X → ∆ K − 1 be the surrogate and original class probability functions, where ( η c ( x )) m
= p c ( y = m | x ) for m ∈ [ M ] , ( η ( x )) k
= p ( y = k
| x ) for k ∈ [ K ] , ∆ M − 1 and ∆ K − 1 are the M - and K -dimensional simplexes.
Let π c
=
[ π 1 c , · · · , π Mc ] (cid:62) and π =
[ π 1 , · · · , π K ] (cid:62) be vector forms of the surrogate and original class priors, and
Π c ∈ R M
×
K be the matrix form of π m,kc deﬁned in Eq.
Then we have
[Equation]
where the unnormlized version of vector valued function Q c is given by
[Equation]
D a denotes the diagonal matrix with diagonal terms being vector a and · denotes matrix multiplication. Q c is normalized by the sum of all entries, i.e., ( Q c
[Equation] i [Equation] 9: g
←
11: g c ← − α ∇
12: f
← f
− ∇ 14: send
15: end procedure 16: procedure C
18: f ← f aggregation 19: broadcast
20: end procedure
Note that π , π c , and Π c are all ﬁxed, Q c is deterministic and can be identiﬁed with the knowledge of π , Π c , and an estimate of π c .
In addition, we note that if M c < M , the last M − M c entries of Q c are padded with zero by construction.
We further study the property of Q c in the following lemma.
Lemma 2.
For each client c ∈ [ C ] , the transition function Q c : ( X → ∆ K ) →
( X → ∆ M ) deﬁned in Theorem 1 is an injective function.
In this sense, the transition function Q c naturally bridges the class probability of our desired task η ( x ) and that of the learnable surrogate task η c ( x ) given only unlabeled datasets.
This motivates us to embed the estimation of η ( x ) into the estimation of η c ( x ) at each client.
Local model modiﬁcation
Second, we modify the model for the clients.
Let f ( x ) be the original global model output that estimates η ( x ) .
At each client c ∈ [ C ] , we implement Q c by adding a transition layer following f
and then we have g c ( x ) =
Q c ( f ( x )) .
Based on the model transformation,
Eq.
(7) can be reformulated as follows:
[Equation]
[Equation]
Therefore, local updates on the surrogate client model g c naturally yield updates on the underlying model f .
Now we can aggregate these surrogate clients with our selected standard FL scheme.
A detailed algorithm, using FedAvg as an example, is given in Algorithm 1.
Remarks The relationship elucidating the supervised and surrogate FL paradigms (for a single client) is visually shown in
Figure 2.
The blue part illustrates a supervised FL scheme: the global model is f ( x ) parameterized by θ ; the loss is R ( f ; y ) in Eq.
(2) where labels y are observed.
However, we cannot observe y and thus employ a surrogate FedUL scheme which is shown in green: the global model is g ( x ) directly modiﬁed from f ; the loss is J ( g ; y ) in Eq.
(7) where y are surrogate labels.
3. T HEORETICAL A NALYSIS
In what follows, we provide theoretical analysis on the convergence and optimal model recovery for the proposed method.
Convergence of the algorithm
In supervised FL, convergence analysis has been studied with regularity conditions (see Section 3.2.2 in Kairouz et al.
for a comprehensive survey), e.g. bounded gradient dissimilarity (Karimireddy et al., 2020) that associates the client model with the server model for non-IID data distributions.
In our analysis, we focus on the behaviour of our surrogate learning objective J with the effect from transformation Q c .
Proposition 3 (Theorem V in Karimireddy et al.
Assume that J ( g ) in Eq.
(7) satisﬁes (A1) - (A3) in Appendix C.3.
Denote g ∗ =
argmin g ∈G (cid:98) J ( g ) , where (cid:98) J
( g ) = 1 C (cid:80)
Cc =1 (cid:98)
J c
( g ) :
=
n c (cid:80) n c i =1 (cid:96) ( g ( x i ) , y i ) .
Let the global step-size α g ≥ 1 and local step-size α l ≤
[Equation]
FedUL given by Algorithm 1 is expected to have contracting gradient: with the initialized model g 0 ,
F :=
J ( g 0 ) − J ( g ∗ ) and constant M = σ (cid:113) 1 + Cη 2 g , in R rounds, the learned model g R satisﬁes
[Equation]
Note that the proposition statement is w.r.t.
the convergence of g learned from the surrogate task.
By model construction, g is built from Q c ( f ) .
By Lemma 2, the convergence of trained model g to the optimal model g ∗ in the function class G implies the convergence of the retrieved model f to the optimal model f ∗ = argmin f ∈F (cid:98) J ( f ) in the corresponding function class F (induced by G ), where (cid:98) J ( f ) = 1 C (cid:80)
Cc =1 (cid:98) J c
( Q c ( f )) := 1 n c (cid:80) n c i =1 (cid:96) ( Q c ( f ( x i )) , y i ) is the empirical version of Eq.
Optimal model recovery
In our context, the model is optimal in the sense that the minimizer of the surrogate task J ( f ; y ) coincide with the minimizer of the supervised task R ( f ; y ) , as if all the data have been labeled.
Based on the convergence of surrogate learning, we show that FedUL can recover this optimal model.
Theorem 4.
Assume that a proper loss function, e.g., the cross-entropy loss, is chosen for (cid:96) , and the model used for g is very ﬂexible, e.g., deep neural networks, so that g ∗∗ ∈ G where g ∗∗ = argmin g J ( g ; y ) .
Let f ∗ = argmin f ∈F (cid:98) J ( f ) be the optimal classiﬁer learned with FedUL by
Algorithm 1, and f (cid:63) =
argmin f R ( f ; y ) be the optimal classiﬁer learned with supervised FL.
By assumptions in Proposition 3, we have f ∗ =
f (cid:63) .
E XPERIMENTS
In this section, we conduct experiments to validate the effectiveness of the proposed FedUL method under various testing scenarios.
Compared with alternative strategies, FedUL consistently achieves superior results on both benchmark and real-world datasets.
The detailed experimental settings and additional supportive results are presented in the Appendix D & E.
4. B ENCHMARK E XPERIMENTS
Settings: As proof-of-concepts, we ﬁrst perform experiments on widely adopted benchmarks MNIST (LeCun et al., 1998) and CIFAR
(Krizhevsky et al., 2009).
For all experiments, we split 20% training data for validation and model selection.
We generate M sets of U training data according to Eq.
(5) for each client c ∈ [ C ] .
The total number of training samples n c on each client is ﬁxed, and the number of instances contained in each set indexed by m is also ﬁxed as n c /M .
In order to simulate the real world cases, we uniformly select class priors π m,kc = p mc ( y = k ) from range [0 . 1 , 0 . 9] and then regularize them to formulate a valid Π c ∈ R M c × K as discussed in Section 3.1.
[Table]
For model training, we use the cross-entropy loss and Adam (Kingma & Ba, 2015) optimizer with a learning rate of 1 e − 4 and train 100 rounds.
If not speciﬁed, our default setting for local update epochs (E) is 1 and batch size (B) is 128 with 5 clients (C) in our FL system.
We use LeNet (LeCun et al., 1998) and ResNet18 (He et al., 2016) as backbone for MNIST and CIFAR10, respectively.
Here, we investigate two different FL settings: IID and Non-IID , where the overall label distribution across clients is the same in the IID setting, whereas clients have different overall label distributions in the non-IID setting.
Specially, for Non-IID setting, we impose class-prior shift following:
• The classes are divided into the majority and minority classes, where the fraction of each minority class is less than 0 .
08 while the fraction of each majority class is in the range [0 . 15 , 0 . 25] .
• The training data for every client are sampled from 2 majority classes and 8 minority classes.
• The test data are uniformly sampled from all classes.
Baselines: We compare our method with the following baselines: (1) Federated Pseudo labeling (FedPL)
(Diao et al., 2021):
At each client of a FedAvg system, we choose the label with the largest class prior in a U set as the pseudo labels for all the data in this set.
During local training, the pseudo labels are boosted by Mixup (Zhang et al., 2017) high-conﬁdence data and low-conﬁdence data, where the conﬁdence is given by model predictions.
(2) Federated LLP (FedLLP) (Dulac-Arnold et al., 2019):
We update each client model by minimizing the distance between ground truth label proportion and the predicted label proportion, and aggregate them using FedAvg; (3) FedLLP with virtual adversarial training (FedLLP-VAT)
(Tsai & Lin, 2020):
We add a consistency term to FedLLP as regularization for optimizing client model in FedAvg.
(4) FedAvg with 10% labeled data (FedAvg 10%): Followng he classical unsupervised classiﬁcation literature (Chen et al., 2020), we compare with supervised training using FedAvg on only 10% of total data samples with true labels.
The detailed baseline settings are depicted in Appendix D.
Results and Analysis: Table 1 shows the results of our method and baselines on two benchmark datasets under IID and Non-IID settings.
The results are reported in mean error (std) format over clients with three independent runs.
Given the limited number of total data available in the benchmark datasets, we compare our method with baselines in an FL system with ﬁve clients and vary the number of sets in each client M ∈ { 10 , 20 , 40 } .
In this regard, each set contains a sufﬁcient number of data samples to train the model.
Note that our proposed FedUL method outperforms other baseline methods by a large margin under both Non-IID and IID settings.
Moreover, we have the following observations: 1) When varying the number of sets M ∈ { 10 , 20 , 40 } , FedUL not only achieves lower classiﬁcation errors but also shows more stable performance compared to baselines; 2) As we use FedAvg as the default aggregation method for all the strategies, the performances of FedUL, FedPL, and FedAvg all degrade under the
[Figure]
inter client Non-IID setting, as expected.
The error rates of FedLLP and FedLLP-VAT are high and unstable, which demonstrates that the EPRM based methods are inferior as discussed in Section 1.
A BLATION S TUDIES
In the following, we present a comprehensive investigation on the ablation studies of the proposed
FedUL method on MNIST benchmark.
Robustness with Noisy Priors: To investigate the robustness of our method, we further apply
FedUL in a noisy environment.
Speciﬁcally, the class priors π m,kc are perturbed by a random noise γ .
In this case, the noisy class prior (cid:101) π m,kc = π m,kc · ((2 (cid:15) − 1) γ + 1) , where γ uniform randomly take values in [0 , 1] and (cid:15) ∈ { 0 , 0 . 2 , 0 . 4 , 0 . 8 , 1 . 6 } , so that the method would treat noisy (cid:101) π m,kc ∈
[ π m,kc (1 − γ ) , π m,kc (1 + γ )]
as the true π m,kc during the whole learning process.
The experimental setup is the same as before except for the replacement of π m,kc .
Note that we tailor the noisy π m,kc to [0 , 1] if it surpasses the range.
Here we conduct experiments with ﬁve clients and ten sets in every client.
From Figure 3(a), we can see that FedUL performs stably even with very challenging noise perturbation (the noise ratio (cid:15) > 0 ).
Effects of the Number of Accessible Non-IID Data: The purpose of FL is utilizing more data to empower model performance.
While how unlabeled data, especially unlabeled Non-IID data, can contribute to FL model performance needs investigation.
To this end, we analyze the effects of the number of accessible Non-IID data.
We set each client with n c = 500 samples ∀ c ∈
[ C ] and conduct experiments by changing the amount of accessible Non-IID data by increasing the number of clients
C ∈ { 10 , 20 , 30 , 40 , 50 } .
As shown in Figure 3(b), the model performance improves by a large margin as the number of accessible Non-IID data increases.
Analysis of Local Updating Epochs: The frequency of aggregation may sometimes inﬂuence the model performance in an FL system.
We would like to explore if FedUL is stable when the local clients are updated with different local epochs E .
In Figure 3(c), we explore E ∈ { 1 , 2 , 4 , 8 , 16 } with ﬁxed B = 128 and C = 5 .
The results demonstrate that with different E , the performance of
FedUL are reasonably stable.
Effects of Batch Size: The training batch size is always an important factor in machine learning tasks, especially in an FL system.
Hence here, we study how batch size affects our model performance.
Speciﬁcally, we train the model using three different batch sizes B ∈ { 16 , 64 , 128 , 512 , ∞} with ﬁxed E = 1 and C = 5 .
Here B = ∞ stands for loading all of the samples from the training data into a single batch.
The results in Figure 3(d) indicate that FedUL can effectively work under a wide range of batch sizes.
4. C ASE S TUDY ON R EAL - WORLD P ROBLEM
To demonstrate the feasibility of our proposed FedUL scheme to the real problem, we test FedUL on a medical application for the diagnosis of brain disease.
In the rest of this section, we introduce the problem formulation, data and setting, and then discuss the results.
Real Problem Formulation: For a real-world scenario, we consider a clinical setting for disease classiﬁcation in C different healthcare systems.
Suppose each healthcare system has M hospitals located in different regions, each associated with a unique class priors due to population diversities. [Figure] [Table] [Figure]
[Table]
[Figure]
(Flanders et al., 2020).
Medical data are not shareable between the healthcare systems for privacy.
We assume patient-wise diagnosis information is unavailable, but each hospital provides the prevalence of these patients (a.k.a.
class priors of the disease classes).
Dataset and Settings: We use the RSNA Intracranial Hemorrhage Detection dataset (Flanders et al., 2020)
(Fig. 4) to simulate this real-world scenario.
This dataset contains large-scale Magnetic
Response Image (MRI) data from several hospitals.
There are 106 k samples in the dataset belonging to six classes, including ﬁve types of intracranial hemorrhage (Epidural, Intraparenchymal, Intraventricular, Subarachnoid, and Subdural) and healthy control type.
We model an FL system with ﬁve healthcare systems (a.k.a clients).
Each healthcare system contains 21 .
2 k data samples.
To show the effect on the number of samples in each set, we vary the number of hospitals (a.k.a sets) in each system M = 12 , 24 , and 48 .
Here we generate each π m,kc for the k -th class in m -th set at c -th client from range [0 . 1 , 0 . 9] and check that the generated class priors are not all identical.
For model training, we use the cross-entropy loss and Adam (Kingma & Ba, 2015) optimizer with a learning rate of 1 e − 4 and train 100 rounds.
We use ResNet18 (He et al., 2016) as backbone for experiments.
Results and Analysis: As indicated in our proof of concept (Table 1), FedPL consistently outperforms the other two LLP baselines, i.e., FedLLP and FedLLP-VAT.
Here we focus on comparing
FedUL with FedPL.
We also include the supervised baseline FedAvg with 10% data.
We report the results of each hospital in mean error (std) format over three runs in Table 2.
We can see that our method FedUL consistently outperforms FedPL and FedAvg 10% for all the hospitals and under different site number settings.
The superiority of FedUL in this challenging real-world FL system further indicates the effectiveness and robustness of our algorithm.
These results are inspiring and bring the hope of deploying FedUL to the healthcare ﬁeld, where data are often very costly to be labeled, heterogeneous, and distributed.
C ONCLUSION
In this paper, we propose FedUL, a novel FL algorithm using only U data.
FedUL works as a wrapper that transforms the client data and modify the client model to form a surrogate supervised learning task, which can be easily solved using existing FL paradigms.
Theoretically, we prove that the optimal model learned by supervised FL can be recovered from the FedUL training scheme under mild conditions.
Through extensive experiments on benchmark and real datasets, we show FedUL can signiﬁcantly overperform the baseline methods under different settings.
As a wrapper, we expect
FedUL can be further improved by incorporating advanced FL aggregation or optimization schemes for the non-IID setting (Wang et al., 2020;
Li et al., 2020; 2021; Karimireddy et al., 2020).
E THICS S TATEMENT
Collecting sample-wise labels for training can be costly and may violate data privacy.
Considering data in the real world are often collected distributed, we provide a pioneer and promising solution to perform multi-class classiﬁcation on distributed U data in the FL system.
Our method FedUL can be widely applied to data-sensitive ﬁelds, such as healthcare, politics, and ﬁnance, where the sample-wise label is not available, but populational prevalence can be accessed.
With FedUL, we hope the future FL can enjoy signiﬁcant labeling cost reduction and improve its privacy.
Therefore, we believe our proposed method and its potential application have a positive impact on society.
The real datasets used in this work are publicly available with appropriate prepossessing to de-identify patients’ identities.
We provide proper references to the datasets, libraries, and tools used in our study.
This study does not have any legal compliance, conﬂict of interest or research integrity issues.
R EPRODUCIBILITY S TATEMENT
We provide the codes to reproduce the main experimental results via the anonymous download link indicated in Appendix.
We further provide experimental details and additional information of the datasets in the supplementary materials.
Supplementary Material
Roadmap
The appendix is organized as follows.
We point to the shareable code and preprocessed data in Section A .
We introduce the notations used in our paper in Section B. Our proofs are presented in Section C.
We provide the detailed experimental settings in Section D. More results on the benchmark dataset are stated in Section E
A D ATA AND C ODE A VAILABILITY
The trained models, guidelines for data downloading and preparation, code for training, validation, testing of all the methods, readme ﬁle, libraries requirement, and environment setup for our benchmark experiments are available at the anonymous link : https://drive.google.com/drive/folders/1kcMO0_XoxoPFhv1ZgcZ6bV_HTzZo-zPs?usp=sharing.
Duetosizelimitation
[Table]
wecouldnotsharethepreprocessedlargescalereal-worlddatasets.
Weare veryhappytoreleaseallmaterialsintheﬁnalversion.
B N OTATION T ABLE
[Table]
m
[Table]
m
[Table]
[ M c ]
[Table]
c
[Table]
th
[Table]
unlabeled
[Table]
surrogate
[Table]
C P ROOFS
In this appendix, we prove all theorems.
C. P ROOF OF T HEOREM
By the Bayes’ rule, ∀ m ∈
[ M ] we have:
[Equation]
By transforming (10) to the matrix form, we complete the proof.
C. P ROOF OF L EMMA
Given our data generation process in Section 3.1, we have that M c ≥ k and Π c ∈ R M c
×
K is a full column rank matrix.
With D π c and D − 1 π being non-degenerate, we obtain that Q c ( η ( x )) presents an injective mapping from η ( x ) to η c ( x ) .
C. P ROOF OF P ROPOSITION
The assumptions (A1) - (A3) are listed as follows.
(A1)
Bounded gradient dissimilarity: ∃ constants G ≥ 0 and B ≥ 1 s.t.
C (cid:88)
C c =1 (cid:107)∇ J
[Equation]
g . (A2) β -smooth function J c satisﬁes
[Equation]
(A3) Unbiased stochastic gradient with bounded variance:
[Equation] [Equation]
, where (cid:98) J c ( g ; U c )
= 1 n c (cid:80) n c i =1 (cid:96) ( g ( x i ) , y i ) .
Note that this proposition statement is w.r.t.
the convergence of model g learned from the surrogate supervised learning task.
The proof follows directly from Theorem V in Karimireddy et al.
C. P ROOF OF T HEOREM
When a proper loss (Reid & Williamson, 2010) is used for (cid:96) , the mapping of g ∗∗ is the unique minimizer of J ( g ) .
We show an example of the a widely used proper loss function: the cross-entropy loss (De Boer et al., 2005) deﬁned as
[Equation]
We can see that the cross-entropy loss is non-negative by its deﬁnition.
Therefore, minimizing J ( g ) can be obtained by minimizing the conditional risk E p ( y | x ) [ (cid:96) ( g ( x ) , y ) | x ] for every x ∈ X .
So we are now optimizing
[Equation]
By using the Lagrange multiplier method (Bertsekas, 1997), we have
[Equation]
The derivative of L with respect to g is
[Equation]
By setting this derivative to 0 we obtain
[Equation]
Since g ∈ ∆ M − 1 is the M -dimensional simplex, we have (cid:80) Mm =1 ( g ∗∗ ( x )) m
= 1 and (cid:80)
Mm =1 p ( y = j | x ) = 1 .
Then
[Equation]
Therefore we obtain λ = − 1 and ( g ∗∗ ( x )) m
= p
y = m | x ) =
( η ( x )) m , ∀ m = 1 , . .
, M and ∀ x ∈ X , which is equivalent to g ∗∗ ( x ) =
¯ η ( x ) .
Similarly, we obtain f (cid:63) ( x ) =
η ( x ) .
Given Proposition 3 and the assumption that the model used for g is sufﬁciently ﬂexible, we obtain the classiﬁcation model learned with FedUL by Algorithm 1 g ∗ =
g ∗∗ .
By our model construction, g is built from Q c ( f ) .
We have f ∗ =
f ∗∗ = argmin
f J ( f ) which is induced by the convergence of g .
Given Theorem 1 and Lemma 2, we obtain that g ∗∗ ( x ) =
¯ η ( x ) if and only if f ∗∗ ( x ) =
η ( x ) .
So we obtain f ∗ =
f (cid:63) .
D E XPERIMENTAL DETAILS
We illustrate our model architectures and training details of the benchmark and real-world experiments in this section.
D. M ODEL A RCHITECTURE
For our benchmark experiment on MNIST, we use a six-layer convolutional neural network.
The network details are listed in Table D.2.
For our benchmark experiment on CIFAR10 and real-world experiment, we use ResNet18 as our backbone.
The network details are listed in Table D.3.
[Table]
D. T RAINING D ETAILS
Benchmark Experiments:
We implement all the methods by PyTorch and conducted all the experiments on an NVIDIA TITAN X GPU.
Please note for all experiments, we split 20% training data for validation and select model.
During training process, we use Adam optimizer with learning rate 1 e − 4 and cross-entropy loss, we set batch size to 128 and training rounds to 100 .
To avoid overﬁtting, we use L1 regularization.
The detailed weight of L1 regularization used in benchmark experiments is shown in Table D.4.
The detailed number of samples in each client for MNIST and CIFAR10 under IID and Non-IID setting is given in Table D.5 and Table D.6, respectively.
We describe details of the baseline methods and corresponding hyper-parameters as follows.
• FedPL (Diao et al., 2021): this method is based on empirical classiﬁcation risk minimization.
The learning objective is
[Equation]
where
[Equation]
is the "ﬁx" loss, where ( x + b , y + b ) represents for a sample x
+ b with high conﬁdence pseudo label y + b
and
[Equation]
is the "mix" loss, in which
[Equation]
where a is the mixup hyper- parameter and x − b y − b ) represents for a sample x −
b with low conﬁdence
pseudo
label
y
− b Following the default implementation in (Diao et al., 2021),
Layer
Details
Conv2D(3, 64, 7, 2, 3), BN(64), ReLU Conv2D(64, 64, 3, 1, 1), BN(64), ReLU
Conv2D(64, 64, 3, 1, 1), BN(64)
Conv2D(64, 64, 3, 1, 1), BN(64), ReLU
Conv2D(64, 64, 3, 1, 1), BN(64)
Conv2D(64, 128, 3, 2, 1), BN(128), ReLU
Conv2D(128, 128, 3, 1, 1), BN(64)
Conv2D(64, 128, 1, 2, 0), BN(128)
Conv2D(128, 128, 3, 1, 1), BN(128), ReLU
Conv2D(128, 128, 3, 1, 1), BN(64)
Conv2D(128, 256, 3, 2, 1), BN(128), ReLU
Conv2D(256, 256, 3, 1, 1), BN(64)
Conv2D(128, 256, 1, 2, 0), BN(128)
Conv2D(256, 256, 3, 1, 1), BN(128), ReLU
Conv2D(256, 256, 3, 1, 1), BN(64)
Conv2D(256, 512, 3, 2, 1), BN(512), ReLU
Conv2D(512, 512, 3, 1, 1), BN(512)
Conv2D(256, 512, 1, 2, 0), BN(512)
Conv2D(512, 512, 3, 1, 1), BN(512), ReLU Conv2D(512, 512, 3, 1, 1), BN(512)
AvgPool2D
[Table]
[Table]
we set hyper-parameter a = 0 .
Moreover, we set the weight of "mix" loss λ = 0 .
and the conﬁdence threshold τ = 0 .
• FedLLP (Dulac-Arnold et al., 2019): this baseline method is based on empirical proportion risk minimization.
The learning objective is
[Equation]
is the aggregated proportion risk, π m c and (cid:98)
π
m c are the true and predicted label proportions for the m -th
U set U mc in c -th client, (cid:98)
π mc
= 1 M c (cid:80) x
∼U mc f ( x ) . For each client c , d prop is deﬁned as
[Equation]
[Table] [Table] • FedLLP-VAT (Tsai & Lin, 2020): this method is based on empirical proportion risk minimization, integrated with a consistency regularization term.
The learning objective is
[Equation]
is the aggregated proportion risk with consistency regularization, π mc and (cid:98)
π mc are the true and predicted label proportions for the m -th
U set U mc in c -th client, d prop is a distance function, and
[Equation]
is the consistency loss, d cons is a distance function, (cid:98) x is a perturbed input from the original one x .
Following the default implementation in their paper Tsai & Lin, we set the hyper-parameter the weight of consistency loss α = 0 .
05 and the perturbation weight µ = 6 .
for FedLLP-VAT on CIFAR10 benchmark experiment.
For MNIST benchmark, we set the hyper-parameter the weight of consistency loss α = 5 e − 4 and the perturbation weight µ = 1 e − 2 .
Real-world Experiments:
We perform a study case on an important clinical task — intracranial hemorrhage diagnosis.
Intracranial hemorrhage, described as bleeding that occurs inside the cranium, is a severe health problem.
The distribution of intracranial hemorrhage depends on the patient’s ethnicity, such as age (Camacho et al., 2015).
Thus, considering age distributions vary in different regions, this dataset and problem naturally ﬁts into our setting.
We use the RSNA Intracranial Hemorrhage Detection dataset (Flanders et al., 2020)
(Fig. 4), which contains large-scale Magnetic Response Image (MRI) data from several hospitals.
There are 106 k samples in the dataset belonging to six classes, including ﬁve types of intracranial hemorrhage (Epidural, Intraparenchymal, Intraventricular, Subarachnoid, and Subdural) and healthy control type.
In our experiments, we simulate a real-world scenario with ﬁve healthcare systems (a.k.a.
clients) and assume the number of hospitals (a.k.a sets) in each healthcare systems varies M = 12 , 24 , and 48 . [Table]
At ﬁrst, we randomly split the whole dataset for ﬁve healthcare systems equally, so each healthcare systems is assigned 21 .
2 k samples.
Then we generate π mc for m -th set in c -th client from range [0 . 1 , 0 . 9] for all the hospitals with U data only.
We check that the generated class priors are not all identical.
Finally, we load samples for each the hospitals following the generated π .
Detailed dataset statistics is given in Table D.7.
During training process, we use Adam optimizer with learning rate 1 e − 4 and cross-entropy loss, we set batch size to 128 and training rounds to 100 .
To avoid overﬁtting, we use L1 regularization.
We set the weight of L1 regularization as 2 e − 6 , 8 e − 7 and 4 e − 7 for 10, 20 and 40 sets, respectively.
In real-world experiments, we use the same hyper-parameters as benchmark experiments for all methods except FedLLP-VAT.
For FedLLP-VAT, we set the hyper-parameter the weight of consistency loss α = 0 .
05 and the perturbation weight µ = 6 . 0 .
Training Efﬁciency:
We further evaluate the training efﬁciency of our method with comparison to baseline methods.
The detailed one epoch training time (mean) and forward FLOPs are given in Table D.8.
The computation time is evaluated on a PC with one Nvidia TITAN X GPU.
We can see that our method slightly increase the forward FLOPs and can achieve higher training efﬁciency compared with FedPL and FedLLP-VAT.
[Table] E M ORE E XPERIMENTAL R ESULTS ON B ENCHMARK D ATASET MNIST E. D ETAILED S TATISTICS OF F IGURE 3( A )
We show detailed statistics of ﬁgure 3(a) in Table E.9.
With different noisy class priors, even challenging (cid:15) = 0 .
8 and (cid:15) = 1 . 6 , our method still achieves low error rate, which strongly verify the robustness and effectiveness of our method.
[Table]
E. D ETAILED S TATISTICS OF F IGURE 3( B )
We show detailed statistics of ﬁgure 3(b) in Table E.10.
With more Non-IID data can be accessed, our method performs better, which validate that more unlabeled Non-IID data can be helpful for FL.
[Table]
E. D IFFERENT C OMBINATIONS OF E AND B
We compare our method with different combinations of E and B on benchmark dataset MNIST with the Non-IID setting.
The classiﬁcation error rate (mean and std) are consistently small as presented (cf.
Table E.11), which indicates our method can work under a wide range combinations of E and B [Table]
[Table]
Here we conduct experiments on benchmark datasets under IID and Non-IID setting with 5 clients and different sets in each client (10, 20, 30, 40, 50). The experimental results are shown in Table
F.12, which further indicates the effectiveness of our method even under this challenging setting. [Table] G B ENCHMARK E XPERIMENTS WITH M ORE C LIENTS To study the inﬂuence of the number of clients
[Table]
here we further conduct benchmark experiments on both MNIST and CIFAR10 with 10 clients.
The detailed results are shown in Table G.13, in which we has the similar observations as in Table 1.
[Table]