F ED MT: F EDERATED L EARNING WITH M IXED - TYPE L ABELS
Anonymous authors
Paper under double-blind review
A BSTRACT
In federated learning (FL), classiﬁers (e.g., deep networks) are trained on datasets from multiple centers without exchanging data across them, and thus improves sample efﬁciency .
In the classical setting of FL, the same labeling criterion is usually employed across all centers being involved in training.
This constraint greatly limits the applicability of FL.
For example, standards used for disease diagnosis are more likely to be different across clinical centers, which mismatches the classical
FL setting.
In this paper, we consider an important yet under-explored setting of
FL, namely FL with mixed-type labels where different labeling criteria can be employed by various centers, leading to inter-center label space differences and challenging existing FL methods designed for the classical setting.
To effectively and efﬁciently train models with mixed-type labels, we propose a theory-guided and model-agnostic approach that can make use of the underlying correspondence between those label spaces and can be easily combined with various FL methods such as FedAvg.
We present convergence analysis based on over-parameterized
ReLU networks.
We show that the proposed method can achieve linear convergence in label projection, and demonstrate the impact of the parameters of our new setting on the convergence rate.
The proposed method is evaluated and the theoretical ﬁndings are validated on benchmark and medical datasets.
I NTRODUCTION
Federated learning (FL) enables centers to jointly learn a model while keeping data at each center.
It avoids the centralization of data which is restricted by regulations such as CCPA (Legislature, 2018),
HIPAA (Act, 1996), and GDPR (Voigt et al., 2018) and has gained popularity in various applications.
The widely used FL methods, such as FedAvg (McMahan et al., 2017), FedAdam (Reddi et al., 2020), and others use iterative optimization algorithms to achieve jointly model training across centers.
At each round, local center performs stochastic gradient descent (SGD) for several steps then centers communicate their current model weight to a central server to be aggregated.
When training a classiﬁer in the classical FL setting, the datasets across all centers are annotated with the same labeling criterion.
However, in real applications such as healthcare, standards for disease diagnosis may be different across clinical centers due to varying levels of expertise or technology available at different sites.
For example, when diagnosing ADHD with brain imaging, the labels are usually acquired over a long period of behavior studies.
Different centers may follow different diagnosis and statistical manuals (McKeown et al., 2015) and it is difﬁcult to ask centers to relabel data using a uniﬁed criterion as some behavior studies cannot be repeated.
This leads to different label spaces across centers.
In addition, the center with the most complex labeling criterion, whose label space is desired for future prediction, typically only has limited labeled samples due to labeling difﬁculty or cost.
In this paper, we aim to answer the following important question:
With limited samples from the desired label space, how to leverage the commonly used FL pipeline ( e.g., FedAvg) and data from other centers in different label spaces to jointly learn an FL model in the desired label space, without additional feature exchanging and data relabeling?
Problem Setting: We study an FL problem for a given classiﬁcation task.
Each center has one labeling criterion, and the criteria across centers can be different.
Samples do not overlap across centers.
As shown in Fig.
1, ﬁrst, label spaces are not necessarily nested.
One class from the desired
[Figure]
[Figure] [Figure]
$
[Figure]
(%) ' #$ (( %& ",# " ) ' #$ (* %& !",# # ! )
# ' % = ,-./01(# ( , # # " ) ' #$ (",(# " ) # # # (%) '
#$ (!",*# # )
FedMT(
[Figure]
) FedMT(
[Figure]
The
[Figure]
label space !
= {! " ,!
% ,⋯,! ! } !
* $! = {$! " , $! % ,⋯, $! ' }
[Figure]
space may overlaps with different classes in another space and vice versa ( e.g., , disease diagnoses often exhibit imperfect agreement).
Second, following the motivated healthcare example, we assume limited amount of labeled data ( < 5% ) in the desired label space is available 1 .
Moreover, for the ease of experiment design, we consider the case where these data are stored in one ‘ specialized center ,’ and this center can be treated as the server to coordinate FL but still perform local model updating like the other clients, i.e., the centers with the other labeling criteria.
All the centers jointly train an
FL model following the standard FL training protocol as shown in Fig.
1 (b).
Prior methods for dealing with different labels spaces include personalized FL (Collins et al., 2021), but they fail to leverage the correspondence across different label spaces.
Transfer learning (Yang et al., 2019) which pretrains a model on one space and ﬁnetunes the pretrained model on other spaces can be an alternative solution in FL, but sub-optimal pretraining may lead to negative transfer (Chen et al., 2019).
Therefore, to address the limitation of above methods, we want to a) simultaneously leverage different types of labels and their correspondence and b) learn FL model end-to-end.
To the best of our knowledge, other possible centralized methods meet our needs a) and b) are restricted to coarse to ﬁne label spaces that have hierarchical structures (Touvron et al., 2021; Chen et al., 2021a), which does not hold for the general problem of our interest, or require pulling all data features together for similarity comparison using more sophisticated training strategies (Hu et al., 2022).
These methods cannot be simply extended to widely used FL methods ( e.g., FedAvg) and require feature sharing across centers which increases privacy risks.
To address the above limitations, we propose a method called FedMT , which is a versatile strategy that can be easily combined with various FL pipelines such as FedAvg.
Speciﬁcally, we use models with the same architecture whose output dimension is the number of classes in the desired label space across all centers.
To use client data from the other label space for supervision, we align two spaces either with label (probability) projection that projects the label (class scores) to the other space.
We further show that our methods has the bonus to handle label noise.
Contributions: Our contributions are three folds.
Methodologically , we propose a novel FL method,
FedMT , which is a computationally efﬁcient and versatile solution Theoretically , we present the convergence of FedMT in FL with over-parameterized ReLU neural networks, and explore the impact of amount of data from desired label space and different noise levels; Empirically , we demonstrate the superior results in this challenging setting over prior art with extensive experiments on benchmark and medical datasets.
R ELATED W ORK
Federated learning FL is emerging as a learning paradigm for distributed clients that bypasses data sharing to train local models collaboratively.
To aggregate model parameters, FedAvg (McMahan et al., 2017) is the most widely used approach in FL.
Variants of FedAvg have been proposed to improve optimization (Reddi et al., 2020; Rothchild et al., 2020) and for non-iid data (Li et al., 2021; 2020a; Karimireddy et al., 2020).
Recently FL methods have been studied in semi-supervised learning (Jeong et al., 2021; Bdair et al., 2021), weakly supervised learning (Lu et al., 2021), and positive label only learning (Yu et al., 2020).
Theoretical studies of FL have not yet directly addressed neural network algorithms nor accounted for the inﬂuence of individual data samples (Karimireddy et al., 2020; Li et al., 2019; Khaled et al., 2020).
Recently, a theoretical framework for FL on fully supervised neural network regression using FedAvg was proposed in Huang et al.
To the best of our knowledge, this is the ﬁrst work to investigate FL’s theory on neural network classiﬁcation under mixed-type labeled data.
Neural Tangent Kernel (NTK) NTK is an essential tool to study the learnability of neural networks.
NTK was ﬁrst studied by Jacot et al.
(2018) showing the equivalence between training an inﬁnitely wide neural network with gradient descent and kernel regression.
NTK theory has been extended to convolutional neural networks (CNNs) (Arora et al., 2019), graph neural networks (GNNs) (Du et al., 2019; Jiang et al., 2019), recurrent neural networks (RNNs) (Alemohammad et al., 2020).
NTK for
FL was recently studied by Huang et al.
, in the supervised FL regression problem using mean squared error loss.
M ETHODS 3. P RELIMINARIES : C LASSICAL FL
We start by reviewing the classical FL approach, FedAvg, with full supervision (McMahan et al., 2017) for classiﬁcation with the same labeling criterion.
We consider a K -class classiﬁcation problem with the feature space X and the label space Y =
[ K ] .
Let x 2 X
and y 2 Y be the input and output random variables following an underlying joint distribution with density p ( x , y ) .
Let f :
X !
R K be a K -class classiﬁer such that  ( f k ( x ))
=
p ( y = k | x ) , where f k ( x ) is the k -th element of f ( x ) and  ( f k ) = exp( f k ) /
P
Kk
=1 exp( f k 0 ) is the softmax function.
Then the predicted label can be obtained via y pred = arg max k
[ K ] f k ( x ) .
In the classical FL setup, each client c 2 [ C ] has access to a labeled training set D c = { ( x ci , y ci ) } N c
i = of size N c and learns its local model f c by minimizing the following empirical risk:
[Equation]
where ` is the cross-entropy loss , i.e., ` CE ( f c ( x ) , y )
=
P
[Equation]
and ( · ) is the indicator function.
The goal of classical FL is that C clients collaboratively train a global classiﬁcation model f that generalizes well with respect to p ( x , y ) , without sharing their local data D c .
The problem can be formalized as minimizing the aggregated risk:
R ( f ) = 1 C
P
[Equation]
R a c ( f ; D c ) .
FedAvg employs a server to coordinate the iterative distributed training through model parameter averaging as described in Algorithm 1.
3. P ROBLEM F ORMULATION
In contrast to the classical supervised FL classiﬁcation, where all the centers share the same label space, we consider the case where the label spaces across centers may be different.
For the ease of presentation, we assume there are two label spaces, namely Y =
{ Y k } K k =1 and e
Y = { e
Y j } Jj =1 where
K > J .
For the ‘specialized center’ with the desired label space, we denote it as a server and its dataset as D s = { ( x si , y si ) : i 2 [ nK ] } , where y si 2 Y .
We also have in total N labeled data with a different criterion that are stored across C clients.
For c 2 [ C ] , we denote by S c the indices of data from the other label space D c
= { ( x ci , y ci ) :
i 2 S c } on c -th client, where y cie
Y .
Let N c
= | S c | and we have N =
P c 2 [ C ] N c .
Our objective is to train a global classiﬁer in the desired label space f : X 7!
Y using data in the system from different label spaces.
3. P ROPOSED M ETHOD
Motivation Learning under different label spaces can be formulated as a corrupted label learning problem (Van Rooyen & Williamson, 2017; Patrini et al., 2017), where label space with less classes e
Y is treated as the corrupted observation of a true underlying label space Y with more classes, i.e., our desired label space.
Our method is based on loss correction (Van Rooyen & Williamson, 2017;
Patrini et al., 2017), which is a well-established corrupted label learning approach in the centralized domain with statistically consistency guarantees.
Let T : Y 7!
e
Y be a linear transformation, whose pseudo inverse is T  1 .
Under proper assumptions (see Theorem 3.1) and a given function f : X 7!
Y , Patrini et al.
(2017) showed two ways to perform loss correction that T  1 can pull back from functions of corrupted labels to functions of true labels and T can transfer functions of true labels to those of corrupted labels.
Mathematically, this can be written as:
Theorem 3.1 (Loss correction (informal) (Patrini et al., 2017; Van Rooyen & Williamson, 2017)) .
Given a non-singular linear mapping matrix T and a proper loss ` , one can achieve the same minimizer of the original loss under the true label distribution:
[Equation]
E
[Equation]
, y
[Equation]
e
Y ` ( y , T f ) and E x , y 2 Y ` ( y , f ) =
E x , ye
Y T  1 ` ( y , f ) .
In addition, loss correction methods are model agnostic and can be generalized to many loss functions, therefore provide a versatile framework for learning from corrupted labels.
Since our goal is to jointly learn a classiﬁer f : X !
R K for clients and server, the overall loss can be written as:
[Equation]
where b
R s ( f ) and b
R c ( f ) are respectively the empirical risk based on the server’s data and the clients’ data from a different label space.
To minimize the overall loss in (2), we need to align the predictions or losses of both types of labels to the underlying true label space.
Considering the speciﬁc form of cross-entropy (1), we adapt the ﬁndings of Patrini et al.
(2017); Van Rooyen & Williamson (2017) to our FL setting.
For all the clients in FL l 2 [ C ] , we propose leveraging the following two kinds of projections to both server and clients as:
Probability projection :
[Equation]
Label projection :
[Equation]
For clients, we denote T =
Q 2 [0 , 1] J ⇥ K , whose j -th row denotes the mixing weights of classes in the desired label space for j -th class in the other label space.
As both probability projection and label projection are performed locally, the overall loss (2) can be optimized using general FL strategies, such as FedAvg (McMahan et al., 2017) used in this work, or other variants like Li et al.
(2020a; 2021); Karimireddy et al.
Note our method can be easily extended to more than one centers in the desired label space as shown in Appendix D.5.
Extension to Noisy Labels Another advantage of our method is that it has the ﬂexibility to extend to the case where there are noisy observations, that is the label is mislabeled to a wrong one in the same label space.
Let us explain with the case where the server has noisy labels.
Speciﬁcally, on the server, we let T =
T 2 [0 , 1] K ⇥ K be a matrix whose ( i, j ) -th element denotes p ( e y s =
Y j | y s = Y i ) where e y s is the observed noisy label and y s is the true label.
The detailed FedMT algorithm is described in Algorithm 2, T
= I K ⇥ K under noisy label case and T
= I K ⇥ K for noise-free setting.
In this algorithm, it is worth noting that FedMT is easy to implement as it only slightly modiﬁes FedAvg as highlighted in blue.
han et al.,
Server Input: initial f , aggregation step-size ⌘ agg , and global communication round
Client Input: local model f c , local dataset D c , SGD step-size ⌘ sgd , and local updating iterations t (for c 2 [ C ] ) 1: For r = 1 !
R rounds, we run A on each client and B iteratively .
2: procedure A .
M ODEL U PDATE ( r ) 3: f c f .
Receive updated model from PROC .
B 4: for ⌧ = 1 !
t do 5: f c f c  ⌘ sgd · r b
R a c ( f c ; D c ) .
Model updates via SGD 6: send f c  f to PROC .
B 7: procedure B .
M ODEL
A GG ( r ) 8: receive model updates f c  f from PROC .
A 9: f f  ⌘ agg ·
P
Cc =1 ( f c  f ) 10: broadcast f to PROC .
A
Server Input: inputs of Alg 1, server model f s , small noisy dataset D s =
{ x s , y s } where y s 2 Y , projection matrix T
Client Input: inputs of Alg 1 but D c = { x c , y c } where y ce
Y , and projection matrices Q 1:
For r = 1 !
R rounds, we run A on each client and B iteratively .
2: procedure A .M ODEL U PDATE ( r ) 3: f c f .
Receive updated model from PROC .
B 4: for ⌧ = 1 !
t do 5: if probability projection then 6: f c f c  ⌘ sgd · r b
R c ( Qf c ; y c ) 7: f s f s  ⌘ sgd · r b
R s ( T f s ; y s ) 8: else if label projection then 9: f c f c  ⌘ sgd · r b
R c ( f c ; Q  1 y c ) 10: f s f s  ⌘ sgd · r b
R s ( f s ; T  1 y s ) 11: send f l  f to PROC .
B for l 2 { [ C ] , s } 12: procedure B .
M ODEL
A GG ( r ) 13: receive model updates from PROC .
A 14: f f  ⌘ agg ·
P l 2 { [ C ] ,s } ( f l  f ) 15: broadcast f to PROC .
A
3. T HEORETICAL A NALYSIS
Although label projection-based loss correction methods have shown to have good properties under centralized setting (see Theorem 3.1 Patrini et al.
(2017); Van Rooyen & Williamson (2017)), their theoretical convergence under FL have not been explored.
Hence, we focus on the theoretical analysis of label projection in (4), given it has been shown to have better theoretical guarantees than (3) in a previous study (see Theorem 3.1 (Patrini et al., 2017)).
Further, we are interested in investigating the impact of various parameters on theoretical convergence.
In this section, we establish a novel theoretical analysis of FedMT using NTK (Arora et al., 2019; Lee et al., 2019; Huang et al., 2021).
NTK setup Our theoretical results are studied under an over-parameterized one-hidden layer neural network (NN).
Let f : X !
R K be the output of the NN whose k th element is
[Equation]
where  ( z ) = max { z, 0 } is the ReLU activation function and u = [ u 1 , u 2 , .
. . , u M ] 2 R d ⇥ M .
Deﬁnition 3.2 (Initialization) .
We initialize u 2 R d ⇥ M and a km as follows.
For each m 2 [ M ] , u m is sampled from N (0 , I ) .
For each k 2 [ K ] and m 2 [ M ] , a km is sampled from {  1 , +1 } uniformly at random and is not trainable.
In the r -th global round, the server broadcasts the global model weight u
m ( r ) to every client.
Each client c then starts from u m,c (0 , r ) =
u ( r ) and takes t local gradient steps via gradient descent with step size ⌘ local [Equation]
[Equation]
where u m,c ( ⌧ , r ) is the value of u m on c th client at step ⌧ in r -th global round.
Then the client sends  u m,c ( r ) =
u m,c ( t, r )  u m,c (0 , r ) to server and server computes a new u m,c ( r + 1) based on the average of all  u m,c ( r ) via
[Equation]
NTK analysis results Let g ( u , x ) =
( f ( u , x )) and y ci be the multi-hot vector representation of y ci and y si be the one-hot vector representation of y si in R K .
Let Q be the linear mapping from Y to e
Y and T be the same as before.
Following Neural Tangent Kernel (NTK) analysis (Lee et al., 2019;
Arora et al., 2019), we consider the mean squared error loss 3 respectively on the client and server
[Equation]
Lemma 3.3.
Let the notation be the same as before, the NTK kernel G ( r ) based on our proposed novel loss (5) is a block matrix with K row partitions and K column partitions.
The block matrix in the l -th row and m -th column has the following form
[Equation]
CCCCCCA
[Equation]
for l 2 [ K ] and m 2 [ K ] where each sub-block matrix has the following form:
[Equation]
We show detailed derivation of Lemma 3.3.
Note that as J < K , Q 2 R J ⇥ K is not inevitable.
In practice, we compute its pseudo inverse instead.
Although looking at Q alone , the plausible property of Theorem 3.1 may not hold.
We claim that by optimizing b
R c ( f ) together with b
R s ( f ) , we beneﬁt from the gradient of labels in the desired label space, as shown in the expression of G i,jc,s ( r ) in Eq.
Following Huang et al.
(2021), we further conclude the convergence of the our proposed loss in (5).
Theorem 3.4 (Convergence) .
Let
M = ⌦
(   4 ( N + nK ) 4 log(( N + nK ) /  )) , we iid initialize u m (0) , a km as Deﬁnition 3.2.
Let  =
min ( G (0)) denote the smallest eigenvalue of G (0) .
Let  denote the condition number of G (0) .
For C clients, for any ✏ , let
[Equation]
⌘
[Equation]
O   /  t ( N + nK ) , and ⌘ agg =
O (1) , the above algorithm satisﬁes
[Equation]
with probability at least 1   .
The details of the proof are deferred to Appendix B. Note that a smaller value of ⌘ agg ⌘ local  t/ 2 C , e.g., a smaller eigenvalue  and a larger number of clients, leads to slower convergence.
It is clear that  depends on both Q and T , therefore it is important to see its connection with Q and T , we discuss the algorithm convergence under the following two special cases.
Corollary 3.5 (Convergence under various label granularity differences) .
Let k 1 , k 2 , . .
, k J be J non-negative integers such that K =
P
Jj =1 k j .
If the linear transformation from desired label space to the other label space is Q = diag ( 1 > k 1 , 1 > k 2 , . .
. , 1 > k J ) 2 [0 , 1] J ⇥ K ( i.e., the classes in the desired label space are subclasses of classes in the other label space), then the smaller the value of J , the lower the convergence rate.
Proof Sketch:
Based on the form of G ( r ) in (7), it can be seen that (Li et al., 2021)  =  min ( G (0))
 min k,c,s {  min ( G k,kc,c (0)) ,  min ( G k,ks,s (0)) }   min ( G k,kc,c (0)) .
⇢ min j k  1 j  where  0 =
min ( r u g ( u (0) , D ) r > u g ( u (0) , D )) .
Since
P j k j = K , the smaller the value of J , the smaller the value of  and hence the algorithm converges slower.
See detailed proof in Appendix B.
Corollary 3.6 (Convergence under different noise levels) .
We assume noisy labels are corrupted via
[Equation] where T ij = p ( e y s = Y j F | y s =
Y i F ) with observed noisy label
e y s and true y s is the true label, and ⇠ = 1  T ii 2 (0 , 1) for i
[ K ] denotes the noisy level.
In this speciﬁed case, the convergence of FedMT with label projection on NTK is does not depend on the noise level ⇠ .
Proof Sketch:
By using matrix inversion lemma, we ﬁnd T  1 = ( K  1) /
( K  1  K ⇠ ) I K  ⇠ / ( K  1  K ⇠ ) K > K .
Hence
P
Kk =1 T kk
=
1 for any k 0
[ K ] .
See Appendix B for details.
E XPERIMENTS
In this section, we demonstrate the effectiveness of local loss correction to jointly train a model with data from different label spaces.
Compared with alternative training strategies and prior art, our proposed method, FedMT , consistently achieves better classiﬁcation accuracy in predicting labels in the desired label space with limited amount of data from this space as demonstrated on
CIFAR100 (Krizhevsky et al., 2009) and a medical dataset.
4. B ENCHMARK E XPERIMENTS S ETUP
Dataset and setting We use the CIFAR100 (Krizhevsky et al., 2009) dataset to mimic our proposed problem setting.
We assume one center (as server to coordinate FL training) has a small amount of data (  5% of total sample size) with sub-class annotations and other centers are labeled using super-class annotations.
The sub-class space is viewed as our desired label space.
Our objective is to train a classiﬁcation model using FL to predict sub-class labels with all the centers simultaneously.
The CIFAR100 dataset consists of 50K images, associated with K = 100 sub-classes that could be further grouped into J = 20 super-classes.
To study the effect of the number of training samples in desired label space, we randomly select n observations from each of the 100 sub-classes on the server and the rest of the observations in the training set are split into C = 10 subsets completely at random, ensuring that each class is equally represented in each subset.
Each subset corresponds to a dataset stored on one client and we use N c = 4000 .
We use the super-class as our label for all clients.
Experiment results with other values of C and N c is given in Appendix D.
We use ResNet18 (He et al., 2016) as our classiﬁer.
We use SGD optimizer Ruder (2016) with a learning rate of 10  2 , momentum 0 .
9 , and weight decay 5 ⇥ 10  4 .
The learning rate is divided byat 20 , 30 , and 40 epochs.
If not speciﬁed, our default setting for local update epochs is E = 1 .
Based on the superclass information in CIFAR100, we are able to formulate the transition matrix Q for all clients.
Its form and more training details are available in Appendix C.1.
Baselines We compare the following learning approaches: 1)
Single : only the data from the desired label space is used to train the classiﬁer; 2) FedMatch (Jeong et al., 2021): we treat client samples as unlabeled data, perform pseudo-labeling on the unlabeled sets, and augment samples in supervised loss training.
FedRep (Collins et al., 2021): clients train a J -class classiﬁer and the server trains a
K -class classiﬁer that differs in the last layer with different output dimension.
These classiﬁers share the same backbone parameters using FedAvg, the performance is tested using the classiﬁer on the server; 4) FedTrans (Chen et al., 2021b): all clients trains a J -class classiﬁer using FedAvg which is later ﬁne-tuned on server with a new K -class linear layer.
We refer FedMT (P) as our proposed method with probability projection loss; and refer FedMT (L) as our proposed method with label projection loss. [Table] [Figure]
4. B ENCHMARK E XPERIMENTS R ESULTS
Comparison under different amount of data on server We ﬁx number of clients C at 10 , and set different number of samples per-class n on the server in the range of {5, 10, 15, 20, 25}.
The advantage of FL is to enable access to more data to improve model performance.
As shown in
Tab. 1, when n increases (supervision information increases), the accuracy of all approaches improves.
Both FedMT (P & L) signiﬁcantly outperform alternative methods when supervision is limited n < 25 .
We achieve similar but much more stable performance compared with FedMatch when n = 25 .
FedMatch adaptively augments the training data with labels in the desired label space using heuristically derived conﬁdent pseudo labels.
Future work could combine FedMT with similar approaches used in FedMatch to evaluate heuristic results in a theoretically provable framework.
Analysis on convergence To validate our developed theoretical results in Theorem 3.4, we investigate the convergence rate of loss (2) of FedMT with label projection.
In Fig. 2(a), we explore loss as a function of the number of classes J 2 { 5 , 10 , 20 } with C = 10 under noise-free setting, and the convergence is consistent with Corollary 3.5, smaller J leads to lower convergence.
The loss curves in Fig.
2(b) show FedMT convergence was achieved faster when training with more clients for a ﬁxed total number of samples.
This result is consistent with the theoretical result of Theorem 3.4.
As shown in Fig. 2(c), under the default setting, the convergence rates are similar at different noise levels.
The results demonstrate our method is reasonably stable to noise, which is in line with Corollary 3.6.
Effect of varying noise levels Our method has the ﬂexibility to adapt to the case where the server data is noisy.
We use the accuracy of K -class prediction in the held out test set to compare our method with other approaches in CIFAR100, the results are reported in Fig. 2(d).
Given the mislabeling rate in many real applications ( e.g., healthcare) could be as high as 40% (Lettieri et al., 2005), we compare
FedMT with other learning approaches, by varying the noise level ⇠ on the server from {0, 0.1, 0.2, 0.3, 0.4} and set n = 10 on the server.
From the results, our proposed FedMT method signiﬁcantly outperforms all alternative methods both even when the noise level ⇠ is up to 0 .
Second, when increasing noise levels, FedMT shows more stable performance compared to baselines.
Our method also shows stable performance if we have more than 1 client has observations from desired label space and the results is given in Appendix D.5. [Table]
4. T REMOR S EVERITY P REDICTION OF P ARKINSON ’ S D ISEASE
Dataset Our method is not limited to sub- and super-classes as in CIFAR100, to illustrate the effectiveness of our method under the general overlapping class case, we evaluate FedMT for the prediction of tremor severity for patients with Parkinson’s disease (PD).
Tremor is a typical movement disorder occurring on the limbs of patients with PD (Qin et al., 2020).
Surface electromyography (sEMG) is widely used for movement disorder assessment by noninvasively recording electrical signals on the skin.
The patterns of sEMG signals can be used to turn PD diagnosis into different subtypes based on the diagnostic values.
We use the synthetic sEMG
dataset in Qin et al.
In our experiment, we consider K = 5 on the server and J = 3 on the client, according to MDSUPDRS19.
Goetz et al.
In this dataset, the classes in two label spaces has overlap.
The form of Q is given in Appendix C.2.
Setup We randomly sample 9000 observations from the sEMG dataset as client training data.
We hold out 1000 observations for testing.
We then split the training set to C = 50 clients completely at random, but ensuring that samples are balanced within each class in both label spaces.
Each client, therefore, has roughly 60 observations from one of the three classes.
In addition to the training set on clients, we also have n observations per class on the server.
Similar to the benchmark experiment, we vary n on the server.
Following Qin et al.
(2019), we use 12 summary statistics of sEMG as our features: mean absolute value (MAV), mean square value (MSV), root mean square (RMS), variance (VAR), standard deviation (STD), waveform length (WL), Willison amplitude (WAMP), log detector (LOG), slope sign change (SSC), zero crossing (ZC), mean spectral frequency (MSF) and median frequency (MF).
The mathematical deﬁnition of these features is given in Chowdhury et al.
We then use a single hidden layer multiple layer perceptron (MLP) with 128 hidden units and ReLU activation function as our backbone.
We use SGD optimizer with learning rate as 10  3 , and set batch size as 16.
Local update frequency is 1 epoch.
We train the model for 100 communication rounds.
More training details are available in Appendix C.2.
We use the same performance measure and the same alternative approaches as those described in Section 4.1 for comparison.
Results Overall, our quantitative results in Tab.
2 show that FedMT with probability projection performs better than probability projection on the sEMG dataset and consistently outperforms alternative learning approaches.
FedMT improves mean accuracy by a non-negligible margin, demonstrating a synergistic collaborative effect with clients’ data from the other label space.
The superiority of FedMT in this demanding medical data setting further demonstrates the efﬁcacy and robustness of our algorithm.
To test of the performance of FedMT at different degree of overlapping, we also conduct experiments when K = 10 with various noisy levels, the results is given in
Appendix E. Our method also outperforms other methods in this case.
C ONCLUSION
In this work, we propose a new FL framework, FedMT , to address an important yet under-explored mixed-type label setting.
Theoretically, we provide the convergence guarantee of FedMT with the extension of NTK.
Through extensive experiments on a benchmark and a medical dataset, we demonstrate FedMT can outperform alternative methods.
We also emphasize that since FedMT makes minor modiﬁcations to local predictions or labels, it has much more ﬂexibility to integrate with other FL strategies beyond FedAvg.
The performance of our proposed method can be further improved by combining our provable method with other heuristic-based weakly supervised learning approaches.
As a plug-and-play method, FedMT can be applied to non-IID setting by combining with advanced FL schemes, including Li et al.
(2020b; 2021); Karimireddy et al.