Collaborative Learning by Detecting Collaboration Partners
Abstract
Massive amounts of data are naturally dispersed over different clients in many real-world applications, collaborative learning has been a promising paradigm that allows to learn models through collaboration among the clients.
However, leveraging these dispersed data to learn good models is still challenging since data over different clients are heterogeneous.
Previous works mainly focus on learning the centralized model for all clients or learning a personalized model for each client.
When there are numerous clients, the centralized model performs badly on some clients, while learning a personalized model for each client costs unaffordable computational resources.
In this paper, we propose the collaborative learning method to detect collaboration partners and adaptively learn K models for numerous heterogeneous clients.
We theoretically prove that the model learned for each client is a good approximation of its personalized model.
Experimental results on real-world datasets verify the effectiveness of our method.
Introduction
In many real-world applications, massive amounts of data are naturally dispersed over numerous clients and each client only has limited data.
The model trained with local data on each client has poor performance since local data are insufﬁcient.
Leveraging these dispersed data on these clients to build well-generalized models is challenging.
In recent years, collaborative learning has been a promising paradigm that enables the clients to collaboratively learn the models (Blum et al., 2017).
Blum et al.
(2017) formalized two settings of collaborative learning, i.e. , personalized setting and centralized setting.
The personalized setting allows to return a model for each client, while the centralized setting only allows to return one single model for all clients.
They evaluated the efﬁciency of collaborative learning with overhead , i.e. , the ratio between the sample complexity of collaborative learning and its non-collaborative counterpart, and proved that the overhead of personalized setting is O (ln N ) and the overhead of centralized setting is O (ln 2 N ) .
Later, Nguyen & Zakynthinou (2018) and Chen et al.
(2018) proved that the overhead of centralized setting could be improved to O (ln N ) , which matches its lower bound ⌦ (ln N ) .
Generally, the data on different clients come from heterogeneous distributions.
For example, in the credit card fraud behavior detection task, the users’ revenue and expenditure behavior and credit rating information are different among banks in regions with diverse socioeconomic characteristics.
To deal with heterogeneous data, Mohri et al.
(2019) proposed Agnostic Federated Learning which optimizes the performance of the model for the single worst client with the min-max optimization scheme.
Mansour et al.
(2020) and Deng et al.
(2020) advocated that combining local and global models could improve the generalization ability.
Nevertheless, the learned single model may perform badly on clients whose distributions are different from the average distribution over all clients.
To achieve good performance, a personalized model should be learned for each client, but learning these personalized models is impractical when the number of clients N is very large since this costs unaffordable computational resources.
Intuitively, it’s better to return K ( 1 6 K ⌧ N ) appropriate models by analyzing the collaboration relationships among N heterogeneous clients.
We expect that the performance of the returned model is comparable to that of each client’s optimal personalized model.
Based on this intuition, in this paper we propose the collaborative learning method to detect collaboration partners for each client and adaptively learn K models for N heterogeneous clients.
We ﬁrst estimate each client’s contribution based on the derived generalization bound, and then partition the clients into K groups and return a model for each group.
We also theoretically prove that the model returned for each client is a good approximation of its optimal personalized model.
Experimental results on real-world datasets verify the effectiveness of our method.
Related Work
Blum et al.
ﬁrst theoretically studied collaborative learning and deﬁned overhead as the ratio between the sample complexity of collaborative learning and its non-collaborative counterpart.
They proved that the overhead of personalized setting is O (ln N ) and the overhead of centralized setting is O (ln 2 N ) .
Later, Nguyen & Zakynthinou (2018) and Chen et al.
(2018) proved that the overhead of centralized setting could be improved to O (ln N ) by using multiplicative weights methods, which matches its lower bound ⌦ (ln N ) .
When there are an ⌘ fraction of malicious clients, Qiao (2018) proposed an algorithm that achieves O ( ⌘ N + ln N ) overhead, which is proved to be the worst-case optimal.
Since the communication cost between clients and the central server is expensive, Blum et al.
(2021) developed algorithms by using distributed boosting to reduce the communication cost without compromising the sample complexity.
To deal with data heterogeneity, Mohri et al.
(2019) proposed Agnostic Federated Learning, which optimizes the performance of the model for the single worst client with the min-max optimization scheme.
Mansour et al.
(2020) proved that one single model generalizes badly on the clients whose distributions vary from the average distribution.
Mansour et al.
(2020) and Deng et al.
(2020) advocated that the mixture of local and global models could improve the generalization ability.
Li et al.
(2020) highlighted the detrimental effects of heterogeneity by observing that heterogeneity leads to poor convergence rate.
They proposed FedProx to tackle heterogeneity and provided convergence guarantee.
Wang et al.
(2020b) proposed the normalized averaging method FedNova to eliminate objective inconsistency while preserving fast convergence rate.
Dinh et al.
(2020) proposed a personalized algorithm pFedMe by using Moreau envelops as regularized loss functions and theoretically showed it could obtain quadratic speedup.
Huang et al.
(2020) proposed a novel framework FEDAMP which allows each client to collaboratively train its own model without using the global model and provided the convergence guarantee.
These works mainly focus on learning the global model which performs well on the average distribution.
However, it may perform badly on the clients whose distributions are different from the average distribution.
In a concurrent work (Even et al., 2022), the authors proposed methods to achieve the tradeoff between personalization and coordination, which tackles the same problem in our paper from a different perspective.
Client selection has also been studied in some works.
Considering that different groups of clients may have different learning objectives, Ghosh et al.
(2020) proposed IFCA which alternately estimates the cluster identities of the clients and optimizes the models for each cluster.
Fraboni et al.
(2021) introduced clustered sampling with better client representativity to reduced variance of the clients stochastic aggregation weights and provided two cluster approaches based on sample size and model similarity.
Tuor et al.
(2020) proposed a method for selecting relevant data for model training, where they used the model trained on the benchmark dataset which is task-speciﬁc to evaluate the data relevance.
Cui et al.
(2022) studied a rational collaboration scheme called collaboration equilibrium where each client collaborates with certain members who maximally improve the model learning and isolates the others who make little contribution.
However, these works are heuristic and lack theoretical support.
Preliminaries
Let X be the instance space and Y = { 0 , 1 } is the label space.
In the collaborative learning setting, there are N clients { C 1 , . . .
, C N } with distributions {D 1 , . . .
, D N } .
In practice, client C i only has access to m i examples S i
=
x i 1 , y i , . .  x im i , y
im i  drawn from distribution D
i , i = 1 , . . .
, N .
Let M = P
Ni =
m i be the total number of examples.
We consider the hypothesis space H with VC dimension d .
A hypothesis h 2 H is a mapping h : X ! Y .
Let l be the loss function and l ( h ( x ) , y ) is the loss of hypothesis h 2 H on example ( x , y ) .
We assume that l is upper bounded by a constant µ .
For client C i , denote L D i ( h ) as the expected risk of a hypothesis h 2 H with respect to distribution D i and ˆ L S i ( h ) is the empirical risk of a hypothesis h 2 H with respect to sample S i :
[Equation]
For client C i , its goal is to pursue the model h ?
i that minimizes the expected risk on its local distribution
D i , i.e. , h ?
i =
arg min
h 2 H L D i ( h ) .
Nevertheless, it is impractical to ﬁnd h ?
i since C i only has access to limited m i examples.
Thus the key incentive of clients C i participating in collaboration is the desire of seeking the model that could generalize well on its local distribution D i with samples { S 1 , . . .
, S N } from all the clients.
Our Work
In collaborative learning scenarios, the common strategy is to train the model over the weighted union of all samples, i.e. , S ↵ =
P
Nj =1 ↵ j S j (McMahan et al., 2017; Mohri et al., 2019).
Here ↵ = ( ↵ 1 , . .
. , ↵ N ) 2  N where  N is the N -simplex, i.e. , P
Nj =1 ↵ j = 1 and ↵ j > 0 for j = 1 , . . . , N .
The weighted empirical risk ˆ L ↵ ( h ) associated to hypothesis h 2 H can be denoted as ˆ L ↵ ( h ) =
P
Nj =1 ↵ j ˆ L S j ( h ) .
In this way, the weighted empirical risk ˆ L ↵ i ( h ) for client C i can be denoted as ˆ L ↵ i ( h ) =
P
Nj [-=
 α-]=1 ↵ ij ˆ L S j ( h ) with the corresponding collaboration vector ↵ i =
( ↵ i 1 , . . .
, ↵ iN ) .
The model for C i can be learned by minimizing ˆ L ↵ i ( h ) .
Let ˆ h ↵ i = arg min h 2 H ˆ L ↵ i
( h ) be the hypothesis that minimizes the weighted empirical risk ˆ L ↵ i ( h ) .
We provide the generalization bound of ˆ h ↵ i for client C
i in the following Theorem 1.
Theorem 1.
Let H be the hypothesis space with VC-dimension d. Let h ?
i =
arg min
h 2 H L D i ( h ) be the hypothesis that minimizes the expected risk on D i and
ˆ h ↵ i = arg min
h 2 H ˆ L ↵ i ( h ) is the hypothesis that minimizes the weighted empirical risk.
For any given  2 (0 , 1) and 8 i 2 { 1 , . . . , N } , the following bound holds with probability at least
[Equation]
Here d H ( D i , D j ) = sup h 2 H
L D i ( h )  L D j ( h )  is the Integral Probability Metrics (IPM) which measures the divergence between distributions D i and D j .
We leave the proof of Theorem 1 in Appendix A due to space limitation.
Theorem 1 reveals that for client C i the gap between ˆ h ↵ i and h ?
i depends on the weight ↵ i , the number of examples m i and the
IPM divergence.
It further suggests that the optimal model for client C i can be obtained by choosing appropriate ↵ i to minimize the right-hand side of Eq.(1).
The optimal ↵ ?
i =
( ↵ ?
i 1 , . .
. , ↵ ? iN ) for client C i can be derived with the following Theorem 2.
Theorem 2.
For client C i , let ⌅ ji = d H
( D i , D j ) where j 2 { 1 , . . .
, N } , 8 i 2 { 1 , . . . , N } .
Given VC dimension d , total sample size M , constants µ and  , let  = µ q 8( d log(2 M ) + log 8  ) .
For client C i , sort { ⌅ 1 i , . . .
, ⌅ Ni } in ascending order to get { ⌅  (1) i
[Equation]
[Equation] , where  ( j ) 2 { 1 , . . . , N } .
The optimal ↵ ?
i =
{ ↵ ?
i 1 , . .
. , ↵ ? iN } for client C i is given by
[Equation]
Here [ · ] + = max ( · , 0) , ⇣ is the larger root of equation
P q 6 q i m  ( q ) ⇣ ⇣  ⌅  ( q ) i ⌘ 2 =  2 , and q i = arg max t
[Equation]
We leave the proof of Theorem 2 in Appendix B. Theorem 2 can be interpreted as follows.
The optimal ↵ ?
i makes the balance between the divergence and the number of examples.
If ↵ ? ij > 0 , then C j is beneﬁcial to C i in the collaboration process, i.e. , C i can utilize the data on C j to relieve the scarcity of its local data.
↵ ?
ij represents the contribution that client C j makes to C i .
We use the directed graph A
= ( V, E ) to represent the collaboration relationships among clients, where | V | =
N , node i denotes client C i and the weight of edge from j to i is ↵ ?
ij .
Here we provide an example in Figure 1 to give a more concrete interpretation.
If there exists an edge from node j to node i , then C i needs the data from C j .
In Figure 1, the contribution that C 2 and C 4 need from all the clients are more similar than that of C 4 and C 5 , i.e. , the incoming edges of C 2 and C 4 are more similar than that of C 4 and C 5 .
We could probably return the same model for C 2 and C 4 while it is inappropriate to return the same model for C 4 and C 5 .
[Figure]
We refer ˆ h ↵ ?
i with respect to the optimal ↵ ?
i as the personalized model for client C
i .
When the number of clients N is large, learning a personalized model for each client is impractical since this costs unaffordable computational resources.
An alternate way is to learn K ( 1 6 K ⌧ N ) appropriate models which are expected to be comparable to the personalized models.
Intuitively, if client C i and C j have similar incoming edges, then we can return the same model for them since they need similar contribution from other clients.
In this way, C i and C j are called collaboration partners .
This motivates us to partition N clients into K groups based on the idea that collaboration partners should be in the same group.
If we set K = 2 in Figure 1, then the group structure is most likely to be { C 1 , C 2 , C 3 , C 4 } and { C 5 , C 6 , C 7 , C 8 } as shown in Figure 1.
4.1 Collaboration with Modularity Maximization
To evaluate the incoming-edge similarity among clients, we construct matrix U which is deﬁned as
[Equation]
where A is the adjacency matrix of the directed graph A , D in is the diagonal matrix of in-degrees in graph A and  is the discounting parameter.
We set  = 12 as suggested in Satuluri & Parthasarathy (2011).
Large U ij represents large similarity between clients, which provides the possibility of detecting the communities for the clients and partitioning collaboration partners into groups.
Modularity has been used as an objective function to evaluate the quality of group partitions by measuring the density of intra-group edges as compared to inter-group edges (Newman & Girvan, 2004), which is formulated as follows:
[Equation]
Here G = { G 1 , . . .
, G K } is the group partition.
W = P i P j w ij is the sum of all edge weight.
W G k in = P i,j 2 G k w ij is the total edge weight within group G k and W G k vol =
P i 2 G k
P j w ij is the total edge weight attached to nodes in group G k .
Higher modularity implies better group partitions and modularity maximization has been widely used for detecting meaningful group partitions (Clauset et al., 2004; Newman, 2006a,b).
Although maximizing modualarity is NP-hard, approximation guarantees can be obtained by introducing the technique of rounding semideﬁnite programs.
We rewrite Modularity as follows:
[Equation]
where d i =
P j w ij is the degree of vertex i , and g i is the group membership of client C i where  ( g i , g j )
= 1 if g i
=
g j ( i.e. , client C i , C j belong to the same group) and 0 otherwise.
Let e j 2 R K be the unit vector with 1 in the j -th coordinate and 0 elsewhere.
Let ⌫ i be the variable that indicates the group membership of client C
i , i.e. , ⌫ i =
e j means C i belongs to group G j .
Then the modularity maximization problem can be formulated as the following vector programming:
[Equation]
Let M denote the modularity matrix with its elements M ij = w ij 2 W  d i d j 4 W 2 .
Denote M + =
{ ( i, j ) | M ij > 0 } and M  = { ( i, j ) | M ij < 0 } .
Noticing that
P ij M ij = 0 , the above vector programming can be relaxed to the following SemiDeﬁnite Program (SDP) by adding a constant P
M + M ij to the objective and relaxing the constraints.
The proof can be found in Appendix C.
[Equation]
Let OPT Q ( G ) be the maximum value of modularity deﬁned on the similarity matrix U .
Dinh et al.
(2015) provided the following result to guarantee the approximation property for the solution of the
SDP in Eq.(7).
Lemma 1 implies that we could ﬁnd reasonable group partitions by solving the SDP in Eq.(7) for maximizing modularity on U .
Lemma 1.
Given matrix U , let Q ( G ) be the modularity value of the group partition G obtained by solving the SDP in Eq.(7) using the rounding techniques in Charikar et al.
Then Q ( G ) satisﬁes Q ( G ) >  OPT Q ( G )  (1   ) , where  = 0 .
766 is the approximation factor.
After obtaining the group partitions, we train models for the groups according to their properties.
Since we assign the clients into K groups, there may exist some clients which have small similarity with other clients in the same group.
We should pay attention to these clients.
We call edge e ij 2 U the weak edge if its weight satisﬁes w ij < 1 N , else strong edge otherwise.
Given group partition G = { G 1 , . . .
, G K } , let Z in be the number of weak edges within all groups and N k is the number of clients in group G k where P
Kk =1 N k = N .
For client C i in group G k , we call C i the bad client if C i is isolated after removing all the weak edges within group G k .
Generally, we keep the largest part when a group is divided into several disjoint parts after removing all weak edges within this group, and clients that do not belong to the largest part are bad clients .
Otherwise, we call it the good client .
In Figure 1, C 1 , C 2 , C 3 , C 4 are in the same group since they have relatively similar incoming edges, and C 5 , C 6 , C 7 are in the same group for the same reason.
C 8 is most likely to be a bad client since it has less similar incoming edges with C 5 , C 6 , C 7 in the same group.
To guarantee the performance of the learned models, we pick out the bad clients since they have small similarity with other clients in the same group.
We return ˆ h ↵ Gk for clients in group G k which is obtained by minimizing ˆ L ↵ Gk ( h ) where ↵ G k
= | G k \B| P
C i 2 G k \B ↵ ?
i .
We refer this method as A daptive C ollaborative L earning with M odularity M aximization (ACLMM).
The whole process is summarized in Algorithm 1, where we use the rounding techniques introduced in Charikar et al.
(2003) to solve the SDP in Eq.(7) for modularity maximization to obtain the group partition.
Let B be the set of bad clients in the group partition returned by solving the SDP.
We provide the following
Lemma 2 to bound the number of bad clients.
Lemma 2.
Given the group partition G = { G 1 , . . .
, G K } returned by Algorithm 1, assume N k > 2 p Z in , 8 k 2 { 1 , . .
, K } .
Let N min =
min k N k , then |B| satisﬁes |B| 6 N min  p N 2 min  4 Z in, where Z in
[Equation]
i and  = 0 .
766 is a constant.
The proof of Lemma 2 can be found in Appendix C. Lemma 2 indicates that the number of weak edges within all groups Z in depends on the maximum modularity OPT Q ( G ) and the number of
1: procedure TRAIN ( K , A ) 2:
Compute similarity matrix U according to Eq.(3).
Obtain G = { G 1 , . .
, G K } with DIVIDE ( K, U ).
Pick out bad clients into set B .
Compute ↵ G k
= | G k \B| P
C i 2 G k \B ↵ ?
i for G k .
Train model
ˆ h ↵ Gk by minimizing ˆ L ↵ Gk ( h ) and return ˆ h ↵ Gk to all clients
C i in group G k .
7: end procedure 8: function DIVIDE ( K, U ) 9:
Pick t random hyperplanes and use projection to divide the nodes into 2 t groups.
Take the better solution of t = 2 and t = 3 with respect to the SDP relaxation Eq.(7).
11: return Group partition G = { G 1 , . . .
, G K } .
12: end function groups K .
If each group G k is not too small, i.e. , N k > 2 p Z in , 8 k 2 { 1 , . .
, K } , then the number of bad clients |B| is upper bounded by the number of weak edges.
We further theoretically analyze the generalization bound of the models returned for the clients.
Given group partition G = { G 1 , . . .
, G K } returned by solving the SDP, we remove all the weak edges within all groups.
For any good clients C i and C j in the same group, there always exits a shortest path p ij between them since good clients in the same group are connected through strong edges.
Let ⌘ denote the maximum p ij between any good clients
C i and C j in group G k , 8 k 2 { 1 , . .
. , K } .
Denote ⌧ = min
e ij 2 E in w ij where E in is the set of strong edges within all groups.
We provide Theorem 3 to show that the model returned by Algorithm 1 for each good client is an approximation of its personalized model.
Theorem 3.
Let G =
{ G 1 , . . .
, G K } be the group partition returned by solving the SDP.
ˆ h ↵ Gk is the model returned by Algorithm 1 for client C
i in group G k .
Let upp(ˆ h ↵ Gk ) be the upper bound of the expected risk of ˆ h ↵ Gk on C i according to the right-hand side of Eq.(1) and upp(ˆ h ↵ ?
i ) is the upper bound of the expected risk of the personalized model ˆ h ↵ ?
i on C i according to the right-hand side of Eq.(1).
The following result holds except for the bad clients in B :
[Equation]
We leave the proof of Theorem 3 in Appendix C due to space limitation.
Theorem 3 reveals that the generalization gap between the model returned by Algorithm 1 for any good client and its personalized model relies on the minimum weight of strong edges ⌧ within all groups and the maximum length of the shortest paths between good clients in the same group ⌘ .
These are further determined by the intrinsic properties of U with respect to modularity and the number of groups K .
4.2 Collaboration with Clustering
Suppose we have the prior knowledge about the collaboration partnerships among the clients, i.e. , there exists a potential partition P ?
= { P ? 1 , . . .
, P ?
K } .
In this case, we try to develop the method with tighter bound for the approximated models.
We use  ( P ) to evaluate the quality of partitions which is deﬁned as  ( P ) =
[Equation]
d ( ↵ ?
i , ¯ ↵ k ) , where P = { P 1 , . .
, P K } is the group partition, d ( · , · ) is the distance measure and ¯ ↵ k is the center of group P k .
If a partition P is of high quality, then  ( P ) is small.
Take Figure 2 for example where K = 2 , the value  ( P ) of the graph A in
Figure 2 is smaller than that in Figure 1 since clients in the same group in Figure 2 have more similar incoming edges.
Thus the example in Figure 2 has better structures than that in Figure 1.
Let OPT  ( P ) =
min P  ( P ) be the minimum value of  ( P ) over all group partitions and
P ?
= { P ? 1 , . . .
, P ?
K } is the corresponding optimal group partition.
The distance between two group partitions P = { P 1 , . .
, P K } and P {+
+}
= { P 0 1 , . . .
, P 0 K } is denoted as dist ( P , P 0 ) =
min  2 S K 1 N P
Kk =
I
( P k = P 0  ( k ) ) , where S K is the set of permutations.
Balcan et al.
(2009) proposed the ( 1 +  , ✏ )- approximation-stability property to show that if a given group partition
P satisﬁes  ( P ) 6 (1 +  )OPT  ( P ) , then we have dist ( P , P ? )
6 ✏ .
This provides a way to ﬁnd good group partitions by optimizing  ( P ) .
We assume that { ↵ ? 1 , . . .
, ↵ ? N } satisfy the ( 1 +  , ✏ )-approximation-stability property with respect to  ( P ) .
In this case, there still exist some bad clients and we need to ﬁnd them out.
Let d 1 ( ↵ ? i )
=
min k d ( ↵ ?
i , ¯ ↵ ? k ) be the distance of ↵ ?
i to its closest group center and d 2 ( ↵ ? i ) =
min j 6 = k d  ↵ ?
i , ¯ ↵ ?
j  is the distance of ↵ ?
i to its second-closest group center.
Denote ¯ d = 1 N OPT  ( P ) as the average distance of all clients.
Given any constant t where t > 2 , let d ?
=
¯ d ✏ t be the critical distance .
We call C i the bad client if d 1 ( ↵ ? i ) > d ? or d 2 ( ↵ ?
i )  d 1 ( ↵ ?
i ) 6 t 2 d ? , else we call it the good client .
It is easy to understand that good clients are those tightly coupled to clients in the same group and loosely coupled to clients in different groups.
To guarantee the performance of the learned models, we pick out the bad clients.
We return ˆ h ↵ Pk for clients in group P k which is obtained by minimizing ˆ L ↵ Pk ( h ) where ↵ P k = | P k
\B|
P C
i 2 P k \B ↵ ?
i .
We refer this method as A daptive C ollaborative L earning with C lustering (ACLC).
The whole process is summarized in Algorithm 2, where  > 1 and t > 2 are given constants.
We use the method in Balcan et al.
(2009) to obtain the group partition, where the constant-factor K -median approximation algorithm is done as that in Arya et al.
Let B be the set of bad clients.
We provide the following Lemma 3 to bound the number of bad clients |B| in the group partition P produced by Algorithm 2.
Lemma 3.
Assume that { ↵ ? 1 , . .
. , ↵ ? N } satisfy the ( 1 +  , ✏ )-approximation-stability property.
Let
P = { P 1 , . .
, P K } be the group partition produced by Algorithm 2.
Then the number of bad clients |B| satisﬁes |B| <
(6 + t  ) ✏ N , where t > 2 and  > 1 are given constants.
The proof of Lemma 3 can be found in Appendix D. Lemma 3 reveals that the number of bad clients is related to  and ✏ , which reveals the degree of difﬁculty in ﬁnding approximate group partitions.
We further theoretically analyze the generalization bound of the models returned for the clients in the following Theorem 4, which shows that the model returned by Algorithm 2 for each good client is an approximation of its personalized model.
Theorem 4.
Assume that { ↵ ? 1 , . . .
, ↵ ? N } satisfy the ( 1 +  , ✏ )-approximation-stability property.
Let
P = { P 1 , . .
, P K } be the group partition produced by Algorithm 2.
ˆ h ↵ Pk is the model returned by Algorithm 2 for client C
i in group P k .
Let upp(ˆ h ↵ Pk ) be the upper bound of the expected risk of ˆ h ↵ Pk on C i according to the right-hand side of Eq.(1) and upp(ˆ h ↵ ?
i ) is the upper bound of the expected risk of the personalized model ˆ h ↵ ?
i on C i according to the right-hand side of Eq.(1).
Then given any constants t > 2 and  > 1 , the following result holds except for the bad clients in B :
[Equation]
We leave the proof of Theorem 4 in Appendix D due to space limitation.
Theorem 4 reveals that the generalization gap between the model returned by Algorithm 2 for any good client and its personalized model relies on the potential structures of the clients’ collaboration relationships, i.e. , the optimal value of objective  ( P ) .
If the collaboration relationships have good structures, i.e. , OPT  ( P ) is small, then the returned model for any good client is close to its personalized model.
1: procedure TRAIN ( K , A , constants ✏ ,  , t,  ) 2:
Obtain P = { P 1 , . .
, P K } and ˆ d with DIVIDE ( K , A , ✏ ,  , t,  ).
Compute the critical distance ˆ d ?
=
ˆ d  t ✏ .
Pick out bad clients into set B .
Compute ↵ P k = | P k
\B| P
C i 2 P k \B ↵ ?
i for P k .
Train model ˆ h ↵ Pk by minimizing ˆ L ↵ Pk ( h ) and return ˆ h ↵ Pk to all clients
C i in group P k .
6: end procedure 7: function DIVIDE ( K , A , ✏ ,  , t,  ) 8:
Run the constant-factor K -median approximation algorithm on { ↵ ? 1 , . .
. , ↵ ? N } to compute  ( P ) .
Compute the value ˆ d =  ( P ) N 6  ¯ d .
Construct the threshold graph G which connects all pairs ( C i , C j ) with d ( ↵ ? i , ↵ ? j ) <
2  ˆ d  t ✏ .
11: for k = 1 to K  1 do 12:
Let v i be the node with the highest degree in G .
Select v i and its neighborhoods to get group P k .
Remove P k from G .
14: end for 15: return P =
{ P 1 , . . .
, P K } and ˆ d , where P K = { ↵ ? 1 , . . .
, ↵ ? N } \  [
K  1 k =1 P k  .
16: end function
Experiments
In the experiments, we use the following four datasets: (1) The MNIST dataset consists of 70 K handwritten digit images in 10 classes, which has a training set of 60 K examples and a test set of 10 K examples.
The CIFAR-10/100 dataset consists of 60 K color images in 10 / 100 classes, with 6000 / 600 images per class.
They have 50 K training images and 10 K test images.
The Federated
Extended MNIST (FEMNIST) dataset is built by partitioning the data in Extended MNIST based on the writer of the digit/character.
For MNIST, we use the same CNN model as that in McMahan et al.
(2017); for CIFAR-10, we use the CNN model which consists of 2 convolutional layers followed by 2 fully connected layers as that in Ghosh et al.
(2020); for CIFAR-100, we use the ResNet-18 model (He et al., 2016); for FEMNIST, we use the same CNN model as that in LEAF (Caldas et al., 2018).
In collaborative learning, using KL divergence to estimate the divergence between distributions is a common practice (He et al., 2020; Yang, 2021).
We use the right hand of Eq.8 to estimate the IPM d H ( D i , D j ) in the experiments since
[Equation]
where d JS ( D i k D j ) is the Jensen-Shannon (JS) divergence (symmetrical KL divergence).
The proof can be found in Appendix E.
First, we partition MNIST into clients using the same method as that in McMahan et al.
and set client number N = 10 where each client has 3 classes, total number of samples M = 2000 .
We partition CIFAR-10/100 using the Dirichlet partition as that in Yurochkin et al.
(2019) and Wang et al.
(2020a).
We set N = 20 and M = 5000 for CIFAR-10 and set N = 50 and M = 20000 for CIFAR-100.
For FEMNIST, we use the default partition proposed in LEAF and choose N =clients randomly from the total 3550 clients (Caldas et al., 2018).
We further set the number of groups K = 3 for MNIST, K = 4 for CIFAR-10, K = 7 for CIFAR-100 and K = 4 for FEMNIST.
We divide N clients into K groups and return a model for each group according to Algorithm 1 on the three datasets respectively, which is denoted as ACLMM.
We use the centralized model as the baseline, where only one single model is trained according to ↵ ij = m j M , 8 i 2 { 1 , . . .
, N } as that in
McMahan et al.
We also implement the personalized model, where each client will obtain the personalized model according to their ↵ ? i .
Since the client number on CIFAR-100 is relatively large, we only perform the personalized model on MNIST, CIFAR-10 and FEMNIST.
The results on four datasets are depicted in Figure 3.
From Figure 3, we ﬁnd that the model learned with our ACLMM performs better than the centralized model and is comparable to the personalized model.
[Figure]
In order to verify the collaboration partners detected by our ACLMM, we take CIFAR-10 for example and visualize the group partition returned by ACLMM in Figure 4(a), where nodes with the same color are in the same group and thicker edge means larger similarity between clients.
We also visualize the similarity matrix U as a color map in Figure 4(b), where the ( i, j ) -th block characterizes the similarity between C i and C j (the lighter the color is, the smaller the similarity is).
Obviously, clients in the same group (having the same node color) in Figure 4(a) are those with large similarities in Figure 4(b), which means they are collaboration partners.
[Figure]
Next, we construct the clients that have intrinsic structures on MNIST and CIFAR-10/100 in the following way.
For MNIST, we set N = 10 with K = 3 .
The ﬁrst group contains 3 clients with samples dominated by digit { 0 , 1 , 2 } .
The second group contains 3 clients with samples dominated by digit { 3 , 4 , 5 } .
The third group contains 4 clients with samples dominated by digit { 6 , 7 , 8 , 9 } .
The number of samples on the clients satisﬁes the Dirichlet distribution as that in Yurochkin et al.
(2019), and Wang et al.
(2020a).
We analogously set N = 20 with K = 4 for CIFAR-10 and set
N = 50 with K = 7 for CIFAR-100.
We set the total number of sample M = 2000 / 5000 /for
MNIST/CIFAR-10/CIFAR-100 respectively.
Since the partition of FEMNIST have ambiguous clustering structures (Ghosh et al., 2020), we did not run clustering experiments on FEMNIST.
[Figure]
We divide N clients into K groups and return a model for each group according to Algorithm 2 on the three datasets respectively, which is denoted as ACLC.
We also implement the centralized model and the personalized model.
We only perform the personalized model on MNIST and CIFAR-10.
The results on three datasets are depicted in Figure 5.
From Figure 5 we can ﬁnd that the model learned with our ACLC performs much better than the centralized model and is comparable to the personalized model.
Furthermore, the gap between the model return by ACLC and the personalized model is small, which shows that our bound in Theorem 4 is tight when the collaboration relationships have intrinsic clustering structures.
Conclusion
In this paper, we propose the adaptive collaborative learning method to deal with heterogeneous data on the clients, which can detect the collaboration partners from numerous heterogeneous clients and adaptively learn K models for them.
We also theoretically prove that the model learned with our method for each client is a good approximation of its personalized model.
Experimental results on real-world datasets verify the effectiveness of our method.
Broader Impact
Our work provides the possibility of lowering computational burden while maintaining comparable model performance in collaborative learning.
We believe our work will be beneﬁcial for collaborative applications, and do not have any negative societal impacts.
Acknowledgment
We would like to thank Chiyu Cai for helpful discussions and anonymous reviewers for their valuable comments.
This work is supported by the National Key Research and Development Program of
China (2018AAA0101100), the National Science Foundation of China (61921006, 62276125), the
Fundamental Research Funds for the Central Universities (022114380013), the project of HUAWEILAMDA Joint Laboratory of Artiﬁcial Intelligence, and the Collaborative Innovation Center of Novel
Software Technology and Industrialization.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information on how to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or [N/A] .
You are strongly encouraged to includ e a justiﬁcation to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets? [Yes] • Did you include the license to the code and datasets?
[No] The code and the data are proprietary.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[No] (c) Did you discuss any potential negative societal impacts of your work?
[Yes] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[No] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[N/A] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[No] 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] (b) Did you mention the license of the assets?
[Yes] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]