With a Little Help from My Friend: Server-Aided Federated Learning with Partial Client Participation
Abstract
Although federated learning (FL) has been a prevailing distributed learning framework in recent years due to its beneﬁts in scalability/privacy and rich applications in practice, there remain many challenges in FL system design, such as data and system heterogeneity.
Notably, most existing works in the current literature only focus on addressing data heterogeneity issues (e.g., non-i.i.d.
datasets across clients), while often assuming either full client or uniformly distributed client participation.
However, such idealistic assumptions on client participation rarely hold in practical
FL systems.
It has been frequently found in FL systems that some clients may never participate in the training (aka partial/incomplete participation) due to various reasons.
This motivates us to fully investigate the impacts of incomplete FL participation and develop effective mechanisms to mitigate such impacts.
Toward this end, by establishing a fundamental generalization error lower bound, we ﬁrst show that conventional FL is not PAC-learnable under incomplete participation.
To overcome this challenge, we propose a new server-aided federated learning (SA-FL) framework with an auxiliary dataset deployed at the server, which is able to revive the PAC-learnability of FL under incomplete client participation.
Upon resolving the PAC-learnability challenge, we further propose the SAFARI (serveraided federated averaging) algorithm that enjoys convergence guarantee and the same level of communication efﬁciency and privacy as state-of-the-art FL.
Introduction
Since the seminal FedAvg (federated averaging) algorithm [1], most works on FL to date are focused on addressing data heterogeneity issues [2–14].
While these algorithms have achieved varying degrees of theoretical and/or empirical success, it does not directly address the underlying challenges of client participation resulted from system-level heterogeneity.
Speciﬁcally, these methods often make “ideal” assumptions on client participation in that, in each communication round, clients either all participate or are sampled uniformly at random in the FL training process (referred to as “full” and “uniform” client participation in this paper, respectively).
However, such assumptions on client participation rarely hold for FL systems in practice due to the aforementioned system-level heterogeneity.
Indeed, studies have found that system-level heterogeneity could signiﬁcantly affect client participation and largely degrade the learning performance of FL [15,16].
In FL, the participation of a client is affected not only through the request/sampling from the server, but also the client’s status (e.g., busy or idle, whether or not in a stable network, privacy concerns, or battery level).
Thus, the client participation is not solely determined by the server.
Even if the server enlists clients uniformly at random, the actual client participation may signﬁcantly deviate from uniform participation.
For example, it is shown in [16] that more than 30% clients never participated in FL, while only 30% of clients contribute to 81% of the total computation even if server uniformly samples clients.
Exacerbating the problem is the fact that the client’s status could be unstable and time-varying.
Unfortunately, in the FL literature, there remains a lack of a theoretical understanding on the impacts of incomplete client participation, let alone how to mitigate such impacts on FL.
This motivates us to ﬁll this gap by fully investigating how to address the challenges of incomplete client participation in this paper.
The major contributions and main results of this paper are summarized as follows:
• We ﬁrst study standard FL under incomplete client participation through the lens of statistical learning theory.
We prove that FL with incomplete client participation is not PAC-learnable by establishing a generalization error lower bound.
Our analysis reveals that no learning algorithm can approach zero generalization error under incomplete participation in FL even in the limit of inﬁnitely many training data and iterations.
• To overcome the non-PAC-learnable challenge, we propose a server-aided federated learning framework (SA-FL) to revive PAC learnability of FL under incomplete client participation.
The key idea of SA-FL is to maintain an auxiliary dataset at the server that mimics the data distributions of the non participating clients.
This allows the system to mitigate the deviation caused by distribution mismatch resulting as a consequence of incomplete client participation.
Under mild conditions, we establish the PAC learnability of SA-FL by proving a new generalization error bound with the joint sever-client dataset. • After showing the PAC learnability of SA-FL, we further propose an efﬁcient training algorithm for SA-FL called SAFARI (server-aided federated averaging).
By carefully designing the update coordination between the server and the clients, we show that SAFARI achieves an O (1 / √ KT ) convergence rate.
PAC learnability analysis for FL with incomplete client participation
The goal of FL is to minimize the following loss function F ( x ) =
E i ∼P
[ F i ( x )] , where F i ( x ) (cid:44) E ξ ∼ P i [ f i ( x , ξ )] .
Here, P represents the distribution of the entire client population, x ∈ R d is the model parameter, F i ( x ) represents the local loss function at client i , and P i is the underlying distribution of the local dataset at client i .
In general, P i (cid:54) =
P j , if i (cid:54) =
j due to data heterogeneity.
However, the loss function F ( x ) or full gradient ∇ F ( x ) can not be directly computed as the exact distribution of data is unknown in general.
Instead, one often considers the following empirical risk minimization (ERM) problem in the form of ﬁnite-sum:
[Equation]
where ˆ F i ( x ) (cid:44) 1 | S i | (cid:80) ξ ∈ S i f i ( x , ξ ) .
Here, M is the total number of clients, S i is the local dataset with cardinality | S i | , which is i.i.d.
and sampled from distribution P i , α i = | S
i | (cid:80) j ∈ [ M ] | S j | (hence (cid:80) i ∈
[ M ] α i = 1 ).
For ease of presentation, we consider a balanced dataset case: α i = 1 M , ∀ i ∈
[ M ] .
Next, we state several basic deﬁnitions from statistical learning theory [17] that are needed here.
Deﬁnition 1 (Generalization Error and Empirical Error) .
Given a hypothesis h ∈ H , a target concept f , an underlying distribution D and a dataset S i.i.d.
sampled from D ( S ∼ D ), the generalization error and empirical error of h are deﬁned as follows R D ( h )
=
E ( x,y )
∼D l ( h ( x ) , f ( x )) and ˆ R D ( h ) = 1 | S | (cid:80) i ∈
S l ( h ( x i ) , f ( x i )) , where l ( · ) is some valid loss function.
Deﬁnition 2 (Optimal Hypothesis) .
For a distribution D and a dataset S ∼ D , we deﬁne h ∗D = argmin h ∈H R D ( h ) and ˆ h ∗D = argmin h ∈H ˆ R D ( h ) .
Deﬁnition 3 (Excess Error) .
For hypothesis h and distribution D , the excess error and excess empirical error are deﬁned as ε D ( h )=
R D ( h ) −R D ( h ∗D ) , and ˆ ε D ( h )=
ˆ R D ( h ) − ˆ R D (ˆ h ∗D ) , respectively.
2.1 FL under incomplete client participation
With the above notations, we consider the FL under incomplete client participation.
Consider an
FL system with M clients in total.
We let P denote the underlying joint distribution of the entire system, which can be decomposed into the summation of the local distributions at each client, i.e.,
P = (cid:80) i ∈
[ M ] λ i P i , where λ i > 0 and (cid:80) i ∈
[ M ] λ i = 1 .
We assume that each client i has n training samples i.i.d.
drawn from P i , i.e., | S i | = n, ∀ i ∈ [ M ] .
Then, S = { ( x i , y i ) , i ∈
[ M × n ] } can be viewed as the dataset i.i.d.
sampled from the joint distribution P .
We consider an incomplete client participation setting, where m ∈ [0 , M ) clients participate in the FL training as a result of some client sampling process F .
We let F ( S ) represent the data ensemble actually used in the training and
D denote the underlying distribution corresponding to F ( S ) .
For convenience, we deﬁne the notion α = mM as the FL system capacity (i.e., only m clients participate in the training).
For FL with incomplete client participation, we establish the following fundamental performance limit of any learner in general.
For simplicity, we use binary classiﬁcation with zero-one loss here.
We state the following impossibility result in Theorem 1 in terms of PAC learnability:
Theorem 1 (Impossibility Theorem) .
Let H be a non-trivial hypothesis space and L : ( X , Y ) ( m × n ) → H be the learner for an FL system.
There exists a client participation process
F , a distribution P , and a target function f ∈ H with min h ∈H R P
( h, f )
= 0 , such that
[Equation]
Proof Sketch.
The proof is based on the method of induced distributions in [17–19].
We ﬁrst show that the learnability of an FL system is equivalent to that of a system that arbitrarily selects mn out of
Mn samples in centralized learning.
Then, for any learning algorithm, there exists a distribution P such that dataset F ( S ) resulting from incomplete participation and seen by the algorithm is always distributed identically for any target functions.
Thus, no algorithm can learn a better predictor than random guessing.
Due to space limitation, we relegate the full proof to supplementary material.
Given the system capacity α ∈ (0 , 1) , the above theorem characterizes the worst-case scenario for
FL with incomplete client participation.
It shows that for any learner (i.e., algorithm)
L , there exists a bad client participation process F and distributions P
i , i ∈
[ M ] over target function f , for which the error of the hypotheses returned by L is constant with non-zero probability.
In other words, FL with incomplete client participation is not PAC-learnable .
One interesting observation here is that the lower bound is independent of the number of samples per client n .
This indicates that even if each client has inﬁnitely many samples ( n → ∞ ), it is impossible to have a zero-generation-error learner under the incomplete client participation situation ( α ∈ (0 , 1) ).
Note that this fundamental result relies on two conditions: heterogeneous dataset and arbitrary client participation.
Under these two conditions, there exists a worst-case scenario where the underlying distribution D of the participating data S D = F ( S ) deviates from the ground-truth P , thus yielding a non-vanishing error.
This result also sheds light on how to motivate client participation in FL effectively and efﬁciently: the participating client’s data should be comprehensive enough to model the complexity of the joint distribution P to close the gap between D and P .
Note that this result is not contradictory to previous works where the convergence of FedAvg is guaranteed since this theorem is not applicable for homogeneous (i.i.d.)
datasets or uniformly random client participation.
As mentioned earlier, most of the existing works rely on at least one of these two assumptions.
However, none of these two assumptions hold for FL with incomplete client participation.
Theorem 1 naturally leads to an important open question: How to make FL with incomplete client participation PAC-learnable?
Toward this end, we propose a new server-aided federated learning framework (SA-FL) in the next subsection.
2.2 Server-aided federated learning (SA-FL)
We consider the same FL problem with incomplete client participation, which induces a dataset
S D ∼ D , with cardinality n S and D (cid:54) = P .
In SA-FL, the server is equipped with an auxiliary dataset
T of cardinality n T , which is i.i.d.
sampled from distribution P .
As a result, the learning process is to minimize R P ( h ) by utilizing ( X , Y ) n T + n S to learn a hypothesis h ∈ H .
For notional clarity, we assume the joint dataset S Q =
( S D ∪ T ) ∼ Q with cardinality n T + n S for some distribution Q .
The intuition of SA-FL is to utilize dataset T as a vehicle to correct potential distribution derivations due to incomplete client participation.
By doing so, the server steers the learning by a small number of representative data, while the clients aid the learning by federation to leverage the huge amount of privately decentralized data ( n S (cid:29) n T ).
For SA-FL to be practical, it is highly desirable that the server only needs to maintain a small dataset from distribution P .
Note that the assumption of access to this dataset is not restrictive since such datasets are already available in many FL systems: although not always necessary for training, an auxiliary dataset is often needed for deﬁning FL tasks (e.g., simulation prototyping) before training and model checking after training (e.g., quality evaluation and sanity checking)
Also, obtaining an auxiliary dataset is affordable since the number of data points required is relatively small (of the order of hundreds, see our experimental results), and hence the cost is low.
Then, SA-FL can be easily achieved or even with manually labelled data thanks to its small size.
It is also worth noting that many works use such auxiliary datasets in FL for security [22], incentive design [23], and knowledge distillation [24].
Before deriving the generalization error bound for SA-FL, we state the following assumption and deﬁnition.
Assumption 1 (Noise Condition) .
Suppose that h ∗ P and h ∗ Q exist.
There exist β P , β Q ∈ [0 , 1] and α P , α Q > 0 such that
[Equation]
This assumption is a traditional noise model known as the Bernstein class condition, which has been widely used in the literature [25–27].
Deﬁnition 4 ( ( α, β ) -Positively-Related) .
Distributions P and Q are said to be ( α, β ) -positivelyrelated
if there exist constants α ≥ 0 and β ≥ 0 such that
[Equation]
Deﬁnition 1 speciﬁes a stronger constraint between distributions P and Q .
It indicates that the difference of excess error for one hypothesis h ∈ H between P and Q is bounded by the excess error of Q in some exponential form.
With the above assumption and deﬁnition, we have the following generation error bound:
Theorem 2 (Generalization Error Bound for SA-FL) .
For an SA-FL system with arbitrary system and data heterogeneity, if distributions P and Q satisfy Assumption 1 and are ( α, β ) -positively-related, then with probability at least 1 − δ for any δ ∈ (0 , 1) , it holds that
[Equation]
where d H denotes the ﬁnite VC dimension for hypotheses class H , and parameters { P, Q, n T , n S , β , β Q } are deﬁned the same as before.
It is known that (see, e.g., [27]) the generalization error bound of centralized learning is (hiding logarithmic factors) (cid:101) O
(( 1 n ) 1 2 − βQ ) with n samples in total and noise parameter β Q .
Note that when β ≥ 1 , the ﬁrst term in Eq.
(6) dominates.
Hence, Theorem 2 implies that the generalization error bound for SA-FL matches that of centralized learning (with dataset size n T + n S ).
Meanwhile, compared with solely training on server’s dataset T , SA-FL exhibits an improvement from (cid:101) O
(( 1 n T ) 1 2 − βQ ) to (cid:101) O (( 1 n [Equation]
β [Equation] βQ ) .
This highlights the beneﬁt of collaboration from the clients.
Note that SA-FL shares some similarity with the domain adaptation problem, where the learning is on Q but the results will be adapted to P .
In what follows, we offer some deeper insights between the two by answering two key questions: 1) What is the difference between SA-FL and domain adaptation (a.k.a. transfer learning)?
and 2) Why is SA-FL from Q to P PAC-learnable, but FL from D to P with incomplete client participation not PAC-learnable (as indicated in Theorem 1)?
To answer these questions, we illustrate the distribution relationships for domain adaptation and federated learning, in Fig. 1, respectively.
In domain adaptation, the target P and source Q distributions often have overlapping support but there also exists distinguishable difference .
In contrast, the two distributions P and Q in FL happen to share exactly the same support with different density, since Q is a mixture of D and P .
As a result, the known bounds in domain adaptation (or transfer learning) are pessimistic in FL.
For example, the dist ( P, Q ) in d A -divergence and Y -divergence both have non-negligible gaps when applied to FL.
Here in Theorem 2, we provide a generalization error bound in terms of the total sample size n T + n S , thus showing the beneﬁt of SA-FL.
[Figure]
P
D
Moreover, for FL, only the auxiliary dataset T i.i.d.
∼ P is directly available for the server.
The clients’ datasets could be used in FL training, but they are not directly accessible due to privacy constraints.
Therefore, previous methods in domain adaptation (e.g., importance weightsbased methods in covariate shift adaptation [28, 29]) are not applicable since they require the knowledge of density ratio between training and test datasets.
The key difference between FL and SA-FL lies in relations among D, P and Q .
For FL, the distance between
D and P under incomplete participation could be large due to system and data heterogeneity in the worst-case.
More speciﬁcally, the support of D could be narrow enough to miss some part of P , resulting in non-vanishing error as indicated in Theorem 1.
For SA-FL, distribution Q is a mixture of P and D ( Q = λ
1 D + λ 2 P , with λ 1 , λ 2 ≥ 0 , λ 1 + λ 2 = 1 ), thus having the same support with P .
Hence, under Assumption 4, the PAC-learnability is guaranteed.
Although we provide a promising bound to show the PAC-learnability of SA-FL in Theorem 2, the superiority of SA-FL over training solely with dataset T in server (i.e., (cid:101) O (( 1 n T ) 1 2 − βP ) ) is not always guaranteed as β → 0 (i.e., Q becomes increasingly different from P ).
In what follows, we reveal under what conditions between P and Q could SA-FL perform no worse than centralized learning in terms of generalization error.
Theorem 3 (SA-FL Being No Worse Than Centralized Learning) .
Consider an SA-FL system with arbitrary system and data heterogeneity.
If Assumption 1 holds and additionally ˆ R P
(ˆ h ∗ Q ) ≤ ˆ R P ( h ∗ Q ) and ε P ( h ∗ Q ) =
O ( A ( n T , δ )) , then with probability at least 1 − δ for any δ ∈ (0 , 1)
, it holds that ε P (ˆ h ∗ Q ) = (cid:101) O (cid:16) ( d H /n T ) 1 2 − βP (cid:17) , where A ( n T , δ ) =
d H n T log( n T d H + 1 n T log( 1 δ )) , and other parameters are as deﬁned in Theorem 2.
Here, we remark that ε P ( h ∗ Q )
=
O ( A ( n T , δ )) is a weaker condition than the ε P ( h ∗ Q ) =
0 condition and the covariate shift assumption ( P Y | X = Q Y | X ) used in the transfer learning literatures [30,31].
Together with the condition ˆ R P (ˆ h ∗ Q ) ≤ ˆ R P ( h ∗ Q ) , the following intermediate result holds: ˆ R P (ˆ h ∗ Q ) − ˆ R P ( h ∗ P ) =
O ( A ( n T , δ ))
(see Lemma 2 in the supplementary material).
Intuitively, this states that “if P and Q share enough similarity, then the difference of excess empirical error between ˆ h ∗ Q and h ∗ P on P can be bounded.”
Thus, the excess error of ˆ h ∗ Q shares the same upper bound as that of ˆ h ∗ P in centralized learning.
Therefore, Theorem 3 implies that, under mild conditions, SA-FL guarantees the same generalization error upper bound as that of centralized learning with dataset T (to see this, set n S = 0 and β = 0 in Eq.
(6)), hence being “no worse than” centralized learning with dataset T .
The SAFARI algorithm
In the previous section, we showed that SA-FL under arbitrary system heterogeneity is PAC learnable and its superiority to standard FL is guaranteed under mild conditions.
This indicates the existence of an algorithm to achieve PAC learnability for SA-FL.
However, it remains to design some efﬁcient algorithms for SA-FL with a comparable level of communication overhead as conventional FL.
In this section, we propose our SAFARI (server-aided federated averaging) algorithm for SA-FL and characterize its convergence guarantees.
As shown in Algorithm 1, SAFARI iteratively performs the following three steps: 1) Server samples a subset of clients as in conventional FL and synchronize the latest global model x t with each participating clients (Line 3).
Each participating client and the server train the model based on local dataset (Lines 4-8).
Speciﬁcally, client initializes its local model with x t and then performs K do 3: do 5:
Synchronization
Local
− 1 : x it,k +
= x
it,k − η ∇ F i ( x it,k , ξ it,k ) .
Send ∆ it
(cid:80) K − 1 k =0 ∇ F i ( x it,k , ξ it,k ) to server. 8: end for 9: do 10:
Local updates: k
K 1 : x 0 t,k +1 = x 0 t,k − η ∇ F ( x 0 t,k , ξ 0 t,k ) , ∆ 0 t = − (cid:80) K − 1 k =0 ∇ F ( x 0 t,k , ξ 0 t,k ) .
Receive ∆ it , i ∈ S t , and normalize it: ˆ∆ it =
c t ∆ it − ∆ 0 t (cid:107) ∆ it − ∆ 0 t (cid:107) .
Server Update: x
[Equation]
= [Equation] it (cid:17) . 13: end for 14: end for local steps by the stochastic gradient descent method.
Then, each client sends its locally accumulated update ∆ it back to the server.
Note the server simultaneously takes K local steps based on its auxiliary dataset (in Line 10).
3) Server aggregates and updates the global model (Lines 11-12).
Upon receiving the local update ∆ it , the server normalizes and rescales it by a hyper-parameter c t .
Then, the server updates the global model by aggregating the normalized update ˆ∆
it and the server’s update ∆ 0 t based on its own auxiliary dataset.
Compared to FedAvg [1] in FL, SAFARI shares the same communication and computation process from the client’s perspective.
Hence, it enjoys the same level of communication efﬁciency and privacy beneﬁts.
Assumption 2.
( L -Lipschitz Continuous Gradient)
There exists a constant L > 0 , such that [Equation]
Assumption 3.
(Unbiased Stochastic Gradient with Bounded Variance)
The stochastic gradient is unbiased, i.e., E [ ∇ F ( x , ξ )]
= ∇ F ( x ) and E [ (cid:107)∇ F ( x , [Equation] ) − [Equation]
F (
[Equation]
2 ] ≤ σ 2 .
With the assumptions above, we are now in the position to analyze the convergence of SAFARI .
Theorem 4 (Convergence Rate for SAFARI ) .
Under Assumptions 2 and 3, let constant learning rate η satisfy ( 12 − 4 LKη − 20 K
( L + 4 KL 3 η ) η 2 ) > 0 .
Then, the sequence { x t } generated by the
SAFARI algorithm satisﬁes:
[Equation]
where c is a constant and x ∗ denotes an optimal solution.
Theorem 4 implies an O (1 /T ) convergence rate to a neighborhood of a stationary point.
Furthermore, by choosing parameters { c t } and the learning rate η appropriately, we have the following convergence rate to a stationary point:
Corollary 1.
If (cid:80) T − 1 t =
c 2 t is bounded and learning rate η = 1 √ KT , the convergence rate of SAFARI is:
[Equation]
In Theorem 4 and Corollary 1, we show the convergence guarantee of SAFARI under no extra assumptions on the data and system heterogeneity (client participation), which corroborates the learnability analysis in Section 2.
Hence, in the worst-case scenarios, convergence rate O (1 / ( K 1 / 2 T 1 / 2 )) is achieved for sufﬁciently large T and K ≤ T .
In comparison, a non-vanishing error term emerges consistently for the same setting in FL [32].
This veriﬁes the superiority of SA-FL over conventional
FL with incomplete client participation.
Note that Corollary 1 requires a convergent series { c 2 t } .
This can be relaxed to (cid:80) T − 1 t =
c 2 t
= O (min { KT, K 3 / 2 T 1 / 2 } ) to maintain the same O (1 / ( K 1 / 2 T 1 / 2 )) rate.
It can be readily veriﬁed that p -series ( c t = t − p ) satisﬁes the condition.  Conclusion
Different from previous works that considered either full or uniform client participation scenarios in federated learning (FL), we considered in this paper a more practical scenario in FL with incomplete client participation.
By establishing a fundamental generalization error lower bound, we ﬁrst showed that conventional FL is not PAC-learnable under incomplete client participation.
To overcome this challenge, we proposed a new server-aided federated learning (SA-FL) framework with an auxiliary dataset deployed at the server, which is able to revive the PAC-learnability of FL under incomplete client participation. Upon resolving the PAC-learnability challenge, we proposed a new SAFARI (server-aided federated averaging) algorithm that enjoys convergence guarantee and the same level of communication efﬁciency and privacy protection as conventional FL. A Proofs
Theorem 1 (Impossibility Theorem) .
Let H be a non-trivial hypothesis space and L : ( X , Y ) ( m × n ) → H be the learner for an FL system.
There exists a client participation process
F , a distribution P , and a target function f ∈ H with min h ∈H R P
( h, f )
= 0 , such that
[Equation]
Proof.
Denote S the dataset with size
Mn i.i.d.
sampled from distribution P , F ( · ) the sampling process of FL system, and ¯ S = F ( S ) the training dataset selected by FL system with size mn .
Consider a distribution P with support on only two points { x 1 , x 2 } such that P P ( x 1 ) = 1 − 4 (cid:15) and P P ( x 2 ) = 4 (cid:15) with (cid:15) = 1 − α 8 .
First we show that the rare points x 2 appears at most (1 − α )
Mn times with constant probability.
Let ˆ s be the number of x 2 points in S , then ˆ s ∼ B
( Mn, (cid:15) ) is a binomial random variable.
By the
Chernoff bound,
[Equation]
So P [ˆ s < (1 − α )
Mn ] > 3 20 .
Next, we consider the following sampling process with dataset
S = { ( x (cid:48) 1 , f ( x (cid:48) 1 )) , . . .
, ( x (cid:48)
M × n , f ( x (cid:48)
M × n )) } : choosing as many data ( x (cid:48) i , f ( x (cid:48) i )) , i ∈ [ mn ] such that x (cid:48)
i = x 1 as possible to form the training set ¯ S .
Let f 1 , f 2 ∈ H be two target functions whose existence is guaranteed by the non-trivial deﬁnition of H and f 1 ( x 1 ) =
f 2 ( x 1 ) , f 1 ( x 2 ) = − f 2 ( x 2 ) , and S be the set of all datasets in ( X , Y )
( M × n ) such that ˆ s < (1 − α ) MN .
Let R ( h s , f ) =
P P [ L ( F ( S ))
( x ) (cid:54) =
f 1 ( x ) ∩ x (cid:54) = x 1 ] , the following holds for these two target functions f 1 and f 2 :
[Equation]
The above result hold in expectation since it holds for any S ∈ S .
Hence, there exists a target function f ∈ H such that E S ∈S R ( h s , f ) ≥ 2 (cid:15).
Note R ( h s , f ) ≤ P ( x (cid:54) = x 1 ) = 4 (cid:15) , then by decomposing the expectation into two parts we obtain:
[Equation]
, f ) P [ R ( h s , f )] ≤ 4 (cid:15) P S ∈S
[ R ( h s , f ) ≥ 4 (cid:15) ] + (cid:15) (1 − P S ∈S
[ R ( h s , f ) ≥ (cid:15) ])
= (cid:15) + 3 (cid:15) P S
[Equation]
[ R ( h s
[Equation]
f )
[Equation]
(cid:15) ] .
That is,
[Equation]
Note R ( h s , f ) =
P P [ L ( F ( S ))
( x ) (cid:54) =
f 1 ( x ) ∩ x (cid:54) = x 1 ] ≤ R ( L ( F ( S )))
=
P P [ L ( F ( S ))
( x ) (cid:54) =
f 1 ( x )] , then we have the ﬁnal results:
[Equation]
Theorem 2 (Generalization Error Bound for SA-FL) .
For an SA-FL system with arbitrary system and data heterogeneity, if distributions P and Q satisfy Assumption 1 and are ( α, β ) -positively-related, then with probability at least 1 − δ for any δ ∈ (0 , 1) , it holds that
[Equation]
where d H denotes the ﬁnite VC dimension for hypotheses class H , and parameters { P, Q, n T , n S , β , β Q } are deﬁned the same as before.
Proof.
[Equation]
Combining with Lemma 1, the proof is complete.
Lemma 1 (Auxiliary Lemma [25, 26, 30, 31]) .
For any m ∈ N and δ ∈ (0 , 1) , deﬁne A ( m, δ ) =
d H m log( md H + 1 m log( 1 δ ))
With probability at least 1 − δ , ∀ h, ˆ h ∈ H ,
[Equation]
where P S ( · ) =
E [ˆ P S ( · )] , S is the i.i.d.
dataset with size m drawn form distribution Q , c ∈ (0 , ∞ ) is a constant.
Theorem 3 (SA-FL Being No Worse Than Centralized Learning) .
Consider an SA-FL system with arbitrary system and data heterogeneity.
If Assumption 1 holds and additionally ˆ R P
(ˆ h ∗ Q ) ≤ ˆ R P ( h ∗ Q ) and ε P ( h ∗ Q ) =
O ( A ( n T , δ )) , then with probability at least 1 − δ for any δ ∈ (0 , 1)
, it holds that ε P (ˆ h ∗ Q ) = (cid:101) O (cid:16) ( d H /n T ) 1 2 − βP (cid:17) , where A ( n T , δ ) =
d H n T log( n T d H + 1 n T log( 1 δ )) , and other parameters are as deﬁned in Theorem 2.
Proof.
Without loss of generality, we use c serve as a generic constant since we focus on the order in terms of the sample number and thus omit the constant factor.
[Equation]
The ﬁrst inequality is due to Lemma 1 and second inequality follows from Lemma 2 and Noise assumption 1.
Then we have the following result, which completes the proof:
[Equation]
Lemma 2.
If ˆ R P (ˆ h ∗ Q ) ≤ ˆ R P ( h ∗ Q ) , with probability at least 1 − δ ,
[Equation]
Proof.
[Equation]
Theorem 4 (Convergence Rate for SAFARI ) .
Under Assumptions 2 and [Equation] let constant learning rate η satisfy ( 12 − 4 LKη − 20 K
( L + 4 KL 3 η ) η 2 ) > 0 .
Then, the sequence { x t } generated by the
SAFARI algorithm satisﬁes:
[Equation]
where c is a constant and x ∗ denotes an optimal solution
[Equation]
Proof.
Let ¯∆[-t = 
-]
[Equation]
Note that ∆ 0 t = − (cid:80) K − 1 k =0 ∇ F ( x 0 t,k , ξ 0 t,k ) .
We have
[Equation]
So
[Equation]
we can bound A 1 as following:
[Equation] [Equation]
where the ﬁrst inequality is due to assumption 1 and {∇ F ( x 0 t,k , ξ 0 t,k ) −∇ F ( x 0 t,k ) } form a martingale difference sequence (see Lemma 4 in [4]).
Hence, we can bound A 2 as following:
[Equation]
Plugging the bound of A 1 and A 2 into the smoothness inequality
[Equation]
we have:
[Equation]
For the server
[Equation]
we have the following results for the norm of parameter changes for one local computation:
[Equation]
[Equation]
Unrolling the recursion
[Equation]
we obtain the following:
[Equation]
Putting the pieces together
[Equation]
we obtain
[Equation]
The last inequality follows from that there exist such constant c if ( 12 − 4 LKη − 20 K ( L + KL 3 η ) η 2 ) >
Summing over t = 0 to T − 1 , we have
[Equation]
B Experiments
In this section, we provide the details of the numerical experiments and some additional experimental results.
B.1 Models and Datasets
We test the SAFARI algorithm by running two models on two different types of datasets, including 1) multinomial logistic regression (LR) on MNIST, and 2) convolutional neural network (CNN) on
CIFAR-10.
Both datasets are chose from a previous FL paper [1], and they are now widely used as benchmarks for FL research [7,33].
MNIST and CIFAR-10 have ten classes of images separately.
In order to impose the heterogeneity of the data, we partition the dataset according to the number of classes ( p ) that each client contains.
We distribute these data to M = 10 clients, and each client only has a certain number of classes.
Speciﬁcally, each client randomly selects p classes of images and then evenly samples training and test data-points within these p classes of images without replacement.
For example, if p = 2 , each client only samples training and test data-points within two classes of images, which causes the heterogeneity among different clients.
If p = 10 , each client contains training and test samples that selects from ten classes.
This situation is almost the same as i.i.d. case.
Hence, the number of classes ( p ) in each client’s local dataset can be used to represent the level of non-i.i.d.
qualitatively.
In addition, to mimic incomplete client participation, we enforce s clients to be exempt from participation, where the index s can be used to represent the degree of incomplete client participation.
Speciﬁcally, we
[Figure]
assume there are M = 10 clients in total, and m = 5 clients participate in each communication round.
These clients are uniformly sampled from M − s clients
Larger incomplete client participation index s means less clients participate in the training.
For both MNIST and CIFAR-10, the learning rate is 0.1, and the local epoch is 1.
For MNIST, the batch size is 64, and the total communication round is 150.
For CIFAR-10, the batch size is 500, and the total communication round is 4000.
To simulate the data heterogeneity, we use p =
[10 , 5 , 2 , 1] as a proxy to represent the degree of non-i.i.d.
on MNIST and CIFAR-10 datasets.
To emulate the effect of incomplete client participation, we set s =
[0 , 2 , 4] to represent the degree of incomplete client participation for the SAFARI algorithm, the FedAvg algorithm, and the SGD algorithm.
Last two algorithms are employed as the baselines to compare with our algorithm.
The hyper-parameter c t in the SAFARI algorithm is set to 0.1 both on MNIST and CIFAR-10.
To compare the effect of the collaboration from server, we add [50 , 100 , 500 , 1000] data to the server’s side for MNIST and [500 , 1000 , 5000] for CIFAR-10.
B.2 Additional Experimental Results
In Figure B.2, we show the test accuracy of FedAvg algorithm on CIFAR-10 for different Non-IID index p and incomplete client participation index s .
In the case of p = 10 , the test accuracy of s =and s = 0 is not much different whereas the test accuracy of s = 4 is 25% lower than that of s =in the case of p = 1 .
Incomplete client participation has no impact on the performance for nearly homogeneous data, but it causes catastrophical performance degradation for highly Non-IID data.
In Figure 3, we show the test accuracy of the SAFARI algorithm, the FedAvg algorithm, and the SGD algorithm on MNIST for incomplete client participation s = 4 and different Non-IID index p .
The evidences of the observations are provided visually as follows:
• Compared to FedAvg in the case of p = 1 (see Figure 3(d)), with only 50 data at server’s side (0.1% of the total training data), there is a non-negligible increase of test accuracy for our SAFARI algorithm.
This increase increases as more data is added to the server’s side.
• In nearly homogeneous case when p = 5 or p = 10 (see Figure 3(a) and 3(b)), there is actually no improvement of the test accuracy with these auxiliary data added to the server’s side, comparing SAFARI with FedAvg. • Compared SAFARI with SGD (for centralized learning solely on server’s data) in nearly homogeneous case when p = 5 or p = 10 (see Figure 3(a) and 3(b)) , the collaborations from clients signiﬁcantly improves the performance, especially with less data on the server’s side.
• In highly heterogeneous case when p = 2 or p = {+
+} (see Figure 3(c) and 3(d)), it shows no obvious improvement from the collaboration of clients comparing SAFARI to SGD.
[Figure]
[Table]
In Table 1, we show the comparison between our SAFARI algorithm and FedAvg algorithm on
CIFAR-10 for incomplete client participation s = 4 .
The observations are further illustrated: 1)
There is non-negligible increase of the test accuracy for SAFARI algorithm with small amount of auxiliary data at server’s side.
With 5000 data at server’s side, the test accuracy increases by 16.08%.
There is actually no improvement with these auxiliary data for nearly homogeneous case (e.g., p = 10 or p = 5 ), which is denoted by ‘-’ in the table.
In Table 2, we show the difference between our SAFARI algorithm and SGD, which is for centralized learning solely on server’s data, for incomplete client participation s = 4 on CIFAR-10.
When the size of data on server’s side is small, the collaborations from clients signiﬁcantly improve the performance of the SAFARI algorithm.
Even in the highly heterogeneous case when p = 1 , the test accuracy can be improved by 10% for only 500 data on the server’s side (0.8% of the total training data).
This observation further validates our theoretical analysis in Theorem 2.
[Table]