A Unified Analysis of Federated Learning with Arbitrary Client Participation
Anonymous Author(s)
AffiliationAddress email
Abstract
Federated learning (FL) faces challenges of intermittent client availability andcomputation/communication efficiency.
As a result, only a small subset of clientscan participate in FL at a given time.
It is important to understand how partial clientparticipation affects convergence, but most existing works have either consideredidealized participation patterns or obtained results with non-zero optimality errorfor generic patterns.
In this paper, we provide a unified convergence analysis for FLwith arbitrary client participation.
We first introduce a generalized version of federated averaging (FedAvg) that amplifies parameter updates at an interval of multiple
FL rounds.
Then, we present a novel analysis that captures the effect of clientparticipation in a single term.
By analyzing this term, we obtain convergence upperbounds for a wide range of participation patterns, including both non-stochasticand stochastic cases, which match either the lower bound of stochastic gradientdescent (SGD) or the state-of-the-art results in specific settings.
We also discussvarious insights, recommendations, and experimental results.
Introduction
We consider a federated learning (FL) problem with N clients [14, 17, 30].
Our goal is to minimize:
[Equation] [Equation]
where x ∈ R m is an m -dimensional model parameter.
The local objective F n ( x ) of each client n isusually defined as F n ( x ) := E ξ n ∼D n [ ℓ n ( x , ξ n )] , where D n is the distribution of client n ’s local data,which cannot be observed globally because the data remains private, and ℓ n ( x , ξ ) is the per-sampleloss function for parameter x and input data ξ .
The objective (1) can be extended to a weightedaverage, but we do not write out the weights and consider them as part of ℓ n ( x , ξ ) and F n ( x ) .
Acanonical way of solving (1) is to use federated averaging (FedAvg) [21], which is a form of stochasticgradient descent (SGD) that operates in multiple rounds, where each round includes multiple localupdate steps followed by communication between clients and a server to synchronize the updates.
Partial Participation.
A major challenge in FL is that clients only intermittently participate in thecollaborative training process [3].
The reason is twofold.
First, FL applications usually have a largepool of clients.
It is impractical to have all clients participating in all FL rounds, because it wouldrequire computation and communication that can consume a large amount of energy and also lead tonetwork congestion.
Second, clients may become unavailable from time to time.
For example, if theclient devices are smartphones, they may be willing to participate in FL only when they are charging(usually at night), to avoid draining the battery in outdoor environments where no charger is available.
Open Problem.
The convergence of FL algorithms with realistic client participation patternshas not been fully understood.
Most existing works on partial participation have only consideredidealized scenarios, such as those where clients are randomly selected according to a given probability
[Table]
distribution that is independent across rounds [4, 8, 9, 15, 18, 19, 28].
They cannot incorporate cases
[Table]
where clients are unavailable for a period of time.
Some recent results considering general scenarios
[Table]
either include a constant error term that does not vanish at convergence [5, 29] or are otherwise lesscompetitive than state-of-the-art results obtained in idealized settings [11, 27].
Therefore, we ask:
What classes of participation patterns have guaranteed convergence to zero error?2.
For these classes, can the convergence rate match with that obtained in idealized scenarios?
To answer these questions, we need to overcome two challenges.
First, we need to devise a unifiedanalytical methodology for obtaining the convergence rates under various participation patterns.
Second, we need a general and realistic algorithm for cross-device FL, with appropriate controloptions that can be configured to make the convergence rate competitive.
Our Contributions.
In this paper, we overcome these challenges and give answers to the twoquestions above.
We first introduce a generalized version of FedAvg, which amplifies the parameterupdates after every P rounds, for some P ≥ 1 .
When P = 1 , our setting is the same as FedAvg withtwo-sided learning rates [15, 28], but we focus on an extended setup where P can be greater thanone.
For this generalized FedAvg algorithm, we perform a new convergence analysis that unifies theeffect of partial participation into a single term ( ˜ δ 2 ( P ) in Section 3).
This allows us to decouple ourmain convergence analysis from the analysis on partial participation, which largely simplifies theanalytical procedure.
Our novel methodology gives the results shown in Table 1, for a broad range ofparticipation patterns.
These patterns include non-stochastic and regularized participation, whereall clients participate equally but not simultaneously within P rounds, and stochastic participationsettings including ergodic , stationary and strongly mixing (e.g., Markov process), and independent .
For the stochastic participation, we provide both expected and high probability convergence bounds,where the expected rates match the state-of-the-art FedAvg convergence rates that were derived forthe idealized setting of random participation following an independent distribution.
Furthermore, weprovide new insights related to non-IID data, amplification of updates, “linear speedup” [31], etc.
To summarize, our main contributions are as follows:
• We introduce a generalized FedAvg algorithm which amplifies the updates aggregated over
multiple rounds, where the amplification interval P can be tuned for the best convergence.•
We present novel analysis and unified methodology for obtaining competitive convergence
upper bounds with arbitrary client participation patterns.
• We discuss important insights from both the theoretical and experimental results.
Related Work.
FedAvg [21] is characterized by partial client participation and multiple local updatesin each round.
In the case of full participation, FedAvg is also known as local SGD [10, 12, 20, 23–25]and parallel restarted SGD [32].
Partial participation was considered in [4, 8, 9, 15, 18, 19, 28], withthe assumption that clients are selected to participate probabilistically according to a given distributionthat is independent across rounds.
Some recent works have started to incorporate unavailable clients.
For example, the work in [22] allows inactive clients, but the theoretical result does not guaranteeconvergence if a client participates with zero probability in a certain round.
Similarly, the result in[5, 29] includes a constant error term in the case of general participation in cross-device FL.
Otherworks have obtained convergence rates related to the maximum number of inactive rounds, basedon additional assumptions such as Hessian Lipschitz [11] and bounded gradient norm [27].
Theirtheory also requires all clients to participate once at the beginning (see [11, Remark 5.2]).
In contrast,our result in this paper is based on a minimal set of assumptions.
We consider a simple algorithmwithout requiring initial full participation, and we provide a better rate of convergence to zero error(see
Table 1).
Moreover, existing methods usually have separate analyses and results for each specificclass of participation pattern, whereas we present a unified framework that largely simplifies theanalytical procedure.
Some more discussions on related works are in the supplementary material.
Generalized FedAvg with Amplified Updates
We consider a generalized version of FedAvg as shown in Algorithm 1.
In this algorithm, we defineq nt to be the participation weight of client n in round t .
If q nt = 0 , the client does not participate inthis round.
Each client n participates whenever q nt > 0 , where the weight q nt is applied in the globalupdate step in Line 9.
For mathematical convenience in our analysis, we assume in Algorithm 1 thatall clients compute their local updates in Lines 4–8, regardless of whether they participate in thecurrent round or not.
We emphasize that this is logically equivalent to the practical setting wherenon-participating clients do not compute, because their computations have no effect in subsequentrounds when q nt =
(see Line 9).
In every round t , each client n computes I steps of local updatesaccording to Line 7, where γ > 0 is the local learning rate and g n ( y nt,i ) is the stochastic gradient of
F n ( y nt,i ) such that E (cid:2)
[Equation]
y nt,i ) .
Starting at round t 0 , the global updates are accumulated in u for P rounds (Line 10).
At the end ofevery interval of P rounds, an amplification is applied to the accumulated updates (Line 12).
Since x tis first updated without amplification in rounds t 0 , . . .
, t 0 + P − 1 , Line 12 adds u after multiplyingη − 1 , which is equivalent to setting x t +1 to x t 0 + η u .
When η = 1 , there is no amplification and
Line 12 has no effect in this case.
In general, we allow any η > 0 including values of η that are lessthan one.
When η < 1 , we reduce (instead of amplify) the updates.
Intuitively, amplification withη > 1 is usually preferred over reduction, because we can compute the gradients on similar modelparameters in each round where only a small number of clients participates.
After P rounds, theaccumulated updates provide a better estimation for the overall client population, and amplifyingthe updates allows the model parameter to progress towards the descent direction for the majorityof clients.
We also note that although P is a parameter in our algorithm, the dominant terms of ourconvergence upper bounds shown in Table 1 do not depend on P .
Algorithm 1: with participation 1 Input: γ , η , x 0 , I , P , T ; Output: { x t : ∀ t } 2 Initialize t 0 ← 0 , u ← 0 ; 3 for t ← 0 , . .
, T − 1 do for n ← 1 , . . .
, N in parallel do y nt, 0 ← x t ; for i ← 0 , . . .
, I − 1 do y nt,i +1 ←
y nt,i − γ g n ( y nt,i ) ∆ nt ← y nt,
I − x t ; x t +1 ← x t + (cid:80) Nn =1 q n t ∆ n t ; //update u ← u + (cid:80)
Nn =1 q nt ∆ nt ; //accumulate
if t + 1 − t 0 =
P then x t +1 ← x t +1 + ( η − 1) u ; //amplify t 0 ← t + 1 ; u ← 0 ;
Convergence Analysis and Main Result
We analyze Algorithm 1 as follows.
For the ease of analysis, we assume that (cid:80) N n =1 q n t = 1 forq nt ≥ 0 for all t .
1 We also define ρ such that (cid:80) Nn =1 ( q nt ) 2 ≤ ρ 2 for all t .
Hence, there always exists ρ such that ρ 2 ≤ 1 .
We define Q :=
{ q nt : ∀ n, t } .
Mathematically, we use the conditional expectation 108 E [ ·| Q ] to denote the case where Q is given and the expectation is only over the stochastic gradientnoise.
In the case where Q is stochastic, the full expectation E [ · ] is taken over both the noise and Q .
We make the following minimal set of assumptions that are commonly used in the literature.
Assumption 1 (Lipschitz gradient) .
[Equation]
Assumption 2 (Unbiased stochastic gradient with bounded variance)
[Equation]
Assumption 3 (Bounded gradient divergence) .
[Equation]
The gradient divergence in Assumption 3 is related to the degree of non-IID data distribution acrossclients.
To better interpret how different divergence components affect the convergence, we introducean alternative set of divergence bounds in Assumption 3’ as follows.
We will show later (in Section 4)that if Assumption 3 holds, then Assumption 3’ also holds with properly chosen ˜ β 2 , ˜ ν 2 , and ˜ δ 2 ( P ) .
Assumption 3’ (Alternative gradient divergence bound) .
[Equation]
In the above, ˜ β 2 and ˜ ν 2 capture the gradient divergence in an arbitrary round t , and ˜ δ 2 ( P ) capturesthe divergence of time-averaged gradients over P rounds, which is a function of P .
In particular, ˜ β 2is an upper bound of the divergence between the original objective’s gradient ∇ f ( x ) and the averagedgradient among participating clients weighted by { q tn } .
When all the clients participate, we haveq nt = 1 N and (5) holds with ˜ β 2 = 0 .
The quantity ˜ ν 2 is an upper bound of the divergence betweeneach client’s gradient and the averaged gradient of participating clients, where the average is alsoweighted by { q tn } .
We will show in Section 4.1 that the overall divergence d 2 in Assumption 3 can beexpressed as a sum of ˜ β 2 and ˜ ν 2 , as intuition suggests.
The quantity ˜ δ 2 ( P ) in (7) extends ˜ β 2 in (5) bycomputing the average over P rounds (inside the norm) instead of a single round.
When the weights{ q tn } are properly chosen, as P gets large, the average q tn over P rounds gets close to each other fordifferent n , and ˜ δ 2 ( P ) becomes small.
We will formally analyze this behavior in Section 4.2.
In the following, let F := f
( x 0 ) − f ∗ , where f ∗ is the true minimum of (1), i.e., f ∗ :=
min x f ( x ) .
Theorem 3.1.
When Assumptions 1, 2, and 3’ hold, γ ≤ 1 12 LIP , γη ≤ 1 LIP , and P ≤ T 2 , we havemin t E (cid:104) ∥∇
[Equation]
≤ O
[Equation]
γηIT
[Equation]
Proof Sketch.
We first use smoothness to relate f ( x t 0 + P ) and f ( x t 0 ) , which includes an innerproduct term that can be expanded into several terms.
One of these terms includes ∇ F n ( y nt,i ) −∇ f
( x t 0 ) .
We upper bound this term with three other terms including (cid:13)(cid:13)(cid:13)
y nt,i − (cid:80) Nn
′ =
1 q n ′ t y n ′ t,i (cid:13)(cid:13)(cid:13) 2 ,(cid:13)(cid:13)(cid:13) (cid:80) N n ′ =1 q n ′ t y n ′ t,i − x t 0 (cid:13)(cid:13)(cid:13) 2 , and ˜ δ 2 ( P ) .
The upper bounds of the first two terms are found by solvingrecurrence relations, where the recurrence for the second term includes the first term.
This nestedrecurrence is a uniqueness in this proof.
The full proof is given in the supplementary material.
For two specific (but different) learning rate configurations, we can obtain the following corollaries.
Corollary 3.2.
Choosing γ = 1 12 LIP √ T and η = min (cid:110) 12 P √ LI F σρ ; 12 √ T (cid:111) , for P ≤ T 2 , we have
[Equation]
t
[Equation]
Corollary 3.3.
If
[Equation]
choosing γ =
[Equation] [Equation]
and η
[Equation]
P
[Equation]
LI F
[Equation]
, for P ≤ T 2 , we have
[Equation]
These two corollaries are key building blocks of our unified convergence analysis framework .
Thechoice between them depends on whether and how fast P grows with T .
We will also see in the nextsection that ˜ δ 2 ( P ) is key in capturing the effect of different participation patterns.
After deriving˜ δ 2 ( P ) such that (7) holds, plugging it back to one of the above corollaries gives the results in Table 1.
Interpreting and Applying the Unified Framework
4.1 Discussion on ˜ β 2 and ˜ ν 2 : Decomposition of Divergence
In Corollaries 3.2 and 3.3, we have an interesting observation that the term involving ˜ ν decreaseswith P , whereas the term involving ˜ β does not decrease with P .
This suggests that when P is large,the divergence term ˜ β has the main effect on convergence.
We recall that ˜
ν captures the divergencebetween the gradient of each client and the average gradient of clients that participate in a round t .
Since the learning rate γ is inversely proportional to both I and P , a large P gives a smaller learningrate which makes the divergence (related to ˜ ν ) after I iterations in each round less significant.
Forthe term with ˜ β , increasing P does not have the same effect, because parameter averaging is onlyconducted across the participating clients instead of all the clients, and ˜ β captures the divergence withrespect to all the clients.
The effect of all clients’ participation pattern is captured by the term ˜ δ 2 ( P ) .
From Assumption 3, by adding and subtracting (cid:80) Nn ′ =1 q n ′ t ∇ F n ′ ( x ) inside the norm and expandingthe square, we can obtain
[Equation]
because (cid:80) Nn =1 q nt =
and the inner-product term is zero.
The two terms in the right-hand side(RHS) of (11) are equal to the left-hand side (LHS) of (5) and (6), respectively.
We thus have thefollowing result showing that ˜ β 2 and ˜ ν 2 can be seen as a decomposition of the original divergence d 2 .
Proposition 4.1.
There exist ˜ β 2 and ˜ ν 2 , such that ˜ β 2 + ˜ ν 2 ≤ d 2 while satisfying (5) and (6) .
4.2 Discussion on ˜ δ 2 ( P ) : Effect of Partial Participation
The results in Theorem 3.1 and Corollaries 3.2 and 3.3 include a term of ˜ δ 2 ( P ) .
To guaranteeconvergence to zero error, ˜ δ 2 ( P ) needs to be either equal to zero or decrease in T .
Condition for ˜ δ 2 ( P ) = 0 .
From (1), we know that 0 = 1 N (cid:80) Nn =1 F n ( x ) − f ( x ) =
1 N (cid:80) Nn
=1 ( F n ( x ) − f ( x )) .
Hence,
[Equation]
for an arbitrary constant µ t 0 , which can possibly depend on t 0 .
Then, we obtain
[Equation]
where the last equality is due to (12).
Since (13) is equal to the LHS of (7), we have the following.
Proposition 4.2 (Regularized participation) .
If 1 P (cid:80) t
+ P − 1 t
=
tq nt − µ t
= 0 for some µ t 0 and alln ∈ { 1 , . . .
, N } , t 0 ∈ { 0 , P, 2 P, . . .
} , then (7) holds with ˜ δ 2 ( P ) = 0 .
This implies that if P is chosen so that the averaged participation weights over every interval of
P rounds ( 1 P (cid:80) t 0 + P − 1 t
=
tq nt ) are equal to each other among all the clients n ∈ { 1 , . .
. , N } , i.e.,regularized, then ˜
δ 2 ( P ) = 0 .
Together with Corollary 3.2 and Proposition 4.1, we have the following.
Corollary 4.3 (Regularized participation) .
If the condition of Proposition 4.2 holds for P ≤ T 2 , andwe choose the learning rates according to Corollary 3.2, then
[Equation]
If Proposition 4.2 holds for a finite P that does not depend on T , we obtain a convergence rate of
O (cid:16) σρ √ IT + 1+ σ 2 T (cid:17) for T ≥ 2 P , where all the other constants are ignored in the O ( · ) notation here.
For full gradient descent with σ 2 = 0 , this convergence rate is improved to O (cid:0) 1 T (cid:1) .
Interpreting ˜ δ 2 ( P ) Using Variance.
The condition for ˜ δ 2 ( P ) = 0 may be too stringent for somepractical scenarios of FL.
Next, we show that ˜ δ 2 ( P ) can be expressed as the variance of clientparticipation weights averaged over P rounds.
We first express ˜ δ 2 ( P ) with d 2 by further bounding:
[Equation]
[Equation]
where we first use Jensen’s inequality on the sum over n in (13), then move the term q nt 0 − µ t 0 to theoutside of the norm, where we define q nt 0 :
= 1 P (cid:80) t
+ P − 1 t = tq nt for any n ∈ { 1 , . . .
, N } , and afterwardsbound ∥∇ F n ( x ) − ∇ f ( x ) ∥ 2 by d 2 according to Assumption 3.
We note that (7) holds if the RHSof (15) is upper bounded by ˜ δ 2 ( P ) .
Therefore, we can obtain ˜ δ 2 ( P ) by analyzing the statisticalproperties of q nt 0 − µ t 0 and then choosing ˜ δ 2 ( P ) to be equal to the RHS of (15).
In the following, we assume that µ t 0 is chosen such that µ t
= E [ q nt ] for all n ∈ { 1 , . . .
, N } andt ∈ { t 0 , t 0 + 1 , . .
, t 0 + P − 1 } .
This implies that the mean participation weights of all the clientswithin the same cycle of P rounds are equal.
Note that this condition is the same as unbiased clientsampling in existing works [8, 9, 15, 18, 19, 28], because we consider an unweighted average in (1)and any weighting is included in the local objective function F n ( x ) .
However, differently from theseexisting works, we do not assume independence here (we will only assume independence in a specialcase later).
Under this condition, it is apparent that Var (cid:0) q nt 0 (cid:1) =
E (cid:104)(cid:0)
q nt 0 − µ t 0 (cid:1) 2 (cid:105) is the variance ofq nt averaged over P rounds, for any n ∈ { 1 , . .
. , N } .
We immediately have the following result.
Proposition 4.4 (Ergodic participation) .
If { q nt : ∀ t } is a mean-ergodic process for any n ∈{ 1 , . .
. , N } , which means that lim P →∞ E (cid:104)(cid:0)
q nt 0 − µ t 0 (cid:1) 2 (cid:105) = 0 for any n and t 0 , then there exists˜ δ 2 ( P ) such that lim P →∞ ˜ δ 2 ( P ) = 0 while satisfying (7) in expectation.
Next, we consider the class of stationary and strongly mixing processes.
Informally, a random processis said to be strongly mixing with coefficient α ( P ) if the outcomes that are at least P steps apart arenearly independent, where the distance between the joint probability distribution of the outcomes andtheir independent counterpart is at most α ( P ) .
See [2, Sec. 27] for a formal definition.
A specificexample of stationary and strongly mixing processes is finite-state irreducible and aperiodic Markovchains, which have an α ( P ) that exponentially decreases in P [2, Thm. 8.9].
It is known that thefollowing (generalized) central limit theorem holds for stationary and strongly mixing processes,where we adopt the result to our participation weights.
Lemma 4.5 ([2, Thm. 27.4]) .
If { q nt : ∀ t } is stationary and strongly mixing with α ( P )
=
O ( P − 5 ) ,for any n ∈ { 1 , . . .
, N } , then
q nt 0 ∼ N (cid:16) µ t 0 , ˆ υ 2 P (cid:17) for any t 0 when P → ∞ , where N ( · , · ) denotesthe normal distribution and ˆ υ 2 :=
Var( q nt 0 ) + 2 (cid:80) ∞
p
=
Cov( q nt 0 , q nt 0 + p ) .
From the definition of strongly mixing, we know that Cov( q nt 0 , q nt 0 + p ) in the definition of ˆ υ 2 approaches zero as p gets large, hence (cid:80)
∞
p =1 Cov( q nt 0 , q nt 0 + p ) converges to a finite value.
When P → ∞ and α ( P ) =
O ( P − 5 ) , Lemma 4.5 shows that Var (cid:0) q
nt 0 (cid:1)
=
ˆ
[Equation]
2 P .
Chebyshev’s inequality gives
[Equation]
Noting (15), we have the following, where “w.p.” stands for “with probability”.
Proposition 4.6 (Stationary and strongly mixing participation) .
If { q nt : ∀ t } is stationary and stronglymixing with α ( P )
=
O ( P − 5 ) , for any n ∈ { 1 , . . .
, N } , then as P → ∞ :
Choosing ˜ δ 2 ( P ) =
N 2 d 2 ˆ
υ 2 P satisfies (7) in expectation; ˜ δ 2 ( P ) =
N 2 d 2 ˆ
[Equation]
2 cP satisfies (7) w.p. 1 − c .
Following a similar procedure but applying Hoeffding’s inequality [13] instead of Chebyshev’sinequality, we can obtain the following result for independent participation, which does not require
P → ∞ .
We define υ 2 such
that Var( q nt ) ≤ υ 2 for any t and n .
Proposition 4.7 (Independent participation) .
If { q nt : ∀ t } is independent across t , for any n , then:
Choosing ˜ δ 2 ( P ) =
N 2 d 2 υ 2 P satisfies (7) in expectation; ˜ δ 2 ( P ) =
N 2 d 2 ln( 2 / c ) 2 P satisfies (7) w.p. 1 − c .
Note that the independence here is only assumed across t , so q nt and q n ′ t for the same t but n ̸ = n ′can still be dependent of each other.
Compared to Proposition 4.6, the high-probability bound in
Proposition 4.7 has c in the logarithmic term, which is tighter since 12 ln (cid:0) 2 c (cid:1) ≤ 1 c − 12 (ignoring ˆ υ 2 ).
Choosing P as a Function of T .
The choices of ˜ δ
( P ) in Propositions 4.6 and 4.7 include P in thedenominator.
Hence, we can guarantee convergence to zero error if we choose P to be an increasingfunction of T , which also ensures P → ∞ as T → ∞ .
If we choose P ∝ N 52 √
IT , we can obtainthe following convergence rate from Corollary 3.3 while noting Proposition 4.1.
Corollary 4.8 (Stochastic participation) .
If the conditions of either Proposition 4.6 or Proposition 4.7hold, and we choose P
= ˆ υ 2 N 52 √ IT or P =
υ 2 N 52 √ IT , respectively, and the learning ratesaccording to Corollary 3.3, then the convergence error in expectation over Q satisfies
[Equation]
for I ≤ ρ [Equation] N [Equation]
[Equation]
L F and T → ∞ when using conditions of Proposition 4.6, or I ≤
[Equation]
2 N [Equation] 2 √ L F and
T ≥ 4 υ 4 IN 5 when using condition of Proposition 4.7.
Remark.
Although the upper bound of I in Corollary 4.8 may appear restrictive, note that the optimalsolution x ∗ to (1) remains the same when the objective f ( x ) is scaled by a multiplicative positiveconstant, which in turn scales √ L F .
Thus, we can potentially allow arbitrarily large I by scalingf ( x ) without changing the optimal solution.
See supplementary material for further discussion.
While Corollary 4.8 focuses on the convergence in expectation, similar results can be obtained for thehigh-probability cases as well (see Table 1).
In general, since (7) is used as an upper bound in theproof of Theorem 3.1, if ˜ δ 2 ( P ) is chosen so that (7) holds in expectation or with a certain probability,such as in Propositions 4.4, 4.6 and 4.7, then the final convergence result also holds in expectation orwith a certain probability, respectively.
Discussions and Insights
Achieving Regularized Participation in Practice.
Regularized participation requires that q nt is equal to each other for all the clients n ∈ { 1 , . . .
, N } , for a properly chosen P .
When all the clients are connected to the system and client sampling is performed to limit the computation and communication overhead, this condition holds when the clients are selected accordingto a permutation, i.e., all the clients participate once before the same client can be selected again.
In the case where clients get disconnected from time to time, the system can properly schedule theparticipation of connected clients and their weights { q nt } , so that within a cycle of P rounds, all theclients participate with equal averaged weights.
In practice, it can be sufficient if this regularizedcondition is approximately satisfied, especially when multiple clients have similar datasets.
Interpretation of “Linear Speedup”.
Linear speedup is a desirable property seen in existing worksthat consider idealized client participation [28, 31, 32].
It essence, it means that the same convergence error can be achieved by increasing the number of participating clients S (with S ≤ N ) and reducingthe number of rounds T , while keeping the product ST unchanged.
In our case of arbitrary clientparticipation, the coefficient ρ is a generalization of the 1 / √ S term for linear speedup in existingworks that select a fixed number of S clients in each round.
When S clients participate with equalweight, because (cid:80) Nn =1 q nt = 1 , we have q nt = 1 / S for the participating clients and q nt = 0 for thenon-participating clients, so ρ = (cid:2) (cid:80)
Nn =1 ( q nt ) 2 (cid:3) 1 / 2 = 1 / √ S .
Plugging this ρ back to the convergenceresults, we achieve a convergence rate of O ( σ / √ SIT ) in Corollary 4.3 and O ( (1 + σ 2 ) / √ SIT ) in
Corollary 4.8, for sufficiently large T while ignoring the other variables in O ( · ) , as shown in Table 1.
This shows that we can achieve linear speedup in S in this special case.
We can also generalizeto having at least S min clients participating with equal weight in each round.
Following the sameargument, we have ρ ≤ 1 / √ S min in this case, giving a linear speedup in S min .
In general, we achievea linear speedup factor of 1 / ρ 2 in the case of arbitrary participation.
Matching Lower Bound or State-of-the-Art Results.
We continue to assume sufficiently large
T .
For regularized participation (Corollary 4.3), our convergence rate of O ( σ / √ SIT ) matches thelower bound of convergence error for centralized SGD [1].
The lower bound states that to reacha convergence error of ϵ , there exists an objective function such that at least Ω ( σ 2 / ϵ 2 ) stochasticgradient oracle calls are required.
2 We have SIT oracle calls to reach ϵ
= O ( σ / √ SIT )
according toour result, thus our upper bound matches this lower bound and is asymptoticlly optimal.
Note thatour upper bound still matches the lower bound if we include the coefficient √ L F in O ( · ) and L F inΩ ( · ) .
For stochastic participation (Corollary 4.8), our convergence rate of O ( (1 + σ 2 ) / √ SIT ) matchesstate-of-the-art results of FedAvg [8, 15, 28] that were obtained for the more idealized case of clientsbeing selected to participate according to a specific independent sampling scheme.
Comparison to Waiting for All Clients.
We consider a baseline where all clients wait for P roundsbefore proceeding with SGD.
In every iteration i of round t within these P rounds, each participatingclient n may call the stochastic gradient oracle to obtain g n ( x t ) , but it does not perform local updates.
Instead, it accumulates and averages the sampled instances of g n ( x t ) .
SGD is performed once afterthese P rounds using the average g n ( x t ) from each client n .
Assume that P is large enough so thateach client n computes g n ( x t ) at least M ≥ 1 times that are averaged afterwards.
If the M instancesof g n ( x t ) are all independent from each other, the noise σ 2 (defined in Assumption 2) becomes σ 2 / M .
This gives a convergence rate of O ( σ √ P / √ MNT ) for regularized participation with this baseline,where we have replaced σ and SIT in our bound O ( σ / √ SIT ) with σ / √ M and P / NT , respectively,because all N clients perform SGD once in P rounds now, so there are T / P updates in total.
In general, M cannot be larger than the total number of minibatches in one epoch on each client’sdataset, because otherwise the stochastic gradients are no longer independent from each other andsampling more gradients at the same parameter x t cannot further reduce the noise. For a simplesetting where each client has D minibatches in an epoch, we need to have M ≤ D .
Assume thatall N clients participate for an equal number of rounds, and S clients participate in each round (asabove). The maximum number of stochastic gradient computations in this case is SIP for all Nclients, so we also need to have NM ≤ SIP .
Therefore, M ≤ min { D ; SIP / N } .
When D ≥ SIP / N ,choosing M = SIP / N gives the same rate of O ( σ / √ SIT ) as with Algorithm 1.
However, when
D < SIP / N , which can frequently occur in FL settings where each client only has a small amount ofdata, the convergence rate of this “wait-for-all” baseline is worse than that of Algorithm 1.
Experiments
We ran experiments of training convolutional neural networks (CNNs) with FashionMNIST [26] and
CIFAR-10 [16] datasets, each of which has images in 10 different classes.
We set the total number ofclients to N = 250 .
Similar to existing works [6, 7, 28], we partition the data into clients so that eachclient has data of one majority class label, with 5% of data of other (minority) labels, to simulate asetup with non-IID data distribution which is often encountered in FL scenarios.
We assume thatthe clients’ availability exhibits a periodic pattern inspired by [6, 7].
Namely, in the first 100 rounds,only clients with the first two majority labels are available; in the next 100 rounds, only clients withthe next two majority labels are available, and so on.
In each round, S = 10 clients participate in a regularized manner, out of all the currently available clients.
To speed up initial rounds of training,similar to practical deep learning implementations, we start with standard FedAvg with a relativelylarge initial learning rate.
The initial rates are γ = 0 .
1 and γ = 0 .
without amplification (i.e.,η = 1 ) for FashionMNIST and CIFAR-10, respectively, which were obtained using grid search in aseparate scenario of always participation.
After an initial training of 2 , 000 rounds for FashionMNISTand 4 , 000 rounds for CIFAR-10, we study the performance of different approaches with their ownlearning rates.
We show the results of Algorithm 1 both with and without amplification.
Whenusing amplification, we set η = 10 and P = 500 .
We also compare to a baseline that waits for allthe clients, related to our discussion in Section 5, for two settings where each client computes itsgradients either on a minibatch or on the entire dataset, which we refer to as “wait-minibatch” and“wait-full”, respectively.
The best learning rate of each approach was separately found on a grid thatis { 1 , 0 . 1 , 0 . 01 , 0 . 001 , 0 . 0001 } times the initial learning rate, with the periodic participation patterndescribed above.
Additional setup details and results are given in the supplementary material.
The results are shown in Figure 1.
We make a few key observations as follows.
First, we clearly seethat Algorithm 1 with amplification gives the best performance .
By choosing P = 500 , we match withthe participation period of clients, as described in the setup above.
This corresponds to our settingwith regularized participation.
In this case, Algorithm 1 with amplification allows different groups ofclients to make small progress among themselves first.
After P rounds, each client has made its ownshare of contribution, and the updates get amplified so that the model parameter progresses fastertowards the direction that is overall beneficial for all the clients.
As our theory predicts, this approachgives a desirable convergence rate, which is now verified by experiments.
Next, we observe that the“wait-minibatch” and “wait-full” baselines perform worse than Algorithm 1 with amplification .
Thisobservation aligns with our discussion in Section 5 for the case where each client only has a smallamount of training data.
It is also interesting to note that “wait-full”, which computes the gradient onthe entire dataset, does not provide substantial gain compared to “wait-minibatch” in the experiments.
Although this is different from the theoretical analysis that shows stochastic gradients computed onmore data points have less noise, this behavior is often experienced in practice, because having somedegree of noise can be helpful for escaping from saddle points and local minima.
Finally, we observethat standard FedAvg (i.e., Algorithm 1 without amplification) gives the lowest performance .
This isbecause without amplification, the algorithm cannot put more emphasis on the collective updates bythe cohort of all clients, and the model parameter tends to diverge from the overall optimal direction.
Conclusion
In this paper, we have studied FL with arbitrary client participation.
For a generalized FedAvgalgorithm that amplifies parameter updates every P rounds, we have conducted a comprehensiveconvergence analysis and obtained convergence rates for a variety of client participation patterns.
Our findings suggest that regularized participation with finite P gives the best performance andmatches the lower bound of SGD convergence error when T is sufficiently large.
For stochasticparticipation, first, we have formally proven that convergence is guaranteed when the participationprocess is ergodic.
Then, for two generic classes of participation processes, we have proven thatwith a properly chosen P , our convergence rate matches state-of-the-art FedAvg convergence ratesthat were derived for the idealized case of independent and unbiased participation.
The experimentshave also verified some of our theoretical insights. Our current experiments are mainly for a basicverification of the theory. Since the scenario of arbitrary participation is complex also from a practicalperspective, an advanced empirical study can be an interesting direction for future work.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] Limitations are described where
necessary.
More discussions are also available in the supplementary material.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
This is a
foundational work that is widely applicable to various domains.
The societal impact
would depend on the specific application, which we believe is beyond the scope of thiswork.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes] Included in the
supplementary material.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] Included in thesupplementary material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes] More details are included in the supplementary material.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] The shaded areas in the figures represent the standard
deviation.
More details are explained in the supplementary material.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] Included in the supplementary
material.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes] Mentioned in the supplementary
material.
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects.
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]