L INEAR C ONVERGENCE OF D ECENTRALIZED F ED A VG FOR N ON -C ONVEX O BJECTIVES : T HE I NTERPOLATION R EGIME
Anonymous authors Paper under double-blind review
A BSTRACT
In the age of Bigdata, Federated Learning (FL) provides machine learning (ML) practitioners with an indispensable tool for solving large-scale learning problems.
FL is a distributed optimization paradigm where multiple nodes each having access to a local dataset collaborate (with or without a server) to solve a joint problem.
Federated Averaging (FedAvg) although the algorithm of choice for many
FL applications is not very well understood especially in the interpolation regime, a common phenomenon observed in modern overparameterized neural networks.
In this work, we address this challenge and perform a thorough theoretical performance analysis of FedAvg in the interpolation regime.
Specifically, we analyze the performance of FedAvg in two settings: (i) Server : When the network has access to a server that coordinates the information sharing among nodes, and (ii) Decentralized: The server-less setting, where the local nodes communicate over an undirected graph.
We consider a class of non-convex functions satisfying the Polyak-Lojasiewicz (PL) condition, a condition that is satisfied by overparameterized neural networks.
For the first time, we establish that FedAvg under both Server and Decentralized settings achieves linear convergence rates of
O ( T 3 / 2 log(1 /ϵ )) and O ( T 2 log(1 /ϵ )) , respectively, where ϵ is the desired solution accuracy, and T is the number of local updates at each node.
In contrast to the standard FedAvg analysis, our work does not require bounded heterogeneity, variance, and gradient assumptions.
Instead, we show that sample-wise (and local) smoothness of the local loss functions suffices to capture the effect of heterogeneity in FL training.
Finally, we conduct experiments on multiple real datasets to corroborate our theoretical findings.
I NTRODUCTION
Federated learning (FL) is a distributed machine learning scenario which allows the edge devices to learn a shared model while maintaining the training data decentralized at the edge devices Koneˇcn`y et al.
; McMahan et al.
This avoids the need to share the data with a central server and hence preserves privacy of the individual clients (edge devices).
Assuming a supervised learning setting, where each of the N clients having access to some local data ( x , y ) ∼ D k from distribution
D k with k ∈ [ N ] aim to solve the FL Problem:
min w ∈ R d
Φ( w ) :
= 1 N (cid:80) Nk =
1 Φ k ( w ) ,
where Φ k ( w ) :
=
E ( x ,y )
∼D k l k ( f w ( x ) , y ) is the average loss at the client k ∈ [ N ] for the input feature vector x ∈ X , and the corresponding output label y ∈ Y .
Here, f w ( x ) is the output of the model parameterized by w ∈ R d .
The de-facto standard for solving the above FL Problem is the simple Federated Averaging (FedAvg) algorithm McMahan et al.
In recent years, many works have attempted to characterize the convergence of FedAvg under different settings Stich (2018);
Li et al.
; Woodworth et al.
(2020a)
; Ma et al.
Yu et al. (2019b).
For example, the authors in Stich (2018) show a convergence rate of O (1 /Nϵ ) for minimizing strongly convex functions while Haddadpour & Mahdavi (2019) establishes similar rates for minimizing functions satisfying Polyak-Lojasiewicz (PL) condition.
Here, ϵ refers to the desired solution accuracy.
For minimizing non-convex smooth functions,
FedAvg achieves a convergence rate of O (1 /Nϵ 2 )
Karimireddy et al.
(2020b)
; Yang et al.
However, in practice, it has been observed that FedAvg converges at a much faster rate compared to the rates demonstrated in these works.
To illustrate this fact, in Fig.
1 we plot the behavior of the training loss (on a log scale) as a function of communication rounds for FedAvg to solve classification tasks on MNIST, FMNIST and CIFAR- 10 data sets (for experimental details please see
Section 4).
It is clear from the plots that the losses decrease linearly as a function of communication rounds.
This implies that the standard analyses of FedAvg lacks theoretical explanation of this linear convergence as shown in Fig.
In this work, we attempt to fill this gap and perform a thorough theoretical analysis of FedAvg in the interpolation regime under two settings; the Server setting, when the network has access to a server that coordinates the information sharing among clients, and the Decentralized setting also referred to as the server-less setting, where the local nodes communicate over an undirected graph.
Under both these settings we establish the linear convergence of FedAvg for minimizing a class of non-convex functions satisfying the PL inequality.
We note that PL inequality plays a key role in the training of overparameterized systems.
Specifically, many works have shown that the loss function of an overparamterized neural network satisfies the PL inequality (Bassily et al., 2018; Liu et al., 2020).
Furthermore, our analysis reveals that the standard but restrictive assumptions of bounded gradients (Yu et al., 2019b; Stich, 2018; Li et al., 2019;
Koloskova et al., 2020), heterogeneity (Yu et al., 2019a; Woodworth et al., 2020b; Yu et al., 2019a), and variance (Woodworth et al., 2020b; Qu et al., 2020) can be avoided while guaranteeing this linear convergence of FedAvg.
Next, we list the major contributions of our work.
• We analyze the convergence of FedAvg in the Server setting which consists of a single server communicating with several clients (McMahan et al., 2017).
In this setting, we show that to achieve an ϵ -accurate solution FedAvg in the interpolation regime requires R ∼ O (cid:0) T 3 / 2 log (1 /ϵ ) (cid:1) rounds of communication, where T is the number of local SGD updates.
• We consider the Decentralized setting where N distributed clients communicate over an undirected graph G .
Similar to the Server setting, we show that to achieve an ϵ -accurate solution Decentralized FedAvg requires R ∼ O (cid:0) T 2 log (1 /ϵ ) (cid:1) rounds of communication.
We also characterize the effect of the network topology on the performance of Decentralized FedAvg.
Moreover, our proof technique utilizes a novel induction based analysis for establishing this convergence.
• Our theoretical results under both the Server and the Decentralized settings do not require assumptions such as bounded heterogeneity, gradient and variance.
We show that sample-wise smoothness of the stochastic loss functions suffices to capture the effect of data heterogeneity among different clients (Bassily et al., 2018) while avoiding the need to impose these restrictive assumptions.
• Finally, to corroborate our theoretical findings we present experimental results using various datasets such as MNIST, FMNIST, CIFAR-
10 and Shakespeare, under different settings.
Related Work:
FedAvg first proposed in (McMahan et al., 2017), has been extensively studied in the server setting and with homogeneous data (see Stich (2018); Wang & Joshi (2018); Khaled et al.
; Yu et al. (2019b); Wang et al.
; Yang et al.
Recently, many works have adapted the analyses of FedAvg for minimizing the non-convex losses in the heterogeneous data settings.
We note that most of the works (see Yu et al. (2019a);
Yang et al.
; Karimireddy et al.
(2020b)) establish the convergence rate of O (1 /Nϵ 2 ) to an ϵ -stationary point under the bounded heterogeneity setting.
It is also worth noting that numerous works have proposed variants of FedAvg with different local update rules (e.g., variance reduction, momentum SGD, adaptive updates, etc.)
with the goal of improving the performance of FedAvg (Karimireddy et al., 2020b;
Sharma et al., 2019; Liang et al., 2019; Khanduri et al., 2021; Karimireddy et al., 2020a).
However, in practice
FedAvg remains the algorithm of choice for training large FL systems.
[Table]
A few works that have also analyzed the performance of Fedvg in the decentralized setting as well.
In (Haddadpour & Mahdavi, 2019; Yu et al., 2019a), the authors analyze the convergence of FedAvg under both server and decentralized setting with bounded gradient dissimilarity assumption and establish a convergence rate of O (1 /Nϵ 2 ) for minimizing smooth non-convex objectives.
We note that all the above works provide a sublinear rate of convergence for FedAvg, however, as illustrated in Fig.
1, FedAvg converges at a much faster rate in practice.
To understand this behavior of FedAvg, in this work we analyze the performance of FedAvg under both server and decentralized settings for minimizing a special class of non-convex functions satisfying PL inequality under the interpolation regime.
We note that overparameterized neural networks/systems usually operate in the interpolation regime while their loss functions have been shown to satisfy the PL inequality.
The linear convergence of centralized SGD in the interpolation regime for minimizing PL objectives was first established in Bassily et al.
Recently, (Qu et al., 2020) showed linear convergence rate of FedAvg in the server setting for minimizing strongly-convex objectives in the overparameterized regime.
Similarly, the authors in (Koloskova et al., 2020) have also established the linear convergence of FedAvg in the decentralized setting for minimizing strongly-convex losses in an overparameterized setting.
The above works only focus on analysis of FedAvg for the stronglyconvex objectives in the overparameterized regime while we focus on the more general class of non-convex functions satisfying the PL inequality.
Moreover, compared to other works that assume restrictive bounded gradient, heterogeneity, and variance assumptions, we show that such assumptions can be avoided by using a sample-wise smoothness assumption.
Table 1 presents a summary of the above discussion.
Please refer to Appendix A.2 for a detailed literature review.
Server F ED A VG
In this section, we present the classical FedAvg algorithm in the server setting and prove its convergence.
The FL setup consists of a set of N clients and a central server.
The clients collaborate with the help of the server to solve the FL Problem stated in Section 1.
As noted earlier, a standard algorithm to solve the FL Problem is FedAvg (McMahan et al., 2017).
FedAvg executes in two major steps, namely the local SGD step and the aggregation and broadcast step by the server.
The main steps of FedAvg are discussed below, and are outlined in Algorithm 1.
Initial Broadcast:
The central server broadcasts the initial model parameters denoted w 0 to all the clients at the beginning,
i
e., in the round r = 0 .
See steps 1 and 3 of Algorithm 1. 2.
Local updates: Each client performs T local SGD updates starting from the initial model received from the server.
To compute the stochastic gradient, each client k ∈ [ N ] uniformly samples a batch of size b denoted by B r,tk .
The resulting model after T local updates in the r -th communication round, w r,Tk , is shared with the server.
See steps 6 and 7 of Algorithm 1. 3.
Aggregation and Broadcast:
In the r -th global communication round, the central server computes an average, w r , of the received models from the individual clients.
The server then broadcasts the aggregated model w r to all the clients.
The steps (2) and (3) are repeated for R communication rounds.
See steps 11 , 12 and 3 of Algorithm 1.
S
In this work, we for the first time establish linear convergence of FedAvg in the interpolation setting for non-convex loss functions (satisfying PL condition)
(Bassily et al., 2018).
It is important to note that the proof of convergence provided in Bassily et al.
(2018) for centralized SGD cannot be directly extended to the FL setting of Algorithm 1.
The major challenge in the analysis is presented by the phenomenon referred to as client drift where the local models drift apart with multiple local
SGD updates because of data heterogeneity.
Naturally, the analysis involves bounding the client drift in terms of the loss function, which is challenging, and non-trivial.
Moreover, without making bounded gradient and bounded heterogeneity assumptions, it remains an open challenge on how to control this client drift.
In this work, we show that this drift can in fact be controlled with samplewise smoothness assumption and a careful analysis of the drift term.
To analyze the convergence of
Server FedAvg, we make some standard assumptions as discussed next.
2. A SSUMPTIONS In
this section, we present the assumptions and definitions used in the paper.
Definition 1.
( L -Smoothness): The function Φ ( u ) is said to be L smooth if there exist a constant
L > 0 such that ∥∇ Φ ( u 1 ) − ∇ Φ ( u 2 ) ∥ 2 ≤ L ∥ u 1 − u 2 ∥ 2 for any u 1 , u 2 ∈ R d .
Note that this implies Φ ( u 1 ) ≤ Φ ( u 2 ) + ⟨∇ Φ( u 2 ) , u 1 − u 2 ⟩ + L 2 ∥ u 1 − u 2 ∥ 2 for any u 1 , u 2 ∈ R d .
Definition 2.
( ϵ -accurate solution): A point w ∈ R d is called an ϵ -accurate solution if Φ( w ) − Φ( w ∗ ) ≤ ϵ , where w ∗ ∈ arg min w ∈ R d
Φ( w ) .
A stochastic algorithm is said to achieve an ϵaccurate solution in r rounds if E [Φ( w r ) − Φ( w ∗ )] ≤ ϵ , where expectation is taken over the stochasticity of the algorithm.
Assumption 1.
(Interpolation (Bassily et al., 2018)): We assume that there exists a w ∗ ∈ R d such that the per sample loss Φ k,j ( w ∗ ) =
for all samples j ∈ [ b ] .
Assumption 2.
(PL inequality):
The loss function Φ ( v ) satisfies the PL inequality, i.e., ∥∇ Φ ( v ) ∥ 2 ≥ µ Φ ( v ) for some µ > 0 and for all v ∈ R d .
Further, the local loss functions Φ k ( v ) for all k ∈ [ N ] are also assumed to satisfy the PL inequality, henceforth referred to as local
PL inequality, i.e., ∥∇ Φ k ( v ) ∥ 2 ≥ µ k Φ k
( v ) for some µ k > 0 and for all v ∈ R d .
Assumption 3.
(Sample-wise, Local and Global smoothness): The functions Φ k,j ( · ) for all j ∈ [ b ] , k ∈
[ N ] are assumed to be l k,j -smooth.
The local functions Φ k ( · ) for all k ∈ [ N ] are assumed to be L k -smooth.
The above assumptions imply ∥∇ Φ k,j ( v ) ∥ 2 ≤ 2 l k,j Φ k,j ( v ) and ∥∇ Φ k ( v ) ∥ 2 ≤ 2 L k Φ k ( v ) for all k ∈ [ N ] and j ∈ [ b ] .
We also assume the global loss Φ( · ) to be L -smooth.
Assumption 4.
(Unbiased Gradient and Loss function): We assume that the estimates of the gradient and the loss function at each client k ∈
[ N ] is unbiased, i.e., E [ ∇ Φ k,j ( w )]
= ∇ Φ k ( w ) and
E [Φ k,j ( w )] =
Φ k ( w ) for any j ∈ [ b ] and w ∈ R d .
Note that the above assumptions, including interpolation, PL inequality and smoothness are standard assumptions made by several authors in the past (Bassily et al., 2018; Karimi et al., 2016; Ma et al., 2018; Nguyen & Mondelli, 2020).
For example, the authors in Bassily et al.
Liu et al.
(2022); Karimi et al.
; Haddadpour et al.
(2019) assume PL inequality along with sample-wise smoothness to prove linear convergence of SGD in a centralized setting for the interpolation regime.
Importantly, it must be noted that the PL inequality plays an important role in the analysis of overparameterized neural networks.
Specifically, many works have shown that the overparameteried neural networks satisfy the PL inequality (Liu et al., 2020; Nguyen & Mondelli, 2020; Nguyen et al., 2021;
Allen-Zhu et al., 2019).
Moreover, we note that the assumption on sample-wise smoothness is not very strict since any neural network with smooth activation will satisfy this assumption.
Although the above does not cover neural networks with non-smooth activations (like ReLU), it is known that such activations can be approximated to arbitrary accuracy by a kernel based smooth activation function (Nguyen & Mondelli, 2020, Eq. 2).
Next, we present the first main result of the paper.
The detailed proof is presented in Appendix A.4.
2. C ONVERGENCE OF Server F ED A VG
Theorem 1.
Under Assumptions 1–4, choosing η ≤ min
[Equation]
the iterates generated by Algorithm 1 satisfy
[Equation]
where,
[Equation]
L 2 max b ( b − 1) b(cid:17) and ζ 4 := 8 L 2 (cid:16) 2 l max b + 2 b ( b − 1)
L max b(cid:17) , µ min :=
min k ∈ [ N ] { µ k } , l max := max k,j l k,j and L max := max k L k .
Next, we characterize the sample complexity of FedAvg.
Corollary 1.
Under the setting of Theorem 1 to achieve an ϵ -accurate solution, Algorithmrequires R =
O (cid:0)(cid:0) T 3 / 2 /µ (cid:1) log (cid:0) Φ (cid:0) w 0 (cid:1) /ϵ
(cid:1)(cid:1) rounds of communication.
Corollary 1 establishes the linear convergence of FedAvg in terms of the number of rounds required to achieve an ϵ -accurate solution.
From Corollary 1, we also observe that as the number of local updates, T , increase the global communication rounds also increase.
Note that this is expected since more local updates result in client drift, thereby, requiring more communication rounds to reach the ϵ -accurate solution.
Note that we can control the effect of client drift by choosing the learning rate appropriately, i.e., η ∼ O (1 /T 3 / 2 ) , and as a consequence, we need to choose the global rounds to be of the order of O ( T 3 / 2 ) .
However, note that Corollary 1 captures the worst case behavior of FedAvg.
Ideally, one would expect the convergence performance (in terms of communication rounds) to first improve with T and then worsen as T increases beyond a threshold as also observed in Yu et al.
(2019b;b).
A more fine grained study reveals that this is in fact the case for our analysis as well.
Below, we discuss the effect of the local updates on the performance of FedAvg.
Effect of Local Updates: Our analysis shows that the convergence performance is not always a monotonic function of T .
To see this, we refer the readers to equation 21 in the Appendix, which reveals that for a fixed η ≤ 4 /µ , the first term decreases with T while the second term increases.
This implies that if T ≤ T th the first term dominates (for η small enough) and the convergence performance improves as T increases, however, as T ≥ T th the second term dominates and the convergence performance degrades as T increases.
This implies that an optimal choice of T =
T th exists, and is a solution to (1 − ηµ/ 4) T
=
η 3 L 2 ( T − 1) 3 (cid:0) 2 l max /b + 2 b ( b − 1)
L max /b 2 (cid:1) .
Note that for a fixed η , the optimal T is independent of the communication rounds r .
Later we present experimental results to corroborate this behavior.
For the clarity of presentation and better interpretability of the result,
Corollary 1 only focuses on the general result establishing linear convergence of FedAvg.
We also note that the variance in our theoretical results is implicitly controlled by the smoothness assumption of the loss function.
In the theoretical analysis, the dependence on the batch size, b is captured by the bound on the learning rate η in Theorem 1.
It is clear from above that a smaller batch size implies smaller η and, thereby, a slower convergence rate.
In contrast, a larger batch-size allows us to choose a larger step-size, thereby, leading to a faster convergence rate.
The best known rate for FedAvg for minimizing the strongly-convex objectives in the overaprameterized regime is O (exp( − NR/T )) with bounded variance assumption Qu et al.
These assumptions are seldom satisfied, and hence are of limited applicability.
In contrast, our analysis without the need of any variance or heterogeneity assumptions achieves O (exp( − µR/ 8 T 3 / 2 ) rate for minimizing a much harder class of non-convex problems satisfying PL inequality.
Further, note that the theorem also reveals the effect of data heterogeneity on the performance of FedAvg, for example, the choice of step-size will be limited by the worst case smoothness constants as well as the
PL inequality parameter µ min ’s among all clients.
Finally, we present the proof sketch of Theorem 1 and discuss the major challenges in the proof.
Proof Sketch of Theorem 1 and Challenges: In FedAvg setting, the individual nodes execute multiple local updates.
This leads to a phenomenon termed as “client drift”, wherein, the local models drift apart from each other because of heterogeneity in the local datasets.
The first step is to establish descent in E (cid:2) Φ (cid:0) w r,t +1 (cid:1)(cid:3) in terms of the client drift, see equation 18.
Note that this expression is independent of the variance, and proving this requires a careful application of the smoothness and the PL inequality, as in equation 13 and equation 21.
Next we bound the drift term in Lemma 1.
In standard FedAvg analyses, this drift is controlled using the bounded heterogeneity (or gradient) assumptions.
However, guaranteeing convergence of FedAvg without making bounded heterogeneity (or gradients) assumption is unclear.
Therefore, the challenge lies in bounding this drift in terms of the local loss (see equation 7), and then relate it to the global loss (see Lemma 2).
These steps require us to again carefully use the smoothness and PL inequality.
Note that our analysis is the first to establish fast (linear) convergence of FedAvg for minimizing PL functions in the interpolation regime without imposing any assumptions on the local data distributions.
In the next section, we extend our analysis and results to a decentralized learning scenario.
Decentralized F ED A VG
In this section, we present the Decentralized FedAvg algorithm, and prove its convergence.
The
Decentralized setting consists of N distributed edge devices which are represented using a connectivity graph G ∈
( V , E ) .
Here, V ∈ [ N ] is the vertex set or clients, and E ⊆ {V × V} represents the edges of the graph.
Any edge ( i,
j ) ∈
E represents a connection between node i and j .
Further, the connections are represented using mixing matrix P =
[ p i,k ] ∈ R N × N , where p i,k = 0 if there is no edge between node i and k i.e., ( i, k ) / ∈ E , else p k,i > 0 .
The algorithm for Decentralized FedAvg is presented in Algorithm 2 while in the following, we provide an outline:
Initialization: Each client k ∈ [ N ] initializes the model parameters denoted by w 0 k at the beginning, i.e., in the round r = 0 .
See Step 1 of Algorithm 2. 2.
Local updates: Each client performs T local SGD updates starting from the model parameters obtained by the aggregation of the updates from neighbouring clients.
To compute the stochastic gradient, each client k ∈ [ N ] uniformly samples a batch size b denoted by B r,tk .
The resulting model parameters after T local rounds in the r -th global round are denoted by w r,Tk which is sent to all the neighbouring clients of k .
See Steps 6 and 7 of Algorithm 2.
Note that this procedure is similar to the local update step in the FedAvg case.
Aggregation:
In the r -th global communication round, each client k ∈ [ N ] computes a local average of the model parameters received by its neighbors.
The aggregate model is denoted by w rk .
The steps (2) and (3) above are repeated for R rounds.
See Steps 11 , 12 of Algorithm 2.
Below, we present the convergence result of Decentralized FedAvg algorithm.
3. C ONVERGENCE OF Decentralized F ED A VG
In this section, we prove that the Decentralized FedAvg algorithm converges linearly to the global optimum for any smooth non-convex function satisfying PL inequality in the interpolated regime.
Compared to the FedAvg this case poses several challenges.
In particular, for Decentralized FedAvg we need to handle two drift terms, namely local drift and the global drift .
Local drift refers to the update at each client drifting away from the average obtained from the neighboring clients while do
I do do
S
A
end end the global drift refers to the average obtained from the neighboring clients drifting away from the global average.
To derive the convergence of FedAvg, we employ a novel induction based proof technique and perform a carefully calibrated analysis which can be of independent interest to the research community.
In addition to the Assumptions 1-4, our analysis also relies on the following assumption on the mixing matrix (Koloskova et al., 2020).
Assumption 5.
The mixing matrix P is assumed to be symmetric, i.e., P = P T , and doubly stochastic, i.e., P 1 = 1 , 1 T P = 1 T .
The above assumption covers all networks that are symmetric, for example, fully connected, ring topology etc.
In the following, we provide our main result for the decentralized FedAvg.
The detailed proof is presented in Appendix A.5.
Theorem 2.
Under Assumptions 1-5, choosing
[Equation]
min , µ
[Equation]
, µ
[Equation]
2 ,  (cid:18) µ
[Equation]
(cid:19)
[Equation]
(cid:18)
[Equation]
(cid:19)
[Equation]
ζ
[Equation]
Nµ min ζ 6 T 2 , µ min ζ 7 , (cid:18)
T
[Equation]
(cid:19) 12 , (cid:18)
T 3 ζ(cid:19) 14 (cid:41) ,
[Equation]
the average total loss of the Decentralized FedAvg algorithm satisfies
[Equation]
where λ ≜ (cid:16) 1 + 1 ψ (cid:17) λ 22 , ψ > 1 1 λ 2 2 − 1 , L m := max (cid:8)
L 2 max , 2 L max N (cid:9) , β :=
4 l max ψ 2 N (1+ ψ ) µ min , µ min :=
min k ∈ [ N ] { µ k } , L max := max k L k and Λ := max (cid:0)(cid:0) 1 − ηµ 8 (cid:1) , λ (cid:1) .
In the above, ζ
[Equation]
Ll
[Equation]
bN
[Equation]
b
[Equation] [Equation]
and
[Equation]
Now, in the following, we characterize the sample complexity of Decentralized FedAvg.
Corollary 2.
Under Assumptions 1-5, to achieve an ϵ accurate solution, Algorithm 2 requires
R = O (cid:0) T 2 /µ min log (cid:0) Φ (cid:0) w 0 (cid:1) /ϵ
(cid:1)(cid:1) number of communication rounds.
Corollary 2 shows that even in the Decentralized setting, FedAvg is capable of achieving linear convergence.
Observe from Theorem 2 and the corollary above that an ϵ -accurate solution can be achieved if the number of global communications rounds R scales as O ( T 2 ) .
We know that for the
Server FedAvg algorithm (complete graph), the second maximum eigenvalue λ 2 is 0 .
In that case, the second term on the right hand side of equation 3 goes to zero and we are able to recover the result in Theorem 1 for the Server FedAvg algorithm.
However, R is of the order T 2 as apposed to
T 3 / 2 in the Server case.
This is an artifact of our analysis, and we believe that an order of T 3 / 2 can be recovered, which is relegated to the future work.
By doing a fine grained analysis, one can show that the effect of local updates T on the convergence is quite similar to the server setting.
We do not present the result that we obtain after the fine grained analysis in Theorem 2 and Corollary 2.
Effect of Network Topology : The effect of Decentralized clients is captured through the term involving λ 2 .
In particular, if λ 2 ̸ = 0 , then the convergence is relatively slower due to the second term, i.e., 4 η 4 LL m βT 2 λ 2 ( r + 1) 2 and Γ p .
Although the above result holds good only for networks with symmetric and doubly stochastic mixing matrix, we believe that similar results hold good even in the general setting as well, and can be proved using the technique developed in this paper.
Again, this is the first result establishing linear convergence of FedAvg in the decentralized setting when minimizing non-convex functions satisfying PL-inequality in the interpolation regime.
Next, we present a sketch of the proof and its challenges.
Proof Sketch of Theorem 2 and Challenges: In addition to the challenges mentioned in section 2.2, the Decentralized setting poses several new challenges.
Unlike server setting, as a consequence of the execution of local updates within each communication round, the nodes do not have consensus.
This implies that in the decentralized setting, we need to control the consensus error in addition to the client drift (see Sec. 2.2).
We handle this challenge by bounding the loss in terms of the drift term that captures both local and global drifts as in Lemma 3.
Further, we bound the drift term which depends on the average loss, leading to two coupled equations; this leads to a new challenge in proving the result.
We use a novel application of induction principle to prove linear convergence of both the average loss and the drift.
Many distributed machine learning problems lead to such coupled equation, which can be handled in the way described in this paper.
Hence, the technique can be of independent interest.
In the next section, we present the experimental results.
E XPERIMENTAL R ESULTS
In this section, we experimentally validate our theoretical findings for the server and the decentralized versions of FedAvg.
First, we present the experimental setup for both the settings.
Server FedAvg: Here we consider the FedAvg algorithm with a central server that communicates with 60 or 25 edge devices that run several rounds of local SGD before sharing the model with the server.
We consider the following models to validate our theory: (i) a simple regression and Deep
Neural Network (DNN) models under overparameterized setting for image classification tasks on
CIFAR-10, MNIST, and FMNIST datasets (no. of devices = 60 ), and (ii) DNN under underparameterized and overparameterized settings for next-character prediction task using Shakespeare dataset (no. of devices = 25 ).
Refer to Appendix A.7 for more details of the experimental setup.
Decentralized FedAvg: In this setting, we run Algorithm 2 for the following networks (i) ring, (ii) random doubly stochastic, and (iii) torus topologies.
The neural network architecture and the datasets considered are same as the server setting discussed above.
In the above settings, we compare (a) the performance of server/decentralized FedAvg with both underparametrized and overparameterized neural network models, (b) effect of topology on the convergence ( only decentralized case), and (c) effect of local updates on the convergence.
Figureplots the training loss for FedAvg in the Server and Decentralized setting for underparameterized and overparameterized models on different datasets.
As established in Theorems 1 and 2, the loss of FedAvg in the Server and Decentralized settings diminishes rapidly for the overparameterized models compared to the underparameterized models.
This is due to the fact that the PL inequality is satisfied for overparameterized systems which helps to reach the global optimum at a linear rate as demonstrated by Theorems 1 and 2.
Additional experiments depicting the testing performance of
FedAvg on different models and under different settings are included in Appendix A.7.
Here, random doubly stochastic case has 5 clients while for others we have used 60 clients.
Effect of local updates T : Figure 3 shows plots of the training losses on MNIST dataset for the FedAvg under the Server and Decentralized settings on the overparameterized regression model and the CNN.
From equation 21, we see that as T increases, the convergence speed either decreases or increases depending on the coefficient of T 3 in the second term.
We capture this phenomenon in Fig.
In particular, as T increases, the rate of convergence increases for simple regression model while it decreases/saturates for the CNN based DNN model.
One plausible explanation is that the smoothness constants of simple regression is small, and hence results in smaller second term in equation 21.
However, for CNN based
DNN, the second term dominates, and hence results in slower convergence with T .
Comparison with different topologies in the Decentralized case: Figure 4 shows the training loss versus the communication rounds R for overparameterized CNN model using MNIST dataset with T =for four different topologies.
Since centralized topology has λ 2 = 0 , it outperforms the network with ring topology and a random (doubly) stochastic matrix.
However, the torus topology does not satisfy the conditions required, i.e., symmetric and doubly stochastic matrix, and hence cannot be used for corroborating our theoretical findings.
Nevertheless, we have conducted experiments with torus topology, and Figure 4 shows that the torus has the worst convergence performance.
One reason for this could be that the ring topology has more structure, i.e., it has a symmetric and doubly stochastic mixing matrix P as opposed to the torus topology.
The theoretical analysis of networks with general topology is relegated to our future work.
Conclusion:
In this work, we performed a theoretical analysis of the well known FedAvg algorithm for the class of smooth non-convex overparameterized systems in the interpolation regime.
We considered two settings, namely (i) Server setting where the central server coordinates the exchange of information, and (ii) Decentralized setting where nodes communicate over an undirected graph.
In this regime, it is well know that neural networks with non-convex loss functions typically satisfy an inequality called Polyak-Lojasiewicz (PL) condition.
Assuming PL condition, we showed that in both the settings, the FedAvg algorithm achieves linear convergence rates of O ( T 3 / 2 log(1 /ϵ )) and
O ( T 2 log(1 /ϵ )) , respectively, where ϵ is the desired solution accuracy, and T is the number of local
SGD updates at each node.
As opposed to standard analysis of FedAvg algorithm, we showed that our approach does not require bounded heterogeneity, variance, and gradient assumptions.
We captured the heterogeneity in FL training through sample-wise and local smoothness of loss functions.
Finally, we carried out experiments on multiple real datasets to confirm our theoretical observations.
A A PPENDIX A. N OTATION
We use bold small letters to denote vectors, and capital bold letters for matrices.
We denote the expected value of a random variable X by E [ X ] .
We denote l 2 -norm by ∥ .
∥ 2 and the Frobenius norm by ∥ .
∥ F .
Also ⟨ ., .
⟩ denotes the inner product space.
The cardinality of a any set B is represented by |B| .
We use the standard notation O ( n ) to denote the order of n .
For a vector valued function Φ( w ) , the gradient is denoted by ∇ Φ( w ) , and the Hessian is denoted by ∇ 2 Φ( w ) .
We use 1 to represent a column vector with all ones.
We use [ N ] to denote the set { 1 , 2 , . . .
, N } .
A. R ELATED W ORK
After the introduction of the FedAvg (McMahan et al., 2017), multiple works have analyzed the convergence of FedAvg in the server setting and with homogeneous data, i.e., when the data is i.i.d across clients (see Stich (2018); Wang & Joshi (2018); Khaled et al.
; Yu et al. (2019b); Wang et al.
; Yang et al.
The authors in (Stich, 2018) were the first to obtain a rate of
O (1 /Nϵ ) for strongly convex and smooth problems.
Later (Haddadpour et al., 2019; Haddadpour & Mahdavi, 2019) proved a similar result but for non-convex functions satisfying PL inequality.
In (Woodworth et al., 2020a), the authors analyzed the trade-off between Minibatch and Local SGD in the homogeneous settings and established O (1 /Nϵ 2 ) convergence rates for minimizing smooth convex objectives.
The analysis of FedAvg for the general non-convex settings was first performed in Yu et al.
(2019b) where the authors establish a rate of O (1 /Nϵ 2 ) .
Recently, many works have adapted the analyses of FedAvg for minimizing the non-convex losses in the heterogeneous data settings.
For example, Yu et al. (2019a) extended the results of Yu et al.
(2019b) for the heterogeneous data setting.
Specifically, the authors in (Yu et al., 2019a) utilized a Momentum SGD updates and established the convergence rate of O (1 /Nϵ 2 ) under bounded heterogeneity setting.
The work (Karimireddy et al., 2020b) also provided a tight analysis for FedAvg and established linear speedup with the number of clients.
Recently, (Yang et al., 2021) analyzed the linear speed-up effect of
FedAvg while (Khanduri et al., 2021) analyzed the trade-off between the batch sizes and the local updates.
We note that all these works establish a convergence rate of O (1 /Nϵ 2 ) for minimizing nonconvex smooth losses in the bounded heterogeneity setting.
It is also worth noting that numerous works have proposed variants of FedAvg with different local update rules (e.g., variance reduction, momentum SGD, adaptive updates, etc.)
with the goal of improving the performance of FedAvg (Karimireddy et al., 2020b;
Sharma et al., 2019; Liang et al., 2019; Khanduri et al., 2021; 2020;
Karimireddy et al., 2020a; Das et al., 2022).
However, in practice FedAvg remains the algorithm of choice for training large FL systems.
There are a few works that have analyzed the performance of Fedvg in the decentralized settings.
One of the initial works, (Lian et al., 2017) considered a decentralized parallel SGD (D-PSGD) and provided convergence rate of O (1 /Nϵ 2 ) for minimizing smooth non-convex functions.
Later, (Haddadpour & Mahdavi, 2019) analyzed the convergence of FedAvg under both server and decentralized setting with bounded gradient dissimilarity assumption.
The authors showed a convergence rate of
O (1 /Nϵ 2 ) for minimizing non-convex functions in both the server and decentralized settings.
The authors in Yu et al. (2019a) also extended the analysis of Momentum SGD to decentralized networks and established a convergence of O (1 /Nϵ 2 ) for minimizing non-convex functions.
All the above works provide a sublinear rate of convergence for FedAvg, however, as illustrated in Fig.
1, FedAvg converges at a much faster rate in practice.
To understand this behavior of FedAvg, in this work we analyze the performance of FedAvg under both server and decentralized settings for minimizing a special class of non-convex functions satisfying PL inequality under the interpolation regime.
We note that overparameterized neural networks/systems usually operate in the interpolation regime while their loss functions have been shown to satisfy the PL inequality (Liu et al., 2022).
The linear convergence of centralized SGD in the interpolation regime for minimizing PL objectives was first established in Bassily et al.
Recently, (Qu et al., 2020) showed linear convergence rate of FedAvg in the server setting for minimizing strongly-convex objectives in the overparameterized regime.
Similarly, the authors in (Koloskova et al., 2020) have also established the linear convergence of FedAvg in the decentralized setting for minimizing strongly-convex losses in an overparameterized setting.
The above works only focus on analysis of FedAvg for the stronglyconvex objectives in the overparameterized regime while we focus on the more general class of non-convex functions satisfying the PL inequality.
Moreover, compared to other works that assume restrictive bounded gradient, heterogeneity, and variance assumptions, we show that such assumptions can be avoided by using a sample-wise smoothness assumption.
Table 1 presents a summary of the above discussion.
In a separate line of work, the linear convergence of SGD (and GD) for optimizing overparameterized neural networks/systems with specific activation functions, network widths, and assumptions on data and loss functions has been established (Zou et al., 2020; Li & Liang, 2018; Allen-Zhu et al., 2019; Jacot et al., 2018; Du et al., 2018;
Chizat et al., 2019; Nguyen & Mondelli, 2020).
Recently, the works in (Huang et al., 2021; Deng et al., 2022) have extended some of these specific neural network architectures to FL settings.
However, we note that these works are orthogonal to our setting as we consider a general setting without assuming a specific model to be learned.
A. U SEFUL L EMMAS
In this section, we state two Lemmas that will be used in proving our main results.
Lemma 1.
For any matrices A ∈ C N × N and B ∈ C N × d , we have ∥ AB ∥ 2 F ≤ N ∥
A ∥ 2 op ∥ B ∥ 2 F .
Lemma 2.
(See Lemma 1 in Sun et al.
For any m ∈ N , the mixing matrix P satisfies ∥ P m − Q ∥ op ≤ λ m 2 , where λ 2 is the second largest eigenvalue of the mixing matrix P , and Q := 1 N 11 T .
A. P ROOF OF THEOREM
In this section, we present the proofs for the convergence of Algorithm 1.
A.4. U SEFUL L EMMAS TO P ROVE T HEOREM
To start with, we briefly discuss some Lemmas to prove the main result.
Using the following Lemmas, theorem 1 will be proved in Sec.
A.4.2.
The local model drift is bounded in terms of local loss.
The local model drifts away from the global averaged model during the local updates which is the essence of the following lemma.
Lemma 1.
The local drift 1 N (cid:80) Nk
=1 E (cid:13)(cid:13) w r,tk − w r,t (cid:13)(cid:13) 2 is bounded in terms of local weight i.e., Φ k ( w r,τk ) as follows
[Equation]
where l max := max k,j l k,j and L max :=
max k L k
Proof: Using the step 7 of Algorithm 1, we have
[Equation]
Performing the telescopic sum over w , we get
[Equation]
Averaging over k ∈ [ N ] results in
[Equation]
Using equation 4 and equation 5 in 1 N (cid:80) Nk − 1 (cid:13)(cid:13) w r,tk − w r,t (cid:13)(cid:13) 2 and noting the fact that w r, 0 = w r, 0 k , we get
N
[Equation]
For a sequence X k for k ∈ [ N ] , we have (cid:80) Nk
=1 ∥
X k − X ∥ 2 ≤ (cid:80) Nk
=1 ∥
X k ∥ 2 .
Applying this in the above results in
[Equation]
where F r,τk := ⟨∇ Φ k,j
[Equation]
w
[Equation]
,τ
[Equation] [Equation]
Φ k,j ′ ( w
[Equation] [Equation]
Taking expectation, we get
[Equation]
Further, using smoothness assumption (see assumption 3)
[Equation]
we have
[Equation]
where ( a ) follows from the fact that l max := max k,j l k,j and L max := max k L k .
Next, we show that the local loss is bounded in terms of global average weight.
This is necessary to obtain linear convergence of Algorithm 1.
Lemma 2.
The local average loss E [Φ k ( w r,τk )] is bounded in terms of global average weight i.e., Φ k ( w r ) as follows
[Equation]
Proof: Applying the smoothness assumption (see 3) for Φ k ( u ) , we have Φ
[Equation]
where G r,τ − 1 k :=
[Equation]
(cid:17) .
The last equality follows from step 7 of Algorithm 1 , i.e., w r,τk − w r,τ − 1 k = − η
[Equation]
∇
[Equation]
(cid:17) .
Taking expectation on both sides in the above
[Equation]
we get
[Equation]
where L max := max k L k .
The last term on the right side in equation 9 can be bounded as
[Equation]
, where F r,τ − 1 k
[Equation] [Equation]
(cid:17)(cid:69) , and ( a ) follows from smoothness assumption and the fact that l max := max k,j l k,j and L max := max k L k .
Now, plugging the above in equation 9, we get
[Equation]
where L r,τ − 1 k :=
Φ k (cid:16) w r,τ − 1 k (cid:17) .
Using the local PL inequality i.e., (cid:13)(cid:13)(cid:13) ∇ Φ k (cid:16) w r,τ − 1 k (cid:17)(cid:13)(cid:13)(cid:13) 2 ≥ µ min Φ k (cid:16) w r,τ − 1 k (cid:17) , where µ min :=
min k ∈ [ N ] { µ k } .
(see definition 2), the above can be further bounded as
[Equation]
Choosing η ≤
[Equation]
(cid:19) results in the following
[Equation]
It is easy to see that the above implies
[Equation]
In the next subsection, we provide the proof of 1 using Lemmas proved above.
A.4. C OMPLETING THE P ROOF OF THEOREM
From the Assumption 1, Φ ( w ) can be written as
[Equation]
Now, using the stochastic gradient descent update w r,t + − w r,t = − ηbN (cid:16)(cid:80)
Nk =1 (cid:80)
j ∈B r,t ∇ Φ k,j (cid:0) w r,tk (cid:1)(cid:17) in the above and using Assumption 4, we get Φ
[Equation]
where G r,t := 1 bN (cid:80) Nk =1 (cid:80)
j ∈B r,t ∇ Φ k,j (cid:0) w r,tk (cid:1) .
Taking the expectation conditioning on w r,tk , we get
E (cid:2) Φ (cid:0) w r,t +1 (cid:1)(cid:3) ≤ E (cid:2)
Φ (cid:0) w r,t (cid:1)(cid:3) − η (cid:42) ∇ Φ( w r,t )
[Equation]
N (cid:88)
[Equation] [Equation]
(cid:0)
[Equation]
(cid:1) (cid:43) (cid:124) (cid:123)(cid:122) (cid:125)
[Equation]
η 2 L (cid:32) b
[Equation]
N (cid:88)
[Equation]
(cid:13)(cid:13) (cid:13)(cid:13)(cid:13)(cid:13) (cid:88)
[Equation]
(cid:0)
[Equation]
(cid:1) (cid:13)(cid:13) (cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
(cid:124) (cid:123)(cid:122)
[Equation]
(cid:43) (cid:124) (cid:123)(cid:122) (cid:125)
[Equation]
(cid:33) .
The term A 2 in equation 10 can be bounded as
[Equation]
Now taking the expectation conditioning on w r,tk , we get
[Equation]
Taking the expectation of A 3 in equation
[Equation]
we get
[Equation]
where ( a ) follows from the fact that ⟨ a, b ⟩ ≤ 12 ∥ a ∥ 2 + 12 ∥ b ∥ 2 .
Next, the inner product term A 1 in equation 10 can be written as
[Equation]
where ( a ) follows from smoothness assumption (see 1).
Substituting equation 11, equation 12 and equation 13 in equation 10, we get the following
[Equation]
The term A 4 in equation 14 can be upper bounded as follows
[Equation]
[Table]
where ( a ) follows by adding and subtracting ∇ Φ k,j ( w r,t ) and using the fact that, ∥ a + b ∥ 2 ≤ 2 ∥ a ∥ 2 + 2 ∥ b ∥ 2 , ( b ) follows from Assumption 3, and ( c ) follows from the fact that l max := max k,j l k,j .
Taking the expectation of A 4 , we get the following bound
[Equation]
Now
[Equation]
let us upper bound the term A 5 in equation 14 as
[Equation]
In the above, ( a ) follows by adding and subtracting ∇ Φ k ( w r,t ) and using the fact that, ∥ a + b ∥ 2 ≤ 2 ∥ a ∥ 2 + 2 ∥ b ∥ 2 , and ( b ) follows from Assumption 3 and ( c ) follows from the fact that L max := max k L k .
Substituting upper bounds from equation 15 and equation 16 in equation 14, we get
E (cid:2)
[Equation]
Now, using PL inequality, i.e., ∥∇ Φ ( w ) ∥ 2 ≥ µ Φ ( w ) , ∀ w ∈ R d and rearranging the terms, we get
E (cid:2) Φ (cid:0) w r,t +1 (cid:1)(cid:3) ≤ E
[Equation]
Choosing η ≤ min
[Equation]
, the above can be further bounded as
[Equation]
In order to prove linear convergence
[Equation]
it suffices to show that the second term above,
i
[Equation]
e., 1 N
[Equation]
2 is exponential in Φ ( w ) .
From Lemma 1, it follows that the second term on the right hand side in equation 18, becomes
[Equation]
Substituting equation 8 of Lemma 2, i.e. E [Φ k ( w r,τk )] ≤ (cid:0) 1 − ηµ min
[Equation] [Equation]
k
[Equation]
w r )] , in the above results in
[Equation]
where ( a ) follows by choosing η ≤ 2 µ min and ( b ) follows from the fact that 1 N (cid:80) Nk =
1 Φ k ( w r )
=
Φ ( w r ) .
Using recursion on equation 18, we get
[Equation]
It follows from the update step that 1 N (cid:80) Nk
[Equation]
(cid:13)(cid:13)(cid:13)
[Equation]
− w r
[Equation]
T
[Equation] [Equation]
(cid:13) (cid:13)(cid:13) 2 = 0 for τ = T − 1 .
Using 1 N (cid:80) Nk
=
[Equation]
E
[Equation] [Equation]
) in the above results in
[Equation]
Setting η ≤ 4 µ gives
[Equation]
Using the fact (cid:0) 1 − ηµ 4 (cid:1) T ≤ (cid:0) 1 − ηµ 4 (cid:1) , and choosing η ≤ (cid:20)
[Equation] [Equation]
lmaxb
[Equation]
) (cid:21)results in the following exponential bound
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
A. P ROOF OF T HEOREM
In this section, we first present the overview of the proof.
Then, we will state and prove Lemmas required to prove the Theorem.
The proof mainly consists of three intermediate steps, namely bounding i) the local loss, ii) the loss in terms of future iterates, and iii) the global drift.
In the
Lemma 6, we bound the local loss.
We use L k smoothness (see definition 1) and local PL inequality to show loss at local weight is bounded in terms loss at global average weight and the drift.
A.5. P ROOF OF T HEOREM
We simplify the presentation of the proof by using the following matrix notations.
Let the local average weights be denoted by W rl :=
[ w r 1 , w r 2 , . . .
, w rN ] T ∈ R N × d , where w rk ∈ R d .
The
Aggregation step of Algorithm 2 can be compactly written in matrix form as
[Equation]
where N k :
=
{ i : p k,i > 0 } .
Further, we define the global average as
[Equation]
where the average matrix Q := 1 N 11 T .
Now, let us represent the gradients compactly in the matrix form as
[Equation]
where G ( r,t ) l,j := ∇ Φ l,j (cid:0) w r,tl (cid:1) .
The mixing matrix P also preserves the average, and hence QP =
P .
We start by proving an upper bound on the average loss E (cid:2) Φ (cid:0)
w r +
(cid:1)(cid:3)
in terms of the loss Φ ( w r ) in the r -th communication round, and the drift D r, 0 , as shown in the following Lemma.
Lemma 3.
The average loss is bounded in terms of the drift as follows
[Equation]
where the drift D r, 0 :=
(cid:13)(cid:13)(cid:13) W r, 0 l − W r, 0 (cid:13)(cid:13)(cid:13)
F , and η is chosen according to equation 2.
Proof: The proof is provided in Appendix A.6.
It is easy to see from Lemma 3 that we can obtain the convergence result provided in theoremprovided the drift term on the right hand side of equation 25 is bounded in terms of loss.
More specifically, if D r, 0 ≤ constant × Φ ( w r ) , then the linear convergence stated in Theorem 2 can be easily proved by substitution.
Before proving this, in the following lemma, we provide a recursion of the drift in terms of the average loss and the past drift.
Lemma 4.
The drift is bounded in terms of Φ (cid:0) w τ, 0 (cid:1) as follows
[Equation]
where
[Equation]
Proof: The proof is provided in Appendix A.6.
Next, our task is to show that the recursion in equation 26 satisfies a bound of the form D r, 0 ≤ constant r ×
Φ (cid:0) w 0 (cid:1) , which is the desired result.
Here, the constant is less than one.
We use induction along with carefully choosing η to achieve this goal.
The following lemma provides the desired result.
Lemma 5.
Using equation 26 and equation 25 and by induction on D r +1 ,
we get
[Equation]
[Equation] where L m := max (cid:8) L 2 max , 2 L max N (cid:9) and β := 4 l max ψ 2 N (1+ ψ ) µ min .
Proof: The proof is provided in Appendix A.6.1.
First, note that if the network is fully connected or centralized, i.e., λ 2 = 0 , then the drift term becomes zero, as expected.
Further, the drift increases with the number of clients N and the number of local rounds T .
Nevertheless, it goes down with Λ exponentially provided Λ < 1 .
This ensures that the exponential bound in our main result holds good.
Finally, the proof of Theorem 2 is complete by using equation 26 and equation 27 in equation 25.
In the next subsection, we state and prove some useful Lemmas that are required to prove the main result.
A.5. U SEFUL L EMMAS TO P ROVE T HEOREM
Lemma 6.
The function Φ k ( w r,τk ) satisfies local PL inequality and can be bounded in terms of global average weight
i
[Equation]
e., Φ k ( w r ) as follows
[Equation]
where µ min := min k ∈ [ N ] { µ k } .
Proof: From assumption 1, the function Φ k ( w r,τk ) is written as
[Equation]
We know from step 7 of Algorithm
[Equation]
w r,τk − w r,τ − 1 k
[Equation]
Using this in equation
[Equation]
we get
[Equation]
∇ Φ k (cid:16) w r,τ − 1 k (cid:17) ,b (cid:88) j ∈ B r,τ − 1 k ∇ Φ k,j (cid:16) w r,τ − 1)
k
(cid:17)(cid:43)
+ η 2 L k 2 b(cid:88)
j ∈ B r,τ − 1 k
(cid:13)(cid:13)(cid:13) ∇ Φ k,j (cid:16) w r,τ − 1 k (cid:17)(cid:13)(cid:13)(cid:13)2 + η 2 L k 2 b(cid:88)
j ̸ = j ′ (cid:68) ∇ Φ k,j (cid:16) w r,τ − 1 k (cid:17) , ∇ Φ k,j (cid:16) w r,τ − 1 k (cid:17)(cid:69) .
≤ Φ k (cid:16) w r,τ − 1 k (cid:17) − η (cid:42) ∇ Φ k (cid:16) w r,τ − 1 k (cid:17) ,b (cid:88) j ∈ B r,τ − 1 k ∇ Φ k,j (cid:16) w r,τ − 1) k (cid:17)(cid:43) + η 2 L max 2 b(cid:88)
j ∈ B r,τ − 1 k
(cid:13)(cid:13)(cid:13) ∇ Φ k,j (cid:16) w r,τ − 1 k (cid:17)(cid:13)(cid:13)(cid:13)2 + η 2 L max 2 b(cid:88)
j ̸ = j ′ (cid:68) ∇ Φ k,j (cid:16) w r,τ − 1 k (cid:17) , ∇ Φ k,j ′ (cid:16) w r,τ − 1 k (cid:17)(cid:69) .
where G k ( r, τ ) :
= (cid:13)(cid:13)(cid:13) 1 b (cid:80) j ∈ B r,τ − 1 k ∇ Φ k,j (cid:16) w r,τ − 1 k (cid:17)(cid:13)(cid:13)(cid:13)2 , and L max := max k L k .
Taking expectation with respect to w r,τ − 1 k in the above, gives us
E [
[Equation]
(cid:17)(cid:69)
[Equation]
max
[Equation]
(cid:13)(cid:13)(cid:13) ∇ Φ k,j (cid:16) w r,τ − 1 k (cid:17)(cid:13)(cid:13)(cid:13)
+
[Equation]
L
[Equation] [Equation]
) 2 b(cid:13)(cid:13)(cid:13) ∇ Φ k (cid:16) w r,τ
[Equation]
k (cid:17)(cid:13)(cid:13)(cid:13) (cid:21) .
Applying smoothness assumption of each sample, i.e., (cid:13)(cid:13)(cid:13) ∇ Φ k,j (cid:16) w r,τ − 1 k (cid:17)(cid:13)(cid:13)(cid:13) ≤ 2 l k,j Φ k,j (cid:16) w r,τ − 1 k (cid:17) , we have
E [Φ k ( w r,τk )] ≤ E (cid:20) Φ k (cid:16) w r,τ − 1 k (cid:17) − η (cid:13)(cid:13)(cid:13)
[Equation]
(30) where l max := max k L k .
From the local PL inequality (see definition 2), it follows that (cid:13)(cid:13)(cid:13) ∇ Φ k (cid:16) w r,τ − 1 k (cid:17)(cid:13)(cid:13)(cid:13)2 ≥ µ min Φ k (cid:16) w r,
τ − 1 k (cid:17) for k = { 1 , 2 , . . . , N } , where µ min := min k ∈ [ N ] { µ k } .
Using this in equation 30 results in
[Equation]
By setting η ≤
[Equation]
, the above can be further bounded as
[Equation]
Since w r, 0 k = w rk , the above can be written as
[Equation]
Using the local PL inequality, i
[Equation]
e., Φ k ( w rk ) ≤
[Equation]
22 in equation 31, we have
[Equation]
Now, adding and subtracting the term ∇ Φ k ( w r ) in the above, and using the fact that ∥ a + b ∥ 2 ≤ 2 ∥ a ∥ 2 + 2 ∥ b ∥ 2 , we get
[Equation]
Using L k smoothness assumption (see Assumption 3), we have
[Equation]
Choosing η ≤ 2 µ min and using the fact that L max = max k L k , we get
[Equation]
Corollary 3.
The function Φ k ( w r,τk ) satisfies local PL inequality and can be bounded in terms of global average weight i.e., Φ k ( w r ) as follows
[Equation]
where µ min := min k ∈ [ N ] { µ k } and L max := max k L k .
Proof: The proof directly follows from Lemma 6 by using the smoothness assumption, i.e., ∥∇ Φ k ( w r ) ∥ 2 ≤ 2 L max Φ k ( w r ) .
This completes the proof.
Next, we show that the loss can be bounded in terms of the future iterates as follows.
Lemma 7.
The function Φ (cid:0) w r − 1 , 0 (cid:1) is bounded in terms of the future value of the function as given below
[Equation]
Proof: It follows from the smoothness assumption that
[Equation]
Telescoping the update in step 7 of Algorithm 2 , we get w
[Equation]
i
[Equation]
Averaging over all neighboring nodes i ∈ N k , we get
[Equation]
Averaging over k ∈ [ N ] leads to
[Equation]
Using the above update in equation 35, we get
[Equation]
−
[Equation] [Equation]
The term A 1 in equation 36 can be bounded as
[Equation]
[Table]
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
bN
N (cid:88) k
[Equation]
(cid:88)
[Equation]
(cid:88)
j
[Equation] [Equation]
k ∇ Φ
[Equation]
(cid:16)
w
[Equation]
k (cid:17)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) .
(37) where ( a ) follows from the inequality ⟨ a, b ⟩ ≤ 12 ∥ a ∥ 2 + 12 ∥ b ∥ 2 − 12 ∥ a − b ∥ 2 , and ( b ) follows from the fact that the term (cid:13)(cid:13)(cid:13) 1 bN (cid:80) Nk =
[Equation]
(cid:80) T − 1 τ =0 (cid:80) j ∈ B r − 1 ,τ k ∇ Φ k,j (cid:16) w r − 1 ,τ k (cid:17) − ∇ Φ (cid:0) w r − 1 (cid:1)(cid:13)(cid:13)(cid:13) 2 > 0 .
Next, using equation 37 in equation 36, we get Φ
[Equation]
bN N
[Equation]
k =
T
[Equation]
(cid:88)
τ = (cid:88)
j
[Equation]
B r
[Equation] [Equation]
τ k ∇ Φ k
[Equation]
(cid:16) w r
[Equation] [Equation]
τ k (cid:17)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) .
[Equation] [Equation]
A part of the third term in the above can be bounded as follows
[Equation]
where ( a ) follows from the fact that for any vector z
= ( z 1 , z 2 , . . .
, z N ) , (cid:16)(cid:80)
Ni =
z i (cid:17) 2 ≤
N (cid:80) Ni
=
( z i ) 2 , ( b ) follows from smoothness assumption, and ( c ) follows from the fact that l max := max k,j l k,j .
Next, taking the expectation
[Equation]
Using E [
[Equation]
from Corollary 3, the above can be further bounded as
[Equation]
Using the above result in equation 38 and rearranging
[Equation]
we obtain
E
[Equation]
E
[Equation]
Choosing η ≤ 1 L and rearranging the terms
[Equation]
we get
[Equation]
Further, choosing η ≤
[Equation]
lmaxT
[Equation]
Lmax
[Equation]
L
[Equation]
, the above can be bounded as
[Equation]
The following bound can be obtained by using η ≤
[Equation]
equation 39:
[Equation]
Now, it suffices to bound the drift term in terms of the loss to obtain the linear convergence.
Lemma 8.
The consensus term, i.e., D r, 0 :=
(cid:13)(cid:13)(cid:13) W r
[Equation] [Equation]
− W r,
[Equation]
F satisfies the following bound
[Equation]
where β :=
[Equation] [Equation]
=
[Equation]
N
[Equation]
, and ψ >
[Equation]
Here, λ 2 is the second largest eigenvalue of the mixing matrix P .
Proof: Let,
[Equation] [Equation]
=
[Equation]
Using equation 22 and equation 23, the consensus term can be written as
[Equation]
Recall that Q = 1 N 11 T is the average matrix, P is the mixing matrix and QP = Q .
Using W r, 0 l =
PW r − 1 ,T (see equation 22), substituting for the update in W r − 1 ,T and taking the telescopic sum, we get
[Equation]
Plugging the above in equation 41, and using the generalized Cauchy’s inequality, i.e., ∥ a + b ∥ 2 ≤ (cid:16) 1 + 1 ψ (cid:17) ∥ a ∥ 2 + (1 + ψ )
∥ b ∥ 2 for any ψ ≥ 0 , the consensus term can be upper bounded as
E
[Equation]
P
[Equation]
W
[Equation]
where Ξ := E (cid:13)(cid:13)(cid:0) Q − P 2 (cid:1) W r − 1 , 0 (cid:13)(cid:13) 2 F , and ( a ) follows from Lemma 1 and ( b ) follows from Lemma 2.
Next, consider bounding the following
[Equation] [Equation]
where ( a ) follows from the smoothness assumption and l max := max k,j l k,j .
Substituting the bound in equation 28 of Lemma 7, i.e., E (cid:104)
Φ k (cid:16) w r − 1 ,τ k
[Equation]
k
[Equation]
2 µ min E (cid:13)(cid:13) ∇ Φ k (cid:0) w r − 1 (cid:1)(cid:13)(cid:13) 2 in the above, and writing it in the matrix form, we get
E (cid:13)(cid:13)(cid:13)
[Equation]
Using the above in equation
[Equation] [Equation]
where α := 4 l max
[Equation]
1+ ψ )
[Equation]
First, let us consider bounding E (cid:13)(cid:13)(cid:0) Q − P 2 (cid:1) W r − 1 , 0 (cid:13)(cid:13) 2 F .
Using the update step W r − 1 ,
=
W r − 1 , 0 l
[Equation]
and following a similar approach as used in steps equation 42 to equation 44, we get the following bound
E (cid:13)(cid:13)(cid:0)
[Equation]
Using the above result in equation
[Equation]
Proceeding further in a similar manner as above, we get
D r,
[Equation]
We initialize W 0 , 0 = 0 .
Further, multiplying and dividing by (cid:16) 1 + 1 ψ (cid:17) to the second and the third term in the above, we get
[Equation]
where δ r, 0 := E (cid:13)(cid:13)
∂
Φ (cid:0) W τ, 0 (cid:1)(cid:13)(cid:13) 2 F and λ := (cid:16) 1 + 1 ψ (cid:17) λ 22 .
Using α = 4 l max (1+ ψ ) µ min in equation 45, we have
[Equation]
The term, E (cid:13)(cid:13) ∂
Φ (cid:0) W τ, 0 (cid:1)(cid:13)(cid:13) 2 F in the above, is bounded as follows
[Equation] [Equation]
where ( a ) follows from smoothness assumption and using the fact
that Φ (cid:0) w τ, 0 (cid:1) =
1 N (cid:80) Nk
=1 Φ k (cid:0) w τ, 0 (cid:1) , and L max = max k L k .
Using equation 47 in equation 46, we get
D
[Equation]
Let L m := max (cid:8) L 2 max , 2 L max N (cid:9) and β := 4 l max ψ 2 N (1+ ψ ) µ min .
Therefore, the drift term results in
D r,
[Equation]
A. C OMPLETING THE P ROOF O F T HEOREM
From L -smoothness assumption (see 1) of Φ ( w ) , we have
[Equation]
Using step 7 of Algorithm 2 we have, w r,t +1 i =
[Equation]
w r,
[Equation]
i
[Equation]
Multiplying both sides by p k,i and summing over i ∈ N k , we get
[Equation] [Equation]
Averaging on both sides over k ∈ [ N ] , we get
[Equation]
Using the above update, i.e., w r,t +1 − w r,t in equation 48, we get Φ
[Equation]
where G r,t := (cid:80) Nk
=1 (cid:80) j ∈ B r,tk ∇ Φ k,j (cid:0) w r,tk (cid:1) .
Taking expectation conditioning on w r,tk and past, we get
[Equation]
) where A 2 := 1 b 2 N
[Equation]
Nk =1 (cid:13)(cid:13)(cid:13)(cid:80) j ∈ B r,tk ∇ Φ k,j (cid:0) w r,tk (cid:1)(cid:13)(cid:13)(cid:13) 2 .
This term can be bounded as follows
[Equation]
Taking expectation
[Equation]
we get
[Equation]
Similarly the term A 3 in equation 50 can be bounded by taking expectation as follows
[Equation]
where ( a ) follows from ⟨ a, b ⟩ ≤ 12 ∥ a ∥ 2 + 12 ∥ b ∥ 2 .
Next, we lower bound the term A 1 in equationas
[Equation]
Substituting equation 51, equation 52 and equation 53 in equation
[Equation]
we get the following
[Equation]
where ∆ r,tk := w r,tk − w r,t
and A 5 := (cid:80) N k =1 (cid:13)(cid:13) ∇ Φ k (cid:0) w r,tk (cid:1)(cid:13)(cid:13) 2 .
The term A 4 in equation 54 is bounded as follows
[Equation]
where ( a ) follows by adding and subtracting the term ∇ Φ k,j ( w r,t ) and using the fact that, ∥ a + b ∥ 2 ≤ 2 ∥ a ∥ 2 + 2 ∥ b ∥ 2 , ( b ) follows from Assumption 3, and ( c ) follows from the fact that l max := max k,j l k,j .
Taking expectation, we get
[Equation]
The term A 5 in equation 54 is bounded as
[Equation]
[Equation]
where ( a ) follows by adding and subtracting ∇ Φ k ( w r,t ) , and ( b ) follows from assumption 3 and ( c ) follows from L max :
= max k L k .
Substituting upper bounds from equation 55 and equationin equation 67, we get
[Equation]
Now, using PL inequality (see definition 2), i
[Equation]
e., ∥∇ Φ ( w ) ∥ 2 ≥ µ Φ ( w ) , ∀ w ∈ R d and rearranging, we get
[Equation]
Choosing η ≤ min
[Equation]
(cid:41) , the above can be further bounded as
[Equation]
where ∆ r,tk := w r,tk − w r,tk and ¯
∆ r,tk := w r,tk − w r,t .
In the above, ( a ) follows by adding and subtracting the term w r,tk and using the fact that, ∥ a + b ∥ 2 ≤ 2 ∥ a ∥ 2 + 2 ∥ b ∥ 2 .
First, let us consider the local drift term i.e., (cid:80)
Nk
=
[Equation]
w r,tk − w r,tk (cid:13)(cid:13) in equation 59.
Telescoping the update from step 7 of
Algorithm 2 we have,
[Equation]
Further, consider the local average at node k , i.e., w r,tk
[Equation]
r
[Equation]
Now noting the fact that w r, 0 k = w r, 0 k and using equation 60 and equation 61, we can bound the drift term as
[Equation]
k =
E (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) η b t − 1 (cid:88) τ = (cid:88)
j ∈B r,τk ∇ Φ k,j ( w r,τk ) − η b t − 1 (cid:88) τ
= (cid:88) i ∈N k p k,i (cid:88)
j ∈B r,τi ∇ Φ i,j ( w r,τi ) (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) (a) ≤
N (cid:88) k
=
E   (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) η b t − 1 (cid:88) τ =
G kj ( r, τ ) (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) + (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) η b t − 1 (cid:88) τ
= (cid:88) i ∈N k p k,i G ij ( r, τ ) (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) 2   (b) ≤
N (cid:88) k
=
E  
η 2 t bt − 1 (cid:88) τ
= ∥
G kj ( r, τ ) ∥ 2 + η 2 t bt − 1 (cid:88) τ =
(cid:13)(cid:13) (cid:13)(cid:13)(cid:13) (cid:88) i ∈N k p k,i G ij ( r, τ ) (cid:13)(cid:13) (cid:13)(cid:13)(cid:13) 2   , where G ij ( r, τ ) := (cid:80) j ∈B r,τi ∇ Φ i,j ( w r,τi ) .
In the above, ( a ) follows from the fact that, ∥ a + b ∥ 2 ≤ ∥ a ∥ 2 + ∥ b ∥ 2 , and ( b ) follows from the fact that for any vector z i , (cid:16)(cid:80) Ni
=
z i (cid:17) 2 ≤ N (cid:80) Ni =1 ( z i ) 2 .
The second term in ( b ) can be further bounded using Jensen’s inequality as follows
N (cid:88)
[Equation]
 
[Equation]
where g r,τkj := ∇ Φ k,j ( w r,τk ) , L r,τkj :=
Φ k,j ( w r,τk ) and ( a ) follows from smoothness assumption and ( b ) follows from the fact that mixing matrix P preserves the average and l max :=
max k,j l k,j .
Simplifying the above results in
N (cid:88)
[Equation]
Taking expectation, we get
N (cid:88) k
=
[Equation]
According to equation 34 of Corollary 3 we have, E [Φ k ( w r,τk )] ≤ 2 L 2 max µ min E
∥ w rk − w r ∥ 22 + 4 L max µ min E [Φ k ( w r )] .
Using this in equation 62, we get
N (cid:88) k
=
E
[Equation]
Simplifying the above results in
N (cid:88)
[Equation] [Equation]
where ¯ ∆ r := E ∥
w rk − w r ∥ 22 .
Next, let us consider the global drift term i.e., (cid:80)
Nk =1 (cid:13)(cid:13) w r,tk − w r,t (cid:13)(cid:13) 2in equation 59, which can be rewritten in matrix notation as D r,t := (cid:13) (cid:13) W r,tl − W r,t (cid:13) (cid:13) 2 F .
This term is bounded as
[Equation]
where ( a ) follows since QPW r,t = W r,t and PW r,t = W r,tl , ( b ) follows from QP = Q , and ( c ) follows from the update W r,t = W r, 0 − η (cid:80) t − 1 τ =0 ∂ ˆΦ ( W r,τ ) .
Using the fact that ∥ a + b ∥ 2 ≤ 2 ∥ a ∥ 2 + 2 ∥ b ∥ 2 in the above, we get
[Equation]
The term E
[Equation]
∂ ˆΦ ( W r,τ ) (cid:13)(cid:13)(cid:13) 2 F in the above can be bounded as
[Equation]
where ( a ) follows from the smoothness assumption and the fact that l max := max k,j l k,j .
Using equation 28 of Lemma 6, i.e., E [Φ
[Equation]
w
[Equation] [Equation]
[Equation]
E ∥∇ Φ k ( w r ) ∥ 2 in the above, we get
[Equation]
The result above can be written in matrix form as,
[Equation]
Substituting the above result in equation
[Equation]
we get
[Equation]
where γ := 2 l max N µ min .
[Equation]
∆ r
[Equation]
T
[Equation]
1 −
[Equation]
k (cid:13)(cid:13)(cid:13)
[Equation]
(cid:13)(cid:13)(cid:13)
[Equation]
(cid:13)(cid:13)(cid:13) 2 (cid:19) (66) where ( a ) follows from the fact that (cid:0) 1 − ηµ 4 (cid:1) T ≤ (cid:0) 1 − ηµ 4 (cid:1) , (cid:13)(cid:13)(cid:13) ∆ r,T − 1 − τ k (cid:13)(cid:13)(cid:13) 2 = 0 and (cid:13)(cid:13)(cid:13) ¯ ∆ r,T − 1 − τ k (cid:13)(cid:13)(cid:13) 2 = 0 for τ = T − 1 .
Now choosing η < 4 µ and substituting equation 63 and equation 65 in equation 66, we get
[Equation]
The term E (cid:13)(cid:13) ∂
Φ (cid:0) W r
[Equation]
0 (cid:1)(cid:13)(cid:13) 2 F be bounded as
E (cid:13)(cid:13)(cid:13) ∂ ˆΦ (cid:0) W r, 0 (cid:1)(cid:13)(cid:13)(cid:13)
F =
N (cid:88) k
=
E ∥∇ Φ k ( w r ) ∥(a) ≤ N (cid:88) k
= 2 L max E [Φ k ( w r )]
= 2 L max N E [Φ ( w r )] , where ( a ) follows from smoothness assumption and ( b ) follows from the fact
that Φ ( w r ) =
N (cid:80) Nk
=
Φ k ( w r ) .
Using the above result in equation
[Equation]
we get
[Equation]
Choosing η ≤
[Equation]
in the above result in
[Equation]
Again choosing η ≤ (cid:34)(cid:32)
[Equation]
+
[Equation]
λ 22 γT 2 L(cid:33)(cid:35), the above results in
E
[Equation]
Φ
[Equation]
It is easy to see that E (cid:13)(cid:13)
( Q − P ) W r, 0 (cid:13)(cid:13) 2 F = E (cid:13)(cid:13)(cid:13) W r, 0 l − W r
[Equation]
0 (cid:13)(cid:13)(cid:13)
F = D r, 0 .
Using this above, gives us
[Equation] [Equation]
From Lemma 8, we have
[Equation]
From Lemma 7, we know that
[Equation]
Using the above result on Φ (cid:0) w τ, 0 (cid:1) in equation 69, we get
[Equation]
Let L m = max { 2 L m , 3 L m } .
The above can be further bounded as
[Equation]
This completes the proof.
A.6. P ROOF OF P ROPOSITION
Note that we need to prove the following set of inequalities hold good for all r
[Equation]
where λ = (cid:16) 1 + 1 ψ (cid:17) λ 22 , Φ (cid:0) w 0 (cid:1) =
Φ (cid:0) w 0 , 0 (cid:1) and Λ = max (cid:0)(cid:0) 1 − ηµ 8 (cid:1) , λ (cid:1) .
We use induction method to prove that the above set of inequalities hold good for all r .
Since D 0 , 0 = 0 , the inequalities hold good for r = 0 .
Next, assuming that the above inequalities hold good for every communication rounds in { 1 , 2 , . . .
, r } , we need to prove that the respective inequalities hold for
D r +1 , 0 and Φ (cid:0) w r +1 (cid:1) .
Towards this, consider the following
[Equation]
where ( a ) follows by substituting equation 71, equation 72 and using Λ := (cid:0) 1 − ηµ 8 (cid:1) .
Let us recall from equation 40 of Lemma 8 that
[Equation]
Substituting for D τ, 0 from equation 71 in the first term of equation
[Equation]
we get
[Equation]
where ( a ) follows from the fact that λ ≤ Λ .
Now picking η 2 ≤
[Equation]
results in
[Equation]
Next, substituting for Φ (cid:0) w τ +1 (cid:1) from equation 72 in the second term of equation 73, we get r (cid:88) τ =
[Equation]
The last inequality follows from the fact that τ ≤ r , and λ ≤ Λ .
By choosing η 4 ≤ 1 4 LL m βT 3 ( r +1) 3 λ , we get r (cid:88)
[Equation]
Using equation 74 and equation 75 in equation 73, and after some algebraic manipulations, we get the following desired result
[Equation]
Using the above result in the upper bound for Φ (cid:0) w r +1 (cid:1) , we get the desired bound on .
A
A DDITIONAL E XPERIMENTS
In this section, we provide the details of the experimental setup and some additional results for experiments carried on different datasets for both Server and Decentralized setting.
We have used
NVIDIA DGX
A 100 to implement all our experiments.
The experimental setup consists of the following model and data set:
Overparameterized regression: We consider a model with 3 linear layers and no activation function with
231490 trainable parameters
Note that
this formulation models a simple regression prob-
lem.
We condsider a image classification task on MNIST dataset and evaluate the performance of FedAvg under different settings
Deep neural network
In this case, we consider an image classification
task
on CIFAR– 10 dataset.
Each edge device implements a three hidden layer convolutional neural network (CNN) followed by two linear layers with 1046426 trainable model parameters.
In the overparameterized setting, for the
CIFAR- 10 , MNIST and FMNIST, each edge device implements a three hidden layer convolutional neural network (CNN) with 256 , 128 and 64 filters followed by three linear layers havingtrainable parameters for CIFAR-10 and two linear layers for MNIST and FMNIST withtrainable parameters.
For Shakespeare dataset, LSTM models are used at each edge device.
We consider an embedding layer with embedding size of 10 followed by 2 LSTM layers with 256 hidden neurons and one linear layer.
On the other hand, in the underparameterized setting, we consider a comparatively smaller neural network.
For the CIFAR-10, MNIST, FMNIST datasets each device implements two hidden layer CNN network with 25 and 52 filters followed by two linear layers for
CIFAR-10 and one linear layer for MNIST and FMNIST datasets.
For the Shakespeare dataset each device has embedding layer followed by one LSTM layer with 56 hidden neurons and a linear layer.
For the experiments, we chose T = 10 and tune for the learning rate in the range η ∈ [0 .
001 : 0 . 01] for CIFAR-10, MNIST, FMNIST datasets whereas we choose η = 0 .
8 for the Shakespeare dataset.
Each device has access to 490 training samples and 90 test samples for CIFAR-10 whereas for
MNIST and FMNIST datasets, 540 samples are used for training and 80 samples are used for testing.
Figure 5 show the training loss on FMNIST and MNIST dataset for server and decentralized FedAvg settings.
Figure 6 show the testing accuracy for FedAvg in the server setting for four different datasets.
As expected the convergence speed of underparameterized case is slower than the overparameterized case.
Similarly, figure 7 show plots for testing accuracy for FedAvg in the decentralized setting.
Finally, in Figure 8, we compare the training loss and testing accuracy for server and decentralized
FedAvg algorithm against the communication rounds for classification task on CIFAR-10 dataset.
It is clear from the figures that the centralized case achieves a very good performance at a faster rate as opposed to the decentralized case, i.e., the ring topology.