PersA-FL : Personalized Asynchronous Federated Learning
Abstract
We study the personalized federated learning problem under asynchronous updates.
In this problem, each client seeks to obtain a personalized model that simultaneously outperforms local and global models.
We consider two optimization-based frameworks for personalization: (i) Model-Agnostic Meta-Learning ( MAML ) and (ii) Moreau Envelope ( ME ).
MAML involves learning a joint model adapted for each client through fine-tuning, whereas ME requires a bi-level optimization problem with implicit gradients to enforce personalization via regularized losses.
We focus on improving the scalability of personalized federated learning by removing the synchronous communication assumption.
Our main technical contribution is a unified proof for asynchronous federated learning with bounded staleness that we apply to MAML and ME frameworks.
For the smooth and non-convex functions class, we show the convergence of our method to a first-order stationary point.
Introduction
Federated Learning ( FL ) is designed to facilitate distributed training of machine learning models across devices by exploiting the data and computation power available to them [19, 15].
The common underlying assumption that determines the superiority of vanilla FL to individual local training is that the data points of all clients are coming from the same distribution, i e., homogeneous data across clients.
In FL with heterogeneous data, an ideal scenario is to learn a globally common model easily adaptable to local data on each client, i e., model fusion.
This approach is known as Personalized
Federated Learning ( PFL ), which strives to exploit both the shared and unshared information from the data of all clients.
Fallah et al.
[8] suggest the MAML formulation as a potential solution for
PFL , and propose Per-FedAvg algorithm for collaborative learning with MAML personalized cost function.
Dinh et al.
[6] present pFedMe algorithm for PFL via adopting a different formulation for personalization, namely Moreau Envelopes ( ME ).
Several recent works have approached PFL mainly through optimization-based [13, 23, 37, 27, 5, 12], or structure-based [4, 32] techniques.
In cross-device FL , devices are naturally prone to update and communicate models under less restrictive rules, whereas clients may apply updates in an asynchronous fashion, i.e., staleness.
Hogwild!
[26] is one of the first efforts to model asynchrony in distributed setup with delayed updates.
Multiple works have studied asynchronous training under different setups and assumptions [22].
Specifically, some recent seminal works have studied the convergence of asynchronous SGD -based methods, and show their convergence under certain assumptions on maximum or average delay [1, 18].
More closely, FL under stale updates has been thoroughly studied in [35, 25, 2].
In this work, we study the PFL problem under asynchronous communications to improve training concurrency, performance, and efficiency.
Through the integration of two personalization formulations, MAML & ME , we propose PersA-FL , an algorithm that allows PFL under asynchronous communications with the server.
We present the client algorithm under three different options for the local updates, each addressing a separate formulation, (A) vanilla AFL , (B) PersA-FL: MAML , and (C)
PersA-FL: ME .
We show the convergence rate of PersAFL based on the maximum delay and personalization budget.
Problem Setup
We consider a set of n clients and one server, where each client i ∈ [ n ] holds a private function f i : R d → R , and the goal is to obtain a model w ∈ R d that
[Equation]
with f i ( w ) :=
E Ξ i ∼ p
i
[ ℓ i ( w, Ξ i )] , where ℓ i : R d × S i
→ R is a cost function that determines the prediction error of some model w ∈ R d over a single data point ξ i ∈ S i on client i , where ξ i is a realization of Ξ i ∼ p i , i.e., p i is the client i ’s data distribution over S i , for i ∈ [ n ] .
Let D i be a data batch with samples drawn from the distribution p i .
Then, the unbiased stochastic cost associated with data batch D i can be denoted as follows:
[Equation]
where for simplicity, we assume that the size of all batches is larger than b .
A solution of (1) is a common model for all the clients; hence no adaptation or fusion to each client’s data.
Next, we elaborate on the personalization concept and discuss an alternative problem formulation for (1).
2.1 Personalized Federated Learning
When the data distributions of different clients share some similarities, e.g., bounded variance in their heterogeneity, and the number of data points on each client is limited, joint training with fusion improves the performance compared to individual locally trained models or vanilla FL .
Therefore, learning a shared model with little fine-tuning, e.g., a few steps of SGD with respect to the local cost, may result in a proper personalized model.
Fallah et al.
[8] proposed Per-FedAvg algorithm, which modifies the training loss function by taking advantage of the fact that fine-tuning will occur after training.
The MAML formulation assumes a limited computational budget for personalization (fine-tuning) at each client.
It then offers to look for an initial (global) parameter that performs well after it is updated with one or a few steps of SGD .
In other words, [8] define the MAML loss function for PFL as follows:
[Equation]
with F ( b ) i ( w ) :
= f i ( w − α ∇ f i ( w )) where α ≥ 0 is the MAML personalization stepsize.
Solving (3) yields a global (meta) model that can be used to create a personalized model by applying one step of gradient descent with respect to individual loss functions.
The degree of fine-tuning determines the personalization budget, which often controls the trade-off between having a local (personalized) or generic model, i.e., exploiting the shared and local knowledge simultaneously.
In Problem (3), stepsize α determines the personalization budget, where α = 0 implies vanilla FL in Problem (1).
See [14, 31, 9] for the study of multi-step MAML .
In a nutshell, Per-FedAvg proposes to minimize
F ( b ) ( w ) via a similar paradigm as FedAvg .
Hence, each client i computes the personalized gradient of its MAML cost in (3), which can be written as follows:
[Equation]
4) where in Per-FedAvg , the authors propose to compute a biased estimation of (4) using stochastic gradients/Hessian.
We will elaborate on the stochastic approximation in Section 3.
Algorithm & Convergence Result
In this section, we propose Algorithms 1 & 3 to solve Problem 3.
We move the result ME to Appendix
⋄ Server Algorithm: Let us denote w 0 ∈ R d as the initial parameter at the server, where the objective is to minimize the cost function in either (1), (3), or (8).
Each client i ∈ [ n ] may communicate with the server when the underlying connection is stable.
Clients may request to download the server’s parameters at any time, and the server will send the most recent model after receiving the request.
All underlying delays for the communications between the server and clients are modeled as download and upload delays.
We consider variable t as a counter for the updates at the server level.
Algorithmrepresents the server updates in PersA-FL .
The server performs an iterative algorithm where at each round t ≥ 0 , remains on hold until receives an update ∆ i t ∈ R d from some client i t ∈ [ n ] .
After receiving the update from client i t , the server updates its parameter according to Step 4 of Algorithm 1, where β ≥ 0 is the server stepsize.
Algorithm {+
+}
[Personalized] Asynchronous Federated Learning ( Client i ) q , D ′′ i,q ) (cid:105) ∇ ˜ f
i (cid:16) w i,q − α ∇ ˜ f i ( w i,q , D ′ i,q ) , D i,q (cid:17) ▷ Option C ( PersA-FL: ME ) 9: end for 10: ∆ i ← w i, 0 − w i,Q 11: client i broadcasts ∆ i to the server 12: until not interrupted by the server
Note that we drop the time index from the iterates of the client algorithm for clarity of exposition. ⋄ Client Algorithm :
Client i repeats an iterative procedure which is composed of three phases, (i) downloading the most up-to-date model from the server as in Step 3, (ii) performing Q local updates starting from the parameters of the downloaded model with respect to the cost function of the underlying problem, (1), (3), or (8), as in Steps 5-9, and (iii) uploading the sum of updates on the server as in Step 11.
Note that η ≥ 0 is the local stepsize, a hyperparameter.
The main idea for the local updates is to perform Q sequential SGD steps on the local cost.
By performing this option, we aim to minimize the MAML cost function in (3).
As we saw in Section 2, the full gradient can be computed according to (4).
Following [8], we sample three data batches to compute a biased estimation of (4) as follows:
[Equation]
We will discuss the variance and bias of this estimator in Subsection B.2.
Next, we present the convergence result of our method for the three formulations.
3.1 Convergence Results
Now, we show the convergence of our method for MAML setup.
Recall that the server updates its model at round t using the updates sent by client i t ∈ [ n ] .
We denote Ω( t ) as the timestep of the round at which client i t has received the server’s parameters before applying its Q local updates.
In other words, (Ω( t ) , t ) denote the download and upload rounds for client i t .
Now, we introduce the assumption of maximum delay.
Assumption 1 (Bounded Staleness) .
For all steps t ≥ 0 , the staleness or effective delay between the model version at the download step Ω( t ) and upload step t is bounded by some constant τ , i.e.,
[Equation]
and the server receives updates uniformly, i.e., i t ∼ Uniform([ n ]) .
Assumption 2 (Local Cost Properties) .
For all clients i ∈ [ n ] , function f i : R d → R is bounded below, twice differentiable, and the following properties hold for all w, u ∈ R d ,  
[Equation]
Assumption 2 summarizes all first-order and second order assumptions in [8].
We elaborate on all in
Appendix B.
Theorem 1 ( PersA-FL: MAML ) .
Let Assumptions 1 and 2 hold, α ≥ 0 , β = 1 , and η = 1 Q √ L b T .
Then, the following property holds for the joint iterates of Algorithms 1 & 3 under Option B on
Problem (3) : for any timestep T ≥ 64 L b at the server
[Equation]
The proof of this theorem can be found in Appendix D.
The last term in the above rate, i.e., O (cid:16) α 2 σ 2 g b (cid:17) accounts for personalization with biased gradient estimation.
Moreover, compared to Per-FedAvg , the second term of this rate is different, which accounts for the maximum delay in asynchronous updates.
For instance, under a fixed personalization budget α ≥ 0 , our method requires T =
O ( ε − 2 ) and
b = O ( ε − 1 ) to reach an ε -approximate first-order stationary solution.
The last expression can also be controlled through a combined stepsize α and batch size b .
It is consistent with intuition, meaning more samples are needed to obtain a higher degree of personalization.
We present a detailed analysis of our method in Appendix B.
Conclusion
This work studied the personalized federated learning problem for the heterogeneous data setting under asynchronous communications with the server.
We considered the MAML and ME formulations to account for personalization.
We proposed the PersA-FL algorithm to solve this problem under stale updates.
We showed the convergence rate of our method for smooth non-convex functions.
The extensions of our method to the buffered aggregation setups remain for future studies.
Acknowledgments and Disclosure of Funding
This work was partly done while Mohamamd Taha Toghani interning at Yahoo!
Research.
Part of this material is based upon work supported by the National Science Foundation under Grants # and #2213568.
A Problem Setup, Algorithm, and Comparisons: Extension
As an alternative option to MAML formulation in (3), Dinh et al.
[6] suggest solving the following optimization problem:
[Equation]
where each function F ( c ) i ( w ) is a local cost of personalized parameter θ i ∈
R d by using the Moreau
Envelope as a regularized loss function, and parameter λ ≥ 0 determines the degree of personalization.
In this setup, λ = 0 is equivalent to local training with no collaboration and as λ → ∞ , the formulation in (8) converges to vanilla FL in (1) with no personalization which is similar to the case in (3) with α = 0 .
For non-extreme values of λ , the clients jointly learn a global model w and personalized parameters θ i , which are regularized to remain close to w .
Note that the gradient of
F ( c ) i ( w ) can be written as follows (please check out Appendix E to see the proof):
[Equation]
where for large λ , ˆ θ i ( w ) is the exact solution to an optimization problem.
Therefore, solving (8) through a similar approach to FedAvg or Per-FedAvg , itself requires minimizing Problem (10) which is potentially intractable.
Dinh et al.
[6] propose a bi-level optimization algorithm called pFedMe , to minimize the optimization problem in (8) by alternating minimization over θ i and w .
The main idea behind pFedMe is to integrating the computation of an inexact solution to (10) inside an FL -type method.
We will explain and use this inexact approximation in the presentation of our method (Option C) in Section 3.
A.1 Asynchronous vs Synchronous Schedule
So far, we have discussed the three different formulations for collaborative learning that we will consider in our method.
As we described the FedAvg algorithm in Subsection ?
, at each round t , the parameter w t , which is the most recent version of the global parameter in the server, will be sent to a subset of the clients.
Then, the server halts the training process until all selected clients receive this parameter, perform local updates, and transmit their updates back to the server.
This synchronization procedure restricts the algorithm flow to the slowest client at each round.
Nevertheless, asynchronous updates and communications can be described in this described framework.
Let us provide a comparison using the example in Figure 1 which illustrates the communication and update schedule for synchronous (left) & asynchronous (right) aggregations for n = 5 clients in
FL with Q = 3 local updates.
As shown in this Figure, for every update at the server-lever under synchronized updates (left figure), the server has to wait for all the selected clients.
Nevertheless, these clients build their local updates based on the recent version of the server’s parameter.
On the contrary, in the asynchronous scenario (right figure), the server updates the global parameter once it receives a new update from some client.
The main challenge for the asynchronous setup is the staleness between download and upload time from/to the server.
We design PersA-FL based on the second communication scenario.
A.2 Algorithm: Extension
Here, we present the client algorithm consisting all options. ⋄ Client Algorithm : Let us explain the operations of i -th client using the pseudo code in Algorithm 3.
Client i repeats an iterative procedure which is composed of three phases, (i) downloading the most up-to-date model from the server as in Step 3, (ii) performing Q local updates starting from the parameters of the downloaded model with respect to the cost function of the underlying problem, (1), (3), or (8), as in Steps 5-13, and (iii) uploading the sum of updates on the server as in Step 15.
Note that η ≥ 0 is the local stepsize, a hyperparameter.
The main idea for the local updates is to perform Q sequential SGD steps on the local cost.
Below, we list our stochastic estimation for the full gradients of each loss function introduced in Section 2:
• Option A: This option intends to minimize (1).
Therefore, for each client i at each local round q , we sample an independent data batch from p i and compute an unbiased estimation of the vanilla loss as in (2). • Option B: By performing this option, we aim to minimize the MAML cost function in (3).
As we saw in Section 2, the full gradient can be computed according to (4).
Following [8], we sample three data batches to compute a biased estimation of (4) as follows:
[Equation]
We will discuss the variance and bias of this estimator in Subsection B. • Option C:
Finally, we invoke this option to minimize the ME personalized loss in (8).
As we mentioned earlier, the full gradient of this cost is (9), where for a fixed w , we may obtain ˆ θ i ( w ) by minimizing (10).
Instead, following [6], we define the stochastic approximation ˜
h
i ( θ i , w, D i ) as in Step 11, and minimize this function with respect to θ i to obtain an approximate solution ˜ θ i ( w ) where the gradient’s norm is less than some threshold ν ≥ 0 .
Therefore, we approximate (9) with the following estimator:
[Equation]
Let us denote the expectation of ˜ h
i ( . )
as h i ( . ) .
Then, for λ > L , the expected function is ( λ + L ) -smooth and ( λ − L ) -strongly convex due to the properties of Moreau Envelopes [6].
Then according to the property of [3, 6], for some ν ≤ 1 (e.g., 10 − 5 ), we can find ˜ θ i ( w ) in
O ( λ + L λ − L log( 1 ν )) iterations.
We will also discuss the properties of (12) in Subsection B.3.
Next, we present the convergence result of our method for the three formulations.
A.3 Comparison
Table 1 illustrates the properties of our proposed method and provides a comparison between our algorithm and underlying analysis with some related seminal works.
As shown in this table, building
1: input: number of local steps Q , local stepsize η , MAML stepsize α , Moreau Envelope (ME) regularization parameter λ , minimum batch size b , estimation error ν .
2: repeat 3: read w from the server ▷ download phase 4: w i, 0 ← w 5: for q = 0 to Q − 1 do ▷ local updates 6: sample a data batch D i,q from distribution p i ▽ 3 options: ▷ Option A (AFL) 7: w i,q +1 ← w i,q − η ∇ ˜
f i ( w i,q , D i,q ) ▷ Option B ( PersA-FL: MAML ) 8: sample two data batches D
′ i,q , D ′′ i,q from distribution p i 9: w i,q +1 ← w i,q − η (cid:104)
I − α ∇ 2 ˜ f
i ( w i,q , D ′′ i,q ) (cid:105) ∇ ˜
f i (cid:16) w i,q − α ∇ ˜ f i ( w i,q ,
D ′ i,q ) , D i,q (cid:17) ▷ Option C ( PersA-FL: ME ) 10: ˜ h
i ( θ i , w i,q , D i,q ) :=
˜ f
i ( θ i , D i,q ) + λ 2 ∥
θ
i − w i,q ∥11: minimize ˜ h
i ( θ i , w i,q , D i,q ) w.r.t.
θ i up to accuracy level ν to find ˜ θ
i ( w i,q ) : (cid:13)(cid:13)(cid:13) ∇ ˜ h
i (cid:16) ˜ θ i ( w i,q ) , w i,q , D i,q (cid:17)(cid:13)(cid:13)(cid:13) ≤ ν 12: w i,q +1 ← w i,q − ηλ ( w i,q − ˜ θ
i ( w i,q )) 13: end for 14: ∆ i ← w i, 0 − w i,Q 15: client i broadcasts ∆ i to the server ▷
[Table]
phase 16: until not interrupted by the server upon the results in [8, 6], we extend the capability of FL to staleness.
Table 1 also contains the convergence results for our proposed algorithm, which we will discuss in more details in Section ?? .
[Table]
This Work
[Table]
[Equation]
[Table]
PersA-FL: ME
[Equation]
B Convergence Results
In this section, we introduce the technical theorems and lemmas to show the convergence of our method for the three described scenarios.
First, we introduce the common assumptions we will use in our analysis for all the three choices of Algorithm 3.
As mentioned earlier, we require some additional assumptions to show the convergence of MAML , which we will introduce in Subsection B.2.
After stating the assumptions, we will present the convergence results.
Recall that the server updates its model at round t using the updates sent by client i t ∈ [ n ] .
We denote Ω( t ) as the timestep of the round at which client i t has received the server’s parameters before applying its Q local updates.
In other words, (Ω( t ) , t ) denote the download and upload rounds for client i t .
Now, we introduce the assumption of maximum delay.
Assumption 3 (Bounded Staleness) .
For all server steps t ≥ 0 , the staleness or effective delay between the model version at the download step Ω( t ) and upload step t is bounded by some constant τ , i.e.,
[Equation]
and the server receives updates uniformly, i.e., i t ∼ Uniform([ n ]) .
The above assumption is standard in the analysis of asynchronous methods, specifically in heterogeneous settings [25, 35, 2, 18, 29, 1].
Assumption 3 guarantees that all clients remain active over the course of training.
However, they have transient delays and perform updates with staleness.
Next, we present our only assumption on the function class, i.e., smooth non-convex.
Assumption 4 (Smoothness) .
For all clients i ∈ [ n ] , function f i : R d → R is bounded below, differentiable, and L -smooth, i.e., for all w, u ∈ R d ,
[Equation]
[Equation]
The smoothness assumption is conventional in the analysis of non-convex functions.
We also assume boundedness from below, which is reasonable since the ultimate goal is to minimize the functions.
We also denote f ⋆ = min i ∈ [ n ] f ⋆i , where according to this definition, we can immediately see that f ⋆ ≤ min w ∈ R d F ( b ) ( w ) and f ⋆ ≤ min w ∈ R d F ( c ) ( w ) .
Now, we present our assumptions on bounded stochasticity and heterogeneity.
Assumption 5 (Bounded Variance) .
For all clients i ∈ [ n ] , the variance of a stochastic gradient ∇ ℓ i ( w, ξ i ) on a single data point ξ i ∈ S i is bounded, i.e., for all w ∈ R d
[Equation]
Assumption 5 is standard in the analysis of SGD -based methods and has been used in many relevant works [30, 25, 16, 33, 17, 18, 31].
Since we perform updates using data batches, we also need to show the stochastic variance for the sampled batches.
Recall that for simplicity; we assumed that all batch sizes are larger than b ≥ 1 , thus, we have:
[Equation]
Next, we present the bounded heterogeneity assumption.
Assumption 6 (Bounded Population Diversity) .
For all w ∈ R d , the gradients of local functions f i ( w ) and the global function f ( w ) satisfy the following property:
[Equation]
The above assumption measures the population diversity (heterogeneity) between the gradients.
In heterogeneous settings, this bound indicates the similarity between different distributions.
Fallah et al. [8] show connections between heterogeneity and the Wasserstein distance between the distributions under certain assumptions.
The above assumptions are sufficient to prove the convergence of our method (Algorithms 1 & 3) under Option A and Option C.
Therefore, we present the convergence analyses starting from our results on AFL .
B.1 Asynchronous Federated Learning (Option A)
We now demonstrate the convergence rate of our method for the cost function in (1).
Theorem 2 ( AFL ) .
Let Assumptions 3-6 hold, β = 1 , and η = 1 Q √ LT .
Then, the following property holds for the joint iterates of Algorithms 1 & 3 under Option A on Problem (1) : for any timestep
T ≥ 160 L ( Q +7)( τ +1) 3 at the server
[Equation]
The proof of Theorem 2 is provided in Appendix C.
This theorem suggests a convergence rate of
O (cid:16) 1 √ T (cid:17) + O (cid:16)
Qτ 2 T (cid:17) for asynchronous federated learning AFL .
Our analysis removes the unnecessary boundedness assumption on the gradient norm.
Remark 1.
Selecting β
= 1 in Theorem 2, results in a sub-optimal first-order stationary rate for smooth non-convex cost functions.
However, this is an arbitrary choice for the value of β and can be relaxed to any β =
O (1) similar to [25].
Next, we present the convergence of PersA-FL: MAML along with some technical lemmas borrowed from [8].
B.2 Personalized Asynchronous Federated
As we discussed in Section 3, we require the second-order derivatives of the local functions to compute the gradients of the personalized costs in (3).
Accordingly, we consider similar assumptions for the second-order derivatives as Assumptions 4-6.
Assumption 7 (Second-Order Properties) .
For all clients i ∈ [ n ] , the following properties hold for the Hessian of each f i : R d → R , the variance of a stochastic Hessian ∇ 2 ℓ i ( w, ξ i ) on a single data point ξ i ∈ S i , and the global Hessian ∇ 2 f ( w ) : for all w, u ∈ R d ,
[Equation]
Assumption 7 is conventional in the analysis of methods with access to second-order information [7, 8, 28, 31].
Finally, we adopt another assumption from [11, 8, 10] on the gradient norm to simplify the analysis for the MAML cost.
Assumption 8 (Bounded-Gradient) .
There exists a constant G such that for all clients i ∈ [ n ] , and any parameter w ∈ R d , [Equation]
[Equation]
To the best of our knowledge, seminal works on MAML loss mainly consider this assumption to simplify the properties of the personalized function.
Note that we consider Assumptions 7-8 only in the analysis of PersA-FL
(Algorithms 1 & 3) under Option B.
Under Assumptions 4 and 8, the properties in (21) and (18) can be simply derived with γ h = 2 L and γ g = 2 G [8].
Before stating the convergence of PersA-FL: MAML , let us state some technical lemmas on the personalized MAML cost function.
Lemma 1 ([8], Lemma 4.2 - Smoothness: MAML ) .
Let Assumptions 4 and 8 hold.
Then, F ( b ) i in (3) is
L b -smooth, i.e., for all clients i ∈ [ n ] , and any parameters w, u ∈ R d ,
[Equation]
where L b := L (1+ αL ) 2 + αρG .
Lemma 1 indicates that the personalized cost in (3) is also smooth.
The smoothness parameter
L b depends on the personalization hyperparameter α .
Increasing the value of α results in higher smoothness constant L b .
The smoothness property of MAML cost under multi-step personalization (instead of one) is shown in [31][Lemma 3].
Lemma 2 ([8], Lemma 4.3 - Bounded Variance: MAML ) .
Let Assumptions 4, 5, 7, and 8 hold, and data batches D , D ′ , D ′′ be randomly sampled according to data distribution p i .
Then, the following properties hold for the stochastic personalized gradient ∇ ˜ F ( b ) i ( w, D ′′ , D ′ , D ) :
[Equation]
for all w ∈ R d , where σ 2 b :=
[Equation]
1+ αL
[Equation]
2 σ
[Equation]
g
[Equation]
Lemma 2 highlights two important results.
First, the stochastic gradient in (11) is a biased estimation of the full gradient 4.
The biasness is controlled by two factors, personalization stepsize α , and batch size b .
Therefore, we obtain an unbiased estimation under no personalization, i.e., α = 0 .
However, as we select a larger α , we require more samples to reduce the error imposed by biased gradient estimations.
Second, similar to Assumption 5 on the vanilla cost; we have a tight variance based on α and b .
Lemma 3 ([8], Lemma 4.4 - Bounded Population Diversity: MAML ) .
For all w ∈ R d , the gradients of local personalized functions F ( b ) i ( w ) and the global function F ( b ) ( w ) satisfy the following property: [Equation]
[Equation]
The above lemma determines the heterogeneity of the personalized gradients ∇ F ( b ) i ( w ) based on the heterogeneity of gradient and Hessian.
One can see the connection of this bound with
O ( γ 2 g )
+ α 2 O ( γ 2 h ) , whereby setting α = 0 , we recover the same heterogeneity in terms of O ( · ) notion.
Lemma 4 (Bounded-Gradient: MAML ) .
For all clients i ∈ [ n ] , and any parameter w ∈ R d ,
[Equation]
This lemma indicates that the bound on the norm of personalized gradients potentially increases under a larger personalization budget α .
Building upon the results in Lemmas 2-(27), we are now ready to present the convergence result for
PersA-FL: MAML .
Theorem 3 ( PersA-FL: MAML ) .
Let Assumptions 3-8 hold, α ≥ 0 , β = 1 , and η = 1 Q √ L b T .
Then, the following property holds for the joint iterates of Algorithms 1 & 3 under Option B on Problem (3) : for any timestep T ≥ 64 L b at the server
[Equation]
The proof of this theorem can be found in Appendix D. Theorem 3 shows a convergence rate of
O (cid:16) 1 √ T (cid:17) + O (cid:16) τ 2 T (cid:17) + O (cid:16) α 2 σ 2 g b (cid:17) for PersA-FL algorithm under MAML setup.
Now, let us compare this rate with the convergence rate of AFL and Per-FedAvg , as in Table 1.
The last term in the above rate, i.e., O (cid:16) α 2 σ 2 g b (cid:17) accounts for personalization with biased gradient estimation.
Moreover, compared to Per-FedAvg , the second term of this rate is different, which accounts for the maximum delay in asynchronous updates.
For instance, under a fixed personalization budget α ≥ 0 , our method requires T =
O ( ε − 2 ) and
b = O ( ε − 1 ) to reach an ε -approximate first-order stationary solution.
The last expression can also be controlled through a combined stepsize α and batch size b .
It is consistent with intuition, meaning more samples are needed to obtain a higher degree of personalization.
Next, we will present the analysis of PersA-FL: ME .
B.3 Personalized Asynchronous Federated Learning: Moreau Envelope Setup (Option C)
In this subsection, we show three technical lemmas on the bounded variance of stochasticity and heterogeneity as well as smoothness for ME formulation (8) and then present the convergence rate of
PersA-FL for this personalization framework.
The proof of all results in this subsection is provided in Appendix E.
First, we present the smoothness property of ME loss.
Lemma 5 (Smoothness: ME) .
Let Assumption 4 holds and λ ≥ κL for some κ > 1 .
Then, F ( c ) i in (8) is L c -smooth, where L c = λκ − 1 .
According to Lemma 5, we limit our exploration to λ > L which satisfies the smoothness constraint for the ME formulation.
In fact, according to Appendix E, one can also see that originally, each F ( c ) i ( · ) is λLλ − L -smooth which is also smaller than L c = λκ − 1 .
As we mentioned in Section 2, when λ → ∞ ,
ME framework converts to vanilla FL .
The smoothness property in Lemma 5 is tight because, L c → L if λ → ∞ .
Corollary 1 ([6], Proposition 1) .
If λ ≥ 2 L , then Lemma 5 implies that F ( c ) i in (8) is λ -smooth.
Lemma 6 (Bounded Variance: ME) .
Let Assumptions 4 and 5 hold, λ ≥ κL (for some κ > 1 ), and the data batch D be randomly sampled according to data distribution p i .
Then, the following properties hold for the stochastic personalized gradient ∇ ˜ F ( c ) i ( w, D ) : for all w ∈ R d ,
[Equation]
This lemma is analogous to Lemma 2 in Subsection B.2.
In Lemma 6, we show an upper bound on the variance and bias of the stochastic gradient compared to the full gradient.
Note that when λ → ∞ , we know that ˆ θ
i ( w ) → w .
Therefore, by fixing ˜ θ
i ( w ) = w , it is guaranteed that ν
= 0 , thus our gradient estimation becomes unbiased and the variance similar to (17).
Lemma 7 (Bounded Population Diversity: ME) .
Let personalization hyperparameter λ ≥ 7 L .
Then, for all w ∈ R d , the gradients of local personalized functions F ( c ) i ( w ) and the global ME function
F ( c ) ( w ) satisfy the following property:
[Equation]
Lemma 7 provides a bound on population diversity of ME as a factor of γ 2 g .
Similar to what we explained so far, for λ → ∞ , the heterogeneity bound turns into γ 2 g .
Remark 2.
In the analysis for Theorem 4, we consider bounded population diversity as in Assumption 6, average bounded diversity.
[6][Assumption 3] and [34][6.1.1 Assumptions and Preliminaries, (vii)] consider a slightly stronger version of this assumption, namely uniformly “bounded heterogeneity” which is defined as follows:
[Equation]
Under the modified assumption in (31) , we can improve γ 2 c := 16 λ 2 λ 2 − 8 L 2 γ 2 g .
Now, we present our convergence result of PersA-FL: ME under Assumption [-3-
 Theorem-]3Theorem 4 ( PersA-FL: ME ) .
Let Assumptions 3-6 hold, λ ≥ 7 L , β = 1 , and η = 1 Q √ L c T .
Then, the following property holds for the joint iterates of Algorithms 1 & 3 under Option C on Problem (8) : for any timestep T ≥ 288 L c ( Q +7) ( τ +1) 2 at the server
[Equation]
This theorem proposes a convergence rate of O (cid:16) 1 √ T (cid:17) + O (cid:16) τ 2 T (cid:17) + O (cid:16) λ [Equation] 2 ( λ − L ) 2 (cid:17) for PersA-FL under ME formulation.
Again, under the exact same reasoning as Lemma 6, we know that ν
=when λ → ∞ , thus the convergence rate simply reduces to vanilla AFL with no personalization.
Let us compare this convergence rate with the rate of pFedMe [6] in Table 1.
By comparing the last terms in both rates, O (cid:16) λ 2 ν 2 ( λ − L ) 2 (cid:17) and O (cid:18) λ 2 ( 1 b + ν 2 ) ( λ − L ) 2 (cid:19) , one can see that the additional term 1 b in the convergence rate of pFedMe , implies that even under λ → ∞ , the last term does not vanish unless with large data batches, i.e., b = O ( ε ) .
Therefore, from the personalization perspective, our analysis provides a tighter bound compared to pFedMe .
C Asynchronous Federated Learning
Proof of Theorem 2.
First, we present a set of useful inequalities we will use in the proof.
For any set of m vectors { w i } mi =1 such that w i ∈ R d , and a constant α > 0 , the following properties hold: for all i, j ∈ [ m ] :
[Equation]
32d
[Equation]
Now, let us rewrite the update rule of the joint iterates in Algorithms 1 & 3 Option A at time t as follows:
• Client update:
[Equation]
• Server update:
[Equation]
For simplicity, we denote ˜ ∇ f i ( w ) = ∇ ˜
f i ( w, D i ) .
Therefore, at round t , the server updates its parameter by receiving ∆ i t from some client i t ∈ [ n ] , as follows:
[Equation]
Moreover, Due to Assumption 4, we can infer that f is L -smooth, thus
[Equation]
First, we provide a lower bound on term S a 1 in (37).
Prior to show the bound, let us denote ˜ g ti =
Q − 1 (cid:80) [Equation] = [Equation]
i
[Equation]
i =
[Equation]
ti , g ti =
Q
[Equation] q = [Equation] ∇ f i (cid:16) w Ω( t ) i,q (cid:17) , and g t = 1 n n (cid:80) i =1 g ti .
Therefore,
[Equation]
=
[Equation]
∇ f [Equation] t [Equation]
n [Equation] i = [Equation]
ti
[Equation] (39) = E [Equation]
∇ f [Equation] [Equation] n
[Equation]
i =
[Equation]
p i
[Equation] ti [Equation] (cid:42)
[Equation] w [Equation] [Equation]
n (cid:88) i = [Equation] ti [Equation]
(40) = [Equation] [Equation]
(cid:13)(cid:13) [Equation] (cid:13)(cid:13) [Equation]
[Equation] ) (cid:13)(cid:13) [Equation] (cid:13)(cid:13) g t
[Equation] ( w t ) (cid:13)(cid:13)(42) = [Equation] ( w
[Equation] ) (cid:13)(cid:13) [Equation] (cid:13)(cid:13) g
[Equation] ( w t ) (cid:13)(cid:13) [Equation]
Moreover, the following holds for S a 2 in (37):
[Equation]
Now, according to (37), (43)
[Equation]
and (44)
[Equation]
we have:
[Equation]
where we bound S a 3 and S a 4 as follows:
[Equation] [Equation]
n
[Equation]
(cid:88)
[Equation]
(cid:13)(cid:13)
[Equation]
( w
[Equation]
(cid:13)(cid:13)
[Equation]
(47) =
[Equation]
n
[Equation]
(cid:88) i =
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
(cid:88)
[Equation]
=
[Equation]
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
=n
[Equation]
(cid:88) i =
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
Q
[Equation]
(cid:88) q
= (cid:104)
[Equation]
(cid:16)
[Equation]
( t ) i
[Equation]
(cid:17)
[Equation]
f i ( w t ) (cid:105)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) (49) (32d)
[Equation]
n n (cid:88) i =
Q
[Equation]
(cid:88) q
= (cid:13)(cid:13)(cid:13)
[Equation]
(cid:16)
[Equation]
( t ) i,
[Equation]
(cid:17)
[Equation]
( w
[Equation]
(cid:13)(cid:13)(cid:13) 2 , (50)
S a
[Equation]
= n (cid:88) i = (cid:13) (cid:13)(cid:13)
Q − 1 (cid:88) q = ˜ ∇ f i (cid:16) w
[Equation]
( t ) i,q (cid:17) (cid:13) (cid:13)(cid:13)(51) (32d) ≤
Q n (cid:88) i =
[Equation]
[Equation]
[Equation]
(cid:35) ⇒ (54)
[Equation]
[Equation]
Therefore, due to (45)-(50) and (59)-(60)
[Equation]
we have
[Equation]
Thus
[Equation]
it is sufficient to bound the following expression in S a 5 :
[Equation]
Now, we show a bound on the evolution of local updates at an arbitrary round s ≥ 0 ,
i
[Equation]
e., the distance between w si,q and w s :
[Equation]
Note that we can select stepsize η ≤ 1 4 L ( Q +1) such that
[Equation]
[Equation] therefore, due to (72)-(78), we have:
[Equation]
for all q ∈ [ Q ] and s ≥ 0 .
We now will use (72)-(84) to provide a bound on the expression in S a 7 .
Again, note that according to Algorithms 1 &
[Equation]
we have:
[Equation]
∇ f (cid:16) w Ω( s ) (cid:17) (cid:13)(cid:13)(cid:13)(cid:35) .
Let ϕ = 8 η 2 Q 2 (1+2 Q )(1+ β 2 ) , then according to (67)-(89) nϕ E
[Equation]
] ≤ τ (cid:34) [Equation] (cid:88) s = t − τ (cid:13)(cid:13)
[Equation]
(cid:35)
[Equation]
nQ
[Equation]
(cid:88) i =
[Equation]
Q
[Equation]
(cid:88) q
= (cid:13)(cid:13)(cid:13) w
[Equation]
( t ) − w Ω( t ) i,q (cid:13)(cid:13)(cid:13) 2 .
(90) ≤ τ
[Equation]
+ τ
[Equation]
(cid:88) s
= t −
[Equation]
E (cid:13)(cid:13)(cid:13) ∇
[Equation]
(cid:16) w Ω( s ) (cid:17) (cid:13)(cid:13)(cid:13)(91) +
[Equation]
(cid:13)(cid:13)(cid:13)
[Equation]
(cid:16)
[Equation]
( t ) (cid:17) (cid:13)(cid:13)(cid:13)(92) ≤ ( τ
[Equation]
1) (cid:2)
[Equation]
a
[Equation]
γ [Equation] g (cid:3)
[Equation]
E (cid:13)(cid:13)(cid:13)
[Equation]
f (cid:16) w Ω( t ) (cid:17) (cid:13)(cid:13)(cid:13) 2 + τ t − 1 (cid:88) s
= t − τ
E (cid:13)(cid:13)(cid:13) ∇ f (cid:16) w Ω( s ) (cid:17) (cid:13)(cid:13)(cid:13)
[Equation]
Thus, by combining (61)-(94)
[Equation]
we have the following inequality:
[Equation]
where by rearranging
[Equation]
we obtain the following inequality:
[Equation]
Now, note that for any s ≥ 0 ,[-
-]
[Equation]
Therefore
[Equation]
we add up the inequality in (99)-(103), for t = 0 , 1 , . . .
T − [Equation] and obtain
[Equation]
Thus [Equation] by setting β = 1 and η = 1 Q √ LT
[Equation]
we can simply see that
[Equation]
4 L ( Q +1) , (109) for T ≥ 160 L ( Q +7)( τ +1) 3 .
Therefore, we can conclude the final result in Theorem 2 under this choice of η and β .
D
Proof of Theorem 3.
To simplify (11), we denote ˜ ∇ F ( b ) i
[Equation]
( w, D ′′ i , D ′ i , D i ) .
Then similar to (36), at round t , the update rule for Option B can be written as follows:
[Equation]
According to Lemma 1,
[Equation] [Equation]
Similar to the inequalities in (38)-(43), we first show a lower bound on term S b 1 in (111).
We also denote ˜ g
ti =
Q − 1 (cid:80)
[Equation]
i =1 g ti for simplicity.
Note that ˜ g ti and g ti are the stochastic and deterministic gradients of the personalized cost functions F ( b ) i at stale parameters.
According to these definitions, we have
[Equation]
where as we discussed in (24), µ b measures the unbiasedness in the estimation of the personalized stochastic gradient.
[Equation]
(cid:43)(cid:35) =
E (cid:104)(cid:68) ∇ F ( b ) ( w t ) , ˜ g t (cid:69)(cid:105) (116) =
Q E (cid:13)(cid:13)(cid:13) ∇ F ( b ) ( w t ) (cid:13)(cid:13)(cid:13) 2 + E (cid:68) ∇ F ( b ) ( w t ) , E (cid:2)
˜
g t − g t (cid:3)(cid:69) (117) + E (cid:68) ∇ F ( b ) ( w t ) , g t − Q ∇ F ( b ) ( w t ) (cid:69) (118) (32c) ≥ Q E (cid:13)(cid:13)(cid:13) ∇ F ( b ) ( w t ) (cid:13)(cid:13)(cid:13) 2 − 1 4 E (cid:13)(cid:13)(cid:13) ∇ F ( b ) ( w t ) (cid:13)(cid:13)(cid:13) 2 − (cid:13)(cid:13) E (cid:2) g t − ˜ g t (cid:3)(cid:13)(cid:13)(119) −4 E (cid:13)(cid:13)(cid:13) ∇ F ( b ) ( w t ) (cid:13)(cid:13)(cid:13) 2 − E (cid:13)(cid:13)(cid:13) g t − Q ∇ F ( b ) ( w t ) (cid:13)(cid:13)(cid:13)(120) (112) − (114) ≥ 2 Q −
E (cid:13)(cid:13) (cid:13) ∇ F ( b ) ( w t ) (cid:13)(cid:13) (cid:13) 2 − E (cid:13)(cid:13) (cid:13) g t − Q ∇ F ( b ) ( w t ) (cid:13)(cid:13) (cid:13)
[Equation]
and
[Equation]
Therefore, according to (111), (121), and (122),
[Equation]
where similar to (47)-(50) [Equation] we can bound S b 3 as follows:
[Equation]
Moreover, we can show an upper bound on S b 4 akin to (126)-(129):
S b 4 = n (cid:88)
[Equation]
[Equation]
[Equation]
i
[Equation]
t )
i
[Equation]
i
[Equation]
(cid:13)(cid:13)(cid:13)
[Equation]
(cid:13)(cid:13)(cid:13)
[Equation]
b )
i
[Equation]
i
[Equation]
(cid:17)
[Equation]
[Equation]
(cid:0)
[Equation] t [Equation] (cid:13)(cid:13)(cid:13)+ (cid:13)(cid:13)(cid:13) ∇ F ( b ) i (cid:0) w t (cid:1) − ∇ F ( b ) (cid:0) w t (cid:1) (cid:13)(cid:13)(cid:13)+ (cid:13)(cid:13)(cid:13) ∇ F ( b ) (cid:0) w t (cid:1) (cid:13)(cid:13)(cid:13)(cid:35) ⇒ (129)
E [ S b 4 ] (129) ≤ 4 Q n (cid:88) i =
Q − 1 (cid:88) q =
E p i (cid:34)(cid:13)(cid:13) (cid:13)
˜ ∇ F ( b ) i (cid:16) w Ω( t ) i,q (cid:17)
− ∇ F ( b ) i (cid:16) w Ω( t ) i,q (cid:17) (cid:13)(cid:13) (cid:13)(cid:35) (130) + 4 Q n (cid:88) i =
Q − 1 (cid:88) q =
E (cid:13)(cid:13)(cid:13) ∇ F ( b ) i (cid:16) w Ω( t ) i,q (cid:17)
− ∇ F ( b ) i (cid:0) w t (cid:1) (cid:13)(cid:13)(cid:13)(131)
+ 4 Q n (cid:88) i =
Q − 1 (cid:88) q =
E (cid:13)(cid:13)(cid:13) ∇ F ( b ) i (cid:0) w t (cid:1) − ∇ F ( b ) (cid:0) w t (cid:1) (cid:13)(cid:13)(cid:13)(132) +
4 Q n (cid:88) i =
Q − 1 (cid:88) q =
[Equation]
Therefore, due to (123)-(125) and (134)-(135), we have
[Equation]
Now, we provide an upper bound on S b 5 in (139) as follows:
[Equation]
where the first term determines the evolution of local updates and the second term considers the effect of asynchronous updates.
Therefore, using Lemma
[Equation]
we have
[Equation]
[Equation]
[Equation]
(cid:16)
[Equation]
(cid:16) [Equation] (cid:17)(cid:13)(cid:13)(cid:13) [Equation]
(cid:13)(cid:13)(cid:13)
[Equation] (cid:16) [Equation] t )
i [Equation] r (cid:17)(cid:13)(cid:13)(cid:13) [Equation]
(cid:21) [Equation]
[Equation]
[Equation]
So, according to (136)-(155),
[Equation]
where by adding the terms in (156)-(158), for t = 0 , 1 , . . .
T −
[Equation]
and rearranging them
[Equation]
we obtain the following inequality:
[Equation]
Finally, we can conclude the proof by fixing β = 1 and η := 1 Q √ L b T for T ≥ 64 L b , hence η ≤ 1 8 βL b Q .
E Personalized Asynchronous Federated Learning: ME
We start by showing (12) [Equation] where according to the definitions in (8) and (10), we have
[Equation]
therefore,
[Equation]
Before, presenting the proof of Theorem
[Equation]
we proceed by providing the proof of Lemmas 5,
[Equation]
and 7.
Proof of Lemma 5.
Let w, v be two arbitrary vectors in R d .
Then, we have:
[Equation]
( w ) (cid:105) − λ (cid:104) v − ˆ θ i ( v ) (cid:105) (168) (163) =
∇ f i (cid:16) ˆ θ i ( w ) (cid:17) − ∇ f i (cid:16) ˆ θ i ( v ) (cid:17) ⇒ (169) (cid:13)(cid:13)(cid:13) ∇ F ( c ) i ( w ) −
∇ F ( c ) i ( y ) (cid:13)(cid:13)(cid:13) =
(cid:13)(cid:13)(cid:13) ∇ f i (cid:16)
ˆ θ i ( w ) (cid:17) − ∇ f i (cid:16) ˆ θ i ( v ) (cid:17)(cid:13)(cid:13)(cid:13) (170) (14) ≤ L (cid:13)(cid:13)(cid:13)
ˆ θ i ( w ) − ˆ θ i ( v ) (cid:13)(cid:13)(cid:13) (171) (163) =
L (cid:13)(cid:13)(cid:13)(cid:13) w − 1 λ ∇ f i (cid:16) ˆ θ i ( w ) (cid:17) − v + 1 λ ∇ f i (cid:16) ˆ θ i ( v ) (cid:17)(cid:13)(cid:13)(cid:13)(cid:13) (172) ≤ L ∥ w − v ∥ + L λ (cid:13)(cid:13)(cid:13) ∇ f i (cid:16) ˆ θ i ( w ) (cid:17) − ∇ f i (cid:16) ˆ
θ i ( v ) (cid:17)(cid:13)(cid:13)(cid:13) (173) =
L ∥ w − v ∥ + L λ (cid:13)(cid:13)(cid:13) ∇ F ( c ) i ( w ) − ∇ F ( c ) i ( y ) (cid:13)(cid:13)(cid:13) ⇒ (174) (cid:13)(cid:13)(cid:13)
[Equation]
which means F ( c ) i is λLλ − L -smooth.
Note that for λ ≥ κL , for some κ > 1 ,
[Equation]
This concludes the statement of Lemma 5.
Proof of Lemma 6.
According to Step 11 of Algorithm 3, let us introduce full and stochastic auxiliary cost functions
h i ( · ) and ˜ h
i ( · ) as follows:
[Equation]
where due to (177), we have
[Equation]
hence, we can show (28) as follows:
[Equation]
You can find the proof of (29) in [6][Appendix A.2].
Proof of Lemma 7.
First, note that we have
[Equation] [Equation]
n
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
w
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) [Equation]
[Equation]
n
[Equation]
n
[Equation]
i = (cid:13)(cid:13) (cid:13) (cid:13) (cid:13)(cid:13)
[Equation]
(cid:88) j =
[Equation]
w
[Equation]
(cid:105) (cid:13)(cid:13) (cid:13) (cid:13) (cid:13)(cid:13) (190) (
[Equation]
[Equation]
So, we simplify the upper bound as follows:
[Equation]
Note that we can bound (199) and (200) according to Lemma 18:
[Equation]
and also given the fact that function f ( · ) as well as each function f i ( · ) are L -smooth, we can bound (196), (197), and (198) as follows: [Equation] [Equation]
[Equation]
Therefore, according to (208)-(207), we have n n
[Equation]
i = [Equation] n [Equation]
w [Equation] ⇒ [Equation]
n [Equation] i [Equation]
w
[Equation]
which concludes the proof.
Now, we are ready to state the proof of Theorem 4.
Proof of Theorem 4.
We write ˜ ∇ F ( c ) i
[Equation]
= ∇ ˜ F
[Equation]
i ( w, D i ) to simplify (12).
Then, the update rule for Algorithms 1 & 3 under Option C can be written as follows:
[Equation]
where similar to (111)-(141), we can show that:
[Equation]
with L c , µ c , σ c , γ c as defined in Lemmas 5, 6, and 7.
Thus, to show the convergence rate of our method for the cost function in (8), it would only be sufficient to provide an upper bound on S c 1 .
First, note that similar to (67)-(71), we have
[Equation]
Now [Equation] if we introduce stepsize η such that η ≤ 1 4 L c ( Q +1) , similar to (72)-(77) and (85)-(89)
[Equation]
the following two inequalities holds for S c 2 and S c 3 :
[Equation]
where by denoting ϕ = 8 η 2 Q 2 (1+2 Q )(1+ β 2 ) , we have
[Equation]
Then, according to (211)-(217)
[Equation]
we obtain
E
[Equation]
[Equation]
where by averaging the terms in (218)-(222), for t = 0 , 1 , . . .
T − 1 , and rearranging them (similar to (105)-(107), we can conclude the following inequality:
[Equation]
Finally, by fixing η =
[Equation]
√ [Equation] c T , for T ≥ 288 L c ( Q +7)( τ +1) 2 , we obtain the sublinear convergence rate in Theorem 4.
F Numerical Experiments
In this section, we discuss the concurrency, speed-up, and accuracy of our proposed method in heterogeneous settings under delayed communications.
Let us first start by explaining our simulation setup for communications with delays.
We consider a set of n = 50 different clients, where each client holds a set of random delays at the upload and download stage.
We set the random delays to be generated such that the upload delay is 4 to 6 times higher than the download delay on average.
We assume that the time for local updates is negligible compared to the time for communication and aggregation.
Then, we plot the number of active users (not idle) over the training process under asynchronous communications.
The orange chart in Figure 2 (a) describes the ratio of active users, which on average is roughly 80% over time.
Therefore, we consider the same ratio for client sampling in the synchronous updates for methods such as FedAvg and plot the activity of users in the same figure with green color.
As shown in Figure 2 (a) , the degree of concurrency for asynchronous methods is dramatically higher than their synchronous counterparts.
We also generate heterogeneous data from MNIST [21] and CIFAR-10 [20] datasets on the clients, i.e., each client holds a distinguished and unbalanced distribution of images from different classes.
For instance, client i ∈ [ n ] may mostly have samples from odd numbers in MNIST, while client j ∈ [ n ] mainly holds digits zero to four.
Over the underlying communication setup and heterogeneous data setting, we compare the speed and accuracy of FedAvg , Per-FedAvg , pFedMe , AFL , PersA-FL:
MAML , PersA-FL: ME , where the first three methods are synchronous and the rest are asynchronous.
For MNIST and CIFAR-10, we consider 2-layer and 4-layer convolutional networks [20] with pooling and dropout as well as cross-entropy loss.
For all algorithms, we consider Q = 10 local updates, and select the best λ ∈ { 15 , 20 , 25 , 30 } for ME and α ∈ { 0 . 001 , 0 .
02 } for MAML .
Moreover, we pick β ∈ { 0 . 7 , 0 . 8 , 0 . 9 , 1 . 0 , 1 . 1 , 1 . 2 } and fix η = 0 .
For both experiments, we consider the exact same communication setup and repeat each experiment 3 times and plot the test accuracy curve over time until one of the algorithms stops.
Figure 2 (b) & (c) compares the performance and speed of these algorithms.
It is worth mentioning that in the implementation of algorithms with MAML , we approximated the
Hessian-vector products via the following first-order formulation: for some small δ > 0 ,
[Equation]
δ
[Equation]
Moreover, in the bi-level optimization problem for the ME formulation, we applied a constant K =steps of SGD to obtain ˜ θ i ( w ) .
[Figure]