P ARALLEL F EDERATED L EARNING OVER H ETEROGENEOUS D EVICES
Anonymous authors
Paper under double-blind review
A BSTRACT
Federated Learning (FL) is a cutting-edge distributed machine learning framework that enables multiple devices to collaboratively train a shared model without exposing their own data.
In the scenario of device heterogeneity, the synchronous FL suffers from a latency bottleneck induced by network stragglers, which hampers the training efficiency significantly.
In addition, due to the diverse structures and sizes of local models, simple and fast averaging aggregation is not feasible anymore.
Instead, complicated aggregation operations, such as knowledge distillation, are required.
The time cost for complicated aggregation becomes a new bottleneck that limits the computational efficiency of FL.
In this work, we claim that the root cause of training latency actually lies in the aggregation-then-broadcasting workflow of the server.
By swapping the computational order of aggregation and broadcasting, we propose a new parallel federated learning (PFL) framework that unlocks the edge nodes during global computation and the central server during local computation.
This fully asynchronous and parallel pipeline enables handling device heterogeneity and network stragglers, allowing flexible device participation as well as achieving scalability in computation.
We theoretically prove that synchronous and asynchronous PFL can achieve a similar convergence rate as FL. Extensive experiments empirically show that our framework can tolerate both network stragglers and complicated aggregation, which brings up to 5 .
56 × speedup compared with traditional FL.
I NTRODUCTION
Federated Learning (FL) is an emerging distributed machine learning framework that enables numerous devices to collaboratively train a shared model without exposing their private data (Shokri & Shmatikov, 2015;
Kone ˇ cn ` y et al., 2016; Kairouz et al., 2019; Li et al., 2020).
The main advantage of FL is the decoupling of model training from the necessity of directly accessing to the training data.
Thus, in many data-sensitive scenarios, such as applications in biomedicine (Xu et al., 2021;
Courtiol et al., 2019; Xu et al., 2021) and finance (Long et al., 2021; Pfitzner et al., 2021), FL can significantly reduce the concern of privacy and security.
The most popular FL system FedAvg (McMahan et al., 2017) works in synchronous training, in which each round includes the following steps (Kairouz et al., 2019): 1) the central server broadcasts the latest global model to edge devices; 2) each edge device then updates its local model with the private data, and uploads the local model update to the central server; 3) once all local updates are received, the central server conducts the aggregation operation to produce the next global model.
FedAvg works in a serial computation manner, in which no local model optimization is allowed during communication and aggregation while no global computation is allowed during local training and local updates transmission, as shown in Fig.1-(a).
Synchronous FL (SyncFL) faces two main challenges that hinder the training efficiency.
First, device heterogeneity inevitably introduces network stragglers, which take much longer time to complete local training than common ones.
According to the rule that aggregation is conducted after all local updates are received, the slowest-responding client, due to either the slowest training speed or dropping out mid-round, becomes the bottleneck of SyncFL system.
Second, the weighted average manner is widely applied for global aggregation, which is simple and fast.
As a result, in most existing FL schemes, the aggregation time spent in the central server is ignored.
However, for [Figure] the scenario of FL over heterogeneous devices, directly averaging of local models is not feasible anymore, since local models may have diverse structures and sizes.
Recently, some methods attempt to address the model heterogeneity by performing knowledge distillation in the server side (Zhu et al., 2021b;
Lin et al., 2020), leading to complicated aggregation operation.
When the time cost for aggregation is significant, in SyncFL, as shown in Fig.1-(a), devices are blocked to wait for the central server, which limits the utilization ( i.e. , the fraction of processors actively computing at any time) of the system.
Accordingly, the time cost for global aggregation becomes another key factor influencing the training efficiency of FL.
The network latency induced by straggler has been well-studied, while the training latency induced by complicated aggregation is overlooked by the literature.
The state-of-the-art synchronous and asynchronous FL training strategies attempt to remedy the network latency issue from the edge side.
A solution called DGA is offered in (Zhu et al., 2021a), which allows extra local optimization during the process of communication, and thus saves the overall time cost to some extent.
However, as a synchronous approach, DGA still suffers from the high network latency induced by network stragglers.
An alternative strategy is to allow clients to update the global model asynchronously by dividing fast and slow clients into different subsets.
Avdyukhin et al.
(Avdiukhin & Kasiviswanathan, 2021b) propose an asynchronous local SGD model, where all the client iterates evolve in synchrony with respect to the global clock, but communicate with the server in asynchrony at arbitrary time intervals.
Instead, we claim that the bottleneck of training latency actually lies in the workflow of the server.
In the literature, this research line has not been explored yet.
In this paper, we propose a simple yet efficient scheme called parallel federated learning (PFL) to achieve training efficiency over heterogeneous devices, which can solve the above challenges simultaneously.
We claim that the bottleneck of training efficiency actually lies in the workflow of the server.
The popular FedAvg conducts serial computation between edge nodes and the central server: uploading → global aggregation → broadcasting → local optimization .
In this pipeline, as shown in Fig.1-(a), the server would be blocked to wait for receiving all local updates in order to perform the global aggregation; the edge nodes would be blocked to wait for receiving the broadcasted global model and then can begin the next local optimization.
By carefully examining the above process, it can be found that the cause of training inefficiency actually lies in the dependency of server on all local updates uploading and the dependency of clients on global aggregation.
An intuitive idea is to decouple these dependencies such that the edge clients and server can run in parallel.
Without redesigning the whole FL framework, we propose a very simple operation—exchange the execution order of global aggregation and broadcasting.
The new workflow of the server becomes: uploading (server receives only the completed local updates) → broadcasting (server broadcasts the stale global model in the buffer)
→ global aggregation (server averages the currently received local updates to get the next global model).
Under this setup, for an edge device, once the local optimization is completed, it uploads the local update to the server; right after the server receives the new local update, it immediately broadcasts the current global model (stale one) in the buffer to the edge device; the edge device no longer needs to wait for the aggregation result but starts Server: Node 1: Node 2:
{+Node 3: ) 
+}
( g  10  20  30  Aggregation Communication Local Training
Communication Aggregation Local Training Local Training the next local optimization upon the received global model; the global aggregation on new updates are then carried out.
In this way, the global aggregation is decoupled from the communication and can be performed in parallel with local training.
PFL includes both synchronous and asynchronous versions, as shown in Fig. 1-(b) and Fig.
We theoretically prove that PFL can achieve the similar convergence rate as FedAvg, and empirically show that our framework can tolerate both stragglers and complicated aggregation tasks, which brings 1 .
77 × to 5 .
56 × speedup.
In summary , PFL enjoys the following merits:
• Handling Device Heterogeneity and Network Stragglers.
The server and edge clients conduct model updates in a fully asynchronous fashion, which is favorable in handling device heterogeneity.
Moreover, the server no longer needs to wait for the straggling devices, which contribute their local updates to aggregation whenever they are available to the server.
• Flexible Device Participation.
In cross-device FL, it is expected that each client would only participate in an arbitrary number of update rounds.
Our PFL can achieve flexible device participation naturally.
Once a local client gets disconnected, it can download the global model into the buffer, upon which it can conduct a local model update and rejoin the collaborative training circle.
• Scalability in Computation.
In PFL, the global aggregation is executed in parallel with communication and local updates, thus it can be generalized to conduct more complicated operations beyond simple averaging.
For instance, we can employ knowledge distillation to deal with the model heterogeneity; use homomorphic encryption to boost security.
M ETHODOLOGY
[-Node 3: (1) ( ) (1) 
-]
[- (1) ( ) 
-]
[-g   (1) ) 1( g   ) 0( g  (2) ( ) (2) 
-]
[- (3) ( ) (3) 
-]
[- (1) ( ) 
-]
[-g   (1)  (3) ( ) 0 g   (2) ( ) 0 g   (2)  ( ) ( ) (1) 
-]
We consider federated learning with N edge devices.
Each device i has access to a private dataset
S i
= { ( x ij , y ij ) } m i j =
including m i training samples from an unknown and fixed distribution P i over
X × Y .
The sample number m i and the data distribution P i can be diverse across edge devices.
The overall goal is to train a global model f ( ω ) using m = (cid:80) Ni
=
m i from all the edge nodes, where ω is denoted as the model parameters to be optimized.
Formally, for each device i , an objective function
F ( i ) ( ω ) is defined for optimization over S i :
[Equation]
2. P ARALLEL F EDERATED L EARNING
The proposed PFL can be conducted in both synchronous and asynchronous modes.
For the synchronous mode, in each round, the server needs to wait until receiving all updates of edge devices prior to conducting the global aggregation; while for the asynchronous mode, the server allows asynchronous updates.
2.1. S YNCHRONOUS PFL
Due to the limitation of synchronized aggregation, our synchronous PFL (SPFL) cannot handle the bottleneck induced by network stragglers, which instead is tailored to alleviate the training latency induced by complicated aggregation.
Specifically, for the t -th round, SPFL works as follows:
• Local Update: Based on the received global model ω ( g ) t − 1 , each edge node i updates the local model by stochastic gradient descent using its private data:
[Equation]
where G ( i ) t − 1 represents the gradient updates computed by the i -th node based on ω ( g ) t − 1 .
• Uploading: Once the edge node i finishes the local training, it uploads the gradient update δ ( i ) to the central server.
• Broadcasting:
Right after receiving the uploaded signals { δ ( i ) } Ni =1 from all edge devices, the central server sends back to them the stale global model in the buffer.
Note that it is the aggregation result of the last round but not the up-to-date one.
If the aggregation procedure of the last round has not yet completed, the server needs to wait until its completion prior to broadcasting the global model.
• Global Aggregation: Along with broadcasting, the central server performs the aggregation operation based on the received local updates:
[Equation]
When the central server is performing aggregation, the edge nodes are performing local training.
Accordingly, by exchanging the execution order of broadcasting and aggregation, we decouple local training and global aggregation and achieve parallel speedup.
2.1. A SYNCHRONOUS PFL
Compared with the synchronous mode, the asynchronous mode enjoys more degree of freedom.
In the proposed asynchronous PFL (APFL), we attempt to simultaneously remedy the training latency induced by network stragglers on the edge side and complicated aggregation operation on the server side.
In APFL, each edge device and the central server have different clocks.
To handle this issue, we define the global clock: whenever any device communicates with the central server, the global clock will increase by 1 .
In other words, each global clock corresponds to only one device.
Specifically, for the t -th global clock, the proposed PFL works as follows:
• Local Update: Edge device i completes the communication with the central server, which receives the global model denoted as ω ( g ) t − 1 .
Based on ω ( g ) t − 1 , edge device i updates the local model by stochastic gradient descent using its private data:
[Equation]
Uploading:
After completing the local training, edge device i then uploads the gradient update δ ( i ) to the central server.
During the local training and the uploading process of device i , there would be n other devices communicating with the central server.
Therefore, when the central server receives the local update of the device i , the global clock becomes t + n + 1 . • Broadcasting:
Right after receiving the uploaded signal δ ( i ) from edge device i , the central server sends back to it the stale global model in the buffer, which is the most recent aggregation result. It would be ω ( g ) t + n if the central server is idle, or ω ( g ) t + n − r if the central server is performing aggregation, where r denotes the number of local devices that previously arrived at the server but did not participate in aggregation. The former is a special case of the latter with r = 0 .
For simplicity, we denote the stale global model as ω ( g ) t + n − r .
If the local model uploaded in the previous clock by that device has not completed aggregation, then the device must wait until it has.
Note that the local signal δ ( i ) received this time does not participate in the current aggregation process.
Therefore, the edge device i no longer needs to wait for the global aggregation, which immediately starts the next local training relying on ω ( g ) t + n − r .
The central server then performs the next aggregation in parallel with the broadcasting and the new local training on device i .
• Global Aggregation: The new broadcasting-then -aggregation mechanism, which sends to clients the stale global models but not the latest ones, makes the global aggregation decoupled from the communication.
However, this fully asynchronous manner leads to a scenario that, when the global aggregation is being executed, there are some local updates arriving.
This situation is easy to handle: the arrivals just wait for the current aggregation to be done and then launch a new aggregation.
The averaging operation is thus performed on the updates of the new received edge devices:
[Equation]
where C t represents the devices that arrived during the last aggregation, p is the number of devices in C t .
Normally, C t contains only one device. 2.1. C OMPARISON OF SYNCHRONOUS PFL AND ASYNCHRONOUS PFL The main difference between SPFL and APFL is whether the central server needs to wait for all devices. SPFL is subject to the requirement of synchronized aggregation, while APFL is not.
This results in SPFL and APFL having different speed-up ratios compared to standard federated learning.
Specifically, we divide the training time of federated learning into four parts: the time cost T l for local training, the time cost T c for communication, the time cost T w for waiting, and the time cost T g for global aggregation. Standard federated learning works in a serial manner, for which the training time of each round is T = T l + T c + T w + T g . In SPFL, the aggregation of the central server is performed in parallel with the local training of the edge nodes and the communication process, the training time for each round is T = max( T l + T c + T w , T g ) .
In APFL, there is no waiting time, we have T = max( T l + T c , T g ) .
2. C ONVERGENCE A NALYSIS
In this subsection, we present the convergence analysis to demonstrate in theory that the proposed synchronous/asynchronous PFL both can get a similar convergence as FedAvg. It is worth noting S p ee dup S p ee dup
Ratio of / w l T
T
Ratio of / l g T T MIFA SPFL PFL
[Figure]
[Figure]
that there is a standard routine for FL convergence analysis, thus we just follow it without the necessity of creating a new one.
We take the following standard assumptions (Stich, 2018; Avdiukhin & Kasiviswanathan, 2021a; Nguyen et al., 2022):
Smoothness: All local functions f i ( i ∈
[ N ]) are L -smooth.
• Bounded second moment: There exists a constant G max > 0 such that:
[Equation]
where ∇ F ( i ) ( x ) is an unbiased stochastic gradient of f ( i ) at x . • Bounded variance: There exists a constant σ ≥
such that:
[Equation]
Analysis for Synchronous PFL.
In synchronous PFL, we only exchange the execution order of global aggregation and broadcasting, which makes each edge device receive the stale global model.
At first glance, it looks a bit like asynchronous FL.
The difference is that in asynchronous FL, only part of the devices (stragglers) upload the stale gradient updates.
In synchronous PFL, the global model received by the devices is only from the last round.
Due to the above similarities, we can analyze the convergence of synchronous PFL with the help of existing studies on asynchronous FL.
Here, we follow (Avdiukhin & Kasiviswanathan, 2021a), but give a sharper bound.
Definition 1 (Synchronous PFL Sequence) Let ω ( g ) t represents the global model of synchronous
PFL, G ( i ) t represents the uploaded gradient updates computed by the i -th client based on ω ( g ) t , the global model ω ( g ) t can be expressed as:
[Equation]
where ω ( g ) [-− 
 =-]0 and ω ( g ) 1 are two different initialized models.
Definition 2 (Virtual Sequence)
We construct a virtual sequence in which each device at round t uses the ω ( g ) t − 1 to calculate the uploaded gradient updates, this virtual sequence’s global model can be expressed as:
[Equation]
Similar virtual sequences have been used before in decentralized optimization in various contexts (Lian et al., 2017; Yuan et al., 2016;
Nedi ´ c et al., 2018; Stich, 2018; Avdiukhin & Kasiviswanathan [Equation] 2021a).
Proposition 3 (Distance Bound)
For ζ t of virtual sequence and ω ( g ) t , ω ( i ) t in synchronous PFL we have:
[Equation]
[Table]
This Proposition shows that ζ t are close to ω ( g ) t and ω ( i ) t for all devices.
Its proof can be seen in Appendix.
Theorem 4 Let f max = f (cid:16) ω ( g )
(cid:17) − f ( ω ⋆ ) , where ω ⋆ is the minimizer for f , γ = √ N/ √ T we have:
[Equation]
Compared to the corresponding result for Basu et al.
and Avdiukhin & Kasiviswanathan (2021a), O (cid:16) 1 √ NT (cid:17) , when N is fixed, our results get a sharper convergence rate.
And illustrates a clearer relationship between T and N , i.e. as the number of nodes N increases, the number of communication rounds T should be larger as well.
Analysis for Asynchronous PFL.
Convergence of asynchronous PFL can be demonstrated in a similar way as above, with the difference that in asynchronous PFL, the global model received by the device will be more than one clock staler than the current global model when aggregated.
For asynchronous PFL, we can use almost the same method of Avdiukhin & Kasiviswanathan (2021a) to prove the convergence of our method.
Our contribution does not lie in the novelty of the method of proof, but rather in pointing out that our approach can be fully accommodated in existing proofs of asynchronous FL and achieve greater speed-up ratios.
For convenience, we use a notion similar to that used in synchronous PFL to describe the global model and the local model.
Except for local models, we use w t to represent the local model, since each global clock only corresponds to one device.
Definition 5 (Asynchronous PFL Sequence) Let ω ( g ) t represents the global model of asynchronous PFL, G ( i ) t represents the uploaded gradient updates computed by the i -th client based on ω ( g ) t , the global model ω ( g ) t can be expressed as: [Equation] where τ − δ t denotes the clock of the global model received by the device participating in the t -th round of aggregation, and δ t can represent the degree of asynchrony.
Proposition 6 (Distance Bound)
For ζ t of virtual sequence and ω ( g ) t , ω t , we have:
[Equation]
{+Theorem 7 Let f max = f (cid:16) ω ( g ) 
+}
(cid:17) − f ( ω ⋆ ) , where ω ⋆ is the minimizer for f , if T > N 3 and δ ≤ T 1 / 4 /N 3 / 4 we have:
[Equation]
where δ = max( δ t ) .
[Table]
In this section [Table] we provide a thorough evaluation of the proposed PFL over heterogeneous devices.
We first compare the convergence speed and accuracy of the PFL under average-based aggregation.
Next, to investigate the speedup ratio of the PFL under different scenarios, experiments are carried out with different T w /T l and T g /T l , where T w represents the reconnection time of the straggler devices, T l represents the local training time [Table] and T g represents the aggregation time.
We then show that our method is also feasible for distillation-based aggregation.
Experimental results show that our method is resilient to stragglers and can achieve a parallel pipeline to improve computational efficiency without sacrificing accuracy.
3. E XPERIMENTAL S ETUP
Straggler Devices and Sample Rate.
Following the previous study (So et al., 2022), we randomly select p × N devices where p is the dropout rate.
To constrain the asynchronous level between devices, we set T w /T l = 3 , which represents the ratio of waiting time to local training time.
We consider the worst-case scenario (Bonawitz et al., 2017), where the straggler devices artificially drop after receiving the global model.
According to the realistic FL system (Bonawitz et al., 2019), we set p = 0 .
Datasets and Models.
We conduct experiments on three datasets, including MNIST, CIFAR-10, and Tiny-Imagenet (100,000 images with 200 classes).
For all three datasets, we use the same ResNet-18 architecture for a fair and clear comparison.
Number of devices and Communication bandwidth.
In our experiments, we train up to N =devices.
We are particularly focused on mobile devices in FL, the real measured bandwidth of a mobile phone is 4 MB/s.
The cost of time to upload and broadcast updated local and global models, denoted as T c .
Baselines.
We analyze and compare the performance of PFL with three baseline schemes:
FedAvg (McMahan et al., 2017), DGA (Zhu et al., 2021a), and MIFA (Gu et al., 2021).
DGA allows extra local optimization during the communication process to implicitly reduce wasted computing power during communication.
MIFA is an asynchronous communication FL method to solve straggler problems.
These two methods address the challenge of training efficiency in FL from two different perspectives.
Data Partition: Following Avdiukhin & Kasiviswanathan (2021a), for IID setup, we randomly split the data equally into N sub-datasets; for Non-IID setup, each device has a corresponding class, where µ fraction of local data is from the corresponding class, while the rest are randomly selected from other classes.
In our experiments, µ = 0 . 3 .
3. P ERFORMANCE E VALUATION
To fully explore the communication efficiency of FL, we simulate all the three scenarios mentioned in the above section, concretely, 1) devices will randomly drop out, 2) the upload and download time of the models is measured by bandwidth 4 Mb/s, and 3) the central server needs to perform additional complex computations such as knowledge distillation or toxic device identification.
Average-based Aggregation.
Although various novel aggregation methods have been proposed,
FedAvg is still the most widely used method, so we first investigate the performance of our method under the average-based approach.
To simulate complex operations on the central server in the real environment without introducing additional effects, we perform additional complex tasks on the central server to demonstrate the superiority of the training efficiency of our approach.
Specifically, an additional stage of toxic device identification is added before the central server performs the aggregation.
The local models uploaded from each device need to be validated by the public dataset on the central server before participating in aggregation.
This is a very efficient but time-consuming detection method and, therefore, is rarely used in real-world scenarios.
As can be seen in Table 1, our method can achieve almost the same accuracy as FedAvg, while achieving a speed increase of approximately 3 . 91 × .
Speed-up is measured in run time per global clock between device and central server, normalized by the run time of FedAvg.
DGA does not reduce the computation time for each communication but rather speeds up convergence by allowing the nodes to perform local training while communicating.
Therefore, it has an acceleration ratio of only 1 × .
MIFA can get good speedups in the presence of straggler, e.g. 1 . 86 × faster on the CIFAR-10 dataset.
Our approach can improve the computational efficiency of federated learning in three dimensions simultaneously, the time for per global clock is T = max( T l + T c , T g ) , for example, when on the CIFAR-10 dataset, our method can obtain a 3 .
28 × speedup.
Impact of T w /T l on speedup ratio.
T w /T l represents the ratio of reconnection time of straggler devices to local training time.
The longer the dropout time, the greater its impact on the overall computational efficiency of SyncFL and the greater its impact on the accuracy of AsyncFL.
Therefore, it is necessary to investigate the effect of different T w /T l on computational efficiency, here we set T w /T l = 1 and T w /Tl
= 1 , 2 , 3 , 4 , 5 , and the experiment is carried out on the CIFAR-10 withdevices.
As can be seen in Fig.
3, as T w /T l increases, the computational speed-up ratio between our method and MIFA’s method gets higher and higher, and when T w /T l = 5 , the computational efficiency of our PFL can be improved by up to 5 . 56 × , while MIFA can only improve the speed-up ratio by 3 . 16 × .
Impact of T g /T l on speedup ratio.
T g /T l represents the ratio of the central server’s computation time to the local device’s training time, which is related to the number of devices and the computation capacity of the central server, and is further complicated when the computation power of the edge devices is different from each other.
To simplify the computational model, we assume that the local training time is the same for each edge device.
Here, we set T g /T l = 1 and T w /Tl = 0 .
5 , 1 , 2 , 4 , and the experiment is carried out on the CIFAR-10 with 100 devices.
As can be seen in Fig.
3, the speed-up ratio of our method reaches its maximum 2 .
52 × when T g /T l
= 1 .
Distillation-based Aggregation.
To verify the effectiveness of our method for complex aggregations, we have chosen the first distillation-based aggregation method (Lin et al., 2020).
The distillation-based approach makes better use of information from edge devices, but introduces serious computational reductions in efficiency that make it difficult to apply to real-world applications.
Following (Lin et al., 2020), we added additional public datasets to the central server and then fine-tuned the global model by distillation after coarse aggregation was completed by averaging operations.
In this experiment, the straggler node is ignored.
As can be seen from Table 2, our method is still feasible under distillation-based aggregation and can improve the computational efficiency of federated learning without sacrificing accuracy, achieving 2 × speedup when the training time for edge devices is the same as the aggregation time for the central server.
C ONCLUSION
The traditional FL works in a serial computation manner, in which communication is deemed a major bottleneck, leading to poor training efficiency.
Most existing methods attempt to improve training efficiency either by performing additional computations or by reducing the communication frequency or data amount.
In this paper, alternatively, we propose a simple yet efficient scheme called parallel federated learning (PFL).
Instead of remedying the efficiency issue from the edge side as done in existing methods, we claim that the bottleneck of computational efficiency actually lies in the workflow of the server.
Through exchanging the execution order of global aggregation and broadcasting, global aggregation is decoupled from the communication and can be performed in parallel with local training.
We theoretically prove that PFL can achieve the same convergence rate as FedAvg.
The proposed scheme has the potential to serve as a new baseline FL framework enabling a wide range of applications.
A A PPENDIX
In the Appendix, we first give the details of the convergence analysis that are missing in the text
Section 2.2.
[-Aggregation No actions Waiting time (1) ) 1( g   (1)  (1)  Local Training (3) ( ) 0 g   (2) ) 
 ( g   (3) ) 1( g   (2)  (2)  (3)  (3) -]And conduct an experiment to investigate the influence of stale global model.
B C ONVERGENCE ANALYSIS OF PFL
Proof for Synchronous PFL.
Proposition 3 (Distance Bound)
For ζ t of full synchronous FL and ω ( g ) t , ω ( i ) t of Algorithm 1, we have:
[Equation]
Proof:
[Equation] [Equation]
Similarly, E
[Equation]
− ω ( i
[Equation]
≤ γ [Equation] G 2max .
Combining these bounds, we have the following.
[Equation]
Theorem 4 Let f max = f (cid:16) ω ( g )
(cid:17) − f ( ω ⋆ ) , where ω ⋆ is the minimizer for f , we have:
[Equation]
Proof: Similar to Avdiukhin & Kasiviswanathan (2021a)
Theorem 2.4.
From smoothness Lipschitz condition on the gradients:
[Equation]
First
[Equation]
we bound ζ t
[Equation]
for ζ t
[Equation]
we have:
[Equation]
[Equation]
By the smoothness property:
[Equation]
The last term in Eq.
21 can be rewritten as:
[Equation]
[Table]
[Equation]
[Equation]
Substituting this into the Eq.
[Equation]
get:
[Equation]
The second term in Eq.
23 can be simplified by avg
[Equation]
[Equation]
For the third term in Eq
[Equation]
23 we have:
[Equation]
For the fourth term in Eq.
23 we have:
[Equation]
Substituting Eq.
24, 25, 26 into the Eq.
[Equation]
get:
[Equation]
Move E (cid:2) || ∇ f ( ζ t ) || 2 (cid:3) to the left of the inequality;
[Equation]
Taking the sum over all iterations:
[Equation]
Finally
[Equation]
we can bound || ∇ f ( ω ( g ) t )
|| in terms of || ∇ f ( ζ t )
|| as:
[Equation]
Substituting this into the inequality above on 1 T (cid:80) Tt =
0 E (cid:104) ∥∇ f ( ζ t ) ∥ 2 (cid:105) gives the claimed bound:
[Equation]
Using the step size γ = √ N/ √ T , we get:
[Equation]
Proof for Asynchronous PFL.
We use the same notation in synchronous PFL for convenience.
In asynchronous PFL, we can use almost the same method of Avdiukhin & Kasiviswanathan (2021a) to prove the convergence of our method.
Our contribution does not lie in the novelty of the method of proof, but rather in pointing out that our approach can be fully accommodated in existing proofs of asynchronous FL and achieve greater speed-up ratios.
Definition 5 (Asynchronous PFL Sequence) Let ω ( g ) t represents the global model of asynchronous
PFL, G ( i ) t represents the uploaded gradient updates computed by the i -th client based on ω ( g ) t , the global model ω ( g ) t can be expressed as:
[Equation]
Proposition 6 (Distance Bound)
For ζ t of virtual sequence and ω ( g ) t , ω t of Algorithm 1, we have:
[Equation]
Proof:
[Equation]
Similarly, E (cid:20)(cid:13)(cid:13)(cid:13) ω ( g ) t − ω t (cid:13)(cid:13)(cid:13) 2 (cid:21) ≤ γ 2 ( δ − 1) 2 G 2max .
Combining these bounds, we have the following.
[Equation]
Theorem 7 Let f max = f (cid:16) ω ( g ) {+
+}
(cid:17) − f ( ω ⋆ ) , where ω ⋆ is the minimizer for f , we have:
[Equation]
Proof: Similar to Theorem 4 B.
[-Theorem 8 Let f max = f (cid:16) ω ( g ) 
 (cid:17) − f ( ω ⋆ )-]If T > N 3 and δ ≤ T 1 / 4 /N 3 / 4 , we get:
[Equation]
C S TUDY ON THE I NFLUENCE OF THE S TALE G LOBAL M ODEL
In our scheme, the key to achieving parallel federated learning is to send the stale global model to the edge devices.
It would be interesting to investigate the influence of the stale global model on the final performance.
In this section, we conduct a convergence analysis through empirical experiments with respect to both accuracy and communication rounds on the CIFAR-10 dataset withedge devices.
The convergence curve is provided in Fig. It can be found that, in the early training stage, the proposed PFL suffers from lower accuracy and stronger oscillations compared with FedAvg; while with the communication round increases, the PFL finally achieves similar performance to FedAvg. This is consistent with the intuition that, when the algorithm converges, there is no significant difference between using the stale global model or using the up-to-date global model. D S PECIAL CASE IN SYNCHRONOUS PFL FOR T g = T l + T c + T w
Here we give a timing diagram for a special case in synchronous PFL when T g =
T l + T c + T w .
As can be seen in Fig.
5, in this setting, the time spent on each round is T g .
[Figure]
[Figure]
[Figure]
[Figure]