A CCELERATING F EDERATED L EARNING C ONVERGENCE VIA O PPORTUNISTIC M OBILE R ELAYING
Anonymous authors
Paper under double-blind review
A BSTRACT
This paper studies asynchronous Federated Learning (FL) subject to clients’ individual arbitrary communication patterns with the parameter server.
We propose
FedMobile, a new asynchronous FL algorithm that exploits the mobility attribute of the mobile FL system to improve the learning performance.
The key idea is to leverage the random client-client communication in a mobile network to create additional indirect communication opportunities with the server via upload and download relaying.
We prove that FedMobile achieves a convergence rate
O ( 1 √ NT ) , where N is the number of clients and T is the number of communication slots, and show that the optimal design involves an interesting trade-off on the best timing of relaying.
Our analysis suggests that with an increased rate of client-client communication opportunities, asynchronous FL converges faster using FedMobile.
Experiment results on a synthetic dataset and two real-world datasets verify our theoretical findings.
I NTRODUCTION
Federated learning (FL) McMahan et al.
(2017) is a distributed machine learning paradigm where a number of clients with decentralized data work collaboratively to learn a common model under the coordination of a centralized server.
The majority of FL algorithms McMahan et al.
Li et al.
; Khaled et al.
; Yang et al.
Yu et al.
(2019) study the synchronous communication setting where clients can synchronize and communicate with the server periodically and simultaneously.
The communication frequency is thus a key design parameter of the FL algorithm under the implicit assumption that the communication channel between the clients and the server is always and universally available.
This assumption, however, hardly holds in many realworld systems where clients have only sporadic communication opportunities with the server, and the communication patterns vary among the clients.
For example, mobile clients (e.g., mobile devices, sensors, vehicles) can communicate with the server (e.g., base stations, sensing hubs, road side units) only when they meet the server (i.e., entering the communication range of the server).
In such mobile systems, FL must be executed asynchronously obeying clients’ individual communication patterns with the server.
The literature on asynchronous FL is much smaller than that on its synchronous counterpart.
Although some insights have been derived (e.g., Avdiukhin & Kasiviswanathan (2021) proves that the convergence rate of asynchronous FL can match that of synchronous FL under the same communication interval), the performance of asynchronous FL is significantly limited by the communication patterns of individual clients as they are arbitrary and not an algorithm parameter.
With only occasional client-server meetings, asynchronous FL can converge slowly or even fail to converge.
This paper studies asynchronous FL in a mobile network setting and demonstrates that the communication patterns of the mobile clients can be “virtually” reshaped to improve the FL convergence performance by exploiting a key attribute of the mobile FL system, namely mobility .
Mobility causes the sporadic client-server meetings but at the same time also creates (the largely neglected) communication opportunities among the clients .
This latter consequence of mobility enables a client to indirectly communicate with the server by using another client as the relay, thereby creating additional communication opportunities (both upload and download) with the server.
In particular, the client’s local model updates can be uploaded to the server sooner if the relay client meets the server before the sending client does.
Likewise, the client can receive a fresher global model from a relay client that more recently met the server.
We present a motivating example, namely FL in Vehicular
Ad Hoc Networks (VANET), in the supplementary material where the considered setting and the assumptions made are justified.
Main Contributions .
Our study starts with a simplified yet practical setting where a client is allowed to use at most one upload relay communication and at most one download relay communication between any of its two consecutive meetings with the server (considering the additional cost due to client-to-client communication).
We design a new asynchronous FL algorithm with opportunistic relaying, called FedMobile , that addresses the following key questions: (1) when to upload (download) via relaying and who should be the relay?
(2) how to relay the local model updates to (the global model from) the server?
We prove the convergence of FedMobile and reveal an interesting trade-off of the best timing of relaying for both the upload and download cases, and that the mobility improves the convergence speed of asynchronous FL.
Furthermore, we extend FedMobile to allow multiple relay communications between two consecutive server meetings.
Extensive experiments on a synthetic dataset and two real-world datasets verify our theoretical findings.
We note that this paper is focused on the theoretical understanding of the convergence speed of FL under opportunistic mobile relays.
Relaying incentives and the security issues associated with relaying are orthogonal to the research topic of this paper.
Nevertheless, in the supplementary material, we also present an extension where clients manipulate data before relaying it to the server.
Such data manipulation can be used to reduce the relaying cost as well as enhance privacy during relaying.
R ELATED W ORK
FedAvg McMahan et al.
(2017) uses local stochastic gradient descent (SGD) to reduce the number of communications between the server and clients.
Convergence has been analyzed for FedAvg Li et al.
(2019); Khaled et al.
; Yang et al.
Yu et al.
and its variants (e.g., momentum variants Wang et al.
Liu et al.
and variance-reducing variants Kone ˇ cn ` y et al.
Karimireddy et al.
in both iid and non-iid data settings.
However, the majority of works on
FL study the synchronous setting and treat the communication frequency as a tunable parameter of the algorithm.
Asynchronous distributed optimization/learning was studied in the past several years.
These works
Mitliagkas et al.
; Hadjis et al.
; Chen et al.
; Zheng et al.
; Dai et al.
(2018) consider a single SGD step by the distributed nodes with iid data distributions, which are not typical settings of FL.
The literature on asynchronous FL is much smaller but different works have vastly different focuses.
For example, some existing works Chen et al.
; Smith et al.
(2017); Nishio & Yonetani (2019) still assume universal communication at all times and the asynchronicity is the result of an algorithm decision rather than a constraint.
Some other works van Dijk et al.
Chai et al.
; Chen et al.
; Li et al.
(2021) use asynchronous model aggregation to address the “straggler” problem encountered in synchronous FL.
The asynchronous setting closest to ours is studied in Avdiukhin & Kasiviswanathan (2021);
Basu et al.
(2019), which considered arbitrary communication patterns.
However, these works only considered the interaction between the server and the clients, neglecting the interaction among clients that may be leveraged to improve the FL performance.
Our work is remotely related to decentralized FL Lalitha et al.
; Zhang et al.
; Xing et al.
(and some hybrid FL works Guo et al.
(2021)) where clients can also communicate among themselves during the training process, typically by using a type of gossip algorithm to exchange local model information.
However, these works assume a fixed topology of clients and the communication among the clients is still synchronous and periodic.
M ODEL AND P RELIMINARIES
We consider a mobile FL system with one server and N mobile clients.
The mobile clients work together to train a machine learning model by solving a distributed optimization problem as follows:
[Equation]
where f i : R d → R is a non-convex loss function for client i and F i is the estimated loss function based on a mini-batch data sample ζ i drawn from client
i ’s own dataset.
Mobility Model .
We consider a discrete time system where time is divided into slots of equal length.
Clients move independently in a network and make contact with the server only at certain time slots.
At each time t and for each client i , let τ last i ( t ) be the last time when client i meets the server by t (including t ), and τ next i ( t ) be the next time when client i will meet the server (excluding t ).
At any t , client i knows τ next i ( t ) but may not know the future meeting times.
The communication pattern is arbitrary and different for different clients but we assume that the time interval between any two consecutive server meetings for any client is bounded by ∆ , i.e., τ next i ( t ) − τ last i ( t ) ≤ ∆ , ∀ t, ∀ i .
Clients can also meet among themselves due to mobility.
When two clients meet, they can communicate with each other via, e.g., device-to-device (D2D) communication protocols Asadi et al.
For the ease of analysis, we assume that at each time t , a client can meet at most one other client (the extension to multiple clients is straightforward).
Let ρ ∈ [0 , 1] be the probability of a client meeting another client at a time slot.
When ρ = 0 , clients do not meet with each other, thus degenerating to the conventional case.
We assume that the client-server meetings remain unchanged regardless of the value of ρ .
Please see the use case in the supplementary material for the justification of the system assumptions.
Asynchronous Federated Learning .
Because FL cannot be executed synchronously in our setting, we consider an asynchronous FL algorithm similar to Avdiukhin & Kasiviswanathan (2021);
Basu et al.
under arbitrary communication patterns.
Note, however, that client meetings due to mobility have not come into the picture yet.
Local Model Update .
For any client i , when it meets the server at t , it downloads the current global model x t .
The client then uses x ti = x t as the initial model to train a new local model using its own local dataset until it meets the server again.
This is done by using a mini-batch SGD method:
[Equation]
where g si = ∇ F i ( x si , ζ si ) is the stochastic gradient on a randomly drawn mini-batch ζ si and η is the learning rate.
Here, we assume that a client performs one step SGD at each time slot to keep the notations simple.
Let m ti ∈ R
d be the cumulative local updates (CLU) of client i at time t since its last meeting with the server, which is updated recursively as follows:
[Equation]
Global Model Update .
At any t , let S t denote the set of clients who meet the server ( S t may be empty).
These clients upload their CLUs to the server who then updates the global model as
[Equation]
The updated global model x t is then downloaded to each client i in S t , and the client starts its local training with a new initial model x ti = x t .
F ED M OBILE
In the vanilla asynchronous FL described in the above section, a client can upload its CLU and download the global model only when it meets the server.
When the meeting intervals are long, however, this information cannot be exchanged in a timely manner, thus hindering the global training process.
To overcome this issue, we take advantage of mobility and propose FedMobile that creates indirect client-server communication opportunities, thereby improving the FL convergence speed.
In a nutshell, a client i can use another client j as a relay to upload its CLU to the server (or download the global model from the server) if client j ’s next (or last) server meeting is earlier (or later) than client i ’s.
Thus, the CLU can be passed to the server sooner (or the client can train based on a fresher global model).
We call such client j an upload (or download) relay for client i .
For now, we assume that a client can only use at most one upload relay and at most one download relay between any two consecutive server meetings, considering the extra cost incurred due to the client-client communication.
However, FedMobile can be easily extended (we discuss it in
The General Case section).
Next, we describe separately how FedMobile handles uploading and downloading, which are essentially the dual cases to each other.
[Figure]
4. U PLOADING CLU VIA R ELAYING
When to upload via relaying and who should be the relay?
Since a client can use the upload relay at most once between its two consecutive server meetings, the timing of uploading via relaying is crucial.
If client i uploads too early (i.e., close to τ last i ), then the CLU has little new information since the client has run just a few mini-batch SGD steps.
If client i uploads too late (i.e., close to τ next i ), then the CLU will be uploaded to the server late even if a relay is used.
To make this balance,
FedMobile introduces a notion called upload search interval defined by two parameters θ and Θ , where 0 ≤ θ ≤ Θ ≤ ∆ (See Fig. 1).
Client i will upload its CLU via a relay only during the interval [ τ last i + θ, τ last i + Θ] .
In addition, not every other client that client i meets during the search interval is qualified.
We say that client j is a semi-qualified upload relay if τ next j ≤ τ last i + Θ , i.e., client j is able to relay client i ’s CLU to the server before the end of the search interval.
We further say that client j is a qualified upload relay if in addition τ next j < τ next i , i.e., client j can indeed deliver the
CLU earlier than client i ’s own server meeting 1 .
FedMobile picks the first qualified upload relay during the search interval.
Note that it is possible that no qualified upload relay is met, in which case no uploading via relaying is performed.
How to relay the local updates to the server?
FedMobile ensures that the same piece of information is delivered to the server exactly once by devising a simple yet storage-efficient mechanism.
Upon a CLU exchange event at time t between a sender client i and a relay client j :
RESET (by sender): After sending its current CLU m ti , sender client i resets its CLU to m ti :=
COMBINE (by relay): After receiving m ti from client i , client j combines m tj and m ti to produce a new m tj , i.e., m tj := m tj + m ti .
In this way, FedMobile essentially offloads the uploading task of m ti from client i to client j , who, by our design, has a sooner server meeting time than client i .
Remark : (1) The subsequent local training is not affected at all for both the sender and the relay.
That is, for client i (client j ), g si ( g sj ) will be the same for all s up to τ next i ( t ) ( τ next j ( t ) )
regardless of the CLU exchange.
The relay client’s storage requirement remains the same because COMBINE produces a CLU of the same size as before.
In fact, the relay can combine multiple sender clients’
CLUs between two consecutive server meeting times (these exchanges occur at different time slots), yet the storage requirement is the same.
(3) A client can be both a sender and a relay between two consecutive server meeting times.
If a client i has already received the CLU(s) from some other client(s) before it sends m ti to client j , then m ti actually also contains CLU(s) of other clients.
Remark :
For higher communication efficiency and/or better privacy protection, the clients may send an altered CLU to the relay for uploading.
We discuss this extension and provide its convergence analysis in the supplementary material.
4. D OWNLOADING G LOBAL M ODEL VIA R ELAYING
When to download via relaying and who should be the relay?
Similar to the uploading CLU case, when to download a global model via relaying also involves a trade-off.
FedMobile introduces a download search interval to make this balance, which is defined by two parameters ω and Ω , where 0 ≤ ω ≤ Ω ≤ ∆ (see Fig. 1).
Given client i ’s next server meeting time τ next i , client i will only download a new global model via relaying during the search interval [ τ next i − Ω , τ next i − ω ] .
Similarly, we say that client j is a semi-qualified download relay if τ last j ≥ τ next i − Ω , i.e., client j ’s global model is less than Ω time slots older than client
i ’s next global model directly from the server.
We further say that client j is a qualified download relay if in addition τ last j > τ last i , i.e., client j ’s global model is indeed fresher than client i ’s current global model that it received directly from the server.
FedMobile picks the first qualified download relay during the search interval.
Again, it is possible that no qualified download relay is met, in which case no downloading via relaying occurs.
How to relay the global model to the client?
To be able to relay a global model to other clients, every client keeps a copy of the most recent global model that it received (from either the server or another client).
We denote this copy for client j by x ψ j ( t ) , where ψ j ( t ) is the time version (or timestamp) of the global model.
Upon a global model exchange at time t between a receiver client i and a relay client j :
REPLACE (by receiver): After receiving x ψ j ( t ) from client j , client i replaces its local model with x ψ i ( t ) , i.e., x ti := x
ψ j ( t ) , and resumes the local training steps.
Remark : (1) Client i also replaces its global model copy with x ψ j ( t ) since it is a fresher version by our design.
Thus, ψ i ( t ) is updated to ψ j ( t ) .
(2) An alternative to the current downloading scheme is that relay client j simply sends its current local model x tj to client i , who then replaces its current local model x ti with x tj , i.e., x ti :=
x tj .
In this way, the clients do not have to keep a copy of the most recent global model, thereby reducing the stored data.
The convergence analysis is not affected by this change and the same convergence bound can be proved (see the proof of Theorem 1).
C ONVERGENCE A NALYSIS
Our convergence analysis will utilize the following standard assumptions.
Assumption 1 (Lipschitz Smoothness) .
There exists a constant L > 0 such that ∥∇ f i ( x ) − ∇ f i ( y ) ∥
≤ L ∥ x − y ∥ , ∀ x, y ∈ R d and ∀ i = 1 , ..., N .
Assumption 2 (Unbiased Local Gradient Estimate) .
The local gradient estimate is unbiased, i.e., E ζ F i ( x, ζ ) =
∇ f i ( x ) , ∀ x and ∀ i
= 1 , · · · , N .
Assumption 3 (Bounded Variance) .
There exists a constant σ > 0 such that E [ ∥∇ F i ( x, ζ i ) − ∇ f i ( x ) ∥ 2 ] ≤ σ 2 , ∀ x ∈ R d and ∀ i = 1 , ..., N .
Assumption 4 (Bounded Second Moment) .
There exists a constant G > 0 such that E [ ∥∇ F i ( x, ζ i ) ∥ 2 ] ≤ G 2 , ∀ x ∈ R d and ∀ i = 1 , ..., N .
The real sequence of the global model is calculated as
[Equation]
where we define ϕ i ( t ) to be the time slot up to when all corresponding gradients of client i have been received at time t .
In the vanilla asynchronous FL case, ϕ i ( t ) is simply τ last i ( t ) − 1 .
In FedMobile, typically ϕ i ( t ) >
τ last i ( t ) − 1 because more information can be uploaded earlier than t due to relaying.
We also define the virtual sequence of the global model, which is achieved in the imaginary ideal case where all local gradients are uploaded to the server instantly at every slot,
[Equation]
First, we bound the difference ( t − 1) − ϕ
i ( t ) , which characterizes how much CLU information of client i is missing compared to the virtual sequence.
Lemma 1.
Assuming at least one semi-qualified upload relay client exists in every upload search interval, then we have ( t − 1) −
[Equation]
, ..., N, ∀ t .
Next, we bound the difference t − ψ i ( t ) , which characterizes the version difference between the current global model and client i ’s copy of the global model.
Lemma 2.
Assuming at least one semi-qualified download relay exists in every download search interval, we have t − ψ i ( t ) ≤ max { ∆ − ω, Ω } ≜ D ( ω, Ω; ∆) , ∀ i = 1 , ..., N, ∀ t .
The following lemma then bounds the model differences in the real sequence and the virtual sequence.
Lemma 3.
The difference of the real global model and the virtual global model is bounded as follows
[Equation]
For each client i , the difference of its local model and the virtual global model is bounded as follows
[Equation]
Now, we are ready to bound the convergence of the real model sequence achieved in FedMobile.
Theorem 1.
Assuming at least one semi-qualified upload (download) relay client exists in every upload (download) search interval, by setting η ≤ 1 /L , after T time slots, we have
[Equation]
Remark :
The convergence bound established in Theorem 1 contains three parts.
The first part diminishes as T approaches infinity.
Both the second and third terms are constants for a constant η , with the second term depending on our algorithm parameters δ , Θ , ω and Ω .
For a given final step
T , one can set η = √ N L √ T so that the bound becomes
[Equation]
Furthermore, if T ≥ N 3 , the above bound recovers the same O ( 1 √ NT ) convergence rate of the classic synchronous FL Yu et al.
Next, we discuss the above convergence result in more detail and investigate how the mobility affects the convergence.
To this end, we consider for now a fixed client-client meeting rate ρ , and investigate how the convergence result depends on our algorithm parameters.
Bound Optimization.
Since the convergence bound can be improved by lowering C ( θ, Θ; ∆) and
D ( ω, Ω; ∆) , we first investigate them as functions of the algorithm parameters.
Proposition 1.
(1) C ( θ, Θ; ∆) is non-decreasing in Θ and non-increasing in θ .
Moreover, ∀ θ, Θ ,
C ( θ, Θ; ∆) ≥ ∆2 , and the lower bound is attained by choosing θ =
Θ = ∆2 .
(2) D ( ω, Ω; ∆) is non-decreasing in Ω and non-increasing in ω .
Moreover, ∀ ω, Ω , D ( ω, Ω; ∆) ≥ ∆2 , and the lower bound is attained by choosing ω
= Ω = ∆2 .
Proposition 1 implies that one should use shorter search intervals, namely Θ − θ and Ω − ω , to lower
C ( θ, Θ; ∆) and D ( ω, Ω; ∆) .
However, the best C and D are no smaller than ∆2 , which are achieved by choosing θ =
Θ = ∆2 and ω
= Ω = ∆2 .
In other words, a short upload search interval around the time τ last i + ∆2 and a short download search interval around the time τ next i − ∆2 improve the FL convergence.
This suggests that the best timing for uploading is at exactly τ last i + ∆2 and the best timing for downloading is at exactly τ next i − ∆2 , which are neither too early or too late in both cases.
Probability of Meeting a Semi-Qualified Relay.
The convergence bound in Theorem 1, however, is obtained under the assumption that a client can meet at least one semi-qualified upload (download) relay client in each upload (download) search interval, which may not always hold.
How easily a client can find a semi-qualified relay client depends on the client-client meeting rate.
In the VANET example, this rate depends on the D2D communication range.
Let Q u ( θ, Θ) (and Q d ( ω, Ω) ) be the probability that at least one semi-qualified uploading (download) relay is met in an uploading (download) search interval with length Θ − θ (and Ω − ω ).
They are characterized as
Proposition 2.
Assuming sufficiently many clients in the system and that clients meet each other uniformly randomly.
Let P int ( · ) be the distribution of server meeting intervals.
Then Q u ( θ, Θ) = 1 − (cid:81)
Θ − θ t =
(1 − ρq u
(Θ − θ − t )) and Q d ( ω, Ω) = 1 − (cid:81) Ω − ω t =
(1 − ρq d ( t )) , where q u ( · ) and q d ( · ) are distributions computed based on P int ( · ) .
Proposition 2 states an intuitive result that one should use a larger search interval to increase the probability of meeting a semi-qualified relay.
This, however, is not desirable for lowering the convergence bound according to Proposition 1.
This is exactly where mobility can help improving the
FL convergence: by increasing the client-client meeting rate ρ , a shorter search interval Θ − θ (or Ω − ω ) can be used to achieve the same Q u (or Q d ), but a smaller C (or D ) is obtained.
In fact, by relaxing the constraint that a client can only meet one other client at a time slot, with sufficiently many clients in the system, both Q u and Q d can approach 1 even if the search interval is just one slot.
T HE G ENERAL C ASE
FedMobile can be easily extended to allow multiple upload (download) relay communications between any two consecutive server meetings.
We will still have the upload (download) search interval, and FedMobile will simply pick the first K qualified upload (download) relays to perform the upload (download) relay communications.
The exact mechanisms of how uploading (downloading) is performed will be slightly changed to handle out-of-order information and avoid redundant information.
Our convergence analysis still holds correctly for this generalized case, although the bound may become looser compared to the actual performance.
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
Figure 2 illustrates the key ideas behind FedMobile.
Figure 2(a)
plots hypothetical sequences of CLUs received by the server from a representative client between two consecutive server meetings.
The virtual sequence is the ideal case where each local stochastic gradient g ti is uploaded to the server instantly after it is computed.
Therefore, the received CLU curve is a steady staircase.
In the vanilla asynchronous FL, the client uploads the CLU only when it meets the server.
Therefore there is a big jump at the next server meeting time but it is all 0 before.
FedMobile with at most one upload relaying allows some local stochastic gradient information to be received by the server earlier.
The generalized FedMobile allows more CLUs to be relayed and received by the server at earlier time slots.
One can imagine that when ρ is large enough, it becomes much easier for the client to find qualified relays that can quickly upload CLUs to the server at every time slot.
Therefore, the received CLU curve approaches that in the ideal case.
Similarly, Figure 2(b) plots the hypothetical sequence of the global models received by the client.
E XPERIMENTS
Datasets and Models .
We conduct experiments on a synthetic dataset and two real-world datasets, i.e., FMNIST Xiao et al.
(2017) and CIFAR10 Krizhevsky et al.
Synthetic dataset : The synthetic dataset is generated on a least-squares linear regression problem.
Each data sample has a 200 dimensional feature vector and its real-valued label is calculated as the vector product of the feature and an underlying linear weight plus a 0-mean Gaussian noise.
The FL system has 50 clients with each client having 40 data samples.
FMNIST : The FL system has 50 clients with each client having 400 data samples.
The data samples are randomly allocated to the clients according to a widely used method to generate non-i.i.d.
datasets for FL Chen & Chao (2020).
We use LeNet
LeCun et al.
(1998) as the backbone model.
CIFAR10 :
The FL system has 50 clients with each client having 600 data samples.
The data allocation method is the same as that used for FMNIST.
We use ResNet-
He et al.
as the backbone model.
Benchmarks .
The following benchmarks are considered in our experiments.
(1) ASYNC :
This is the state-of-the-art asynchronous FL method proposed in Avdiukhin & Kasiviswanathan (2021) to handle arbitrary communication patterns.
(2) Virtual-U : This method assumes that each client can upload its local updates immediately to the server at every slot via an imaginary channel but can only download the global model at their actual communication slots.
(3) Virtual-D : This method assumes that each client can download the global model from the server at every slot via an imaginary channel but can only upload their CLUs at their actual communication slots.
We also consider several variations of FedMobile.
(1) FedMobile : This is the proposed algorithm combining both upload and download relay communications.
(2) FedMobile-U :
This is the proposed algorithm with only upload relaying.
(3) FedMobile-D : This is the proposed algorithm with only download relaying.
Communication Patterns .
We simulate two asynchronous communication patterns with the server.
Fixed-Interval .
Client i ∈ { 1 , ..., 50 } communicates with the server at slots i + 50 n, ∀ n = 0 , 1 , 2 , ... .
Random-Interval .
Client i ∈ { 1 , ..., 50 } has the first communication with the server at slot i , and then communicates with the server with a random interval between 30 and 50 slots.
For communications among the clients, clients randomly meet with each other.
Details can be found in the supplementary material.
[Figure]
Results on Synthetic Data : Fig.
3 reports the results (averaged over three repeats) on the synthetic data with a fixed-interval communication pattern.
The detailed configurations can be found in the supplementary material.
Fig.
3(a) compares FedMobile and its variations with ASYNC in terms of the test loss.
As can been seen, incorporating either upload or download relaying into asynchronous
FL improves the FL performance, and a further improvement can be achieved when uploading and downloading are combined.
Fig.
3(b)/(c) illustrates the impact of upload/download search interval on the FL convergence.
In both cases, the best search timing is around the middle point of the two consecutive server meeting times, confirming our theoretical analysis in Theorem 1 and Proposition 1.
Fig. 3(d) shows the impact of ρ on the FL convergence.
As predicted by our analysis, a higher ρ in the system improves FL convergence since more timely relay communication opportunities are created.
In Figs.
3(e)(f)
, we allow clients to use multiple relays to create more communication opportunities with the server whenever possible.
The results show that further improvement can indeed be achieved.
We also conduct experiments on the ideal relaying scenarios, namely Virtual-U and Virtual-D , to illustrate what can be achieved in the ideal case.
The results verify our hypothesis that more communication opportunities with the server benefits convergence.
It is also interesting to note that virtual uploading/downloading has a more significant impact on the early/late FL slots, suggesting that an adaptive design may better balance the FL performance and the resource cost.
Results on Real-World Datasets : We now report the results (averaged over three repeats) on realworld datasets.
Fig.
4 shows that FedMobile achieves a better convergence speed than ASYNC on
FMNIST and CIFAR10 under both the Fixed-Interval and Random-Interval settings.
For a specific example, to achieve 70% accuracy on FMNIST under the fixed-interval setting, ASYNC needs about 178 slots while FedMobile only needs 115 slots.
The required time slot decreases by 35.4%, which demonstrates the significant improvement of FedMobile .
Fig.
5 and Fig.
6 further show the experiment results on FMNIST under the fixed-interval setting and the random-interval setting, respectively.
The results largely resemble those in the synthetic data case.
In particular, Fig. 5(a) and Fig.
6(a) demonstrate that while upload and download relay communications individually can improve the FL convergence performance, combining them results in an additional benefit.
Fig.(b)(c) show that the best timing of upload and download relaying should be neither too early nor too late.
However, the difference between the three download search intervals in Fig. 5(c) is less distinguishable.
Fig. 5(d) and Fig.
6(b) show that using multiple upload relay communications further improves the FL convergence performance.
However, we observed in our experiments that this is not the case with using multiple download relay communications.
This is likely due to the complexity of real-world datasets, which causes the assumptions for our theoretical analysis to be violated.
This represents a limitation of our current analysis and requires further investigation.
We defer related experiment results to the supplemental material.
Finally, Fig. 6(c) confirms that a higher client-client meeting rate (e.g., larger D2D communication range of mobile devices) of the system improves the convergence of asynchronous FL with the help of FedMobile .
[Figure]
[Figure] [Figure]
C ONCLUSION
This paper offers a new perspective on how to design asynchronous FL algorithms in practical systems where client-server communication is not always available.
We advocate the role of client mobility, and hence the resulting random client-client communication opportunities, in the timely information exchange for the model training in asynchronous FL.
The experiment results are mostly consistent with our theoretical results and demonstrate that mobility indeed improves the convergence of asynchronous FL.
We believe that FedMobile can promote distributed machine learning, especially FL, in many real-world systems such as mobile gaming, mobile sensing and smart vehicular networks.
B E XPERIMENT D ETAILS
We implement FL simulation on the Pytorch framework and perform the model training on one
Geforce RTX 3080 GPU.
The code files of data generation and the proposed method are attached in the supplemental material.
All experiment results are averaged over 3 repeats.
The real datasets (FMNIST and CIFAR10) which we utilized in experiment section are both public.
Both of them are licensed under MIT License.
To simulate communications among the clients, at each time slot, we uniformly sample ρN clients over total N clients and randomly construct ρN 2 client pairs.
Any two clients in the same pair are simulated to communicate.
Synthetic dataset : The synthetic dataset is generated on a least-squares linear regression problem.
Each data sample has a 200 dimensional feature vector and its real-valued target is calculated as the vector product of the feature and an underlying linear weight plus a 0-mean Gaussian noise.
The FL system has 50 clients with each client having 40 data samples.
The experiment configuration details are shown in Table 1.
[Table]
θ =
[Table]
, Θ
[Table]
ω = 5 .
Ω =
FMNIST(Fashion-MNIST) :
The FL system has 50 clients with each client having 400 data samples.
We utilize the Dirichlet function ( α = 0 . 3 ) which is typically utilized to simulate the level of non-IID in FL.
We use LeNet as the backbone model.
The training configuration details are summarized in Table 2.
[Table]
θ = 10 , Θ =
[Table]
ω = 5 .
Ω =
CIFAR10 :
The FL system has 50 clients with each client having 600 data samples.
The data allocation method is the same as that used for FMNIST.
We use ResNet-9 as the backbone model.
The architecture of ResNet-9 is given in Table 4.The training configuration details are summarized in Table 3.
[Table]
[Table]
C A DDITIONAL E XPERIMENTS C. V IRTUAL -U AND V IRTUAL -D
Virtual-U and Virtual-D work as the ideal cases for upload relaying and download relaying and hence we conduct experiments to investigate their performance on the real-world dataset.
Fig.reports the performance of Virtual-U and Virtual-D on FMNIST with both fixed and random interval communication patterns.
Similar to Fig. 3(g) for the synthetic data, Figs. 8 (a)(b) show that
Virtual-U can greatly improve the convergence performance.
However, different from Fig. 3(h) for the synthetic data, Figs.
8 (c)(d) show that Virtual-D fail to converge on the real-world dataset.
We conjecture that this is likely due to the complexity of the real-world dataset where some of the assumptions needed for our theoretical analysis do not strictly hold.
However, as Figs.
5 andshow, using one-time download relaying still has benefits on the convergence even in the real-world dataset.
C. A LTERNATIVE D OWNLOAD R ELAYING S CHEME
In Section 4.2, we discussed how to relay the global model to the client.
In particular, each client has to keep a copy of the most recent global model that it received (from either the server or another client).
Therefore, when needed, the client can relay this global model to another client.
We mentioned an alternative download relaying scheme where clients do not have to keep this copy.
Instead, the client simply performs the local training on the received global model (from either the server or another client).
When needed, the client relays this local model to another client.
This way saves the clients storage space.
We also proved in Theorem 1 that this alternative download relaying scheme achieves the same convergence bound as the original one.
Here, we illustrate its performance via
[Figure] [Figure]
experiments in Fig.
As we can see, the alternative download relaying scheme still outperforms
ASYNC .
C. D OWNLOAD T IME E FFECT
In FMNIST experiments, the difference between three download search interval is not obvious.
To better show the impact of download relaying timing, we consider an ideal scenario where each client can virtually download the global model 1/30/49 time slot(s) after its last communication with server.
The result in Fig.
10 clearly shows that the best timing of download relaying should be neither too early nor too late.
C. C LIENT -C LIENT M EETING R ATE
Fig.
6(d) shows the effect of client-client meeting rate under the random-interval communication pattern.
Here, we provide the experiment results under the fixed-interval pattern.
Fig.
11 shows that the higher client-client meeting rate improves the convergence performance.
C. D IFFERENT I NTERVAL L ENGTH AND N UMBER OF C LIENTS
To further validate the proposed method FedMobile, we run additional experiments with different interval length/number of clients.
In Fig.
12, holding other training settings unchanged, we extend the fixed-interval to 70 time slots and update FedMobile’s parameters ( θ = 20 , Θ = 60 , ω = 10 , Ω = 30 ).
In Fig.
13, we increase the number of clients to 100.
Both results validate that our proposed method can outperform the baseline method.
C. E XPONENTIALLY D ISTRIBUTED S ERVER M EETING I NTERVALS
We consider the client communication pattern where the server meeting intervals are exponentially distributed with mean being 30 time slots.
In addition, we assume that the distribution is truncated
(maximum is 80 time slots) so that the intervals between consecutive server meetings are bounded.
We consider both the case where the next server meeting time is known to the clients and the case where the next server meeting time is unknown.
In the latter case, the expected value of the exponential distribution is used to estimate their next server meeting time.
Fig.
14 shows that FedMobile can still outperform the baseline under exponentially distributed server meeting intervals.
D P ROOFS D. P ROOF OF L EMMA
It is obvious that if the server meeting time interval τ next i ( t ) − τ last i ( t ) ≤ Θ , then ( t − 1) − ϕ i ( t ) =
t − τ last i ( t ) ≤ Θ already holds.
Otherwise, for all t ≤ τ last i ( t ) + Θ , then ( t − 1) − ϕ i ( t ) =
t − τ last i ( t ) ≤ Θ also holds.
Thus, we only need to consider the case τ next i ( t ) − τ last i ( t ) >
Θ and for time slot t > τ last i ( t ) + Θ .
In this case, a semi-qualified relay client is also a qualified relay client because
[Equation]
By the assumption that at least one semi-qualified relay exists in the search interval, at least one qualified relay must exist.
This further implies that the qualified relay client is able to upload a CLU before t .
Because this CLU contains gradients of client i for at least θ steps since τ last i ( t ) , we have ϕ
i ( t ) ≥ τ last i ( t )
+ θ − 1 .
Therefore,
[Equation]
To summarize the above cases, ( t − 1) − ϕ i ( t ) ≤ max { ∆ − θ, Θ } is established.
D. P ROOF OF L EMMA
Let t ′
be the meeting time between receiver client i and relay client j .
Clearly, τ next i
[Equation]
t ′ ≤ τ next i − ω by the definition of the search interval.
For t ≤ t ′ , client i has not met client j yet, so ψ i ( t ) =
τ last i ( t ) .
Therefore, t − ψ i ( t ) =
t − τ last i ( t ) ≤ t ′ − τ last i ( t ) ≤ τ next i ( t ) − ω − τ last i ( t ) ≤ ∆ − ω (13)
For t > t ′ , client i has met client j , so ψ i ( t ) ≥ τ last j ( t ) .
Therefore, t − ψ i ( t ) ≤ t − τ last j ( t ) ≤ t − ( τ next i ( t ) − Ω) ≤ Ω (14)
To sum up, t − ψ i ( t ) ≤ max { ∆ − ω, Ω }
D. P ROOF OF L EMMA
Consider any time t , by the definition of virtual and real sequences, we have
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
( t )
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
t
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
N
[Equation]
(cid:88)
i
[Equation]
(( t − 1)
[Equation]
ϕ
[Equation]
( t ))
[Equation]
G
[Equation]
(16) ≤ C 2 ( θ, Θ; ∆)
η 2 G(17) where the first ≤ is due to the Cauchy-Schwarz inequality, the second ≤ is due to both the CauchySchwarz inequality and Assumption 4, and the last ≤ is due to Lemma 1.
Consider client i at any time t with the global model version ψ i ( t ) .
Therefore client i is doing the local training steps using the global model x ψ i ( t ) as the initial model.
Thus, we use x ψ i ( t ) as the anchor model to investigate the difference between v t and x ti .
[Equation]
[Equation]
ψ i ( t ) − x ψ i ( t ) (cid:13)(cid:13)(cid:13) 2 (cid:21)
[Equation]
E (cid:20)(cid:13)(cid:13)(cid:13) x
t ′ i − x ti (cid:13)(cid:13)(cid:13) 2 (cid:21) (20) ≤ 

 E    (cid:13)(cid:13)(cid:13) (cid:13) (cid:13)(cid:13)
N
N (cid:88) i =
t − 1 (cid:88) s
= ψ
i ( t ) ηg
[Equation]
i (cid:13)(cid:13)(cid:13) (cid:13) (cid:13)(cid:13) 2  
[Equation]
E (cid:20)(cid:13) (cid:13)(cid:13) v ψ i ( t ) − x ψ i ( t ) (cid:13) (cid:13)(cid:13) 2 (cid:21)
[Equation]
E (cid:20)(cid:13) (cid:13)(cid:13) x
ψ i ( t ) i − x t i (cid:13) (cid:13)(cid:13)(cid:21)   (21) ≤ 3 (cid:16) D
[Equation]
( ω
[Equation]
Ω
[Equation]
∆)
η
[Equation]
G
[Equation]
+ C
[Equation]
( θ
[Equation]
Θ
[Equation]
∆)
[Equation]
∆)
η
[Equation]
(cid:17) (22) =3(2 D 2 ( ω, Ω; ∆) + C 2 ( θ, Θ; ∆))
η 2 G(23)
Note that t − t ′ ≤ t − ψ i ( t ) always holds.
If the client i has not met its download relay, t ′ =
ψ i ( t ) .
Otherwise (the client i has met its download relay), t ′ is the meeting time and therefore, t ′ >
ψ i ( t ) .
Note that this bound holds even if a relay client j simply sends its local model x sj to the client i at some time t ≤ ψ ( t ) ≤ s ≤ t .
D. P ROOF OF T HEOREM
We first analyze the convergence of the virtual global model sequence.
By the smoothness of f ( x ) (Assumption 1), we have
[Equation]
The second term on the right-hand side of equation 24 can be expressed as follows,
[Equation]
The third term on the right-hand side of equation 24 can be bounded as follows,
[Equation]
Substituting these into equation 24 yields
[Equation]
Dividing both sides by η 2 and rearranging the terms
[Equation]
we have
[Equation]
Taking the sum over t = 0 , 1 , ..., T − 1 and dividing both sides by T , we have
[Equation]
where f ∗ =
min x f ( x ) .
Now, for the real sequence, we have
[Equation] [Equation]
Substituting the bound on the virtual sequence into the above equation
[Equation]
we have,
[Equation]
This completes the proof.
D. P ROOF OF P ROPOSITION
Let π u ( s ) be the fraction of clients whose next server meeting time is in s time slots.
For sufficiently many clients, the distribution π u =
[ π u (0) π u (1) ...
π u (∆)] is time-invariant and can be calculated by solving the stationary distribution of a Markov chain that describes the client state in terms of the remaining server meeting time.
Specifically, π u is the solution to
[Equation] [Equation]
Furthermore, we let q u ( t ) =
(cid:80) ts =
π u ( s ) be the cumulative distribution function.
Suppose in the ( t + 1) -th slot in the upload search interval, a client is met.
The probability that this client is a semiqualified upload relay is q u (Θ − θ − t ) .
Therefore, the probability that no semi-qualified upload client is met in the ( t + 1) -th slot is 1 − ρq u
(Θ − θ − t ) .
The probability that no semi-qualified upload relay client is met during the entire upload search interval is thus (cid:81)
Θ − θ t =
(1 − ρq (Θ − θ − t )) .
Similarly, let π d ( s ) be the fraction of clients whose last server meeting time was s time slots ago.
The invariant distribution can be solved according to another Markov chain as π d     ...
P ′ int (1) 1 − P
′ int (1) · · ·
P ′ int (2)  · · · ...
P ′ int (∆) · · ·  1 − P
′ int (∆) 
 =
[Equation]
d (49) where P ′ int ( t ) =
P int ( t ) (cid:80) ∆ s = t P int ( s ) .
Let q d ( t ) =
(cid:80) ts =
π d ( s ) be the cumulative distribution function.
Suppose in the t -th slot in the download search interval, a client is met.
The probability that this client is a semi-qualified download relay is q d ( t ) .
Therefore, the probability that no semi-qualified download client is met in the t -th slot is 1 − ρq d ( t ) .
The probability that no semi-qualified download relay client is met during the entire download search interval is thus (cid:81) Ω − ω t =
(1 − ρq d ( t )) .
Finally, the probabilities of meeting at least one semi-qualified relay are obtained.
E R ELAYING M ANIPULATED CLU
In this section, we present an extension of FedMobile where clients upload manipulated CLUs via relaying.
The manipulation operations can be either quantization/compression (in order to reduce the relayed data size) or perturbation (in order to add privacy protection).
To avoid confusions, we let n ti denote the cumulative gradient update of client i ’s own, which is not combined with any CLUs received from other clients.
We call n ti the private-CLU of client i .
In general, there is a difference between the CLUs before and after manipulation.
We denote this difference by ϵ t i .
In the case of quantization, a quantized private-CLU is used for relaying.
Therefore, the difference is ϵ ti =
Q ( n ti ) − n ti where Q ( · ) is the quantizer operator.
In the case of perturbation, a noise ϵ ti is directly added to n ti to create a noisy CLU n ti + ϵ ti for relaying.
To facilitate presentation, we call ϵ ti noise in both cases.
When to upload via relaying and who should be the relay?
This is the same as the original
FedMobile strategy.
How to relay the local updates to the server?
Upon a CLU exchange event at time t between a sender client i and a relay client j :
RESET (by sender): Before sending the CLU to the relay client j , the sender client first records the noise ϵ ti .
Then the manipulated CLU, i.e., ˜ m ti = m ti + ϵ ti , is sent to the relay.
The sender then resets its CLU to m ti := − ϵ ti and its private-CLU to n ti := 0 .
COMBINE (by relay): After receiving ˜ m ti from client i , client j combines m tj and (cid:101) m ti to produce a new m tj , i.e., m tj :
= m tj + (cid:101) m ti .
In this way, the server could get the client i ’s manipulated CLU sooner.
Then the manipulated CLU will later be corrected when the client i meets the server.
E. C ONVERGENCE A NALYSIS
In this subsection, we prove the convergence of FedMobile with manipulated CLU uploading.
We first state an additional assumption on the noise term ϵ ti .
Assumption 5 (Bounded Error) .
The noise term ϵ i is bounded, i.e., E [ ∥ ϵ ti ∥ 2 ] ≤ q E [ ∥ n ti ∥ 2 ] , ∀ i = 1 , ..., N , ∀ t for some positive real constant q .
The corrupted real sequence of the global model can be written as
[Equation]
where we define U t as the set of clients for whom only corrupted CLUs have been received by the server via the relay, and | U t | is the size of U t .
The real sequence x t is the imaginary real sequence where the noise is not added.
Lemma 4.
The difference of the corrupted real global model and the virtual global model is bounded as follows
[Equation]
For each client i , the difference of its local model and the virtual global model is bounded as follows
[Equation]
Proof : Consider any time t
[Equation]
let t
i be the meeting time of client i and its relay, by the definition of corrupted real sequence, we have
[Equation]
Similar to the proof of Lemma 3, consider client i at any time t with the global model version ψ i ( t ) .
Therefore client i is doing the local training steps using the global model x ψ i ( t ) as the initial model.
Thus
[Equation]
we use x ψ i ( t ) as the anchor model to investigate the difference between v t and x ti .
[Equation]
t
[Equation]
t )
[Equation]
(cid:20)(cid:13)(cid:13)(cid:13)
[Equation]
(cid:13)(cid:13)(cid:13)
[Equation]
(cid:21) (66)
[Equation]

[Equation]
(cid:20)(cid:13)(cid:13)(cid:13)
[Equation]
[Equation] [Equation]
∆)
[Equation]
Theorem 2.
With manipulated CLU uploading, assuming at least one semi-qualified upload (download) relay client exists in every upload (download) search interval, by setting η ≤ 1 /L , after T time slots, we have
[Equation]
Proof : The proof of Theorem 2 follows the proof of Theorem 1 by replacing Lemma 3 with Lemma 4.
E. E XPERIMENT R ESULTS
We test two types of manipulation.
In the first type, we directly add Gaussian noises to the relayed
CLU.
Fig.
15 shows the convergence curves under different amount of noises.
(e.g. Gaussian Noise
N 1 (0 , 0 . 01) and Gaussian Noise N 2 (0 , 0 . 001) ).
In the second type, we utilize the low precision quantizer in Alistarh et al.
(2017) to quantize the CLU before relaying.
Here the quantization level is defined as s .
Fig.
16 shows the convergence curves under different quantization levels.
In both cases, FedMobile still outperforms the baseline method provided that the added noise is small or the adopted quantization level is low.