S WIFT : R APID D ECENTRALIZED F EDERATED L EARNING VIA W AIT -F REE M ODEL C OMMUNICATION
A BSTRACT
The decentralized Federated Learning (FL) setting avoids the role of a potentially unreliable or untrustworthy central host by utilizing groups of clients to collaboratively train a model via localized training and model/gradient sharing.
Most existing decentralized FL algorithms require synchronization of client models where the speed of synchronization depends upon the slowest client.
In this work, we propose S WIFT : a novel wait-free decentralized FL algorithm that allows clients to conduct training at their own speed.
Theoretically, we prove that
S WIFT matches the gold-standard iteration convergence rate O (1 / √ T ) of parallel stochastic gradient descent for convex and non-convex smooth optimization (total iterations T ).
Furthermore, we provide theoretical results for IID and non-IID settings without any bounded-delay assumption for slow clients which is required by other asynchronous decentralized FL algorithms.
Although S WIFT achieves the same iteration convergence rate with respect to T as other state-of-the-art (SOTA) parallel stochastic algorithms, it converges faster with respect to run-time due to its wait-free structure.
Our experimental results demonstrate that S WIFT ’s run-time is reduced due to a large reduction in communication time per epoch, which falls by an order of magnitude compared to synchronous counterparts.
Furthermore, S WIFT produces loss levels for image classification, over IID and non-IID data settings, upwards of 50% faster than existing SOTA algorithms.
Code for S WIFT can be found on GitHub at https://github.com/umd-huang-lab/SWIFT .
I NTRODUCTION
Federated Learning (FL) is an increasingly popular setting to train powerful deep neural networks with data derived from an assortment of clients.
Recent research (Lian et al., 2017; Li et al., 2019;
Wang & Joshi, 2018) has focused on constructing decentralized FL algorithms that overcome speed and scalability issues found within classical centralized FL (McMahan et al., 2017; Savazzi et al., 2020).
While decentralized algorithms have eliminated a major bottleneck in the distributed setting, the central server, their scalability potential is still largely untapped.
Many are plagued by high communication time per round (Wang et al., 2019).
Shortening the communication time per round allows more clients to connect and then communicate with one another, thereby increasing scalability.
Due to the synchronous nature of current decentralized FL algorithms, communication time per round, and consequently run-time, is amplified by parallelization delays.
These delays are caused by the slowest client in the network.
To circumvent these issues, asynchronous decentralized FL algorithms have been proposed (Lian et al., 2018; Luo et al., 2020; Liu et al., 2022; Nadiradze et al., 2021).
However, these algorithms still suffer from high communication time per round.
Furthermore, their communication protocols either do not propagate models well throughout the network (via gossip algorithms) or require partial synchronization.
Finally, these asynchronous algorithms rely on a deterministic bounded-delay assumption, which ensures that the slowest client in the network updates at least every τ iterations.
This assumption is satisfied only under certain conditions (Abbasloo &
Chao, 2020), and worsens the convergence rate by adding a sub-optimal reliance on τ .
[Table]
To remedy these drawbacks, we propose the S hared W a I
t-
F ree T ransmission (S WIFT ) algorithm: an efficient, scalable, and high-performing decentralized FL algorithm.
Unlike other decentralized
FL algorithms, S WIFT obtains minimal communication time per round due to its wait-free structure.
Furthermore, S WIFT is the first asynchronous decentralized FL algorithm to obtain an optimal
O (1 / √ T ) convergence rate (aligning with stochastic gradient descent) without a bounded-delay assumption .
Instead, S WIFT leverages the expected delay of each client (detailed in our remarks within Section 5).
Experiments validate S WIFT ’s efficiency, showcasing a reduction in communication time by nearly an order of magnitude and run-times by upwards of 35%.
All the while, S WIFT remains at state-of-the-art (SOTA) global test/train loss for image classification compared to other decentralized FL algorithms.
We summarize our main contributions as follows.
▷ Propose a novel wait-free decentralized FL algorithm (called S WIFT ) and prove its theoretical convergence without a bounded-delay assumption.
▷ Implement a novel pre-processing algorithm to ensure non-symmetric and non-doubly stochastic communication matrices are symmetric and doubly-stochastic under expectation.
▷ Provide the first theoretical client-communication error bound for non-symmetric and non-doubly stochastic communication matrices in the asynchronous setting.
▷ Demonstrate a significant reduction in communication time and run-time per epoch for CIFARclassification in IID and non-IID settings compared to synchronous decentralized FL.
R ELATED W ORKS Asynchronous Learning
HOGWILD!
(Recht et al., 2011), AsySG-Con (Lian et al., 2015), and AD-PSGD
(Lian et al., 2017) are seminal examples of asynchronous algorithms that allow clients to proceed at their own pace.
However, these methods require a shared memory/oracle from which clients grab the most up-to-date global parameters (e.g. the current graph-averaged gradient).
By contrast, S WIFT relies on a message passing interface (MPI) to exchange parameters between neighbors, rather than interfacing with a shared memory structure.
To circumvent local memory overload, common in IoT clusters (Li et al., 2018), clients in S WIFT access neighbor models sequentially when averaging. The recent works of (Koloskova et al., 2022; Mishchenko et al., 2022) have improved asynchronous SGD convergence guarantees which no longer rely upon the largest gradient delay. Like these works, S WIFT similarly proves convergence without a bounded-delay assumptions.
However, S WIFT differs as it functions in the decentralized domain as well as the FL setting.
Decentralized Stochastic Gradient Descent (SGD) algorithms are reviewed in Appendix D.
Communication Under Expectation.
Few works in FL center on communication uncertainty.
In (Ye et al., 2022), a lightweight, yet unreliable, transmission protocol is constructed in lieu of slow heavyweight protocols.
A synchronous algorithm is developed to converge under expectation of an unreliable communication matrix (probabilistic link reliability).
S WIFT also convergences under expectation of a communication matrix, yet in a different and asynchronous setting.
S WIFT is already lightweight and reliable, and our use of expectation does not regard link reliability.
Communication Efficiency.
Minimizing each client i ’s communication time per round C i is a challenge in FL, as the radius of information exchange can be large (Kairouz et al., 2021).
M ATCHA (Wang et al., 2019) decomposes the base network into m disjoint matchings.
Every epoch, a random sub-graph is generated from a combination of matchings, each having an activation probability p k .
Clients then exchange parameters along this sub-graph.
This requires a total communicationtime complexity of O ( T (cid:80) mk =1 p k max j ∈N i C j ) , where N i are client
i ’s neighbors.
LD-SGD (Li et al., 2019) and PA-SGD (Wang & Joshi, 2018) explore how reducing the number of neighborhood parameter exchanges affects convergence.
Both algorithms create a communication set C s (defined in Appendix D) that dictate when clients communicate with one another.
The communication-time complexities are listed in Table 1.
These methods, however, are synchronous and their communicationtime complexities depend upon the slowest neighbor max j ∈N i C j .
S WIFT improves upon this, achieving a complexity depending on a client’s own communication-time per round.
Unlike ADPSGD Lian et al.
(2018), which achieves a similar communication-time complexity, S WIFT allows for periodic communication, uses only local memory, and does not require a bounded-delay assumption.
P ROBLEM F ORMULATION
Decentralized FL.
In the FL setting, we have n clients represented as vertices of an arbitrary communication graph G with vertex set V = { 1 , . . . , n } and edge set E ⊆
V × V .
Each client i communicates with one-hop neighboring clients
j such that ( i, j ) ∈ E .
We denote the neighborhood for client i as N i , and clients work in tandem to find the global model parameters x by solving:
[Equation]
The global objective function f ( x ) is the weighted average of all local objective functions
f i ( x ) .
In
Equation 1, p i , ∀ i ∈ [ n ] denotes the client influence score.
This term controls the influence of client i on the global consensus model, forming the client influence vector p = { p i } ni =1 .
These scores also reflect the sampling probability of each client.
We note that each local objective function f i ( x ) is the expectation of loss function ℓ with respect to potentially different local data ξ i = { ξ i,j } Mj =1 from each client i ’s distribution D i , i.e., ξ i,j ∼ D
i .
The total number of iterations is denoted as T .
Existing Inter-Client Communication in Decentralized FL.
All clients balance their individual training with inter-client communications in order to achieve consensus while operating in a decentralized manner.
The core idea of decentralized FL is that each client communicates with its neighbors (connected clients) and shares local information.
Balancing individual training with inter-client communication ensures individual client models are well-tailored to personal data while remaining (i) robust to other client data, and (ii) able to converge to an optimal consensus model.
Periodic Averaging.
Algorithms such as Periodic Averaging SGD (PA-SGD) (Wang & Joshi, 2018) and Local Decentralized SGD (LD-SGD) reduce communication time by performing multiple local updates before synchronizing.
This process is accomplished through the use of a communication set C s , which defines the set of iterations a client must perform synchronization,
[Equation]
We adopt this communication set notation, although synchronization is unneeded in our algorithm.
S HARED W AIT -F REE T RANSMISSION (S WIFT ) F EDERATED L EARNING
In this section, we present the S hared W a I
t- F ree T ransmission (S WIFT ) Algorithm.
S WIFT is an asynchronous algorithm that allows clients to work at their own speed.
Therefore, it removes the dependency on the slowest client which is the major drawback of synchronous settings.
Moreover, unlike other asynchronous algorithms, S WIFT does not require a bound on the speed of the slowest client in the network and allows for neighborhood averaging and periodic communication.
A S WIFT Overview.
Each client i runs S WIFT in parallel, first receiving an initial model x
i , communication set C s , and counter c i ← 1 .
S WIFT is concisely summarized in the following steps:
(0) Determine client-communication weights w i via Algorithm 2 in Appendix B.2.
(1) Broadcast the local model to all neighboring clients.
(2) Sample a random local data batch of size M .
(3) Compute the gradient update of the loss function ℓ with the sampled local data.
(4) Fetch and store neighboring local models, and average them with one’s own local model
if c i ∈ C s .
(5) Update the local model with the computed gradient update, as well as the counter c i ←
c i + 1 .
(6) Repeat steps (1) - (5) until convergence.
A diagram and algorithmic block of S WIFT are depicted in Figure 1 and Algorithm 1 respectively.
Active Clients, Asynchronous Iterations, and the Local-Model Matrix.
Each time a client finishes a pass through steps (1) - (5) , one global iteration is performed.
Thus, the global iteration t is increased after the completion of any client’s averaging and local gradient update.
[Figure]
The client that performs the t -th iteration is called the active client, and is designated as i t (Line 6 of Algorithm 1).
There is only one active client per global iteration .
All other client models remain unchanged during the t -th iteration (Line 16 of
Algorithm 1).
In synchronous algorithms, the global iteration t increases only after all clients finish an update .
S WIFT , which is asynchronous, increases the global iteration t after any client finishes an update.
In our analysis, we define local-model matrix
X t ∈ R d × n as the concatenation of all local client models at iteration t for ease of notation,
[Equation]
Inspired by PA-SGD (Wang & Joshi, 2018), S WIFT handles multiple local gradient steps before averaging models amongst neighboring clients (Line 10 of Algorithm 1).
Periodic averaging for
S WIFT , governed by a dynamic client-communication matrix, is detailed below.
Initial model x
Output : Consensus model 1 n (cid:80) ni =1 x Ti p
Broadcast to all k
Sample a batch active
’s ξ ti
D
Compute the gradient update : g ( x ti ξ ti t ) ← 1 M (cid:80)
Mm =1 ∇ ℓ ( x ti t , ξ ti t ,m ) if current step falls in the predefined communication set, i.e., c i t ∈ C s then
Fetch and store the latest models { tk from i t ’s neighbors ti
← (cid:80) k w ti t k x tk + w ti t x ti else
Active model remains x t ← x ti
← x t
− γg ( x ti ξ ti
← x tj
∀ j ̸ = c
← c
Wait Free.
The backbone of S WIFT is its wait-free structure.
Unlike any other decentralized FL algorithms, S WIFT does not require simultaneous averaging between two clients or a neighborhood of clients.
Instead, each client fetches the latest models its neighbors have sent it and performs averaging with those available (Lines 11-12 of Algorithm 1).
There is no pause in local training waiting for a neighboring client to finish computations or average with, making S WIFT wait-free.
Update Rule.
S WIFT runs in parallel with all clients performing local gradient updates and model communication simultaneously.
Collectively, the update rule can be written in matrix form as
[Equation]
where γ denotes the step size parameter and the matrix G ( x ti t , ξ
ti t ) ∈ R
d × n is the zero-padded gradient of the active model x ti t .
The entries of G ( x ti t , ξ ti t ) are zero except for the i t -th column, which contains the active gradient g ( x ti t , ξ ti t ) .
Next, we describe the client-communication matrix W ti t .
Client-Communication Matrix.
The backbone of decentralized FL algorithms is the clientcommunication matrix W (also known as the weighting matrix).
To remove all forms of synchronization and to become wait-free, S WIFT relies upon a novel client-communication matrix W ti t that is neither symmetric nor doubly-stochastic, unlike other algorithms in FL (Wang & Joshi, 2018; Lian et al., 2018;
Li et al., 2019; Koloskova et al., 2020).
The result of a non-symmetric and non-doubly stochastic client-communication matrix, is that averaging occurs for a single active client i t and not over a pair or neighborhood of clients.
This curbs superfluous communication time.
Within S WIFT , a dynamic client-communication matrix is implemented to allow for periodic averaging.
We will now define the active client-communication matrix W ti t in S WIFT , where i t is the active client which performs the t -th global iteration.
W ti t can be one of two forms: (1) an identity matrix
W ti t
=
I n if c i t / ∈ C s or (2) a communication matrix if c i t ∈ C s with structure,
[Equation]
The vector w ti t ∈ R n denotes the active client-communication vector at iteration t , which contains the communication coefficients between client i t and all clients (including itself).
The clientcommunication coefficients induce a weighted average of local neighboring models.
We note that w ti t is often sparse because clients are connected to few other clients only in most decentralized settings.
Novel Client-Communication Weight Selection.
While utilizing a non-symmetric and nondoubly-stochastic client-communication matrix decreases communication time, there are technical difficulties when it comes to guaranteeing the convergence.
One of the novelties of our work is that we carefully design a client-communication matrix W ti t such that it is symmetric and doubly-stochastic under expectation of all potential active clients i t and has diagonal values greater than or equal to 1 /n .
Specifically, we can write
[Equation]
where, we denote ¯ W t as the expected client-communication matrix with the following form,
[Equation]
[Equation]
Note that ¯ W t is column stochastic as the entries of any column sum to one.
If we ensure that ¯ W t is symmetric, then it will become doubly-stochastic.
By Equation 7, ¯ W t becomes symmetric if,
[Equation]
To achieve the symmetry of Equation 8, S WIFT deploys a novel pre-processing algorithm:
the
Communication Coefficient Selection (CCS) Algorithm.
Given any client-influence vector p
i , CCS determines all client-communication coefficients such that Equations 5 and 8 hold for every global iteration t .
Unlike other algorithms, CCS focuses on the expected client-communication matrix, ensuring its symmetry.
CCS only needs to run once, before running S WIFT .
In the event that the underlying network topology changes, CCS can be run again during the middle of training.
In
Appendix B.2, we detail how CCS guarantees Equations 5 and 8 to hold.
The CCS Algorithm, presented in Appendix B.2, is a waterfall method: clients receive coefficients from their larger-degree neighbors.
Every client runs CCS concurrently, with the following steps:
(1) Receive coefficients from larger-degree neighbors.
If the largest, or tied, skip to (2) .
(2) Calculate the total coefficients already assigned s w as well as the sum of the client influence scores for the unassigned clients s p .
(3) Assign the leftover coefficients 1 − s w to the remaining unassigned neighbors (and self) in a manner proportional to each unassigned client
i ’s percentage of the leftover influence scores p
i /s p .
(4) If tied with neighbors in degree size, ensure assigned coefficients won’t sum to larger than one.
(5) Send coefficients to smaller-degree neighbors.
S WIFT T HEORETICAL A NALYSIS
Major Message from Theoretical Analysis.
As summarized in Table 1, the efficiency and effectiveness of decentralized FL algorithms depend on both the iteration convergence rate and communicationtime complexity; their product roughly approximates the total time for convergence.
In this section, we will prove that S WIFT improves the SOTA convergence time of decentralized FL as it obtains SOTA iteration convergence rate (Theorem 1) and outperforms SOTA communication-time complexity.
Before presenting our theoretical results, we first detail standard assumptions (Kairouz et al., 2021) required for the analysis.
Assumption
[Equation]
smooth
[Equation]
and
[Equation]
Assumption 2 (Unbiased stochastic gradient) .
[Equation] ∇ [Equation] x
[Equation]
∇ f [Equation] ( x ) for each i ∈ V .
Assumption 3 (Bounded inter-client gradient variance) .
The variance of the stochastic gradient is bounded for any x with client i sampled from probability vector p and local client data ξ sampled from D i .
This implies there exist constants σ, ζ ≥ 0 (where ζ = 0 in IID settings) such that:
[Equation]
As mentioned in Section 4, the use of a non-symmetric, non-doubly-stochastic matrix W ti t causes issues in analysis.
In Appendix C, we discuss properties of stochastic matrices, including symmetric and doubly-stochastic matrices, and formally define ρ ν , a constant related to the connectivity of the network.
Utilizing our symmetric and doubly-stochastic expected client-communication matrix (constructed via Algorithm 2), we reformulate Equation 4 by adding and subtracting out ¯ W t ,
[Equation]
Next, we present our first main result in Lemma 1, establishing a client-communication error bound.
Lemma 1 (Client-Communication Error Bound) .
Following Algorithm 2, the product of the difference between the expected and actual client communication matrices is bounded as follows:
[Equation]
Remark.
One novelty of our work is that we are the first to bound the client-communication error in the asynchronous decentralized FL setting.
The upper bound in Lemma 1 is unique to our analysis because other decentralized works do not incorporate wait-free communication (Lian et al., 2017; Li et al., 2019; Wang & Joshi, 2018; Lian et al., 2018).
Now, we are ready to present our main theorem, which establishes the convergence rate of S WIFT :
Theorem 1 (Convergence Rate of S WIFT ) .
Under assumptions 1, 2 and 3 (with Algorithm 2), let ∆ f := f
[Equation]
¯ x
[Equation]
− f [Equation] ¯ x ∗ ) , step-size γ , total iteration T , and average model ¯ x t be defined as
[Equation]
Then, for the output of Algorithm [Equation] it holds that
[Equation]
Iteration Convergence Rate Remarks.
(1) We prove that S WIFT obtains a O (1 / √ T ) iteration convergence rate, matching the optimal rate for SGD (Dekel et al., 2012; Ghadimi & Lan, 2013;
Lian et al., 2017; 2018).
(2) Unlike existing asynchronous decentralized SGD algorithms, S WIFT ’s iteration convergence rate does not depend on the maximal bounded delay.
Instead, we bound any delays by taking the expectation over the active client.
The probability of each client i being the active client is simply its sampling probability p
i .
We therefore assume that each client i is expected to perform updates at its prescribed sampling probability rate p i .
Clients which are often delayed in practice can be dealt with by lowering their inputted sampling probability.
(3) S WIFT converges in fewer total iterations T with respect to n total clients compared to other asynchronous methods (Lian et al., 2018)
( T = Ω( n 4 p 2 max ) in S WIFT versus T = Ω( n 4 ) in AD-PSGD).
Similar to AD-PSGD,
S WIFT achieves a linear speed-up in computational complexity as the number of clients increase.
Communication-Time Complexity Remarks.
(1) Due to its asynchronous nature, S WIFT achieves a communication-time complexity that relies only on each client’s own communication time per round C i .
This improves upon synchronous decentralized SGD algorithms, which rely upon the communication time per round of the slowest neighboring client max j ∈N i C j .
(2) Unlike AD-PSGD (Lian et al., 2018), which also achieves a communication-time complexity reliant on C i , S WIFT incorporates periodic averaging which further reduces the communication complexity from T rounds of communication to |C s | .
Furthermore, S WIFT allows for entire neighborhood averaging, and not just one-to-one gossip averaging.
This increases neighborhood information sharing, improving model robustness and reducing model divergence.
Corollary 1 (Convergence under Uniform Client Influence) .
In the common scenario where client influences are uniform, p i = 1 /n ∀ i = ⇒ p max
= 1 /n , S WIFT obtains convergence improvements: total iterations T with respect to the number of total clients n improves to T = Ω( n 2 ) as compared to T = Ω( n 4 ) for AD-PSGD under the same conditions.
E XPERIMENTS
Below, we perform image classification experiments for a range of decentralized FL algorithms (Krizhevsky et al., 2009).
We compare the results of S WIFT to the following decentralized baselines:
• The most common synchronous decentralized FL algorithm: D-SGD (Lian et al., 2017).
• Synchronous decentralized FL communication reduction algorithms: PA-SGD (Wang & Joshi, 2018) and LD-SGD (Li et al., 2019).
• The most prominent asynchronous decentralized FL algorithm:
AD-PSGD (Lian et al., 2018).
Finer details of the experimental setup are in Appendix A. Throughout our experiments we use two network topologies: standard ring and ring of cliques (ROC).
ROC- x C signifies a ring of cliques with x clusters.
The ROC topology is more reflective of a realistic network, as networks usually have pockets of connected clients.
These topologies are visualized in Figures 7 and 8 respectively.
6. B ASELINE C OMPARISON
To compare the performance of S WIFT to all other algorithms listed above
[Figure]
we reproduce an experiment within (Lian et al., 2018).
With no working code for AD-PSGD to run on anything but an extreme supercomputing cluster (Section 5.1.2 of (Lian et al., 2018)), reproducing this experiment allows us to compare the relative performance of S WIFT to AD-PSGD.
[Figure]
client ring.
Table in Figure 2a showcases that S WIFT reduces the average epoch time, relative to D-SGD, by 35% ( C 0 and C 1 ).
This far outpaces AD-PSGD (as well as the other synchronous algorithms), with
AD-PSGD only reducing the average epoch time by 16% relative to D-SGD.
Finally, Figure 2 displays how much faster S WIFT achieves optimal train and test loss values compared to other decentralized baseline algorithms.
S WIFT outperforms all other baseline algorithms even without any slow-down (which we examine in Section 6.2), where wait-free algorithms like S WIFT especially shine.
6. V ARYING H ETEROGENEITIES Varying Degrees of Non-IIDness
Our second experiment evaluates S WIFT ’s efficacy at converging to a well-performing optima under varying degrees of non-IIDness.
We vary the degree (percentage) of each client’s data coming from one label.
The remaining percentage of data is randomly sampled (IID) data over all labels.
A ResNet-18 model is trained by 10 clients in a 3-cluster ROC network topology.
We chose 10 clients to make the label distribution process easier: CIFAR-10 has 10 labels.
[Table]
As expected, when data becomes more non-IID
[Figure]
the test loss becomes higher and the overall accuracy lower (Table 2).
We do see, however, that S WIFT converges faster [Figure] and to a lower average loss, than all other synchronous baselines (Figure 3).
In fact,
S WIFT with C 1 converges much quicker than the synchronous algorithms.
This is an important result:
S WIFT converges both quicker and to a smaller loss than synchronous algorithms in the non-IID setting
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
Varying Heterogeneity of Clients
In this experiment, we investigate the performance of S WIFT under varying heterogeneity, or speed, of our clients (causing different delays).
This is done with 16 clients in a ring topology.
We add an artificial slowdown, suspending execution of one of the clients such that it takes a certain amount of time longer (slowdown) to perform the computational portions of the training process.
We perform tests in the case where a client is two times (2x) and four times (4x) as slow as usual.
We then compare how the decentralized algorithms fare under these circumstances compared to the normal setting with no added slowdown.
[Figure]
[Table] [Figure] [Table] [Figure]
[Table] [Table] [Figure] [Table]
[Figure]
[Table] [Table] [Figure] [Table]
[Table]
[Table] [Table] [Figure]
[Table]
[Figure]
[Table]
[Table]
[Table] [Table]
In Table 3, the average epoch, communication, and total time is displayed.
Average total time includes computations, communication, and any added slowdown (wait time).
S WIFT avoids large average total times as the slowdown grows larger.
The wait-free structure of S WIFT allows all non-slowed clients to finish their work at their own speed.
All other algorithms require clients to wait for the slowest client to finish a mini-batch before proceeding.
At large slowdowns (4x), the average total time for S WIFT is nearly half of those for synchronous algorithms.
Thus, S WIFT is very effective at reducing the run-time when clients are slow within the network.
Figure 4 shows how S WIFT is able to converge faster to an equivalent, or smaller, test loss than
D-SGD for all slowdowns.
In the case of large slowdowns (4x), S WIFT significantly outperforms
D-SGD, finishing in better than half the run-time.
We do not include the other baseline algorithms to avoid overcrowding of the plotting space.
However, S WIFT also performs much better than PA-SGD and LD-SGD as shown in Table 3.
These results show that the wait-free structure of S WIFT allows it to be efficient under client slowdown.
6. V ARYING N UMBERS OF C LIENTS & N ETWORK T OPOLOGIES
Varying Numbers of Clients In our fourth experiment, we determine how S WIFT performs versus other baseline algorithms as we vary the number of clients.
[Figure]
A v e r a g e C
o mm un i ca ti on T
i m e ( s )
[Figure]
In Table 4, the time per epoch for S WIFT drops by nearly the optimal factor of 2 as the number of clients is doubled.
For all algorithms, there is a bit of parallel overhead when the number of clients is small, however this becomes minimal as the number of clients grow to be large (greater than 4 clients).
In comparison to the synchronous algorithms, S WIFT actually decreases its communication time as the number of clients increases.
This allows the parallel performance to be quite efficient, as shown in Figure 5.
[Figure]
[Table] [Figure] [Table] [Figure]
[Figure]
[Table]
Varying Topologies Our fifth experiment analyzes the effectiveness of S WIFT versus other baseline decentralized algorithms under different [Table] and more realistic, network topologies.
In this experiment setting, we train 16 clients on varying network topologies (Table 5 and Figure 6)
[Table]
[Figure]
[Figure] [Figure] o ss
[Figure] [Figure]
[Figure] [Figure]
T e s t L o ss
[Figure]
[Figure]
C ONCLUSION
S WIFT delivers on the promises of decentralized FL: a low communication and run-time algorithm ( scalable ) which attains SOTA loss ( high-performing ).
As a wait-free algorithm, S WIFT is wellsuited to rapidly solve large-scale distributed optimization problems.
Empirically, S WIFT reduces communication time by almost 90% compared to baseline decentralized FL algorithms.
In future work, we aim to add protocols for selecting optimal client sampling probabilities.
We would like to show how varying these values can: (i) boost convergence both theoretically and empirically, and (ii) improve robustness under local client data distribution shift.
E THICS S TATEMENT
We propose a novel wait-free decentralized Federated Learning algorithm with strong theoretical guarantees and improved empirical results. Our contributions add to the sparse foundational literature in asynchronous decentralized Federated Learning. Therefore, our work does not have direct societal or ethical consequences.
We would like to note that it is imperative for users of Federated Learning algorithms in real-world distributed learning applications to respect data privacy.
R EPRODUCIBILITY
Our code can be found on GitHub at https://github.com/umd-huang-lab/SWIFT .
We ran five trials for S WIFT and each baseline algorithm we compared it to with random seeds over each experiment (Sections 6.1, 6.2, 6.2, 6.3).
Our plots include error bars from these five trials for each experiment.
As stated in Section 6, we perform image classification experiments on the
CIFAR-10 dataset (Krizhevsky et al., 2009).
In Table 6 we describe the hyperparameters we use in all experiments.
Appendix A describes further experimental setup details (computational resources used, how we partition data, and more).
Finally we provide pseudocode for S WIFT in Algorithm 1.
A CKNOWLEDGMENTS
Bornstein, Rabbani and Huang acknowledge support by the National Science Foundation NSF-IISFAI program, DOD-ONR-Office of Naval Research, DOD-DARPA-Defense Advanced Research
Projects Agency Guaranteeing AI Robustness against Deception (GARD), Adobe, Capital One and
JP Morgan faculty fellowships.
Rabbani is additionally supported by NSF DGE-1632976.
Bedi acknowledges the support by Army Cooperative Agreement W911NF2120076.
Bornstein additionally thanks Michael Blankenship for both helpful discussions regarding parallel code implementation and inspiration for S WIFT ’s name.
Supplementary Material A A DITIONAL E XPERIMENTAL D ETAILS AND S ETUP A. C OMPUTATIONAL S PECIFICATIONS
We train our consensus model on a network of nodes.
Each node has an NVIDIA GeForce RTX
Ti GPU.
All algorithms are built in Python and communicate via Open MPI, using MPI4Py (Python bindings for MPI).
The training is also done in Python, leveraging Pytorch.
A. D ATA P ARTITIONING
For all experiments, the training set is evenly partitioned amongst the number of clients training the consensus model.
While the size of each client’s partition is equal, we perform testing with data that is both (1) independent and identically distributed (iid) among all clients and (2) sorted by class and is thus non-iid.
For the iid setting, each client is assigned data uniformly at random over all classes.
In the non-iid setting, each client is assigned a subset of classes from which it will receive data exclusively.
The c classes are assigned in the following steps:
(1) The class subset size n c , for all clients, is determined as the ceiling of the number of classes per client n c = ⌈ cn ⌉ .
Each class k within the class subset will take up 1 /n c of the client’s total data partition if possible.
The classes within each client’s class subset are assigned cyclically, starting with the first client.
The first client selects the first n c classes, the second client selects the next n c classes, and so on.
Classes can, in some cases, be assigned to multiple clients.
If the final class has been assigned
and more clients have yet to be assigned any classes, classes can be re-assigned starting at the first class.
Each client is assigned data from the classes in their class subset cyclically ( 1 /n c of its partition for each class), starting with the first client.
If no more data is available from a specific class, the required data to fill its fraction of the partition is replaced by data from the next class.
Since we follow this data partitioning process within our experiments, each client is assigned equal partitions of data.
Therefore, following the works (Lian et al., 2018; Wang et al., 2019; Ye et al., 2022;
Li et al., 2019), we set the client influence scores to be uniform for all clients p i = 1 /n ∀
i ∈ V .
A. E XPERIMENTAL S ETUP
Below we provide information into the hyperparameters we select for our experiments in Section 6.
[Table]
In Table 6, one can see that the step-size decay column is split into the following sections: rate, E , and frequency.
The rate is the decay rate for the step-size.
For example, in the Baseline row, the step-size decays by 1/10.
The term E is the epoch at which decay begins during training.
For example, in the
Baseline row, the step-size decays at E = 81 and 122 .
Frequency simply is how often the step-size decays.
For example, in the Vary Topology row, the step-size decays every 10 epochs.
A.
N ETWORK T OPOLOGIES
Ring Topology.
Please refer to Figure 7.
Client
Client Client
Client Client
Client Client
Client
Ring of Cliques Topology.
Please refer to Figure 8.
Client
Client Client
Client Client
Client Client
Client
Client 7 Client
Client Client
Client Client
Client Client
Client Client
Client Client
Client Client
Client
Client 1 Client
Client
Client Client Client Client
Client Client Client Client
Client Client Client
Client Client
Client
Client
Client
Like the works of (Jeon et al., 2021; Bellet et al., 2021), we believe that the ring of cliques network topology is a realistic topology in the decentralized setting.
In many real-world setting, like one’s home (smart appliances, smart speakers/displays, phones, etc.), devices are connected together in a small clique.
Only a small amount of these devices have connections to other devices outside the cluster (like a phone or router).
We wanted to utilize this network topology due to this realistic nature.
Furthermore, (Bellet et al., 2021) shows that a ring of clique topology can be used in the decentralized setting to reduce the impact of label distribution skew.
B A LGORITHM D ETAILS AND N OTATION B. N OTATION T ABLE
Definition
Notation
Definition
Notation
Global Iteration t
Number of Clients n Active Client at t
i t
Parameter Dimension d Client ( i,j ) Communication Coefficient at t w ti,j
Data Dimension f
Local Model of Client i at t x ti ∈ R d
Step-Size γ
Local Model of Active Client at t x ti t ∈ R d
Total
[Table]
T
[Table] ξ ti t ∈ R f
[Table] Size [Table]
M
[Table] g ( x t i t ,ξ t i t ) ∈ R d
[Table]
C s [Table] w ti ∈ R n
[Table]
D
[Table]
X t ∈ R d × n
[Table]
f ( x )
[Table]
G ( x ti t ,ξ ti t ) ∈ [Table] d × n
[Table]
p i Expected
[Table]
¯ [Table] [Table] t
[Table]
ξ t ) ∈ [Table]
d × n
[Table] p ∈ R n
[Table]
W ti t ∈ R
[Table]
n
[Table]
i ∈
[Table]
I
[Table]
Input : Client Influence Score (CIS) p i ∈ R , Client Degree d i ∈ R ,
Client Neighbor Set J i
= {∀ j : client j is a one-hop neighbor of client i } , ∀
i
Output : Client-Communication Vector w i ∈ R n , ∀ i ∈ [ n ] do
Initialize Client
Vector w i
=
[ w 1 ,i , w 2 ,i , · · · , w n,i ] ← (1 /n )
e i else w i =
[ w 1 ,i , w 2 ,i , · · · , w n,i ] ←
Exchange j
J storing all exchanged s jw , s Store s
←
{ s iw , s jw } s p ← max { s ip , s j
J E
Set w ,i p j Recompute s iw = (cid:80) nm =1 w m,i and s
ip = (cid:80) j ∈{ J SE ∪ i }\ J E P Jj
Set
− s else
− s
B. S WIFT P RE - PROCESSING : S ETTING C LIENT -C OMMUNICATION W EIGHTS
Below we present the the algorithmic pseudocode here for our novel client-communication selection algorithm in Algorithm 2.
As a note, we include different client-communication vector initializations if the client influence scores are uniform versus non-uniform.
The reason for this is to ensure that the selfweight for each client i , w i,i , has a value greater than 1 /n .
This naturally occurs when the CIS are uniform, however is not so when they are non-uniform.
In Algorithm 2, the terms s w and s p play a pivotal role in satisfying Equations 5 and 8 respectively.
Equation 8 is satisfied by assigning weights amongst client i and its neighbors j as follows
[Equation]
Assigning weights proportionally with respect to neighboring client influence scores and total neighbors ensures higher influence clients receive higher weighting during averaging.
B. O PTIMAL S TEP -S IZE U NDER U NIFORM C LIENT I NFLUENCE
The defined step-size γ and total iterations T for S WIFT is
[Equation]
Therefore, γ can be rewritten as
[Equation]
When the client influence scores are uniform (i.e, p i = 1 /n ∀ i ∈ V ), one can see that our step-size becomes
[Equation]
This mirrors the optimal step-size in analysis of gradient descent convergence to first-order stationary points O (1 /L )
(Nesterov, 1998).
B. I NSTANTANEOUS C OMMUNICATION AND O VERCOMING C OMPUTATIONAL D ELAY
Similar to AD-PSGD (Lian et al., 2018), we assume that model transmission is instantaneous between clients.
Within (Lian et al., 2018), two algorithms (Algorithms 2 and 3) are provided within the appendix to provide realistic multi-thread implementation of AD-PSGD.
The computational thread of Algorithm 2 is explicitly told to wait at Line 5 until the gradient buffer is empty.
Unless a gradient queue is utilized, which alters the effective batch-size from the constant M , the computational thread must wait for model transmission if there exists communication delay.
Important future work for S WIFT , and within the field of asynchronous decentralized FL in general, is to remove this instantaneous delay assumption and build protocols to overcome it (and theoretically guarantee convergence in its presence).
We would like to mention that eliminating computational delay is a key novelty of S WIFT .
Delayed computations are handled by SWIFT through its communication matrix.
Unlike AD-PSGD, no model overwriting can occur while a client is performing gradient computations (Line 9 of Algorithm 1).
This only occurs due to our novel non-doubly stochastic communication matrix.
Therefore, client will never be computing gradients with a delayed model.
This, of course, assumes that model transmission is instantaneous.
SWIFT deals with a slow client i (having delayed computations) by allowing a faster client j to reuse a stored model of client i until client i finally finishes its model update.
The stored model is still up to date (and not delayed) since client i has not finished its gradient computations, updated its model, and sent out its updated model to neighboring clients (as it experiences delayed computations).
If a client has not received a message from a neighbor, then their stored model for that neighbor is still its most up-to-date model.
C P ROPERTIES OF C OMMUNICATION M ATRICES
Stochastic Matrices.
Within our work, we use a non-symmetric, non-doubly-stochastic matrix W ti t for client averaging.
Utilizing W ti t comes with some analysis issues (it is non-symmetric), however it provides the wait-free nature of S WIFT .
Interestingly, W ti t does have some unique properties: it is column-stochastic.
Lemma 3 proves that the product of stochastic matrices converges exponentially to a stochastic vector with common ratio ν ∈ [0 , 1) .
Symmetric and Doubly-Stochastcic Matrices.
As mentioned in Section 5, we utilize Algorithm 2 to select client weights such that we have a symmetric and doubly-stochastic communication matrix ¯ W t under expectation.
By Lemma 2, there exists a scalar ρ ∈ [0 , 1) such that max {| λ 2 (cid:0) ( ¯ W t ) ⊺ ¯ W t (cid:1) | , | λ n (cid:0) ( ¯ W t ) ⊺ ¯ W t (cid:1) |} ≤ ρ, ∀ t.
This parameter ρ reflects the connectivity of the underlying graph topology.
The value of ρ is inversely proportionate to how fast information spreads in the client network.
A small value of ρ results in information spreading faster ( ρ = 0 in centralized settings).
Within our analysis, we denote the parameter ρ ν as a combination of ρ and ν :
[Equation]
[Equation]
D R EVIEW OF E XISTING I NTER -C LIENT C OMMUNICATION IN D ECENTRALIZED FL
The predecessor to decentralized FL is gossip learning (Boyd et al., 2006; Heged˝us et al., 2021).
Gossip learning was first introduced by the control community to assist with mean estimation of decentrally-hosted data distributions (Aysal et al., 2009; Boyd et al., 2005).
Now, SGD-based gossip algorithms are used to solve large-scale machine learning tasks (Lian et al., 2015; 2018; Ghadimi et al., 2016; Nedic & Ozdaglar, 2009; Recht et al., 2011; Agarwal & Duchi, 2011).
A key feature of gossip learning is the presence of a globally shared oracle/memory with whom clients exchange parameters at the end of training rounds (Boyd et al., 2006).
While read/write-accessible shared memory is well-suited for a single-organization ecosystem (i.e. all clients are controllable and trusted), this is unrealistic for more general edge-based paradigms.
Below, we review newer decentralized SGD algorithms, such as D-SGD
(Lian et al., 2017), PA-SGD (Wang & Joshi, 2018), and LD-SGD Li et al.
These algorithms theoretically and empirically outperform their centralized counterparts, especially under heterogeneous client data distribution.
Decentralized SGD (D-SGD) (Lian et al., 2017)
One of the foundational decentralized Federated
Learning algorithms is Decentralized SGD.
In order to minimize Equation 1, D-SGD orchestrates a local gradient step for all clients before performing synchronous neighborhood averaging.
The
D-SGD process for a single client i is defined as:
[Equation]
The term g ( x tj , ξ tj ) denotes the stochastic gradient of x tj with mini-batch data ξ tj sampled from the local data distribution of client j .
The matrix W is a weighting matrix, where W ij is the amount of x t
+1 i which will be made up of client j ’s local model after one local gradient step (e.g. if W ij = 1 / 2 , then half of x t +1 i will have been composed of client j ’s model after its local gradient step).
The weighting matrix only has a zero value W ij = 0 if clients i and j are not connected (they are not within the same neighborhood).
The values of W ij are generally selected ahead of time by a central host, with the usual weighting scheme being uniform.
In D-SGD, model communication occurs only after all local gradient updates are finished.
These gradient updates are computed in parallel.
Periodic Averaging SGD (PA-SGD) (Wang & Joshi, 2018)
The Periodic Averaging SGD algorithm is an extension of D-SGD.
In order to save communication costs when the number of clients grows to be large, PA-SGD performs model averaging after an additional I 1 local gradient steps.
Thus, the communication set for PA-SGD is defined as:
[Equation]
The special case of I 1 = 0 reduces to D-SGD.
The PA-SGD process for a single client i is defined as:
[Equation]
otherwise .
[Equation]
Compared with D-SGD, PA-SGD still suffers from the inefficiency of having to wait for the slowest client for each update.
However, PA-SGD saves communication costs by reducing the frequency of communication.
Local Decentralized SGD (LD-SGD) (Li et al., 2019)
Continuing to generalize the foundational decentralized Federated Learning algorithms is Local Decentralized SGD.
LD-SGD generalizes
PA-SGD by allowing multiple chunks of singular D-SGD updates, as described in Equation 14, in between the increased local gradient steps seen in PA-SGD.
The number of D-SGD chunks is dictated by a new parameter I 2 .
In this case, the communication set for LD-SGD is defined as
[Equation]
For example, in the case I 1 = 3 , I 2 = 2 , LD-SGD will take three local gradient steps and then perform two D-SGD updates (which consists of a local gradient step and then averaging) as shown in
Figure 9.
The special case of I 2 = 1 reduces to PA-SGD.
The LD-SGD process for a single client i is defined as:
[Equation]
otherwise
[Equation]
In the literature, there also exist other asynchronous decentralized learning methods such as in Cao &
Ba¸sar (2020); Bedi et al.
(2019a;b), but they are limited to convex objectives and hence not applicable to the setting in our work.
E P ROOF OF THE M AIN T HEOREM
Before beginning, we quickly define the expected gradient E i t (cid:2) G
( x ti t , ξ i t ∗ ,t ) (cid:3) as
[Equation]
Proof of Theorem 1.
In this theorem, we characterize the convergence of the average of all local models.
Using the Gradient Lipschitz assumption with Equation 9 yields
[Equation]
We first denote the average over all local models as ¯ x t :=
X t 1 n n .
Taking the expectation with respect to the updating client i t yields
[Equation] [Equation]
Taking the expectation over all local data E ξ ∼D i yields
[Equation]
By properties of the inner product
[Equation]
Bounding Term A :
Given the update equation, term A can be transformed into
[Equation]
Due to the symmetric and doubly stochastic property of ¯ W t this reduces to E
[Equation]
∼D
i
[Equation]
i
[Equation]
[Equation] yields [Equation] Since n ≥ 2 (we assume at least 2 devices are running the algorithm)
we find the following result
[Equation]
Thus
[Equation]
we have bounded Term A .
Substituting this back into Equation 24 results in
[Equation]
[Equation]
Taking the sum from t = 0 to t =
T − 1 yields
[Equation] [Equation]
Using the Lipschitz Gradient assumption
[Equation]
the following term is bounded as
[Equation]
Placing this back into Equation 32, and rearranging, yields
[Equation]
Given that ¯ x 0 = x 0 i for all clients i , one can see that
[Equation]
Using the result of Equation 38 condenses Equation 37 into
[Equation]
[Equation]
Bounding Term B.
The recursion of our update rule can be written as
[Equation]
The recursion equation for the expected consensus model ¯ x t and expected local model for client i can be computed by multiplying by 1 n n and e i respectively
[Equation]
[Equation]
[Table] ,j ) (cid:18) 1 n n − t (cid:89) q = j + ¯ W q e i (cid:19)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) + (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) t (cid:88) j
=
{+G ( x ji j , ξ i j ∗ ,j ) t (cid:89) q = j + ( W qi q − ¯ W q )( 1 n n − e i ) (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) 
+}
(cid:19) (46) =
2 γ 2 E ξ ∼D
i
T − 1 (cid:88) t = n (cid:88) i = p
i (cid:18) t (cid:88) j
=
(cid:13)(cid:13) (cid:13)(cid:13)(cid:13)(cid:13) G ( x j i j , ξ i j ∗ ,j ) (cid:18) 1 n n − t (cid:89) q =
j + ¯ W q e i (cid:19)(cid:13)(cid:13) (cid:13)(cid:13)(cid:13)(cid:13) (cid:124) (cid:123)(cid:122) (cid:125) :
= B+ t (cid:88)
j = (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) G ( x ji j , ξ i j ∗ ,j ) t (cid:89) q =
j + ( W qi q − ¯ W q )( 1 n n − e i ) (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation] [Equation]
Bounding Term B 1 .
Using Lemma 2,
[Equation]
j = (cid:13) (cid:13)(cid:13) G ( x ji [Equation]
i
[Equation] ) (cid:13) (cid:13) (cid:13)(cid:13)(cid:13) (cid:13) (cid:13)(cid:13)(cid:13)(cid:18) 1 n n − t (cid:89) q =
j + ¯ W q e i (cid:19)(cid:13)(cid:13) (cid:13) (cid:13)(cid:13)(cid:13) (48) =
E ξ ∼D i t (cid:88)
j = (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
M
M (cid:88) m =
[Equation]
[Equation]
) (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:18) 1 n n − t (cid:89) q = j + ¯
[Equation]
e
[Equation]
(cid:19)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) (49) ≤ E ξ ∼D i t (cid:88)
[Equation]
= (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
M
[Equation]
(cid:88)
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
[Equation]
t − j (50)
Using Lemma 4, the equation above becomes =
[Equation]
[Equation]
[Equation]
Taking the expectation over worker i j yields the desired bound
[Equation]
Bounding Term B 2 .
Using Lemma 2,
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table] [Equation] − [Equation]
For any α j,j ′ > 0 we find ≤
[Table]
(cid:88)
j =
[Equation]
+ [Equation] [Equation] 56) ≤
[Equation]
j
[Equation]
=
[Equation]
By applying inequality of arithmetic and geometric means to the term in the last step
[Equation]
we can choose α j,j ′ > 0 s
[Equation]
t.
[Equation]
Using Lemma 4 (and the expectation E ξ ∼D i that was omitted above but is present) yields
[Equation]
Taking the expectation over worker i j yields the desired bound
[Equation]
Bounding Term B 3 .
[Equation]
Due to the structure of ϕ 1 ⊺ , multiplying this matrix by 1 n n or e i yields the same result.
Using this, as well as the double stochasticity of ¯ W
[Equation]
we find
[Equation]
(cid:21)
[Equation]
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
( x ji j , [Equation] i
[Equation]
∗ ,j )( 1 n n − t (cid:89) q = j + ¯ W q e i ) (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) (cid:124) (cid:123)(cid:122) (cid:125) =
B(cid:21) (71)
Using the result of Lemma 3, as our communication graph G is uniformly strongly connected and [ W ti t ] i,i ≥ 1 /n by construction, we see that [Equation]
[Equation]
Using this result, we find that
[Equation]
We can remove the -1 exponent by doubling the constant out front
[Equation]
Finally, using the fact that G ( x ji j , ξ i j ∗ ,j ) is all zeros except for one column, the i j -th column, yields the desired result
[Equation]
Utilizing Lemma 4 yields
[Equation]
By properties of geometric series
[Equation]
and taking the expectation over worker i j , we find
[Equation]
Using the bound of B 1 in the main proof above
[Equation]
we arrive at the final bound of B
E ξ ∼D
i
[Equation]
Bounding Term B 4 .
Following similar steps as bounding Term B 2 we find
[Equation]
Once again
[Equation]
we can use the proof of Lemma 1 to bound this result
[Equation]
Finishing Bound of Term B .
Putting all terms together
[Equation]
we find that Term B is bounded above by T −
[Equation]
[Equation]
+
[Equation]
−
[Equation]
) n
[Equation]
Simplifying results in
[Equation]
Using Lemma 5 results in
[Equation]
[Table]
[Equation]
[Table]
i = p i ∇ f i ( x ji ) (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) + 6 Ln (cid:88) i
= p
i (cid:13)(cid:13)(cid:13) ¯ x j − x ji (cid:13)(cid:13)(cid:13) 2 + 3 ζ 2 (cid:19)(cid:18)
7 2 ρ t − j + ( t − j ) ρ t − j 2 + 384 ν 2( t − j ) (cid:19) (87) ≤
[Table]
2 σ 2 Tρ ν
M + 16( n − 1) γn
T − 1 (cid:88)
j = (cid:18) (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) n (cid:88) i
= p
i ∇ f i ( x ji ) (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
+ 6 Ln (cid:88) i =
p i (cid:13)(cid:13)(cid:13) ¯ x j − x ji (cid:13)(cid:13)(cid:13) 2 + 3 ζ 2 (cid:19)(cid:18) ∞ (cid:88) h
=  2 ρ h + hρ h 2 + 384 ν 2 h
(cid:19) (88) ≤
[Table]
2 σ 2 Tρ ν
M + 16 ρ ν γ
T − 1 (cid:88)
j = (cid:18) (cid:13) (cid:13)(cid:13)(cid:13)(cid:13) n (cid:88) i
= p
i ∇ f i ( x ji ) (cid:13) (cid:13)(cid:13)(cid:13)(cid:13) + 6 Ln (cid:88) i
= p
i (cid:13)(cid:13)(cid:13)
[Equation]
[Equation]
Using Equation 38 ends with
[Equation] [Equation]
Now subtract the final term on the right hand side from both sides T −(cid:88) t
[Equation]
By Lemma 6, we can divide z from both sides
[Equation]
Finishing Bound of Term A .
Substituting the bound of B above into Equation 41 yields
[Equation]
Rearranging terms simplifies the inequality above to
[Equation]
From Lemma
[Equation]
we find that (1 − 32 ρ ν ϕγnz ) ≥ 0 .
Therefore
[Equation]
the second term of the right hand side above can be removed.
[Equation]
Rearranging the inequality above and dividing by T yields
[Equation]
From Lemmas 6 and
[Equation]
the inequality above becomes
[Equation]
Substituting in the defined step-size γ (as well as its bound) yields
[Equation]
The final desired result is shown as
[Equation]
F A DDITIONAL L EMMAS
Lemma 2 (From Lemma 3 in Lian et al.
2018 (Lian et al., 2018)) .
Let W t be a symmetric doubly stochastic matrix for each iteration t .
Then
[Equation]
Lemma 3 (From Corollary 2 in Nedic and Olshevsky 2014 (Nedi´c & Olshevsky, 2014)) .
Let the communication graph G be uniformly strongly connected (otherwise known as B -strongly-connected for some integer B > 0 ), and A ( t ) ∈ R n × n be a column stochastic matrix with [ A ( t )]
i,i ≥ 1 /n ∀ i, t .
Define the product of matrices A ( t ) through A ( s ) (for t ≥ s ≥ 0 ) as A ( t : s ) :
= A ( t ) . . .
A ( s ) .
Then, there exists a stochastic vector ϕ ( t ) ∈ R n such that
[Equation]
will always hold for the following values of C and ν ,
[Equation]
[Equation]
Lemma 4.
Under Assumption 1, the following inequality holds
[Equation]
Lemma 5.
Under Assumption 1, the following inequality holds
[Equation]
Lemma 6.
Given the defined step-size γ and total iterations T in Theorem 1, the term z is and bounded by
[Equation]
Lemma 7.
Given the defined step-size γ and total iterations T in Theorem 1, the term ϕ is and bounded by
[Equation]
Lemma 8.
Given the defined step-size γ and total iterations T in Theorem
[Equation]
we find the following bound
[Equation]
G L EMMA P ROOFS
Proof of Lemma 1.
These steps are shown in the Main Theorem proof.
Due to the symetry and double stochasticity of ¯ W we find
[Equation]
[Table]
[Equation]
Due to the structure of ϕ 1 ⊺
[Equation]
multiplying this matrix by 1 n n or e i yields the same result.
Using this
[Equation]
as well as the double stochasticity of ¯ W
[Equation]
we find
[Equation]
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) (cid:124) (cid:123)(cid:122) (cid:125) =
B(cid:21) (107)
Using the result of Lemma
[Equation]
as our communication graph G is uniformly strongly connected and [ W ti t ] i,i ≥ 1 /n by construction, we see that [Equation]
[Equation]
Using this result
[Equation]
we find that
[Equation]
[Equation]
We can remove the -1 exponent by doubling the constant out front
[Equation]
Finally, using the fact that G ( x ji j , ξ i j ∗ ,j ) is all zeros except for one column [Equation] the i j -th column [Equation]
yields the desired result
[Equation]
Utilizing Lemma 4 yields
[Equation]
By properties of geometric series [Equation] and taking the expectation over worker i j
[Equation]
we find [Equation] [Equation] Using the bound Equation 54 (term B 1 ) in the main proof above
[Equation]
we arrive at the final bound
E ξ ∼D i t (cid:88) j =
[Equation]
E i j (cid:13)(cid:13)(cid:13) ∇ f i j
( x ti j ) (cid:13)(cid:13)(cid:13) 2 ν 2( t − j ) + 4( n − 1) n t (cid:88)
j = E i j (cid:13)(cid:13)(cid:13) ∇ f i j ( x ji j ) (cid:13)(cid:13)(cid:13) 2 ρ t − j (114) ≤ 4( n − 1) σ Mn (cid:18) (1 − ρ ) + (1 − ν 2 ) (cid:19) + 4( n
[Equation]
1) n t (cid:88) j = E i j (cid:13)(cid:13)(cid:13) ∇ f i j ( x ji j ) (cid:13)(cid:13)(cid:13)(cid:18) ρ t − j + 128 ν 2( t − j ) (cid:19) (115)
Thus, we have our desired result
[Equation]
[Equation] Proof of Lemma [Equation] [Equation]
[Equation]
Proof of Lemma 5.
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
(cid:88) [Equation] ( x [Equation]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
(cid:19) (123)
The first term on the right hand side can be bounded by
E
[Equation]
(cid:13) (cid:13)
[Equation]
= [Equation] (cid:18) [Equation]
+
[Equation]
) (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
[Equation]
[Equation]
[Equation]
[Equation]
Proof of Lemma 6.
It is trivial to see that z < 1 .
We now determine the lower bound of z given n ≥ 2 , p max ≥ 1 /n [Equation] and ρ ν ≥ 775 /
[Equation]
Proof of Lemma 7.
Given n ≥ 2 and ρ ν ≥ 775 / 4 , and the definition of γ and T , one can see ϕ =
[Equation]
Proof of Lemma 8.
Given n ≥ 2 , p max ≥ 1 /n Lemma 6, and Lemma 7, one can see
[Equation]