Asynchronous Speedup in Decentralized Optimization
Abstract
In decentralized optimization, nodes of a communication network each possess a local objective function, and communicate using gossip-based methods in order to minimize the average of these per-node functions.
While synchronous algorithms are heavily impacted by a few slow nodes or edges in the graph (the straggler problem ), their asynchronous counterparts are notoriously harder to parametrize.
Indeed, their convergence properties for networks with heterogeneous communication and computation delays have defied analysis so far.
In this paper, we use a continuized framework to analyze asynchronous algorithms in networks with delays.
Our approach yields a precise characterization of convergence time and of its dependency on heterogeneous delays in the network.
Our continuized framework benefits from the best of both continuous and discrete worlds: the algorithms it applies to are based on event-driven updates.
They are thus essentially discrete and hence readily implementable.
Yet their analysis is essentially in continuous time, relying in part on the theory of delayed ODEs.
Our algorithms moreover achieve an asynchronous speedup : their rate of convergence is controlled by the eigengap of the network graph weighted by local delays, instead of the network-wide worst-case delay as in previous analyses.
Our methods thus enjoy improved robustness to stragglers.
Introduction
We study the following optimization problem:
[Equation]
1) where each individual function f i : R d → R for i ∈ [ n ] is held by an agent i .
We consider asynchronous and decentralized optimization methods that do not rely on a central coordinator.
This is particularly relevant in large-scale systems in which centralized approaches suffer from a communication bottleneck at the central controller.
Decentralized optimization is relevant to supervised learning of models in data centers, but also to more recent federated learning scenarios where data and computations are distributed among agents that do not wish to share their local data.
We focus on asynchronous operations because of their scalability in the number of agents in the system, and their robustness to node failures and to stragglers .
In the case of empirical risk minimization, f i represents the empirical risk for the local dataset of node i , and f the empirical risk over all datasets.
Another important example, that plays the role of a toy problem for both decentralized and/or stochastic optimization is that of network averaging, corresponding to f i ( x ) =
∥ x − c i ∥where c
i is a vector attached to node i .
In this case, the solution of Problem (1) reads ¯ c = 1 n (cid:80) n i =1 c i .
We assume that agents are located at the nodes of a connected, undirected graph G =
( V, E ) with node set V =
[ n ] .
An agent i ∈ V can compute first-order quantities (gradients) related to its local objective function f i , and can communicate with any adjacent agent in the graph.
Our model of asynchrony derives from the popular randomized gossip model of [5].
In this model, nodes update their local values at random activation times using pairwise communication updates.
This asynchronous model makes the idealized assumption of instantaneous communications, and hence does not faithfully represent practical implementations.
To alleviate this drawback, several works [2, 23, 24, 34, 37, 39] introduce communication and computation delays in either pairwise updates, or in asymmetric gossip communications.
However, all these works provide convergence guarantees that either require global synchronization between the nodes, or are implicitly determined by an upper bound on the worst-case delay in the whole graph.
Indeed they assume that i) for some k max > 0 , for all edges ( ij ) ∈ E , each communication between agents i and j overlaps with at most k max other communications in the whole graph, and ii) either agents i or graph edges ( ij ) are activated for agent interaction sequentially in an i.i.d. manner.
Thus assuming distributed asynchronous operation where individual nodes schedule their interactions based only on local information, the k max constraint can only be enforced by requiring individual nodes to limit their update frequency to 1 / ( nτ max ) .
Consequently, the resulting algorithms have temporal convergence guarantees proportional to τ max .
They are thus not robust to stragglers , i.e. slow nodes or edges in the graph that induce large τ max .
To understand the scope for improvement over such methods, recall that for synchronous algorithms with updates performed every τ max seconds, for L -smooth and σ -strongly convex functions f i , the time required to reach precision ε > 0 for 1 n (cid:80)
f i is lower-bounded by [32]:
[Equation]
where κ = L/σ is the condition number of the functions f i and Diam( G ) is the diameter of graph
G .
In this article we seek better dependency on individual delays in the network.
Specifically we consider the following
Assumption 1 (Heterogeneous delays) .
There exist τ ij for ( ij ) ∈ E and τ comp i for i ∈ V such that communications between two neighboring agents i and j in the graph take time at most τ ij , and a computation at node i takes time at most τ comp i .
Under such heterogeneous delay assumptions, how robust to stragglers can decentralized algorithms be?
One can adapt the proof of [32] to Assumption 1 to generalize (2) as:
[Equation]
where D ( τ ) =
sup ( i,
j ) ∈
V 2 dist( i, j ) for dist( i, j ) =
inf (cid:110)
τ comp i + τ comp j + (cid:80) p − 1 k =
τ i k
i k +1 (cid:111) , where the inf is taken over all ( i = i 0 , . .
, i p = j ) ∈
V p
+1 that verify ∀ 1 ⩽ k ⩽ p, ( i k , i k +1 ) ∈ E .
Here dist( i, j ) is the time distance between nodes i and j , and D ( τ ) is the diameter of graph G for this distance.
D ( τ ) is the generalization of τ max Diam( G ) to the heterogeneous-delay setting.
This lower bound suggests that robustness to stragglers is possible: indeed if a fraction of the nodes or edges is too slow (large delay τ ij ), this may not even impact this lower bound, since the shortest path between two nodes may always take another route.
We aim at building decentralized algorithms with performance guarantees that enjoy such robustness to individual delay bounds.
However, since we focus on fully decentralized algorithms, our performance guarantees will not be expressed in terms of some diameter D ( τ ) as in (3) but instead in terms of some spectral characteristics of the graph at hand 1 .
We thus seek performance guarantees similar to (3) with in place of D ( τ ) the term λ 2 (∆ G ( ν )) − 1 for some parameters ν ij that depend on delay characteristics local to edge ( ij ) , where the graph Laplacian ∆ G is defined as follows.
Definition 1 (Graph Laplacian) .
Let ν = ( ν ij ) ( ij ) ∈
E be a set of non-negative real numbers.
The
Laplacian of the graph G weighted by the ν ij ’s is the matrix ∆ G ( ν ) with ( i, j ) entry equal to − ν ij if ( ij
) ∈ E , (cid:80) k ∼ i ν
ik if j = i , and 0 otherwise.
In the sequel ν ij always refers to the weights of the
Laplacian, and λ 2 (∆ G ( ν )) denotes this Laplacian’s second smallest eigenvalue.
Contributions and related works 2.1 Contributions
We consider the network averaging problem , for which we introduce Delayed Randomized Gossip in
Section 3.
Building on recent works on continuized gradient descent for Nesterov acceleration [12], we analyze Delayed Randomized Gossip in the continuized framework, that allows a continuous-time analysis of an algorithm even though the latter is based on discrete, hence practically implementable operations.
Our analysis leads to explicit stability conditions that have the appealing property of being local , i.e. they require each agent to tune its algorithm parameters to delay bounds in its graph neighborhood.
These conditions ensure a linear rate of convergence determined by λ 2 (cid:0) ∆ G ( ν ) (cid:1) , for weights of order ν = 1 / ( (cid:80) ( kl ) ∼ ( ij ) τ kl ) 2 .
This dependency of weights in the Laplacian on local delay bounds is what we call the asynchronous speedup , since it implies a scaling that is no longer proportional to τ max .
We provide extensions in Section 4 of the delayed randomized gossip we introduce next, to the more general decentralized optimization problem defined in Equation (1), and generalize our setup to capacity constraints, beyond the Poisson point process assumption, in Section 5, that both further illustrate the generality of our approach.
We also highlight a phenomenon reminiscient of Braess’ paradox in Section 6: sparsifying the graph may accelerate convergence speed. 2.2 Related works
Decentralized Optimization and Gossip Algorithms.
Gossip algorithms [5, 9] were initially introduced to compute the global average of local vectors with local pairwise communications only (no central coordinator), and were generalized to decentralized optimization.
Two types of gossip algorithms appear in the literature: synchronous ones, where all nodes communicate with each other simultaneously [9, 19, 32, 33], and randomized ones [5, 28].
A third category considers directed (nonsymmetric) communication graphs [2, 40] which are much easier to implement asynchronously.
In the synchronous framework, the communication speed is limited by the slowest node ( straggler problem), whereas the classical randomized gossip framework of [5] assumes communications to happen instantaneously, and thus does not address the question of how to deal with delays.
[2, 23, 34, 37, 39] introduce delays in the analysis of decentralized algorithms; as mentioned in the introduction, their analyses and algorithms are not robust to stragglers, relying on a single upper bound on the delays of all edges.
[38] study how sparsifying the communication graph can lead to faster decentralized algorithm.
Their approach is different from ours in Section 6: they do not consider asynchronous algorithms with physical constraints (delays and capacity), but synchronous algorithms where sequentially matchings are built in the graph.
Yet, we observe similar phenomenon as theirs in Section 6.
We refer the reader to [29] for a more complete survey of gossip algorithms.
Handling Asynchrony.
The dynamics of asynchronous optimization algorithms are significantly more complex than their synchronous counterparts.
Their study goes back to the monograph of [4], where asynchrony is modelled through a global ordering of events, providing the formalism classically used.
Most of the recent literature is then derived from a distributed asynchronous variant of SGD called HOGWILD!
[22, 25] introduce alternative orderings of the iterates (before-read and after-read) that correspond to different views of the same sequence of updates, and which simplify the analysis through the use of perturbed or virtual iterates [14, 25, 27, 35, 42], even though proofs and convergence guarantees under realistic assumptions on the intricacies between iterates, delays and choices of coordinates, are a challenging problem [7, 36].
We refer the interested reader to [1] for a more exhaustive survey of advances in asynchrony, both in shared-memory and decentralized models.
In this paper, we deal with asynchrony and delays from a different viewpoint: the analysis is inspired by time-delayed ODE systems [30], and the assumptions related to delays and asynchrony (such as
Assumption 1) do not need to be translated into discrete-time ones, as in the above references.
Finally, we believe our continuous time framework to be particularly adequate for the study and design of asynchronous algorithms, in the decentralized setting as in this paper, but also in centralized settings where it may remove the need to introduce a discrete ordering of events and thus avoid difficulties that lead to unrealistic assumptions, such as the after/before-read approaches [22].
Delayed Randomized Gossip for Network Averaging 3.1 Randomized gossip
Let G =
( V, E ) be a connected graph on the set of nodes V =
[ n ] , representing a communication network of agents.
Each agent i ∈ V is assigned a real vector x 0 ( i ) ∈ R d .
The goal of the averaging (or gossip) problem is to design an iterative procedure allowing each agent in the network to estimate the average ¯ x = 1 n (cid:80) ni
=1 x 0 ( i ) using only local communications, i.e. , communications between adjacent agents in the network.
In randomized gossip [5], time t is indexed continuously by R + .
A Poisson point process [18] (abbreviated as P.p.p.
in the sequel) P = { T k } k ⩾ 1 of intensity I > 0 on R + is generated: T 0 =
and ( T k +1 − T k ) k ⩾ 0 are i.i.d.
exponential random variables of mean 1 /I .
For positive intensities ( p ij ) ( ij )
∈
E such that (cid:80) ( ij ) ∈
E p ij
=
I , for every k ⩾ 0 , at T k an edge ( i k j k ) is activated with probability p i k j k /I , upon which adjacent nodes i k and j k communicate and perform a pairwise update.
The P.p.p. assumption implies that edges are activated independently of one another and from the past: the activation times of edge ( ij ) form a P.p.p. of intensity p ij .
To solve the gossip problem, [5] proposed the following strategy: each agent i ∈
V keeps a local estimate x t ( i ) of the average and, upon activation of edge ( i k j k ) at time T k ∈ R + , the activated nodes i k , j k average their current estimates:
[Equation]
Writing f ( x ) = (cid:80) ( [Equation] E p [Equation]
f
[Equation]
[Equation]
for f [Equation] ( x [Equation]
[Equation] x ( j ) ∥ 2 and x = ( x ( i ))
i ∈ V , [12] observe that local averages (4) correspond to stochastic gradient steps on f :
[Equation]
for step sizes K i k j k = p ikjk 2 .
Yet, this continuous-time model with P.p.p. activations implicitly assumes instantaneous communications, or some form of waiting.
Indeed, the gradient is computed on the current value of the parameter, which is x T k − .
In the presence of (heterogeneous) communication delays (Assumption 1), a more realistic update uses the parameter x S k at a previous time S k < T k , to account for the time it takes to compute and communicate the gradient.
In this case, the updates write
[Equation]
3.2 The continuized framework
Our approach uses the continuized framework [12], which amounts to consider continuous-time evolution of key quantities, with discrete jumps at the instants of Poisson point processes.
This gives the best of both continuous (for the analysis and assumptions) and discrete (for the implementation) worlds.
From now on and for the rest of the paper, we assume that Assumption 1 holds.
Edges ( ij ) ∈
E locally generate independent P.p.p.
P ij of intensity p ij > 0 (random activation times, with i.i.d.
intervals, exponentially distributed with mean 1 /p ij ).
As mentioned previously,
P = (cid:83) ( ij ) ∈
E P ij is a P.p.p. of intensity
I = (cid:80) ( ij ) ∈
E p ij , and noting P =
{ T 1 < T 2 < . . .
} , at each clock ticking T k , k ⩾ 1 , an edge ( i k j k ) is chosen with probability p i
k j k /I .
This time T k corresponds to a communication update between nodes i k and j k started at time T k − τ i
k j k 3 .
Assumptionensures that the communication started at time T k − τ ij takes some time τ ( k ) ⩽
τ i k j k and is thus completed before time T k so that the update at time T k is indeed implementable.
Consequently, the sequence ( x t ) t generated by Algorithm 1 writes as:
[Equation]
[Equation]
1: Step size K ij > 0 and intensity p ij > 0 , Initialization T 1 ( ij ) ∼ Exp
( p ij ) 2: for ℓ = 1 , 2 , . . .
do 3:
T ℓ +1 ( ij ) =
T ℓ ( ij ) + Exp( p ij ) .
4: end for 5: for ℓ = 1 , 2 , . . .
do 6:
At time T ℓ ( ij ) − τ ij for, i sends ˆ x i
= x T ℓ ( ij ) − τ ij ( i ) to j and j sends ˆ x j = x T ℓ ( ij ) − τ ij ( j ) to i .
At time T ℓ ( ij ) , x T ℓ ( ij ) ( i ) ← x T ℓ ( ij ) − ( i ) − K ij p ij (cid:0) ˆ x i − ˆ x j (cid:1) , x T ℓ ( ij ) ( j ) ← x T ℓ ( ij ) − ( j ) − K ij p ij (cid:0) ˆ x j − ˆ x i (cid:1) , (8) 8: end for Algorithm 1 is the pseudo-code for Delayed Randomized Gossip , from the viewpoint of two adjacent nodes i and j . The times T ℓ ( ij ) for ℓ ⩾ 1 denote the activation times of edge ( ij ) . They follow a
P.p.p. of intensity p ij , and are sequentially determined by adjacent nodes i and j .
Formally, this decentralized and asynchronous algorithm corresponds to a jump process solution of a delayed stochastic differential equation . Defining N (d t, ( ij )) as the Poisson measure on R + × E of intensity I d t ⊗U p where U p is the probability distribution on E proportional to ( p ij ) ( ij ) ∈
E ( U p (( ij ))
=
p ij /I ), we have:
[Equation]
3.3 Convergence guarantees
Theorem 1 (Delayed Randomized Gossip) .
Assume the following bound on K ij , ( ij ) ∈ E holds, and let γ > 0 such that: 4 :
[Equation]
where ν ij ≡ K ij , ( ij ) ∈ E , and τ max = max ( ij ) ∈
E τ ij .
For any T ⩾ 0 , for ( x t ) t ⩾ 0 generated with delayed randomized gossip (Algorithm 1) or equivalently by the delayed SDE in Equation (7) , we have:
[Equation]
An essential aspect of Theorem 1 lies in the explicit sufficient conditions for convergence it establishes for our proposed schemes, and on how they only rely on (upper bounds on) individual delays.
We now discuss the asynchronous speedup obtained by fine-tuning algorithm parameters according to delays.
For many graphs of interest such as grids, hypergrids, trees...and bounded edge parameters ν ij , in the large network limit n → ∞ one has λ 2 (∆ G ( ν ))
→ 0 5 and so λ 2 (∆ G ) ∧ 1 /τ max = λ 2 (∆ G ) .
The asynchronous speedup consists in having a rate of convergence as the eigengap of the Laplacian of the graph weighted by local communication constraints: the term λ 2 (∆ G ( K )) , where each K ij is impacted only by local quantities.
As mentioned in the introduction, this quantity should be understood as the analogue in decentralized optimization of the squared diameter of the graph (using time distances) in (3) in centrally coordinated algorithms and as expected, gossip algorithms are affected by spectral properties of the graph.
In Theorem 1, these properties reflect delay heterogeneity across the graph: here, λ 2 (∆ G ( K )) − 1 the mixing time of a random walk on the graph where jumping from node i to j takes a time ˜
τ
ij = K − 1 ij .
In contrast, previous analyses (of synchronous or asynchronous algorithms) involve the mixing time of a random walk with times between jumps set to a quantity that is linearly dependent on τ max .
We coin this discrepancy the asynchronous speedup .
Equation (9) suggests a scaling of p ij ≈ 1 /τ ij , giving local weights K ij of order 1 / (degree ij τ ij ) where degree ij is the degree of edge ( ij ) in the edge-edge graph.
On the other hand, synchronous algorithms are slowed down by the slowest node: the equivalent term would be of order λ 2 (∆ G (1 / (degree ij τ max )) .
Indeed, for a gossip matrix W ∈ R V × V ( W is a symmetric and stochastic matrix), the equivalent factor in synchronous gossip [9] is λ 2 (∆ G ( W ij τ max )) , and W ij is usually set as 1 / degree ij in order to ensure convergence.
Finally, assume that all τ ij are equal to τ max , and set p ij = 1 /τ max .
We then recover λ 2 (∆ G (1 / (degree ij τ max )) in the rate of convergence γ , thus yielding the same rates as synchronous algorithms [9] and asynchronous algorithms that only use a global upper bound on the delays [2, 23, 24, 34, 37, 39]: albeit being asynchronous, these algorithms do not take advantage of an asynchronous speedup in their convergence speed.
The same asynchronous speedup is present in the decentralized optimization problem, dealt with in Section 4, where we generalize our arguments using an augmented-graph approach [15].
This asynchronous speedup also holds under the presence of capacity constraints (Section 5), where we replace P.p.p. ’s by more general truncated P.p.p. ’
s.
3.4 A delayed ODE for mean values in gossip
Before proving Theorem 1 in Appendix D, we provide some intuition for its conditions and the resulting convergence rate.
We do this by studying the means of the iterates, that verify a delayed linear ordinary differential equation, easier to study than the process itself, for which we provide stability conditions.
Denoting y t
= E [ x t ] ∈ R n × d , for t ⩾ 0 , where ( x t ) t ⩾ 0 is generated using delayed randomized gossip updates (6), we have:
[Equation]
Indeed, for any t ⩾ 0 and d t > 0 ,
[Equation]
and the right-handside simplifies as − d t (cid:80) ( ij )
∈
E K ij ∇ f ij ( x t − τ ij )
+ o (d t ) .
Taking the mean, dividing by d t and making d t → 0 leads to the delayed ODE verified by y t =
E [ x t ] .
Such delaydifferential ODEs are classical [30] yet their stability properties are notoriously hard to characterize.
This is typically attacked by means of Lyapunov-Krasovskii functionals or Lyapunov-Razumikhin functions [13].
Alternatively, sufficient conditions for convergence and stability guarantees on ( y t ) can be obtained, under specific conditions, by enforcing stability of the original system after linearizing it with respect to delays [26].
Linearizing in the sense of [26] means making the approximation y t − τ ij =
y t − τ ij d y t d t .
Under this approximation, we have:
[Equation]
For any weights ν ij and vector z , (cid:80) ij ν ij ∇ f ij ( z ) =
∆ G ( ν ) z .
Thus the delay-linearized ODE reads
[Equation]
This delay-linearized ODE (12) provides intuition on the behavior of E [ x t ] .
Indeed, (12) is stable provided that ρ (∆ G ( { τ ij K ij } )) < 1 , in which case it has a linear rate of convergence of order λ 2 (∆ G ( { K ij } ) .
Even though this stability condition and the rate of convergence are only heuristics, since (12) is obtained through an approximation of the delayed ODE verified by E [ x t ] (11), this stability condition for the delay-linearized system implies stability of the original delayed system under assumptions on the matrices and delays involved[26], that hold in our case, leading to the following, proved in Appendix C.
Proposition 1.
Assume that the spectral radius of the weighted Laplacian ∆ G ( { τ ij K ij } ) verifies ρ (∆ G ( { τ ij K ij } )) <
Then the delayed ODE (11) is stable.
Consequently, the stability conditions (necessary conditions on step sizes K ij in Equation (9)) obtained in Theorem 1 are very natural.
Indeed, a simple way to enforce ρ (∆ G ( { τ ij K ij } ) <
1 based on local conditions consists in imposing (cid:80)
j τ ij K ij < 1 for all i .
This is a weaker condition than the one stated in Theorem 1, but it only gives stability of the means.
Furthermore, the rate of convergence of delayed randomized gossip in Theorem 1, that takes the form of the eigengap of a weighted graph
Laplacian, is also that of any solution of the delay-linearized ODE (12).
Extension to decentralized optimization
In this section we extend our results to decentralized optimization, going beyond the quadratic objective functions considered for network averaging.
4.1 Delayed Decentralized Optimization
Consider the decentralized optimization problem (1).
We make the following assumptions on the individual objective functions f i therein :
[Equation] see [6] for definitions.
Let f ( z ) :
= (cid:80) i ∈ [ n ] f i ( z ) for z ∈ R d and F ( x )
= (cid:80) i ∈ [ n ] f i ( x i ) for x = ( x 1 , · · · , x n ) ∈ R n × d where x i ∈ R d corresponds to node i ∈ [ n ] .
Definition 2 (Fenchel Conjugate) .
For any function g : R p
→ R , its Fenchel conjugate is denoted by g ∗ and defined on R p by g ∗ ( y ) =
sup x ∈ R p ⟨ x, y ⟩ − g ( x ) ∈ R ∪ { + ∞} .
Our algorithm for delayed decentralized optimization is built on delayed randomized gossip for network averaging, augmented with local computations.
Each node i ∈ V keeps two local variables: the communication variable x i ( t ) , used to run delayed randomized gossip, and a computation variable y i ( t ) , used to make local computation updates in the following way.
Local computations.
Each node i generates a Poisson point process P comp i =
{ T comp1 ( i ) <
T comp2 ( i ) , . . .
} of intensity p comp i .
At the clock tickings T comp k
( i ) , a local computation update is made corresponding to a computation started at a time T comp k ( i ) − τ comp i , where τ comp i is the upper bound on the time to perform an elementary computation at node i , introduced in
Assumption 1.
Thus by assumption the computation started at time
T comp k
( i ) − τ comp
i is completed by time T comp k
( i ) so that the update can be performed at that time.
The precise form of this update is given by Equation (17).
Communications.
In parallel of these local computations, a Delayed Randomized Gossip is run on the graph.
Dedicated P.p.p.
( P ij ) ( ij )
∈ E with respective intensities ( p ij ) ( ij ) ∈
E are associated to communication updates of all network edges, and used to perform updates as prescribed by
Equation (8) in Delayed Randomized Gossip .
The resulting Delayed Decentralized Optimization algorithm, or DDO for short, is described in
Algorithm 3 and is a combination of Algorithm 1 for communication updates along edges ( ij ) ∈ E with Algorithm 2 for local computation updates at nodes i ∈ V .
4.2 Convergence guarantees
The process ( x ( t ) , y ( t ))
∈ R 2 n × d defined by algorithm DDO , Algorithm 3, satisfies the following convergence guarantees that generalize Theorem 1 to decentralized optimization beyond the case of quadratic functions.
Theorem 2 (Delayed Decentralized Optimization) .
Under the regularity assumptions (13) , assume further that for all 1 ⩽ i ∈ V and ( ij ) ∈ E , we have:
[Equation]
Let τ max := max (cid:0) max ( ij )
∈
E τ ij , max i ∈ V τ comp i (cid:1) .
Then for γ > 0 such that
[Equation]
[Equation]
(15) the process ( x ( t ) , y ( t )) generated by DDO satisfy
[Equation]
where ¯ x ⋆ = ( x ⋆ , . . .
, x ⋆ ) ⊤ ∈ R n × d for x ⋆ minimizer of f = (cid:80) i f i .
DDO is based on a dual formulation and uses an augmented graph representation introduced in [16] to decouple computations from communications, as detailed in the proof.
The dual gradient computations in Algorithm 2 can be expensive in general; they could be avoided by using a primal-dual approach for the computation updates [21].
The convergence guarantees we obtain resemble classical ones:
Interpreting γ as the reciprocal of the time scale for convergence, we recognize in its upper bound (15) an “optimization factor” κ − 1 comp :
= σ/L , and a “communication factor” κ − 1 comm =
λ 2 (cid:0) ∆ G ( K ) (cid:1) .
Our method is non-accelerated, so the computation factor κ comp , the condition number of the optimization problem, is expected.
The communication factor captures the delay heterogeneity in the graph as in Delayed Randomized
Gossip , leading to the asynchronous speedup discussed in Section 3 after Theorem 1.
Previous approaches have considered accelerating decentralized optimization by obtaining √ κ comp instead of
κ comp and/or (cid:112) κ ′ comm instead of κ ′ comm for κ ′ comm a communication factor in the rate of convergence [15, 20, 32].
Our result yields a speedup of a different nature: we obtain a communication factor κ comm that can be arbitrarily larger than previously considered κ ′ comm for networks with huge delay heterogeneity.
, node i computes g i = ∇ ϕ ∗
i ( y i ( T comp ℓ ( i ) −
τ comp i )) (takes a time less than τ comp i ) and keeps ˆ x i
= x
i ( T comp ℓ ( i ) − τ comp i ) in memory, where ϕ i = f i − σ 4 ∥ .
∥ 2 .
At time T comp ℓ ( i ) , y i
[Equation]
comp
[Equation]
(17) 10: end for
Handling communication and computation capacity limits 5.1 Communication and computation capacity constraints
A given node or edge in the network may be able to handle only a limited number of communications or computations simultaneously.
In Delayed Randomized Gossip and DDO algorithms, such con- straints could be violated when some P.p.p.
generates many points in a short interval.
We extend our algorithms and resulting convergence guarantees to take into account these additional constraints.
In the continuized framework, this constraint can be enforced by truncating the P.p.p.
that handles activations (Definition 3).
We formalize communication and capacity constraints in Assumption 2, and show that asynchronous speedup is still achieved in this setting in Theorem 3.
In the previous sections, step size parameters K ij , K comp i of the algorithms could be tuned to counterweight the effect of delays for arbitrary intensities p ij .
With the introduction of capacity constraints we will see that the local optimizers at every node must also bound the intensities p ij , p comp i based on local quantities.
The resulting rate of convergence is the same as in Theoremsand 2, up to a constant factor of 1 / 2 .
We formalize communication and computation capacity constraints as follows.
Assumption 2 (Capacity constraints) .
For some q ij , q comm i , q comp i ∈ N ∗ ∪{∞} , i ∈ V and ( ij ) ∈ E ,
Computation Capacity: Node i can compute only q comp i gradients in an interval of time of length τ comp i ; 2.
Communication Capacity, edge-wise limitations: Only q ij messages can be exchanged simultaneously between adjacent nodes i ∼ j in an interval of time of length τ ij ; 3.
Communication Capacity, node-wise limitations: Node i can only send q comm i messages in any interval of time of length τ comm i = max [Equation]
∼ i [Equation] ij . Taking into account these constraints in the analysis boils down to replacing P.p.p. processes ( P ij ) ( ij ) ∈ E , ( P comp i ) i ∈ V of intensities ( p ij ) , ( p comp i ) in the DDO algorithm, by truncated Poisson point processes ( ˜ P ij , ˜ P comp i ) (see Definition 3).
More precisely, for every edge ( ij ) ∈ E ( resp.
node i ∈ V ), let n ij ( t ) be the number of communications occurring along ( ij ) between times t − τ ij and t ( resp.
n comm i,j the number of communications node i is involved in between times t and t − τ ij , n comp i the number of computations node i is involved with between times t and t − τ comp i ).
Without capacity constraints, these quantities are discrete Poisson random variables (of mean p ij τ ij for n ij ( t ) , e.g. ).
5.2 Convergence guarantees
As in Section 4, we consider communication and computation update rules as in Algorithm 3 ( DDO algorithm).
In the presence of capacity constraints, a communication alongside edge ( ij ) ∈
E at a clock ticking t ∈ P ij occurs and does not break the communication capacity constraints if and only if n ij ( t ) <
q ij (for edge-wise limitations), n comm i,j ( t ) <
q comm i and n comm j,i ( t ) <
q comm j (for node-wise limitations) are satisfied.
The realistic implementation of these truncated Point processes is discussed in Section 7.
Under capacity constraints, we have the following guarantees for our algorithm, defined as in
Algorithm 3 (Algorithm 1 for communications and Algorithm 2 for local computations), where communications and computations that violate the capacity constraints are dropped.
Theorem 3.
Assume for any i ∈ V and ( ij ) ∈
E :
[Equation]
where c = 1 / (1 − (cid:112) ln(6) / 2) is a numerical constant.
Then, if the assumptions of Theoremdescribed in Equation 14 additionally hold, for γ verifying
[Equation]
we have:
[Equation]
[Table]
[Equation]
The same guarantees as without the capacity constraints thus hold, up to a constant factor 1 / 2 in the rate of convergence.
The conditions on the activation intensities (18) suggest that graph sparsity is beneficial: for q comm i small, 2 (cid:80)
j ∼ i p ij τ comm i ⩽ q comm i translates into p ij scaling with the inverse of the edge-degree of ( ij ) , so large degrees thus slow down the convergence.
The new conditions (18) are easily enforced with the natural choice of intensities p ij ( resp.
p comp i ) of order 1 /τ ij ( resp. τ comp i ).
Taking q comm i = 1 , we recover the behavior of loss networks [17], where a node cannot concurrently communicate with different neighbors.
Gossip on loss networks was previously studied in [11], to obtain some form of asynchronous speedup.
Comparatively, our present algorithms are structurally simpler and their analysis in the continuized framework yields faster convergence speeds.
Braess’s Paradox and Experiments
In this section, we investigate how the local step sizes K ij and Poisson intensities p ij used in
Theorems 1, 2 and 3 should be tuned for a fixed choice of communication delays.
Consider the line graph with constant delays
τ i,i +
= τ .
Add edge (1 , n ) in order to close the line, with a delay τ 1 n = τ ′ with arbitrarily large τ ′ .
If the added Poisson intensity p 1 n satisfies τ 1 n p 1 n → ∞ , then according to Theorem 1, we have K 12 → 0 and K n − 1 ,n → 0 .
Consequently, since γ = O (∆ G ( K ) , we have γ → 0 : the weighted graph becomes close to disconnected.
By adding an edge to the graph, the convergence speed of delayed randomized gossip is degraded.
In order to alleviate the phenomenon, we would need to virtually delete the edge, by setting p 1 n = 0 .
Figure 1 illustrates this phenomenon in the more general setting: one can sparsify the communication graph by solving a regularized optimization problem over the p ij in order to maximize λ 2 (∆ G ( K ))
( K being a function of p ), leading to both faster consensus and smaller communication complexity (and thus lower energy footprint).
In road-traffic, removing one or more roads in a road network can speed up the overall traffic flow.
This phenomenon, called Braess’s paradox [10], also arises in loss networks [3].
In our problem, this translates to removing an edge ( ij ) with a non-negligible Poisson intensity p ij .
We take G 1 a dense
Erd ˝ os-Rényi random graph (Figure 1(a)) of parameters n = 30 , p = 0 .
Delays τ ij are taken equal to 0.01 with probability 0.9, and to 1 with probability 0.1.
Initially, intensities are set as p (1) ij = 1 /τ ij .
Maximizing:
[Equation]
over ( p ij ) ij , we obtain intensities p (2) and a graph G 2 (Figure 1(b)), sparser than G 1 : we delete edges that have a null intensity ( i.e. such that p (2) ij = 0 ).
We then run our delayed gossip algorithm for initialization x 0 a Dirac mass ( x 0 ( i )
=
I i = i 0 ), on G 1 (blue curves) and G 2 , for the choice of K ij as in Theorem 1.
The green curve is the synchronous gossip algorithm [9] on G 1 , to illustrate the asynchronous speedup, where each iteration takes a time τ max = 1 .
In Figure 1(c), the error to the consensus is measured as a function of the continuous time, while it is measured in terms of number of updates in Figure 1(d) and in terms of energy (defined as (cid:80) k : T k <t τ i k j k at time t : the energy consumed by a communication is assumed to be proportional to the time the communication took) in
Figure 1(e).
As expected, in terms of number of updates in the whole graph and energy spent, the sparser graph is more effective: slow and costly edges were deleted.
Perhaps more surprising, but supported by our theory (Theorem 2) and the resulting Braess’s paradox, this also holds in Figure 1(c): even though in the same amount of time, less updates are made in the sparser graph G 2 than in G 1 , delayed randomized gossip is still faster on G 2 than G 1 .
Making less updates and deleting some communications make all other communications more efficient.
We believe that this phenomenon could be exploited for efficient design of large scale networks, beyond the maximization the spectral gap regardless of physical constraints as in [41] for instance.
[Figure]
Decentralized generation of truncated Poison point processes 7.1 Practical Implementation and Limitations of Algorithm 3 ( DDO)
In order for DDO (or Delayed Randomized Gossip) to be implemented (with or without truncated
P.p.p. ), some conditions are required, leading to some limitations that we discuss below.
Communication steps at some time t are of the form − η ij ( x i ( t − τ ij ) − x j
( t − τ ij ))
(on node i ): the (delayed) iterates of local variables x i and x j for this update are required to have the exact same delay.
This comes from our dual formulation.
Secondly, it is not clear who has the initiative of communications in our algorithm, since we treat delays and communications edge-wise.
However, we propose here a practical implementation of DDO , taking into account all these limitations: nodes have the initiative of communications and delayed variables are enforced to have the same constant delays (equal to an upper-bound on these delays).
Computation Updates at node i let N comp i ( t ) be the number of computations launched at node i between times t and t − τ comp i .
Initialize P comp i as { t 0 } where t 0 is a random time of exponential law of parameter p comp
i
At any time t ∈ P comp i
Compute a random time T of exponential law of parameter p comp i , and add t + T to P comp i ; 2.
Computation: at time t , if N comp i ( t ) <
q comp i , i computes ∇ g ∗ i comp ( y comp i ( t )) and saves into memory node-variable x i ( t ) ; 3.
Update: at time t + τ comp i , the computation is finished, and the computation update (17) can be performed
Communication Updates at node i let N comm i ( t ) and N ij ( t ) for j ∼ i respectively be the number of communications started by node i between times t and t − τ comm i and the number of communications started between nodes i and j in [ t − τ ij , t ] .
Initialize P comm i as { t 0 } where t 0 is a random time of exponential law of parameter (cid:80)
j ∼ i p ij / 2 .
At any time t ∈ P comm i :
Compute a random time T of exponential law of parameter (cid:80)
j ∼ i p ij / 2 , and add t
+ T to P comm i ;
Local synchronization: at time t , if N comm i ( t ) <
q comm i , i chooses an adjacent node j with probability
p ij / (cid:80) k ∼ i p ik , and sends a ping (smallest message possible).
We assume that sending a ping takes a time upper-bounded by τ ping ij .
Upon reception of this ping , if N comm j ( t ) <
q comm j , j returns the same ping to i .
i and j are thus synchronized at time t + 2 τ ping ij .
Until the return of the feedback by node j , i assumes in its request list that this communication will happen.
Communication: if N ij ( t + 2 τ ping ij ) <
q ij , i sends x i ( t ) to j and j sends x j ( t ) to i , while each agent keeps in memory the vector sent.
For this to be possible, at a time s , node i needs to keep in memory its local values at times between s − max j ∼ i τ ping ij and s (a small number of values usually, so that is not too restrictive).
Update: at time t + 2 τ ping ij + τ ij , update (8) can thus be performed.
The induced process has the same law as the one studied and analyzed thanks to the classical property that a P.p.p.
with intensity p has its distribution unchanged after translation of all its points by some constant τ ij .
The communication/computation scheme above emphasizes the fact that quantities τ ij , τ comp i are upper bounds on the delays of local communication/computations.
The delay is here τ ij + 2 τ ping ij instead of simply τ ij yet these are of the same order since τ ping ij ⩽ τ ij . Conclusion
We introduced a novel analysis framework for the study of asynchronous algorithms in the presence of delays, establishing that an asynchronous speedup can be achieved in the network averaging problem, and in decentralized optimization.
Our results hold for explicit choices of algorithm parameters based on local network characteristics.
They derive from the continuous-time analysis and assumptions handled in our continuized framework.
The explicit conditions and convergence rates we obtain allow us to further discuss counter-intuitive effects akin to the Braess paradox, such as the possibility to speed up convergence by suppressing communication links.
A Discussion of our assumptions
Continuous time clocks Implicitly we have assumed that agents in the networks share a continuoustime clock.
We now discuss how critical this assumption is.
One may wonder if clock skews, drifts or shifts between agents could lead to unwanted behavior (instability).
First, computer clocks are synchronized via Network Time Protocol (NTP) or other more recent and more robust protocols, justifying the assumption of a common shared clock.
However, failures in these protocols could lead to slight skews, that need to be addressed.
Looking at our algorithm, it in fact appears that, using timestamps, nodes do not need to be perfectly synchronized, as long as time in each local clock passes at the same speed.
Shifts in time-synchronization thus do not appear to be the core practical difficulty: clock drifts (clocks that do not have the same frequency and where time thus passes at a different speed) are the remaining issue.
However, this can be dealt with simply by augmenting pairwise delays to take that into account.
Local upper bounds of the local delays Our rates depend on local upper bounds of the local delays, that need to be known in order to tune the coefficients.
The drawback is thus that, if a node/edge has an erratic behavior (fast but rarely slow), we only use the upper-bound on the delays, which can lead to slower convergence.
However, we believe that this is a drawback of most (if not all) asynchronous algorithms, where delays are dealt with by diminishing step sizes.
These step sizes are tuned using upper bounds on the delays.
Even if we did not artificially force the delays to be equal to the local upper bound τ ij , we would have to tune K ij based on these upper bounds, resulting in the same rates of convergence.
A natural extension would be to consider whether step sizes can be adapted to the physical delay as in [27] for asynchronous SGD, therefore obtaining an asynchronous speedup and guarantees without requiring any knowledge on the delay upper bounds.
B Delayed coordinate gradient descent in the continuized framework
Let G be a σ -strongly convex function on R D .
For k = 1 , ..., m , let E k be a subspace of R d , and assume that:
[Equation]
For x ∈ R D , let x k denote its orthogonal projection on E k and let ∇ k G := ( ∇ G ) k , and assume that the subspaces E 1 , ..., E m are orthogonal.
For k, ℓ ∈ [ m ] , we say that k and ℓ are adjacent and we write k ∼ ℓ if and only if ∇ k ∇ ℓ G is not identically constant equal to 0 .
This induces a symmetric graph structure on the coordinates k ∈ [ m ] .
In the context of gossip network averaging, m = | E | and each subspace E k corresponds to an edge e k
=
( i k j k ) of the graph; in that context, we have k ∼ ℓ if and only if edges e k and e ℓ share a node.
In the network averaging problem previously described, the function G used is g ( λ ) = 12 ∥
Aλ ∥ for λ ∈ R E × d the edge variables.
Subspaces are E ij of dimension d for ( ij ) ∈ E (and m = | E | ) corresponding to variables of λ associated to edge ( ij ) .
B.1 Algorithm and assumptions B.1.1 Continuized delayed coordinate gradient descent algorithm
For k ∈ [ m ] , let P k be a P.p.p. of intensity
p k
denoting the times at which an update can be performed on subspace E k .
For t ∈ P k let ε k ( t ) ∈ { 0 , 1 } be the indicator of whether the update is performed or not.
Let also η k be some positive step size for k ∈ [ m ] .
Consider then the following continuous-time process X ( t ) , where X k ( t ) (the projection of X ( t ) on E k ) evolves according to:
[Equation]
where P k (d t ) corresponds to a Dirac at the points of the P.p.p.
P k .
In words, ( X ( t )) t ⩾ 0 is a jump process that takes coordinate gradient descent steps along subspaces ( E k ) k ∈ [ m ] at the times of independent Poisson point processes ( P k ) k ∈ [ m ] .
We introduced variables ( ε k ( t ))
k ∈ [ m ] ,t ∈P k with values in { 0 , 1 } to represent capacity constraints: ε k ( t )
=[-
-]
if the update at time t ∈
P
k cannot be performed due to some constraint saturation; these variables ε k ( t ) will be essential in our treatment of communication and computation capacity constraints in Section 5.
B.1.2 Regularity assumptions
G is σ -strongly convex, and L k -smooth on E k for k ∈ [ m ] .
Furthermore, there exist non-negative real numbers M k,ℓ and M ℓ,k for k ∼ ℓ such that for all k = 1 , ..., m and x, y ∈ R D , we have:
[Equation]
When G is L k smooth on E k as we assume, the above condition is verified by the choice M i,j = L j , i ∼ j .
If ∇ k G is M k -Lipschitz, Condition (21) is verified by the choice M k,ℓ =
M k .
Assumption (21) however allows for more freedom, and is particularly well suited for our analysis.
In particular for decentralized optimization, it will be convenient to take M kℓ = √ L k L ℓ .
B.1.3 Assumptions on
variables ε k ( t ) , t ∈ P k
For t ∈ P k , random variable ε k ( t ) is σ (cid:0) P ℓ ∩ [ t − τ k , t ) , ℓ ∈ [ m ]) -measurable, and there exists a constant ε k > 0 such that:
[Equation]
Furthermore, we assume that ε k ( t ) is negatively correlated with each quantity N ℓ ( t − τ k , t ) =
|P ℓ ∩ [ t − τ k , t ] | , i.e. that for all k, ℓ ∈ [ m ] ,
[Equation]
In our subsequent treatment of communication and capacity contraints, we shall see that the above assumptions are verified for ϵ k ( t ) the indicator that t is a point a truncated P.p.p.
˜
P k defined as follows:
Definition 3 ( Truncated P.p.p. ) .
Let ( P k )
⩽ k ⩽ m be P.p.p. of respective intensities ( p k )
1 ⩽ k ⩽ m , ( τ k )
⩽ k ⩽ m non-negative delays.
Let N k be the Poisson point measures associated to P k , k ∈ [ m ] .
For ( C r ) 1 ⩽ r ⩽ M subsets of [ m ] , we define the truncated Poisson point measures ( ˜ N ) 1 ⩽ k ⩽ m of intensities ( p k ) {+
+}
⩽ k ⩽ m and parameters ( τ k ) k , ( q k,r ) k ∈ [ m ] ,r ∈
[ M ] as:
[Equation]
and we let ˜ P k be the point process associated to this point measure.
B.2 Convergence guarantees and analysis
The main result of this Section is the following
Theorem 4 (Delayed Coordinate Gradient Descent) .
Under the stated assumptions on regularity of
G and on variables ϵ k ( t ) , assume further that the step sizes η k are given by η k
= K
k p k L k where for all k ∈ [ m ] ,
[Equation]
and let γ ∈ R + be such that:
[Equation]
where τ max := max k ∈ [ m ] τ k .
Then for any T > 0 the solution X ( t ) to Equation (20) verifies
[Equation]
Proof.
We proceed in three steps.
The first step consists in upper bounding, for t ⩾ 0 , the quantity d E [ G ( X ( t ))]
d t .
We then introduce in Step 2 a Lyapunov function inspired by the Lyapunov-Krasovskii functional [13]), and by using the result proved in the first step, we show that it verifies a delayed ordinary differential inequality.
The last step then consists in deriving the desired result from this delayed differential inequality.
Step 1 To bound d E [ G ( X ( t ))]
d t , we study infinitesimal increments between t and t + d t for d t → 0 .
This approach is justified by results on stochastic ordinary differential equation with Poisson jumps , see [8].
For t ⩾ 0 , let F t be the filtration induced by P k ∩ [0 , t ) ,
k ∈
[ m ] i.e. , the filtration up to time t .
By convention, for non-positive t , we write X ( t ) =
X (0) .
The following inequalities are written up to o (d t ) terms, that we omit to lighten notations.
Finally, we write
[Equation]
We have, using local smoothness properties of G and the fact that for a P.p.p.
P of intensity p , P ( P ∩ [ t, t + d t ] = ∅ )
= 1 − p d t + o (d t ) and P (# P ∩ [ t, t + d t ] = 1) =
p d t + o (d t ) :
[Equation]
First, we rewrite − ε k ( t
[Equation]
and bound the second term there by
[Equation]
where we used the Cauchy-Schwarz inequality and then local Lipschitz property (21) of ∇ k G .
Writing
[Equation]
where N ℓ is the Poisson point measure associated to P ℓ
[Equation]
we have (where we use a triangle inequality for integrals):
[Equation]
For the first term, since both ε k ( t ) and N ℓ (d s ) for s in the integral are independent from X ( t − τ k ) (and thus from g k,t − τ k ) [Equation] and where we write N ℓ ( u, v ) the number of clock tickings of P ℓ in the interval [ u, v ) [Equation]
we obtain:
[Equation]
Furthermore, using our negative correlation assumption,
E [ N [Equation] − [Equation]
[Equation]
t )] ⩽ E [ N ℓ [Equation] t − τ k , t )] E [ ε k ( t )]
= p
[Equation]
k E [ ε k
[Equation]
t )] , and since ε k ( t ) and g k,t − τ k are independent, E [ [Equation]
k
[Equation]
t )] [Equation] For the second term, since the process ( ε ℓ ( s ) g ℓ,s − τ ℓ ) s is predictable (in the sense that it is independent from N u (d s ) for all u ), we have
[Equation]
Hence,
[Equation]
Combining all our elements and taking d t → 0 , we hence have:
[Equation]
Step 2 Now, introduce the following Lyapunov function:
[Equation]
that we wish to upper-bound by some constant, where γ is as in (25).
We have:
[Equation]
Integrating the bound (27) on d E [ G ( X ( t ))
d t , we obtain, using (cid:82) T 0 (cid:82) t ( t − τ )
+ h [Equation] u
[Equation]
d
[Equation]
⩽ [Equation] (cid:82) T {+
+} h ( t )d t for non-negative h :
[Equation]
where
[Equation]
Remark now that we have
[Equation]
Indeed, (28) is equivalent to
[Equation]
which follows from the assumed bounds (24) on K k and the fact that γ ⩽ 1 /τ max , assumed in (25).
we then have, using (28) and the fact that, by strong convexity, G ( X ( t )) − G ( x ⋆ ) ⩽
[Equation]
∥∇ G
[Equation]
where we used the assumption (25) that γ ⩽ σ min k ∈ [ m ] (cid:0)
K k ε k L k (cid:1) .
Step
The proof is then concluded by using the following lemma, to control solutions of this delayed ordinary differential inequality.
Lemma 1.
Let h : R → R + a differentiable function such that:
[Equation]
for some positive constants a, b, τ verifying τb < 1 . Then:
[Equation]
Proof.
Let δ ( t )
=
h ( t ) − h ( t − τ ) .
For any t ⩾ 0 , we have:
[Equation]
Let c = τa 1 − τb
(solution of x = τ ( a + bx ) ) and t
= inf { t > 0 | δ ( t ) ⩾ c } ∈ R ∪ {∞} .
Assume that t 0 is finite.
Then, by continuity, δ ( t 0 ) =
c and:
[Equation]
as for all s < t 0 , δ ( s ) < c .
This is absurd, and thus t 0 is not finite: ∀ t > 0 , δ ( t ) <
c , giving us h ( t ) ⩽ c ( t + τ ) /τ for all t ⩾ 0 .
To conclude the proof of Theorem (4), we apply Lemma 1 to h ( T ) =
L γT with a = G ( X (0))
− G ( x ⋆ ) , b = γ and τ = τ max to obtain that for all T > 0 ,
[Equation]
The result of Theorem 4 follows by dividing this inequality by (cid:82) T 0 e γt d t = e γT − 1 γ
[Equation]
where we used that for x ⩾ 0 , e x − 1 x ⩾ e x/ 2 .
C Proof of Proposition
First, we explain how the updates of randomized gossip can also be derived from coordinate gradient descent steps.
Let A ∈ R V × E be such that for all ( ij ) ∈ E , Ae ij = µ ij ( e i − e j ) for arbitrary µ ij ∈ R , where ( e ij ) ( ij ) ∈ E and ( e i ) i ∈
V are the canonical bases of R E and R V .
Then, let g ( λ ) =
∥
Aλ ∥for λ ∈ R E × d , so that the coordinate gradient ∇ ij g ( λ ) writes ∇ ij g ( λ ) = µ ij (( Aλ ) i − ( Aλ ) j ) .
Thus, provided that for some λ T k − ∈ R E × d , x T k − − ¯ x =
Aλ T k − , the local averaging defined in
Equation (4) is equivalent to x T k
− ¯ x =
Aλ T k , where
λ T k = λ
T k − −
K ikjk p ikjk µ 2 ikjk ∇ i k j k g ( λ T k − ) for
K i k
j k = p ikjk 2 .
Hence, the gossip algorithm of [5] can be viewed as a simple block-coordinate gradient descent on variables λ ∈ R E × d indexed by the edges of the graph instead of the nodes.
Proof of Proposition 1.
For A ∈ R V × E as defined in Section 3.1 for non-null weights µ ij , define the following delayed ODE:
[Equation]
For ( y t ) solution of (11), if there exists λ 0 such that Aλ 0 = y 0 , then y t =
Aλ t for all t , where λ t is solution of (29) initialized at the value λ 0 .
Then, since AA ⊤ is the Laplacian of graph G with weights µ 2 ij > 0 , A is of rank n − 1 .
For all λ , Aλ is in the orthogonal of R 1 ( 1 ∈ R V is the vector with all entries equal to 1 ), so that Im( A ) is exactly the orthogonal of R 1 .
Finally, since for ( y t ) a solution of (11), y t − ( 1 ⊤ y 0 ) 1 is also solution of (11) and takes values in the orthogonal of R 1 , it is sufficient to prove stability of (29).
To that end, we use Theorem 1 of [26].
For z ∈ R E , let D ( z ) ∈ R E
× E be the diagonal matrix with diagonal equal to z .
Let M = D
( Kµ 2 )
A ⊤ A .
Then, the delayed ODE (29) writes as:
[Equation]
and ODE that takes the same form as Equation (7) in [26], for D ← ( ij ) =
τ ij , D → ( ij ) = 0 and D ( ij ) =
τ ij ,
R = E and with our matrix M .
In order to ensure that M is symmetric and positive semi-definite, we take µ 2 ij =
K ij , to have M =
A ⊤ A .
The assumptions of Theorem 1 of [26] are verified, so that the delayed ODE (29) is table if ρ ( D ( τ ) M ) < 1 .
We then write ρ ( D ( τ ) M )
=
ρ ( D ( √ τ )
A ⊤ AD ( √ τ ) =
ρ ( AD ( √ τ )( AD ( √ τ ))
⊤ ) , and notice that AD ( √ τ )( AD ( √ τ ))
⊤ is the Laplacian of graph G with weights µ 2 ij τ ij =
K ij τ ij , concluding the proof.
D Proof of Theorem
In the proof, we use the assumed bounds τ ij on actual delays in our algorithm to ensure that communications between i and j started at a time t − τ ij induce communication updates at time t .
Our algorithms thus behave exactly as if individual communication delays coincide with these upper bounds τ ij , which allows us to analyze algorithms with constant, albeit heterogeneous delays.
In contrast an analysis in discrete time would use a global iteration counter, and discrete-time delays would not be constant,making the analysis either much more involved or unable to capture the asynchronous speedup described above.
Proof.
Theorem 1 is obtained by applying a general result on delayed coordinate descent in the continuized framework that we detail in Section B.
Specifically, we consider the function:
[Equation]
for some A ∈ R V × E such that Ae ij = µ ij ( e i − e j ) for all ( ij ) ∈ E , where we let µ ij = − µ ji by convention.
As in Section 3.4, there exists λ ∈ R E ×
d such that x 0 − ¯ x = Aλ .
Let ( λ t ) t ⩾ 0 be defined with λ [-
-]0 = λ , and the delayed coordinate gradient steps at the clock tickings of the P.p.p. ’s:
[Equation]
For all t ⩾ 0 , we then have x t =
¯ x + Aλ t , where we recall that the process ( x t ) follows the delayed randomized gossip updates (8) of Algorithm 1.
Then, for all t ⩾ 0 , we have g ( λ t ) = 1 2 ∥
Aλ t ∥ 2 = 12 ∥ x t − ¯ x ∥ 2 .
The result of Theorem 1 follows from a control of E [ g ( λ t )]
that is a direct consequence of Theoremin next section with the specific choices m = | E | and coordinate blocks corresponding to edges.
The assumptions of Theorem 4 are verified with L ij = 2 µ 2 ij , M ( ij ) , ( kl ) =
(cid:112) L ij L kl , and strong convexity parameter λ 2 (∆ G ( ν ij = µ 2 ij )) for the specific choice µ 2 ij =
K ij , as is shown in Lemmas ??
in the Appendix, giving us exactly Theorem 1.
Corollary 1.
Under the same assumptions as Theorem 1, for ( x t ) t ⩾ 0 generated with delayed randomized gossip, define (˜ x t ) t ⩾ 0 as the exponentially weighted averaging along the trajectory of ( x t ) :
[Equation]
Then, for all T ⩾ 0 ,
[Equation]
E Proof of Theorem
Proof.
Following the augmented graph approach [16], for each “physical” node i ∈ V , we associate a “virtual” node i comp , corresponding to the computational unit of node i .
We then consider the augmented graph G + =
( V + , E + ) , where V + = V ∪ V comp (for V comp =
{ i comp , i ∈ V } ) and
E + = E ∪ E comp (for E comp = { ( ii comp ) , i ∈ V } ).
For i ∈ V , function f
i is then split (using σ -strong convexity) into a sum of two σ/ 2 -strongly convex functions: f i = ϕ
i + ϕ
i comp where ϕ i comp ( x i ) =
f i ( x i ) − σ 4 ∥ x i ∥ 2 and ϕ i ( x i ) =
ϕ comm ( x i )
=
σ 4 ∥ x i ∥ 2 .
The optimization objective (1) min x 1 = ...
= x n n n (cid:88) i =
[Equation]
f i ( x i ) , x = ( x 1 , . . . , x n ) ∈ R V × d can then be rewritten as
[Equation]
ϕ
[Equation]
x
[Equation]
, under the constraint x i = x j for ( ij ) ∈
E
+ .
This constraint can then be rewritten as A ⊤ x = 0 for
A ∈ R E + × V + such that for all ( ij ) ∈ E + , Ae ij = µ ij ( e i − e j ) , as was done for network averaging, considering the augmented graph instead of the original graph.
Using Lagrangian duality, denoting
F ∗ A ( λ ) :
= F ∗ ( Aλ ) for λ ∈ R E + × d where F ∗ is the Fenchel conjugate of F , we have:
[Equation]
Thus F ∗ A ( λ ) is to be minimized over the dual variable λ ∈ R E + × d .
The rest of the proof is divided in two steps: in the first, we derive the updates of the DDO algorithm from coordinate gradient descent steps on dual variables, and in the second step we apply Theorem 4 to prove rates of convergence for these coordinate gradient descent steps on function F ∗ A .
The partial derivative of F ∗ A with respect to coordinate ( ij ) ∈
E + of λ ∈ R E + × d reads:
[Equation]
Consider then the following step of coordinate gradient descent for F ∗ A on coordinate ( i
k j k ) ∈
E + of λ , performed when edge ( i k j k ) is activated at iteration k (corresponding to time t k ):
[Equation]
corresponding to an instantiation of delayed coordinate gradient descent in the continuized framework, on function F ∗ A , for P.p.p. of intensities ( p ij ) for ( ij ) ∈ E and p comp i for ( ii comp )
∈ E comp .
Denoting v t =
Aλ t ∈ R V + × d for t ⩾ 0 , we obtain the following formula for updating coordinates i k , j k of v when i k j k activated, irrespectively of the choice of µ ij in matrix A :
[Equation]
Such updates can be performed locally at nodes i and j after communication between the two nodes (if ( ij ) is a ‘physical edge’), or locally (if ( ij ) is ‘virtual edge’).
We refer in the sequel to this scheme as the Coordinate Descent Method.
While λ ∈ R E × d is a dual variable defined on the edges, v ∈ R n × d is also a dual variable, but defined on the nodes.
The primal surrogate of v is defined as x = ∇ F ∗ ( v )
i.e. x i = ∇ f ∗ i ( v i ) at node i .
It can hence be computed with local updates on v .
The decentralized updates of Algorithm 3 (computational updates in Algorithm 2, communication updates in Algorithm 1) are then direct consequences of Equation (31).
The last step of the proof consists in applying Theorem 4 in order to obtain Theorem 2.
The function
F ∗ A we introduced satisfies the assumptions of Theorem 4 with coordinate blocks corresponding to edges E + : The regularity assumptions are satisfied with smoothness parameter L ij = 8 µ 2 ij σ − 1 and local Lipschitz coefficients M ( ij ) , ( kl ) =
(cid:112) L ij L kl for any ( ij ) , ( kl
) ∈ E
+ , as shown in Lemmas ??
and ??
in the Appendix.
F ∗ A is moreover σ -strongly convex 6 with σ derived using Lemmas ??
and ??
, and the weights associated to matrix A are chosen so that µ 2 ij = ε ij K ij σ 2 µ 2 ij .
Finally, the output of the algorithm at node i is the primal surrogate of variable x
i ( t ) (associated to ϕ i ), which is equal to ∇ ϕ i ( x i ( t ))
=
σ 2 x i ( t ) .
F Proof of Theorem
Proof.
The algorithm under capacity constraints is obtained by applying coordinate gradient descent in the continuized framework to the same dual problem as in Section 4, but with random variables “ ε k ( t ) ” that are not taken constant equal to 1.
Here, for ( ij ) ∈ E and t ∈ P ij , we have
[Equation]
while for i ∈ V and t ∈ P comp i , [Equation] [Equation]
We apply Theorem 4 as in the proof of Theorem 2, leading to the same stability conditions on the step sizes K ij , K comp i , while the rate of convergence is multiplied by a lower bound ε on all E [ ε ij ( t )] and E [ ε ii comp ( t )] .
Let us finally compute such a lower bound ε .
For ( ij ) ∈ E , n ij ( t ) is stochastically dominated by Z ij a Poisson random variable of parameter p ij τ ij , while n comm i ( t ) and n comm j ( t ) are respectively dominated by Z i and Z j , Poisson random variables of parameters τ ij (cid:80) k ∼ i p ki and τ ij (cid:80)
ℓ ∼ j p ℓj , so that:
[Equation]
We now prove that P ( Z ij ⩾ q ij ) , P ( Z i ⩾ q comm i ) , P ( Z j ⩾ q comm j ) are all inferior to 1 / 6 .
For
P ( Z ij ⩾ q ij ) , using that for Z a Poisson variable of parameter µ and x ⩾ 0 ,
[Equation]
we have P ( Z ij ⩾ q ij ) ⩽ e [Equation]
⩽ e − 2(1 − 1 /c ) 2 if q ij ⩾ 2 , and this quantity is equal to 1 / 6 , by definition of c .
Then, if q ij = 1 , using P ( Z µ ⩾ 1) = 1 − e − µ , we have that P ( Z ij ⩾ q ij ) ⩽ p ij τ ij ⩽ 1 /c ⩽ 1 / 6 .
We proceed in the same way for P ( Z i ⩾ q comm i ) , P ( Z j ⩾ q comm j ) .
Hence, E [ ε ij ( t )] ⩾ 1 / 2 under our assumptions on the Poisson intensities.
Similarly, we prove that E [ ε ii comp ( t )] ⩾ 1 / 2 , and this concludes the proof.