Optimal Complexity in Non-Convex Decentralized Learning over Time-Varying Networks
Abstract
Decentralized optimization with time-varying networks is an emerging paradigm in machine learning.
It saves remarkable communication overhead in large-scale deep training and is more robust in wireless scenarios especially when nodes are moving.
Federated learning can also be regarded as decentralized optimization with time-varying communication patterns alternating between global averaging and local updates.
While numerous studies exist to clarify its theoretical limits and develop efficient algorithms, it remains unclear what the optimal complexity is for non-convex decentralized stochastic optimization over time-varying networks.
The main difficulties lie in how to gauge the effectiveness when transmitting messages between two nodes via time-varying communications, and how to establish the lower bound when the network size is fixed (which is a prerequisite in stochastic optimization).
This paper resolves these challenges and establish the first lower bound complexity.
We also develop a new decentralized algorithm to nearly attain the lower bound, showing the tightness of the lower bound and the optimality of our algorithm.
1. Introduction
Decentralized optimization.
Decentralized optimization is an emerging learning paradigm in which each node only communicates with its immediate neighbors per iteration.
By avoiding the central server and maintaining a more balanced communication between each pair of connected nodes, decentralized approaches can significantly speedup the training process of large-scale machine learning models [4, 10, 43].
Although decentralized optimization has been extensively studied in literature, its performance limits with time-varying communication patterns has not been fully explored.
This paper provides a better understanding in optimal complexity for non-convex decentralized stochastic optimization over time-varying communication networks
Time-varying communication pattern.
Decentralized optimization over time-varying communication networks is ubiquitous in applications.
In large-scale deep neural network training, sparse and time-varying network topologies such as one-peer exponential graph [4, 42] and EquiRand [32] endow decentralized learning with a state-of-the-art balance between communication efficiency and convergence rate.
In wireless signal processing, time-varying topologies naturally emerge when the nodes (such as cellphones, drones, robots, etc.) are moving [37, 38].
Federated learning [22, 33] can also be regarded as a special decentralized learning paradigm which admits a time-varying communication pattern alternating between global averaging and local updates.
Prior results in theoretical limits.
A series of pioneering works have attempted to establish the optimal complexity in decentralized optimization over static communication networks.
In the deterministic regime, [30, 31, 34] clarified the theoretical limits and proposed algorithms to (nearly) attain these limits.
In the stochastic regime, recent works [21, 47] have established the optimal complexity in the non-convex setting.
However, there are few studies on theoretical limits in decentralized optimization over time-varying communication networks.
A recent useful work [14] establishes the optimal complexity over time-varying networks for deterministic and strongly-convex problems.
While this bound is inspiring, its analysis, as well as all existing results in literature to our knowledge, cannot be easily extended to the stochastic setting due to challenges below.
Challenges.
When considering a static network topology, it is known that the optimal complexity in decentralized optimization is typically proportional to diameter D of the static topology [30].
Clarifying how the diameter D affects the algorithmic convergence is the key to justifying the influence of the communication network on the optimal convergence rate.
However, it is unclear in literature how to gauge, or even define, the graph diameter for a sequence of time-varying networks.
Furthermore, this paper considers decentralized stochastic optimization where the network size n is a fixed constant .
A fixed n is a prerequisite in distributed stochastic optimization which enables distributed algorithms to achieve the linear speedup in convergence rate O ( σ/ √ nT ) where σ indicates the gradient noise and T is the algorithmic iteration.
In decentralized deterministic optimization, however, size n does not appear in the convergence rate.
Thus, it does not need to be fixed and can be varied freely to simplify the lower-bound analysis.
In fact, references [14, 30, 31] all tune n delicately to derive the optimal complexity for decentralized deterministic optimization over static or time-varying networks.
Therefore, the analysis in [14, 30, 31] cannot be extended to decentralized stochastic setting in which the network size n is fixed.
Main results.
This paper overcomes the above two challenges and successfully establishes the optimal complexity for decentralized stochastic optimization over time-varying network topologies.
• Inspired by the graph diameter of a static network topology, we introduce a novel effective graph diameter to gauge how efficient a message is transmitted between two farthest nodes via a sequence of time-varying decentralized communications.
• We provide the first lower bound complexity for decentralized non-convex stochastic optimization over time-varying networks.
The derivation of this lower bound is based on a novel family of sun-shaped network topologies .
Given any fixed network size n , we can always construct a sequence of time-varying sun-shaped topologies that maintains the optimal relation between the effective graph diameter and the network connectivity.
• We prove that the established lower bound complexity can be nearly attained (up to logarithmic factors) by integrating multiple gossip communications [19, 28, 45] and gradient accumulation [21, 27, 30] to the vanilla stochastic gradient tracking approach [8, 20, 24, 26, 41].
It implies that our complexity bound is tight and the proposed algorithm is nearly optimal.
All established results in this paper as well as those of existing state-of-the-art decentralized learning algorithms over time-varying networks are listed in Table 1. [Table]
Other related works.
Decentralized optimization can be tracked back to [36].
Decentralized gradient descent [16, 25, 44], diffusion [7, 29] and dual averaging [9] are early popular decentralized methods.
Other advanced variants extend decentralized methods to data-heterogeneous scenarios [1, 13, 20, 35, 40], adaptive momentum settings [18, 23, 46], or asynchronous implementations [17].
When the network topology is time-varying, reference [14] establishes optimal convergence rate under the deterministic and strongly-convex setting.
References [14, 15] develop decentralized methods with Nesterov acceleration to nearly achieve such optimal convergence rate.
In the stochastic and non-convex setting, the convergence rate of decentralized SGD over general time-varying networks is clarified in [12].
Other references [32, 39, 42] study specific sparse and time-varying network topologies that can further save communication overheads in decentralized SGD.
However, none of these works provides the optimal complexity for non-convex decentralized learning over time-varying networks.
2. Problem setup
Problem setup.
Consider the following problem with a network of n computing nodes:
[Equation]
where
[Equation]
Function f i ( x ) is local to node i , and random variable ξ i denotes the local data that follows distribution D i .
Each local data distribution D i can be different across all nodes.
Assumptions.
The optimal convergence rate is established under the following assumptions.
• Function class.
We let the function class F ∆ L denote the set of all functions satisfying the following assumption for any dimension d ∈ N + and initialization point x (0) ∈ R d .
Assumption 1 ( C OST FUNCTIONS )
We assume each f i has L -Lipschitz gradient, i.e. ,
[Equation]
for all i ∈ [ n ] ,
x
y ∈ R
d ,
and f
x
− inf x ∈ R d f ( x ) ≤ ∆ with f = 1 n (cid:80) ni =1 f i .
• Gradient oracle class.
We assume each worker i has access to its local gradient ∇ f i ( x ) via a stochastic gradient oracle O
i ( x ; ζ i ) subject to independent randomness ζ i , e.g. , the minibatch sampling ζ i ≜ ξ i ∼ D i .
We further assume that the output O i ( x, ζ i ) is an unbiased estimator of the full-batch gradient ∇ f i ( x ) with a bounded variance
Formally, we let the stochastic gradient oracle class O σ 2 denote the set of all oracles
O i satisfying Assumption 2.
Assumption 2 ( G RADIENT STOCHASTICITY )
We assume local gradient oracle O i satisfies
[Equation]
and
[Equation]
for any x ∈ R d and i ∈ [ n ] .
• Decentralized communication.
Let V =
[ n ] denote the set of n computing nodes.
For any communication round t ≥ 0 , we assume nodes are connected through a time-varying communication network represented by a graph G t
=
( V, E t ) , where E t ⊆ {
( j, i ) ∈
V × V : i ̸ =
j } is the set of links activated at round t .
If a directed link ( j, i ) ∈
E t , then node j can transmit information to node i at round t .
In decentralized communication protocols, each node i can only receive messages with its immediate neighbors via links in E t .
• Weight matrix class.
To characterize the decentralized communication in algorithm development, we associate each time-varying communication graph G t with a weight matrix W t (also known as the gossip matrix [25, 44]).
As in [14, 21, 47], we consider a sequence of time-varying weight matrices { W t } ∞ t =0 ⊆
R n × n satisfying Assumption 3.
Assumption 3 ( W EIGHT MATRIX )
For any t ≥ 0 , W t =
[ w ti,j ] ni,j =1 satisfies
if ( j, i ) / ∈ E t
and i ̸
= j , then w ti,j = 0 ; 2.
W t 1 n = 1 n and 1 ⊤ n W t = 1 ⊤ n where 1 n = [1 , . .
, 1] ⊤ ∈ R n ; 3.
there exists a fixed constant β ∈ [0 , 1) such that ∥ W t − 1 n 1 ⊤ n /n ∥ 2 ≤ β .
Note that a weight matrix W t satisfying Assumption 3 is not necessarily symmetric or positive semi-definite.
The constant β is the connectivity measure that gauges how well the network topology G t is connected.
Constant β → 0 (which implies W t → 1 n 1 n 1 ⊤ n ) indicates a wellconnected topology while β → 1 (which implies W t → I ) indicates a poor connection.
We let W n,β denote the class of all weight matrices W t ∈ R n × n satisfying Assumption 3.
• Algorithm class.
We consider an algorithm A in which each node i assesses an unknown local function f i via the independent stochastic gradient oracle O i ( x ; ζ i ) ∈
O σ 2 .
Each node i running algorithm
A will maintain a local model copy x ( t ) i at round t .
We assume A to follow the partial averaging policy,
i
e. , each node communicates at round t via protocol
[Equation]
with
some W t =
[ w
ti,j
] n
i,j = ∈ W n β where y and z are the input and output of the communication protocol.
In addition, we assume A to follow the zero respecting policy
Informally speaking, the zero -respecting policy requires that the number of non-zero
entries
of local model copy x ( t ) i can only be increased by either sampling its own stochastic gradient oracle or interacting with the neighboring nodes.
We let A { W t } ∞ t =0 be the set of all algorithms following the partial averaging and zero-respecting policies.
With the above classes, this paper will clarify the following question: Given loss functions { f i } ni
=1 ⊆ F ∆ L , stochastic gradient oracles { O i } ni =1 ⊆
O σ 2 , a sequences of time-varying networks { G t } ∞ t =0 and its associated weight matrices { W t } ∞ t =0 ⊆
W n,β , what is the optimal complexity to solve problem (1) , and what decentralized algorithm
A ∈
A { W t } ∞ t =0 can achieve it?
Notations.
We let [ n ] := { 1 , 2 , · · · , n } .
For any network G =
([ n ] , E ) and node i ∈ [ n ] , we let
N G ( i ) denote { j : ( j, i )
∈
E or j = i } , i.e. , the neighborhood set of node i in network G .
Similarly, for a subset of nodes I ⊆
[ n ] , we use N G ( I ) to denote its neighborhood set ∪ i ∈I N G
( i ) .
3. Sun-shaped graphs and effective distance/diameter
As we have discussed in the Challenge paragraph in Section 1, it is unknown in literature (1) how to gauge the graph diameter for a sequence of time-varying network topologies, and (2) how to develop time-varying network topologies that can maintain the optimal relation between graph diameter and the network connectivity when the network size n is fixed.
This section will resolve these two challenges by introducing a novel family of sun-shaped time-varying graphs.
Definition 1 ( S UN - SHAPED GRAPH )
Given any positive integers n ≥ 2 and C ⊆
[ n ] , the sunshaped graph over nodes [ n ] with center set C , denoted by S n, C , is an undirected graph in which the neighborhood N S n, C
( i ) of node i ∈ [ n ] is given by
[Equation]
i ∈ C
[Equation]
C ∪ { i } otherwise .
The center set C in S n, C constitutes a complete subgraph.
Nodes in the complete set [ n ] \C are connected to each node in C , but there is no connection between any pair of nodes in [ n ] \C .
Note that a sun-shaped graph S n, C with |C| = 1 corresponds to a star graph while |C| = n or |C| = n −corresponds to a complete graph.
S n, C can be regarded as an intermediate state between the star and complete graphs when 2 ≤ |C| ≤ n − 2 , see the illustration in Figure 1.
We next introduce effective graph diameter to gauge how efficient a message is transmitted between two farthest nodes via a sequence of time-varying decentralized communications.
Definition 2 ( E FFECTIVE DISTANCE / DIAMETER )
We define the effective distance dist { G t } ∞ t =
( i, j ) between two nodes i ̸
=
j over a sequence of networks { G t } ∞ t =0 to be the smallest number of rounds with which a message sent from node i or j at some round t can be received by the other one via decentralized communications ( i.e. , communicating over { G t ′ } ∞ t ′ = t ).
Formally, we define dist { G t } ∞ t [Equation] [Equation] i
[Equation]
some
[Equation]
Similarly, we define the effective distance between two disjoint subsets of nodes I 1 , I 2 ⊊ [ n ] as
[Equation]
We define the effective diameter to be the largest effective distance between any two nodes, i.e. ,
[Equation]
The definitions of effective distance and effective diameter are specific to the time-varying networks.
We remark that when the networks are static, i.e. , G t = G for any t ≥ 0 , then the effective distance/diameter reduces to the canonical distance/diameter in a static graph.
The following fundamental theorem establishes the relation between the effective distance with respect to a sequence of sun-shaped graphs and the connectivity measure β .
Theorem 3 Given a fixed n ≥ 2 , two disjoint subsets of nodes I 1 , I 2 ⊊ [ n ] , and any β ∈ [0 , 1 − 1 n ] , there exists a sequence of sun-shaped graphs {S n, C t } ∞ t =
such that
(1) the graph S n, C t at round t has an associated weight matrix W t ∈ W n,β , i.e. , W t ∈ R n × n , 1 ⊤ n W t = 1 ⊤ n , W t 1 n = 1 n , and ∥ W t − 1 n 1 n 1 ⊤ n ∥ 2 ≤ β ; (2) the effective distance between I
and I 2 satisfies[-dist {S n, C t } ∞ t =
-]
[Equation]
In particular, if 1 − ( |I 1 | + |I 2 | ) /n
= Ω(1) , then dist {S n, C t } ∞ t =
( I 1 , I 2 ) =
Θ((1 − β ) − 1 ) .
4. Lower Bound
With the help of Theorem 3, we are ready to establish the lower bound for non-convex decentralized stochastic optimization over time-varying networks.
All proof details are in Appendix B.
Theorem 4 For any L > 0 , n ≥ 2 , β ∈ [0 , 1 − 1 n ] , and σ > 0 , there exists a set of loss functions { f i } ni
=1 ⊆
F ∆ L , a set of stochastic gradient oracles { O i } ni =1 ⊆
O σ 2 , and a sequence of weight matrices { W t } ∞ t =0 ⊆
W n,β resulted from the sun-shaped graphs, such that it holds for the output ˆ x of any A ∈ A { W t } ∞ t =0 starting form x (0)
that
[Equation]
Input: Initialize x (0) i
= x (0) and h (0) i = 1 nR (cid:80)
ni
=1 (cid:80) R − 1 r =
O i ( x ( k +1) ; ζ ( k +1 ,r ) i ) for any i ∈ [ n ] ; initialize x (0) =
[ x (0)1 , · · · , x (0) n ] ⊤ , h (0) =
[ h (0)1 , · · · , h (0) n ] ⊤ , and ˜ g (0) =
h (0) ; the decentralized gossip communication rounds R for k = 0 , · · · , K − 1 do
Update x ( k +1)
=
Multi-Consensus ( x ( k ) − γ h ( k ) , 2 kR, (2 k + 1) R )
Query stochastic gradients ˜ g ( k +1) i
= 1 R (cid:80) R − 1 r =
O i ( x ( k +1) i ; ζ ( k +1 ,r ) i ) at each node i
Update h ( k +1) =
Multi-Consensus ( h ( k ) + ˜ g
( k +1)
− ˜ g ( k ) , (2 k + 1) R, (2 k + 2) R ) end for
Algorithm
Input: Variable z ( t 1 ) ; index t 1 and tfor t = t 1 , · · · , t 1 do
Update z ( t +1) =
W t z ( t ) end for return Variable z ( t 2 )
Remark 5 While the lower bound is established for β ∈ [0 , 1 − 1 /n ] ⊂ [0 , 1) , it approaches to [0 , 1) as n goes large.
Such interval is broad enough to cover most weight matrices (generated through the Laplacian rule W =
I − L/d max ) resulted from common topologies such as grid, torus, hypercube, exponential graph, complete graph, Erdos-Renyi graph, geometric random graph, etc.
whose β lies in the interval [0 , 1 − 1 /n ] when n is sufficiently large.
5. Upper Bound
This section presents a decentralized algorithm that achieves the lower bound established in Theorem 4 up to logarithmic factors.
The new algorithm is a direct extension of the vanilla decentralized stochastic gradient tracking (DSGT) [20, 40].
Inspired by the algorithm development in [14, 21], we add two additional components to DSGT: gradient accumulation and multiple-consensus communication.
The main recursions are listed in Algorithm 1 which utilizes the fast gossip average step [19] in Algorithm 2.
We call the new algorithm as MC-DSGT where “MC” indicates “multiple consensus”.
All proofs are in Appendix C. Since each node takes R gradient queries and R gossip communications at round k , it holds that T
= KR when MC-DSGT finishes after K rounds.
The following theorems clarify the convergence rate of MC-DSGT where T = KR .
Theorem
Given L > 0 , n ≥ 1 , β ∈ [0 , 1) , σ > 0 , by choosing the learning rate γ as in (40) , the convergence of Algorithm 1 can be bounded for any { f i } ni =1 ⊆ F ∆ L and any { W } ∞ t =0 ⊆
W n,β that
[Equation]
+ [Equation] [Figure] where ρ ≜ β R ∈ [0 , 1) , ¯ x ( k ) =
1 n (cid:80) ni
=1 x ( k ) i , and T = KR is the total number of gradient queries and gossip communications at each node.
If we further set R as in (41) , then the rate becomes
[Equation]
The rate (3) matches with the lower bound (2) up to logarithm factors.
Therefore, our established lower bound is tight and hence optimal.
The comparison between MC-DSGT with other state-ofthe-art algorithms for non-convex decentralized stochastic optimization is listed in Table 1.
6. Experiments
We consider the logistic regression with a non-convex regularization term [2, 40].
The problem formulation is given by min x 1 n (cid:80) ni
=
f i ( x ) + ρr ( x ) where
[Equation]
[ x ] k denotes the k -the entry of x ∈ R d , { ( h i,j , y i,j ) } mj
=1 is the local dataset at node i where h i,j ∈
R d , y i,j ∈ {± 1 } is a feature vector and label, respectively.
The regularization r ( x ) is a smooth but non-convex function and ρ > 0 is the regularization weight.
We consider two real datasets: MNIST and COVTYPE.binary.
We binarize MNIST dataset by considering datapoints with labels 2 and 4 .
The regularization weight ρ is chosen as 0 .
2 (MNIST) and 0 .
015 (COVTYPE.binary).
We partition the two datasets non-uniformly such that a half of the nodes contain 80% positive datapoints while the other half hold 80% negative datapoints.
We compare decentralized stochastic gradient descent (DSGD)
[12], decentralized stochastic gradient tracking (DSGT) [40] and Algorithm 1 (MC-DSGT) with random time-varying sun-shaped graphs with ( n, |C| ) equal to (16 , 1) for MNIST and (32 , 4) for COVTYPE.binary.
We set R = 2 and 4 in
MC-DSGT for MNIST and COVTYPE.binary, repspectively
The performance of algorithms over MNIST and COVTYPE.binary is illustrated in the left and right plot in Figure 2, respectively.
The error metric is taken as ∥∇ f
(¯ x ) ∥ 2 with ¯ x = 1 n (cid:80) ni
=1 x ( k ) i .
In both experiments, we find the convergence rate as well as the robustness to time-varying network topology of MC-DSGT outperforms DSGD and DSGT, which coincides with our theory.
7. Conclusion
This paper provides the first optimal complexity for non-convex decentralized stochastic optimization over time-varying networks.
We also generalize DSGT with multiple consensus under timevarying networks to match the optimal bound up to logarithm factors.
Future works include establishing the optimal rate for (strongly) convex stochastic scenarios over time-varying networks.
Appendix A. Sun-shaped Graph
Proof [Proof of Theorem 3] It is easy to see that when |I 1 | + |I 2 | = n , dist {S n, C t } ∞ t =
( I 1 , I 2 ) = 1 for any graphs {S n, C t } ∞ t =0 .
Thus in this case, we can simply let C t =
[ n ] and W t = βI n +(1 − β ) 1 n 1 ⊤ n for any t ≥ 0 .
It is easy to see that W t ∈ W n,β .
Next we consider |I 1 | + |I 2 | < n .
Let k = ⌈ n (1 − β ) ⌉ ∈ [1 , n ] .
Case 1.
If k = n , i.e. , 0 ≤ β < 1 n , then we again let C t =
[ n ] with associate weight matrix
W t = βI n + (1 − β ) 1 n 1 ⊤ n for all t ≥ 0 .
It is easy to see that
[Equation]
where the last identity is because 0 ≤ 1 − ( |I 1 | + |I 2 | ) /n ≤ 1 and (1 − β ) − 1 = Θ(1) .
Case 2.
If 1 ≤ k ≤ n − 1 , then 1 n ≤ β ≤ 1 − 1 n .
Let J 0 , . .
, J p − 1 with p =
⌊ ( n −|I 1 |−|I 2 | ) /k
⌋ be disjoint subsets of [ n ] \ ( I 1 ∪ I 2 ) such that each J q ( 0 ≤ q ≤ p − 1 ) exactly contains k nodes.
Such {J q } p − 1 q =0 always exists due to p × k ≤ n − |I
| − |I 2 | .
Now let C t =
J t mod p , i.e. , {S n, C t } ∞ t =0 = {S n, J 0 , . . . , S n, J p − 1 , S n, J 0 . . . , S n, J p − 1 , S n, J 0 , . . . } .
It is easy to see that for any center set C with |C| =
k , the Laplacian L ( S n, C ) of graph S n, C has eigenvalues:
[Equation]
We thus let the associated weight matrices to be W t
= I n − δn L ( S n, C t ) with δ = n (1 − β ) / ⌈ n (1 − β ) ⌉ ∈ (0 , 1] for any t ≥ 0 .
Since δ < 1 , { W t } ∞ t =0 are positive semi-definite.
Therefore, we have
[Equation]
The rest is to verify dist {+{S n, C t } ∞ t =
 ( I 1 , I 2 ) = Θ+}
[Equation]
| [Equation]
[Equation]
+
[Equation]
By the construction of sunshaped graphs, starting from any round t , the neighborhood of I 1 (or I 2 ) satisfies N S n, C t
[Equation]
n
[Equation]
Therefore, we conclude that
[Equation]
On one hand, we easily see
[Equation]
On the other hand, since n (1 − β ) ≥ 1 , we have ⌈ n (1 − β ) ⌉ ≤ 2 n (1 − β ) and further (cid:22) n
[Equation]
[Equation]
where the last step is due to ⌊ x ⌋ + 1 ≥ ( x + 1) / 2 for any x ≥ 0 .
Combining (6) and (7) with (5), we reach dist {S n, C t } ∞ t =
( I 1 , I 2 ) =
Θ (cid:16) 1 − ( |I 1 | + |I 2 | ) /n
1 − β + 1 (cid:17) .
Appendix B. Lower Bound B
1. Proof of Theorem
Without loss of generality, we assume algorithms to start from x (0)
= 0 .
We denote the j -th coordinate of a vector x ∈ R d by [ x ] j for j = 1 , . . .
, d , and let prog( x ) be prog( x ) :
= (cid:40)if x = 0; max 1 ≤ j ≤ d { j : [ x ] j ̸ = 0 } otherwise .
Similarly, for a set of points X = { x 1 , x 2 , . . .
} , we define prog( X ) :
= max x ∈X prog( x ) .
As described in [5, 6], a zero chain function f satisfies prog( ∇ f ( x )) ≤ prog( x ) + 1 , ∀ x [Equation] R d
[Equation]
which implies that, starting from x = 0 , a single gradient evaluation can only make at most one more coordinate for the model parameter x be non-zero.
We prove the two terms of the lower bound in Theorem 4 separately by constructing two hardto-optimize instances.
We first state some key zero-chain functions that will be used to facilitate the analysis.
Lemma 7 (Lemma 2 of [3]) Let [ x ] j denote the j -th coordinate of a vector x ∈ R d , and define function
[Equation]
where for ∀ z ∈ R ,
[Equation]
Then h satisfy the following properties:
h ( x ) − inf x h ( x ) ≤ δ 0 d , ∀ x ∈ R d with δ 0 = 12 ; 2.
h is ℓ 0 -smooth with ℓ 0 = 152 ; 3.
∥∇ h ( x ) ∥ ∞ ≤ g ∞ , ∀ x ∈ R
d with g ∞ = 23 ; 4.
∥∇ h ( x ) ∥ ∞ ≥ 1 for any x ∈ R d with [ x ] d = 0 .
Lemma 8 (Lemma 4 of [11]) Let functions
h 1 ( x ) := − 2 ψ (1) ϕ
([ x ] 1 ) +(cid:88) j even, 0 <j<d (cid:16) ψ ( − [ x ] j ) ϕ ( − [ x ] j +1 ) − ψ ([ x ] j ) ϕ ([ x ] j +1 ) (cid:17) and h 2 ( x ) :=(cid:88) j odd, 0 <j<d (cid:16) ψ ( − [ x ] j ) ϕ ( − [ x ] j +1 ) − ψ ([ x ] j ) ϕ
([ x ] j +1 ) (cid:17) .
Then h 1 and h 2 satisfy the following properties:
12 ( h 1 + h 2 )
= h , where h is defined in Lemma 7. 2.
For any x ∈ R d , if prog( x ) is odd, then prog( ∇ h 1 ( x ))
≤ prog( x ) ; if prog( x ) is even, then prog( ∇ h 2 ( x ))
≤ prog( x ) .
h 1 and h 2 are also ℓ 0 -smooth with ℓ 0 = 152 .
Given Lemmas 7 and 8, we now construct two instances that lead to the two terms in lower bound (2), respectively.
Instance 1.
The proof of the first term Ω(( ∆ Lσ 2 nT ) 12 ) essentially follows the first example in proving Theorem 1 of [21].
We provide the key steps for the sake of being self-contained.
(Step 1.)
Let f i
=
Lλ 2 h ( x/λ ) /ℓ 0 , ∀ i ∈ [ n ] be homogeneous and hence f
= Lλ 2 h
( x/λ ) /ℓwhere
h is defined in Lemma 7 and λ > 0 is to be specified.
Since ∇ 2 f i
=
L ∇ 2 h/ℓ 0 and h is ℓ 0 -smooth by Lemma 7, we know f
i is L -smooth for any λ > 0 .
By Lemma 7, we have
[Equation]
Therefore, to ensure f i ∈ F ∆ L , it suffices to let
[Equation]
(Step 2.)
We construct the stochastic gradient oracle O i on worker i , ∀ i ∈ [ n ] as the follows:
[ O i ( x ; Z )] j = [
[Equation]
f i ( x )] j (cid:18)
[Equation]
x )
[Equation]
(cid:19)(cid:19) , ∀ x ∈ R d , j = 1 , . . . , d with random variable Z ∼ Bernoulii ( p ) independent of x and f i , and p ∈ (0 , 1) to be specified.
It is easy to see O i is an unbiased stochastic gradient oracle.
Moreover, since f i is zero-chain, we have prog( O i ( x ; Z )) ≤ prog( ∇ f i ( x ))
≤ prog( x ) + 1 and hence
E [ ∥ [ O i ( x ; Z )] −
[Equation]
Therefore
[Equation]
to ensure O i ∈ O σ
[Equation]
it suffices to let
[Equation]
(Step 3.)
Let x ( t ) i , ∀ t ≥ 0 and i ∈ [ n ] , be the t -th query point of worker i .
Since algorithms satisfy the zero-respecting property, as discussed in [5, 6, 21], within T gradient queries on each worker, algorithms can only return model ˆ x such that
[Equation]
which implies
[Equation]
By Lemma 2 of [21]
[Equation]
we have
[Equation]
On the other hand, when prog(ˆ x ) < d , by Lemma 7, it holds that
[Equation]
Therefore, by combining (11) and (12), we have
[Equation]
Let
[Equation]
and d = (cid:36)(cid:18)
[Equation] [Equation]
Then (8) naturally holds and p = min { g 2 ∞ σ 2 (cid:16) ∆ Lσ 2 3 nTℓ
[Equation]
g 2 ∞ (cid:17) 12 , 1 } by (9).
Without loss of generality, we assume d ≥ 2 , which is guaranteed when T = Ω( σ 2 nL ∆ ) .
Then, using the definition of p , we have that
[Equation]
which, combined with (13), leads to
[Equation]
Instance 2.
The proof for the second term Ω( c ∆ LT (1 − β )) utilizes weight matrices defined on the sun-shaped graphs described in Theorem 3.
(Step 1.)
Let functions
[Equation]
and ℓ 2 ( x ) :
= n ⌈ n/ 4 ⌉ (cid:88)
j odd [Equation]
[Equation]
j [Equation] d [Equation]
By Lemma 8, ℓ 1 and ℓ 2 defined here are 2 ℓ 0 -smooth.
Furthermore, let f i =  
[Equation]
if i ∈ I 1 ≜ { j : 1 ≤ j ≤ ⌈ n 4 ⌉} [Equation] Lλ
[Equation]
if i ∈ I 2 ≜ { j : n − ⌈ n 4 ⌉ + 1 ≤ j ≤ n } , else.
where λ > 0 is to be specified.
To ensure f i ∈ F ∆ L for all 1 ≤ i ≤ n , it suffices to let
{+Lλ 2 ∆ 
 d 2 ℓ≤ ∆ , i.e. , dλ+}
[Equation]
With the functions defined above, we have f ( x ) = 1 n (cid:80) ni
=
f i ( x ) =
Lλ 2 ℓ ( x/λ ) / (2 ℓ 0 ) and prog( ∇ f i ( x )) (cid:40)
= prog( x ) +if { prog( x ) is even
and i ∈ I 1 } ∪ { prog( x ) is odd and i ∈
I 2 } ≤ prog( x ) otherwise .
Therefore, to make progress, i.e. , to increase prog( x ) , for any gossip algorithm A , it must take the gossip communications to transmit information between I 1 to I 2 alternatively.
Namely, it takes at least dist { G t } ∞ t =
( I 1 , I 2 ) rounds of decentralized communications for any possible gossip algorithm
A to increase prog(ˆ x ) by 1 .
Therefore, we have
[Equation]
(Step 2.)
We consider a gradient oracle that return lossless full-batch gradients, i.e., O i ( x ) = ∇ f i ( x ) , ∀ x ∈ R d , i ∈ [ n ] .
For the construction of graphs and weight matrices, we consider the sequence of sun-shaped graphs { G t := S n, C t } ∞ t =0 and their associated weight matrices { W t } ∞ t =0 ∈
W n,β investigated in Theorem 3.
Since 1 − ( |I 1 | + |I 2 | ) /n = Ω(1) , by Theorem 3, we have dist { G t } ∞ t =
( I 1 , I 2 ) =
Θ((1 − β ) − 1 ) .
Suppose dist { G t } ∞ t =
( I 1 , I 2 ) ≥ 1 / ( C (1 − β )) with some absolute constant C , then by (15), we have prog
[Equation] [Equation]
(Step 3.)
We finally show the error E [ ∥∇ f ( x ) ∥ 2 ] is lower bounded by Ω (cid:16) ∆ L (1 − β ) T (cid:17) , with any algorithm
A ∈
A { W t } ∞ t =0 .
For any T ≥ 1 / ( C (1 − β )) =
Ω((1 − β ) − 1 ) , let
[Equation]
and
[Equation]
Then (14) naturally holds.
Since prog(ˆ x ) < d by (16), following (12) and using (17), we have
[Equation]
Appendix C. Upper Bound C.1. Preliminary
Notation.
We first introduce necessary notations as follows.
• x ( k ) =
[( x ( k ) 1 ) ⊤ ; ( x ( k ) 2 ) ⊤ ; · · · ; ( x ( k ) n ) ⊤ ] ∈ R n × d ; • ˜ g ( k ) ≜ ∇ F ( x ( k ) ; ξ ( k,r ) ) =
[ ∇ F 1 ( x ( k ) 1 ; ξ ( k,r ))
⊤ ; · · · ; ∇ F n ( x ( k ) n ; ξ ( k,r ) n ) ⊤ ] ∈ R n × d ; • ∇ f ( x ( k ) )
=
[ ∇ f 1 ( x ( k ) 1 ) ⊤ ; ∇ f 2 ( x ( k ) 2 ) ⊤ ; · · · ; ∇ f n ( x ( k ) n ) ⊤ ] ∈ R n × d ; • ¯ x ( k ) =
[(¯ x ( k ) )
⊤ ; (¯ x ( k ) ) ⊤ ; · · · ; (¯ x ( k ) )
⊤ ] ∈ R n × d where ¯ x ( k ) = 1 n (cid:80) ni
=1 x ( k ) i ; • W t =
[ w ti,j ] ∈ R n × n is the weight matrix; • 1 n = [1 , 1 , · · · , 1] ⊤ ∈ R n ; • Given two matrices x , h ∈ R n × d , we define inner product ⟨ x , h ⟩ = tr( x T h ) and the Frobenius norm ∥ x ∥ 2 F = ⟨ x , x ⟩ ; •
Given W ∈ R n × n , we let ∥ W ∥ 2 = σ max ( W ) where σ max ( · ) denote the maximum sigular value.
Smoothness.
Since each f i ( x ) is assumed to be L -smooth, it holds that f ( x ) = 1 n (cid:80) ni
=
f i ( x ) is also L -smooth.
As a result, the following inequality holds for any x, y ∈ R d :
[Equation]
Gradient noise.
For stochastic gradient oracles satisfying Assumption 2, by independence, it holds for any k ≥ 0 and R ≥ 1 that
[Equation]
and
[Equation]
where ˜ g ( k ) ≜ 1 n (cid:80) ni =1 ˜ g
[Equation]
=
[Equation]
Network weighting matrix.
Since each weight matrix W t ∈ W n,β , it holds that
[Equation]
Following (20), it holds for a sequence of weight matrices W t 1 , . . .
, W t 2 − 1 that
[Equation]
Therefore, when t 2 − t 1 grows, (cid:81) t 2 − 1 t = t 1 W t exponentially converges to 1 n 1 n 1 ⊤ n .
Submultiplicativity of the Frobenius norm.
For any matrix W ∈ R n × n and z ∈ R n × d , it holds that
[Equation]
To verify it, by letting z j be the j -th row of z , we have ∥ W z ∥ 2 F = (cid:80) dj
=1 ∥ W z j ∥ 22 ≤ (cid:80) dj =1 ∥ W ∥ 22 ∥
z j ∥ 22 = ∥ W ∥ 22 ∥ z ∥ 2 F .
C.2. Proof of Theorem
Our proof is adapted from the proof of [40, Theorem 1], which presents the convergence rate of stochastic decentralized gradient tracking with single consensus operation and a static weight matrix.
We generalize the proof to suit multiple consensus and time-varying weight matrices.
We use the matrix-form notations of the algorithm mostly for convenience.
At the beginning of phase k , the three quantities of interests are x ( k ) , h ( k ) and ˜ g ( k ) ≜ ∇ F ( x ( k ) ; ξ ( k,r ) ) , and the update rule for any k ≥ 0 is
[Equation]
where W ( k ) R ≜ (cid:81) ( k +1)
R − 1 t = kR
W t for any k ≥ 0 and R ≥ 1 .
By (21), we have ∥ W ( k ) R − 11 ⊤ /n
∥ 2 ≤ β R for any k ≥ 0 .
By multiplying 1 n 1 ⊤ n /n to the left-side of (23) and (24), we have
[Equation]
Since ¯ h (0) =
˜ g (0) , by iterating (25) over 0 , . . .
, k − 1 , it holds that ¯ h ( k ) =
˜ g ( k ) for any k ≥ 0 .
We will use the following descent lemma, which is adapted from [40, Lemma 3].
Lemma 9 ( D ESCENT L EMMA )
Under Assumption 1, 2, 3, if 0 < γ ≤ 12 L , then we have for any k ≥ 0 , E [ f
[Equation]
where Π ≜ I − 1 n 1 n 1 ⊤ n .
By iterating Lemma 9 over k = 0 , . . .
, K , we obtain
[Equation]
where ∆ ≥ f ( x (0) ) − min x f ( x ) .
We next turn to bound the consensus error E [ ∥ Π x ( k ) ∥ 2 F ] , which relies on the following recursion bound of consensus errors.
Lemma 10 ( R ECURSION OF C ONSENSUS E RROR )
Under Assumption 1, 2, 3, denoting ρ ≜ β R , it holds for 0 < γ ≤ 1 − ρ 2 24(1+ ρ 2 ) L that E
[Equation]
Proof Multiplying Π to the left side of (23) and (24), we have
[Equation]
Therefore, following (27) [Equation] by using ∥ Π W (2 k ) R a ∥ F ≤ ρ ∥ Π a ∥ F for any a ∈ R n × n and −⟨ a , b ⟩ ≤ 1 − ρ 2 1+ ρ 2 ∥ a ∥ 2 F + 1+ ρ 2 1 − ρ 2 ∥ b ∥ 2 F for any a , b ∈ R n × n , we have
[Equation]
Following (28)
[Equation]
we can bound ∥
Π h ( k +1)
∥ 2 F as follows:
[Equation]
[Equation]
where the inequality follows ∥ Π W (2 k ) R a ∥ F ≤ ρ ∥ Π a ∥ F ≤ ρ ∥
a ∥ F and E [˜ g ( k +1)
| h ( k ) , ˜ g ( k ) ]
= ∇ f
( x ( k +1) ) .
We next bound the terms in (29) one by one.
By using the similar derivation to [40,
Lemma 5]
[Equation]
we can easily reach
[Equation]
where we use ¯ x ( k +1)
− ¯ x ( k ) =
− γ ˜ g
( k ) and E [ ∥ ˜ g ( k )
∥ 2 F ] ≤ E [ ∥ ¯ g ( k ) ∥ 2 F ] + σ 2 /R =
n E [ ∥ ¯ g ( k ) ∥ 2 ] + σ 2 /R .
Combining (30) and (31) together, we reach
E
[Equation]
[Equation]
+
[Equation]
We next turn to bound E [ ⟨ Π W (2 k +1) R
[Equation] ( ∇ f ( x [Equation] k ) )
− ˜ g ( k ) ) ⟩ F ] in (29).
For any k ≥ 1 , since h ( k ) =
W (2 k − 1) R R
( h ( k − 1) + ˜ g ( k ) − ˜ g ( k − 1) ) , E [ ∇ f
x ( k ) )
− ˜ g ( k ) | h ( k − 1) , ˜ g ( k − 1) ]
= 0 , we reach
[Equation]
f ( x ( k )
[Equation]
− ˜ g ( k ) ) ⟩ F ] .
Since (cid:13)(cid:13)(cid:13)(cid:13)(cid:16)
[Equation]
we further have
[Equation]
It is easy to see that (33) also holds for k = 0 .
We finally bound the last term
E
[Equation]
x
[Equation]
k ) ))
[Equation]
] in (29).
Since ∥ Π W (2 k +1)
R a ∥ F ≤ ρ ∥ a ∥ F for any a ∈ R n × d , we have
[Equation]
≤ ρ
[Equation]
[Equation]
34) where we us ∥
Π x ( k +1)
∥
F ≤ ρ ∥ Π x ( k ) ∥
F + γρ ∥ Π h ( k ) ∥ F and ¯ x ( k +1)
− ¯ x ( k ) =
− γ ˜ g
( k ) in the last inequality.
By Young’s inequality
[Equation]
we have for any η
[Equation]
η 2 > 0 that
[Equation]
[Equation]
Plugging (35) and (36) into (34), we have
[Equation]
Plugging (32), (33), and (37) into (29), we reach
E [
[Equation]
Letting η 1 =
2(1 − ρ 2 ) 9(1+ ρ 2 ) and η 2 = 1 − ρ 2 9(1+ ρ 2 )
[Equation]
then it holds for any 0 ≤ γ ≤ 1 − ρ 2 24(1+ ρ 2 ) L that
[Equation]
≤
[Equation]
n, which, combined with (38), leads to the conclusion. Letting a ( k ) ≜ [ 1 n E [ ∥ Π x ( k ) ∥ 2 F ] , 1 nL 2 E [ ∥ Π h ( k ) ∥ 2 F ]] ⊤ ∈ R 2 , b ( k ) ≜ [0 , 12 γ 2 ρ 2 1 − ρ 2 E [ ∥ ¯ g ( k ) ∥ 2 F ] + 6 σ 2 RL 2 ] ⊤ ∈ R 2 for any k ≥ 0 , and
[Equation]
by Lemma 10, it holds that
[Equation]
where ⪯ indicates entry-wise inequality.
Since γ ≤
[Equation]
−
[Equation] [Equation]
9 ρ 2 (1+ ρ 2 ) L , one can check that there exists v 1 , v 2 ≥ 0 such that M [ v 1 , v 2 ] ⊤ ≺ [ v 1 , v 2 ] ⊤ .
Therefore, by [40, Lemma 9], we have for any k ≥that
[Equation]
Therefore, we reach
[Equation]
Since a (0) = 0 by our initialization, considering the first entry of the above, we have
L
[Equation]
When γ ≤
[Equation]
Hence, plugging (39) into (26) yields
[Equation]
Plugging
[Equation]
(cid:40)
[Equation]
=
[Equation]
and T = KR into the above
[Equation]
we reach
[Equation]
Furthermore, if one set
[Equation]
=
[Equation]
so that
[Equation]
then we obtain
[Equation]