Revisiting Optimal Convergence Rate for Smooth and Non-convex Stochastic Decentralized Optimization
Anonymous Author(s)
AffiliationAddress email
Abstract
While numerous effective decentralized algorithms have been proposed with theoretical guarantees and empirical successes, the performance limits in decentralizedoptimization, especially the influence of network topology and its associated weightmatrix on the optimal convergence rate, have not been fully understood.
While
Lu and Sa [39] have recently provided an optimal rate for non-convex stochasticdecentralized optimization for weight matrices associated with linear graphs, theoptimal rate with general weight matrices remains unclear.
This paper revisits non-convex stochastic decentralized optimization and establishesan optimal convergence rate with general weight matrices.
In addition, we alsoestablish the first optimal rate when non-convex loss functions further satisfythe Polyak-Lojasiewicz (PL) condition.
Following existing lines of analysis inliterature cannot achieve these results.
Instead, we leverage the Ring-Lattice graphto admit general weight matrices while maintaining the optimal relation between thegraph diameter and weight matrix connectivity.
Lastly, we develop a decentralizedalgorithm to nearly-attain the above two optimal rates.
Introduction
1.1 Motivation.
Decentralized optimization is an emerging paradigm for large-scale machine learning.
By letting each node average with its neighbors, decentralized algorithms save great communicationoverhead compared to traditional approaches with a central server.
While numerous effectivedecentralized algorithms (see, e g., [46, 11, 57, 35, 5, 37, 73]) have been proposed in literatureshowing both theoretical guarantees and empirical successes, the performance limits in decentralizedoptimization have not been fully clarified.
This paper provides further understandings in optimalconvergence rate in decentralized optimization and develops algorithms to achieve these rates.
Weight matrix and connectivity measure.
Assume n computing nodes are connected by somenetwork topology (graph).
We associate this topology with a weight matrix W =
[ w ij ] ni,j =1 ∈ R n × nin which w ij ∈ (0 , 1) if node j is connected to node i otherwise
w ij = 0 .
We introduce β :=
∥ W − 1 n 1 n 1 Tn ∥ ∈ (0 , 1) , where 1 n = (1 , . .
. , 1) T ∈ R n with all entries being 1 , as the connectivitymeasure to gauge how well the network topology is connected.
Quantity β → 0 (which implies
W → 1 n 1 n 1 T n ) indicates a well-connected topology while β → 1 (which implies W → I ) indicatesa badly-connected topology.
It is the weight matrix W and its connectivity measure β that bringmajor challenges to analysis in decentralized optimization.
Decentralized optimization.
Decentralized approaches are built upon the partial averaging x + i = w ij x j (or x + =
Wx in a more compact manner) where N i is the set of neighbors of node
i
References
[Table]
LB [39]
[Table]
[Equation]
[Table]
Theorem
[Equation]
∈ [0 , cos(
[Equation]
/n )]
[Table]
UB
DSGD [26] β ∈ [0 , 1)
[Table]
D 2 /ED [61]
[Equation]
[Equation]
[Table]
DSGT [25] β ∈ [0 , 1)
[Table]
DeTAG [39] β ∈ [0 , 1)
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
(including node i itself).
The sparsity and connectivity of W significantly influence the efficiency andeffectiveness in partial averaging.
Decentralized gradient descent [46], diffusion [11, 54] and dualaveraging [17] are well-known decentralized methods.
Other advanced algorithms, which can correctthe bias caused by heterogeneous data distributions, include decentralized ADMM [40, 58], explicitbias-correction [57, 74, 34], gradient tracking [45, 15, 49, 69], and dual acceleration [55, 63].
In the stochastic regime, decentralize SGD [11, 54, 35, 26] and its momentum and adaptive variants[37, 73, 42] have attracted a lot of attentions.
After being extended to directed topologies [5, 70],time-varying topologies [26, 43, 64], asynchronous settings [36], and data-heterogeneous scenarios[61, 68, 2, 25], decentralize SGD has much more applicable scenarios with significantly improvedperformance.
Decentralized algorithms are also closely related to federated learning methods [41, 59,71, 24, 33] when they admit weight matrices alternating between 1 n 1 n 1 Tn and the identity matrix.
Prior understandings in theoretical limits.
A series of pioneering works have shed lights on thetheoretical limits in decentralized optimization.
These works establish the optimal convergence ratefor convex or non-stochastic decentralized optimization [55, 56, 60, 28], and propose decentralizedalgorithms that (nearly) match these optimal bounds [55, 56, 63, 60, 29, 28].
However, there are fewstudies on the theoretical limits in non-convex stochastic decentralized optimization, which is quitecommon in large-scale deep learning, even when the costs are smooth.
A recent work [39] establishes the optimal complexity for smooth and non-convex stochastic decentralized optimization.
While this optimal bound is inspiring, it is valid for a restrictive family of weightmatrix W associated with the linear graph whose connectivity measure β
= cos( π/n ) 1 .
However, thelinear graph is just one of the commonly used topologies.
A more difficult but fundamental questionstill remains open: what is the optimal convergence rate for smooth and non-convex stochasticdecentralized optimization with a general weight matrix (that does not satisfy β = cos( π/n ) )?
Itis not known whether the lower bound in [39] still holds for other commonly-used weight matricesresulted from grids, toruses, hypercubes, etc.
whose connectivity measure β ̸
= cos( π/n ) .
Challenges.
Following existing lines of analysis, to our knowledge, cannot provide an assuringanswer.
It is known that the communication complexity in decentralized optimization is typicallyproportional to the diameter D of the network topology [55].
Establishing the relation between D andβ is key to clarifying the influence of the weight matrix on optimal convergence rate.
Existing works[55, 56, 39] utilize the linear graph to establish the optimal relation D = Θ(1 / √ 1 − β ) .
However,
References
[Table]
Theoremβ ∈
[Table]
cos( π/n )
[Table]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
weight matrices associated with linear graph are quite limited .
Given a fixed network size n (which isa prerequisite when discussing complexities in distributed stochastic optimization [39, 22, 27, 19]),they have to satisfy β = cos( π/n ) .
Weight matrices with other β cannot be implied from the lineargraph.
To examine the lower bounds with a more general weight matrix W , [60] proposes a widerclass of networks named linear-star graph, which, however, leads to suboptimal relation between Dand β .
Therefore, a network topology that admits general weight matrices W while maintaining theoptimal relation between D and β is critical to clarify
the above fundamental question.1.2 Main results.
This paper successfully identifies the desired network topology which facilitatesthe theoretical limits over the general weight matrices.
Our contributions are summarized as follows.
• We discover that ring-lattice graph maintains the optimal relation between D and β for a muchbroader class of weight matrices.
Given any fixed n and β ∈ [0 , cos( π/n )] , we can alwaysconstruct a weight matrix W ∈ R n × n associated with some ring-lattice graph, whose connectivitymeasure is β , that satisfies D = Θ(1 / √ 1 − β ) .
Since cos( π/n ) approaches to 1 when n is large,the ring-lattice graph admits far broader weight matrices than linear graph used in [55, 56, 39].• With the above results we provide the optimal convergence rate for smooth and non-convexstochastic decentralized optimization that holds for any weight matrix with β ∈ [0 , cos( π/n )] .In
contrast, the optimal rate in [39] only applies to weight matrix satisfying β = cos( π/n ) .
Wealso provide the optimal convergence rate when the non-convex cost functions further satisfy the
) condition [23] for any weight matrix with β ∈ [0 , cos( π/n )] .
• We prove that the above two optimal complexities can be nearly-attained (under mild assumptionsand up to the condition number L/µ as well as other logarithmic factors) by simply integratingmultiple gossip communication [38, 53] and gradient accumulation [55, 52, 39] to the vanilladecentralized SGD (D-SGD) algorithm.
Compared to DeTAG [39], the extended D-SGD avoidsthe gradient tracking and saves half of the communication overheads.
Compared to DAGD [52], theextended D-SGD achieves the near-optimality using constant mini-batch sizes and has theoreticalguarantees in the non-convex (as well as the PL) scenario
All established results in this paper as well as those of existing state-of-the-art decentralized algorithmsare listed in Tables 1 and 2.1.3 Other related works.
Some other related works are discussed as follows.
Lower bounds and optimal algorithms.
The lower bounds and optimal algorithms for stochasticand convex optimization has been intensively studied in [31, 32, 50, 1, 16].
References [9, 10]established the lower bounds to find stationary solutions for non-convex optimization, and [19, 3]proposed algorithms with near-optimal convergence rate for non-convex finite-sum minimization.
In deterministic decentralized optimization, the optimal convergence rate was established for stronglyconvex optimization [55], non-smooth convex optimization [56], and non-convex optimization [60].
Furthermore, the lower bounds and optimal algorithms over time-varying topologies are studied in[29, 52, 53].
In stochastic decentralized optimization, works [72] and [26] established the lowerbounds for strongly-convex decentralized SGD.
[39] proves the optimal complexities for non-convexproblems when weight matrices satisfy β = cos( π/n ) .
Ring-lattice Graph.
The ring-lattice graph was used in [65, 20, 66] as a basis model to generate thesmall world graph.
However, its relation between graph diameter and connectivity measure is notclarified to our knowledge.
A related work [67] examines the robustness of the ring-lattice graph,but does not discuss its connectivity measure.
Furthermore, ring-lattice has never been utilized toestablish the influence of the weight matrix on the optimal rate in decentralized optimization before.
Problem setup
Problem setup.
Consider the following problem with a network of n computing nodes:
[Equation]
Function f i ( x ) is local to node i , and random variable ξ i denotes the local data that follows distribution
D i .
Each local data distribution D i can be different across all nodes.
Assumptions.
The optimal convergence rate is established under the following assumptions.
• Function class.
We assume each local loss function is L -smooth, i.e.,
[Equation]
for some constant L > 0 .
We let the class F L denote the set of all functions satisfying (2).
We
may also assume the global loss f ( x ) = 1 n (cid:80) ni
=
f i ( x ) satisfies the µ -PL condition:
[Equation]
where f ⋆ is the optimal function value in problem (1) and µ > 0 is some constant.
We let the
function class F L,µ denote the set of all functions satisfying both (2) and (3).
• Oracle class.
We assume each node i interacts with f i ( x ) via the stochastic gradient oracle ˜ g i ( x, ξ ) where ξ is a random variable.
In addition, we assume oracle ˜
g
i ( x, ξ ) satisfies
[Equation]
We let O σ 2 denote the class of all such stochastic gradient oracles.
• Weight matrix class.
We let W n,β denote the class of all gossip matrices W ∈ R n × n satisfying
[Equation]
where 1 n = (1 , . .
. , 1) T ∈ R n with all entries being 1 .
If W is positive semi-definite, thenconnectivity measure β is its second largest eigenvalue. • Algorithm class.
We consider an algorithm A in which each node i assesses an unknown local function f i via the stochastic gradient oracle ˜
g
i ( x ; ξ i ) ∈
O σ 2 .
Each node i running algorithm
A will maintain a local model copy x
( k ) i at iteration k .
We assume A to follow the partial averaging policy, i.e., each node communicates via protocol
[Equation]
for some W = [ w ij ] ni,j =1 ∈ W n,β where [ n ] := { 1 , · · · , n } , and y and z are the input and outputvariables of the communication protocol.
In addition, we assume A to follow the zero-
respectingpolicy [9, 10].
Informally speaking, the zero-respecting policy requires that the number of non-zero entries of local model copy x ( k ) i can only be increased by sampling its own stochasticgradient oracle or interacting with the neighboring nodes.
The zero-respecting policy is widelyfollowed by all methods in Tables 1 and 2 and their momentum and adaptive variants.
We let A Wbe the set of all algorithms following the partial averaging and zero-respecting policies.
Lower bound metrics.
With the above classes, we now introduce the measures for convergencerate’s lower bound.
Given loss functions { f i } n i =1 ⊆ F L (or { f i } n i =1 ⊆ F L,µ ), stochastic gradientoracles { ˜ g i } ni =1 ⊆
O σ 2 , a weight matrix W ∈ W n,β , an algorithm A ∈
A W , the number of gradientqueries (or communications rounds) T , we let ˆ x A, { f i } ni =1 , { ˜ g i } ni =1 ,W,T (which will be denoted as ˆ xwhen there is no ambiguity) be the output of algorithm A with ˆ x ∈ Span (cid:0) {{ x ( t ) j } nj =1 } Tt =1 (cid:1) .
Thelower bound for smooth and non-convex stochastic decentralized optimization is defined as
[Equation]
The lower bound under the PL condition is defined as
[Equation]
Fixed network size n .
This paper considers stochastic decentralized optimization where the networksize n is a fixed constant .
A fixed n is a prerequisite in distributed stochastic optimization whichenables distributed algorithms to achieve the linear speedup in convergence rate O ( σ/ √ nT ) , see thealgorithms listed in Tables 1 and 2.
In decentralized deterministic optimization, however, size n doesnot appear in the convergence rate.
Thus, it does not need to be specified and can be varied freelywhen establishing the lower bounds [55, 56].
Ring-Lattice graph
The main challenge in analyzing theoretical limits in decentralized optimization lies in clarifying theinfluence of decentralized network topology and its associated weight matrix.
It is intuitive that thegraph diameter D affects the convergence rate.
In the worst case, one node needs to take at least Dcommunication steps to transmit its own gradient information to the other.
If the relation between Dand β is established, the influence of the weight matrix on lower bounds will become clear.
However,existing results either establishes suboptimal relation between D and β [60] or admits restrictiveweight matrices [55, 56, 39].
Now we show that the ring-lattice graph maintains the optimal relationbetween D and β for a broad class of weight matrices.
All proof details are in Appendix B.
Definition 1 (R ING -L ATTICE G RAPH [66]) .
Given any positive integers n, k such that k is evenand 2 ≤ k < n − 1 , the k -regular ring-lattice graph over nodes { 1 , · · · , n } , denoted by R n,k , is anundirected graph in which each node i ∈ [ n ] is connected with the set of nodes { ( i + ℓ ) mod n : 1 ≤| ℓ | ≤ k/ 2 } where ( i + ℓ ) mod n is defined as( i + ℓ ) mod n = [Equation] if [Equation]
i +
[Equation] n [Equation] i
[Equation] i [Equation] Apparently, R n,k is a k -regular graph.
When k = 2 , ring-lattice will reduce to the ring graph.
While
Definition 1 excludes the case in which k
= n − 1 since n − 1 is not necessarily a even number, thecomplete graph can be exceptionally viewed as the ring lattice with k = n − 1 .
Therefore, for general2 ≤ k ≤ n − 1 , R n,k can be regarded as an intermediate state between the ring and complete graphs,see the illustration in Fig.
The following lemma clarifies the diameter of R n,k .
Lemma 1.
The diameter of the ring-lattice graph R n,k is D
=
Θ
( n/k ) .
It is derived from the above lemma that the diameters of R n, 2 and R n,n − 1 are Θ( n ) and Θ(1) ,respectively, which are consistent with the ring and complete graphs.
The following fundamental theorem establishes the relation between D and connectivity measure β .
Theorem 1.
Given a fixed n and any β ∈ [0 , cos( π/n )] , we can always construct a ring-latticegraph R n,k so that
(1) it has an associated weight matrix W ∈ W n,β , i.e., W ∈ R n × n and ∥ W − 1 n 1 n 1 Tn ∥ =
β ,(2) its diameter D satisfies D = Θ(1 / √ 1 − β ) .
Remark 1.
The works [55, 56, 39] utilize the linear graph to establish the same relation D =Θ(1 / √ 1 − β ) .
However, given fixed network size n , the linear graph admits very limited weightmatrices with β = cos( π/n ) .
Remark 2.
Sun and Hong [60] propose a slightly wider class of networks named linear-star graph,which, however, leads to a suboptimal relation D = Ω(1 /
[ n (1 − β )]) [60, Proof of Corollary 3.1].
Remark 3.
It is worth noting that for deterministic decentralized optimization where n may vary, onecan tune n to make β = cos( π/n ) lie in the full interval (0 , 1) as shown in [55, 56].
In other words,[55, 56] proved that D = Θ(1 / √ 1 − β ) for any β ∈ (0 , 1) if n is a varying parameter.
However,this result is not valid in stochastic scenarios where n is required to be fixed.
Lower bounds with general weight matrices
This section will establish the lower bounds for smooth and non-convex decentralized stochasticoptimization.
All proof details are in Appendix C. When the cost function does not satisfy the PLcondition (3), we will derive that the convergence rate is lower bounded by Ω( σ √ nT + DT ) .
This resulttogether with Theorem 1 leads to:
Theorem 2.
For any L > 0 , n ≥ 2 , β ∈ [0 , cos( π/n )] , and σ > 0 , there exists a set of loss functions{ f i } ni =1 ⊆ F L , a set of stochastic gradient oracles { ˜ g i } ni =1 ⊆
O σ 2 , and a weight matrix W ∈ W n,β ,such that it holds for any A ∈ A W starting form x (0)
that
[Equation]
Remark 4 ( Comparison with existing bounds).
Comparing to the lower bound established in[39], Theorem 2 admits a much broader class of weight matrices with β ∈ [0 , cos( π/n )] .
While theinterval [0 , cos( π/n )] ⊂ [0 , 1) , it approaches to [0 , 1) as n goes large.
Such interval is broad enoughto cover most weight matrices (generated through the Laplacian rule W =
I − L/d max ) resultedfrom common topologies such as line, ring, grid, torus, hypercube, exponential graph, completegraph, Erdos-Renyi graph, geometric random graph, etc.
whose β lies in the interval [0 , cos( π/n )]when n is sufficiently large, see details in Appendix C.
Remark 5 ( Transient iteration complexity).
Due to the topology-incurred overhead in convergence rate, a decentralized algorithm has to experience a transient stage to achieve linear-speedup.
Transient iterations are referred to those iterations before an algorithm reaches linear-speedupstage.
To reach linear speedup in the lower bound (8) , T has to satisfy 1 T √ 1 − β ≤ σ √ nT and hence
T =
O ( n (1 − β ) σ 2 ) , which is the transient iteration complexity shown in the first line in Table 1.
Thesmaller the transient iterations are, the faster the algorithm can achieve linear speedup.
Transientiteration complexity is an important metric to evaluate how sensitive the algorithm is to β .
Require:
Initialize x (0) i = 0 , the weight matrix W , and the rounds of gossip
R .
for k = 0 , 1 , 2 , ..., K , every node i do
Sample { ξ ( k,r ) i } Rr =1 independently and let g ( k ) i = 1 R R (cid:80) r =1 ∇ F ( x ( k ) i ; ξ ( k,r ) i ) ; ▷ grad.
accumulation
Update ϕ
( k +1)
i =
x ( k ) i − γg ( k ) i ; ▷ local stochastic gradient descent
Update x ( k +1)
i =
FastGossipAverage ( { ϕ ( k +1) i } ni =1 , W, R ) ;
▷ multiple gossips
Algorithm
Require: { ϕ i } ni =1 , W , R , and β ; let z (0) i =
z ( − 1) i =
ϕ
i and η = 1 − √ 1 − β1+ √ 1+ β 2 .
for r = 0 , 1 , 2 , ..., R − 1 , every node i do
Update z ( r +1) i
= (1 + η ) (cid:80) j ∈N i w ij z
( r ) i − ηz
( r − 1) i ; ▷ fast gossip averaging
Output: x i =
z ( R ) i ;
The lower bound in (8) ignores constants L and f ( x (0) )
− f ⋆ since the influences of gradient noiseσ , network size n , and network topology β on the convergence rate typically attract more attentions.
A lower bound with constants L and f ( x (0) )
− f ⋆ is given in Appendix C.
When the cost function satisfies PL condition (3), we prove the convergence rate is lower bounded byΩ( σ √ nT + exp( − CT/D )) where C is some constant.
This result together with Theorem 1 leads to:
Theorem 3.
For any L > 0 , n ≥ 2 , β ∈ [0 , cos( π/n )] , and σ > 0 , there exists a set of loss 206 functions { f i } ni =1 ⊆ F L,µ , a set of stochastic gradient oracles { ˜ g i } ni =1 ⊆
O σ 2 , and a weight matrix
W ∈ W n,β , such that it holds for any A ∈ A W
starting from x (0) that
[Equation]
where C is a universal constant.
Rate (9) results in an ˜ O
((1 − β ) − 12 ) transient complexity.
Note that this lower bound also applies to strongly-convex stochastic decentralized optimization, seethe construction details in Appendix C.
DSGD with multiple gossip communications
This section will present a decentralized algorithm that achieve the lower bounds established in Sec.
4up to logarithmic factors.
The new algorithm is a direct extension of the vanilla decentralized SGD(DSGD)
Inspired by the algorithm development in [52, 39], we add two additional components to DSGD: gradient accumulation and multiple-gossip communication.
The main recursions arelisted in Algorithm 1 which utilizes the fast gossip average step [38] in Algorithm 2.
We call the newalgorithm as MG-DSGD where “MG” indicates “multiple gossips”.
All proofs are in Appendix D.
The DeTAG algorithm proposed in [39] applies the gradient accumulation and multiple-gossipcommunication techniques to gradient tracking, which achieves the optimal rate in non-convex(without PL condition) stochastic decentralized optimization.
This paper finds in MG-DSGD thatthe multiple gossip technique can significantly reduce the influence of data heterogeneity, and thegradient tracking step can be saved to achieve the convergence optimality.
Similar observations alsoappeared in [52] albeit in the strongly-convex scenario.
Removing gradient tracking from algorithmupdates brings two benefits.
First, it saves half of the communication overhead and memory storageper iteration.
Second, it enables numerous effective acceleration techniques designed exclusively fordecentralized SGD such as Quasi-global momentum [37], DecentLaM [73], adaptive momentum[42], etc. which cannot be easily extended to gradient tracking.
Let T be the toal number of gradient queries (or decentralized communications) at each node.
Sinceeach node takes R gradient queries and R communications at round k , it holds that T = KRwhen MG-DSGD finishes after K rounds.
The following theorems clarify the convergence rate of MG-DSGD where T = KR
We omit the initialization constants below.
Our analysis techniquesrelate to the classic SGD analysis but with inaccurate oracles in the nonconvex case [18].
Theorem 4.
Given L > 0 , n ≥ 2 , β ∈ [0 , 1) , σ > 0 , and let A denote Algorithm 1.
Assuming that1 n (cid:80) ni =1 ∥∇ f i ( x ) − ∇ f ( x ) ∥ 2 ≤ b 2 for any x ∈ R n and the number of gossip rounds R is set as in
Appendix D, the convergence of A can be bounded for any { f i } ni =1 ⊆ F L and any W ∈ W n,β that
[Equation]
¯ x ( k ) = 1 n n (cid:88) i =
x ( k ) i .
Remark 6.
The rate (10) matches with the lower bound (8) up to logarithm factors.
Furthermore, weremark that the gradient similarity assumption is not required to obtain the lower bound in Theorem 2. Due to these two restrictions, MG-DSGD is a nearly-optimal (not fully-optimal) algorithm.
However,while the analysis in MG-DSGD needs to assume bounded gradient similarity b 2 , it only affects theconvergence rate as a logarithm term hidden in the ˜ O ( · ) notation.
Theorem 5.
Under the same assumptions as in Theorem 4 and setting R as in Appendix D, the 242 convergence of A can be bounded for any loss functions { f i } n i =1 ⊆
F L,µ , and any W ∈ W n,β by
[Equation]
Remark 7. Comparing (11) and lower bound (9) , we find its dependence on σ 2 , n , T , and 1 − β isoptimal while that on L and µ is not.
One reason is that our algorithm does not involve any Nesterovacceleration [47] mechanism. The established upper bound in a recent work [52] is closer to thelower bound (9) by utilizing Nesterov acceleration, see Table 2. However, it requires an increasinglylarge mini-batch of data per iteration, and only works for the strongly-convex scenario. It is notknown whether the acceleration mechanisms in [52] can be extended to the non-convex and PLscenario.
The bound (11) is state-of-the-art under the general PL condition to our knowledge.
Experiments
This section will validate our theoretical results by empirically comparing different decentralizedalgorithms DSGD [35], D 2 [62], DSGT [76], DeTAG [39] and MG-DSGD in deep learning.
Implementation setup.
We implement all decentralized algorithms with PyTorch [48] 1.6.0 using
NCCL 2.8.3 (CUDA 10.1) as the communication backend, in which the partial averaging primitiveis implemented by NCCL send/receive primitives.
For parallel SGD, we used PyTorch’s native
Distributed Data Parallel (DDP) module.
All the models and training scripts in this section run onservers with 8 NVIDIA V100 GPUs with each GPU treated as one node.
Similar to existing [-works[
 ],-]works[39], we use Ring graph throughout all experiments.
Tasks and training details.
A series of experiments are carried out with CIFAR-10 [30] and
ImageNet [14] to compare the aforementioned methods.
We utilize ResNet variants [21] on CIFAR10 and ResNet-50 on ImageNet.
For training protocol, we train total 300 epochs and the learningrate is warmed up in the first 5 epochs and is decayed by a factor of 10 at 150 and 250-th epoch.
Forlearning rate, we tuned a strong baseline in the PSGD setting (5e-3 for single node) and used thesame setting in all decentralized methods.
The batch size is set to 128 on each node.
All CIFAR-10experiments are repeated three times with different seeds.
For fair comparison, we maintain the totalnumber of gradient queries the same for algorithms in experiments, i.e., T = KR .
Performance with homogeneous data.
Table 3 compares the proposed MG-DSGD with centralizedparallel SGD and other baselines with homogeneous data distributions.
For DeTAG and MG-DSGD,the communication round R is set as 2.
We also performed an ablation study on the effect ofmomentum acceleration.
Except for the ResNet-20 scenario with momentum in which MG-DSGDand DeTAG achieve the same performance, MG-DSGD consistently outperforms baseline methodsin other scenarios.
Moreover, MG-DSGD is more communication-efficient than DSGT and DeTAG.
[Table]
[Table] [Figure]
DSGDDSGTD2DeTAGMG
[Figure]
Performance with heteogeneous data. Following [75, 37], we simulate data heterogeneity amongnodes with a Dirichlet distribution-based partitioning. A quantity α controls the degree of the dataheterogeneity. The training data for a particular class tends to becomes more heterogeneous as α → 0 ,while the homogeneity intensifies as α → ∞ .
We tested α = 0 .
1 / 1 / 10 for all compared methodsas corresponding to a setting with high/mild/low heterogeneity.
As shown in Table 4, DeTAG and MG-DSGD outperform the other baselines by visible margins, and MG-DSGD is with slightly betteraccuracies.
The left plot in Fig.
2 illustrates that MG-DSGD achieve a competitive accuracy to DeTAG using much less communication budgets on CIFAR-10 with α = 1 .
[Table]
Large-batch training in ImageNet.
To train ResNet-50 on ImageNet,utilization of the large-batch is necessary to speedup the trainingprocess.
Since MG-DSGD is based on DSGD, we can extend thelarge-batch acceleration techniques introduced in [73, 37] to MGDSGD to achieve better performance.
These techniques are developedexclusively for DSGD and cannot be integrated to DeTAG easily.
Following [73], we train total 120 epochs.
The learning rate is warmedup in the first 20 epochs and is decayed in a cosine annealing scheduler.
The results are listed in Table 5, showing that MG-DSGD has better accuracy than other approaches.
Fig. 2(right) illustrates how different algorithms perform in terms of communication budgets used.
More results.
In Appendix E, we depict the accuracy curves in terms of epochs (rather thancommunication volumes).
The ablation on the influence of inner loop R is also listed there.
Conclusion
This paper revisits the theoretical limits in smooth and non-convex stochastic decentralized optimization.
Existing lower bound proved in [39] is only valid for special weight matrices with connectivitymeasure β = cos( π/n ) .
It is not known what the lower bound is when using weight matrices withother β .
To address this issue, this paper establishes the lower bound for any weight matrix withβ ∈ [0 , cos( π/n )] .
We also clarify the lower bound when loss functions satisfy the PL condition.
Anew algorithm is proposed to match these lower bounds up to logarithm factors.
Checklist
For all authors
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and
scope?
[Yes
(b) Did you describe the limitations of your work
[Yes See Section {+
+}
(c) Did you discuss any potential negative societal impacts of your work
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Section 2.
(b) Did you include complete proofs of all theoretical results?
[Yes] See appendix.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[No]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See Section 6.
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[Yes] See Section 6.
(d) Did you include the total amount of compute and the type of resources used (e g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See Section 6.
If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] See Section 6.
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’reusing/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A] (c
[Table] [Table]