Revisiting Optimal Convergence Rate for Smooth and Non-convex Stochastic Decentralized Optimization
Anonymous Author(s)
AffiliationAddress email
Abstract
While numerous effective decentralized algorithms have been proposed with theoretical guarantees and empirical successes, the performance limits in decentralizedoptimization, especially the influence of network topology and its associated weightmatrix on the optimal convergence rate, have not been fully understood.
While
Lu and Sa [33] have recently provided an optimal rate for non-convex stochasticdecentralized optimization for weight matrices associated with linear graphs, theoptimal rate with general weight matrices remains unclear.
This paper revisits non-convex stochastic decentralized optimization and establishesan optimal convergence rate with general weight matrices.
In addition, we alsoestablish the first optimal rate when non-convex loss functions further satisfy the Polyak-Lojasiewicz (PL) condition.
Following existing lines of analysis in literaturecannot achieve these results.
Instead, we leverage the Ring-Lattice graph to admitgeneral weight matrices while maintaining the optimal relation between the graphdiameter and weight matrix connectivity.
Lastly, we develop a new decentralizedalgorithm to attain the above two optimal rates up to logarithm factors.
Introduction
1.1 Motivation.
Decentralized optimization is an emerging paradigm for large-scale machine learning.
By letting each node average only with its neighbors, decentralized algorithms save remarkablecommunication overhead compared to traditional approaches with a central server.
While numerouseffective decentralized algorithms (see, e.g., [39, 7, 46, 29, 4, 31, 62]) have been proposed in literatureshowing both theoretical guarantees and empirical successes, the performance limits in decentralizedoptimization have not been fully clarified.
This paper provides further understandings in optimalconvergence rate in decentralized optimization and develops algorithms to achieve these rates.
Weight matrix and connectivity measure.
Assume n computing nodes are connected by somenetwork topology (graph).
We associate this topology with a weight matrix W =
[ w ij ] ni,j =1 ∈ R n × nin which w ij ∈ (0 , 1) if node j is connected to node i otherwise
w ij = 0 .
We introduce β :=
∥ W − 1 n 1 n 1 Tn ∥ ∈ (0 , 1) , where 1 n = (1 , . .
. , 1) T ∈ R n with all entries being 1 , as the connectivitymeasure to gauge how well the network topology is connected.
Quantity β → 0 (which implies
W → 1 n 1 n 1 T n ) indicates a well-connected topology while β → 1 (which implies W → I ) indicatesa badly-connected topology.
It is the weight matrix W and its connectivity measure β that bringmajor challenges to analysis in decentralized optimization.
Decentralized optimization.
Decentralized approaches are built upon the partial averaging x + i = w x j (or x + =
Wx in a more compact manner) where N i is the set of neighbors of node
i
References
[Table]
LB [33]
[Table]
/n
[Table]
Theorem
[Table]
UB
DSGD [20] β ∈ [0 , 1)
[Table]
D 2 /ED
[Table]
∈ [0 , 1)
[Table]
DSGT [19] β ∈ [0 , 1)
[Table]
DeTAG [33] β ∈ [0 , 1)
[Table]
MG
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table] [Table]
[Equation]
[Table]
[Table]
(including node i itself).
The sparsity and connectivity of W significantly influence the efficiencyand effectiveness in partial averaging.
Decentralized gradient descent [39], diffusion [7, 43] and dualaveraging [12] are well-known decentralized methods.
Other advanced algorithms, which can correctthe bias caused by heterogeneous data distributions, include decentralized ADMM [34, 47], explicitbias-correction [46, 63, 28], gradient tracking [38, 10, 41, 58], and dual acceleration [44, 52].
In the stochastic regime, decentralize SGD [7, 43, 29, 20] and its momentum and adaptive variants[31, 62, 36] have attracted a lot of attentions.
After being extended to directed topologies [4, 59],time-varying topologies [20, 37, 53], asynchronous settings [30], and data-heterogeneous scenarios[50, 57, 2, 19], decentralize SGD has much more applicable scenarios with significantly improvedperformance.
Decentralized algorithms are also closely related to federated learning methods [35, 48,60, 18, 27] when they admit weight matrices alternating between 1 n 1 n 1 Tn and the identity matrix.
Prior understandings in theoretical limits.
A series of pioneering works have shed lights on thetheoretical limits in decentralized optimization.
These works establish the optimal convergence ratefor convex or non-stochastic decentralized optimization [44, 45, 49, 22], and propose decentralizedalgorithms that (nearly) match these optimal bounds [44, 45, 52, 49, 23, 22].
However, there are fewstudies on the theoretical limits in non-convex stochastic decentralized optimization, which is quitecommon in large-scale deep learning, even when the costs are smooth.
A recent work [33] establishes the optimal complexity for smooth and non-convex stochastic decentralized optimization.
While this optimal bound is inspiring, it is valid for a restrictive family of weightmatrix W associated with the linear graph whose connectivity measure β
= cos( π/n ) 1 .
However, thelinear graph is just one of the commonly used topologies.
A more difficult but fundamental questionstill remains open: what is the optimal convergence rate for smooth and non-convex stochasticdecentralized optimization with a general weight matrix (that does not satisfy β = cos( π/n ) )?
Itis not known whether the lower bound in [33] still holds for other commonly-used weight matricesresulted from grids, toruses, hypercubes, etc.
whose connectivity measure β ̸
= cos( π/n ) .
Challenges.
Following existing lines of analysis, to our knowledge, cannot provide an assuringanswer.
It is known that the communication complexity in decentralized optimization is typicallyproportional to the diameter D of the network topology [44].
Establishing the relation between D andβ is key to clarifying the influence of the weight matrix on optimal convergence rate.
Existing works[44, 45, 33] utilize the linear graph to establish the optimal relation D = Θ(1 / √ 1 − β ) .
However,
References
[Table]
[Equation] [Table] [Equation] [Table]
[Equation]
[Table]
[Equation] [Table] [Equation] [Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
weight matrices associated with linear graph are quite limited .
Given a fixed network size n (which isa prerequisite when discussing complexities in distributed stochastic optimization [33, 16, 21, 13]) [Table] they have to satisfy β = cos( π/n ) .
Weight matrices with other β cannot be implied from the lineargraph.
To examine the lower bounds with a more general weight matrix W , [49] proposes a widerclass of networks named linear-star graph, which, however, leads to suboptimal relation between Dand β .
Therefore, a network topology that admits general weight matrices W while maintaining theoptimal relation between D and β is critical to clarify
the above fundamental question.1.2 Main results.
This paper successfully identifies the desired network topology which facilitatesthe theoretical limits over the general weight matrices.
Our contributions are summarized as follows.
• We discover that ring-lattice graph maintains the optimal relation between D and β for a muchbroader class of weight matrices.
Given any fixed n and β ∈ [0 , cos( π/n )] , we can alwaysconstruct a weight matrix W ∈ R n × n associated with some ring-lattice graph, whose connectivitymeasure is β , that satisfies D = Θ(1 / √ 1 − β ) .
Since cos( π/n ) approaches to 1 when n is large,the ring-lattice graph admits far broader weight matrices than linear graph used in [44, 45, 33].• With the above results, we provide the optimal convergence rate for smooth and non-convexstochastic decentralized optimization that holds for any weight matrix with β ∈ [0 , cos( π/n )] .
Incontrast, the optimal rate in [33] only applies to weight matrix satisfying β = cos( π/n ) .•
We also provide the optimal convergence rate when the non-convex cost functions further satisfythe Polyak-Lojasiewicz (PL) condition [17].
It is the first result on optimal convergence rate undersuch setting to our knowledge and is valid for any weight matrix with β ∈ [0 , cos( π/n )] .•
We prove that the above two optimal complexities can be attained (under mild assumptions andup to logarithmic factors) by simply integrating multiple gossip communication and gradientaccumulation to the vanilla decentralized SGD (D-SGD) algorithm.
Compared to DeTAG proposedin [33], the extended D-SGD avoids the gradient tracking step and saves half of the communicationoverheads per round.
All established results in this paper as well as those of existing state-of-the-art decentralized algorithmsare listed in Tables 1 and 2.1.3 Other related works.
Some other related works are discussed as follows.
Lower bounds and optimal algorithms.
The lower bounds and optimal algorithms for stochasticand convex optimization has been intensively studied in [25, 26, 42, 1, 11].
References [5, 6]established the lower bounds to find stationary solutions for non-convex optimization, and [13, 3]proposed algorithms with near-optimal convergence rate for non-convex finite-sum minimization.
Indeterministic decentralized optimization, the optimal convergence rate was established for stronglyconvex optimization [44], non-smooth convex optimization [45], and non-convex optimization [49].
Furthermore, authors in [23] proved the lower bounds over time-varying topologies.
In stochastic decentralized optimization, works [61] and [20] established the lower bounds for strongly-convexdecentralized SGD under the homogeneous and heterogeneous settings, respectively.
[33] proves theoptimal complexities for non-convex problems when weight matrices satisfy β = cos( π/n ) .
Ring-lattice Graph.
The ring-lattice graph was used in [54, 14, 55] as a basis model to generate thesmall world graph.
However, its relation between graph diameter and connectivity measure is notclarified to our knowledge.
A related work [56] examines the robustness of the ring-lattice graph,but does not discuss its connectivity measure.
Furthermore, ring-lattice has never been utilized toestablish the influence of the weight matrix on the optimal rate in decentralized optimization before.
Problem setup
Problem setup.
Consider the following problem with a network of n computing nodes:
[Equation]
Function f i ( x ) is local to node i , and random variable ξ i denotes the local data that follows distribution
D i .
Each local data distribution D i can be different across all nodes.
Assumptions.
The optimal convergence rate is established under the following assumptions.
• Function class.
We assume each local loss function is L -smooth, i.e.,
[Equation]
for some constant L > 0 .
We let the class F L denote the set of all functions satisfying (2).
We
may also assume the global loss f ( x ) = 1 n (cid:80) ni
=
f i ( x ) satisfies the µ -PL condition:
[Equation]
where f ⋆ is the optimal function value in problem (1) and µ > 0 is some constant.
We let the
function class F L,µ denote the set of all functions satisfying both (2) and (3).
• Oracle class.
We assume each node i interacts with f i ( x ) via the stochastic gradient oracle˜ g i ( x, ξ ) where ξ is a random variable.
In addition, we assume oracle ˜
g
i ( x, ξ ) satisfies
[Equation]
i
[Equation]
x
[Equation]
f i
[Equation]
x
[Equation] and [Equation] We let O σ 2 denote the class of all such stochastic gradient oracles.
• Weight matrix class.
We let W n,β denote the class of all gossip matrices W ∈ R n × n satisfying
[Equation]
where 1 n = (1 , . .
. , 1) T ∈ R n with all entries being 1 .
If W is positive semi-definite, thenconnectivity measure β is its second largest eigenvalue.• Algorithm class.
We consider an algorithm A in which each node i assesses an unknown local function f i via the stochastic gradient oracle ˜
g
i ( x ; ξ i ) ∈
O σ 2 .
Each node i running algorithm
A will maintain a local model copy x
( k ) i at iteration k .
We assume A to follow the partial averaging policy, i.e., each node communicates via protocol
[Equation]
for some W = [ w ij ] ni,j =1 ∈ W n,β where [ n ] := { 1 , · · · , n } , and y and z are the input and outputvariables of the communication protocol.
In addition, we assume A to follow the zero-respecting
policy [5, 6].
Informally speaking, the zero-respecting policy requires that the number of non-zeroentries of local model copy x ( k ) i can only be increased by sampling its own stochastic gradientoracle or interacting with the neighboring nodes.
The zero- respecting policy is widely followedby all methods in Tables 1 and 2 and their momentum and adaptive variants.
We let A W be theset of all algorithms following the partial averaging and zero-respecting policies.
Lower bound metrics.
With the above classes, we now introduce the measures for convergencerate’s lower bound.
Given loss functions { f i } ni =1 ⊆ F L (or { f i } ni =1 ⊆ F L,µ ), stochastic gradientoracles { ˜ g i } ni =1 ⊆
O σ 2 , a weight matrix W ∈ W n,β , an algorithm A ∈
A W , the number of gradientqueries (or communications rounds) T , we let ˆ x A, { f i } ni =1 , { ˜ g i } ni =1 ,W,T (which will be denoted as ˆ xwhen there is no ambiguity) be the output of algorithm A with ˆ x ∈ Span (cid:0) {{ x ( t ) j } nj =1 } Tt =1 (cid:1) .
Thelower bound for smooth and non-convex stochastic decentralized optimization is defined as
[Equation]
The lower bound under the PL condition is defined as
[Equation]
Fixed network size n .
This paper considers stochastic decentralized optimization where the networksize n is a fixed constant .
A fixed n is a prerequisite in distributed stochastic optimization whichenables distributed algorithms to achieve the linear speedup in convergence rate O ( σ/ √ nT ) , see thealgorithms listed in Tables 1 and 2.
In decentralized deterministic optimization, however, size n doesnot appear in the convergence rate.
Thus, it does not need to be specified and can be varied freelywhen establishing the lower bounds [44, 45].
Ring-Lattice graph
The main challenge in analyzing theoretical limits in decentralized optimization lies in clarifying theinfluence of decentralized network topology and its associated weight matrix.
It is intuitive that thegraph diameter D affects the convergence rate.
In the worst case, one node needs to take at least Dcommunication steps to transmit its own gradient information to the other.
If the relation between Dand β is established, the influence of the weight matrix on lower bounds will become clear.
However,existing results either establishes suboptimal relation between D and β [49] or admits restrictiveweight matrices [44, 45, 33].
Now we show that the ring-lattice graph maintains the optimal relationbetween D and β for a broad class of weight matrices.
All proof details are in Appendix B.
Definition 1 (R ING -L ATTICE G RAPH [55]) .
Given any positive integers n, k such that k is evenand 2 ≤ k < n − 1 , the k -regular ring-lattice graph over nodes { 1 , · · · , n } , denoted by R n,k , is anundirected graph in which each node i ∈ [ n ] is connected with the set of nodes { ( i + ℓ ) mod n : 1 ≤| ℓ | ≤ k/ 2 } where ( i + ℓ ) mod n is defined as( i + ℓ ) mod n =   i + ℓ if 1 ≤ i + ℓ ≤ n ; i + ℓ + n if i + ℓ < 1; i + ℓ − n if i + ℓ > n.
Apparently, each node in R n,k has degree k .
Ring-lattice is a k -regular graph.
When k = 2 , ring-lattice will reduce to the ring graph.
While
Definition 1 excludes the case in which k
= n − 1 since n − 1 is not necessarily a even number, thecomplete graph can be exceptionally viewed as the ring lattice with k = n − 1 .
Therefore, for general2 ≤ k ≤ n − 1 , R n,k can be regarded as an intermediate state between the ring and complete graphs,see the illustration in Fig.
The following lemma clarifies the diameter of R n,k .
Lemma 1.
The diameter of the ring-lattice graph R n,k is D
=
Θ
( n/k ) .
It is derived from the above lemma that the diameters of R n, 2 and R n,n − 1 are Θ( n ) and Θ(1) ,respectively, which are consistent with the ring and complete graphs.
The following fundamental theorem establishes the relation between D and connectivity measure β .
Theorem 1.
Given a fixed n and any β ∈ [0 , cos( π/n )] , we can always construct a ring-latticegraph R n,k so that
(1) it has an associated weight matrix W ∈ W n,β , i.e., W ∈ R n × n and ∥ W − 1 n 1 n 1 Tn ∥ =
β , (2) its diameter D satisfies D = Θ(1 / √ 1 − β ) .
Remark 1.
The works [44, 45, 33] utilize the linear graph to establish the same relation D =Θ(1 / √ 1 − β ) .
However, given fixed network size n , the linear graph admits very limited weightmatrices with β = cos( π/n ) .
Remark 2.
Sun and Hong [49] propose a slightly wider class of networks named linear-star graph,which, however, leads to a suboptimal relation D = Ω(1 /
[ n (1 − β )]) [49, Proof of Corollary 3.1].
Remark 3.
It is worth noting that for deterministic decentralized optimization where n may vary, onecan tune n to make β = cos( π/n ) lie in the full interval (0 , 1) as shown in [44, 45].
In other words,[44, 45] proved that D = Θ(1 / √ 1 − β ) for any β ∈ (0 , 1) if n is a varying parameter.
However,this result is not valid in stochastic scenarios where n is required to be fixed.
Lower bounds with general weight matrices
This section will establish the lower bounds for smooth and non-convex decentralized stochasticoptimization.
All proof details are in Appendix C. When the cost function does not satisfy the PLcondition (3), we will derive that the convergence rate is lower bounded by Ω( σ √ nT + DT ) .
This resulttogether with Theorem 1 leads to:
Theorem 2.
For any L > 0 , n ≥ 2 , β ∈ [0 , cos( π/n )] , and σ > 0 , there exists a set of loss functions{ f i } ni =1 ⊆ F L , a set of stochastic gradient oracles { ˜ g i } ni =1 ⊆
O σ 2 , and a weight matrix W ∈ W n,β ,such that it holds for any A ∈ A W starting form x (0)
that
[Equation]
Remark 4 ( Comparison with existing bounds).
Comparing to the lower bound established in[33], Theorem 2 admits a much broader class of weight matrices with β ∈ [0 , cos( π/n )] .
While theinterval [0 , cos( π/n )] ⊂ [0 , 1) , it approaches to [0 , 1) as n goes large.
Such interval is broad enoughto cover most weight matrices (generated through the Laplacian rule W =
I − L/d max ) resultedfrom common topologies such as line, ring, grid, torus, hypercube, exponential graph, completegraph, Erdos-Renyi graph, geometric random graph, etc.
whose β lies in the interval [0 , cos( π/n )]when n is sufficiently large, see details in Appendix C.
Remark 5 ( Linear speedup).
The first term 1 / √ nT dominates (8) when T is sufficiently large,which will require T
= O
(1 / ( nϵ 2 ))
(it is inversely proportional to n ) iterations to reach accuracy ϵ .
Therefore, if the term involving nT is dominating the rate for some T , we say the algorithm is in itslinear-speedup stage.
Linear speedup is a fundamental property that distributed algorithms enjoy.
Remark 6 ( Transient iteration complexity).
Due to the topology-incurred overhead in convergence rate, a decentralized algorithm has to experience a transient stage to achieve linear-speedup.
Transient iterations are referred to those iterations before an algorithm reaches linear-speedupstage.
To reach linear speedup in the lower bound (8) , T has to satisfy [Equation] σ √ nT and hence
T =
O ( n (1 − β ) σ 2 ) , which is the transient iteration complexity shown in the first line in Table 1.
Thesmaller the transient iterations are, the faster the algorithm can achieve linear speedup.
Transientiteration complexity is an important metric to evaluate how sensitive the algorithm is to β .
The lower bound in (8) ignores constants L and f ( x (0) )
− f ⋆ since the influences of gradient noiseσ , network size n , and network topology β on the convergence rate typically attract more attentions.
A lower bound with constants L and f ( x (0) )
− f ⋆ is given in Appendix C.
Require:
Initialize x (0) i = 0 , the weight matrix W , and the rounds of gossip
R .
for k = 0 , 1 , 2 , ..., K , every node i do
Sample { ξ ( k,r ) i } Rr =1 independently and let g ( k ) i = 1 R R (cid:80) r =1 ∇ F ( x ( k ) i ; ξ ( k,r ) i ) ; ▷ grad.
accumulation
Update ϕ
( k +1)
i =
x ( k ) i − γg ( k ) i ; ▷ local stochastic gradient descent
Update x ( k +1)
i =
FastGossipAverage ( { ϕ ( k +1) i } ni =1 , W, R ) ;
▷ multiple gossips
Algorithm i }
=1 , W, Require: { ϕ i } ni =1 , W , R , and β ; let z (0) i =
z ( − 1) i =
ϕ
i and η = 1 − √ 1 − β1+ √ 1+ β 2 .
for r = 0 , 1 , 2 , ..., R − 1 , every node i do
Update z ( r +1) i
= (1 + η ) (cid:80) j ∈N i w ij z
( r ) i − ηz
( r − 1) i ; ▷ fast gossip averaging
Output: x i =
z ( R ) i ;
When the cost function satisfies PL condition (3), we prove the convergence rate is lower bounded byΩ( σ √ nT + exp( − CT/D )) where C is some constant.
This result together with Theorem 1 leads to:
Theorem 3.
For any L > 0 , n ≥ 2 , β ∈ [0 , cos( π/n )] , and σ > 0 , there exists a set of loss 210 functions { f i } ni =1 ⊆ F L,µ , a set of stochastic gradient oracles { ˜ g i } ni =1 ⊆
O σ 2 , and a weight matrix
W ∈ W n,β , such that it holds for any A ∈ A W
starting from x (0) that
[Equation]
where C is independent of n , T , σ , and β .
Rate (9) results in an ˜ O
((1 − β ) − 12 ) transient complexity.
Expression (9) is the first lower bound for smooth and non-convex stochastic decentralized optimization under the PL condition to our knowledge.
A detailed lower bound with constants L and µ isgiven in Appendix C.
In fact, we can futher show that this lower bound also applies to strongly-convexstochastic decentralized optimization, see the construction details in Appendix C.
DSGD with multiple gossip communications
This section will present a decentralized algorithm that achieve the lower bounds established in Sec.
4up to logarithmic factors.
The new algorithm is a direct extension of the vanilla decentralized SGD(DSGD)
Inspired by the algorithm development in [33], we add two additional componentsto DSGD: gradient accumulation and multiple-gossip communication.
The main recursions are listedin Algorithm 1 which utilizes the fast gossip average step [32] in Algorithm 2.
We call the newalgorithm as MG-DSGD where “MG” indicates “multiple gossips”.
All proofs are in Appendix D.
The DeTAG algorithm proposed in [33] applies the gradient accumulation and multiple-gossipcommunication techniques to gradient tracking, which achieves the optimal rate in non-convex(without PL condition) stochastic decentralized optimization.
Surprisingly, we find in MG-DSGDthat the multiple gossip technique can significantly reduce the influence of data heterogeneity, and thegradient tracking step can be saved to achieve the convergence optimality.
Removing gradient trackingfrom algorithm updates brings two benefits.
First, it saves half of the communication overhead andmemory storage per iteration.
Second, it enables numerous effective acceleration techniques designedexclusively for decentralized SGD such as Quasi-global momentum [31], DecentLaM [62], adaptivemomentum [36], etc. which cannot be easily extended to gradient tracking.
Let T be the number of gradient queries (or decentralized communications) at each node after
MG-DSGD finishes. Since each node takes R gradient queries and R communications at round k , it holds that T = KR when MG-DSGD finishes after K rounds.
The following theorems clarify theconvergence rate of MG-DSGD where T = KR .
We omits L and initialization constants below.
Theorem 4.
Given L > 0 , n ≥ 2 , β ∈ [0 , 1) , σ > 0 , and let A denote Algorithm 1.
Assuming that1 n (cid:80) ni =1 ∥∇ f i ( x ) − ∇ f ( x ) ∥ 2 ≤ b 2 for any x ∈ R n and the number of gossip rounds R is set as in
Appendix D, the convergence of A can be bounded for any { f i } ni =1 ⊆ F L and any W ∈ W n,β that
[Equation]
where ¯ x ( k ) = 1 n n (cid:88) i =
x ( k ) i .
The rate (10) matches with the lower bound (8) up to logarithm factors.
The comparison between MG-DSGD with other state-of-the-art algorithms is listed in Table 1. MG-DSGD matches DeTAG inconvergence rate and saves half of the communications, see experiments in Fig.
While the analysisin MG-DSGD needs to assume bounded data heterogeneity b 2 , it only affects the convergence rate asa logarithm term hidden in the ˜ O ( · ) notation.
Theorem 5.
Under the same assumptions as in Theorem 4 and setting R as in Appendix D, the 246 convergence of A can be bounded for any loss functions { f i } ni =1 ⊆
F L,µ , and any W ∈ W n,β by
[Equation]
It is observed that the rate (11) matches with the lower bound (9) up to a logarithm factor.
Thecomparison between MG-DSGD with other state-of-the-art algorithms is listed in Table 2.
It is notknown in [33] whether DeTAG can achieve the rate in (11) when PL condition is satisfied.
Experiments
This section will validate our theoretical results by empirically comparing different decentralizedalgorithms DSGD [29], D 2 [51], DSGT [65], DeTAG [33] and MG-DSGD in deep learning.
Implementation setup.
We implement all decentralized algorithms with PyTorch [40] 1.6.0 using
NCCL 2.8.3 (CUDA 10.1) as the communication backend, in which the partial averaging primitiveis implemented by NCCL send/receive primitives.
For parallel SGD, we used PyTorch’s native
Distributed Data Parallel (DDP) module.
All the models and training scripts in this section run onservers with 8 NVIDIA V100 GPUs with each GPU treated as one node.
Similar to existing {+works[
 ],+} we use Ring graph throughout all experiments.
Tasks and training details.
A series of experiments are carried out with CIFAR-10 [24] and
ImageNet [9] to compare the aforementioned methods.
For CIFAR-10 dataset, it consists of 50,000training images and 10,000 validation images in 10 classes. For ImageNet dataset, it consists of1,281,167 training images and 50,000 validation images in 1000 classes. We utilize ResNet variants[15] on CIFAR-10 (ResNet-20 with 0.27M parameters and ResNet-18 with 11.17M parameters) and ResNet-50 on ImageNet.
For training protocol, we train total 300 epochs and the learning rate iswarmed up in the first 5 epochs and is decayed by a factor of 10 at 150 and 250-th epoch.
For learningrate, we tuned a strong baseline in the PSGD setting (5e-3 for single node) and used the same settingin all decentralized methods.
The batch size is set to 128 on each node.
All CIFAR-10 experimentsare repeated three times with different seeds.
For fair comparison, we maintain the total number ofgradient queries the same for algorithms in experiments, i.e., T = KR .
Performance with homogeneous data.
Table 3 compares the proposed MG-DSGD with centralizedparallel SGD and other baselines with homogeneous data distributions.
For DeTAG and MG-DSGD,the communication round R is set as 2.
We also performed an ablation study on the effect ofmomentum acceleration.
Except for the ResNet-20 scenario with momentum in which MG-DSGDand DeTAG achieve the same performance, MG-DSGD consistently outperforms baseline methodsin other scenarios.
Moreover, MG-DSGD is more communication-efficient than DSGT and DeTAG.
Performance with heteogeneous data. Following previous works [64, 31], we simulate dataheterogeneity among nodes with a Dirichlet distribution-based partitioning. A quantity α controlsthe degree of the data heterogeneity. The training data for a particular class tends to concentrate in a
[Table] [Table]
[Figure]
single node as α → 0 , i.e. becoming more heterogeneous, while the homogeneous data distributionis achieved as α → ∞ .
We tested α = 0 .
1 / 1 / 10 for all compared methods as corresponding to asetting with high/mild/low heterogeneity.
As shown in Table 4, DeTAG and MG-DSGD outperformthe other baselines by visible margins, and MG-DSGD is with slightly better accuracies.
The leftplot in Fig.
2 illustrates that MG-DSGD achieve a competitive accuracy to DeTAG using much lesscommunication budgets on CIFAR-10 with α = 1 .
[Table]
Large-batch training in ImageNet.
To train ResNet-50 on ImageNet,utilization of the large-batch is necessary to speedup the trainingprocess.
Since MG-DSGD is based on DSGD, we can extend thelarge-batch acceleration techniques introduced in [62, 31] to MGDSGD to achieve better performance.
These techniques are developedexclusively for DSGD and cannot be integrated to DeTAG easily.
Following [62], we train total 120 epochs.
The learning rate is warmedup in the first 20 epochs and is decayed in a cosine annealing scheduler.
The results are listed in Table 5, showing that MG-DSGD has better accuracy than other approaches.
Fig. 2(right) illustrates how different algorithms perform in terms of communication budgets used.
More results.
In Appendix E, we depict the accuracy curves in terms of epochs (rather thancommunication volumes).
The ablation on the influence of inner loop R is also listed there.
Conclusion
This paper revisits the theoretical limits in smooth and non-convex stochastic decentralized optimization.
Existing lower bound proved in [33] is only valid for special weight matrices with connectivitymeasure β = cos( π/n ) .
It is not known what the lower bound is when using weight matrices withother β .
To address this issue, this paper establishes the lower bound for any weight matrix withβ ∈ [0 , cos( π/n )] .
We also clarify the lower bound when loss functions satisfy the PL condition.
Anew algorithm is proposed to match these lower bounds up to logarithm factors.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See Section[-
-]
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Section 2.
(b) Did you include complete proofs of all theoretical results?
[Yes] See appendix.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[No]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes] See Section 6.
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[Yes] See Section 6.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See Section 6.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] See Section 6.
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
[Table]
[Table] [Table]