Beyond spectral gap: The role of the topology in decentralized learning
Anonymous Author(s)
AfﬁliationAddress email
Abstract
In data-parallel optimization of machine learning models, workers collaborateto improve their estimates of the model: more accurate gradients allow them touse larger learning rates and optimize faster.
We consider the setting in whichall workers sample from the same dataset, and communicate over a sparse graph(decentralized).
In this setting, current theory fails to capture important aspectsof real-world behavior.
First, the ‘spectral gap’ of the communication graph isnot predictive of its empirical performance in (deep) learning.
Second, currenttheory does not explain that collaboration enables larger learning rates than trainingalone.
In fact, it prescribes smaller learning rates, which further decrease as graphsbecome larger, failing to explain convergence in inﬁnite graphs.
This paper aimsto paint an accurate picture of sparsely-connected distributed optimization whenworkers share the same data distribution.
We quantify how the graph topologyinﬂuences convergence in a quadratic toy problem and provide theoretical resultsfor general smooth and (strongly) convex objectives.
Our theory matches empiricalobservations in deep learning, and accurately describes the relative merits ofdifferent graph topologies.
Introduction
Distributed data-parallel optimization algorithms help us tackle the increasing complexity of machinelearning models and of the data on which they are trained.
We can classify those training algorithms aseither centralized or decentralized and we often consider those settings to have different beneﬁts overtraining ‘alone’.
In the centralized setting, workers compute gradients on independent mini-batchesof data and they average those gradients between all workers.
The resulting lower variance in theupdates enables larger learning rates and faster training.
In the decentralized setting, workers averagetheir models with only a sparse set of ‘neighbors’ in a graph instead of all-to-all, and they may haveprivate datasets sampled from different distributions.
As the beneﬁt of decentralized learning, weusually focus only on the (indirect) access to other worker’s datasets, and not of faster training.
While decentralized learning is typically studied with heterogeneous datasets across workers, sparse(decentralized) averaging between is also useful when worker’s data is identically distributed(i.i.d.)
As an example, sparse averaging is used in data centers to mitigate communication bottlenecks [1].
In fact the D-SGD algorithm [11], on which we focus in this work, performs well mainlyin this setting, while algorithmic modiﬁcations [14, 19, 20] are required to yield good performanceon heterogeneous objectives.
When the data is i.i.d., the goal of sparse averaging is to optimize faster,just like in centralized (all-to-all) graphs.
Yet, current decentralized learning theory poorly explains the i.i.d. case.
Analyses typically show that,for small enough learning rates, training with sparse averaging behaves the same as with all-to-allaveraging [11, 8]. Compared to training alone with the same small learning rate , all-to-all averaging
￿￿ ￿￿￿ ￿￿￿￿ " Steps until loss < ￿ .
￿￿ ￿ . ￿￿￿ ￿ .
￿￿ ￿ .
￿ ￿ Learning rate !
Fully connected
Ring
Alone (disconnected)
Current theory uses lower learning rates but decentralized averaging enables higher learning rates ￿ .
￿￿￿ ￿ .
￿￿ ￿ .
￿ ￿ Learning rate !
￿ -ring (spectral gap ￿ ) ￿ -ring (spectral gap ￿ )
￿ -ring (spectral
[Figure]
￿ .6 ￿ )
[Figure]
[Figure]
.g.
￿ . ￿￿ )
[Figure] [Figure]
reduces the gradient variance by the number of workers.
In practice, however, such small learningrates would never be used.
In fact, a reduction in variance should allow us to use a larger learningrate than training alone, rather than imposing a smaller one.
Contrary to current theory, we showthat averaging reduces the variance from the start, instead of just asymptotically.
Lower varianceincreases the maximum learning rate, which directly speeds up convergence.
We characterize howmuch averaging with various communication graphs reduces the variance, and show that centralizedperformance is not always achieved when using optimal large learning rates.
The behavior we explainis illustrated in Figure 1.
In current convergence rates, the graph topology appears through the spectral gap of its averaging(gossip) matrix.
The spectral gap poses a conservative lower bound on how much one averagingstep brings all worker’s models closer together.
The larger, the better.
If the spectral gap is small,a signiﬁcantly smaller learning rate is required to make the algorithm behave close to SGD withall-to-all averaging with the same learning rate.
Unfortunately, we experimentally observe that, bothin deep learning and in convex optimization, the spectral gap of the communication graph is notpredictive of its performance under realistically tuned learning rates.
The problem with the spectral gap quantity is clearly illustrated in a simple example.
Let thecommunication graph be a ring of varying size.
As the size of the ring increases to inﬁnity, its spectralgap goes to zero, since it becomes harder and harder to achieve consensus between all the workers.
This leads to the optimization progress predicted by current theory to go to zero as well.
Yet, thisbehavior does not match the empirical behavior of the rings with i.i.d. data.
As the size of the ringincreases, the convergence rate actually improves (Figure 1), until it saturates at a point that dependson the problem.
In this work, we aim to accurately describe the behavior of i.i.d.
distributed learning algorithms withsparse averaging, both in theory and in practice.
• We quantify the role of the graph in a quadratic toy problem designed to mimic the initial phase of deep learning (Section 3), showing that averaging enables a larger learning rate.
• From these insights, we derive a problem-independent notion of ‘effective number of
neighbors’ in a graph that is consistent with time-varying topologies and inﬁnite graphs, and is predictive of a graph’s empirical performance in both convex and deep learning.
• We provide convergence proofs for convex and (strongly) convex objectives that only mildly
depend on the spectral gap of the graph (Section 4), and consider the whole spectrum instead. At
its core, our analysis does not enforce global consensus, but only between workers that
are close to each other in the graph.
Our theory for the ﬁrst time shows that sparse averaging
provably enables larger learning rates and thus speeds up optimization.•
We verify that our insights are relevant in deep learning.
We can accurately describe the performance of a variety of topologies, while their spectral gap does not (Section 5).
Related work
Decentralized SGD
This paper studies decentralized SGD.
Koloskova et al.
[8] obtain the tightestbounds for this algorithm in the general setting where workers optimize heterogeneous objectives.
Contrary to their work, we focus primarily on the case where all workers sample i.i.d.
data from thesame distribution.
This important case is not described in a meaningful way by their analysis: whilethey show that gossip averaging reduces the asymptotic variance suffered by the algorithm, the fastinitial linear decrease term in their convergence rate depends on the spectral gap of the gossip matrix.
This key term does not improve through collaboration and gives rise to a smaller learning rate thantraining alone.
Besides, as discussed above, this implies that optimization is not possible in the limitof large graphs, even in the absence of heterogeneity: for instance, the spectral gap of an inﬁnite ringis zero, which would lead to a learning rate of zero as well.
These rates suggest that decentralized averaging speeds up the last part of training (dominated byvariance), at the cost of slowing down the initial (linear convergence) phase.
Beyond the workof Koloskova et al.
[8], many papers focus on linear speedup (in the variance phase) over optimizingalone, and prove similar results in a variety of settings [11, 18, 12].
All these results rely on thefollowing insight: while linear speedup is only achieved for small learning rates, SGD eventuallyrequires such small learning rates anyway (because of, e.g., variance, or non-smoothness).
Thisobservation leads these works to argue that “topology does not matter”.
This is the case indeed, butonly for very small learning rates, as shown in Figure 1.
In practice, averaging speeds up both theinitial and last part of training.
This is what we
Gossiping in inﬁnite graphs An important feature of our results is that they only mildly dependon the spectral gap, and so they apply independently of the size of the graph.
Berthier et al.
[3]study acceleration of gossip averaging in inﬁnite graphs, and obtain the same conclusions as we do:although spectral gap is useful for asymptotics, it fails to accurately describe the transient regime ofaveraging.
This is especially limiting for optimization (compared to of just averaging), as new localupdates need to be averaged at every step.
The transient regime of averaging deeply matters.
Indeed,it impacts the quality of the gradient updates, and so it rules the asymptotic regime of optimization.
The impact of the topology Some of the works on linear speedup [11] argue that the topology ofthe graph does not matter.
This is only true for asymptotic rates in speciﬁc settings, as illustrated in
Figure 1.
Neglia et al.
[16] investigate the impact of the topology on decentralized optimization, andcontradict this claim.
Compared to us, they make different noise assumptions, which in particulardepend on the spectral distribution of the noise over the eigenvalues of the Laplacian (thus mixingcomputation and communication aspects).
Although they show that the topology has an impact in theearly phases of training (just like we do), they still get an unavoidable dependence on the spectral gapof the graph.
Our results are different in nature, and show the beneﬁts of averaging and the impact ofthe topology through the choice of large learning rates.
Another line of work studies the interaction of topology with particular patterns of data heterogeneity [4, 2], and how to optimize graphs with this heterogeneity in mind.
These works “only” show abeneﬁt from one-step gossip averaging and this is thus what they optimize the graph for.
In contrast,we show that it is possible to beneﬁt from distant workers beyond direct neighbors, too.
This is anorthogonal direction, though the insights from our work could be used to strengthen their results.
Time-varying topologies Time-varying topologies are popular for decentralized deep learning indata centers due to their strong mixing [1, 21].
The beneﬁt of varying the communication topologyover time is not easily explained through standard theory, but requires dedicated analysis [23].
Whileour proofs only cover static topologies, the quantities that appear in our analysis can be computed fortime-varying schemes, too.
With these quantities, we can empirically study static and time-varyingschemes in the same framework.
A toy problem: D-SGD on isotropic random quadratics
Before analyzing decentralized stochastic optimization through theory for general convex objectivesand deep learning experiments, we ﬁrst investigate a simple toy example that illustrates the behaviorwe want to explain in the analysis.
In this setting, we can exactly characterize the convergence ofdecentralized SGD.
We also introduce concepts that will be used throughout the paper.
We consider n workers that jointly optimize an isotropic quadratic E d ⇠ N d (0 , 1) 12 ( d > x ) 2 = 12 k x k 2 125 with a unique global minimum x ?
= 0 .
The workers access the quadratic through stochastic gradientsof the form g ( x ) = dd > x , with d ⇠ N d (0 , 1) .
This corresponds to a linear model with inﬁnitedata, and where the model can ﬁt the data perfectly, so that stochastic noise goes to zero close tothe optimum.
We empirically ﬁnd that this simple model is a meaningful proxy for the initial phaseof (over-parameterized) deep learning (Section 5).
A beneﬁt of this model is that we can computeexact rates for it.
These rates illustrate the behavior that we capture more generally in the theory of
Section 4.
Appendix C contains a detailed version of this section that includes full derivations.
The stochasticity in this toy problem can be quantiﬁed by the noise level
[Equation]
which is equal to ⇣ = d + 2 , due to the random normal distribution of d .
The workers run the D-SGD algorithm [11].
Each worker i has its own copy x i 2 R d of the model,and they alternate between local model updates x i x i  ⌘ g ( x i ) and averaging their modelswith others: x i
P nj =1 w ij x j .
The averaging weights w ij are summarized in the gossip matrix
W 2 R n ⇥ n .
A non-zero weight w ij indicates that i and j are directly connected.
In the following,we assume that W is symmetric and doubly stochastic:
[Equation]
On our objective, D-SGD either converges or diverges linearly.
Whenever it converges, i.e. when thelearning rate is small enough, there is a convergence rate r such that
[Equation]
with equality as t ! 1 (proofs in Appendix C).
When the workers train alone ( W = I ), theconvergence rate for a given learning rate ⌘ reads:
[Equation]
The optimal learning rate ⌘ ?
= 1 ⇣ balances the optimization term (1  ⌘ ) 2 and the stochastic term( ⇣  1) ⌘ 2 .
In the centralized (fully connected) setting ( w ij = 1 n 8 i, j ), the rate is simple as well:
[Equation]
Averaging between n workers reduces the impact of the gradient noise, and the optimal learning rategrows to ⌘ ?
=
n n + ⇣  1 .
D-SGD with a general gossip matrix W interpolates those results.
To quantify the reduction of the ( ⇣  1) ⌘ 2 term in general, we introduce the problem-independentnotion of effective number of neighbors n W (  ) of the gossip matrix W and decay parameter  .
Deﬁnition A.
The effective number of neighbors n W (  ) =
lim t !
{+1 P ni =
+}
Var[ y ( t ) i ]
[Equation]
ni
[Equation]
1 Var[ z ( t ) i ] measures theratio of the asymptotic variance of the processes
[Equation]
where y ( t ) 2 R n and ⇠ ( t ) ⇠ N n (0 , 1) (4) and
[Equation]
where z ( t ) 2 R n and ⇠ ( t ) ⇠ N n
[Equation] [Equation]
We call y and z random walks because workers repeatedly add noise to their state, somewhat like
SGD’s parameter updates.
It should not be confused with a ‘random walk’ over nodes in the graph.
Since averaging with W decreases the variance of the random walk by at most n , the effective numberof neighbors is a number between 1 and n .
The decay  modulates the sensitivity to communicationdelays.
If  = 0 , workers only beneﬁt from averaging with their direct neighbors.
As  increases,
[Figure]
[Figure] ￿￿ ￿￿ [Figure] multi-hop connections play an increasingly important role.
As  approaches 1, delayed and undelayednoise contributions become equally weighted, and the reduction tends to n for any connected topology.
For regular doubly-stochastic symmetric gossip matrices W with eigenvalues  1 , . . . ,  n , n W (  )has a closed-form expression [Equation] The notion of variance reduction in random walks, however, naturally extends to inﬁnite topologiesor time-varying averaging schemes as well.
Figure 2 illustrates n W for various topologies.
In our exact characterization of the convergence of D-SGD on the isotropic quadratic toy problem(Appendix C), we ﬁnd that the effective number of neighbors appears in place of the number ofworkers n in the fully-connected rate of Equation 3. The rate is the unique solution to [Equation] For fully-connected and disconnected W , n W (  ) = n or 1 respectively, irrespective of  , and
Equation 7 recovers Equations 2 and 3. For other graphs, the effective number of workers depends onthe learning rate. Current theory only considers the case where n W ⇡ n , but the small learning ratesthis requires can make the term (1  ⌘ ) 2 too large, defeating the purpose of collaboration by leadingto smaller rates r than alone. Beyond this toy problem, we ﬁnd that the proposed notion of effective number of neighbors is alsomeaningful in the analysis of general objectives (Section 4) and in deep learning (Section 5).
Theoretical analysis
In the previous section, we have derived exact rates for a speciﬁc function.
Now we presentconvergence rates for general (strongly) convex functions that are consistent with our observations inthe previous section.
We obtain rates that depend on the level of noise, the hardness of the objective,and the topology of the graph.
We will assume the following randomized model for D-SGD:
[Equation]
with probability 12 ,
[Equation]
where f ⇠ ( t ) i represent sampled data points and the gossip weights w ij are elements of W .
This randomized model yields a clean analysis, but similar results hold for standard D-SGD (Appendix D.4).
Assumption B.
The stochastic gradients are such that: ( I ) ⇠ ( t ) i and ⇠ ( ` ) j are independent for all t, ` andi
= j . ( II ) E [ f ⇠ ( t ) i ]
= f for all t, i ( III ) E kr f
⇠ ( t ) i ( x ? )
k 2   2 for all t, i , where x ? is a minimizer off .
( IV ) f
⇠ ( t ) i is convex and ⇣ -smooth for all t, i .
( V ) f is µ -strongly-convex for µ  0 and L -smooth.
The smoothness ⇣ of the stochastic functions f ⇠ deﬁnes the level of noise in the problem (the lower,the better).
The ratio ⇣ /L compares the difﬁculty of optimizing with stochastic gradients to thedifﬁculty with the true global gradient (before reaching the ‘variance region’ of distance
O
(  2 ) to theoptimum).
Assuming better smoothness for the global average objective than for the local functionsis key to showing the beneﬁt of averaging between workers.
Without communication, convergenceto the variance region is ensured for learning rates ⌘  1 / ⇣ .
If ⇣ ⇡ L , there is little noise andcooperation does not help before k x ( t )  x ? k 2 ⇡  2 .
Yet, in noisy regimes ( ⇣  L ), such as in
Section 3 in which ⇣ =
d + 2  1 =
L , averaging enables larger step-sizes up to min(1 /L, n/ ⇣ ) ,greatly speeding up the initial training phase.
This is precisely what we prove in Theorem I.
If the workers always remain close ( x i ⇡ 1 n ( x 1 + . . .
+ x n ) 8 i , or equivalently 1 n 11 > x ⇡ x ), D-SGD behaves the same as SGD on the average parameter 1 n
P n i =1 x i , and the learning rate dependson max( ⇣ /n, L ) , showing a reduction of variance by n .
To maintain “ 1 n 11 > x ⇡ x ”, however, werequire a small learning rate.
This is a common starting point for the analysis of D-SGD, in particularfor the proofs in Koloskova et al.
On the other extreme, if we do not assume closeness betweenworkers, “ Ix ⇡ x ” always holds.
In this case, there is no variance reduction, but no requirement fora small learning rate either.
In Section 3, we found that, at the optimal learning rate, workers are notclose to all other workers, but they are close to others that are not too far away in the graph.
We capture the concept of ‘local closeness’ by deﬁning an averaging matrix M .
It allows us toconsider semi-local averaging beyond direct neighbors, but without fully averaging with the wholegraph.
We ensure that “ Mx ⇡ x ”, leading to some improvement in the smoothness between ⇣ and⇣ /n , interpolating between the two previous cases.
Each matrix M implies a requirement on thelearning rate, as well as an improvement in smoothness.
Based on Section 3, we therefore focuson a speciﬁc family of matrices that strike a good balance between the two: We choose M as thecovariance of a decay-  ‘random walk process’ with the graph, meaning that
[Equation]
Varying  induces a spectrum of averaging neighborhoods from M = W 2 (  = 0 ) to M = 1 n 11 > (  = 1 ).
also implies an effective number of neighbors n W (  ) : the larger  , the larger n W (  ) .
Theorem I provides convergence rates for any value of  , but the best rates are obtained for a speciﬁc that balances the beneﬁt of averaging with the constraint it imposes on closeness between neighbors.
In the following theorem, we assume that M ii =
M jj for all i, j , so that M ii  1 = n W (  ) : theeffective number of neighbours deﬁned in (6) is equal to the inverse of the self-weights of matrix M .
Otherwise, all results hold by replacing n W (  ) with min
i M ii  1 .
Theorem I.
If Assumption B holds, and the learning rate satisﬁes
[Equation]
then the iterates obtained by (8) verify
[Equation]
The bound on the learning rate (10) represents the tension between ( I ) reducing the noise ⇣ byaveraging with more people (larger n W (  ) ), which is the ﬁrst term in the minimum, and ( II ) stayingclose to all of them.
A large spectral gap 1   2 ( W ) reduces the second constraint, but we allownon-trivial learning rates ⌘ > [-
-]0 even when  2 ( W )
= 1 (inﬁnite graphs) as long as  < 1 .
￿ ￿￿ ￿￿ ￿￿ ￿￿ ￿￿ ￿ .
￿￿￿ ￿ .
￿￿￿ ￿ . ￿￿￿ ￿ .
￿￿￿ ￿ . ￿￿￿ ￿ .
￿￿￿ " Learning rate given by Theorem ￿ ( L = 1 . 0 , ⇣ = 2000 )
Effective number of neighbors n W (  ) !
￿ ￿￿ ￿￿ ￿￿ ￿￿ ￿￿ ￿ .
￿￿￿ ￿ .
￿￿￿ ￿ . ￿￿￿ ￿ .
￿￿￿ . ￿￿￿ .
￿￿￿
Effective number of neighbors n W (  ) !
Ring
Torus ( ￿ x8)
Hypercube
Restrictedbynoise
Restrictedbyconsensus
[Figure]
Theorem I gives a rate for each parameter  that controls the local neighborhood size. The task thatremains is to ﬁnd the  parameter that gives the best convergence guarantees (the largest learningrate). As explained before, one should never reduce the learning rate in order to be close to others,because the goal of collaboration is to increase the learning rate.
We should therefore pick  suchthat the ﬁrst term in Equation (10) dominates.
This intuition is summarized in Corollary II, whichcompares the performance of D-SGD with centralized SGD with fewer workers.
Corollary II.
D-SGD is as fast as centralized mini-batch SGD with O ( n W (  ))
workers, assumingthat ⇣  nL , and that the parameter  is the highest  such that 2 n W (  ) 2 1   2 ( W )  16 ⇣ L .
This correspondsto a learning rate ⌘ = n W (  ) / 8 ⇣ .
The typical D-SGD learning rates [8] are of order O (min(1 /T, 1   2 ( W ))) , which are much smallerthan the learning rate of Corollary II when  2 ( W ) is large or the number of iterations large.
Weuse the condition ⇣  nL only to present results in a simpler way.
The condition 2 n W (  ) 2 1   2 ( W ) onlydepends on the size and topology of the graph, and can easily be computed in many cases.
Thus, toobtain the best guarantees, we start from  = 0 and then increase it until either n W (  ) ⇡ n , the totalsize of the graph, or the two terms in the minimum match.
This is how we obtain Figure 3.
Proof sketch (Theorem I).
The proof relies on a simple argument: rather than bounding k x ( t )  x ?
k 2or k 1 n 11 > x ( t )  x ?
k 2 , we analyze k x ( t )  x ? k 2 M .
This term better captures the beneﬁt of averagingthan k x ( t )  x ? k 2 , thus leading to better smoothness constants, as long as k x ( t ) k
I  M is not too large.
This yields fast rates without the need to guarantee that iterates between very distant workers remainclose, which would be prohibitively expensive.
Theorem I is a special case of a more general theorem presented in Appendix D.
This version, amongother things, covers different choices of parameters, unbalanced communication and computationprobabilities (thus allowing for local steps), and the convex ( µ = 0 ) case.
Experimental analysis
While in the previous sections we have discussed isotropic quadratics or convex and smooth functions,the initial motivation for this work comes from observations in deep learning.
First, it is crucialin deep learning to use a large learning rate in the initial phase of training [10].
Contrary to whatcurrent theory prescribes, we do not use smaller learning rates in decentralized optimization thanwhen training alone (even when data is heterogeneous.)
And second, we ﬁnd that the spectral gap ofa topology is not predictive of the performance of that topology in deep learning experiments.
￿ .
￿ ￿ . ￿ ￿ .
￿￿ ￿ . ￿￿ ￿ .
￿ ￿ . ￿￿￿ ￿ .
￿￿ ￿ .
￿ " Cifar- ￿￿ training loss after ￿ .
￿ k steps ( ⇠ ￿￿ epochs)
Learning rate !
Binary tree
Fully connected
Hypercube
Ring
Social network
Solo
Star
Time-varying exponential
Torus ( ￿ x8)
Two cliques
G o ss i p m a t r i x
M e a s u r e
d
c
o v .
o n C
i f a r - ￿ ￿
C o v a r i a n c e i n r a n d o m w a l k
[Figure]
and corresponding ‘effective number of neighbors’.
In this section, we experiment with a variety of 32-worker topologies on Cifar-10 [9] with a VGG-11model [17].
Like other recent works [13, 20], we opt for this older model, because it does notinclude BatchNorm [6] which forms an orthogonal challenge for decentralized SGD.
Please refer to
Appendix E for full details on the experimental setup.
Our set of topologies (Appendix B) includesregular graphs like rings and toruses, but also irregular graphs such as a binary tree [20] and socialnetwork [5], and a time-varying exponential scheme [1].
We focus on the initial phase of training,25k steps in our case, where both train and test loss converge close to linearly.
Using a large learningrate in this phase is found to be important for good generalization [10].
Figure 4 shows the loss reached after the ﬁrst 2.5k SGD steps for all topologies and for a dense gridof learning rates.
The curves have the same global structure as those for isotropic quadratics Figure 1:(sparse) averaging yields a small increase in speed for small learning rates, but a large gain overtraining alone comes from being able to increase the learning rate.
The best schemes support almostthe same learning rate as 32 fully-connected workers, and get close in performance.
" Cifar- ￿￿
) ￿ ￿ .
￿ ￿ .
￿ ￿ . ￿ ￿ .
￿ ￿ . ￿ ￿ .
￿ . ￿ ￿ .
￿ .
￿ ￿ Spectral gap ! ⇥ ⇥ ⇥ ⇥ ⇥ ￿ .
￿ ￿ . ￿ ￿ .
￿ . ￿ .
￿ ￿ . ￿ ￿ .
￿ ￿ .
￿ ￿ ￿ ￿￿￿ Effective num.
neighbors (  = ￿ .
￿￿￿ , tuned) ! ⇥ ⇥ ⇥ ⇥ ⇥
After ﬁtting a decay parameter  = 0 .
that captures problem speciﬁcs, the effective number of neighbors (left) as measured by variance reduction in a random walk (like in Section 3) explains the relative performance of these graphs much better than the spectral gap of these topologies (right).
We also ﬁnd that the random walks introduced in Section 3 are a good model for variance betweenworkers in deep learning.
Figure 5 shows the empirical covariance between the workers after
SGD steps.
Just like for isotropic quadratics, the covariance is accurately modeled by the covariancein the random walk process for a certain decay rate  .
Finally, we observe that the effective number of neighbors computed by the variance reduction in arandom walk (Section 3) accurately describes the relative performance under tuned learning rates ofgraph topologies on our task, including for irregular and time-varying topologies.
This is in contrastto the topology’s spectral gaps, which we ﬁnd to be not predictive.
We ﬁt a decay rate  = 0 .
951 thatseems to capture the speciﬁcs of our problem, and show the correlation in Figure 6.
In Appendix F.1, we replicate the same experiments in a different setting.
There, we use larger graphs(of 64 workers), a different model and data set (an MLP on Fashion MNIST [22]), and no momentumor weight decay.
The results in this setting are qualitatively comparable to the ones presented above.
Conclusion
We have shown that the sparse averaging in decentralized learning allows larger learning rates to beused, and that it speeds up training.
With the optimal large learning rate, the workers’ models are notguaranteed to remain close to their global average.
Enforcing global consensus is unnecessary in thei.i.d.
setting and the small learning rates it would require are counter-productive.
With the optimallearning rate, models do remain close to some local average in a weighted neighborhood around them.
The workers beneﬁt from a number of ‘effective neighbors’, smaller than the whole graph, that allowthem to use a large learning rate while retaining sufﬁcient consensus within the ‘local neighborhood’.
Based on our insights, we encourage practitioners of sparse distributed learning to look beyond thespectral gap of graph topologies, and to investigate the actual ‘effective number of neighbors’ thatis used.
We also hope that our insights motivate theoreticians to be mindful of assumptions thatartiﬁcially limit the learning rate.
We show experimentally that our conclusions hold in deep learning, but extending our theory tothe non-convex setting is an important open direction that could reveal interesting new phenomena.
Furthermore, an extension of our semi-local analysis to the heterogeneous setting where workersoptimize different objectives could shed further light on the practical performance of D-SGD.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e
g., data splits, hyperparameters,
how they
were chosen)?
[N/A]
(c) Did you report error bars (e
g., with respect to the random seed after running experi-
ments multiple times)?
[Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[No]
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[No]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]