O N THE E FFECT OF C ONSENSUS IN D ECENTRALIZED D EEP L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
Decentralized training of deep learning models enables on-device learning over networks, as well as efﬁcient scaling to large compute clusters.
Experiments in earlier works revealed that decentralized training often suffers from generalization issues: the performance of models trained in a decentralized fashion is in general worse than the performance of models trained in a centralized fashion, and this generalization gap is impacted by parameters such as network size, communication topology and data partitioning.
We identify the changing consensus distance between devices as a key parameter to explain the gap between centralized and decentralized training.
We show that when the consensus distance does not grow too large, the performance of centralized training can be reached and sometimes surpassed.
We highlight the intimate interplay between network topology and learning rate at the different training phases and discuss the implications for communication efﬁcient training schemes.
Our insights into the generalization gap in decentralized deep learning allow the principled design of better training schemes that mitigate these effects
I NTRODUCTION
Highly over-parametrized deep neural networks show impressive results in machine learning tasks, which also lead to a dramatic increase in the size, complexity, and computational power of the training systems.
In response to these challenges, distributed training algorithms (i.e. data-parallel large mini-batch SGD) have been developed for use in data-center (Goyal et al., 2017; You et al., 2018;
Shallue et al., 2018).
These SOTA training systems in general use the All-Reduce communication primitive to perform exact averaging on the local mini-batch gradients computed on different subsets of the data, for the later synchronized model update.
However, exact averaging with All-Reduce is sensitive to the communication hardware of the training systems, causing the bottleneck in efﬁcient deep learning training.
To this end, decentralized training has become an indispensable training paradigm for efﬁcient large scale training in the data-center, alongside its orthogonal beneﬁts on preserving users’ privacy for edge AI
(Bellet et al.,
[Table] [Table]
Several very recent papers aim to address the communication overhead in data-center training by compressing the gradients (Koloskova et al., 2020a; Vogels et al., 2020) or designing better a communication topology (Assran et al., 2019).
However, decentralized training for deep learning models still often results in a severe loss in generalization performance, even after hyper-parameter ﬁne-tuning (see our Table 1 as well as
Tables 1, 2, 3 in Assran et al., 2019).
This phenomenon is poorly understood even in relatively straightforward i.i.d.
data distribution scenarios (i.e. the data-center case), to which very few works are dedicated (but none of them provide insights into the generalization gap).
In this work, we theoretically identify the consensus distance, i.e. the average discrepancy between the nodes, as the key parameter that captures the joint effect of decentralization.
We show that there exists a critical consensus distance: when the consensus distance is lower than this critical value, the optimization is almost unhindered.
With the insight derived from optimization convergence,
we further question the existence of the critical consensus distance for deep learning in terms of generalization, and identify the training phase when the critical consensus distance matters.
We believe that the answers to these questions are valuable in practice, as they offer the possibility to design training strategies that strike appropriate trade-off between targeted generalization performance and affordable communication resources.
• We show that tracking the consensus distance over the training phases can explain the generalization gap between centralized and decentralized training.
• We show theoretically that when the consensus distance does not exceed a critical value, then decentralization exerts negligible impact to the optimization.
We argue how this critical value depends crucially on the learning rate and the mixing matrix.
• Through the lens of consensus distance, we empirically investigate what is the desirable level of consensus distance during different phases of training, in order to ensure high generalization performance.
Our extensive experiments on Computer Vision (CV) tasks (CIFAR-10 and ImageNet-32) as well as Natural Language Processing (NLP) tasks (transformer models for machine translation) do conﬁrm that a critical consensus distance indeed exists, and that consensus distances higher than this critical value heavily penalize the ﬁnal generalization performance.
Surprisingly, we ﬁnd that large consensus distance might be beneﬁcial in later training phases after optimization has plateaued, leading to improved generalization, which is consistent with the observations in centralized Post-Local SGD (Lin et al., 2020b).
• Based on our ﬁndings, we propose practical guidelines on how to achieve favorable generalization performance with low communication expenses, on arbitrary communication networks.
While our numerical study mainly focuses on the data-center setting with homogeneous nodes, our ﬁndings also apply to decentralized training over time-varying topologies and the more difﬁcult heterogeneous setting alike.
Our ﬁndings not only explain previously proposed ad-hoc solutions (e.g. using All-Reduce in the ﬁrst training phase as in Assran et al., 2019)
but our insights will allow the principled design of better decentralized training systems in future work.
R ELATED W ORK
Gossip averaging (Kempe et al., 2003; Xiao & Boyd, 2004; Boyd et al., 2006) forms the backbone of many decentralized learning algorithms.
The convergence rate of gossip averaging towards consensus among the nodes can be expressed in terms of the (expected) spectral gap of the mixing matrix.
Lian et al.
(2017) combine SGD with gossip averaging for deep learning and show that the leading term in the convergence rate O (cid:0) 1 nε 2 (cid:1) is consistent with the convergence of the centralized mini-batch
SGD (Dekel et al., 2012) and the spectral gap only affects the asymptotically smaller terms.
Similar results have been observed very recently for related schemes (Scaman et al., 2017; 2018; Koloskova et al., 2019; 2020a;b).
As the communication topology also impacts the cost per round (number of peer-to-peer communications), sparse topologies have been proposed and studied recently (Assran et al., 2019; Wang et al., 2019; Nadiradze et al., 2020).
Whilst a few recent works focus on the impact of the topology on the optimization performance (Luo et al., 2019; Neglia et al., 2020), we here identify the consensus distance as a more canonical parameter that can characterize the overall effect of decentralized learning, beyond only the topology, through which we are able to provide deeper understanding on the more ﬁne-grained impact of the evolution of the actual consensus distance, on the generalization performance of deep learning training.
Prior work identiﬁed the consensus distance as an important parameter that can affect optimization performance and convergence, and provide approaches to increase consensus: for instance Scaman et al.
; Sharma et al.
(2019) perform multiple consensus steps per round, and Tsitsiklis (1984);
Nedi´c & Ozdaglar (2009); Duchi et al.
; Yuan et al.
(2016) choose carefully tuned learning rates.
However, these work do not provide insights into how consensus distance at different training phases impacts the decentralized training, which is the main target of this work.
T HEORETICAL U NDERSTANDING
In this section we consider decentralized training with stochastic gradient descent (D-SGD) without momentum, but
we
3. N OTATION AND S ETTING
The agents are tasked to solve a sum-structured optimization problem f : R d → R of the form
[Equation]
where the components f i : R d → R are distributed among the n nodes and are given in stochastic form: f i ( x ) :=
E ξ ∼D i [ F i ( x , ξ )] , where D i denotes the local data distribution on node i ∈ [ n ] .
For data-center settings, where data is re-shufﬂed periodically among nodes, these distributions are identical, but in other scenarios there can be differences between nodes.
In D-SGD, each agent i ∈ [ n ] maintains local parameters x ( t ) i ∈ R d , and updates them as:
[Equation]
that is, by a stochastic gradient step based on a sample ξ ( i ) i ∼ D i , followed by gossip averaging with neighboring nodes in the network encoded by the mixing weights w ij .
As parameters can differ across nodes, we deﬁne ¯ x := 1 n (cid:80) ni
=1 x i and X :=
[ x 1 , . . .
, x n ] ∈ R d × n , and ¯ X :=
[¯ x , . .
, ¯ x ] ≡ X 1 n 11 (cid:62) .
Assumption 1 (Mixing matrix) .
Every sample of the (possibly randomized) mixing matrix W = { w ij } ∈ R n × n is doubly stochastic and there exists a parameter p > 0 such that
[Equation]
This assumption covers a broad variety of settings (see e.g. Koloskova et al., 2020b), such as D-SGD with ﬁxed (constant) mixing matrix with spectral gap ρ , with parameter p = 1 − (1 − ρ )
= Θ( ρ ) , but also for randomly chosen mixing matrices, for instance random matchings.
Assumption 2 ( L -smoothness) .
Each function f i ( x ): R d → R , i ∈ [ n ] is differentiable and there exists a constant L ≥
such that for each x , y ∈ R
d : (cid:107)∇ f i ( y ) − ∇ f i ( x ) (cid:107) ≤ L (cid:107) x − y (cid:107) .
Assumption 3 (Bounded noise σ and diversity ζ ) .
There exists constants σ 2 , ζ 2 s.t. ∀ x 1 , . . .
x n ∈ R d
[Equation]
[Equation]
D ECENTRALIZED C ONSENSUS O PTIMIZATION
Under the above assumptions, which are standard in decentralized optimization, the convergence rate of (D-SGD) has been shown to be:
Theorem 1 (Koloskova et al.
(2020b)) .
Let f i be L -smooth and stepsize γ ≤ γ max
= O (cid:0) pL (cid:1) .
Then there exists an optimal stepsize γ ≤ γ max such that 1 T (cid:80) T − 1 t =
E (cid:13)(cid:13) ∇ f
(¯ x ( t ) )
(cid:13)(cid:13) 2 2 ≤ ε for
T =
O (cid:18) σ [Equation] nε [Equation]
√ pσ
[Equation]
ζ pε
[Equation]
[Equation] + p [Equation] In comparison, for centralized mini-batch SGD (C-SGD) we are allowed to choose a potentially much larger stepsize γ ≤ O (cid:0) 1 L (cid:1) , and can bound the number of iterations by O (cid:0) σ 2 nε 2 + 1 ε (cid:1) .
While asymptotically both these rates are equivalent, they differ in low accuracy setting when ε is not too small.
That is, especially in the ﬁrst phase of optimization where the lower order terms matter.
To measure differences between agents, we use the consensus distance Ξ 2 t := 1 n (cid:80) ni =1 (cid:13)(cid:13)
¯ x ( t ) − x ( t ) i (cid:13)(cid:13) 2 .
Remark 2 (Critical Consensus Distance (CCD)) .
If the consensus distance is bounded by
[Equation]
for all t , then in D-SGD we may choose larger stepsizes γ ≤ γ (cid:48) max
= O (cid:0) 1 L (cid:1) and recover the convergence rate of C-SGD, that is O (cid:0) σ 2 nε 2 + 1 ε (cid:1) (
Dekel et al., 2012; Bottou et al., 2018).
We denote Γ 2 t as critical consensus distance (CCD).
The proof can be found in the Appendix A.1.
Note that the CCD does not depend on the graph topology and that Γ 2 t > 0 , which means that we do not need perfect consensus between agents to recover the C-SGD rate, but we allow consensus distance Ξ 2 t ≥ 0 (i.e. the Ξ 2 t = 0 ∀ t , as we have for centralized optimization is sufﬁcient, but not necessary).
We now estimate the magnitude of the consensus distance in D-SGD and compare it to CCD.
Proposition 3 (Typical consensus distance) .
Let φ 2 t := 1 n (cid:80) ni
=1 (cid:13)(cid:13) ∇ f i ( x ( t ) i )
(cid:13)(cid:13) 2 .
Then under the assumption that γ, p are constant, and the φ t do not change too fast between iterations, i.e. not decreasing faster than exponentially: φ 2 t ≤ (1+ p/ 4) φ 2 t +1 , the consensus distance in D-SGD satisﬁes
[Equation]
We give the proof in the Appendix A.2. While these assumptions do not hold in epochs with learning rate decay, we observed in practice that when the learning rate is constant indeed the gradients do not change too fast and found it to be a reasonable approximation to capture the practical behavior (see Figure 5(b)).
3. C ONTROLLING THE CONSENSUS DISTANCE
In this section we investigate scenarios where the typical consensus distance derived in Propositioncan be smaller than its critical value (CCD).
This reveals two orthogonal strategies to control the consensus distance in D-SGD.
We here assume diversity ζ = 0 as with iid training data, and that the stepsize γ ≤ O (cid:0) 1 L (cid:1) as for C-SGD, and give a more reﬁned discussion in the appendix.
Learning rate decay (changing γ ).
We observe that when γ = O (cid:0) pnL (cid:1) then Ξ 2 t ≤ Γ 2 t (if the noise σ is small, especially for σ = 0 , then the weaker assumption γ =
O (cid:0) pL (cid:1) is sufﬁcient).
However, choosing the stepsize too small can impact performance in practice.
In C-SGD the constraint on the stepsize is loose ( γ ≤ 1 L ).
Yet, our observations show that after sufﬁcient learning rate decay, the desired CCD can be reached.
More gossip iterations (changing p ).
We observe that when 11 − p =
O (1 + γLn ) , then Ξ 2 t ≤ Γ 2 t (again, when the noise σ is small, especially when σ 2 = 0 , a weaker condition 11 − p
=
O (1 + γL ) is sufﬁcient).
Whilst designing new mixing topologies to control p might not be possible due to practical constraints (ﬁxed network, denser graphs increase latency, etc.)
a simple and commonly used strategy is to use repeated gossip steps in every round.
Lemma 4 (Repeated gossip) .
Suppose W = W k . . .
W 1 , for k (possibly randomized) mixing matrices with parameter p each.
Then the mixing parameter for W is at least p W ≥ 1 − (1 − p ) k .
From this, we see that the mixing parameter can be improved exponentially when applying more gossip steps.
To ensure p W ≥ 1 −
[Equation]
, at most k ≤
[Equation]
p
[Equation]
repetitions are required.
These arguments show, that we can—at least in theory—recover the convergence behavior of C-SGD by controlling the consensus distance.
We will now present numerical evidence, that corroborates these ﬁndings for deep learning tasks.
I NSPECTING C ONSENSUS D ISTANCE FOR D ECENTRALIZED T RAINING
Motivated by theoretical analysis in Section 3, we empirically investigate to what extent our theory holds and how consensus distance interacts with deep learning training .
First we introduce and justify our experimental design in Section 4.1.
We describe the implementation in Section 4.2.
In
Section 4.3, we present our ﬁndings on image classiﬁcation benchmark with standard SGD optimizer, which is the main focus of this work; a preliminary study on Transformer with Adam optimizer and inverse square root learning rate schedule can be found in Section 4.4.
4. E XPERIMENT D ESIGN : C ONTROLLED DECENTRALIZED TRAINING PHASES .
Phase-wise training.
As consensus distance evolves throughout training, identifying its impact at every training step is infeasible.
Also, since consensus distance and critical consensus distance (CCD) have different but signiﬁcant dependencies on learning rate (Remark 2 and Proposition 3), we would expect observations to be distinct over different learning rate phases but rather consistent within each phase.
On CV tasks, stage-wise learning rate schedule is the common practice for SOTA training as described in Section 4.2: thus the training can be naturally divided into phases with the corresponding learning rate, in each of which key training dynamics are signiﬁcantly different from the others, such as Ξ t (Figure 1), φ t (Figure 5(b)) and L -smoothness (Figure 5(c)).
The transformer (NLP task) has no well-deﬁned training phases due to the conventional inverse square root learning rate, thus for the sake of simplicity, we consider the entire transformer training as one phase as a preliminary study.
Individual phase investigation.
In order to eliminate the coupling of effects from other phases, in each experiment we place only one phase under consensus distance control, while performing exact averaging (All-Reduce) 1 on model parameters for the other unstudied phases.
For the ease of presentation, the term “phase- x ” refers to a training phase between x − 1 -th and x -th learning rate decay.
The notation “dec-phase- x ” indicates that only in “phase- x ” the model is trained with a decentralized communication topology, while for other phases we perform All-Reduce on model parameters.
We compare the result of individual decentralized investigation with that of All-Reduce centralized training (on all training phases), so as to identify when (which phase) and how decentralized training inﬂuences ﬁnal generalization performance 2 .
4. E XPERIMENTAL S ETUP
Datasets and models.
We empirically study the decentralized training behavior on the following two tasks, on CNN and transformer architectures: (1) Image Classiﬁcation for CIFAR-10 (Krizhevsky & Hinton, 2009) and ImageNet-32 (i.e. image resolution of 32 ) (Chrabaszcz et al., 2017), with the standard data augmentation and preprocessing scheme (He et al., 2016); and (2) Neural Machine
Translation for the Multi30k dataset (Elliott et al., 2016).
For Image Classiﬁcation, ResNet-20 (He et al., 2016) with different widths are used on CIFAR (default width of 1 ) and ImageNet-32 (width factor of 3 ).
For Neural Machine Translation, a down-scaled transformer architecture (by 2 × w.r.t.
the base model in Vaswani et al.
(2017)) is used.
Weight initialization schemes follow Goyal et al.
; He et al.
(2015) and Vaswani et al.
2017) respectively.
Unless mentioned otherwise, our experiments are repeated over three random seeds.
Training schemes.
We use mini-batch SGD with a Nesterov momentum of 0 .
9 without dampening for image classiﬁcation task, and Adam for neural machine translation task.
Unless mentioned otherwise we use the optimal learning rate (lr) for centralized training 3 in order to observe the impact of decentralization on normal centralized training.
• For image classiﬁcation experiments, unless mentioned otherwise the models are trained forand 90 epochs for CIFAR-10 and ImageNet-32 respectively; the local mini-batch size are set toand 64 .
By default, all experiments use learning rate scaling and warmup scheme (Goyal et al., 2017).
The learning rate is always gradually warmed up from a relatively small value (i.e. 0 . 1 ) for the ﬁrst 5 epochs.
Besides, the learning rate will be divided by 10 when the model has accessed speciﬁed fractions of the total number of training samples (He et al., 2016); we use { 12 , 34 } and { 13 , 23 , 89 } for CIFAR and ImageNet respectively.
All results in tables are test top-1 accuracy.
• For experiments on neural machine translation, we use standard inverse square root learning rate schedule (Vaswani et al., 2017) with local mini-batch size 64 .
The warmup step is set to 4000 for the mini-batch size of 64 and is linearly scaled down by the global mini-batch size.
Consensus distance control.
For consensus control, we adopt the “more gossip iterations” strategy introduced in Section 3.3.
That is, we perform multiple gossip steps (if needed) until reaching the desired target consensus distance value.
Two metrics are considered to set the consensus distance target value during the speciﬁed training phase
: • constant consensus distance (main approach 4 ): the target consensus distance Ξ for a phase is the maximum consensus distance Ξ max of the current phase in normal (uncontrolled) decentralized training, multiplied by a factor.
For a given topology, the smaller the factor, the tighter consensus.
We demonstrate in Table 4 of Section 4.3 that the decentralization
[Figure]
[Figure] [Figure] [Table] • adaptive consensus distance (in Appendix C.3.1): the target consensus distance Ξ for the current step is the averaged local gradient norm φ avg t scaled by a factor.
For stability, we use the exponentially moving averaged value φ ema t of φ avg t (accumulated during the corresponding phase).
[Figure]
We use a ring as our (main) decentralized communication topology 5 as it is a particular hard instance with large spectral gap (cf.
Table 7), allowing to study a wide range of target consensus distances by modifying the number of communication rounds.
4. F INDINGS ON COMPUTER VISION TASKS
In this section we present our empirical ﬁndings and provide insights into how consensus distance at different phases impacts the training generalization for CV tasks (i [Table] e. CIFAR-10, Imagenet-32).
Critical consensus distance exists in the initial training phase and ensures good optimization and generalization.
In the initial training phase, both training and generalization performance are heavily inﬂuenced by the consensus distance 6 (“dec-phase-1” in Figure 2 and Table 2).
A smaller consensus distance in the early phase results in considerably faster optimization (training loss) and higher generalization performance (test accuracy) and these advantages persist over the entire training.
When the consensus is more loose (e.g. 1/2 Ξ max for CIFAR-10), although the optimization (training performance) can eventually catch up with the centralized convergence (c.f. Figure 2(a) and 2(b)), a considerable generalization gap still remains ( 92 .
36 v.s. 92 .
82 for the setup in Figure 2) as shown in Table 2.
This pattern is consistent 7 in ImageNet-32 experiments, as shown in Table 3.
These observations to some extent are also consistent with the insights of the critical learning phase described in Jastrzebski et al.
(2020); Frankle et al.
Golatkar et al.
for traditional (centralized) training, where it is argued that the initial learning phase is crucial for the ﬁnal generalization.
Notably, perfect consensus distance is not required to recover the centralized training performance.
For instance, 1/4 Ξ max is sufﬁcient in CIFAR-10 experiments to approach the optimal centralized training performance in both optimization and generalization at the end.
Smaller distances (e.g.
1/8 Ξ max , 1/16 Ξ max ) do not bring signiﬁcant gain ( 92 . 77 and 92 . 72 respectively in Table 9).
The performance saturates (c.f. 92 .
74 for 1/4 Ξ max ) with signiﬁcantly increased communication overhead (e.g. Figure 9 of Appendix C.1).
This conﬁrms that our analysis and discovery in Section 3 are In the case of ImageNet-32 with n =32 , 1/2 Ξ max has already been tight enough to recover the centralized training performance But a signiﬁcant can be Ξ max and 1/2 Ξ max .
[Table]
[Table]
[Table]
sensible in the initial training phase: there exists a critical consensus distance for the training, beyond which the impact of decentralization is negligible.
A non-negligible consensus distance at middle phases can improve generalization over centralized training.
Surprisingly, it is not always the case that the generalization performance improves with the shrinking consensus distance.
We observe that at the phase right after initial training plateaus (e.g. phase-2 for CIFAR-10, phase-3 for Imagenet-32), a non-negligible consensus distance 8 actually boosts the generalization performance over the centralized training which has been deemed optimal.
In CIFAR-10 dec-phase-2 experiments (Table 2), the generalization performance increases monotonically with the evaluated consensus distance and is consistently superior to that of the centralized training (e.g. 93 . 04 , 92 . 99 , 92 .
over 92 .
82 for n = 32 ).
Analogous observation can be obtained in
Imagenet-32 dec-phase-3 experiments (Table 3).
It coincides with the observations ﬁrstly introduced in post-local SGD (Lin et al., 2020b), where for better generalization, consensus distance is created among local machines by less frequent model parameter synchronization (All-Reduce) in late training phases (e.g. phase-2, phase-3 for CIFAR).
Thus non-negligible consensus distance at middle phases can be viewed as a means of injecting proper noise as argued in Lin et al.
(2020b), which reduces communication cost and in the meanwhile beneﬁts generalization.
At the last phase of training, the consensus degree marginally impacts the generalization.
Similar to the initial training phase, the ﬁnal convergence phase seems to favor small consensus distances in CIFAR-10 experiments.
However, its impact is less prominent in comparison: for dec-phase-3, performance of a smaller consensus distance (1/4 Ξ max ) is only 0 .
25% and 0 .
37% higher than that of Ξ max for n = 32 , 64 respectively (Table 2).
In Imagenet-32 experiments, dec-phaseperformance is not even affected by changes in consensus.
Quality propagation across phases.
Our previous experiments only considered a single phase of decentralized training.
We now evaluate the continued impact of consensus across the sequence of multiple phases.
In Table 4, we control the consensus distance for both phase-1 and phase-2 when training CIFAR-10.
When one phase is held under every speciﬁc consensus control, our previous ﬁndings for the other phase still hold.
For instance, when we apply 1/2 Ξ max ,1/4 Ξ max consensus control to phase-2 (middle column in Table 4), we can still observe that a smaller consensus distance in phase-1 results in a higher performance as in our previous ﬁnding.
Hence our previous ﬁndings are valid in more general cases of decentralized training.
[Table]
Longer training cannot close the generalization gap caused by large consensus distances in initial training phase.
As discussed above, large consensus distances in the initial phase can result in signiﬁcant generalization loss due to the optimization difﬁculty.
Table 5 investigates whether a prolonged training on the initial phase can address this difﬁculty: we prolong the phase-1 for
CIFAR-10 with a range of consensus distances and leave the other training phases centralized.
We can observe that although longer training is beneﬁcial for each consensus distance, it cannot recover the generalization gap resulting from large consensus distance.
For instance, the maximum gain (among all evaluated cases) of increasing the epoch number from 150 to 250 is 0 .
31% at 1/2 Ξ max , which is lower than the average gain (around 0 .
6% ) of merely reducing the consensus distance from Ξ max to 1/2 Ξ max .
Table 13 evaluates cases where dec-phase-2 and dec-phase-3 are prolonged.
We ﬁnd that longer training in these two phases bring about negligible performance gain.
Practical guidelines: prioritizing the initial training phase.
Apart from effectiveness (generalization/test performance), efﬁciency (time) stands as the other crucial goal in deep learning, and thus how to allocate communication resource over the training becomes a relevant question.
[Figure]
As indicated by our ﬁrst empirical ﬁnding (and theory in Section 3), initial training phase bears the greatest importance over all other training phases; therefore the communication expenditure should be concentrated on the initial phase to maintain a consensus distance lower than the CCD.
A list of communication topologies with superior spectral properties, e.g. exponential graph (Assran et al., 2019) and random matching (Nadiradze et al., 2020), can be utilized 9 to achieve fast convergence in gossip averaging.
The late training phases should be less prioritized for communication resources, due to the generalization beneﬁts from a reasonable consensus distance in the middle phases.
Providing a rigorous way to quantify the optimal consensus distance is non-trivial, and we leave it as future work.
In Table 6 we show that the above mentioned guideline is practically feasible: as long as the quality of the initial phase is ensured, we can afford to slacken the consensus control for later phases, in particular the middle phase.
For instance, when the number of epochs is 150, a consensus control of 1/4 Ξ max in the initial phase with uncontrolled middle and ﬁnal phase is adequate to recover the centralized training performance ( 92 .
v.s. 92 . 82 ).
Note that here the noise injection from the uncontrolled middle phase also contributes positively to the performance.
Table 16 in Appendix C.3.
additionally justiﬁes the applicability of applying this guideline on exponential graph, and we leave more complicated communication scheme design as future work.
4. P RELIMINARY STUDY ON TRAINING TRANSFORMER MODELS
The critical consensus distance also exists in NLP tasks.
Figure 4(c) demonstrates that 1/4 Ξ max target control on a ring is sufﬁcient to recover the centralized training performance.
Besides, the target consensus distance in this case can be reached by exponential graph (and thus target test performance, as shown in Figure 4(a) and 4(b)).
These justify the importance of designing an efﬁcient communication topology/scheme in practice so as to effectively reach the critical consensus distance.
[Figure]
A T HEORY
In this section, we prove the claims from Section 3.
A. P ROOF OF R EMARK 2, C RITICAL C ONSENSUS D ISTANCE
The proof of this claim follows by the following Lemma:
Lemma 5 (Koloskova et al.
(2020b), Descent lemma for non-convex case) .
Under the given assumptions, and for any stepsize γ < 14 L , the iterates of D-SGD satisfy
[Equation]
Proof.
By replacing Ξ t in the above inequality with (4), we simplify:
[Equation]
This inequality now matches (up to differences in the constants) the standard recursion that one can derive for C-SGD (Dekel et al., 2012; Bottou et al., 2018; Stich & Karimireddy, 2019).
A. P ROOF OF P ROPOSITION 3, TYPICAL CONSENSUS DISTANCE
We need an auxiliary (but standard) lemma, to estimate the change of the consensus distance between iterations.
Lemma 6 (Consensus distance) .
It holds
[Equation]
We give the proof of this statement shortly below.
First, let us consider how this lemma allows the proof of the claim.
For this, we ﬁrst consider a particular special case, and assume φ t ≤ φ , for a constant φ .
In this case, we can easily verify by unrolling the recursion:
[Equation]
Now
[Equation] for the claim in the main text [Equation] we use assumption that φ t are changing slowly, that is, not decreasing faster than exponentially: φ 2 t ≤ (1 + p/ 4) φ 2 t +1 .
With this assumption
[Equation]
and observing (1 − p/ 2) i (1 + p/ 4) i ≤ (1 − p/ 4) i , we can unroll as before
[Equation]
Proof of Lemma 6.
We use the following matrix notation here
[Equation]
[Table]
[Equation]
[Table]
A. S UFFICIENT BOUNDS TO MEET CRITICAL CONSENSUS DISTANCE
In this section, we show that the claimed bounds in Section 3.3 are sufﬁcient conditions to reach the
CCD.
According to Remark 3, there exists an absolute constant C , (w
[Equation]
l.o.g.
C ≥ 2 )
such that
[Equation]
By smoothness, φ
[Equation]
n n
[Equation]
i
[Equation]
Supposing (1 − p ) γ 2 ≤ p 2 6 CL
[Equation]
we can therefore estimate
[Equation]
and hence
[Equation]
The claimed bounds can now easily be veriﬁed, by plugging the provided values into (6).
For simplicity in the main text we assume that ζ = 0 (we are in the datacenter training scenario).
Small γ .
By choosing γ ≤ p 4 nLC , we check that our previous constraint γ 2 C ≥ 2 ≤ p 2 6 CL 2 is satisﬁed, and
[Equation]
Small p .
By choosing 1 − p ≤ 1 5 C (1+ γLn ) , we note that p
[Equation]
Moreover, our previous constraint (1 − p ) γ 2 ≤ γ 2 5 C ≤ p 2 6 L 2 C is satisﬁed (note that γ ≤ 14 L throughout).
Hence
[Equation]
In the above calculations we for the simplicity assumed that ζ = 0 .
For the general non-iid data case when ζ > 0 we can calculate similar bounds on γ , p .
These bounds would have similar dependence on parameters, and would be stricter.
Indeed, the typical consensus distance would be also inﬂuenced by non-iidness of the data ζ
and it is therefore harder to satisfy the CCD condition.
A. P ROOF OF L EMMA 4, REPEATED GOSSIP
By the assumption stated in the lemma, it holds for each component W i of the product W =
W k . . .
W 1 , i ∈ [1 , k ] that
[Equation]
Now lets estimate the parameter p W .
Using that W i are independent
[Equation]
2 F , where we deﬁned Y = XW k . . .
W 2 and used that W i 1 n 11 (cid:62) =
1 n 11 (cid:62) , so
[Equation]
Therefore,
[Equation]
Applying the same calculations to the rest, we conclude that 1 − p W =
(1 − p ) k .
B D ETAILED E XPERIMENTAL S ETUP
Comments on large-batch training.
Coupling the quality loss issue of the decentralized training with the large-batch training difﬁculty is non-trivial and is out of the scope of this paper.
Instead, we use reasonable local mini-batch sizes (together with the number of workers (denoted as n )), as well as the well-developed large-batch training techniques (Goyal et al., 2017), to avoid the difﬁculty of extreme large-batch training.
Multi-phase experiment justiﬁcation.
The averaged local gradient norm φ t as well as the Lsmoothness of ResNet-20 on CIFAR-10 for a ring and a complete graph ( n = 32 ) are shown in
Figure 5 and Figure 6 respectively.
The estimation procedure is analogous to that in Santurkar et al.
Lin et al.
(2020a): we take 8 additional steps long the direction of current update, each with 0 .
2 of normal step size.
This is calculated at every 8 training steps.
The smoothness is evaluated as the maximum value of L satisfying Assumption
[Figure]
[Figure] [Figure]
[Figure]
C A DDITIONAL R ESULTS C. U NDERSTANDING ON C ONSENSUS A VERAGING P ROBLEM
We study a host of communication topologies: (1) deterministic topologies (ring, and complete graph) and (2) undirected time-varying topologies (illustrated below).
• Random matching (Boyd et al., 2006).
At each communication step, all nodes are divided into non-overlapping pairs randomly.
Each node connects all other nodes with equal probability.
• Exponential graph (Assran et al., 2019).
Each is assigned a rank from 0 to n − 1 .
Each node i periodically communicates with a list nodes with rank i + 2 0 , i + 2 1 , . .
, i + 2 (cid:98) log 2 ( n − 1) (cid:99) .
In the one-peer-per-node experiments, each node only communicates to one node by cycling through its list.
The formed graph is undirected, i
e., both transmission and reception take place in each communication.
• Bipartite exponential graph (Lian et al., 2018; Assran
et al
In order to avert deadlocks
(Lian et al.,
the node with an odd rank i cycles through nodes with even
ranks i + 2 0 − 1 , i + 2 1 − 1 , . .
, i + 2 (cid:98) log 2 ( n − 1) (cid:99) − 1 by
transmitting
a
message and waiting for a response.
while
the nodes with even ranks only await messages and reply upon reception
Table 7 displays the spectral gap and node degree of studied topologies [Figure] and Figure 7 provides the convergence curves for different communication topologies on graph scales.
Figure 8 in addition visualizes the spectral gap (in expectation) for different communication topologies. [Table] [Figure] [Table]
[Figure] [Table] [Figure] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Figure]
Table 8 examines these topologies on a standard deep learning benchmark with different graph scales, while Figure 9 visualizes the required communication rounds (per gradient update step) for a range of consensus distance targets
[Figure]
[Table]
C. U NDERSTANDING THE D ECENTRALIZED D EEP L EARNING T RAINING FOR CV T ASKS
We use ring as our underlying decentralized communication topology in this subsection.
Consensus distance    # o f c
o mm .
r o un d s t
o r e a c h t h e d i s t a n c e Elaborated results on consensus distance control. Table 9 is the elaborated version of Tablewith more evaluated consensus distances.
[Table] [Figure] [Table] SlowMo cannot fully address the decentralized optimization/generalization difﬁculty.
Table 10 studies the effectiveness of using SlowMo for better decentralized training.
We can witness that even though the performance of decentralized training can be boosted to some extent, it cannot fully address the quality loss issue brought by decentralized training [Table] [Table]
On the ineffectiveness of tuning learning rate.
Table 11 displays the results of training ResNet20 on CIFAR-10 (32 nodes), with ﬁne-tuned learning rate on phase-1; learning rate tuning cannot address the test quality loss issue caused by the large consensus distance (i e. over the CCD) [Table] Distance control on SGD without momentum. Table 12 contains the consensus distance control results for SGD without momentum.
We can observe a consistent pattern as in Table 2.
[Table]
Prolonged training for dec-phase-2 and dec-phase-3. Table 13 shows the results for prolonged dec-phase-2 and dec-phase-3 on CIFAR-10 with ResNet20. We can observe although longer training duration increases the performance, the improvement is rather small
[Table]
[Table]
The impact of half cosine learning rate schedule. Table 14 examines the existence of the critical consensus distance with half cosine learning schedule (this scheme is visited in He et al.
as a new paradigm for CNN training).
We can witness from Table 14 that the effect of critical consensus distance can be generalized to this learning rate schedule: there exists a critical consensus distance in the initial training phase (as revealed in the inline Figure of Table 14) and ensures good optimization and generalization
[Table]
[Figure]
[Table]
Ring ( Ξ max ) Ring ( 1 / 2Ξ max )
Ring
[Table]
C.2. A DAPTIVE CONSENSUS DISTANCE CONTROL
In Table
[Table]
we apply the adaptive consensus distance control in the experiments.
The observations are consistent with those in constant consensus distance control experiments
[Table] [Table]
C. C ONSENSUS CONTROL WITH OTHER TOPOLOGIES
In Table
[Table]
we exert consensus control with an exponential graph as the base communication topology.
We can observe that our ﬁndings from main experiments with a ring base topology are valid.
[Table]
C.3. T HE E XISTENCE OF THE O PTIMAL C ONSENSUS D ISTANCE FOR N OISE I NJECTION .
Table 17 uses a different communication topology
(i
[Table]
e. time-varying exponential graph) for decentralized optimization.
Here exponential graph with large spectral gap is applied to CIFAR-10 dec-phasetraining.
We apply the adaptive consensus distance control in this set of experiments.
We can observe that increasing consensus distance further by taking local steps improves generalization, however, too many local steps diminish the performance.
For instance, for ratio=2, the performance peaks at local update steps 2 and drops at local update 4.
It points out that an optimal consensus distance is required to inject proper stochastic noise for better generalization.
[Figure]
[Table]
C. R ESULTS FOR T RAINING T RANSFORMER ON M ULTI 30 K
We additionally report the decentralized training results, for a downsampled transformer models (by the factor of 2 w.r.t.
the base model in Vaswani et al.
(2017)) on Multi30k (Elliott et al., 2016).
Figure 10 shows that the straightforward application of Adam in the decentralized manner does encounter generalization problems, which are attributed to the fact that the different local moment buffers (in addition to the weights) become too diverse.
Tuning the learning rate schedule cannot address the issue of decentralized Adam, as shown in the Figure 10(b).
[Figure]
[Figure] [Figure] [Figure]