S LOW M O : I MPROVING C OMMUNICATION -E FFICIENT D ISTRIBUTED SGD WITH S LOW M OMENTUM
A BSTRACT
Distributed optimization is essential for training large models on large datasets.
Multiple approaches have been proposed to reduce the communication overhead in distributed training, such as synchronizing only after performing multiple local SGD steps, and decentralized methods ( e.g. , using gossip algorithms) to decouple communications among workers.
Although these methods run faster than
A LL R EDUCE -based methods, which use blocking communication before every update, the resulting models may be less accurate after the same number of updates.
Inspired by the BMUF method of Chen & Huo (2016), we propose a slow momentum (S LOW M O ) framework, where workers periodically synchronize and perform a momentum update, after multiple iterations of a base optimization algorithm.
Experiments on image classiﬁcation and machine translation tasks demonstrate that S LOW M O consistently yields improvements in optimization and generalization performance relative to the base optimizer, even when the additional overhead is amortized over many updates so that the S LOW M O runtime is on par with that of the base optimizer.
We provide theoretical convergence guarantees showing that S LOW M O converges to a stationary point of smooth non-convex losses.
Since BMUF can be expressed through the S LOW M O framework, our results also correspond to the ﬁrst theoretical convergence guarantees for BMUF.
I NTRODUCTION
Distributed optimization (Chen et al., 2016; Goyal et al., 2017) is essential for training large models on large datasets (Radford et al., 2019; Liu et al., 2019; Mahajan et al., 2018b).
Currently, the most widely-used approaches have workers compute small mini-batch gradients locally, in parallel, and then aggregate these using a blocking communication primitive, A LL R EDUCE , before taking an optimizer step.
Communication overhead is a major issue limiting the scaling of this approach, since A LL R EDUCE must complete before every step and blocking communications are sensitive to stragglers (Dutta et al., 2018;
Ferdinand et al., 2019).
Multiple complementary approaches have recently been investigated to reduce or hide communication overhead.
Decentralized training (Jiang et al., 2017; Lian et al., 2017; 2018; Assran et al., 2019) reduces idling due to blocking and stragglers by employing approximate gradient aggregation ( e.g. , via gossip or distributed averaging).
Approaches such as Local SGD reduce the frequency of communication by having workers perform multiple updates between each round of communication (McDonald et al., 2010; McMahan et al., 2017; Zhou & Cong, 2018; Stich, 2019; Yu et al., 2019b).
It is also possible to combine decentralized algorithms with Local SGD (Wang & Joshi,
; Wang et al., 2019).
These approaches reduce communication overhead while injecting additional noise into the optimization process.
Consequently, although they run faster than large minibatch methods, the resulting models may not achieve the same quality in terms of training loss or generalization accuracy after the same number of iterations.
Momentum is believed to be a critical component for training deep networks, and it has been empirically demonstrated to improve both optimization and generalization (Sutskever et al., 2013).
Yet, there is no consensus on how to combine momentum with communication efﬁcient training algorithms.
Momentum is typically incorporated into such approaches by having workers maintain separate buffers which are not synchronized (Lian et al., 2017; 2018; Assran et al., 2019; Koloskova et al., 2019a).
However, recent work shows that synchronizing the momentum buffer, using periodic
[-Base optimizer step: x ( i ) t,k +
-]
[-Update slow momentum: u t +
-]
[-Update outer iterates: x t +1 , 
-]
A LL R EDUCE or a decentralized method, leads to improvements in accuracy at the cost of doubling the communication overhead (Yu et al., 2019a).
In block-wise model update ﬁltering (BMUF), nodes perform multiple local optimization steps between communication rounds (similar to local
SGD), and they also maintain a momentum buffer that is only updated after each communication round (Chen & Huo, 2016).
Although it is now commonly used for training speech models, there are no theoretical convergence guarantees for BMUF, and it has not been widely applied to other tasks ( e.g. , in computer vision or natural language processing).
Inspired by BMUF, we propose a general framework called slow momentum (S LOW M O ) to improve the accuracy of communication-efﬁcient distributed training methods.
S LOW M O runs on top of a base algorithm, which could be local SGD or a decentralized method such as stochastic gradient push (SGP) (Nedi´c & Olshevsky, 2016;
Assran et al., 2019).
Periodically, after taking some number τ of base algorithm steps, workers average their parameters using A LL R EDUCE and perform a momentum update.
We demonstrate empirically that S LOW M O consistently improves optimization and generalization performance across a variety of base algorithms on image classiﬁcation and neural machine translation tasks—training ResNets on CIFAR-10 and ImageNet, and training a transformer on WMT’16 En-De.
Ultimately, S LOW M O allows us to reap the speedup and scaling performance of communication-efﬁcient distributed methods without sacriﬁcing as much in accuracy.
We also prove theoretical bounds showing that S LOW M O converges to a stationary point of smooth non-convex functions at a rate O (1 / √ mTτ ) after Tτ total inner optimization steps and T S LOW M O updates with m worker nodes, for a variety of base optimizers.
Thus, S LOW M O is order-wise no slower than stochastic gradient descent.
BMUF and the recently-proposed Lookahead optimizer (Zhang et al., 2019) can be expressed through the S LOW M O framework, and so our results also translate to the ﬁrst theoretical convergence guarantees for both of these methods.
T HE S LOW M OMENTUM (S LOW M O ) F RAMEWORK
S LOW M O is a framework intended for solving stochastic optimization problems of the form
[Equation]
using m worker nodes, where the loss function term F i and samples ξ i from the distribution
D i are available at the i th worker.
S LOW M O builds on top of a base optimization algorithm and has a nested loop structure shown in Algorithm 1.
Each worker maintains a local copy of the parameters, x ( i ) t,k at worker i after the k th inner step of the t th outer iteration.
We assume that all workers are initialized to the same point x 0 , 0 , and the framework also uses a slow momentum buffer u t which is initialized to u 0 = 0 ; although each worker stores a copy of u t locally, these are always synchronized across all nodes, so we omit the superscript to simplify the notation.
Within each outer iteration, workers ﬁrst take τ steps of the base optimizer.
The base optimizer could be a method which involves no communication, such as SGD (with or without momentum) or a decentralized algorithm which involves some communication, such as stochastic gradient push (SGP) (Assran et al., 2019).
We denote these updates by x ( i ) t,k +
= x ( i ) t,k − γ t d ( i ) t,k where γ t is the base optimizer (fast) learning rate and d ( i ) t,k is the update direction used at worker i .
If the base optimizer is SGD then d ( i ) t,k = ∇ F i ( x ( i ) t,k ; ξ ( i ) t,k ) .
For other base optimizers which may use additional Input: Base optimizer with learning rate γ t ; Inner loop steps τ ; Slow learning rate α ; Slow momentum factor β ; Number of worker nodes m .
Initial point x 0 , 0 and initial slow momentum buffer u 0 = 0 .
1 for t ∈ { 0 , 1 , . . .
, T − 1 } at worker i in parallel do
Reset/maintain/average base optimizer buffers for k ∈ { 0 , 1 , . .
. , τ − 1 } do
{+Base optimizer step: x ( i ) t,k +
+}
= x ( i ) t,k − γ t d ( i ) t,k end
Exact-Average: x t,τ = 1 m (cid:80) mi
=1 x ( i ) t,τ
{+Update slow momentum: u t +
+}
= β u t + 1 γ t ( x t, 0 − x t,τ )
{+Update outer iterates: x t +1 , 
+}
= x t, 0 − αγ t u t + 9 end x t, 0 <latexit sha1_base64="oF5Kwxixi4yTIqBJhyqy7gdCBNo=">AAAB+XicbVDLSsNAFL2pr1pfUZduBovgQkpSBbssuHFZwT6gDWEynbRDJ5MwMymW0D9x40IRt/6JO//
[Table]
[Table]
[Table]
[Table]
+iQMk7ieagAsnbwfVZ7rdvuDYijq5wnPCegmEkQsEArdQv7R75AUfwh6AUUF8BjoIwSyd97JfKbuWkblGjbsWdIifeaa1+TL2ZUiYzXPRLb/4gZqniETIJxnQ9N8FeBhoFk3xS9FPDE2DXMORdSyNQ3PSy6QsTemCVAQ1jbStCOlW/T2SgjBmrwHbmN5rfXi7+5XVTDOu9TERJijxin4vCVFKMaZ4HHQjNGcqxJcC0sLdSNgINDG1qRRvC16f0f9KqVrxapXp5XG7UZ3EUyB7ZJ4fEI6ekQc7JBWkSRm7JPXkkT86d8+A8Oy+frXPObGaH/IDz+gGd0peR</latexit> local momentum or communication, d ( i ) t,k represents the full update applied at worker i on this step.
Speciﬁc examples of d ( i ) t,k for different base optimizers are presented in Table C.1 in Appendix C.
After the τ base optimizer steps, the workers calculate the average x t,τ = x t, 0 − γ
[Equation]
k
[Equation] d [Equation] k using A LL R EDUCE (line 6), and then they perform a slow momentum update (lines 7–8),
[Equation]
[Equation] Although the workers perform this update locally, in parallel, we again omit superscripts because the values of x t, 0 , x t,τ , and hence u t +1 and x t +1 , 0 are always identical across all workers, since they follow the A LL R EDUCE in line 6.
Note that the difference x t, 0 − x t,τ is scaled by 1 γ t in (2) to make the slow momentum buffer invariant to the fast learning rate γ t , which may change through training, e.g. , when using a learning rate schedule.
The outer update in line 8 uses the product αγ t of the slow and fast learning rates.
We use the distinction between slow and fast because the base optimizer step is applied τ times for each outer update, but this is not intended to imply that one learning rate is necessarily bigger or smaller than the other.
We give speciﬁc examples of learning rates and other hyperparameters used in the experiments in Section 4 below.
A speciﬁc S LOW M O algorithm instance is obtained by specifying the base algorithm and the hyperparameters α , β , γ , and τ .
We can recover a number of existing algorithms in this framework.
When the base algorithm is SGD, τ = 1 , α = 1 , and β ∈ [0 , 1) , we recover standard large mini-batch SGD with learning rate γ and momentum β .
When the base algorithm is SGD, τ > 1 , α = 1 , and β = 0 , we recover Local SGD (McDonald et al., 2010; Stich, 2019; Yu et al., 2019b; Wang & Joshi, 2018).
When the base algorithm does not involve communication among workers, τ > 1 and β > 0 , we recover BMUF (Chen & Huo, 2016).
We also obtain interesting novel distributed algorithms.
In particular, the experiments in Sectiondemonstrate that using S LOW M O with a decentralized base algorithm like SGP and reasonable values of τ consistently leads to improved optimization and generalization performance over the base method alone, without a signiﬁcant increase in runtime.
We also observe empirically that, for a ﬁxed number of iterations, S LOW M O combined with SGP is superior to S LOW M O combined with SGD.
The above are all distributed algorithms.
Perhaps surprisingly, S LOW M O also encompasses a recently-introduced non-distributed method: if we have m = 1 worker with SGD/Adam as the base algorithm, α ∈ (0 , 1] , β = 0 , and τ > 0 , we recover the Lookahead optimizer of Zhang et al.
(2019), which also has a nested loop structure.
Section 5 provides theoretical convergence guarantees when using the S LOW M O framework to minimize smooth non-convex functions, and thus provides the ﬁrst theoretical convergence guarantees in the literature for BMUF and Lookahead in this setting.
R ELATED W ORK
The idea of reducing communication overhead by using A LL R EDUCE to synchronize parameters after every τ > 0 optimizer steps has been considered at least since the work of McDonald et al.
(2010), and has been more recently referred to as Local SGD in the literature.
Elastic-average SGD (Zhang et al., 2015) uses a related approach, but with a parameter server rather than A LL R EDUCE .
Lin et al.
(2018) apply Local SGD for distributed training of deep neural networks and propose postlocal SGD, which starts by running A LL R EDUCE -SGD for some epochs before switching to Local
SGD, to improve generalization at the cost of additional communication.
Decentralized methods use approximate distributed averaging over a peer-to-peer topology, rather than A LL R EDUCE .
This decouples communication but also injects additional noise in the optimization process since the models at different workers are no longer precisely synchronized.
Lian et al.
(2017) present decentralized parallel SGD (D-PSGD), where each worker sends a copy of its model to its peers at every iteration, and show it can be faster than parameter-server and A LL R EDUCE methods for training deep neural networks.
Lian et al.
(2018) study an asynchronous extension,
AD-PSGD.
Assran et al.
(2019) study stochastic gradient push (SGP), and propose its asynchronous counterpart overlap SGP (OSGP), which achieve a further speedup over D-PSGD and AD-PSGD by using less coupled communication.
D-PSGD, AD-PSGD, and SGP all have similar theoretical convergence guarantees for smooth non-convex functions, showing a linear scaling relationship between the number of workers and the number of iterations to reach a neighborhood of a ﬁrst-order stationary point.
Although the theory for all three methods only covers the case of SGD updates without momentum, implementations use momentum locally at each worker, and workers only average their model parameters (not momentum buffers).
Yu et al.
(2019a) prove that linear scaling holds when workers average their parameters and momentum buffers, although this doubles the communication overhead.
We refer to this approach as double-averaging below.
Scaman et al.
(2019) establish optimal rates of convergence for decentralized optimization methods in the deterministic, convex setting.
Richards & Rebeschini (2019) provide guarantees on the generalization error of non-parametric least-squares regression trained using decentralized gradient descent, showing that there are regimes where one can achieve a linear speedup.
Neither of these results apply directly to the setting considered in this paper, which focuses on smooth non-convex stochastic optimization, and extending this line of work to non-convex settings is an interesting direction.
Mahajan et al.
(2018a) propose an approach to distributed learning of linear classiﬁers ( i.e. , convex problems) where, in parallel, workers minimize locally formed approximate loss functions, and then the resulting minimizers are averaged to determine a descent direction.
Methods which ﬁt in the
S LOW M O framework, including Local SGD, BMUF (Chen & Huo, 2016), and the serial Lookahead optimizer (Zhang et al., 2019), can be seen as related to this approach, where the actual loss function at each worker is used rather than an approximate one, and where the descent direction is used in a momentum update rather than a (deterministic) line search method.
Note that various approaches to gradient compression have been proposed to reduce the communication overhead for A LL R EDUCE and decentralized learning methods (Alistarh et al., 2007; Wen et al., 2007;
Bernstein et al., 2019; Karimireddy et al., 2019; Koloskova et al., 2019b; Vogels et al., 2019).
However, it is presently not clear to what extent compression may be beneﬁcial for methods like
BMUF, D-PSGD, SGP, and OSGP, which perform averaging on the model parameters rather than on gradients.
Combining S LOW M O with compression techniques is an interesting and important direction for future work.
Finally, although momentum methods are known to achieve accelerated rates for deterministic optimization, currently the theoretical understanding of the beneﬁts of momentum methods (both serial and parallel) is limited (Bottou et al., 2018).
Loizou & Richt´arik (2017) show that accelerated convergence rates can be achieved when the objective is a quadratic ﬁnite sum (i.e., least squares problem) that satisﬁes an interpolation condition.
Can et al.
(2019) show that accelerated rates can also be achieved in the more general setting of smooth, strongly convex objectives, in a regime where the noise level is below an explicit threshold, and where the rate of convergence is measured in terms of the 1-Wasserstein metric of the distribution of trajectories produced by the momentum method.
In the setting of smooth non-convex functions, Gitman et al.
(2019) establish stability and asymptotic convergence results for the quasi-hyperbolic momentum method (Ma & Yarats, 2019), which can be viewed as interpolating between SGD and a stochastic momentum method.
Extending these results, which focus on the serial setting, to obtain decentralized momentum methods with guaranteed acceleration, is an important direction for future work.
E XPERIMENTAL R ESULTS
We evaluate the effectiveness of S LOW M O on three datasets: image classiﬁcation on CIFARand ImageNet, and neural machine translation on WMT’16-En-De.
All experiments use NVIDIA
DGX-1 servers as worker nodes.
Each server contains 8 NVIDIA V100 GPUs and the servers are internetworked via commodity 10 Gbps Ethernet.
On CIFAR-10 (Krizhevsky et al., 2009), we train a ResNet-
(He et al., 2016) using 32 V
GPUs, located on 32 different worker nodes.
The total mini-batch size is 4096 , and we train forepochs.
The learning rate ( γ t ) linearly increases during the ﬁrst 5 epochs, following the warm-up strategy in Goyal et al.
and then decays by a factor of 10 at epochs 100 , 150 , and 175 .
The (fast) learning rate was tuned separately for each base optimizer.
All experiments were run 5 times with different random seeds, and the mean metrics are reported.
On ImageNet (Krizhevsky et al., 2012), we train a ResNet-
(He et al., 2016) using 32 worker nodes ( i.e. , 256 GPUs).
The total mini-batch size is 8192 , and we train for 90 epochs.
The learning rate schedule is identical to (Goyal et al., 2017), i.e. , linear warm-up in the ﬁrst 5 epochs and decay by a factor of 10 at epochs 30 , 60 and 80 .
On WMT’16-En-De, we train a transformer model (Vaswani et al., 2017) using 8 worker nodes ( i.e. , 64 GPUs).
The model is trained with 200 k token batches, and we train for 25 epochs.
We follow the experimental setting of Ott et al.
For each task, we consider several baselines: (i) Local SGD /Local
Adam, where worker nodes independently run single-node SGD (with Nesterov momentum) or Adam and periodically average model parameters; (ii) stochastic gradient push (SGP), the state-of-the-art synchronous decentralized training method; and (iii) Overlap-SGP (OSGP), an asynchronous version of SGP.
For each baseline, we examine its performance with and without S LOW M O .
Recall that Local SGD and Local Adam with S LOW M O are equivalent to BMUF.
Local SGD and Local Adam do not involve communication during the inner loop (base optimizer) updates, while SGP and OSGP involve gossiping with one peer at every step.
In addition, we also evaluate the performance of AR-SGD/AR-Adam, the traditional A LL R EDUCE implementation of parallel SGD/Adam.
Details of all baseline methods are provided in Appendices A and C.
In general, the hyperparameters of S LOW M O (slow learning rate α , slow momentum β , and number of inner loop steps τ ) need to be tuned for each base optimizer and task.
The results in Table 1 all use α = 1 , which we found to be consistently the best.
For Local SGD (with or without S LOW M O ), we set τ = 12 , and for all other baseline methods we use τ = 48 .
Using τ > 12 for Local SGD resulted in signiﬁcantly worse loss/accuracy on ImageNet and WMT’16 En-De.
Note also that all of our baselines (or base optimizers) leverage a local momentum scheme, following previous works (Assran et al., 2019; Koloskova et al., 2019a).
When using these methods with
S LOW M O , there are different ways to handle the base algorithm local momentum buffers at the beginning of each outer loop (line 2 in Algorithm 1): zeroing, averaging among workers, or maintaining the current local value.
Appendix B.4 provides an empirical comparison.
For the experiments reported here, when using SGD with Nesterov momentum as the base algorithm (CIFAR-10 and Imagenet)
we zero the base algorithm buffer, and when using Adam as the base algorithm (WMT’ En-De) we maintain the current value of the Adam buffers.
We also tried to apply S LOW M O on top of AR-SGD base optimizer, but we did not observe any improvement in that setting.
Optimization and Generalization Performance.
Table 1 shows the best training loss and the validation accuracy/BLEU score for each baseline, with and without S LOW M O .
Using S LOW M O consistently improves both the optimization and generalization performance across all training tasks and baseline algorithms.
Figure 2 presents validation error/loss per epoch to give a sense of convergence speed.
Observe that SGP with S LOW M O substantially improves convergence, compared to
SGP alone.
We observe a similar phenomenon when comparing the training curves; see Appendix B. [Table]
Communication Cost.
Table 2 shows the average training time per iteration on ImageNet and
WMT’16.
For SGP/OSGP, since the additional communication cost due to averaging in line 6 of
Algorithm 1 is amortized over τ = 48 iterations, S LOW M O maintains nearly the same speed as the corresponding base algorithm.
For methods like Local SGD or Local Adam, which already compute an exact average every τ iterations, using SlowMo ( i [Table] e. , using β > 0 ) does not increase the amount of communication.
In other words, using S LOW M O on top of the base algorithm improves training/validation accuracy at a negligible additional communication cost.
Effects of τ .
The most important hyper-parameter in S LOW M O is the number of base optimizer steps τ before each S LOW M O update, since it inﬂuences both the accuracy and the training time.
Figure 3 presents the validation accuracy and average iteration time of SGP-S LOW M O for different values of τ on ImageNet and WMT’16.
It can be observed that the validation performance does not monotonically increase or decrease with τ .
Instead, there is a best value.
On both ImageNet and
WMT’16, we ﬁnd τ = 48 to be a good tradeoff between speed and accuracy.
Moreover, S LOW M O is pretty robust to the choice of τ ; even if τ = 96 for ImageNet and τ = 192 for WMT’16, SGP with S LOW M O achieves better validation accuracy/loss than SGP alone.
We further investigate the effect of other hyperparameters (the slow learning rate α , slow momentum β ) as well as the different strategies for handling base algorithm buffers in Appendix B. [Figure] [Figure] [Figure]
[Figure]
is better), and for WMT’16 En-De we plot validation NLL (lower is better). Increasing τ amortizes communication cost over more iterations, so the average time per iteration decreases. We hypothesize that moderate values of τ have a regularizing effect, improving loss and accuracy, and when τ is too large performance is degraded because workers’ local models drift too far apart. Comparison with Double-Averaging Momentum.
As mentioned in Section 3, Yu et al.
(2019a) propose an alternative momentum scheme, double-averaging, to improve the convergence of Local
SGD and D-PSGD.
We empirically compare it with S LOW M O in terms of the validation accuracy and average training time per iteration on ImageNet.
When the base algorithm is SGP, double averaging achieves 75 .
54% validation accuracy and takes 402 ms per iteration on average, while
S LOW M O -SGP ( τ = 48 ) reaches 75 .
73% validation accuracy while taking 302 ms per iteration on average.
Similarly, when the baseline algorithm is Local SGD with τ = 12 , double-averaging reaches 72 .
04% and takes 405 ms per iteration, while S LOW M O reaches 73 .
24% and takes onlyms per iteration.
T HEORETICAL R ESULTS
This section provides a convergence guarantee for S LOW M O and shows that it can achieve a linear speedup in terms of number of workers.
Let f i ( x ) =
E ξ i ∼ D i [ F i ( x ; ξ i )] denote the expected objective function at worker i , and let f ( x ) = 1 m (cid:80) mi =
f i ( x ) .
Our analysis is conducted for a constant learning rate γ t = γ under the following standard assumptions.
Assumption 1 ( L -smooth) .
Each local objective function f i ( x ) is L -smooth, i.e., (cid:107)∇ f i ( x ) − ∇ f i ( y ) (cid:107) ≤ L (cid:107)
x − y (cid:107) , for all x ,
y ∈ R d and i ∈ { 1 , 2 , . .
, m } .
Assumption 2 (Bounded variance) .
There exists a ﬁnite positive constant σ 2 such that
E
[Table]
In order to generalize the analysis to various base algorithms, we deﬁne d t,k = 1 m (cid:80) mi =1 d ( i ) t,k as the average descent direction across the m workers and make the following assumption.
Assumption 3.
There exists a ﬁnite positive constant V such that E (cid:107) d t,k − E t,k [ d t,k ] (cid:107) 2 ≤ V , where E t,k denotes expectation conditioned on all randomness from stochastic gradients up to the k -th step of t -th outer iteration.
As mentioned in Section 2, the analytic form of d t,k depends on the choice of base algorithm.
Therefore, the value of V also changes.
For instance, when the base algorithm is Local-SGD, then d t,k = 1 m (cid:80) mi =1 ∇ F i ( x ( i ) t,k ; ξ ( i ) t,k ) .
It follows that
[Equation]
The above value ( V = σ 2 /m ) can also be applied to other base algorithms, such as D-PSGD, SGP, and OSGP.
More details are provided in Appendix C.
Our main convergence result is stated next.
Proofs of all results in this section appear in Appendix D.
Theorem 1 ( General Result ) .
Suppose all worker nodes start from the same initial point x 0 , 0 , and the initial slow momentum is u 0 = 0 .
If we set α , β , γ t = γ , τ and T so that αγ 1 − β = (cid:112) mτT and the total iterations τT satisﬁes τT ≥ mL 2 (cid:16) 1 + √ 3 max (cid:110) 3 τ (1 − β − α ) α ,
[Equation]
τβ 1 − β , 1 (cid:111)(cid:17) , then under
Assumptions 1 to 3, we have that:
τT T
[Equation]
[Table]
[Equation]
[Table]
[Equation]
where f inf = inf x f ( x ) .
Consistent with AR-SGD.
Recall that AR-SGD is equivalent to taking τ = 1 , α = 1 , and β =and using SGD with learning rate γ as the base optimizer.
In this case, all terms on the RHS but the ﬁrst one vanish, V = σ 2 /m , and (5) is identical to the well-known rate of O (1 / √ mTτ ) for SGD.
Effect of the base optimizer.
The second term in (5) only depends on the base optimizer.
It measures the bias between the full batch gradient ∇ f ( x t,k ) and the expected update averaged across all workers E t,k [ d t,k ] .
For the base optimizers considered in this paper, this term relates to the discrepancies among local models and can be easily found in previous distributed optimization literature.
In particular, under the same assumptions as Theorem 1, one can show that this term vanishes in a rate of 1 / ( Tτ ) for D-PSGD, SGP, OSGP and Local-SGD; see Appendix C.
As an example, we provide the convergence analysis for the extreme case of Local SGD, where there is no communication between nodes during each inner iteration.
Intuitively, using other base algorithms should only make this term smaller since they involve more communication than Local SGD.
Corollary 1 ( Convergence of BMUF, i.e. , Local SGD with S LOW M O ) .
Under the same conditions as Theorem 1, if the inner algorithm is Local SGD and there exists a positive ﬁnite constant ζ such that 1 m (cid:80)
[Equation]
f [Equation] x ) − ∇ f i ( x ) (cid:107) 2 ≤ ζ 2 , then
[Equation]
Linear speedup.
Corollary 1 shows that when the total number of steps Tτ is sufﬁciently large:
T ≥ m 3 τ 3 , the convergence rate will be dominated by the ﬁrst term O (1 / √ mTτ ) .
That is, in order to achieve an (cid:15) error, the algorithm requires m times less steps when using m times more worker nodes.
This also recovers the same rate as AR-SGD.
Extension to single-node case.
As mentioned in Section 2, when there is only one node and the slow momentum factor is β = 0 , the S LOW M O -SGD is the Lookahead optimizer.
One can directly apply Theorem 1 to this special case and get the following corollary.
Corollary 2 ( Convergence of Lookahead ) .
Under the same conditions as Theorem 1, if the inner optimizer is AR-SGD and β = 0 , then one can obtain the following upper bound:
[Equation]
F ASTER S LOW M O : R EMOVING THE P ERIODIC A LL R EDUCE
S LOW M O helps improve both the optimization and generalization of communication-efﬁcient algorithms.
When the base optimizer is SGP or OSGP, S LOW M O also comes at the expense of higher communication cost, since it requires performing an exact average every τ iterations.
1 Although the communication cost can be amortized, here we go one step further and propose a SGP-S LOW M O variant, named SGP-S LOW M O -noaverage, where we remove the exact average when we perform the S LOW M O update, i.e we skip line 6 in Algorithm 1.
We empirically evaluate this variant on the
ImageNet and WMT’16 datasets, using α = 1 , β = 0 .
6 and τ = 48 .
Surprisingly, we observe that SGP-S LOW M O -noaverage achieves similar performances on Imagenet ( 75 . 78% , compared to 75 .
73% for SGP-S LOW M O ) and only slightly degrades the validation NLL on
WMT’16 ( 2 . 11 , compared to 2 . 10 ), while preserving the iteration time of the base algorithm (ms per iteration on ImageNet and 1227 ms per iteration on WMT’16) since this variant does not require additional communication.
These results suggest that the slow momentum updates, and not the momentum buffer synchronization, contribute the most to the performance gain of S LOW M O .
We leave further investigation of S LOW M O -SGP-noaverage for future work.
C ONCLUDING R EMARKS
In this paper, we propose a general momentum framework, S LOW M O , for communication-efﬁcient distributed optimization algorithms.
S LOW M O can be built on the top of SGD, as well as decentralized methods, such as SGP and (asynchronous) OSGP.
On three different deep learning tasks, we empirically show that S LOW M O consistently improves the optimization and generalization performance of the corresponding baseline algorithm while maintaining a similar level of communication efﬁciency.
Moreover, we establish a convergence guarantee for S LOW M O , showing that it converges to a stationary point of smooth and non-convex objective functions.
Since BMUF (Chen &
Huo, 2016) can be expressed through S LOW M O framework (by setting the base optimizer to be Local SGD or Local Adam), to the best of our knowledge, we provide the ﬁrst convergence guarantee for BMUF in the literature.
A E XPERIMENT D ETAILS A. I MPLEMENTATION D ETAILS
All methods are implemented in PyTorch 1.0 (Paszke et al., 2017), and our experiments use
CUDA 9.2, CUDNN 7.3, and NCCL 2.2.13.
The ImageNet experiments build on the example from https://github.com/pytorch/examples/imagenet .
The WMT’16 En-De experiments build on https://github.com/pytorch/fairseq .
For SGP and OSGP we use the implementations available at https://github.com/facebookresearch/stochastic_ gradient_push .
A. CIFAR-
For the CIFAR-10 experiments, we train a ResNet-18, the implementation of which is available at https://github.com/kuangliu/pytorch-cifar/blob/master/models/ resnet.py .
In all base algorithms, we use a Nesterov momentum parameter of 0 . 9 and set the weight decay factor as 10 − 4 .
For each base algorithm, we tune the (fast) learning rate from { 0 . 01 , 0 . 025 , 0 .
05 , 0 . 1 , 0 . 15 } and linearly scale it with the number of workers ( i.e. , 32 ).
We found that, with a total batch size 4096 , the best learning rate for AR-SGD is 0 . 01 , for OSGP/SGP is 0 . 05 , and for Local SGD is 0 . 025 .
When applying S LOW M O to these base algorithms, we ﬁx α = 1 and τ = 12 and tune the value of β from { 0 . 4 , 0 . 5 , 0 . 6 , 0 . 7 , 0 . 8 } .
It turns out that for OSGP, SGP, and Local SGD, the best values of β are all equal to 0 .
More discussion on the effects of α and β can be found in Appendix B.3.
A. I MAGE N ET
For the ImageNet experiments, we use the same learning-rate, schedule, momentum, and weight decay as those suggested in Goyal et al.
(2017) for SGD.
In particular, we use ResNet50 (He et al.
and train it for 90 epochs with a reference learning-rate of 0.1 with respect to a 256 sample batch, and scale this linearly with the batch-size.
We decay the learning-rate by a factor of 10 at epochs 30, 60, 80.
We use a Nesterov momentum parameter of 0.9.
We use weight decay 10 − 4 .
When using S LOW M O , we set the slow learning rate to α = 1 and explore different numbers of inner steps, τ ∈ { 12 , 48 } and different slow momentum value β ∈ { 0 . 5 , 0 . 6 , 0 .
7 } when the base optimizer is SGP/OSGP and β = 0 .
7 when the base optimizer is LocalSGD.
We also explore a larger set of τ values τ ∈ { 12 , 24 , 48 , 96 } in the ablation experiments.
A. WMT16 E N -D E
For the WMT16 En-De experiments, we follow the experimental protocol described in Ott et al.
All experiments are based on the big transformer model (Vaswani et al., 2017) with 6 blocks in the encoder and decoder networks.
For these experiments, the base optimizer used is Adam (Kingma & Ba, 2015) with beta1 = 0 .
9 , beta2 = 0 .
98 , and (cid:15) =
10 − 8 and trained for 25 epochs.
We use the same learning rate schedule as Ott et al.
(2018), i.e. , the learning rate increases linearly for 4 , 000 steps to 10 − 3 , after which it is decayed proportionally to the inverse square root of the number of steps.
We use label smoothing with weight 0.1 for the uniform prior distribution.
For S LOW M O , we explore { 0 . 5 , 1 . 0 } as the slow learning rate α .
We observe that α =gives better performance and therefore report results for α = 1 unless stated otherwise.
We explore different numbers of inner steps, τ ∈ { 12 , 48 } and different slow momentum value β ∈ { 0 . 1 , 0 . 3 , 0 . 5 , 0 . 6 , 0 . 7 } .
We also explore a larger set of τ values, i.e. τ ∈ { 12 , 24 , 48 , 96 , 192 } , in the ablation experiments.
[Table]
B A DDITIONAL E MPIRICAL R ESULTS B. V ALIDATION NLL ON WMT’16 E N -D E
We show the Validation NLL on WMT’16 En-De in Table B.1, corresponding to the experiments in
Table 1.
We observe that S LOW M O improves the validation NLL (along with BLEU score) of SGP and Local Adam.
B. A DDITIONAL T RAINING C URVES
We present the training loss-versus-epochs curves in Figure B.1, corresponding to the validation curves in Figure 2.
It can be observed that S LOW M O substantially improves the convergence speed of SGP.
[Figure]
base algorithm.
We ﬁx α = 1 , τ =for these three plots.
Shaded areas in (a) and (b) show the min-max values across all worker nodes.
The corresponding validation curves are presented in Figure 2.
Note that the training losses in these three ﬁgures are evaluated right after the S LOW M O update ( i.e. , Eq.
B. E FFECT OF S LOW L EARNING R ATE α AND S LOW M OMENTUM F ACTOR β
In this section we evaluate the impact of slow learning rate α and slow momentum β hyperameters.
In Figure B.2a, we perform a parameter sweep over α and β on CIFAR-10 dataset, using OSGP as the base algorithm of S LOW M O .
One can observe that when the value of β is ﬁxed, α = 1 always gives the highest validation accuracy; when the value of α is ﬁxed, there is a best value of β ranging from 0 .
4 to 0 .
We further validate this claim on the WMT’16-En-De dataset.
Figure B.2b shows that α =gives lower validation loss than α = 0 .
5 for ﬁxed β when using SGP or Local Adam as the base algorithms.
When running S LOW M O -Adam with β > 0 .
5 and α = 1 . 0 , or with β > 0 .
7 and α = 0 .
5 , the validation loss was substantially worse and so is not plotted here.
Motivated by the above observations, we stick to ﬁx α = 1 and ﬁne-tune β for S LOW M O methods on all training tasks. [Figure]
[Figure]
B ASE O PTIMIZER M OMENTUM B UFFER S TRATEGIES
As described in Section 2, the base optimizer may have some associated buffers.
SGD with momentum uses a momentum buffer, and Adam tracks estimates of the ﬁrst and second moments of the gradient.
The slow momentum buffer is updated every τ steps according to Eq.(2).
There are several strategies that can be used to update the base optimizer buffers at the outer iteration level (line 2 in Algorithm 1).
Here, we explore three strategies: 1) reset the base optimizer buffers to zero ; 2) maintain the base optimizer buffers to their current values; 3) average the base optimizer buffers across workers, which requires additional communications.
We evaluate the impact of these strategies on ImageNet and WMT’16 in Table B.2 and Table B.3.
On ImageNet, we observe that the different buffer strategies achieve similar training and validation performance.
However, the averaging strategy comes at the cost of higher communication overhead (an additional call to A LL R EDUCE for each buffer averaged).
Based on these results, we choose the reset strategy as the default in our experiments.
On WMT’16, we ﬁnd that the reset buffer strategy underperforms both the maintain and average approaches.
When using Adam as base optimizer, reseting the second moment to zeros hurts the optimization performance.
This is not surprising since it is recognized that warming up the Adam buffer is important.
Averaging buffers achieves the best results but comes at a signiﬁcantly higher communication cost.
We therefore select the maintain strategy as the default one when using Adam.
[Table] [Table]
S TANDARD D EVIATIONS ON CIFAR-
Since experiments for CIFAR-10 were ran for 5 times with different random seeds, here we report the standard deviations on the validation accuracy in Table B.4, as a complementary to Table 1.
[Table]
C B ASELINE A LGORITHMS
In this section we give a detailed description of each baseline algorithms used throughout the paper, provide theoretical justiﬁcation on how to incorporate the update rules of D-PSGD, SGP and OSGP into the analysis of S LOW M O , and also derive the analytic form of V used in Assumption 3 for each method.
A summary of the base optimizer update directions is given in Table C
[Table]
Base Optimizer
Update directions
d ( i ) t,k (and possible additional buffers h
( i ) , v ( i ) )
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Equation]
k
[Equation]
[-(cid:80) j ∈N in ( i ) k p ( i,j ) [ x ( i ) t,k − γ t ˆ h ( i ) t,k +1 / ( (cid:113) ˆ v ( i ) t,k +
-]+
[Equation]
C. SGP AND OSGP
Algorithms 2 and 3 present the pseudo-code of SGP and OSGP (Assran et al., 2019).
To be consistent with the experimental results of the original paper, we also use local Nesterov momentum for each worker node.
The communication topology among worker nodes is a time-varying directed exponential graph Assran et al.
That is, if all nodes are ordered sequentially, then, according to their rank ( 0 , 1 , . . .
, m − 1 ), each node periodically communicates with peers that are 2 0 , 2 1 , . . .
, 2 (cid:98) log 2 ( m − 1) (cid:99) hops away.
We let each node only send and receive a single message ( i.e. , communicate with 1 peer) at each iteration.
[Table]
[Table]
[Table]
[Table]
[Table]
. , m } ; count since last = 0.
1 for k ∈ { 0 , 1 , . . .
, K − 1 } at worker i in parallel do
Compute mini-batch gradients: ∇ F i ( z ( i ) k ; ξ ( i ) k )
Update local momentum: h ( i ) k
+1 =
β 0 h ( i ) k
+ ∇ F i ( z ( i ) k ; ξ ( i ) k ) x
( i ) k
+
= x ( i ) k − γ [ β 0 h ( i )
k +1 + ∇ F i
( z ( i ) k ; ξ ( i ) k )]
Non-blocking send ( p ( j,i ) k
x ( i ) k
+ 12 , p ( j,i ) k w ( i ) k ) to out-neighbors x ( i ) k
+1 =
p ( i,i ) x
( i ) k w ( i ) k
+1 =
p ( i,i ) w ( i ) k if count since last =
s then
Block until ( p ( i,j ) k x ( j ) k + 12 , p ( i,j ) k w ( j ) k ) is received from in-neighbors count since last = else count since last = count since last + end if Receive buffer non-empty then for ( p ( j,i ) k (cid:48)
x ( i ) k (cid:48) + 12 , p ( j,i ) k (cid:48) w ( i ) k (cid:48) ) in the receive buffer do x ( i ) k
+1 =
x ( i ) k
+1 + p
( i,j ) k (cid:48)
x ( j ) k (cid:48)
+ w ( i ) k
+1 =
w ( i ) k
+1 + p
( i,j ) k (cid:48)
w ( j ) k (cid:48)
+ end end
Update de-biased model parameters: z ( i ) k
+1 =
x ( i ) k
+1 /w
( i ) k
+ 21 end
Note that although the implementation of SGP is with Nesterov momentum, the theoretical analysis in Assran et al.
(2019) only considers the vanilla case where there is no momentum.
Accordingly, the update rule can be written in a matrix form as
[Equation]
where X k =
[ x (1) k , . .
, x ( m ) k ] ∈ R d × m stacks all model parameters at different nodes and Z k =
[ z (1) k , . .
, z (
m ) k ] ∈ R d × m denotes the de-biased parameters.
Similarly, we deﬁne the stochastic gradient matrix as ∇ F ( Z k ) =
[ ∇ F 1 ( z (1) k ; ξ ( i ) k ) , . .
. , ∇ F m ( z ( i ) k ; ξ ( i ) k )]
∈ R d × m .
Moreover, P k ∈ R m × m is deﬁned as the mixing matrix which conforms to the underlying communication topology.
If node j is one of the in-neighbors of node i , then p ( i,j ) > 0 otherwise p ( i,j ) = 0 .
In particular, matrix P k is column-stochastic.
If we multiply a vector 1 /m on both sides of the update rule (9), we have
[Equation]
where x k = X k 1 /m denotes the average model across all worker nodes.
Recall that in S LOW M O , we rewrite the updates of the base algorithm at the k th steps of the t th outer iteration as
[Equation]
So comparing (10) and (11), we can conclude that d t,k = 1 m (cid:80) m i
=1 ∇ F i
( z ( i ) t,k ; ξ ( i ) t,k ) .
As a consequence, we have E t,k
[ d t,k ] = 1 m (cid:80) mi =1 ∇ f i ( z ( i ) t,k ) .
Since mini-batch gradients are independent, it follows that
[Equation]
Similarly, for OSGP, one can repeat the above procedure again.
But the deﬁnition of X k , Z k and ∇ F ( Z k ) will change, in order to account for the delayed messages.
In this case, we still have the update rule Eq.
But x k is no longer the averaged model across all nodes.
It also involves delayed model parameters.
We refer the interested reader to Assran et al.
for futher details.
C. D-PSGD
In the case of decentralized parallel SGD (D-PSGD), proposed in Lian et al.
(2017), the update rule is quite similar to SGP.
However, the communication topology among worker nodes is an undirected graph.
Hence, the mixing matrix P k is doubly-stochastic.
Each node will exchange the model parameters with its neighbors.
The update rule can be written as
[Equation]
Again, we have that X k =
[ x (1) k , . .
, x ( m ) k ] ∈ R d × m stacks all model parameters at different nodes and ∇ F ( X k ) =
[ ∇ F 1 ( x (1) k ; ξ ( i ) k ) , . .
. , ∇ F m ( x ( i ) k ; ξ ( i ) k )]
∈ R
d × m denotes the stochastic gradient matrix.
By multiplying a vector 1 /m on both sides of (15), we have
[Equation]
As a result, using the same technique as (12)-(14), we have V = σ 2 /m for D-PSGD.
C. L OCAL SGD
We further present the pseudo-code of Local SGD in Algorithm 4, and the pseudo-code of doubleaveraging momentum scheme in Algorithm 5.
Input: learning rate γ ; momentum factor β 0 ; Communication period τ ; Number of worker nodes m all nodes m } .
1 for k ∈ { 0 , 1 , . . . , K − 1 } at worker i in parallel do Compute mini-batch gradients: ∇ F i ( x ( i ) k ; ξ ( i ) k ) Update local momentum: h ( i ) k
+1 =
β 0 h ( i ) k + ∇ F i ( x ( i ) k ; ξ ( i ) k ) x
{+( i ) k + 
 = x ( i ) k − γ [ β 0 h ( i ) k +1 + ∇ F i+}
( z ( i ) k ; ξ ( i ) k )]
if k mod τ = 0 then
A LL R EDUCE model parameters: x ( i ) k
+
= 1 m (cid:80) mj
=1 x ( j ) k + else x ( i ) k +1 = x ( i ) k
+ end 10 end learning rate γ momentum factor β period k ∈ { 0 , 1 , . . . , K − 1 } at worker i in parallel do Compute mini-batch gradients: ∇ F i ( x ( i ) k ; ξ ( i ) k ) Update local momentum: h ( i ) k
+ 12 = β 0 h ( i ) k
+ ∇ F i ( x ( i ) k ; ξ ( i ) k ) x
( i ) k
{++ 
+}
= x ( i ) k − γ [ β 0 h ( i ) k
+ 12 + ∇ F i
z ( i ) k ; ξ ( i ) k )] if k mod τ = 0 then
A LL R EDUCE model parameters: x ( i ) k
{++
+}
= 1 m (cid:80) mj
=1 x ( j ) k
+ A LL R EDUCE momentum buffers: h ( i ) k +
= 1 m (cid:80) mj
=1 h ( j ) k
+ else x ( i ) k
+1 =
x ( i ) k
+ h ( i ) k +1 =
h ( i ) k
+ end 12 end
D P ROOFS D. E QUIVALENT U PDATES
To begin, recall that d t,k = 1 m (cid:80) mi =1 d ( i ) t,k , and that the local updates are
[Equation]
(18) for k = 0 , . . .
, τ − 1 , followed by an averaging step to obtain x t,τ = 1 m (cid:80) m i =1 x ( i ) t,τ .
Therefore, we can write the update rule of the base optimizer as
[Equation]
Combining this with (2) and (3), we have
[Equation] [Equation]
Let y t,
= x t, 0 + β 1 − β ( x t, 0 − x t − 1 , 0 ) , ∀ t .
Then by rearranging terms we get
[Equation]
Now, let us further extend the auxiliary sequence to all values of k (cid:54)
= 0 as follows:
[Equation]
It is easy to show that y t,τ = y t
+1 , 0 .
In the sequel, we will analyze the convergence of sequence { y t,k } instead of { x t,k } .
D. P RELIMINARIES
In the table below, we list all notations used in this paper.
[Table]
Throughout the theoretical analysis, we will repeatedly use the following facts:
• Fact 1 : (cid:104) a , b (cid:105) =
12 (cid:107) a (cid:107) 2 + 12 (cid:107) b (cid:107) 2 − 12 (cid:107) a − b (cid:107) 2 ; • Fact 2 :
According to Young’s Inequality, for any a > 0 , we have ± (cid:104) a , b (cid:105) ≤ 2 a (cid:107) a
[Equation]
• Fact 3 : (cid:107)
a + b
[Equation]
[Equation]
; • Fact 4 : Suppose { a i } ki =1 is a set of non-negative scalars and s = (cid:80) ki =1 a i .
Then according to Jensen’s Inequality, we have
[Equation]
D. G ENERAL T REATMENT
Since each local objective f i is L -smooth
[Equation]
the function f = 1 m (cid:80) mi =1 f i is also L -smooth.
It follows that
[Equation]
where E t,k denotes a conditional expectation over the randomness in the ( t, k ) -th iteration, conditioned on all past random variables.
For the ﬁrst term on the right hand side:
[Equation]
where (27) comes from Fact 4 (24)
[Equation]
and a > 0 is constant.
For simplicity, we directly set a = 0 .
Eqn.
(28) uses Fact 2 (cid:104) a , b (cid:105)
=
12 (cid:107)
[Equation]
(cid:107)
[Equation]
+ 12 (cid:107)
[Equation]
(cid:107)
[Equation]
− 12 (cid:107)
[Equation]
− b (cid:107) 2 .
Furthermore, according to the deﬁnition of y t,k , it can be shown that
[Equation]
Substituting (32) into (29), it follows that
[Equation] [Equation]
Moreover
[Equation]
for the second term in (26)
[Equation]
we have
[Equation]
Then, plugging (33) and (34) into (26),
[Equation]
where γ eff = αγ/ (1 − β ) .
Taking the total expectation,
[Equation]
Summing from k = 0 to k = τ − 1 , we have
[Equation]
where N 1 ( t, k ) , N 2 ( t, k )
[Equation]
and N 3 ( t ) are as deﬁned in (36).
Summing from t = 0 to t =
T − 1 and dividing both side by total iterations K
=
τT ,
[Equation]
T − 1 (cid:88) t = τ − 1 (cid:88) k
=
N 2 ( t, k ) + 1 K
T − 1 (cid:88) t = τ − 1 (cid:88) k
=
N 3 ( t ) .
Now, we are going to further expand the expressions of the last two terms in (39).
D.3. B OUNDING N 2 ( t, k )
Using the fact (cid:107) a + b (cid:107) 2 ≤ 2 (cid:107) a (cid:107) 2 + 2 (cid:107) b (cid:107) 2 , we have (cid:13) (cid:13) (cid:13) (cid:13)(cid:13)(cid:13) k − 1 (cid:88)
j = d t,j (cid:13) (cid:13) (cid:13) (cid:13)(cid:13)(cid:13) ≤(cid:13) (cid:13) (cid:13) (cid:13)(cid:13)(cid:13) k − 1 (cid:88)
j = ( d t,j − E t,j [ d t,j ]) (cid:13) (cid:13) (cid:13) (cid:13)(cid:13)(cid:13) +(cid:13) (cid:13) (cid:13) (cid:13)(cid:13)(cid:13) k − 1 (cid:88) j =
E t,j [ d t,j ] (cid:13) (cid:13) (cid:13) (cid:13)(cid:13)(cid:13) (40) ≤ 2 (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) k − 1 (cid:88)
j = ( d t,j − E t,j [ d t,j ]) (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
where the last inequality comes from Fact 3.
Then, taking the total expectation and summing over the t -th outer iteration,
[Equation]
where (43) uses the following fact:
[Equation]
As a result
[Equation]
we end up with the following
[Equation]
where K = τT denotes the total steps
[Equation]
D.3. B OUNDING N 3 ( t )
From the update rule (2), (3) and (18), we have
[Equation]
(cid:125) [Equation] +
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
(cid:88) [Equation] [Equation] (cid:32)
[Equation]
(cid:88)
[Equation]
E
[Equation]
] (cid:33)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) (cid:124) (cid:123)(cid:122) (cid:125) T(55) [Table] E
[Equation]
= E  [Equation] (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) t
[Equation]
(cid:88) s
=
β t
[Equation]
[Equation]
(cid:32)
[Equation]
(cid:88) k = ( d s [Equation] k
[Equation]
E s
[Equation]
k [ d s,k ]) (cid:33)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) 2   (56) =
t
[Equation]
1 (cid:88) s =
[Equation]
[Equation]
[Equation]

[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
τ
[Equation]
(cid:88) k
= ( d s,k − E s,k
[ d s,k ]) (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) 2   (57) =
t −
[Equation]
(cid:88) s =
[Equation]
( t
[Equation]
[Equation]
s )
τ
[Equation]
(cid:88) k
=
E (cid:104) (cid:107) d s
[Equation]
k − E s
[Equation]
[ d s,k ]) (cid:107)
[Equation]
(cid:105) (58) ≤ τV t − 1 (cid:88)
[Equation] [Equation] where (57) and (58) are derived using the same routine as (46) to (48).
Similarly, for the second term T 2 in (55), according to Fact 4,
[Equation]
Substituting (59) and (62) back into (55) and summing over the t -th outer iteration, we have
[Equation]
D.3. F INAL R ESULTS
Plugging (51) and (68) back into (39), one can obtain
E [ f ( y T, 0 ) − f ( y 0 , 0 )]
[Equation]
69) where C 1 = 1 / 2 − γ eff L − 4 γ 2 eff L 2 τ ( τ − 1)(1 − β − α ) 2 /α 2 − 8 γ 2 eff
L 2 τ 2 β 2 / (1 − β ) 2 .
When the constants satisfy
[Equation]
we have
E
[Equation]
[Equation]
After rearranging terms
[Equation]
we get
K
[Equation]
Furthermore, since y 0 , 0 = x 0 , 0 − β x − 1 , 0 / (1 − β ) = x 0 , 0 and f ( y T, 0 ) ≥ f inf , the above upper bound can be simpliﬁed as
[Equation]
If we set γ eff = (cid:112) mK
[Equation]
then
[Equation]
Recall the learning rate constraint is
[Equation]
When γ eff = √ m √
K , the constraint can be rewritten as
[Equation]
After rearranging
[Equation]
we have
[Equation] [Equation]
Furthermore, note that
[Equation] [Equation]
Therefore, when K ≥ mL
[Equation]
, the condition (79) must be satisﬁed.
D. S PECIAL C ASE 1: B LOCKWISE M ODEL U PDATE F ILTERING (BMUF)
In this case, the inner optimizer is Local-SGD.
That is,
[Equation]
Since all worker nodes are averaged after every τ iterations, we have x ( i ) t,
= x t, 0 , ∀ i .
Besides, it is easy to validate that V = σ 2 /m .
According to previous literature on the convergence of Local-SGD (Wang & Joshi, 2018; Yu et al., 2019a)
[Equation]
we can directly get the following upper bound.
[Equation]
When γLτ ≤
[Equation]
we have 1 / (1 − 12 γ 2 L 2 τ 2 ) ≥ 3 / 2 .
It follows that
[Equation]
Substituting (88) into (73) and setting α 1 − β γL = (cid:112) m/K , we have
[Equation]
D. S PECIAL C ASE 2: L OOKAHEAD
In this case
[Equation]
the inner optimizer is SGD and m = 1 .
Thus
[Equation]
we have β = 0 , E t,k [ d t,k ] = ∇ f ( x t,k ) , and V = σ 2 .
Therefore,
[Equation]
It can be observed that when α = 1 or τ = 1 , the above upper bound reduces to the case of vanilla mini-batch SGD.
If we set αγL = (cid:112) 1 /K , then we have
[Equation]
If the total iterations K is sufﬁciently large, then the ﬁrst term will dominate the convergence rate.