D ISTRIBUTED T RAINING ACROSS THE W ORLD
Anonymous authors
Paper under double-blind review
A BSTRACT
Traditional synchronous distributed training is performed inside a cluster since it requires high bandwidth and low latency network ( e.g . 25Gb Ethernet or Inﬁniband).
However, in many application scenarios, training data are often distributed across many geographic locations , where physical distance is long and latency is high.
Traditional synchronous distributed training cannot scale well under such limited network conditions.
In this work, we aim to scale distributed learning under high-latency network .
To achieve this, we propose D elayed and T emporally
S parse (DTS) update that enables synchronous training to tolerate extreme network conditions without compromising accuracy.
We benchmark our algorithms on servers deployed across three continents in the world: London (Europe), Tokyo (Asia), Oregon (North America) and Ohio (North America).
Under such challenging settings, DTS achieves 90 × speedup over traditional methods without loss of accuracy on ImageNet.
I NTRODUCTION
Deep neural networks have demonstrated much success in solving large-scale machine learning problems (Krizhevsky et al., 2012; Simonyan & Zisserman, 2014;
He et al., 2016).
However, training deep neural networks may take days or even weeks to converge.
In order to enable training in a reasonable time, distributed training is an important technique and gains increasing attention (Li et al., 2014;
Dean et al., 2012; Recht et al., 2011; Goyal et al., 2017; Jia et al., 2018).
To maintain a good scalability, a low latency and high bandwidth network is essential for most modern distributed systems.
Existing frameworks (Chen et al., 2015; Xing et al., 2015; Moritz et al., 2015; Abadi et al., 2015; Akiba et al., 2017; Paszke et al., 2017; Sergeev & Del Balso, 2018)
all require high-end network infrastructure such as 25Gbps Ethernet or Inﬁniband where bandwidth is as large as 10 to 100 Gbps and latency is as small as 1 us.
[Figure]
while the performance of conventional algorithms degrades quickly.
Bandwidth is easy to increase ( e.g . stacking hardware) but latency is hard to improve (physical limits).
For example, if we have two servers located at Shanghai and Boston respectively, even at the speed of light and direct air distance, it still takes 78ms ∗ to send and receive a packet.
In real world scenario, the latency can be only worse (around 700ms) because indirect routing between internet service providers (ISP) and queuing delay in switches.
Such high latency cause severe scalability † issue for distributed training.
As shown in Fig.
1, traditional distributed training algorithm scales poorly under such large latency.
In many scenarios, the training data involves privacysensitive information, such as personal medical history (Jochems et al., 2016; 2017) and keyboard input history (McMahan et al., 2016; Konen et al., 2016; Bonawitz et al., 2019), thus cannot be centralized to a data center due to security and legacy concerns (GDPR, 2016).
When datasets are distributed across many
[Table] [Table]
London
Tokyo California Ohio 277m 63Mbps 17Mbp183ms 140ms 23Mbp 35ms 13Mbp
Table 1-1-1 Latency Bandwidth Scalability
Inside 3 25600 819.2 Across 158750 29 8.
Latency (us) Bandwidth (Mbps) Scalability (1024 GPUs)
Across the world Inside a cluster
London
Tokyo Oregon
Ohio 102ms 210ms 97ms 70ms
Table 1-1 Latency
Scalability
Across the world 2274 Inside a cluster 23.
Latency
Speed 479, 2,
Inside a cluster
Across the world 159,666x longer 98x worse µs µs img/s img/s locations across public clouds, private clusters and even edge devices, it is impossible to setup a low latency network under long-distance connections and thereby hurts the scalability of training.
In this work, we enable scalable distributed training across different geographical locations with long distance and high latency network connection.
We propose Delayed Update to tolerate latency by putting off the synchronization barrier to a later iteration; we also propose Temporally Sparse
Update to amortize the latency and alleviate congestion by reducing the synchronization frequency.
To ensure no loss of accuracy, we design a novel error compensation to overcome the staleness for both vanilla and momentum SGD.
We focus on the widely adopted synchronous update using data parallelism.
As shown in Fig.
1, DTS can maintain high scalability even when latency is as high as 1000ms.
This result is also better than existing state of art technologies such as ECD-PSGD (results copied directly from their original paper (Tang et al., 2018a))
We benchmark our DTS under a challenging settings: training a deep neural network on four AWS
P3 instances located in different continents of the world (Fig. 2).
The measured latency is as large as 277ms (compared to internal latency 2us).
In this case, the naive distributed synchronous SGD (SSGD) can only achieve a poor scalability of 0.008.
It means in this setting, distributed training with 100 servers is even slower than single machine (0.8 v.s. 1.0).
Meanwhile DTS achieves scalability of 0.72 without compromising the accuracy.
In conclusion, our contributions are listed below:
• We propose delayed update to tolerate the latency and temporally sparse update to amortize the latency.
While preserving the accuracy, delayed update tolerates up to 6 seconds latency and temporally sparse update reduced the trafﬁc congestion by 20 × .
• We theoretically justify the convergence rate of our proposed algorithms.
We show that both algorithms can be as fast original SGD while scaling well under high latency.
• With servers and data distributed in four different countries across the world, we can train ResNet-50 on ImageNet with scalability.
To our best knowledge, DTS is the ﬁrst work that can achieve scalable synchronous distributed training under such high latency.
R ELATED W ORK
Distributed Learning becomes ever more important as the sizes of both datasets and models increase.
Many studies have been made to explore the efﬁciency, both at the algorithm level (Li et al., 2014;
Dean et al., 2012; Recht et al., 2011; Goyal et al., 2017; Jia et al., 2018)
and at the framework level (Chen et al., 2015; Akiba et al., 2017; Abadi et al., 2015; Paszke et al., 2017; Sergeev &
Del Balso, 2018).
In most of distributed algorithms, each node performs computation and exchange updates through network.
A key component to improve scalability is to reduce the communication-tocomputation ratio.
The communication cost is determined by latency and bandwidth.
Conventional studies focus on reducing bandwidth requirements as the latency of internal trafﬁc is usually low.
Gradient quantization / compression has been proposed to reduce the data to be transferred.
Onebit gradient gradient (Seide et al., 2014) achieves 10 × speedup using 20 GPUs on text-to-speech task.
QSGD (Alistarh et al., 2016) and Terngrad (Wen et al., 2017) further improves the trade-off between accuracy and gradient precision.
But the theoretical limit of the quantization cannot go beyond 32.
To overcome the limitation, gradient sparsiﬁcation using predeﬁned static threshold (Strom, 2015; Aji &
Heaﬁeld, 2017) and dynamic compression ratio (Chen et al., 2018) demonstrate that 99% gradients can be pruned with negligible degradation on model performance.
(Sattler et al., 2018) combines both quantization and compression to push the ratio to a new level.
DGC (Lin et al., 2017) and
DoubleSequeeze (Tang et al., 2019) explore how to compensate the error to preserve the accuracy better.
The convergence of quantization and compression is also discussed in (Tang et al., 2019; Jiang & Agrawal, 2018;
Alistarh et al., 2018).
However, latency remains an issue, especially when servers are not in the same physical location.
Learning on decentralized data has become increasingly popular recently as the growing awareness of data privacy (GDPR, 2016).
For example, Federated Learning (Google, 2017) aim to jointly train a model without centralizing the data and have been used to train models for medical treatments across multiple hospitals (Jochems et al., 2016), analyze patient survival situations from various countries (Jochems et al., 2017) and build predictive keyboards to improve typing experience (McMahan et al., 2016; Google, 2017; Bonawitz et al., 2019).
However, existing federated learning works do not scale well under high latency network.
An orthogonal exploration is Decentralized Training where only partial synchronization is performed in each update, such as AD-PSGD (Lian et al., 2017) and
D 2 (Tang et al., 2018b).
None of them has been evaluated on large learning tasks yet.
Asynchronous SGD (ASGD) is derived from (Tsitsiklis et al., 1986) and has advantages in unstable latency and fault tolerance.
Different from synchronous SGD (SSGD), ASGD relaxes synchronization by allowing training on inconsistent models and powers many successful applications such as
HOGWILD!
(Recht et al., 2011), BUCKWILD!
(De Sa et al., 2015) and dist-belief (Dean et al., 2012).
However, most of them are implemented through parameter server (Dean et al., 2012;
Chilimbi et al., 2014;
Li et al., 2014) and leads to problems like communication congestion and resource congestion when scaling up.
Moreover, training on inconsistent models leads to different behaviors compared to training on a single device.
Though there are studies discussing the convergences (Recht et al., 2011; De Sa et al., 2015) and showing that ASGD converges as good as SSGD (Lian et al., 2015), synchronous SGD (SSGD) is usually preferred in practice (Goyal et al., 2017; Sun et al., 2019) because it has consistent behaviors when increasing the number of machines and therefore is easy to develop and deploy.
A PPROACH
To motivate this section, we ﬁrst review the mechanism of synchronous distributed training: at each step, each node will ﬁrst compute gradients locally, then they wait for the collective operation to transmit gradients to each other to calculate the average.
They use the averaged gradient to update the weight and continue to the next step.
As shown in right of Fig.
3a, when the communication increases , the whole training process would be drastically slowed down.
To address the problem, our proposed algorithm contains two parts: delayed update (Fig. 3b) and temporally sparse update (Fig. 3c).
With the delayed update, instead of waiting for average gradients to come back, it puts off the synchronization barrier to steps later to tolerate the latency.
With the temporally sparse update, the synchronization frequency is reduced to amortize latency.
Throughout the paper, we use the following notions:
• γ : learning rate.
• t : iteration steps that the synchronization barrier is delayed.
• p : iteration steps between temporally sparse update intervals.
• u ( i,j ) : momentum at iteration i on worker j . • u (cid:48)
( i,j ) : error compensated momentum at iteration i on worker j . • w ( i,j ) : model weights at iteration i on worker j . • w (cid:48)
( i,j ) : error compensated weights at iteration i on worker j . • ∇ w ( i,j ) : local gradients at iteration i on worker j . • ∇ w ( i ) : global averaged gradients at iteration i .
3. D ELAYED U PDATE
In modern implementations of vanilla SSGD, the transmission is partially pipelined with backpropogration: Synchronizing gradients of n th layer can be performed simultaneously with backpropagating ( n − 1) th layer.
However, it is not enough to cover communication since latency on
Environment Agent
Channel Pruning Policy gradient
[Table]
Q Network Actor Critic
Actions: sparsity
[Table]
Observations / Reward: accuracy,
[Table]
Hardware
[Table]
Xxx
[Table] [Table]
[Table] [Table]
[Table]
Channel Pruning Policy
[Table]
Network Actor Critic
Actions: sparsity
[Table]
Observations /
[Table]
[Table] [Table]
[Table] [Table]
Pruning
[Table]
Critic
[Table] [Table]
[Equation]
T
[Equation]
:Computation :Synchronization Barrier :Locally update :Send gradients to other
[Figure]
Temporally sparse reduces long-distance connection is usually larger ( ≥ 100ms) than propagating a layer ( ≈ 11ms).
To relax the limitation, we design delayed update to put off the the synchronization barrier by n steps as shown in Fig.
3b.
To ensure minimal loss on accuracy, we modify the gradient update formula with error compensations.
To understand, we start with vanilla SGD: Consider a worker with index j , the weights at timestamp n during distributed training are derived via
[Equation]
where ∇ w ( n − 1) is the averaged gradients calculated by gathering all local gradients ∇ w ( n − 1 ,j ) .
In
SSGD, the synchronization has to ﬁnish before ( n + 1) th forwarding.
In delayed update, we delay the barrier to ( n + t ) th iteration and because of the delay, ∇ w ( n − 1) is not received at timestamp n .
We use local gradients to update instead.
[Equation]
Meanwhile, stale gradients ∇ w ( n − t ) arrive.
At timestamp n − t , we performed the gradient step using local gradients ∇ w ( n − t,j ) , which should be the global averaged ∇ w ( n − t ) in SSGD.
We correct the mismatch caused by replacing ∇ w ( n − t,j ) with ∇ w ( n − t ) .
From Eq.
1, it adds a compensation term, which is the difference of two gradients.
[Equation]
Next, we consider momentum SGD.
With momentum m , the weight at timestamp n is updated by
[Equation]
We rewrite the cumulative form to general form
[Equation]
Then, we can derive the compensated momentum
[Equation]
and weights similarly
[Equation]
In original synchronous stochastic gradient descent (SSGD), every worker receives averaged gradients from n − 1 at timestamp n .
This requires collective operation ( e.g . Allreduce) to execute quickly; otherwise, it will slow down the training process.
In delayed update, the barrier of the collective operation is moved from timestamp n to n + t − 1 .
It means as long as the gradients are transmitted within t × T compute , the training is not blocked.
To put some real numbers, assume we are training
ResNet-50 on Nvidia Tesla V100, each step takes around 300ms.
If we delay the update by 20 steps, even with latency as large as 6 seconds, the communication still can be fully covered by computations.
We attach the proof of convergence in Appendix A.2.1 and show this is as fast as SSGD.
3. T EMPORALLY S PARSE U PDATE
Delayed update provides a good solution to tolerate latency.
However, there still remains two issues and network congestion is the ﬁrst one.
In the original SSGD, only one transmission is performed during one iteration.
But when using delayed update, there can be up to t transmissions at the same time.
Without a special hacking to low-level network driver, gradients sent earlier may arrive later, which is not what we want.
The second one is latency variance.
In real world network, especially for those long-distance connections, the latency varies in a large range.
Even though the average latency is small, some accidentally high latency can slow the system.
To address, we adapt temporally sparse update to reduce the communication frequency and design corresponding error compensation to guarantee the accuracy.
As demonstrated in Fig.
3c, temporally sparse update only synchronizes for every p steps.
For those steps where synchronization is not performed, each worker updates its model with local gradients.
For those steps where synchronization is performed, each worker calculates error compensations based averaged information.
As a result, the networking congestion is alleviated and the networking latency is amortized across multiple local update steps.
For vanilla SGD, like the delayed update, we can derive the compensation term for temporally sparse update from Eq. 1, which is the difference between accumulated gradients.
Note the compensation in temporally sparse update only applies when n (mod p ) ≡ 0 .
[Equation]
Bandwidth
Non-computing time
Synchronous SGD
[Equation]
Delayed Update (t)
[Equation]
Temporal Sparsity (p)
[Equation]
Delayed Update (t) +
Temporal Sparsity (p)
[Equation]
For SGD with momentum acceleration, based on accumulated gradients is not enough to compensate both ∇ w ( n,j ) and u ( n,j ) .
We need to transmit an extra term to correct the momentum:
[Equation] [Equation]
Let S ( i,j ) =
(cid:80) ik = n − p
m i − k ∇ w ( k,j ) .
Note S
( i +1 ,j ) =
mS ( i,j ) + ∇ w ( i +1 ,j ) and S ( n − p +1 ,j ) =
∇ w ( n − p,j ) , the formula can be written as:
[Equation]
Note temporally sparse update is different from gradient accumulation (forward and backward
N times, accumulate the gradients and optimizer steps only 1 time).
Gradient accumulation is equivalent to increasing the batch size and requires to adjust learning rate and the parameters of batch normalization to ensure smooth convergence.
Temporally sparsity updates model every iteration and compensate error every N steps.
It does not require any changes to the hyper-parameters.
We prove the convergence of temporally sparse update in Appendix A.2.2 and demonstrate that it enjoys the same convergence rate as SSGD.
3. D ELAYED AND TEMPORALLY SPARSE UPDATE
The delayed technique and temporally sparse update can be combined together.
The compensation formulas for vanilla SGD is
[Equation]
and for momentum SGD is
[Equation]
Original ring allreduce transfers 2 ||
w || packets (Thakur et al., 2005).
Though temporally sparse update doubles data transferred, the temporal sparsity t is always larger than 2.
As a consequence, combing two methods can further improve scalability under challenging networking ( Tab. 1).
D, T, D + T
Training curve | Scalability v.s. latency (same machine)
| Scalability v.s. machine (same latency)
[Table]
[Table] [Table]
[Figure]
[Table]
[Figure] [Figure]
[Table]
latency=1000ms-1 Nodes ideal
L= d=4, t=4 delay=8, temporal= delay=12, t=8 delay=20, t=12    8 8    16 16    31
[Figure]
2 with load balance balanced without load balance unbalanced 0.9 6.24667472793229 0.9 5.
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Table]
[Table] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
D, T, D + T
Training curve | Scalability v.s. latency (same machine)
| Scalability v.s. machine (same latency) 16x4 = 64 cards, batch size 32 latency(ms) speed
Original delay=4 delay=8 delay=12 delay=16 delay=20 0.
[Table]
[Figure]
[Table] [Table]
ideal
L= d=4, t=4 delay=8,
[Table] [Table]
[Figure]
ca
[Figure]
ilit y
[Figure]
servers
[Figure]
Table 2 with load balance balanced without load balance unbalanced 0.9 6.24667472793229 0.9 5.53935234827364 0.85 5.
[Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table]
A cc u r ac y L o ss -4.5%
[Figure] [Figure]
[Figure]
E XPERIMENT 4. E XPERIMENT SETTINGS
We evaluate our DTS on image classiﬁcation task.
We studies ResNet-50 (He et al., 2016) on
ImageNet (Deng et al., 2009).
ImageNet contains 1.2 million training images and 50,000 validation images in 1000 classes.
We train ImageNet model with momentum SGD using warmup strategy (Goyal et al., 2017) and cosine anneal learning rate decay (Loshchilov & Hutter, 2016) with 120 epochs.
Standard data augmentations random crop and ﬂip are adapted.
We set the initial learning rate 0.0125 for which grows linearly w.r.t number of GPUs.
For a fair comparison, the training settings are set the same among all experiments.
When performing ablation studies under different latency, we synthesis the network condition using netem ‡ .
When training across the world, the limited network is benchmarked using qperf § .
We implement DST in PyTorch framework (Paszke et al., 2017) and adapt Horovod (Sergeev & Del Balso, 2018) as the distributed training backend.
To make results reproducible, we conduct our experiment on 16 standard Amazon p.large EC2 instances with eight Nvidia Tesla V100 on each and will release the codebase when less anonymous.
4. R ESULTS OF D ELAYED U PDATE
Fig.
4a shows the performance and scalability for delayed update with different latency.
We start from delay steps 4 and gradually increase it.
The learning curve shows that with our proposed error compensation delayed update demonstrates robust convergence and comparable performance compared vanilla SSGD, even when delay interval is as large as 20.
[Table]
[Table]
D, T, D + T Training curve | Scalability v.s. latency (same machine)
| Scalability v.s. machine (same
[Table]
) 16x4 = 64 cards, batch size
[Table]
speed
[Table]
[Table]
=12 delay=16 delay=20 0.
0.7321 S ca l a b ilit y
Network latency (
[Figure]
Original delay= delay=8 delay= delay= delay=
[Figure]
S ca l a b il it y 0.
Network Latency (ms) 2000 3000 4000
Temporal=
Temporal=12 Temporal Sparsity=4 Temporal=
Temporal= 0 0.2 0.4 0.6 0.
latency=1000ms-1 Nodes ideal
L= d=4, t=4 delay=8, temporal= delay=12, t= delay=20, t=12     8     16     31     63      latency=1000ms-1Nodes
Nodes D=4, T=4 (200ms)
D=20, T=12 (200ms)
SSGD
(0ms) SSGD (200ms)
Ideal 0 0 0 0 1 0.8 0.8 0.8 1 2 1.5 1.5 1.6 0.016 4 3 3 3.1 0.030
Number of servers
[Figure]
Table 2 with load balance balanced without load balance unbalanced 0.9 6.24667472793229 0.9 5.53935234827364 0.85 5.23536863440588 0.85 4.56400742115028 0.8 4.41746119970927 0.8 3.73158046807281 0.7 3.15048025613661 0.7 2.71773154115264 0.6 2.42040902382458 0.6 2.15947329919532 0.5 1.95871011772735 0.5 1.79981186635543 0.4 1.64360026725208 0.4 1.54608167357505 0.3 1.41571093846344 0.3 1.36450079239303 0.2 1.28799022663243 0.2 1.28212446485078 0 1.00005807538185 0 1.
[Table]
[Figure]
A cc u r
[Figure]
[Table] [Table]
[Table] [Table]
Parametes Pruned Away 40% 50% 60% 70% 80% 90% 100%
L2 regularization w/o retrain
L1 regularization w/o retrain L1 regularization w/ retrain
L2 regularization w/ retrain L2 regularization w/ iterative prune and retrain
[Figure]
[Table]
In original SSGD and modern deep learning frameworks, though the transmission is partially pipelined with backpropogration: Synchronizing gradients of n th layer can be performed simultaneously with back-propagating ( n − 1) th layer, it is not enough to fully cover communication since latency on long distance connection is usually larger ( ≥ 100ms) than propagating a layer (11ms).
By delaying the synchronization, delayed update provides much more spare time for transmission and thus scales better.
As shown in Fig.
4a, the larger the delayed interval is, the better tolerance towards latency.
The scalability keeps stable until maximum tolerance is exceeded.
4. R ESULTS OF T EMPORALLY S PARSE U PDATE
The connection of real-world network has many bothering issues such as unstable latency and congestion on routers.
Temporally sparse is introduced to amortize these via reducing the synchronization frequency.
The results are shown in Fig.
4b( Left ): we start with temporal sparsity 4 and increase to 20.
With proposed compensation, the temporally sparse update has a negligible loss on accuracy.
Fig.
4( Right ) exhibits how temporally sparse training scales under different network latency.
The scalability degrades slower than because temporally sparse update amortizes the latency.
4. D ISTRIBUTED T RAINING ACROSS THE W ORLD
With two powerful techniques, we are close to our original target of distributed training across world.
To evaluate DTS, we deploy located at four different locations across the world: Tokyo(Japan),
London (U.K), Oregon (U.S.A) and Ohio(U.S.A).
The latency information is shown in Fig.2.
The real latency across the ring allreduce is about 479ms through Ethernet.
On V100 GPU, ResNettakes around 300ms to forward and backward.
Therefore delayed update with steps more than 4 is already enough to tolerate such a network.
In terms of latency variance and network congestion, we integrate temporally sparse update to alleviate which drastically reduces the bandwidth requirement by t times.
However, there is still a bottleneck on bandwidth for cross-continent connections ( e.g .
Tokyo to London).
Though theoretically bandwidth is achievable by switching to better internet service providers, AWS does not provide such an option.
As an economical alternative, we adapt deep gradient compression (Lin et al., 2017) to work through.
As shown in Tab.
2a, DTS has little effect on training accuracy.
Meanwhile, DTS demonstrates strong scalability (0.72) under high latency (200ms between workers) and this performance is close to what conventional algorithms achieved inside a data center (speedup ratio 0.78 and 1us latency).
C ONCLUSION
In this paper, we propose two novel delayed update and temporally sparse update methods to scale synchronous distributed training on servers located at different continents across the world.
We demonstrate that delayed synchronization and temporally sparse update are both accuracy preserving while tolerate poor network conditions.
We believe that our work will open future avenues for a wide range of decentralized learning applications.
A A PPENDIX A. C ONNECTION TO VARIANCE REDUCTION APPROACH
A lot of recent work has been devoted to the study of variance reduction techniques in convex optimization, including (Johnson & Zhang, 2013; Shalev-Shwartz & Zhang, 2013b; Xiao & Zhang, 2014; Schmidt et al., 2017; Defazio et al., 2014; Mairal, 2015). The vanilla SVRG algorithm has the update
[Equation] 14) where ˜ w is a snapshot reference point updated every constant iteration.
These methods are able to achieve linear convergence rates for smooth strongly-convex optimization problems, which significantly improve the sub-linear rate of vanilla SGD.
Extension to multiple setting are investigated, including acceleration (Shalev-Shwartz & Zhang, 2013a;
Lin et al., 2015; Allen-Zhu, 2017), streaming (Frostig et al., 2015; Lei & Jordan, 2016), synchronous distributed setting (Lee et al., 2015), data augmentation (Bietti & Mairal, 2017).
Extension of variance reduction approaches to general non-convex problems has also been studied in (Allen-Zhu & Hazan, 2016; Reddi et al., 2016).
Despite these theoretical progress, variance reduction technique does not appear to be helpful for training deep neural network (Defazio & Bottou, 2018).
Our proposed method could be viewed as a delayed variant of SVRG in the decentralized setting.
The update (15) in our method mimic the update of vanilla SVRG (14), with the difference that the average gradient is obtained across machine base on local parameters.
This variance reduction technique empirically stabilized the asynchronous gradients and handle the staleness, which is able to achieve state-of-the-art performance in modern deep learning setting.
Moreover, we obtain close linear speed up with respect to the number of machine, which is the essential goal of distributed optimization.
A. T HEORETICAL J USTIFICATION
Our proof closely follows the analysis in (Yu et al., 2019).
We consider the following non-convex optimization problem
[Equation]
where J denotes the number of workers.
Moreover, each f j = E ζ j [ F j ( w, ζ j )] .
In the empirical minimization setting, ζ j corresponds to a mini-batch sampling strategy and f j could again be expressed as a ﬁnite sum f j = 1 K j (cid:80) f k j .
Throughout this paper, we assume the optimization satisﬁes the following assumptions.
Assumption 1 ( L -smooth) .
Each function f j ( x ) is L -smooth, i.e . for all x
, y ∈ R d and i ∈ [1 , J ] ,
[Equation]
Assumption 2 (Bounded gradients and variances) .
There exists a positive constant G and σ such that
[Equation]
We prove that our algorithms have the convergence rate
[Equation]
where J is the number of workers, N is iterations and c is staleness in our algorithm ( t for delayed steps and d for temporal sparsity).
With proper settings, our algorithms are as fast as original SSGD
O (1 / √ JN ) .
The detailed proof follows below.
A.2. P ROOF OF THE CONVERGENCE OF D ELAYED U PDATE A LGORITHM Delayed Update Algorithm
Initialize w 0 and initialize each worker with w 0 ,j = w 0 for j ∈ [1 , J ] .
For iteration n = 0 , 1 , · · · On each local worker j , sample a stochastic gradient ∇ w ( n − 1 ,j ) and update: ◦
If n < t , update with standard SGD rule:
[Equation]
◦ Else:
[Equation]
where
[Equation]
Output ∇ w ( n ) =
1 J (cid:80)
[Equation]
Notably [Equation] the delayed update formula can rewritten as
[Equation]
where the ﬁrst quantity is commonly shared among all the workers and the second term corresponds to local updates
[Equation]
Lemma A.1.
Let ( w ( i,j ) ) be the sequence generated on the j -th worker according to Delayed Update
Algorithm, then for any j 0 ∈ [1 , J ]
[Equation]
Proof.
From the delayed update formula Delayed, we have,
[Equation]
2 (cid:3) =
γ 2 E   (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) 1 J
J (cid:88)
j = n − 1 (cid:88) i =
n − t + ∇ w
i,j ) − n − 1 (cid:88) i = n − t + ∇ w
( i,j 0 ) (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) 2 
 ≤ 2 γ 2 E   (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) 1 J
J (cid:88)
j = n − 1 (cid:88) i =
n − t + ∇ w
i,j ) (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) 2 
 + E (cid:34) (cid:107) n − 1 (cid:88) i = n − t + ∇ w
i,j 0 ) (cid:107)(cid:35) ≤ 2 γ 2 t
J
J (cid:88)
j = n − 1 (cid:88) i = n − t +
E (cid:104)(cid:13) (cid:13) ∇ w ( i,j ) (cid:13) (cid:13) 2 (cid:105) + 2 γ 2 t n − 1 (cid:88) i
= n − t +
E (cid:2) (cid:107)∇ w ( i,j 0 ) (cid:107) 2 (cid:3) ≤ 4 γ 2 t 2 G 2 .
Theorem A.2.
Consider the sequence generated by the Delayed Algorithm, then
[Equation]
Proof.
By smoothness of f , we have
[Equation]
Note that
[Equation]
We
[Equation]
Moreover,
[Equation]
Applying the lemma
[Equation]
we have
[Equation]
j
[Equation] [Equation]
≤ 4 γ
[Equation]
L 2 t
[Equation]
Thus, combining the above inequalities together with γ ≤ 1 L yields
[Equation]
Rearrange the terms yields
[Equation]
Telescoping from n = 1 , ..N yields
[Equation]
Corollary A.2.1.
When the number of iteration N surpass the number of machines J
[Equation]
let the stepsize γ = √ J L √ N , yields
[Equation]
When delayed steps is in a reasonable range ( t <
O ( N 14 J − 34 ) ), the ﬁrst term dominates and delayed update demonstrates the same convergence speed as original SSGD O (1 / √ JN ) .
A.2. P ROOF OF THE CONVERGENCE OF T EMPORALLY S PARSE U PDATE A LGORITHM Temporally Sparse Update Algorithm
Initialize w 0 and initialize each worker with w 0 ,j = w 0 for j ∈ [1 , J ] .
For iteration n = 0 , 1 , · · · On each local worker j , sample a stochastic gradient ∇ w ( n − 1 ,j ) and update: ◦ If n (mod d ) (cid:54) =
update with standard (local) SGD rule:
[Equation]
◦ Else:
[Equation]
where
[Equation]
Output ∇ w ( n ) =
1 J (cid:80)
Jj =1 w ( n,j )
Notably, the temporally sparse update formula can rewritten as
[Equation]
(TS
[Equation]
where the ﬁrst quantity is commonly shared among all the workers and the second term corresponds to local updates.
Lemma A.3.
Let ( w ( i,j ) ) be the sequence generated on the j -th worker according to Temporally
Sparse Update Algorithm, then for any j 0 ∈ [1 , J ]
[Equation]
Proof.
From the update of (TS), we have
[Equation] [Equation]
Theorem A.4.
Consider the sequence generated by the Temporally Sparse Update Algorithm, then
[Equation]
Proof.
Based on Lemma A.3, the proof follows the exact same schema as the Delayed Update
Algorithm, which we omit here.
Corollary A.4.1.
When the number of iteration N surpass the number of machines J , let the stepsize γ = √ J L √ N , yields
[Equation]
When temporal sparsity is in a reasonable range ( d < O ( N 14 J − 34 ) ), the ﬁrst term dominates and temporally sparse update demonstrates the same convergence speed as original SSGD O (1 / √ JN ) .