A C OMMUNICATION -E FFICIENT D ISTRIBUTED G RADIENT C LIPPING A LGORITHM FOR T RAINING D EEP N EURAL N ETWORKS
Anonymous authors
Paper under double-blind review
A BSTRACT
In distributed training of deep neural networks or Federated Learning (FL), people usually run Stochastic Gradient Descent (SGD) or its variants on each machine and communicate with other machines periodically.
However, SGD might converge slowly in training some deep neural networks (e.g., RNN, LSTM) because of the exploding gradient issue.
Gradient clipping is usually employed to address this issue in the single machine setting, but exploring this technique in the
FL setting is still in its infancy: it remains mysterious whether the gradient clipping scheme can take advantage of multiple machines to enjoy parallel speedup in the FL setting.
The main technical difficulty lies at dealing with nonconvex loss function, non-Lipschitz continuous gradient, and skipping communication rounds simultaneously.
In this paper, we explore a relaxed-smoothness assumption of the loss landscape which LSTM was shown to satisfy in previous works, and design a communication-efficient gradient clipping algorithm.
This algorithm can be run on multiple machines, where each machine employs a gradient clipping scheme and communicate with other machines after multiple steps of gradient-based updates.
Our algorithm is proved to have O (cid:0)
Nϵ 4 (cid:1) iteration complexity for finding an ϵ -stationary point, where N is the number of machines.
This indicates that our algorithm enjoys linear speedup.
Our experiments on several benchmark datasets and various scenarios demonstrate that our algorithm indeed exhibits fast convergence speed in practice and validate our theory.
I NTRODUCTION
Deep learning has achieved tremendous successes in many domains including computer vision (Krizhevsky et al., 2012;
He et al., 2016) and natural language processing (Devlin et al., 2018), game (Silver et al., 2016).
To obtain good empirical performance, people usually need to train large models on a huge amount of data, and it is usually very computationally expensive.
To speedup the training process, distributed training becomes indispensable (Dean et al., 2012).
For example, Goyal et al.
(2017) trained a ResNet-50 on ImageNet dataset by distributed SGD with minibatch sizeon 256 GPUs in only one hour, which not only matches the small minibatch accuracy but also enjoys parallel speedup, and hence improves the running time.
Recently, there is an increasing interest in an variant of distributed learning, namely Federated Learning (FL) (McMahan et al., 2017), which focuses on the cases where the training data is non-i.i.d.
across devices and only limited communication is allowed.
McMahan et al.
(2017) proposed an algorithm named Federated Averaging, which runs multiple steps of SGD on each clients before communicating with other clients.
Despite the empirical success of distributed SGD and its variants (e.g., Federated Averaging) in deep learning, they may not exhibit good performance when training some neural networks (e.g.,
Recurrent Neural Networks, LSTMs), due to the exploding gradient problem (Pascanu et al., 2012; 2013).
To address this issue, Pascanu et al.
(2013) proposed to use the gradient clipping strategy, and it has become a standard technique when training language models (Gehring et al., 2017; Peters et al., 2018; Merity et al., 2018).
There are some recent works trying to theoretically explain gradient clipping from nonconvex optimization’s perspective (Zhang et al., 2019; 2020).
These works are built upon an important observation made in (Zhang et al., 2019): for certain neural networks such as LSTM, the gradient does not vary uniformly over the loss landscape (i.e., the gradient is not Lipschitz continuous with a uniform constant), and the gradient Lipschitz constant can scale linearly with respect to the gradient norm.
This is referred to as the relaxed smoothness condition (i.e., ( L 0 , L 1 ) -smoothness defined in Definition 2), which generalizes but strictly relaxes the usual smoothness condition (i.e., L -smoothness defined in Definition 1).
Under the relaxed smoothness
[Table] [Table]
condition, Zhang et al.
(2019; 2020) proved that gradient clipping enjoys polynomial-time iteration complexity for finding the first-order stationary point in the single machine setting
[Table]
and it can be arbitrarily faster than fix-step gradient descent
[Table]
In practice
[Table]
both distributed learning (or FL) and gradient clipping are important techniques to accelerate neural network training.
However, the theoretical analysis of gradient clipping is only restricted to the single machine setting (Zhang et al., 2019; 2020).
Hence it naturally motivates us to consider the following question:
Is it possible that the gradient clipping scheme can take advantage of multiple machines to enjoy parallel speedup in training deep neural networks, with data heterogeneity
across machines and limited communication
In this paper, we give an affirmative answer to the above question.
Built upon the relaxed smoothness condition as in (Zhang et al., 2019; 2020), we design a communication-efficient distributed gradient clipping algorithm.
The key characteristics of our algorithm are: (i) unlike naive parallel gradient clipping algorithm which requires averaging model weights and gradients from all machines for every iteration, our algorithm only aggregates weights with other machines after certain number of local updates on each machine; (ii) our algorithm clips the gradient according to the norm of the local gradient on each machine, instead of the norm of the averaged gradients across machines as in the naive parallel version.
These key features make our algorithm amenable to the FL setting and it is nontrivial to establish desired theoretical guarantees (e.g., linear speedup, reduced communication complexity).
The main difficulty in the analysis lies at dealing with nonconvex objective function, non-Lipschitz continuous gradient, and skipping communication rounds simultaneously.
Our main contribution is summarized as the following:
• We design a novel communication-efficient distributed stochastic local gradient clipping algorithm, namely CELGC, for solving a nonconvex optimization problem under the relaxed smoothness condition.
The algorithm only needs to clip the gradient according to the local gradient’s magnitude and globally averages the weights on all machines periodically.
To the best of our knowledge, this is the first work proposing communication-efficient distributed stochastic gradient clipping algorithms under the relaxed smoothness condition.
• Under the relaxed smoothness condition, we prove iteration and communication complexity results of our algorithm for finding an ϵ -stationary point.
First, comparing with (Zhang et al., 2020), we prove that our algorithm enjoys linear speedup, which means that the iteration complexity of our algorithm is reduced by a factor of N (the number of machines).
Second, comparing with naive parallel verion of the algorithm of (Zhang et al., 2020),
we
bounded gradient assumption, instead they require L -Lipschitz gradient and bounded variance σ 2 .
Under their assumption, their complexity result is O (cid:0) ∆ Lϵ − 2 + ∆
Lσ 2 ϵ − 4 (cid:1) .
Naive Parallel of (Zhang et al. 2020) is different from our algorithm (CELGC) with I = 1 in that the naive version requires averaging gradients across all machines to update the model while CELGC only updates the model using local gradients computed in that machine.
This also means that in each iteration, naive version clips the gradient based on the globally averaged gradient while ours only bases on the local gradient.
prove that our algorithm enjoys better communication complexity.
Specifically, our algorithm’s communication complexity is smaller than naive parallel clipping algorithm if the number of machines is not too large (i.e., N ≤ O (1 /ϵ ) ).
The detailed comparison over existing algorithms under the same relaxed smoothness condition is described in Table 1.
Please refer to (Koloskova et al., 2020) for local SGD complexity results for L -smooth functions.
• We empirically verify our theoretical results by conducting experiments on different neural network architectures on benchmark datasets and on various scenarios including small to large batch-sizes, homogeneous and heterogeneous data distributions, and partial participation of machines.
The experimental results demonstrate that our proposed algorithm indeed exhibit speedup in practice.
R ELATED W ORK
Gradient Clipping/Normalization Algorithms
In deep learning literature, gradient clipping (normalization) technique was initially proposed by (Pascanu et al., 2013) to address the issue of exploding gradient problem in (Pascanu et al., 2012), and it has become a standard technique when training language models (Gehring et al., 2017; Peters et al., 2018; Merity et al., 2018).
Menon et al.
(2019) showed that gradient clipping is robust and can mitigate label noise.
Recently gradient normalization techniques (You et al., 2017; 2019) were applied to train deep neural networks on the very large batch setting.
For example, You et al.
(2017) designed LARS algorithm to train a
ResNet50 on ImageNet with batch size 32k, which utilized different learning rate according to the norm of the weights and the norm of the gradient.
In optimization literature, gradient clipping (normalization) was used in early days in the field of convex optimization (Ermoliev, 1988; Alber et al., 1998; Shor, 2012).
Nesterov (1984) and Hazan et al.
considered normalized gradient descent for quasi-convex functions in deterministic and stochastic cases respectively.
Gorbunov et al.
(2020) designed an accelerated gradient clipping method to solve convex optimization problem with heavy-tailed noise in stochastic gradients.
Mai & Johansson (2021) established the stability and convergence of stochastic gradient clipping algorithms for convex and weakly convex functions.
In nonconvex optimization, Levy (2016) showed that normalized gradient descent can escape from saddle points.
Cutkosky & Mehta (2020) showed that adding a momentum provably improves the normalized SGD in nonconvex optimization.
Zhang et al.
and Zhang et al.
(2020) analyzed the gradient clipping for nonconvex optimization under the relaxed smoothness condition rather than the traditional L -smoothness condition in nonconvex optimization (Ghadimi & Lan, 2013).
However, all of them only consider the algorithm in the single machine setting or the naive parallel setting, and none of them can apply to FL setting where data on different nodes is heterogeneous and only limited communication is allowed.
Communication-Efficient Algorithms in Distributed and Federated Learning
In large-scale machine learning, people usually train their model using first-order methods on multiple machines and these machines communicate and aggregate their model parameters periodically.
When the function is convex, there is scheme named one-shot averaging (Zinkevich et al., 2010; McDonald et al., 2010; Zhang et al., 2013; Shamir & Srebro, 2014), in which every machine runs an stochastic approximation algorithm and averages the model weights across machines only at the very last iteration.
One-shot averaging scheme is communication-efficient and enjoys statistical convergence with one pass of the data (Zhang et al., 2013; Shamir & Srebro, 2014; Jain et al., 2017; Koloskova et al., 2019), but the training error may not converge in practice.
McMahan et al.
(2017) considered the Federated Learning setting where the data is decentralized and might be non-i.i.d.
across devices and communication is expensive.
McMahan et al.
(2017) designed the very first algorithm for FL (a.k.a., FedAvg), which is communication-efficient since every node communicates with other nodes infrequently.
Stich (2018) considered a concrete case of FedAvg, namely local SGD, which runs SGD independently in parallel on different works and averages the model parameters only once in a while.
Stich (2018) also showed that local SGD enjoys linear speedup for stronglyconvex objective function.
There are also some works analyzing local SGD and its variants on convex (Dieuleveut & Patel, 2019; Khaled et al., 2020; Karimireddy et al., 2020;
Woodworth et al., 2020a;b; Gorbunov et al., 2021; Yuan et al., 2021) and nonconvex smooth functions (Zhou & Cong, 2017; Yu et al., 2019a;b; Jiang & Agrawal, 2018; Wang & Joshi, 2018; Lin et al., 2018;
Basu et al., 2019; Haddadpour et al., 2019; Karimireddy et al., 2020).
Recently, Woodworth et al.
(2020a;b) analyzed advantages and drawbacks of local SGD compared with minibatch SGD for convex objectives.
Woodworth et al.
(2021) proved hardness results for distributed stochastic convex optimization.
Reddi et al.
(2021) introduced a general framework of federated optimization and designed several federated versions of adaptive optimizers.
Zhang et al.
(2021) considered to employ gradient clipping to optimize L -smooth functions and achieve differential privacy.
Due to a vast amount of literature of FL and limited space, we refer readers to (Kairouz et al., 2019) and references therein.
However, all of these works either assume the objective function is convex or L -smooth.
To the best of our knowledge, our algorithm is the first communication-efficient algorithm which does not rely on these assumptions but still enjoys linear speedup.
P RELIMINARIES , N OTATIONS AND P ROBLEM S ETUP
Preliminaries and Notations Denote ∥ · ∥ by the Euclidean norm.
We denote f : R d → R as the overall loss function, and f i : R d → R as the loss function on i -th machine, where i = 1 , . . .
, N .
Denote ∇ h ( x ) as the gradient of h evaluated at the point x , and denote ∇ h ( x ; ξ ) as the stochastic gradient of h calculated based on sample ξ .
Definition 1 ( L -smoothness) .
A function h is L -smooth if ∥∇ h ( x ) − ∇ h ( y ) ∥
≤ L ∥ x − y ∥ for all x , y ∈ R d .
Definition 2 ( ( L 0 , L 1 ) -smoothness) .
A second order differentiable function h is ( L 0 , L 1 ) -smooth if ∥∇ 2 h ( x ) ∥ ≤ L 0 + L 1 ∥∇ h ( x ) ∥ for any x ∈ R d .
Definition 3 ( ϵ -stationary point) .
x ∈ R
d is an ϵ -stationary point of the function h if ∥∇ h ( x ) ∥ ≤ ϵ .
Remark:
From definitions, we know that the ( L 0 , L 1 ) -smoothness is strictly weaker than Lsmoothness.
To see this, first, we know that L -smooth functions is ( L 0 , L 1 ) -smooth with L
= L and L 1 = 0 .
However the reverse is not true.
For example, consider the function h
( x ) = x 4 , we know that the gradient is not Lipschitz continuous and hence is not L -smooth, but | h ′′ ( x ) | = 12 x 2 ≤ 12 + 3 × 4 | x | 3 = 12 + 3 | h
( x ) | , so h ( x ) =
x 4 is (12 , 3) -smooth.
Zhang et al.
(2019) empirically showed that the ( L 0 , L 1 ) -smoothness holds for the AWD-LSTM (Merity et al., 2018).
In nonconvex optimization literature (Ghadimi & Lan, 2013; Zhang et al., 2020), the goal is to find an ϵ -stationary point since it is NP-hard to find a global optimal solution for a general nonconvex function.
Problem Setup
In this paper, we consider the following optimization problem:
[Equation]
where N is the number of nodes and each f i ( x ) :
=
E ξ i ∼D i [ F i ( x ; ξ i )] is a nonconvex function where
D i can be possibly different for different i .
This formulation has broad applications in distributed deep learning and FL.
For example, in FL setting, f i stands for the loss function on i -th machine,
D i represents the data distribution on i -th machine, and N machines want to jointly optimize the objective function f .
We make the following assumptions throughout the paper.
Assumption 1.
(i)
Each function f i ( x ) is ( L 0 , L 1 ) -smooth, i.e., ∥∇ 2 f i ( x ) ∥ ≤ L 0 + L 1 ∥∇ f i ( x ) ∥ , for ∀ x ∈ R d and i = 1 , . . .
, N .
(ii) There exists ∆ > 0 such that f ( x 0 ) − f ∗ ≤ ∆ , where f ∗ is the global optimal value of f .
(iii)
For all x ∈ R d , E ξ i ∼D i [ ∇ F i ( x ; ξ i )]
= ∇ f i ( x ) , and ∥∇ F i ( x ; ξ ) − ∇ f i ( x ) ∥ ≤ σ almost surely.
(iv) 1 N (cid:80) Ni =1 ∥∇ f i ( x ) − ∇ f ( x ) ∥ ≤ κ .
Remark:
The Assumption 1 (i) means that that the loss function defined on each machine satisfies the relaxed-smoothness condition, and it holds when we want to train a language model using
LSTMs.
Assumption 1 (ii) and (iii) are standard assumptions in nonconvex optimization (Ghadimi & Lan, 2013; Zhang et al., 2019).
Note that it is usually assumed that the stochastic gradient is unbiased and has bounded variance (Ghadimi & Lan, 2013), but we follow (Zhang et al., 2019) to assume we have unbiased stochastic gradient with almost surely bounded deviation σ .
This is an
1: for t = 0 , . . . , T do 2: Each node i samples its stochastic gradient ∇ F i ( x it ; ξ it ) , where ξ it ∼ D i . Each node i updates it local solution in parallel :
[Equation]
4: if t is a multiple of I then 5:
Each worker resets the local solution as the averaged solution across nodes:
[Equation] [Equation]
) 6: end if 7: end for stronger assumption than the bounded variance, but it is a normal assumption when encountering relaxed smoothness.
Assumption 1 (iv) quantifies the averaged heterogeneity across nodes, which is frequently used in the FL literature (e.g., Yu et al.
(2019a)).
A LGORITHM AND T HEORETICAL A NALYSIS 4. M AIN DIFFICULTY AND THE ALGORITHM DESIGN
We briefly present the main difficulty in extending the single machine setting (Zhang et al., 2020) to the FL setting.
In (Zhang et al., 2020), they split the contribution of decreasing objective value by considering two cases: clipping large gradients and keeping small gradients.
If communication is allowed at every iteration, then we can aggregate gradients on each machine and determine whether we should clip or keep the averaged gradient or not.
However, in FL setting, communicating with other machines at every iteration is not allowed.
This would lead to the following difficulties: (i) the averaged gradient may not be available to the algorithm if communication is limited, so it is hard to determine whether clipping operation should be performed or not; (ii) the model weight on every machine may not be the same when communication does not happen at the current iteration; (iii) the loss function is not L -smooth, so the usual local SGD analysis for L -smooth functions cannot be applied in this case.
To address this issue, we design a new algorithm, namely Communication-Efficient Local Gradient Clipping (CELGC), which is presented in Algorithm 1.
The algorithm calculates a stochastic gradient and then performs multiple local gradient clipping steps on each machine in parallel, and aggregates model parameters on all machines after every I steps of local updates.
Note that the naive version of the parallel gradient clipping algorithm in (Zhang et al., 2020) needs to aggregates model parameters and gradients across all machines at every iteration, and perform one step of gradient clipping operation based on the aggregated gradient.
Conceptually speaking, our algorithm is expected to have better performance.
The reason is that our algorithm is able to skip communication rounds, and does not need to transmit gradient information across machines (note that it only averages the weights).
The remaining issue is that the non-asymptotic convergence guarantees are not established yet.
In other words, we aim to establish iteration and communication complexity for
Algorithm 1 for finding an ϵ -stationary point.
We present our main theoretical results as below.
4. M AIN R ESULTS
Theorem 1.
Suppose Assumption 1 holds and σ + κ ≥ 1 .
Take ϵ ≤ min( AL 0 BL 1 , 0 . 1) be a small enough constant and N ≤ min( 1 ϵ , 14 AL 0 5 BL 1 ϵ ) .
In Algorithm 1, choose I ≤ 1 2 Nϵ , γ ≤
[Equation] } and the fixed ratio γη = 5( σ + κ ) , where A ≥ 1 and B ≥ 1 are constants which will be specified in the proof, and run Algorithm 1 for T
= O (cid:0) ∆ L
Nϵ 4 (cid:1) iterations.
Then we have
[Equation]
Remark: We have some implications of Theorem 1.
When the number of machines is not large (i.e., N ≤ O (1 /ϵ ) ) and the number of skipped communications is not large (i.e., I ≤ O (1 /ϵN ) ), then with proper setting of learning rate, we have following observations.
First, our algorithm enjoys linear speedup, since the number of iterations we need to find an ϵ -stationary point is divided by the number of machines N when comparing the single machine algorithm in (Zhang et al., 2020).
Second, our algorithm is communication-efficient, since the communication complexity is T/I
= O (cid:0) ∆ L 0 ( σ + κ ) 2 ϵ − 3 (cid:1) , which provably improves the naive parallel gradient clipping algorithm of (Zhang et al., 2020) with O (∆ L 0 σ 2 / ( Nϵ 4 ))
communication complexity when
N ≤ O (1 /ϵ ) .
Another important fact is that both iteration complexity and communication complexity only depend on L 0 , independent of L 1 and the gradient upper bound M .
This indicates that our algorithm does not suffer from slow convergence even if these quantities are large.
In addition, local gradient clipping is a good mechanism to alleviate the bad effects brought by a rapidly changing loss landscape (e.g., some language models such as LSTM).
4. S KETCH OF THE P ROOF OF T HEOREM
In this section, we present the sketch of our proof of Theorem 1 and the detailed proof can be found in Appendix B.
The key idea in our proof is to establish the descent property of the sequence { f (¯ x t ) }
Tt =0 in the FL setting under the relaxed smoothness condition, where ¯ x t = 1 N (cid:80) ti =1 x it is the averaged weight across all machines at t -th iteration.
The main challenge is that the descent property of ( L 0 , L 1 ) -smooth function in the FL setting does not naturally hold, which is in sharp contrast to the usual local SGD proof for L -smooth functions.
To address this challenge, we need to carefully study whether the algorithm is able to decrease the objective function in different situations.
Our main technical innovations in the proof are listed as the following.
First, we monitor the algorithm’s progress in decreasing the objective value according to some novel measures.
The measures we use are the magnitude of the gradient evaluated at the averaged weight and the magnitude of local gradients evaluated at the individual weights on every machine.
Please note that our algorithm does not have access to the gradient evaluated at the averaged weight, but it can be served as a proxy in our proof even if we do not have knowledge about it.
To this end, we introduce Lemma 2, whose goal is to carefully inspect how much progress the algorithm makes, according to the magnitude of local gradients calculated on each machine.
The reason is that the local gradient’s magnitude is an indicator of whether the clipping operation happens or not.
For each fixed iteration t , we define J ( t ) =
{ i ∈ { 1 : N } : ∥∇ F i ( x it , ξ it )
∥ ≥ γ/η } and ¯ J ( t ) = { 1 : N }\ J ( t ) .
Briefly speaking, J ( t ) contains all machines that perform clipping operation at iteration t and ¯ J ( t ) is the set of machines that do not perform clip operation at iteration t .
In Lemma 2, we perform one-step analysis and consider all machines with different clipping behaviors at the iteration t .
By considering all cases together and taking the telescoping sum over t = 0 , . . .
, T , we can get an upper bound of the gradient in the ergodic sense.
Second, Zhang et al.
(2020) inspect their algorithm’s progress by considering the magnitude of gradient at different iterations, so they treat every iteration differently.
However, this approach does not work in FL setting since one cannot get access to the averaged gradient across machines at every iteration.
Instead, we treat every iteration of the algorithm as the same but consider the progress made by each machine.
Third, by properly choosing hyperparameters ( η, γ, I ) and using an amortized analysis, we prove that our algorithm can decrease the objective value by an sufficient amount, and the sufficient decrease is mainly due to the case where the gradient is not too large (i.e., clipping operations do not happen).
This important insight allows us to better characterize the training dynamics without worrying too much about the case that where gradient is large(i.e.
, the clipping operation is performed).
With the idea mentioned above, now we present how to proceed with the proof in detail.
Lemma 1 characterizes the ℓ 2 error between averaged weight and individual weights at t -th iteration.
Intuitively, the ℓ 2 error scales linearly in terms of the length of node synchronization interval I .
Lemma 1.
Under Assumption 1, for any i and t , Algorithm 1 ensures ∥ ¯ x t − x it ∥ ≤ 2 γI holds almost surely.
Lemma 4 and Lemma 5 (included in Appendix A) are some properties of ( L 0 , L 1 )
-smooth functions and we need to use them frequently in our paper.
To make sure they work, we need 2 γI ≤ c/L 1 for some c > 0 .
This inequality follows from the choice of parameters in Theorem 1 and details will be shown in Appendix B.
We denote A = 1 + e c − e c − 1 c and B = e c − 1 c .
Let J ( t ) be the index set of i
such that ∥∇ F i ( x it , ξ it )
∥ ≥ γη at fixed iteration t , i.e., J ( t ) =
{ i ∈ [1 , . . .
, N ] | ∥∇ F i ( x it , ξ it )
∥ ≥ γη } .
Lemma 2 characterizes how much progress we can get in one iteration of Algorithm 1, and the progress is decomposed into contributions from every machine (note that J ( t ) ∪ ¯ J ( t ) = { 1 , . . .
, N } for every t ).
Lemma 2.
Let J ( t ) be the set defined as above.
If 2 γI ≤ c/L 1 for some c > 0 ,If AL 0 η ≤ 1 / 2 , then we have
[Equation]
where A = 1 + e c − e c − 1 c and B = e c − 1 c .
Lemma 3 quantifies an upper bound of the averaged ℓ 2 error between local gradient evaluated at the local weight and the gradient evaluated at the averaged weight.
The upper bound contains the noise term in stochastic gradient σ , the data heterogeneity κ , and another error term which scales linearly with the length of node synchronization interval I .
Lemma 3.
Suppose Assumption 1 holds.
If 2 γI ≤ c/L 1 for some c > 0 , then we obtain
[Equation]
almost surely , where A = 1 + e c − e c − 1 c and B = e c − 1 c .
Putting all together Suppose our algorithm runs T iterations.
Taking summation on both sides of Lemma 2 over all t = 0 , . .
, T − 1 , we are able to get an upper bound of (cid:80) T −t =
E [ f (¯ x t +1 ) − f (¯ x t )]
=
E [ f (¯ x T ) − f
(¯ x 0 )] .
Note that E [ f (¯ x T ) − f
(¯ x 0 )] ≥ − ∆ due to
Assumption 1, so we are able to get a upper bound of gradient norm.
For details, please refer to the proof of Theorem 1 in Appendix B.
EXPERIMENTS
We conduct extensive experiments to validate the merits of our algorithm in realistic settings and find the distributed clipping algorithm indeed consistently exhibits substantial speedup compared with the baseline, which is the naive parallel version of the algorithm in (Zhang et al., 2020).
We want to re-emphasize that the major difference is that the baseline algorithm needs to average the model weights and local gradients at every iteration while ours only requires averaging the model weights after I iterations and does not need to average the gradients at all.
This immediately suggests that our algorithm will gain substantial speedup in terms of the wall clock time, which is also supported by our empirical experiments in this section.
We conduct each experiment in two nodes with 4 Nvidia-V100 GPUs for each node.
In our experiments, one "machine" corresponds to one GPU, and we use the word "GPU" and "machine" in this section interchangeably.
We compared our algorithm with the baseline across three deep learning benchmarks: CIFAR-10 image classification with ResNet, Penn Treebank language modeling with LSTM, and Wikitext-2 language modeling with LSTM.
All algorithms and the the training framework are implemented in Pytorch 1.4.
Due to limited computational resources, we choose the same hyperparameters (learning rates, clipping thresholds) according to the best-tuned baselines unless otherwise specified.
For more results including small and large batch-sizes, heterogeneous data distribution, and partial participation of machines, we kindly refer readers to the Appendix C and E.
5. E FFECTS OF SKIPPING COMMUNICATION
We focus on one feature of our algorithm: skipping communication.
Theorem 1 says that our algorithm enjoys reduced communication complexity since every node only communicates with
[Figure] [Figure]
[Figure]
periodically communication skipping affects the convergence of Algorithm we run it with I
CIFAR-10 classification with ResNet-56.
We train the standard 56-layer ResNet (He et al., 2016) architecture on CIFAR-10.
We use SGD with clipping as the baseline algorithm with a stagewise decaying learning rate schedule, following the widely adopted fashion on training the ResNet architecture.
Specifically, we use the initial learning rate η = 0 .
3 , the clipping threshold γ = 1 .
and decrease the learning rate by a factor of 10 at epoch 80 and 120 .
The local batch size at each GPU is 64 .
These parameter settings follow that of Yu et al. (2019a).
The results are illustrated in Figure 1.
Figure 1a shows the convergence of training loss and test accuracy v.s. the number of epochs that are jointly accessed by all GPUs.
This means that, if the x-axis value is 8, then each GPU runs 1 epoch of training data.
The same convention applied to all other figures for multiple GPU training in this paper.
Figure 1b verifies our algorithm’s advantage of skipping communication by plotting the convergence of training loss and test accuracy v s. the wall clock time.
Overall, we can clearly see that our algorithm matches the baseline epoch-wise but greatly speeds up wall-clock-wise.
Language modeling with LSTM on Penn Treebank.
We adopt the 3-layer AWD-LSTM (Merity et al., 2018) to do language modeling on Penn Treebank (PTB) dataset (Marcus et al.,
1993)(word
[Figure]
level).
We use SGD with clipping as the baseline algorithm with the initial learning rate η = 30 and the clipping threshold γ = 7 . 5 .
The local batch size at each GPU is 3.
These parameter settings follow that of Zhang et al.
We report the results in Figure 2.
Though we slightly fall behind the baseline epoch-wise in terms of validation perplexity, we do better in training, and gains substantial speedup (2x faster for I = 16 ) wall-clock-wise.
Language modeling with LSTM on Wikitext-2.
We again adopt the 3-layer AWD-LSTM (Merity et al., 2018) to do language modeling on Wikitext-2 dataset (Marcus et al., 1993)(word level).
We use SGD with clipping as the baseline algorithm with the initial learning rate η = 30 and the clipping threshold γ = 7 . 5 .
The local batch size at each GPU is 10.
These parameter settings follow that of Merity et al.
We report the results in Figure 3.
We can match the baseline in both training loss and validation perplexity epoch wise, but we again obtain large speedup (2.5x faster for I = 16 ) wall-clockwise.
This, together with the above two experiments, clearly show our algorithm’s effectiveness in speeding up the training in distributed settings.
Another observation is that Algorithm 1 can allow relatively large I without hurting the convergence behavior.
5. V ERIFYING P ARALLEL S PEEDUP
Figure 4 shows the training loss and test accuracy v.s. the number of iterations.
In the distributed setting, one iteration means running one step of Algorithm 1 on all machine; while in the single machine setting, one iteration means running one step of SGD with clipping.
In our experiment, we use minibatch size 64 on every GPU in distributed setting to run Algorithm 1, while we also use 64 minibatch size on the single GPU to run SGD with clipping.
In Figure 4, we can clearly find that even with I > 1 , our algorithm still enjoys parallel speedup, since our algorithm requires less number of iterations to converge to the same targets (e.g., training loss, test accuracy).
This observation is consistent with our iteration complexity results in Theorem 1.
C LIPPING OPERATION HAPPENS FREQUENTLY
Figure 5 reports the proportion of iterations in each epoch that clipping is triggered.
We observe that for our algorithm, clipping happens more frequently than the baseline, especially for NLP tasks.
We conjecture that this is because we only used local gradients in each GPU to do the clipping without averaging them across all machines like the baseline did.
This leads to more stochasticity of the norm of the gradient in our algorithm than the baseline, and thus causes more clippings to happen.
This observation highlights the importance of studying clipping algorithms in the distributed setting.
Another interesting observation is that clipping happens much more frequently when training language models than image classification models.
Hence this algorithm is presumably more effective in training deep models in NLP tasks.
C ONCLUSION
In this paper, we design a communication-efficient distributed stochastic local gradient clipping algorithm to train deep neural networks.
By exploring the relaxed smoothness condition which was shown to be satisfied for certain neural networks, we theoretically prove both the linear speedup property and the improved communication complexity.
Our empirical studies show that our algorithm indeed enjoys parallel speedup and greatly improves the runtime performance due to skipping communication rounds in various scenarios.
A PPENDIX A P ROPERTIES OF ( L 0 , L 1 ) FUNCTIONS AND U SEFUL L EMMAS
Lemma 4 (Descent inequality in Zhang et al.
Let f be ( L 0 , L 1 ) -smooth, and c > 0 be a constant.
For any x k and x k +1 , as long as ∥ x k − x k +1 ∥ ≤ c/L 1 , we have f
[Equation]
x k +
[Equation]
) where A = 1 + e c − e c − 1 c and B = e c − 1 c .
Lemma 5 (Zhang et al.
Let f be ( L 0 , L 1 ) -smooth, and c > 0 be a constant.
For any x k and x k +1 , as long as ∥ x k − x k +1 ∥ ≤ c/L 1 , we have
[Equation]
5) where A = 1 + e c − e c − 1 c and B = e c − 1 c .
Lemma 6 (Zhang et al.
Let µ ≥ 0 be a real constant.
For any vector u and v ,
[Equation]
u ,
[Equation]
B P ROOF OF T HEOREM
During the proof of Theorem 1, we need the following simple fact for our algorithm.
Fact Recall the definition of ¯ x t = 1 N (cid:80) Ni =1 x it for a fixed iteration t , Algorithm 1 immediately gives us ¯ x t +1 − ¯ x t = − 1 N (cid:80)
Ni =1 min (cid:16) η, γ ∥∇ F i ( x it ; ξ it ) ∥ (cid:17) ∇ F i ( x it ; ξ it ) , and
hence ∥ ¯ x t +1 − ¯ x t ∥ 2 ≤ γholds for any t .
B. P ROOF OF L EMMA
Lemma 1 restated Under Assumption 1, Algorithm 1 ensures
[Equation]
Proof.
Fix t ≥ 1 and i .
Note that Algorithm 1 calculates the node average every I iterations.
Let t 0 ≤ t to be the largest multiple of I .
Then we know ¯ x t
= x it for all i .
(Note that t − t 0 ≤ I ).
We further notice that
[Equation]
By the definition of ¯ x t , we have
[Equation]
Thus
[Table]
we have
[Table]
The technique we mainly use in this proof is the inequality ∥ (cid:80) Kj =
[Equation]
z j ∥
[Equation]
≤ K (cid:80) Kj =1 ∥
z j ∥ 2 for any
K vectors z 1 , . . .
, z K .
Taking the square root for both sides, then we obtain the desired result.
B. P ROOF OF L EMMA
Note that J ( t ) =
{ i ∈ [1 , . . .
, N ] | ∥∇ F i ( x it , ξ it )
∥ ≥ γη } .
Lemma 2 restated If AL 0 η ≤ 1 / 2 and 2 γI ≤ c/L 1 for some c > 0 , then we have
[Equation]
where A = 1 + e c − e c − 1 c and B = e c − 1 c .
Proof.
By the ( L 0 , L 1 ) -smoothness and invoking Lemma 4 and noting that ∥ ¯ x t +1 − ¯ x t ∥ ≤ γ for any t , we have
[Equation]
For each iteration t , we first consider the case where i ∈ J ( t ) , i.e. ∥∇ F i ( x it , ξ it )
∥ ≥ γη .
We use
Lemma 6 with µ = 25 to obtain the following result
[Equation]
Taking the expectation for both sides of above inequality
[Equation]
then we obtain
[Equation]
Next, we consider the case in which i ∈ ¯ J ( t ) ,
i
[Equation]
e. ∥∇ F i ( x it , ξ it )
∥ ≤ γη at fixed iteration t .
Denote ξ [ t − 1] by the σ -algebra generated by all random variables until ( t − 1) -th iteration, i.e., ξ [ t − 1]
= σ ( ξ 10 , . . .
, ξ N 0 , ξ 11 , . .
. , ξ N 1 , . .
. , ξ 1 t − 1 , . .
. , ξ Nt − 1 ) .
We notice that
[Equation]

[Equation]
i ∈ ¯ J ( t )
[Equation]
F i ( x it ,
[Equation]
it ) (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)
ξ [ t − 1]   ,
E (cid:2)
[Equation]
J ( t )
[Equation]
(cid:3)
N
[Equation]
f (
[Equation]
x t ) (cid:43)  =
[Equation]
(cid:34)(cid:42)
E
[Equation]
N
[Equation]
i
[Equation]
(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)
[Equation] [Equation]
N
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
N
[Equation] [Table] [Equation]
[Table]
[Equation]
[Table]
[Equation]
N
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
) (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
− (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
p
[Equation]
f
[Equation]
(cid:2)
[Equation]
( t )
[Equation]
(cid:3)
N
[Equation]
f (
[Equation]
t ) (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
[Table]
[Equation] [Table] [Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
∥∇ f
(¯ x t ) ∥ 2 − η 2 E (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) 1 N
N (cid:88) i
= p
i
[Equation]
f i ( x it ) (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) + η ( E |
[Equation]
( t ) | )
N
E (cid:104)
[Equation]
γ
[Equation]
I
[Equation]
( AL
+
[Equation]
1 ∥∇ f (
[Equation]
x t )
[Equation] [Equation]
(cid:105) ( e ) ≤ −
[Equation]
E
[Equation]
t
[Equation]
|
[Equation] [Equation]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) 1 N
N (cid:88) i
= p
i ∇ f i (
[Equation]
it )
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) + η ( E | ¯
[Equation]
( t ) | )
N
E (cid:2) 2 γ 2 I 2 (cid:0)
A 2 L 20 +
[Equation]
B 2 L 21 ∥∇ f
(¯ x t )
[Equation]
2 (cid:1)(cid:3) ≤ − η
( E | ¯ J ( t ) | )
[Equation]
N
[Equation] [Equation]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) 1 N
N (cid:88) i =
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation] [Equation]
21 η
[Equation]
f (¯
[Equation]
t )
[Equation]
(cid:3) , (12) where (a) holds due to Lemma 8 with A i =
∇ F i ( x it ; ξ it ) , B i =
I ( i ∈
¯ J ( t )) , 0 ≤ p i ≤ 1 is the corresponding constant defined in Lemma 8 and (cid:80)
Ni =1 p i =
E (cid:2) | ¯ J ( t ) | (cid:3) ; (b) holds by using the basic identity ⟨ x 1 , x 2 ⟩ = 12 (cid:0)
∥ x 1 ∥ 2 + ∥ x 2 ∥ 2 − ∥ x 1 − x 2 ∥ 2 (cid:1) for any two vectors x 1 , x 2 ; (c) holds due to
Jensen’s inequality and (cid:80) Ni =1 p i =
E | ¯ J ( t ) | .
(d) holds due to Lemma 1 and Lemma 5, (e) holds since ( a + b ) 2 ≤
a
+ 2 b 2 .
Next, we compute E ∥ ¯ x t +1 − ¯ x t ∥ 2 .
By definition, we have
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
1 N
N (cid:88) i = ∇ F i ( x it , ξ it ) − E (cid:34)
N
N (cid:88) i = ∇ F i ( x it , ξ it ) (cid:35)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) + 2 η 2 (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) E (cid:32)
N
N (cid:88) i = ∇ F i ( x it , ξ it ) I (cid:0) i ∈
¯ J ( t ) (cid:1)(cid:33)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) ≤ 2 γ 2 ( E | J ( t )
[Equation] [Equation]
(cid:32)
[Equation]
(cid:88)
i
[Equation] [Equation]
(cid:0)
[Equation]
( t
[Equation]
(cid:1)(cid:33)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
( b ) =
[Equation]
γ
[Equation] [Equation]
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
(cid:32)
[Equation]
(cid:88)
[Equation]
(cid:33)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) (13) where (a) holds due to Lemma 7 with A i =
∇ F i ( x it ; ξ it ) , B i =
I (cid:0) i ∈
¯ J ( t ) (cid:1) ; (b) holds due to
Lemma 8 with A i = ∇ F i ( x it ; ξ it ) , B i
=
I ( i ∈
¯ J ( t )) , 0 ≤ p i ≤ 1 is the corresponding constant defined in Lemma 8 and (cid:80)
Ni =1 p i =
E (cid:2) | ¯ J ( t ) | (cid:3) .
Substituting (11), (12) and (13) into (9) yields
[Equation]
where the last inequality holds if AL
η ≤ 1 2 and E ∥ X ∥ 2 ≥ ∥
E X ∥ 2 .
B. P ROOF OF L EMMA
Lemma 3 restated Suppose Assumption 1 holds.
If 2 γI ≤ c/L 1 for some c > 0 , then we obtain
[Equation]
almost surely , where A = 1 + e c − e c − 1 c and B = e c − 1 c .
Proof.
By using triangular inequality and Lemma 5, we obtain
[Equation]
Summing over i = 1 to N and by Assumption 1, we have
[Equation]
B. M AIN P ROOF OF T HEOREM
Proof.
We consider 0 < ϵ ≤ min( AL
BL 1 , 0 . 1) be a small enough constant and choose N ≤ min { 1 ϵ , 14 AL
[Equation]
5 BL 1 ϵ } , γ ≤
Nϵ 28( σ + κ ) min { ϵAL 0 , 1 BL 1 } , ratio γ/η = 5( σ + κ ) and I ≤ 12 Nϵ , we first check several important conditions which were used in previous Lemmas and we will use in our later proof.
[Equation]
2 , where (a) holds because of Nϵ ≤ 1 and (b) is true due to σ + κ ≥ 1 .
Note that Nϵ ≤ 1 , σ + κ ≥ 1 ,
A ≥ 1 , B ≥ 1 , and so we have
[Equation]
where c = 128 .
Recall the descent inequality (Lemma 4), we can explicitly define
[Equation]
We can show that our choice of ϵ guarantees that ϵAL 0 ≤ 1 BL 1 .
Then, the upper bound of the parameter γ becomes
[Equation]
Based on Lemma 2 and Lemma 3, we take summation over t and obtain
[Equation]
where (a) holds due to Lemma
[Equation]
and
[Equation]
(b) is derived by computing
[Equation]
η 28 2 ( σ + κ )
A 2 L28 2 A 2 L25 B 2 L 21 ϵ(cid:124) (cid:123)(cid:122) (cid:125) bound of N 4 N 2 ϵ 2 (cid:124) (cid:123)(cid:122) (cid:125) bound of I≤ η (16) with N ≤ min { 1 ϵ , 14 AL 0 5 BL 1 ϵ } , γ ≤
Nϵ 28( σ + κ ) min { ϵAL 0 , 1 BL 1 } and I ≤ 12 Nϵ .
Now we give the upper bound of P 1 and P 2 .
From the choice of N ≤ min { 1 ϵ , 14 AL 0 5 BL 1 ϵ } , γ ≤
Nϵ 2 28 AL 0 ( σ + κ ) , I ≤ 12 Nϵ and the fixed ratio γη = 5( σ + κ ) , and noting that σ + κ ≥ 1 , we have
[Equation]
Plugging (17) back to (15), we obtain
[Equation]
V (
[Equation]
  , where
[Equation]
[Figure]
Inequality (a) follows by using the standard inequality x 2 ≥ 2 ϵx − ϵ 2 and (b) is true because
[Equation]
To check (18), we compute
[Equation]
where the last inequality holds because we assume N ≤ 14 AL 0 5 BL 1 ϵ
[Equation]
It’s clear that U ( x ) ≤ V ( x ) .
Then we obtain −
[Equation]
Then we rearrange above inequality and get
[Equation]
as long as T ≥ 560 AL 0 ∆
[Equation]
2 Nϵand I ≤ 12 Nϵ .
C E XPERIMENTS DETAILS AND OTHER RESULTS C. CIFAR-10 CLASSIFICATION WITH R ES N ET -
Apart from the 56 layer Resnet used in Section 5, we also trained the 32 layer Resnet on CIFAR-10.
Here, we used a smaller minibatch size of 16 per GPU.
We also used SGD with clipping for the baseline algorithm with a stagewise decaying learning rate schedule, We set the initial learning rate η = 0 .
1 and the clipping parameter γ = 1 . 0 .
We decrease both η and γ by a factor of 10 at epoch 80 and 120 .
These parameter settings follow that of Zhang et al.
Results reported in Figure 6 again shows that our algorithm can not only match the baseline in both training loss and test accuracy epoch-wise, but is way better in terms of wall clock time for moderate values of I .
D U SEFUL L EMMAS
Lemma 7.
Suppose { A i } N i =1 is an sequence of vector-valued independent random variables, { B i } Ni =1 is a sequence of independent random variables where B i ∈ { 0 , 1 } for every i .
Suppose E ∥
A i ∥ 2 < + ∞ for every i .
Then we have
[Equation]
Proof.
It suffices to show that E ∥
A i B i − E
[ A i B i ] ∥ 2 ≤ E ∥
A i − E
[ A i ] ∥ 2 , since both { A i B i − E
[ A i B i ] } Ni =1 and { A i − E
[ A i ] } Ni =1 are sequences of independent random vectors with zero mean.
Define C i = 1 − B i , and hence B i C i = 0 with probablity 1 .
Without loss of generality, we can assume E
[ A i ] = 0 .
Note that
[Equation]
It completes the proof.
Lemma 8.
Suppose { A i } Ni
=1 is an sequence of vector-valued independent random variables, { B i } Ni =1 is a sequence of independent random variables where B i ∈ { 0 , 1 } for every i .
A i and
B j are independent of each other if i ̸
= j .
Then there exists ( p 1 , . .
. , p N ) where 0 ≤ p i ≤ 1 and (cid:80) Ni =1 p i =
E (cid:104)(cid:80) Ni
[Equation]
1 B i (cid:105) such that
[Equation]
Proof.
Without loss of generality, we can assume the sequence { B i } Ni =1 is sorted in a decreasing order, i.e., B 1 ≥ B 2 ≥ . . .
≥ B N , since otherwise we can switch the order of summation in both
LHS and RHS of the Equation (20).
Define (cid:101) T = sup { 1 ≤ i ≤ N : B i = 1 } , and define F s = σ ( A 1 , B 1 , . . .
, A s , B s ) , where σ ( · ) denotes σ -algebra generated by random variables in the argument.
Since { (cid:101) T = s } ∈ F s for all s , we know that (cid:101) T is a stopping time, and (cid:101) T ≤ N almost surely.
Hence we have
E
[Equation]
Note that I ( (cid:101) T ≥ i )
=
1 − I ( (cid:101) T ≤ i − 1) ∈
F i − 1 , and A i is independent of F i − 1 , we know that + ∞
[Equation]
i =
[Equation]
Define p i =
Pr ( (cid:101) T ≥ i ) , we have 0 ≤ p i ≤ 1 and (cid:80) Ni =1 p
i
=
E ( (cid:101) T )
=
E (cid:104)(cid:80) Ni
=
1 B i (cid:105) .
Hence (20) is proved.
E M ORE EXPERIMENT RESULTS E. P ARALLEL S PEEDUP ON NLP T ASKS
In Section 5.2 we have shown the parallel speedup effects of our algorithm on training a Resnet model on the CIFAR10 dataset.
Here we present the same phenomena but on training AWD-LSTMs on doing language modeling on Penn Treebank and Wikitext-2 in Figure 7.
Again, note that in the distributed setting, one iteration means running one step of Algorithm 1 on all machine; while in the single machine setting, one iteration means running one step of SGD with clipping.
For Penn Treebank, we use minibatch size 3 on every GPU in distributed setting to run
Algorithm 1, while we also use 3 minibatch size on the single GPU to run SGD with clipping.
For
Wikitext-2, the corresponding minibatch size is 10 .
Figure 7 again clearly shows that even with I > 1 , our algorithm still enjoys parallel speedup, since our algorithm requires less number of iterations to converge to the same targets (e.g., training loss, validation perplexity).
This observation is consistent with our iteration complexity results in
Theorem 1.
[Figure]
E. G RADIENT N ORM D YNAMICS
The careful reader might notice that in Theorem 1 we require N ≤ O (cid:0) 1 ϵ (cid:1) where ϵ is the desired ℓnorm of the gradient when converging and wonder if this is too strict and impractical.
Therefore, we draw the dynamics of the ℓ 2 norm of gradients over epochs for training the Resnetmodel to do image classification on CIFAR10 in Figure 8.
It can be clearly seen that the gradient ℓ 2 norm decreases to less than 0 .
1 after epoch 100 and eventually reaches around 0 .
06 at the epoch 200 .
Since 1 / 0 . 06 ≈ 17 , this means that the N = 8 setting we used in our experiments is consistent with our theory.
E. H ETEROGENEOUS S ETTING
In the previous experiments, all machines can access the whole dataset.
Yet, in practice, each machine might only access a subset of the whole dataset and the data might not be evenly distributed across machines.
Therefore, we conduct an experiment on this heterogeneous setting.
Specifically, we take the Penn Treebank training set which is a text file and divide it into 8 non-overlapping consecutive parts which contain { 10 . 64% , 11 .
36% } of the whole text file respectively.
We then train the AWD-LSTM of the same structure as in Section 5.1 to do language modeling.
Considering that the heterogeneous setting introduces big difference compared with the setting we adopted in Section 5.1, we finely tuned the initial learning rate for both the baseline and our algorithm and picked the one that gives the smallest training loss.
The results are reported in Figure 9.
Note that, as the size of training data in each machine is different, we draw the training and testing performance curves over the number of iterations each machine runs instead of epochs.
This means that, if the x-axis value is 1000 , then each GPU runs 1000 iterations.
We can see that, though the unbalancing and heterogeneity of the training data deteriorates the performance for both the baseline and our algorithm as compared with Figure 2a, our algorithm is still able to surpass the baseline in training losses though it falls slightly behind in testing; meanwhile, our algorithm obtains significant speedup in terms of wall-clock time.
This suggests that our algorithm is robust to the heterogeneous setting.
[Figure]
[Figure]
E.
P ERFORMANCE VS . C OMMUNICATION R OUNDS
The reader might be interested in how the training and testing performance changes w.r.t.
the number of communications each machine does for both the Naive Parallel SGDClip and our CELGC with different I .
Thus, we show such results in Figure 10.
The results are very close to that w.r.t.
the wall-clock time which is as expected.
E. P ARTIAL P ARTICIPATION
In the previous experiments, when communication between nodes occurs, all nodes send their model and/or gradients for averaging and update accordingly.
Yet, in practice, it might be that only a (different) subset of nodes participate in communication each time.
Therefore, we conduct an experiment on this setting.
Specifically, we take the Penn Treebank dataset and train the AWD-LSTM of the same structure as in Section 5.1 to do language modeling.
However, when the algorithm requires a communication across nodes, we randomly select 6 out of the total 8 nodes to communicate and average the model and/or gradient and update those 6 nodes only; the rest 2 nodes will be left untouched.
Considering that this setting introduces big difference compared with the setting we adopted in Section 5.1, we finely tuned the initial learning rate for both the baseline and our algorithm and picked the one that gives the smallest training loss.
The results are reported in Figure 11.
We can see that, though the partial participation setting slightly deteriorates the performance for both the baseline and our algorithm as compared with Figure 2a, our algorithm is still able to closely match the baseline for both the training loss and the validation [Figure] [Figure]
perplexity epoch-wise, but also note that our algorithm greatly saves wall-clock time.
This indicates that our algorithm is robust in the partial participation setting.
E. L ARGE M INI - BATCH
The distributed learning paradigm is especially desirable with training using large mini-batches.
To this end, we trained a Resnet- 56 model on CIFAR10 with batch-size 256 on each node which sums up to a batch-size of 2048 globally.
Compared with the experment in Section 5.1, considering the large batch-size, we trained for 300 epochs instead of 200 , decrease the learning rate by a factor of 10 at epoch 150 and 250 , and finely tuned the initial learning rate for each variant and picked the one that gives the smallest training loss.
Results are shown in Figure 12.
Similar to small batch case reported in Figure 1, our algorithm with skipped communication is able to match the naive parallel
SGDClip in terms of epochs but greatly speeds up wall-clock-wise.
Also, Figure 12c again verifies the parallel speed-up property of our algorithm.