C ONVERGENT A DAPTIVE G RADIENT M ETHODS IN D ECENTRALIZED O PTIMIZATION
Anonymous authors
Paper under double-blind review
A BSTRACT
Adaptive gradient methods including Adam, AdaGrad, and their variants have been very successful for training deep learning models, such as neural networks, in the past few years.
Meanwhile, given the need for distributed training procedures, distributed optimization algorithms are at the center of attention.
With the growth of computing power and the need for using machine learning models on mobile devices, the communication cost of distributed training algorithms needs careful consideration.
In that regard, more and more attention is shifted from the traditional parameter server training paradigm to the decentralized one, which usually requires lower communication costs.
In this paper, we rigorously incorporate adaptive gradient methods into decentralized training procedures and introduce novel convergent decentralized adaptive gradient methods.
Speciﬁcally, we propose a general algorithmic framework that can convert existing adaptive gradient methods to their decentralized counterparts.
In addition, we thoroughly analyze the convergence behavior of the proposed algorithmic framework and show that if a given adaptive gradient method converges, under some speciﬁc conditions, then its decentralized counterpart is also convergent.
I NTRODUCTION
Distributed training of machine learning models is drawing growing attention in the past few years due to its practical beneﬁts and necessities.
Given the evolution of computing capabilities of CPUs and GPUs, computation time in distributed settings is gradually dominated by the communication time in many circumstances (Chilimbi et al., 2014; McMahan et al., 2017).
As a result, a large amount of recent works has been focussing on reducing communication cost for distributed learning (Alistarh et al., 2017; Lin et al., 2018;
Wangni et al., 2018; Stich et al., 2018; Wang et al., 2018; Tang et al., 2019).
In the traditional parameter (central) server setting, where a parameter server is employed to manage communication in the whole network, many effective communication reductions have been proposed based on gradient compression (Aji & Heaﬁeld, 2017) and quantization (Chen et al., 2010;
Ge et al., 2013; Jegou et al., 2010) techniques.
Despite these communication reduction techniques, its cost still, usually, scales linearly with the number of workers.
Due to this limitation and with the sheer size of decentralized devices, the decentralized training paradigm (Duchi et al., 2011b), where the parameter server is removed and each node only communicates with its neighbors, is drawing attention.
It has been shown in Lian et al.
(2017) that decentralized training algorithms can outperform parameter server-based algorithms when the training bottleneck is the communication cost.
The decentralized paradigm is also preferred when a central parameter server is not available.
In light of recent advances in nonconvex optimization, an effective way to accelerate training is by using adaptive gradient methods like AdaGrad (Duchi et al., 2011a), Adam (Kingma & Ba, 2015) or AMSGrad (Reddi et al., 2018).
Their popularity are due to their practical beneﬁts in training neural networks, featured by faster convergence and ease of parameter tuning compared with
Stochastic Gradient Descent (SGD) (Robbins & Monro, 1951).
Despite a large amount of studies within the distributed optimization literature, few works have considered bringing adaptive gradient methods into distributed training, largely due to the lack of understanding of their convergence behaviors.
Notably, Reddi et al.
(2020) develop the ﬁrst decentralized ADAM method for distributed optimization problems with a direct application to federated learning.
An inner loop is employed to compute mini-batch gradients on each node and a global adaptive step is applied to update the global parameter at each outer iteration.
Yet, in the settings of our paper, nodes can only communicate to their neighbors on a ﬁxed communication graph while a server/worker communication is required in Reddi et al.
Designing adaptive methods in such settings is highly non-trivial due to the already complex update rules and to the interaction between the effect of using adaptive learning rates and the decentralized communication protocols.
This paper is an attempt at bridging the gap between both realms in nonconvex optimization.
Our contributions are summarized as follows:
• In this paper, we investigate the possibility of using adaptive gradient methods in the decentralized training paradigm, where nodes have only a local view of the whole communication graph.
We develop a general technique that converts an adaptive gradient method from a centralized method to its decentralized variant.
• By using our proposed technique, we present a new decentralized optimization algorithm, called decentralized AMSGrad, as the decentralized counterpart of AMSGrad.
• We provide a theoretical veriﬁcation interface, in Theroem 2, for analyzing the behavior of decentralized adaptive gradient methods obtained as a result of our technique.
Thus, we characterize the convergence rate of decentralized AMSGrad, which is the ﬁrst convergent decentralized adaptive gradient method, to the best of our knowledge.
A novel technique in our framework is a mechanism to enforce a consensus on adaptive learning rates at different nodes.
We show the importance of consensus on adaptive learning rates by proving a divergent problem instance for a recently proposed decentralized adaptive gradient method, namely
DADAM (Nazari et al., 2019), a decentralized version of AMSGrad.
Though consensus is performed on the model parameter, DADAM lacks consensus principles on adaptive learning rates.
After having presented existing related work and important concepts of decentralized adaptive methods in Section 2, we develop our general framework for converting any adaptive gradient algorithm in its decentralized counterpart along with their rigorous ﬁnite-time convergence analysis in Section 3 concluded by some illustrative examples of our framework’s behavior in practice.
Notations : x t,i denotes variable x at node i and iteration t .
(cid:107)·(cid:107) abs denotes the entry-wise L 1 norm of a matrix, i.e. (cid:107) A (cid:107)
abs
= (cid:80) i,j | A i,j | .
We introduce important notations used throughout the paper: for any t > 0 , G t :=
[ g t,N ] where [ g t,N ] denotes the matrix [ g t, 1 , g t, 2 , · · · , g t,N ] (where g t,i is a column vector), M t :=
[ m t,N ] , X t :=
[ x t,N ] , ∇ f ( X t ) :
= 1 N (cid:80) Ni =1 ∇ f i ( x t,i ) , U t :=
[ u t,N ] , ˜
U t :=
[˜ u t,N ] ,
V t :=
[ v t,N ] , ˆ V t :=
[ˆ v t,N ] , X t := 1 N (cid:80) Ni =1 x t,i , U t := 1 N (cid:80) Ni
=1 u t,i and ˜
U t := 1 N (cid:80) Ni
=1 ˜ u t,i .
D ECENTRALIZED A DAPTIVE T RAINING AND D IVERGENCE OF DADAM 2. R ELATED W ORK
Decentralized optimization: Traditional decentralized optimization methods include well-know algorithms such as ADMM (Boyd et al., 2011), Dual Averaging (Duchi et al., 2011b), Distributed
Subgradient Descent (Nedic & Ozdaglar, 2009).
More recent algorithms include Extra (Shi et al., 2015), Next (Di Lorenzo & Scutari, 2016), Prox-PDA (Hong et al., 2017), GNSD (Lu et al., 2019), and Choco-SGD (Koloskova et al., 2019).
While these algorithms are commonly used in applications other than deep learning, recent algorithmic advances in the machine learning community have shown that decentralized optimization can also be useful for training deep models such as neural networks.
Lian et al.
(2017) demonstrate that a stochastic version of Decentralized Subgradient Descent can outperform parameter server-based algorithms when the communication cost is high.
Tang et al.
(2018) propose the D 2 algorithm improving the convergence rate over Stochastic Subgradient Descent.
Assran et al.
(2019) propose the Stochastic Gradient Push that is more robust to network failures for training neural networks.
The study of decentralized training algorithms in the machine learning community is only at its initial stage.
No existing work, to our knowledge, has seriously considered integrating adaptive gradient methods in the setting of decentralized learning.
One noteworthy work (Nazari et al., 2019) propose a decentralized version of AMSGrad (Reddi et al., 2018) and it is proven to satisfy some non-standard regret.
Adaptive gradient methods: Adaptive gradient methods have been popular in recent years due to their superior performance in training neural networks.
Most commonly used adaptive methods include AdaGrad (Duchi et al., 2011a) or Adam (Kingma & Ba, 2015) and their variants.
Key features of such methods lie in the use of momentum and adaptive learning rates (which means that the learning rate is changing during the optimization and is anisotropic, i.e. depends on the dimension).
The method of reference, called Adam, has been analyzed in Reddi et al.
(2018) where the authors point out an error in previous convergence analyses.
Since then, a variety of papers have been focussing on analyzing the convergence behavior of the numerous existing adaptive gradient methods.
Ward et al.
(2019), Li & Orabona (2019) derive convergence guarantees for a variant of
AdaGrad without coordinate-wise learning rates.
Chen et al.
(2019) analyze the convergence behavior of a broad class of algorithms including AMSGrad and AdaGrad.
Zou & Shen (2018) provide a uniﬁed convergence analysis for AdaGrad with momentum.
Noticeable recent works on adaptive gradient methods can be found in Agarwal et al.
; Luo et al.
Zaheer et al.
D ECENTRALIZED O PTIMIZATION
In distributed optimization (with N nodes), we aim at solving the following problem
[Equation]
where x is the vector of parameters and f i is only accessible by the i th node.
Through the prism of empirical risk minimization procedures, f i can be viewed as the average loss of the data samples located at node i , for all i ∈ [ N ] .
Throughout the paper, we make the following mild assumptions required for analyzing the convergence behavior of the different decentralized optimization algorithms:
A1.
For all i ∈ [ N ] , f i is differentiable and the gradients is L -Lipschitz,
i
e., for all ( x, y ) ∈ R d , (cid:107)∇ f i ( x ) − ∇ f i ( y ) (cid:107) ≤ L (cid:107) x − y (cid:107) .
A2.
We
assume
that, at iteration t , node i accesses a stochastic gradient g t,i .
The stochastic gradients and the gradients of f i have bounded L ∞ norms,
i
e. (cid:107) g t,i (cid:107) ≤ G ∞ , (cid:107)∇ f i ( x ) (cid:107) ∞
≤ G ∞ .
A3.
The gradient estimators are unbiased and each coordinate have bounded variance, i.e. E [ g t,i ]
= ∇ f i ( x t,i ) and E [([ g t,i − f i ( x t,i )] j ) 2 ] ≤ σ 2 , ∀ t, i,
j .
Assumptions A1 and A3 are standard in distributed optimization literature.
A2 is slightly stronger than the traditional assumption that the estimator has bounded variance, but is commonly used for the analysis of adaptive gradient methods (Chen et al., 2019; Ward et al., 2019).
Note that the bounded gradient estimator assumption in A2 implies the bounded variance assumption in A3.
In decentralized optimization, the nodes are connected as a graph and each node only communicates to its neighbors.
In such case, one usually constructs a N × N matrix W for information sharing when designing new algorithms.
We denote λ i to be its i th largest eigenvalue and deﬁne λ (cid:44) max( | λ 2 | , | λ N | ) .
The matrix W cannot be arbitrary, its required key properties are listed in the following assumption:
A4.
The matrix W satisﬁes:
[Equation]
| λ N | < 1 and ( III ) W i,j =
if node i and node j are not neighbors.
We now present the failure to converge of current decentralized adaptive method before introducing our proposed framework for general decentralized adaptive gradient methods.
2. D IVERGENCE OF DADAM
Algorithm β 2 ) g 2 t,i 7: ˆ v t,i =
β 3 ˆ v t,i +(1 − β 3 ) max(ˆ v t − 1 ,i , v t,i ) 8: x t
+ 12 ,i = (cid:80) Nj =1 W ij x t,j 9: x t +1 ,i = x t + 12 ,i − α m t,i √
ˆ v t,i 10: end for
Recently, Nazari et al.
(2019) initiated an attempt to bring adaptive gradient methods into decentralized optimization with Decentralized
ADAM (DADAM), shown in Algorithm 1.
DADAM is essentially a decentralized version of ADAM and the key modiﬁcation is the use of a consensus step on the optimization variable x to transmit information across the network, encouraging its convergence.
The matrix W is a doubly stochastic matrix (which satisﬁes A4) for achieving average consensus of x .
Introducing such mixing matrix is standard for decentralizing an algorithm, such as distributed gradient descent (Nedic & Ozdaglar, 2009; Yuan et al., 2016).
It is proven in Nazari et al.
that DADAM admits a non-standard regret bound in the online setting.
Nevertheless, whether the algorithm can converge to stationary points in standard ofﬂine settings such training neural networks is still unknown.
The next theorem shows that DADAM may fail to converge in the ofﬂine settings.
Theorem 1.
There exists a problem satisfying A1-A4 where DADAM fails to converge to a stationary points with ∇ f ( ¯ X t ) = 0 .
Proof.
Consider a two-node setting with objective function f ( x ) = 1 / 2 (cid:80) 2 i =
f
i ( x ) and f 1 ( x ) =[ | x | ≤ 1]2 x 2 + 1 [ | x | > 1](4 | x |− 2) , f 2 ( x ) = 1 [ | x − 1 | ≤ 1]( x − 1) 2 + 1 [ | x − 1 | > 1](2 | x − 1 |− 1) .
We set the mixing matrix W =
The optimal solution is x ∗ = 1 / 3 .
Both f 1 and f 2 are smooth and convex with bounded gradient norm 4 and 2, respectively.
We also have L =(deﬁned in A1).
If we initialize with x 1 , 1 = x 1 , 2 = − 1 and run DADAM with β 1 = β 2 = β 3 =and (cid:15) ≤ 1 , we will get ˆ v 1 , 1 = 16 and ˆ v 1 , 2 = 4 .
Since | g t, 1 | ≤ 4 , | g t, 2 | ≤ 2 due to bounded gradient, and (ˆ v t, 1 , ˆ v t, 2 ) are non-decreasing, we have ˆ v t, 1 = 16 , ˆ v t, 2 = 4 , ∀ t ≥ 1 .
Thus, after t = 1 , DADAM is equivalent to running decentralized gradient descent (DGD)
(Yuan et al., 2016) with a re-scaled fand f 2 , i.e. running DGD on f (cid:48)
( x ) = (cid:80) 2 i =1 f (cid:48) i ( x ) with f (cid:48) 1 ( x ) = 0 .
25 f 1 ( x ) and f (cid:48) 2 ( x )
= 0 .
5 f 2 ( x ) , which unique optimal x (cid:48) = 0 . 5 .
Deﬁne ¯ x t
=
( x t, 1 + x t, 2 ) / 2 , then by Th.
2 in Yuan et al.
(2016), we have when α < 1 / 4 , f (cid:48)
(¯ x t ) − f ( x (cid:48) )
=
O (1 / ( αt )) .
Since f (cid:48) has a unique optima x (cid:48) , the above bound implies ¯ x t is converging to x (cid:48)
= 0 .
5 which has non-zero gradient on function ∇ f (0 . 5)
= 0 .
Theorem 1 shows that, even though DADAM is proven to satisfy some regret bounds (Nazari et al., 2019), it can fail to converge to stationary points in the nonconvex ofﬂine setting (common for training neural networks).
We conjecture that this inconsistency in the convergence behavior of DADAM is due to the deﬁnition of the regret in Nazari et al.
The next section presents decentralized adaptive gradient methods that are guaranteed to converge to stationary points under assumptions and provide a characterization of that convergence in ﬁnite-time and independently of the initialization.
C ONVERGENCE OF D ECENTRALIZED A DAPTIVE G RADIENT M ETHODS
In this section, we discuss the difﬁculties of designing adaptive gradient methods in decentralized optimization and introduce an algorithmic framework that can turn some existing convergent adaptive gradient methods to their decentralized counterparts.
We also develop the ﬁrst convergent decentralized adaptive gradient method, converted from AMSGrad, as an instance of this framework.
3. I MPORTANCE AND D IFFICULTIES OF C ONSENSUS ON A DAPTIVE L EARNING R ATES
The divergent example in the previous section implies that we should synchronize the adaptive learning rates on different nodes.
This can be easily achieved in the parameter server setting where all the nodes are sending their gradients to a central server at each iteration.
The parameter server can then exploit the received gradients to maintain a sequence of synchronized adaptive learning rates when updating the parameters, see Reddi et al.
However, in our decentralized setting, every node can only communicate with its neighbors and such central server does not exist.
Under that setting, the information for updating the adaptive learning rates can only be shared locally instead of broadcasted over the whole network.
This makes it impossible to obtain, in a single iteration, a synchronized adaptive learning rate update using all the information in the network.
Systemic Approach: On a systemic level, one way to alleviate this bottleneck is to design communication protocols in order to give each node access to the same aggregated gradients over the whole network, at least periodically if not at every iteration.
Therefore, the nodes can update their individual adaptive learning rates based on the same shared information.
However, such solution may introduce an extra communication cost since it involves broadcasting the information over the whole network.
Algorithmic Approach: Our contributions being on an algorithmic level, another way to solve the aforementioned problem is by letting the sequences of adaptive learning rates, present on different nodes, to gradually consent , through the iterations.
Intuitively, if the adaptive learning rates can consent fast enough, the difference among the adaptive learning rates on different nodes will not affect the convergence behavior of the algorithm.
Consequently, no extra communication costs need to be introduced.
We now develop this exact idea within the existing adaptive methods stressing on the need for a relatively low-cost and easy-to-implement consensus of adaptive learning rates.
3. D ECENTRALIZED A DAPTIVE G RADIENT U NIFYING F RAMEWORK
We now develop a method that implements consensus of adaptive learning rates.
While each node can have different ˆ v t,i in DADAM (Algorithm 1), one can keep track of the min/max/average of these adaptive learning rates and use that quantity as the new adaptive learning rate.
1: Input: α , initial point x 1 ,i = x init , u 12 ,i =
ˆ v 0 ,i , m 0 ,i = 0 , mixing matrix W 2: for t = 1 , 2 , · · · , T do 3: for all i ∈
[ N ] do in parallel 4: g t,i ← ∇ f i ( x t,i ) + ξ t,i 5: m t,i =
β 1 m t − 1 ,i + (1 − β 1 ) g t,i 6: ˆ v t,i =
r t ( g 1 ,i , · · · , g t,i ) 7: x t + 12 ,i = (cid:80) Nj =1 W ij x t,j 8: ˜ u t,i =
(cid:80) Nj =1 W ij ˜ u t − 12 ,j 9: u t,i =
max(˜ u t,i , (cid:15) )
10: x t +1 ,i = x t + 12 ,i − α m t,i √ u t,i 11: ˜ u t + 12 ,i =
˜
u t,i − ˆ v t − 1 ,i + ˆ v t,i 12: end for
The predeﬁnition of some convergent lower and upper bounds may also lead to a gradual synchronization of the adaptive learning rates on different nodes as developed for AdaBound in Luo et al.
In this paper, we present an algorithm framework for decentralized adaptive gradient methods as Algorithm 2, which uses average consensus of ˆ v t,i (see consensus update in line 8 and 11) to help convergence.
Algorithm 2 can become different adaptive gradient methods by specifying r t as different functions.
E.g., when we choose ˆ v t,i = 1 t (cid:80) tk
=1 g 2 k,i , Algorithm 2 becomes a decentralized version of
AdaGrad.
When one chooses ˆ v t,i to be the adaptive learning rate for AMSGrad, we get decentralized AMSGrad (Algorithm 3).
The intuition of using average consensus is that for adaptive gradient methods such as AdaGrad or Adam, ˆ v t,i approximates the second moment of the gradient estimator, the average of the estimations of those second moments from different nodes is an estimation of second moment on the whole network.
Also, this design will not introduce any extra hyperparameters that can potentially complicate the tuning process ( (cid:15) in line 9 is important for numerical stability as in vanilla Adam).
The main convergence result for the framework Algorithm 2 reads as follows:
Theorem 2.
Assume A1-A4.
When α ≤ (cid:15) 0 .
5 16 L , Algorithm 2 yields the following regret bound
[Equation]
where (cid:107) · (cid:107) abs denotes the entry-wise L 1 norm of a matrix (i.e (cid:107) A (cid:107)
abs = (cid:80) i,j |
A ij | ).
The constants
C 1 = max(4 , 4 L/(cid:15) ) , C 2 = 6(( β 1 / (1 − β 1 ))
+ 1 / (1 − λ ) 2 ) LG 2 ∞ /(cid:15) 1 . 5 , C
=
L 2 (1 − λ )
G 2 ∞ /(cid:15) 2 ,
C 4 = 2 / ( (cid:15) 1 . 5 (1 − λ ))
( λ + β 1 / (1 − β 1 ))
G 2 ∞ , C 5 = 2 / ( (cid:15) 2 (1 − λ ))
L ( λ + β 1 / (1 − β 1 ))
G 2 ∞ +4 / ( (cid:15) 2 (1 − λ ))
LG 2 ∞ are independent of d , T and N .
In addition, 1 N (cid:80) Ni =1 (cid:13)(cid:13) x t,i − X t (cid:13)(cid:13)
[Equation]
≤ α 2 (cid:16) 1 1 − λ (cid:17) 2 dG 2 ∞ 1 (cid:15) which quantiﬁes the consensus error.
Theorem 2 how the convergence guarantee is affected by different factors.
In addition, one can specify α to show convergence in terms of T , d , and N .
An immediate result is by setting α = √ N/ √ Td , which is shown in Corollary 2.1.
Corollary 2.1.
Assume A1-A4.
Set α = √ N/ √ Td .
When α ≤ (cid:15) 0 .
5 16 L , Algorithm 2 yields the following regret bound
[Equation]
where V T :=
(cid:80) Tt =1 (cid:107) ( − ˆ V t −
+ ˆ V t − 1 ) (cid:107) abs and C 1 , C 2 , C 3 , C 4 , C 5 are deﬁned in Theorem 2.
Corollary 2.1 shows that if E [ V T ] =
o ( T ) and ¯ U t is upper bounded, then Algorithm 2 is guaranteed to converge to stationary points of the loss function.
Intuitively, this means that if the adaptive learning rates on different nodes do not change too fast, the algorithm can converge.
It is shown in Chen et al.
(2019) that such a condition can be satisﬁed by AdaGrad and AMSGrad.
Besides, if this condition is violated, the algorithm may diverge (e.g., Adam).
Later, we will show convergence of decentralized versions of AMSGrad and AdaGrad by bounding this term as O ( Nd ) and O ( Nd log( T )) , respectively.
The intuition E [ V T ] =
o ( T ) can guarantee divergence is that the correlation between ˆ v t,i and m t,i (due to their shared dependency on historical gradients) can make update direction negatively correlated with true gradient in expectation, leading to a non-negligible bias in updates.
However, the total bias across T iterations introduced by such a correlation is bounded by the term E
[ V T ] .
Thus, if E [ V T ] grows sublinearly with T , convergence can still be guaranteed.
Furthermore, Corollary 2. conveys the beneﬁts of using more nodes in the graph.
When T is large enough such that the term O ( √ N/ √ Td ) dominates the RHS of (3), linear speedup can be achieved by increasing N .
We now present, in Algorithm 3, a notable special case of our algorithmic framework, namely
Decentralized AMSGrad, which is a decentralized variant of AMSGrad.
Compared with DADAM, the above algorithm exhibits a dynamic average consensus mechanism to keep track of the average of { ˆ v t,i } Ni =1 , stored as ˜ u t,i on i th node, and uses u t,i :=
max(˜ u t,i , (cid:15) ) for updating the adaptive learning rate for i th node.
As the number of iteration grows, even though ˆ v t,i on different nodes can converge to different constants, the u t,i will converge to the same number lim
t →∞ 1 N (cid:80) Ni
=
ˆ v t,i if the limit exists.
This average consensus mechanism enables the consensus of adaptive learning rates on different nodes, which accordingly guarantees the convergence of the method to stationary points.
The consensus of adaptive learning rates is the key difference between decentralized AMSGrad and
DADAM and is the reason why decentralized AMSGrad is convergent while DADAM is not.
g t,i ← ∇ f i ( x t,i ) + ξ t,i 5: m t,i =
β 1 m t − 1 ,i + (1 − β 1 ) g t,i 6: v t,i =
β 2 v t − 1 ,i + (1 − β 2 )
g 2 t,i 7: ˆ v t,i =
max(ˆ v t − 1 ,i , v t,i ) 8: x t + 12 ,i = (cid:80) Nj =1 W ij x t,j 9: ˜ u t,i =
(cid:80) Nj =1 W ij ˜ u t − 12 ,j 10: u t,i = max(˜ u t,i , (cid:15) ) 11: x t +1 ,i = x t + 12 ,i − α m t,i √ u t,i 12: ˜ u t + 12 ,i =
˜ u t,i − ˆ v t − 1 ,i + ˆ v t,i 13: end for
One may notice that decentralized AMSGrad does not reduce to AMSGrad for N = 1 since the quantity u t,i in line 10 is calculated based on v t − 1 ,i instead of v t,i .
This design encourages the execution of gradient computation and communication in a parallel manner.
Speciﬁcally, line 4-7 (line 4-6) in Algorithm 3 (Algorithm 2) can be executed in parallel with line 8-9 (line 7-8) to overlap communication and computation time.
If u t,i depends on v t,i which in turn depends on g t,i , the gradient computation must ﬁnish before the consensus step of the adaptive learning rate in line 9.
This can slow down the running time per-iteration of the algorithm.
To avoid such delayed adaptive learning, adding ˜ u t − 12 ,i = ˜
u t,i − ˆ v t − 1 ,i + ˆ v t,i before line 9 and get rid of line 12 in Algorithm 2 is an option.
Similar convergence guarantees will hold since one can easily modify our proof of Theoremfor such update rule.
As stated above, Algorithm 3 converges, with the following rate:
Theorem 3.
Assume A1-A4.
Set α = 1 / √ Td .
When α ≤ (cid:15) 0 .
5 16 L , Algorithm 3 yields the following regret bound
[Equation]
Nd
T
[Equation]
T
[Equation]
where D f :=
E [ f ( Z 1 )] − min x f ( x ) , C (cid:48)
1 = C 1 , C (cid:48) 2 = C 2 , C (cid:48) 3 = C 3 , C (cid:48)
4 =
C 4 G 2 ∞ and C (cid:48)
= C 5 G 2 ∞ .
C 1 , C 2 , C 3 , C 4 , C 5 are constants independent of d , T and N deﬁned in Theorem 2.
In addition, the consensus of variables at different nodes is given by 1 N (cid:80)
[Equation]
[Table]
Theorem 3 shows that Algorithm 3 converges with a rate of O ( √ d/ √ T ) when T is large, which is the best known convergence rate under the given assumptions.
Note that in some related works, SGD admits a convergence rate of O (1 / √ T ) without any dependence on the dimension of the problem.
Such improved convergence rate is derived under the assumption that the gradient estimator have a bounded L 2 norm, which can thus hide a dependency of √ d in the ﬁnal convergence rate.
Another remark is the convergence measure can be converted to 1 T (cid:80)
Tt =1 E (cid:104)(cid:13)(cid:13) ∇ f
( X t ) (cid:13)(cid:13) 2 (cid:105) using the fact that (cid:107) U t (cid:107) ∞
≤ G 2 ∞ (by update rule of Algorithm 3), for the ease of comparison with existing literature.
3. C ONVERGENCE A NALYSIS
The detailed proofs of this section are reported in the supplementary material.
Proof of Theorem 2: We now present a proof sketch for out main convergence result of Algorithm 2.
Step 1: Reparameterization.
Similarly to Yan et al.
; Chen et al.
(2019) with SGD (with momentum)
and centralized adaptive gradient methods, deﬁne the following auxiliary sequence:
[Equation]
with X 0 (cid:44) X 1 .
Such an auxiliary sequence can help us deal with the bias brought by the momentum and simpliﬁes the convergence analysis.
An intermediary result needed to conduct our proof reads:
Lemma 1.
For the sequence deﬁned in (4) , we have
[Equation]
Lemma 1 does not display any momentum term in 1 N (cid:80) Ni
=1 g t,i √ u t,i .
This simpliﬁcation is convenient since it is directly related to the current gradients instead of the exponential average of past gradients.
Step 2: Smoothness.
Using smoothness assumption A1 involves the following scalar product term:
[Equation]
U t (cid:105) which can be lower bounded by:
[Equation]
The above inequality substituted in the smoothness condition f ( Z t +1 ) ≤ f ( Z t ) + (cid:104)∇ f ( Z t ) , Z t +1 −
Z t (cid:105) + L 2 (cid:107) Z t +1 − Z t (cid:107) 2 yields:
[Equation]
where ∆ f := E [ f ( Z 1 )] − E [ f ( Z T +1 )]
D 1 , D 2 and D 3 are three terms, deﬁned in the supplementary material, and which can be tightly bounded from above.
We ﬁrst bound D 3 using the following quantities of interest:
[Equation]
where λ = max( | λ 2 | , | λ N | ) and recall that λ i is i th largest eigenvalue of W .
Then, concerning the term D 2 , few derivations, not detailed here for simplicity, yields:
[Equation]
where q l is the eigenvector corresponding to l th largest eigenvalue of W and (cid:107) · (cid:107) abs is the entry-wise
L 1 norm of matrices.
We can also show that
[Equation]
resulting in an upper bound for D 2 proportional to (cid:80) T − 1 o =0 (cid:107) ( − ˆ [Equation] o − [Equation]
ˆ
[Equation]
o ) (cid:107) abs .
Similarly:
[Equation]
Step 3:
Bounding the drift term variance.
An important term that needs upper bounding in our proof is the variance of the gradients multiplied (element-wise) by the adaptive learning rate:
[Equation]
where Γ fu :
= 1 /N (cid:80) Ni =1 ∇ f i ( x t,i ) / √ u t,i .
Two consecutive and simple bounding of the above yields:
T (cid:88)
[Equation]
=
[Equation]
and
T [Equation]
(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
Then, by plugging the LHS of (6) in (5), and further bounding as operated for D 2 , D 3 (see supplement), we obtain the desired bound in Theorem 2.
Proof of Theorem 3: Recall the bound in (3) of Theorem 2.
Since Algorithm 3 is a special case of Algorithm 2, the remaining of the proof consists in characterizing the growth rate of
E [ (cid:80) Tt =1 (cid:107) ( − ˆ V t −
+ ˆ V t − 1 ) (cid:107) abs ] .
By construction, ˆ V t is non decreasing, then it can be shown that
E [ (cid:80) Tt =1 (cid:107) ( − ˆ V t −
+ ˆ V t − 1 ) (cid:107) abs ] =
E [ (cid:80) Ni =1 (cid:80) dj
=1 ( − [ˆ v 0 ,i ] j
+ [ˆ v T − 1 ,i ] j )] .
Besides, since for all t, i , (cid:107) g t,i (cid:107) ∞ ≤ G ∞ and v t,i is an exponential moving average of g 2 k,i , k = 1 , 2 , · · · , t , we have | [ v t,i ]
j | ≤ G 2 ∞ for all t, i, j .
By construction of ˆ V t , we also observe that each element of ˆ V t cannot be greater than G 2 ∞ , i.e. | [ˆ v t,i ] j | ≤ G 2 ∞ for all t, i, j .
Given that [ˆ v 0 ,i ] j ≥ 0 , we have
[Equation]
Substituting into (3) yields the desired convergence bound for Algorithm 3.
3. I LLUSTRATIVE N UMERICAL E XPERIMENTS
In this section, we conduct some experiments to test the performance of Decentralized AMSGrad, developed in Algorithm 3, on both homogeneous data and heterogeneous data distribution (i.e. the data generating distribution on different nodes are assumed to be different).
Comparison with
DADAM and the decentralized stochastic gradient descent (DGD) developed in Lian et al.
(2017) are conducted.
We train a Convolutional Neural Network (CNN) with 3 convolution layers followed by a fully connected layer on MNIST (LeCun, 1998).
We set (cid:15) =
10 − 6 for both Decentralized AMSGrad and DADAM.
The learning rate is chosen from the grid [10 − 1 , 10 − 2 , 10 − 3 , 10 − 4 , 10 − 5 , 10 − 6 ] based on validation accuracy for all algorithms.
In the following experiments, the graph contains 5 nodes and each node can only communicate with its two adjacent neighbors forming a cycle.
Regarding the mixing matrix W , we set W ij = 1 / 3 if nodes i and j are neighbors and W ij = 0 otherwise.
More details and experiments can be found in the supplementary material of our paper
[Figure]
Homogeneous data: The whole dataset is shufﬂed and evenly split into different nodes.
Such a setting is possible when the nodes are in a computer cluster.
We see, Figure 1(a), that decentralized
AMSGrad and DADAM perform quite similarly while DGD is much slower both in terms of training loss and test accuracy.
Though the (possible) non convergence of DADAM, mentioned in this paper, its performance are empirically good on homogeneous data.
The reason is that the adaptive learning rates tend to be similar on different nodes in presence of homogeneous data distribution.
We thus compare these algorithms under the heterogeneous regime.
Heterogeneous data: Here, each node only contains training data with two labels out of ten.
Such a setting is common when data shufﬂing is prohibited, such as in federated learning.
We can see that each algorithm converges signiﬁcantly slower than with homogeneous data.
Especially, the performance of DADAM deteriorates signiﬁcantly.
Decentralized AMSGrad achieves the best training and testing performance in that setting as observed Figure 1(b).
E XTENSION TO A DA G RAD
In this section, we provide a decentralized version of AdaGrad (optionally with momentum) converted by Algorithm 2, further supporting the usefulness of the decentralization framework.
The required modiﬁcation for decentralize AdaGrad is to specify line 4 of Algorithm 2 as
[Equation]
which is equivalent to ˆ v t,i = 1 t (cid:80) tk
=1 g 2 k,i .
Throughout this section, we will call this algorithm decentralized AdaGrad.
There are two details in the algorithm worth mentioning.
One is that the framework uses momentum m t,i in updates, while original AdaGrad does not use momentum.
The momentum can be turned off by setting β 1 = 0 and the convergence results will hold.
The other one is that in decentralized AdaGrad, we use average instead of sum in ˆ v t,i .
I.e. ˆ v t,i =
1 t (cid:80) tk =1 g 2 k,i .
This is different from original AdaGrad which should use ˆ v t,i = (cid:80) tk =1 g 2 k,i .
The reason is in original
AdaGrad, a constant stepsize ( α independent of t or T ) is used with ˆ v t,i = (cid:80) tk =1 g 2 k,i and this is equivalent to using a well-known diminishing stepsize sequence α t = 1 √ t with ˆ v t,i = 1 t (cid:80) tk
=1 g 2 k,i .
In our convergence analysis which will be presented later, we will use a constant stepsize α = O ( 1 √ T ) to replace the diminishing stepsize sequence α t
= O ( 1 √ t ) .
Such a replacement is popularly used in
SGD analysis to simplify analysis and achieving better convergence rate.
In addition, it is easy to modify our theoretical framework to apply diminishing stepsize sequences such as α t =
O ( 1 √ t ) .
The convergence analysis for decentralized AdaGrad is shown in Theorem 4.
Theorem 4.
Assume A1-A4.
Set α = √ N/ √ Td .
When α ≤ (cid:15) 0 .
5 16 L , decentralized AdaGrad yields the following regret bound
[Equation]
where D (cid:48)
[Equation]
E [
[Equation]
( Z
[Equation] ] − [Equation] 2 = C 2 , C (cid:48) 3 = C 3 , C (cid:48)
4 =
C 4 G 2 ∞ and
C (cid:48)
= C 5 G 2 ∞ .
C 1 , C 2 , C 3 , C 4 , C 5 are deﬁned in Theorem 2 independent of d , T and N .
In addition, the consensus of variables at different nodes is given by 1 N (cid:80) Ni =1 (cid:13)(cid:13) x t,i − X t (cid:13)(cid:13) 2 ≤ NT (cid:16) 1 1 − λ (cid:17) 2 G 2 ∞ 1 (cid:15) .
C ONCLUSION
This paper studies the problem of designing adaptive gradient methods for decentralized training.
We propose a unifying algorithmic framework that can convert existing adaptive gradient methods to decentralized settings.
With rigorous convergence analysis, we show that if the original algorithm satisﬁes converges under some minor conditions, the converted algorithm obtained using our proposed framework is guaranteed to converge to stationary points of the regret function.
By applying our framework to AMSGrad, we propose the ﬁrst convergent adaptive gradient methods, namely
Decentralized AMSGrad.
Experiments show that the proposed algorithm achieves better performance than the baselines.
A P ROOF OF A UXILIARY L EMMAS
Lemma 1.
For the sequence deﬁned in (10) , we have
[Equation]
Proof: By update rule of Algorithm 2, we ﬁrst have
[Equation]
where (i) is due to an interchange of summation and (cid:80) i =1 W ij = 1 .
Then
[Equation]
we have
[Equation]
which is the desired result.
(cid:3)
Lemma A.1.
Given a set of numbers a 1 , · · · , a n and denote their mean to be ¯ a = 1 n (cid:80) ni =1 a i .
Deﬁne b i ( r ) (cid:44) = max( a i , r ) and ¯ b ( r ) = 1 n (cid:80) ni
=
b i ( r ) .
For any r and r (cid:48) with r (cid:48) ≥ r we have n
[Equation]
i = | b i
[Equation]
r )
[Equation]
b
[Equation]
r
[Equation]
| ≥ n (cid:88) i = |
[Equation]
| (8) and when r ≤ min i ∈ [ n ] a i , we have n
[Equation]
Proof : Without loss of generality, assume a i ≤ a j when i < j , i.e. a i is a non-decreasing sequence.
Deﬁne
[Equation]
We need to prove that h is a non-increasing function of r .
First, it is easy to see that h is a continuous function of r with non-differentiable points r = a i , i ∈ [ n ] , thus h is a piece-wise linear function.
Next, we will prove that h ( r ) is non-increasing in each piece.
Deﬁne l
r ) to be the largest index with a ( l ( r ))
< r , and s ( r ) to be the largest index with a s ( r ) < ¯ b ( r ) .
Note that we have for i ≤ l ( r ) , b i ( r ) =
r and for i ≤ s ( r ) b i ( r ) − ¯ b ( r ) ≤ 0 since a i is a non-decreasing sequence.
Therefore, we have
[Equation]
and
[Equation]
Taking derivative of the above form, we know the derivative of h ( r ) at differentiable points is
[Equation]
Since we have s ( r ) ≤ n we know ( l ( r ) − n ) + ( s ( r ) − l
( r )) − ( n − s ( r )) ≤ 0 and thus h (cid:48)
( r ) ≤ 0 , which means h ( r ) is non-increasing in each piece.
Combining with the fact that h ( r ) is continuous, (8) is proven.
When r ≤ a ( i ) , we have b ( i ) =
max( a i , r ) =
r , for all r ∈ [ n ] and ¯ b ( r ) =n (cid:80) ni
=1 a i = ¯ a which proves (9).
(cid:3)
B P ROOF OF T HEOREM
To prove convergence of the algorithm
[Equation]
we ﬁrst deﬁne an auxiliary sequence
[Equation] [Equation]
with X 0 (cid:44) X 1 .
Since E [ g t,i ]
= ∇ f ( x t,i ) and u t,i is a function of G 1: t − 1 (which denotes
G 1 , G 2 , · · · , G t − 1 ), we have
[Equation]
Assuming smoothness (A1) we have
[Equation]
Using Lemma 1 into the above inequality and take expectation over G t given G 1: t −
[Equation]
we have
[Equation]
Then take expectation over G 1: t − 1 and rearrange, we have
[Equation]
In addition
[Equation]
we have
[Equation]
and the ﬁrst term on RHS of the equality can be lower bounded as
[Equation]
[Table]
t (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) −2 (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) 1 N (cid:80) Ni =1 ∇ f i ( x t,i ) − ∇ f ( X t )
U 1 / 4 t (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) ≥ (cid:13)(cid:13)(cid:13)(cid:13) (cid:13) ∇ f ( X t )
U 1 / 4 t (cid:13)(cid:13)(cid:13)(cid:13) (cid:13) − (cid:13)(cid:13)(cid:13)(cid:13) (cid:13) ∇ f ( Z t ) − ∇ f ( X t )
U 1 / 4 t (cid:13)(cid:13)(cid:13)(cid:13) (cid:13) − (cid:13)(cid:13)(cid:13)(cid:13) (cid:13) 1 N (cid:80) Ni =1 ∇ f i ( x t,i ) − ∇ f ( X t )
U 1 / 4 t (cid:13)(cid:13)(cid:13)(cid:13) (cid:13) , (14) where the inequalities are all due to Cauchy-Schwartz.
Substituting (14) and (13) into (11), we get 2 α E
[Equation] [Equation]
+
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
(cid:80)
[Equation]
t
[Equation]
i
[Equation]
U
[Equation] [Equation]
t (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) + (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) ∇
[Equation] [Equation]
t )
[Equation]
( X t )
U 1 / 4 t (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) 2   .
Then sum over the above inequality from t = 1 to T and divide both sides by Tα/ 2 , we have
[Equation]
[Equation]
Now we need to upper bound all the terms on RHS of the above inequality to get the convergence rate.
For the terms composing D 3 in (15), we can upper bound them by
[Equation]
and
[Equation]
using Jensen’s inequality, Lipschitz continuity of f i , and the fact that f = 1 N (cid:80) Ni =1 f i .
Next we need to bound D 4 and D 5 .
Recall the update rule of X t , we have
[Equation]
where we deﬁne W
= I .
Since W is a symmetric matrix, we can decompose it as W = Q Λ Q T where Q is a orthonormal matrix and Λ is a diagonal matrix whose diagonal elements correspond to eigenvalues of W in an descending order, i.e. Λ ii
= λ i with λ
i being i th largest eigenvalue of
W .
In addition, because W is a doubly stochastic matrix, we know λ 1 = 1 and q 1 = 1 N √ N .
With eigen-decomposition of W , we can rewrite D 5 as
N (cid:88) i =
[Equation] [Equation]
In addition, we can rewrite (18) as
[Equation]
where the last equality is because x 1 ,i = x 1 ,j , for all i, j and thus X 1 W = X 1 .
Then we have when l > 1 ,
[Equation]
since Q is orthonormal and X 1 q l = x 1 , 1 1 TN q l = x 1 , 1 √ Nq T 1 q l = 0 , for all l (cid:54) = 1 .
Combining (19) and (21), we have
[Equation]
(22) where the last inequality follows from the fact that g t,i ≤ G ∞ , (cid:107) q l (cid:107)
= 1 , and | λ l | ≤ λ < 1 .
Now let us turn to D 4 , it can be rewritten as
[Equation]
Now we know both D 4 and D 5 are in the order of O ( α 2 ) and thus D 3 is in the order of
O ( α 2 ) .
Next we will bound D 2 and D 1 .
Deﬁne G 1 (cid:44) max t ∈ [ T ] max i ∈ [ N ] (cid:107)∇ f i ( x t,i ) (cid:107) ∞ ,
G 2 (cid:44) max t ∈ [ T ] (cid:107)∇ f ( Z t ) (cid:107) ∞ , G 3 (cid:44) max t ∈ [ T ] max i ∈
[ N ] (cid:107) g t,i (cid:107) ∞ and G ∞ =
max( G 1 , G 2 , G 3 ) .
Then we have
D
[Equation]
=
[Equation]
(cid:88) t =
E (cid:34)(cid:42) ∇ f ( Z t ) , 1 N
N (cid:88) i = ∇ f i ( x t,i ) (cid:12) (cid:32)(cid:112)
U t − 1 √ u t,i (cid:33)(cid:43)(cid:35) ≤
T (cid:88) t =
E   G 2 ∞ 1 N
N (cid:88) i =
d (cid:88)
j = (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) (cid:113)
[ U t ] j − (cid:112) [ u t,i ] j (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) =
T (cid:88) t =
E   G 2 ∞ 1 N
N (cid:88) i =
d (cid:88)
j = (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) (cid:113)
[ U t ] j − (cid:112) [ u t,i ] j (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) (cid:113) [ U t ]
j + (cid:112) [ u t,i ] j (cid:113) [ U t ]
j + (cid:112) [ u t,i ] j   =
T (cid:88) t =
E   G 2 ∞ 1 N
N (cid:88) i =
d (cid:88)
j = (cid:12)(cid:12)(cid:12) (cid:12) (cid:12) (cid:12) [ U t ] j − [ u t,i ]
j [ U t ] j (cid:112) [ u t
[Equation]
where the last inequality is due to [ u t,i ] j ≥ (cid:15) , for all t, i, j .
To simplify notations, deﬁne (cid:107) A (cid:107)
abs = (cid:80) i,j |
A ij | to be the entry-wise L 1 norm of a matrix A , then we obtain
[Equation]
where the second inequality is due to Lemma A.1, introduced Section A, and the fact that U t = max( ˜ U t , (cid:15) )
(element-wise max operator).
Recall from update rule of U t , by deﬁning ˆ V − 1 (cid:44) ˆ V 0 and
U 0 (cid:44) U 1 / 2 , we have for all t ≥ 0 , ˜
U t +1 =
( ˜ U t − ˆ V t − 1 + ˆ V t ) W .
Thus, we obtain
[Equation]
Then we further obtain when l (cid:54) = 1 ,
[Equation]
where the last equality is due to the deﬁnition ˜
U 0 (cid:44) U 1 / 2 = (cid:15) 1 d 1 TN = √ N(cid:15) 1 d 1 TN (recall that q 1 = 1 √ N 1 TN ) and q Ti q j =
when i (cid:54)
=
j .
Note that by deﬁnition of (cid:107) · (cid:107) abs , we have for all
A, B, (cid:107) A +
[Equation]
(cid:107)
[Equation]
A (cid:107) abs + (cid:107) B (cid:107) abs , then
[Equation]
where λ = max( | λ 2 | , | λ N | ) .
Combining (24) and (25), we have
[Equation]
1 + ˆ V o ) (cid:107) abs (cid:35) .
Now we need to bound D 1 , we have
D
[Equation]
=
[Equation] [Equation]
i ]
[Equation] [Equation]
[Equation] [Equation]
[Equation]
(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)
[Equation]
[Equation] [Equation]
where ( a ) is due to [˜ u t − 1 ,i ] j = max([ u t − 1 ,i ] j , (cid:15) ) and the function max( · , (cid:15) ) is 1-Lipschitz.
In addition, by update rule of U t , we have
T (cid:88) t =
[Equation]
=
[Equation]
≤
[Equation] [Equation]
k =
[Equation]
− ˆ
[Equation]
−
[Equation] [Equation]
Combining (26) and (27), we have
[Equation]
[Equation]
What remains is to bound (cid:80)
Tt =1 E (cid:2) (cid:107) Z t +1 − Z t (cid:107) 2 (cid:3) .
By update rule of Z t , we have
[Equation]
[Table]
[Equation] [Equation]
where the last inequality is again due to the deﬁnition that [˜ u t,i ] j = max([ u t,i ] j , (cid:15) ) and the fact that max( · , (cid:15) ) is 1-Lipschitz.
Then, we have
[Equation]
2   ≤ α 2 (cid:18) β1 − β(cid:19) 2 G 2 ∞ √
N (cid:15) 1 − λ E (cid:34) T (cid:88) t
= (cid:107) ( − ˆ V t −
+ ˆ V t − 1 ) (cid:107) abs
(cid:35)
+ 2 α
T (cid:88) t =
E   (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) 1 N
N (cid:88) i = g t,i √ u t,i (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) 2   , where the last inequality is due to (27).
We now bound the last term on RHS of the above inequality.
A trivial bound can be
T (cid:88) t = (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
N
N (cid:88) i = g t,i √ u t,i (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) ≤
T (cid:88) t = dG 2 ∞ 1 (cid:15) , due to (cid:107) g t,i (cid:107) ≤ G ∞ and [ u t,i ] j ≥ (cid:15) , for all j (veriﬁed from update rule of u t,i and the assumption that [ v t,i ] j ≥ (cid:15) , for all i ).
However, the above bound is independent of N , to get a better bound, we need a more involved analysis to show its dependency on N .
To do this, we ﬁrst notice that
[Table]
N (cid:88) i =
d (cid:88) l =
E G t | G 1: t − 1 [[ ξ t,i ] 2 l ] [ u t,i ] l ( c ) ≤ (cid:13)(cid:13) (cid:13) (cid:13) (cid:13)
N
N (cid:88) i = ∇ f i ( x t,i ) √ u t,i (cid:13)(cid:13) (cid:13) (cid:13) (cid:13) + d
N σ(cid:15) , where (a) is due to E G t | G 1: t − 1 [ ξ t,i ] = 0 and ξ t,i is independent of x t,j , u t,j for all j , and ξ j , for all j (cid:54)
=
i , (b) comes from the fact that x t,i , u t,i are ﬁxed given G 1: t , (c) is due to E G t | G 1: t − 1 [[ ξ t,i ] 2 l ≤ σand [ u t.i ] l ≥ (cid:15) by deﬁnition.
Then we have
E   (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) 1 N
N (cid:88) i = g t,i √ u t,i (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) 2   =
E G 1: t − 
E G t | G 1: t −  (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) 1 N
N (cid:88) i = g t,i √ u t,i (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) 2   
 ≤ E G 1: t −  (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
In traditional analysis of SGD-like distributed algorithms, the term corresponding to
E (cid:20)(cid:13)(cid:13)(cid:13) 1 N (cid:80) Ni =1 ∇ f i ( x t,i ) √ u t,i (cid:13)(cid:13)(cid:13) 2 (cid:21) will be merged with the ﬁrst order descent when the stepsize is chosen to be small enough.
However, in our case, the term cannot be merged because it is different from the ﬁrst order descent in our algorithm.
A brute-force upper bound is possible but this will lead to a worse convergence rate in terms of N .
Thus, we need a more detailed analysis for the term in the following.
[Equation]
∇ f i ( x t,i ) (cid:112)
U t +
N
N (cid:88) i = ∇ f i ( x t,i ) (cid:12) (cid:32) 1 √ u t,i − (cid:112)
U t (cid:33)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) 2   ≤ 2 E   (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) 1 N
N (cid:88) i = ∇ f i ( x t,i ) (cid:112)
U t (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) 2   + 2 E  
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) 1 N
N (cid:88) i = ∇ f i ( x t,i ) (cid:12) (cid:32) 1 √ u t,i − (cid:112)
U t (cid:33)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) 2   ≤ 2 E   (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) 1 N
N (cid:88) i = ∇ f i ( x t,i ) (cid:112)
U t (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) 2   + 2 E  
1 N
N (cid:88) i = (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) ∇ f i ( x t,i ) (cid:12) (cid:32) 1 √ u t,i − (cid:112)
U t (cid:33)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Table]
[Table]
Summing over T , we have
[Table]
[Equation]
For the last term on RHS of (31), we can bound it similarly as what we did for D 2 from (24) to (25), which yields
[Equation]
Further
[Equation]
we have
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
 and the last term on RHS of the above inequality can be bounded following similar procedures from (17) to (22), as what we did for D 3 .
Completing the procedures yields
[Equation]
Finally, combining (30) to (33), we get
T (cid:88)
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
where the last inequality is due to each element of U t is lower bounded by (cid:15) by deﬁnition.
Combining all above
[Equation]
we obtain
[Equation]
1 − λ V T (cid:21) +
T
G 2 ∞ √
N 2 (cid:15) 1 .λ 1 − λ E
[ V T ] +
T (cid:32) T (cid:88) t =
L (cid:18) 1 − λ (cid:19)α 2 dG 2 ∞ 1 (cid:15) 1 . 5 +
T (cid:88) t =
L (cid:18) β1 − β(cid:19)α 2 dG 2 ∞ (cid:15) 1 .(cid:33) =
Tα ( E [ f ( Z 1 )] − E [ f ( Z T +1 )]) + 2 Lα d N σ 2 (cid:15) + 8 Lα 1 √ (cid:15) 1 T
T (cid:88) t =
E   (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) ∇ f ( X t )
U 1 / 4 t (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) 2   + 3 α 2 d (cid:32)(cid:18) β1 − β(cid:19)+ (cid:18) 1 − λ (cid:19) 2 (cid:33) LG 2 ∞ (cid:15) 1 . 5 + 8 α 3 L(cid:18)
1 − λ (cid:19) dG 2 ∞ (cid:15)+
T(cid:15) 1 .
G 2 ∞ √
N 1 − λ (cid:32)
Lα (cid:18) β1 − β(cid:19) 2(cid:15) 0 .
5 + λ + β1 − β 1 + 2 Lα (cid:15) 0 .
5 λ (cid:33)
E [ V T ] .
where V T := (cid:80) T t =1 (cid:107) ( − ˆ V t −
+ ˆ V t − 1 ) (cid:107) abs .
Set α = 1 √ dT and when α ≤ (cid:15) 0 .
5 16 L , we further have
[Equation]
+ 6 α 2 d (cid:32)(cid:18) β1 − β(cid:19)
[Table]
+ (cid:18) 1 − λ (cid:19)
[Table]
(cid:33) LG
[Table]
(cid:15)
[Table] [Table]
L
[Table]
(cid:18)
[Table]
−
[Table]
(cid:19) dG
[Table]
(cid:15)
[Table]
+
T(cid:15)
[Table]
G
[Table]
√
N 1 − λ (cid:32)
Lα (cid:18)
[Table] [Table]
(cid:19)
[Table]
(cid:15)
[Table] [Table]
(cid:15)
[Table]
E [
[Table] [Table]
Tα
[Table] [Table]
[Table]
d (cid:32)(cid:18)
[Table] [Table]
(cid:19)
[Table]
(cid:18)
[Table]
(cid:19)
[Table]
LG
[Table]
(cid:15)
[Table] [Table]
dL
[Table]
(cid:18) 1 −
[Table]
(cid:19)
[Table]
(cid:15)
[Table]
+
T(cid:15)
[Table]
N 1 − λ (cid:32)
Lα (cid:18)
[Table]
1 −
[Table]
(cid:19)
[Table]
(cid:15)
[Table]
+
[Table]
+ β
[Table]
1 −
[Table]
Lα (cid:15)
[Table]
(cid:33)
E [ V
[Table] [Table]
N (cid:19)
[Table]
[Equation] [Equation]
where the ﬁrst inequality is obtained by moving the term 8 Lα 1 √ (cid:15) 1 T (cid:80)
Tt =1 E
[Equation]
∇
[Equation]
( X t
[Equation]
U
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13) 2 (cid:35) on the
RHS of (34) to the LHS to cancel it using the assumption 8 Lα 1 √ (cid:15) ≤ 12 followed by multiplying both sides by 2.
The constants introduced in the last step are deﬁned as following
[Equation]
Substituting into Z 1 =
X 1 completes the proof.
[Equation]
C P ROOF OF T HEOREM
Under some assumptions stated in Corollary 2.1, we have that
T
[Equation]
t =
E 
[Equation]
36) where (cid:107) ·
(cid:107) abs denotes the entry-wise L 1 norm of a matrix (i.e (cid:107) A (cid:107) abs
= (cid:80) i,j |
A ij | ) and
C 1 , C 2 , C 3 , C 4 , C 5 are deﬁned in Theorem 2.
Since Algorithm 3 is a special case of 2, building on result of Theorem 2, we just need to characterize the growth speed of E (cid:104)(cid:80)
Tt =1 (cid:107) ( − ˆ V t −
+ ˆ V t − 1 ) (cid:107) abs
(cid:105) to prove convergence of Algorithm 3.
By the update rule of Algorithm 3, we know ˆ V t is non decreasing and thus
E (cid:34)
[Equation]
t = (cid:107)
[Equation]
abs (cid:35) =
E  
T (cid:88) t =
N (cid:88) i =
d (cid:88)
j
= | − [ˆ v t − 2 ,i ] j + [ˆ v t − 1 ,i ] j | 
 =
E  
T (cid:88) t =
N (cid:88) i =
d (cid:88)
j = ( − [ˆ v t − 2 ,i ] j
+ [ˆ v t − 1 ,i ] j ) 
 =
E  
N (cid:88) i =
d (cid:88)
j = ( − [ˆ v − 1 ,i ] j
+ [ˆ v T − 1 ,i ] j ) 
 =
E  
N (cid:88) i =
d (cid:88)
j = ( − [ˆ v 0 ,i ] j + [ˆ
[Equation]
T − 1 ,i ] j ) 
 , where the last equality is because we deﬁned ˆ V − 1 (cid:44) ˆ V 0 previously.
Further, because (cid:107) g t,i (cid:107) ∞
≤ G ∞ for all t, i and v t,i is a exponential moving average of g 2 k,i , k = 1 , 2 , · · · , t , we know | [ v t,i ]
j | ≤ G 2 ∞ , for all t, i, j .
In addition, by update rule of ˆ V t , we also know each element of ˆ V t also cannot be greater than G 2 ∞ , i.e. | [ˆ v t,i ] j | ≤ G 2 ∞ , for all t, i, j .
Given the fact that [ˆ v 0 ,i ] j ≥ 0 , we have
E
[Equation]
Substituting the above into (39)
[Equation]
we have
[Equation]
(37) where we have
[Equation]
and we conclude the proof
[Equation]
D P ROOF OF T HEOREM
The proof follows the same ﬂow as that of Theorem 3.
Under assumptions stated in Corollary
[Equation]
1, set α = √ N/ √ Td , we have that
[Equation]
where (cid:107) · (cid:107) abs denotes the entry-wise L 1 norm of a matrix (i.e (cid:107) A (cid:107)
abs = (cid:80) i,j |
A ij | ) and
C 1 , C 2 , C 3 , C 4 , C 5 are deﬁned in Theorem 2.
Again, Since decentralized AdaGrad is a special case of 2, we can apply Corollary 2.1 and what we need is to upper bound E (cid:104)(cid:80)
Tt =1 (cid:107) ( − ˆ V t −
+ ˆ V t − 1 ) (cid:107) abs (cid:105) derive convergence rate.
By the update rule of decentralized AdaGrad, we have ˆ v t,i = 1 t ( (cid:80) tk
=1 g 2 k,i ) for t ≥ 1 and ˆ v 0 ,i = (cid:15) 1 .
Then we have for t ≥ 3 ,
E
[Equation]
max
[Equation]
where the ﬁrst equality is because we deﬁned ˆ V − 1 (cid:44) ˆ V 0 previously and (cid:107) g k,i (cid:107) ∞
≤ G ∞ by assumption.
Substituting the above into (39), we have
[Equation]
where we have
[Equation]
and we conclude the proof.
[Equation]
E A DDITIONAL E XPERIMENTS AND D ETAILS
In this section, we compare the training loss and testing accuracy of different algorithms, namely
Decentralized Stochastic Gradient Descent (DGD), Decentralized Adam (DADAM) and our proposed
Decentralized AMSGrad, with different stepsizes on heterogeneous data distribution.
We use 5 nodes and the heterogeneous data distribution is created by assigning each node with data of only two labels.
Note that there are no overlapping labels between different nodes.
For all algorithms, we compare stepsizes in the grid [10 − 1 , 10 − 2 , 10 − 3 , 10 − 4 , 10 − 5 , 10 − 6 ] .
Figure 2 shows the training loss and test accuracy for DGD algorithm.
We observe that the stepsize 10 − 3 works best for DGD in terms of test accuracy and 10 − 1 works best in terms of training loss.
This difference is caused by the inconsistency among the value of parameters on different nodes when the stepsize is large.
The training loss is calculated as the average of the loss value of different local models evaluated on their local training batch.
Thus, while the training loss is small at a particular node, the test accuracy will be low when evaluating data with labels not seen by the node (recall that each node contains data with different labels since we are in the heterogeneous setting).
Figure 3 shows the performance of decentralized AMSGrad with different stepsizes.
We see that its best performance is better than the one of DGD and the performance is more stable (the test performance is less sensitive to stepsize tuning).
Figure 4 displays the performance of Decentralized Adam algorithm.
As expected, the performance of
DADAM is not as good as DGD or decentralized AMSGrad.
Its divergence characteristic, highlighted
Section 2.3, coupled with the heterogeneity in the data amplify its non-convergence issue in our experiments.
From the experiments above, we can see the advantages of decentralized AMSGrad in terms of both performance and ease of parameter tuning, and the importance of ensuring the theoretical convergence of any newly proposed methods in the presented setting.
[Figure]
t r a i n i
[Figure] [Figure]
[Figure]
t r a i n i ng l o ss
[Figure]