C OMMUNICATE T HEN A DAPT : A N E FFECTIVE D ECENTRALIZED A DAPTIVE M ETHOD FOR D EEP T RAINING
Anonymous authors
Paper under double-blind review
A BSTRACT
Decentralized adaptive gradient methods, in which each node averages only with its neighbors, are critical to save communication and wall-clock training time in deep learning tasks.
While different in concrete recursions, existing decentralized adaptive methods share the same algorithm structure: each node scales its gradient with information of the past squared gradients (which is referred to as the adaptive step) before or while it communicates with neighbors.
In this paper, we identify the limitation of such adapt-then/while-communicate structure: it will make the developed algorithms highly sensitive to data heterogeneity, and hence deviate their limiting points from the stationary solution.
To overcome this limitation, we propose an effective decentralized adaptive method with a communicate-thenadapt structure, in which each node conducts the adaptive step after ﬁnishing the neighborhood communications.
The new method is theoretically guaranteed to converge to the stationary solution in the non-convex scenario.
Experimental results on a variety of CV/NLP tasks show that the proposed algorithm has a clear superiority to other existing decentralized adaptive methods.
I NTRODUCTION
Decentralized SGD (Lopes & Sayed, 2008; Nedic & Ozdaglar, 2009; Chen & Sayed, 2012; Lian et al., 2017; Assran et al., 2019) is an emerging training approach for deep learning known for its much less communication overhead.
In contrast to parallel SGD in which a global averaging across all computing nodes is required per iteration, decentralized SGD does not involve any global operations.
Building upon partial averaging, in which each node only needs to compute the locally averaged model within its neighborhood, decentralized SGD can save remarkable communications and training time in large-scale distributed deep learning tasks compared to parallel SGD.
Although simple to use, the vanilla decentralized SGD sometimes suffers from the slow convergence.
Inspired by the well-documented success of adaptive methods such as AdaGrad (Duchi et al., 2011a), Adam (Kingma & Ba, 2014) and AMSGrad (Reddi et al., 2019), several decentralized adaptive methods (Nazari et al., 2019; Lin et al., 2021) have been proposed to accelerate decentralized SGD training.
While these algorithms have achieved remarkable success in several practical applications, they have also been observed to not converge to the desired solution (i.e., global optimal solution in the convex scenario or stationary solution in the non-convex scenario) in some other settings.
For example, it has been observed in the convex setting (see Sec. 3) that DAdam (Nazari et al., 2019) and QG-DAdam (Lin et al., 2021) do not converge to the global optimal solution.
This paper studies this situation in detail.
We rigorously uncover the reason why DAdam and QGDAdam fail to achieve the desired solution, and propose a novel decentralized adaptive method to resolve the convergence issue.
In particular, we make the following key contributions:
• We ﬁnd the algorithms DAdam and QG-DAdam, while different in concrete recursions, share a similar structure: each node scales its gradient with the past squared gradients (which is referred to as the adaptive step) before or while it communicates with neighbors.
We identify the limitation of such adapt-then/while-communicate structure: it will make the developed algorithms highly sensitive to data heterogeneity, and hence deviate their limiting points from the desired solution.
• To overcome these limitations, we propose a novel communicate-then-adapt algorithm structure, in which each node will conduct the adaptive step after all neighborhood communications.
It is not just trivially switching the order of communication step and adaptive step.
The key component to guarantee the effectiveness of the communicate-then-adapt structure is the utilization of the
decentralized augmented gradient (DAG) rather than the standard stochastic gradient in algorithm development.
The newly proposed algorithm, which is coined as DAG-Adam, can provably converge to the desired solution.
While this paper mainly focuses on the vanilla Adam algorithm, the core idea behind DAG-Adam can be easilty extended to AMSGrad or other adaptive methods.
• Experimental results on a variety of computer vision and natural language processing tasks show that DAG-Adam outperforms various existing state-of-the-art decentralized adaptive baselines under different network and data conﬁgurations.
Furthermore, the performance of our proposed algorithm is persistently competitive to the centralized counterpart.
Related work on decentralized optimization and deep training.
Decentralized optimization was extensively studied in the control and signal processing community.
The ﬁrst decentralized algorithms on general optimization problems include decentralized gradient descent (Nedic & Ozdaglar, 2009), diffusion (Lopes & Sayed, 2008; Chen & Sayed, 2012; Sayed, 2014), and dual averaging (Duchi et al., 2011b).
After that, various primal-dual algorithms come out to further speed up the convergence, and they are based on alternating direction method of multipliers (Adam) (Shi et al., 2014), explicit bias-correction (Shi et al., 2015; Yuan et al., 2019; Li et al., 2019), gradient tracking (Xu et al., 2015; Di Lorenzo & Scutari, 2016; Nedic et al., 2017; Qu & Li, 2018;
Lu et al., 2019), and dual acceleration (Scaman et al., 2017; Uribe et al., 2020).
In deep learning tasks, decentralize SGD, which was established in (Lian et al., 2017) to achieve the same linear speedup as parallel SGD in convergence rate, has attracted a lot of attentions.
Many efforts have been made to extend the algorithm to directed topologies (Assran et al., 2019), time-varying topologies (Koloskova et al., 2020), asynchronous settings (Lian et al., 2018), and data-heterogeneous scenarios (Tang et al., 2018; Xin et al., 2020; Lin et al., 2021; Yuan et al., 2021).
With careful consensus control (Kong et al., 2021) or periodic global averaging (Chen et al., 2021b), decentralize SGD can achieve 1 .
3 ∼ 2 × training time speedup without severe performance degradation.
Techniques such as quantization/compression (Alistarh et al., 2017; Bernstein et al., 2018; Koloskova et al., 2019a;b; Tang et al., 2019; Liu et al., 2020), periodic updates (Stich, 2019;
Koloskova et al., 2020; Yu et al., 2019), and lazy communication (Chen et al., 2018a; Liu et al., 2019b) were also integrated into decentralized SGD to further reduce communication overheads.
There are also a few studies on accelerated variants of decentralized SGD, and most of them are on (static) momentum acceleration.
(Assran et al., 2019; Gao & Huang, 2020) propose to run a local momentum SGD step ﬁrst before the partial averaging is conducted.
Another work (Yu et al., 2019) imposes an additional partial averaging over momentum to increase stability.
Recent works (Lin et al., 2021; Yuan et al., 2021) developed strategies that can remove the momentum-incurred bias in decentralized momentum SGD.
All these methods are not with adaptive strategies to scale gradients.
Related work on adaptive gradient method.
Adaptive gradient methods, with AdaGrad (Duchi et al., 2011a) and Adam (Kingma & Ba, 2014) as two representatives, have shown strong performance in training deep neural networks.
With adaptive adjustment in the gradient direction and automatic tune in the learning rate, adaptive gradient methods can boost the performance of SGD training signiﬁcantly.
In spite of its remarkable empirical success, Adam suffers from a convergence issue: it may not converge to the desired solution with a ﬁxed mini-batch size (Reddi et al., 2019; Zaheer et al., 2018).
Many algorithms have been proposed to resolve this issue.
AMSGrad (Reddi et al., 2019) preserves the long-term memory of past gradients to improve convergence, and (Zaheer et al., 2018) suggests the usage of increasing mini-batch sizes in convergence guarantees.
(Chen et al., 2018b) has studied the convergence of a family of Adam-type algorithms.
(Zhou et al., 2018) provides a better convergence rate for AMSGrad.
From the empirical side, RAdam (Liu et al., 2019a) and AdamW (Loshchilov & Hutter, 2018) can signiﬁcantly improve Adam performance.
The exploration in decentralized adaptive methods are rather limited.
DAdam (Nazari et al., 2019) is the ﬁrst consensus-based adaptive methods for distributed optimization to our knowledge.
(Lin et al., 2021) proposes QG-DAdam, which utilizes quasi-global momentum to locally approximates the globally descent direction.
A recent work (Chen et al., 2021a) proposes a uniﬁed framework that incorporates various adaptive methods into decentralized setting.
While these methods have shown strong empirical performance in several practical applications, they either suffer from unstable convergence or heavy communications.
We leave more discussion on their limitations in Sec. 2.
Adaptive gradient methods have also been extended to the federated learning setting in which multiple clients cooperate to learn a model under the supervision of a central server (McMahan et al., 2017).
Useful references in this direction can be found in (Xie et al., 2019; Reddi et al., 2020).
A DAPT - THEN / WHILE - COMMUNICATE STRUCTURE AND ITS LIMITATION
Problem.
Suppose n computing nodes cooperate to solve the distributed optimization problem:
[Equation]
In the above problem, f i ( x ) is local to node i , and random variable ξ i denotes the local data that follows distribution D i .
We do not assume each distribution D
i is the same across all nodes.
Network topology and weights.
We assume all computing nodes are connected by a network topology.
We deﬁne w ij , the weight to scale information ﬂowing from node j to node i , as follows:
[Equation]
N i :=
{ j | w ij > 0 } is deﬁned as the set of neighbors of node i which also includes node i itself and the weight matrix W :=
[ w ij ] ni,j =1 ∈ R n × n are denoted as a matrix that stacks the weights of all nodes.
This matrix W characterizes the sparsity and connectivity of the underlying topology.
Partial averaging.
Decentralized methods are based on partial averaging within neighborhood deﬁned by the network topology.
With weights { w ij } and the set of neighbors N i , the partial averaging operation of node i can be expressed as
[Equation]
Partial averaging requires much less communication than global averaging on sparse topologies.
Assumptions.
We will make the following standard assumptions throughout the paper:
A.1 [S MOOTHNESS ]
Each f i ( x ) is L -smooth, i.e., (cid:107)∇ f i ( x ) −∇ f i ( y ) (cid:107)
≤ L (cid:107) x − y (cid:107) for any x, y ∈ R d .
A.2 [G RADIENT NOISE ]
The random sample ξ (
t ) i is independent of each other for any k and i .
We also assume E [ ∇ F ( x ; ξ i )]
= ∇ f i ( x ) and E (cid:107)∇ F ( x ;
ξ i ) − ∇ f i ( x ) (cid:107) 2 ≤ σ 2 .
A.3 [W EIGHT MATRIX ]
The weight matrix W is symmetric and doubly-stochastic, i.e. W 1 =and 1 T W = 1 T , where 1 is an all-ones vector.
Further, we assume (cid:107) W − 1 n 11 T (cid:107) 2 ≤ ρ ∈ (0 , 1) .
A.4 [B OUNDED GRADIENT ]
The loss function F has bounded gradient, i.e., the maximum norm (cid:107)∇ F ( x ; ξ i ) (cid:107) ∞ ≤ G for all x and ξ i , and i ∈ [ n ] .
Notation.
Given a constant n , we let [ n ] := { 1 , · · · , n } .
Given a vector x ∈ R d , we let diag( x ) =
diag { x 1 , · · · , x d } ∈ R d × d be a diagonal matrix.
Given two vectors x ∈ R d and y ∈ R d , we let x (cid:12) y be the element-wise product between x and y .
2. T HE ADAPT - WHILE - COMMUNICATE STRUCTURE AND ITS LIMITATION
Adapt-While-Communicate Structure.
Algorithm 2 in Appendix B lists recursions for DAdam (Nazari et al., 2019).
The critical step in DAdam, which is highlighted in Algorithm 2, is as follows
[Equation]
where m ( t ) i is the momentum, ˆ v ( t ) i is an estimate of ∇ F ( x ( t ) i ; ξ ( t ) i ) (cid:12)∇ F ( x ( t ) i ; ξ ( t ) i ) , and (cid:15) is a small positive constant.
Their updates can be referred to Algorithm 2.
In recursion (4), it is observed that node i in DAdam conducts the adaptive step (i.e., scales the gradient by the square root of exponential moving averages of past squared gradients) while it communicates with neighbors.
We call such structure utilized in DAdam as adapt-while-communicate .
Fixed point condition.
Now we examine the ﬁxed point achieved by DAdam and verify whether it satisﬁes the optimal condition to problem (1).
To this end, we let x ( t ) i → x ∞ i as t → ∞ for i ∈ [ n ] .
To remove the the inﬂuence of gradient noise, we assume each node i can access the fullbatch gradient ∇ f i ( x ) .
If we deﬁne H i := diag( (cid:112) ∇ f i ( x ∞ i ) (cid:12) ∇ f i ( x ∞ i ) ) to be a diagonal matrix associated with node i , the ﬁxed point condition for DAdam is (see derivation in Appendix B.2):
[Equation]
Fixed point in DAdam is deviated from the desired solution.
In the data-heterogeneous scenario where data sample ξ i follows different distribution D i , it holds that f i ( x ) (cid:54) =
f j ( x ) .
If we further assume ∇ f i ( x ) (cid:54) = ∇ f j ( x ) , it follows that H i (cid:54) =
H j , for any i (cid:54) =
j .
With this fact, we conclude that DAdam ﬁxed point satisfying condition (6) does not necessarily satisfy the optimality condition for problem (1),
i.e., 1 n (cid:80) ni
=1 ∇ f i ( x ∞ ) = 0 .
This implies the ﬁxed point of DAdam is not the stationary solution to problem (1).
In particular, the following proposition derives the distance between the DAdam ﬁxed point and global optimal solution to problem (1) when the cost function is strongly-convex (Proof is in Appendix B.3).
Proposition 1 Under Assumptions A.1, A.3 and A.4, if each f i ( x ) is further assumed to be µstrongly-convex, and the full-batch gradient ∇ f i ( x ) is accessed per iteration, then DAdam (Algorithm 2) cannot converge to x (cid:63) , i.e., the global solution to problem (1).
In particular, the distance between DAdam limiting point and x (cid:63) can be characterized as follows:
[Equation]
where B := (cid:107) 1 n (cid:80) ni
=
( H i + (cid:15)I ) − 1 ∇ f i ( x ∞ i ) − 1 n (cid:80) ni
=1 ∇ f i ( x ∞ i ) (cid:107) (cid:54) = 0 is a constant bias term.
Furthermore, the B 2 term in (7) is necessary and cannot be removed by an improved analysis; there exists strongly convex and smooth functions
f i ( x ) such that lim t →∞ (cid:80) ni =1 (cid:107) x
( t ) i − x (cid:63) (cid:107)
= Ω( B 2 ) .
It is observed in Proposition 1 that the distance between the DAdam ﬁxed point and the optimal solution cannot diminish to zero as learning rate γ → 0 .
Such non-vanishing limiting bias Bis caused by the adapt-while-communicate structure in DAdam.
It is worth noting that a recent work (Chen et al., 2021a) has pointed out that DAdam cannot converge to the stationary solution by constructing a counter-example.
However, (Chen et al., 2021a) does not clarify the reason why
DAdam fails to converge to the stationary solution, and how far it can be from the desired solution.
2. T HE ADAPT - THEN - COMMUNICATE STRUCTURE AND ITS LIMITATION
Adapt-Then-Communicate Structure.
Algorithm 3 in Appendix B lists the recursions for QGDAdam (Lin et al., 2021).
The critical step, which is highlighted in Algorithm 3, is as follows
[Equation]
It is observed that node i in the above recursion ﬁrst conducts the adaptive step, and then it communicates with neighbors.
We call the structure utilized in QG-DAdam as adapt-then-communicate .
Fixed point condition and limitation.
Following the settings and arguments in Sec. 2.1, we can derive the ﬁxed point condition for QG-DAdam as follows (see the derivation in Appendix B.5):
[Equation]
where β 1 , β 2 ∈ (0 , 1) are momentum parameters listed in Algorithm 3.
Similar to DAdam, the above condition implies that the ﬁxed point of QG-DAdam is also deviated from the stationary solution.
In particular, the distance between QG-DAdam ﬁxed point and the global solution x (cid:63) can be characterized for the strongly-convex scenario as follows (proof is in Appendix B.6):
Proposition 2 Under the same conditions as in Proposition 1, the distance between QG-DAdam ﬁxed point and the global solution x (cid:63) can be characterized as follows:
[Equation]
) where B := (cid:107) 1 n (cid:80) ni
=1 ( √ 1 − β 2 H
i + (cid:15)I ) − 1 ∇ f i ( x ∞ i ) − 1 n (cid:80) ni
=1 ∇ f i ( x ∞ i ) (cid:107) (cid:54) = 0 is a constant bias.
Furthermore, the B 2 term in (7) is necessary and cannot be removed by an improved analysis; there exists strongly convex and smooth functions
f i ( x ) such that lim t
[Equation]
= Ω
[Equation]
B
[Equation]
Similar to DAdam, it is observed that the distance between the QG-DAdam ﬁxed point and the optimal solution cannot diminish to zero as learning rate γ → 0 .
2. GT-DA DAM : A DECENTRALIZED ADAPTIVE METHOD BASED ON GRADIENT TRACKING
It is observed from the ﬁxed-point condition (6) that DAdam cannot converge to the stationary solution due to the discrepancy between H i ’s.
A recent work (Chen et al., 2021a) proposes a gradient tracking based DAdam that enables each local different H i to converge to ¯ H = 1 n (cid:80) ni
=1 H i in theory, and therefore, its limiting ﬁxed point satisﬁes 1 n (cid:80) ni
=1 ∇ f i ( x ∞ ) = 0 , the optimality condition to problem (1).
GT-DAdam also utilizes the adapt-while-communicate structure.
While it can converge to the stationary solution to problem (1) as learning rate γ → 0 , it incurs twice amount of the communication overhead as DAdam.
The distance between GT-DAdam ﬁxed point and the global solution x (cid:63) can be characterized in Proposition 4 in Appendix B.7 for the strongly-convex scenario.
In the convex scenario, the distance between GT-DAdam ﬁxed point and the global solution can be worse than that of the proposed DAG-Adam, see Table 5 in Appendix C.3 and Fig.1 in Sec. 3.
D ECENTRALIZED A UGMENTED G RADIENT A DAM
Now that we understand that the adapt-while/then-communicate structure in decentralized adaptive methods either deviates the limiting point from the desired solution (e.g., DAdam and QG-DAdam), or incurs more expensive communication overhead (GT-DAdam), this section will propose a new algorithm that can overcome the limitation of the adapt-while/then-communicate structure.
Decentralized SGD interprets as standard SGD.
Without loss of generality, we assume each node i can access the accurate gradient ∇ f i ( x ) in decentralized SGD recursions.
In this scenario, decentralized SGD, according to (Yuan et al., 2016), can be written as
[Equation] [Equation]
In other words, decentralized SGD (12) can be regarded as a standard SGD recursion (13) to solve
[Equation]
n (cid:88) i =
(cid:107) x
i (cid:107) 2 − n (cid:88) i = n (cid:88)
j = w ij x i x j (cid:17) .
We call above as an augmented loss function, which is an approximate penalty problem to the target problem (1).
The term (cid:80) ni =1 (cid:107) x i (cid:107) 2 − (cid:80) i,j w ij x i x j is convex when the weight matrix W
=
[ w ij ] is symmetric and doubly-stochastic, and it equals 0 if x 1 = · · · = x n .
Apparently, it is a regularization term that promotes consensus among all nodes.
Decentralized Augmented Gradient Adam (DAG-Adam).
Inspired by the fact that decentralized
SGD can be interpreted as a standard SGD with an augmented gradient (13), we can easily integrate the augmented gradient to the standard Adam method (Kingma & Ba, 2014).
Decentralized
Augmented Gradient Adam, or DAG-Adam for short is listed in Algorithm 1.
It is observed that all communications occur when the stochastic augmented gradient is computed (see the highlighted part in Algorithm 1).
After that, each node i will conduct an adaptive step, which is untangled with communication.
We call the structure utilized in DAG-Adam as communicate-then-adapt .
This adjustment of the algorithm structure is crucial.
The stochastic gradient that used to compute m ( t ) i and
Initialize x (0) i arbitrarily; let β 1 , β 2 , ∈ [0 , 1) ; let m (0) i
= v (
[Equation]
i = 0 ; set γ, ν properly;
For t = 0 , 1 , 2 , ..., T − 1 , every node i do in parallel
[Equation]
v ( t ) i have incorporated the neighbor’s information, which will correct the deviation in ﬁxed point that introduced by different v i (and H i ).
[Figure]
Note that a parameter ν > 0 is added to the adjust the priority to gradient descent, or to promote consensus.
Empirically, we can set ν = 1 for the convex scenario.
But for the non-convex deep learning problems, the choice of ν will heavily inﬂuence both convergence rate and performance.
We extensively discuss the inﬂuence of ν in Appendix D and empirically examine it in the later experiment section.
While this paper we only focus on the extension of the vanilla Adam method, the core ideas such as the augmented gradient and the communicate-then-adapt structure can be extended to AMSGrad or other adaptive gradient methods.
Fixed point condition.
The stochastic augmented gradient and communicate-then-adapt structure utilized in DAGAdam can remove the inﬂuence of the discrepancy in v ∞ i and hence H i .
Following the settings and arguments in Sec. 2.1, we can derive the ﬁxed point condition for DAG-Adam as follows (see the derivation in Appendix C):
[Equation]
Note that the ﬁxed-point condition (15) for DAG-Adam, when ν = 1 , is exactly the same as that for vanilla decentralized SGD with full-bath gradient (Yuan et al., 2016).
Relation (16) implies that DAG-Adam will converge to the stationary solution to problem (1).
The following proposition characterizes the distance between the DAG-Adam ﬁxed point and the global optimal solution x (cid:63) for the strongly-convex scenario.
Proposition 3 Under the same conditions as in Proposition 1, the distance between DAG-Adam ﬁxed point and x (cid:63) can be characterized as follows:
[Equation]
It is observed in (17) that the distance is on the order of O ( γ 2 ) .
Apparently, DAG-Adam will converge to the global optimal solution when γ → 0 .
Moreover, DAG-Adam has the same communication overhead as in DAdam, and it is more communication-efﬁcient than GT-DAdam.
Propositionalso implies that DAG-Adam has the same limiting bias as decentralized SGD when ν = 1 .
Limiting point justiﬁcation.
We validate the limiting point analysis for DAdam, QG-DAdam, GTDAdam, and DAG-Adam on the full-batch logistic regression problem.
Table 5 in Appendix C. lists the results we establish in Propositions 1 – 4.
When ν = 1 in the DAG-DAdam method, which is a valid choice in convex settings in numerical experiments, the ﬁxed point of DAG-DAdam is closet to the global solution x (cid:63) when the cost function is strongly-convex.
The learning curves of
DAdam, GT-DAdam, QG-DAdam, and DAG-Adam are shown in Fig.
The MSE metric in the y -axis denotes 1 n (cid:80) ni
=1 (cid:107) x
( t ) i − x (cid:63) (cid:107) 2 .
From the ﬁgure, it is observed that the DAG-Adam algorithm can converge as fast as other decentralized adaptive algorithms but to a more accurate limiting point.
This limiting point is exactly the same as vanilla decentralized SGD, which is consistent with our claim in the limiting point analysis.
More experimental setting and results refer to Appendix C.4.
Convergence Analysis.
We provide the convergence analysis for our proposed DAG-Adam for non-convex functions.
The full proof can be found in Appendix E.
Theorem 1 Suppose the parameters γ , ν , and β 2 satisfy the conditions that γ ≤ (cid:15) − 2 ν 2 L , ν < min { (cid:15)/ 2 , κγ } , ( κ is any positive constant) and 1 − β 2 ≤
[Equation]
2 16( G + κC ) 2 .
Under Assumptions A.1A.4 and the augmented gradient shown is bounded, the decentralized augmented gradient generated by DAG-
Adam algorithm satisﬁes that
[Equation]
where the augmented loss function L ( x ) is the one deﬁned in (14), L (cid:63) is the minimum of the augmented loss.
Further, it holds for the globally averaged iterate ¯ x ( t ) that
[Equation]
(19) where ¯ x ( t ) =
1 /n (cid:80) ni
=1 x ( t ) i and ρ is the second largest eigenvalue of W in magnitude.
Remark
The theorem stated two conclusions.
The ﬁrst one is that the augmented gradient in
DAG-Adam will converge to a neighborhood around 0 with constant learning rate γ , and the neighborhood, when T → ∞ , is on the order of O ( σ 2 ) .
The second one is that DAG-Adam will converge to a neighborhood around the stationary solution in terms of the globally averaged iterate ¯ x ( t ) .
The neighborhood, when learning rate γ =
O (1 / √ T ) and T → ∞ , will be on the order of O ( σ 2 ) .
It is thus observed that DAG-Adam will converge to the exact stationary solution with decaying learning rate and increasing mini-batch sizes, which is consistent with the results of Adam using the standard stochastic gradient in (Zaheer et al., 2018).
Remark 2 The non-vanishing term O ( σ 2 ) in (18) and (19) is caused by the vanilla Adam algorithm structure, not the decentralized communication and update in DAG-Adam.
Such term can be corrected by using AMSGrad rather than Adam in DAG-Adam.
When there is no stochastic gradient noise, the augmented gradient will converge to zero.
In consequence, the v ( t ) i will converge to zero over all agents as well.
E XPERIMENTS
In this section, we systematically compare the proposed method, DAG-Adam, with all previous mentioned state-of-the-art decentralized adaptive methods on typical CV and NLP tasks.
Implementation Details.
We implement all the aforementioned algorithms with PyTorch 1.8.
(Paszke et al.
using NCCL 2.9.6 (CUDA 10.2) as the communication backend.
For PAdam, we used PyTorch’s native Distributed Data Parallel (DDP) module.
For the implementation of decentralized methods, we utilize BlueFog.
See Appendix F for more details.
4. I MAGE C LASSIFICATION : P ERFORMANCE OVER VARIOUS CONFIGURATIONS
A series of experiments are carried out with CIFAR-10 (Krizhevsky et al.
and ImageNet (Deng et al.
datasets to evaluate the performance of the proposed DAG-Adam on both homogeneous and heterogeneous dataset over different sizes of network.
We utilize ResNet-20 on
CIFAR-10 and ResNet-50 on ImageNet (He et al., 2016).
Training conﬁguration.
All methods are trained using 90 epochs with mini-batch of size 32 in each computing node.
PAdam uses a learning rate of 0.0125, while the ones for decentralized algorithms are scaled by the network size.
In addition, QG-Adam (Lin et al.
(2021)) uses a learning ratetimes smaller than other decentralized methods to avoid divergence 1 .
During the training procedure,
[Table]
the learning rate is warmed up in the ﬁrst 5 epochs and is then decayed by a factor of 10 at 30, 60 and 80 epochs.
The weight decay is chosen as 5 × 10 − 5 .
We also compared the results of the proposed
DAG-Adam with various ν choices
[Table]
Heterogenerous data conﬁguration.
To simulate the heterogenerous data, the disjoint non-i.i.d.
training data over agents are generated through Dirichlet distribution Dir( α ) , where α stands for the degree of the data heterogeneity (Lin et al., 2021; Yurochkin et al., 2019).
The training data for a particular class tends to concentrate in a single node as α → 0 , i.e. becoming more heterogenous while the homogeneous data is formed as α → ∞ .
See Appendix F for more details.
Performance comparison.
Table 1 compares the proposed DAG-Adam with other methods over networks with both homogeneous data distribution and heterogeneous ones.
Two different network sizes (4 and 8) are tested using ring topology.
With a proper choice of ν , the proposed method consistently outperforms all other methods under comparison, including the centralized optimization method PAdam, no matter what the data distribution is given.
The left and middle ﬁgures in Fig.shows how the test loss and test accuracy evolves in a heterogeneous data distribution with α = 10 .
DAG-Adam (with ν = 1 e − 1 ) converges faster than any other decentralized algorithms.
As typical decentralized algorithm behaves, it was relatively slower compared with centralized PAdam algorithm in the beginning, but it eventually caught up when the algorithm converged.
The right ﬁgure in Fig.
2 plots the consensus error (cid:80) i (cid:107) x
i − ¯ x (cid:107) , which validates that the consensus error in
Theorem 1 is proportional to the choice of ν .
{+80 Epochs 10 
+}
L o ss    80 Epochs 0.
A cc u r a c y
PAdam DAdam
QG-DAdam GT-DAdam
{+DAG-Adam (1e-3) DAG-Adam (1e-2) DAG-Adam (1e-1)    80 Epochs 10 − 
 C o n c e n s u s E rr o r+} [Figure]
10 0 ν 0.
A cc u r a c y α IID 1e+01 3e+ 1e+00 3e1e-01 3e1e-02 3e1e- ν
H e t e r o g e n e o u
s D e g
r
ee
α 62%68%77%80%83%86%88%90%92%91%90%
Sensitivity on ν constant.
As shown in Table 1, the ﬁnal classiﬁcation performance depends on the choice of constant ν in DAG-Adam.
To better understand how ν affects the DAG-Adam performance, we performed a corresponding ablation study with varying heterogeneity degree α as shown in Fig.
For varying heterogeneity degree α , the constant ν that achieving best performance lies in the range of [1 e − 2 , 5 e − 1 ] for CIFAR-10 typically.
Performance over different network scales.
We further examine the performance of the proposed algorithm over different network sizes to see how good it can scale up in large decentralized network case.
As shown in Table 2, under the exponential graph (Assran et al., 2019) and ﬁne-tuning of ν ,
DAG-Adam persistently achieves comparable performance with the centralized method until the node size reaches n = 64 .
Even under that case, the drop of performance is quite small.
[Table]
ImageNet results.
We also validate the performance of our proposed method on a relatively largescale dataset ImageNet.
We use PAdam to tune the learning rate ( 4 × 10 − 4 ) and wight decay ( 1 × 10 − 4 ) hyper-parameters to reach a strong baseline.
Then, the same hyper-parameters are applied for decentralized counterparts.
In the experiment, the algorithms run on 8 nodes (i.e. 8 × 8 =
GPUs) and train total 100 epochs with a cosine decay scheduler.
Our proposed method reaches topaccuracy 75.96 % which is slightly better than PAdam.
[Table]
[Table]
L ANGUAGE M ODELING : P ERFORMANCE OVER DIFFERENT TASKS
BERT (Devlin et al.
(2018)) is a widely used pre-training language representation model for NLP tasks.
We ﬁne-tuned pretrained BERT models (Base/Large: 110/330M parameters) on the SQuAD (Rajpurkar et al. (2016)) dataset.
All experiments are based on NVIDIA BERT 2 implementation with mixed precision support.
We examine the task over both static exponential graph and dynamic exponential graph setting (Assran et al., 2019).
As shown in Table 4, with proper tuned hyperparameter ν , our proposed method reached best performance in both metrics
[Table]
[Table]
[Table]
C ONCLUSION AND FUTURE WORKS
In this paper, we identiﬁed the limitation of adapt-while/then-communicate structure utilized in existing adaptive gradient methods: the limiting discrepancy in the squared local gradient term (i.e., v ∞ i ) can deviate the limiting point from the desired stationary point.
To ﬁx this issue, we proposed a novel DAG-Adam algorithm that is built upon the communicate-then-adapt structure instead.
DAGAdam can provably converge to the desired solution even with heterogeneous data.
Experimental results on a variety of deep learning tasks show that DAG-Adam persistently outperforms existing decentralized adaptive baselines.
However, it is also observed in these experiments that DAG-Adam is highly-sensitive to the hyper-parameter ν .
How to tune ν automatically during the training stage is a crucial topic of the future works.
A N OTATIONS AND P RELIMINARIES
Notations.
We introduce several notations as follows:
[Equation]
(cid:98) f
[Equation]
x )
[Equation]
• W =
[Equation]
ij
[Equation]
∈ R n × n .
• W (cid:44) W ⊗ I d ∈ R nd × nd , where ⊗ means the Kronecker product.
• 1 n = col { 1 , 1 , . .
, 1 } ∈ R n .
• ¯ x = ( 1 n 1 n 1 Tn ⊗ I d ) x ∈ R nd × 1 .
• Given a vector x ∈ R d , we let diag( x ) =
diag { x 1 , · · · , x d } ∈ R d × d be a diagonal matrix.
• Given a positive deﬁnite matrix A , we deﬁne the weighted normal (cid:107) x (cid:107)
A = x T Ax , (cid:107) x (cid:107) as vector (cid:96) 2 norm and (cid:107) A (cid:107) as matrix (cid:96) 2 norm.
Algorithm reformulation.
With the above notation
[Equation]
we rewrite DAG-Adam in Algorithm 1 into the following stacked vector form:
[Equation] [Equation]
[Equation]
where H ( t ) is the diagonal matrix deﬁned as H ( t )
=
Diag( √ v ( t ) ) .
Smoothness.
Since each f i ( x ) is assumed to be L -smooth in Assumption A.1, it holds that f ( x ) =n (cid:80) ni
=
f i ( x ) is also L -smooth.
As a result, the following inequality holds for any x , y ∈ R d :
[Equation]
Algorithm
(Nazari et al., 2019)
Initialize x (0) i arbitrarily; let β 1 , β 2 , β 3 ∈ (0 , 1) ; let m (0)
i
[Equation]
0 ; set γ properly;
For t = 0 , 1 , 2 , ..., T − 1 , every node i do
Sample g ( t ) i
[Equation]
Weighted norm inequality.
In this paper, the weighted norm (cid:107) x (cid:107) 2 A only considers the scenario in which A is a diagonal matrix.
Suppose the diagonal elements in the diagonal matrix
A satisfy that 0 < a ≤ [ A ] i,i ≤ b for all i , one useful inequality is
[Equation]
The proof is strait-forward:
[Equation]
We can use the same argument to show a (cid:107) x (cid:107) TA ≤ (cid:107) Ax (cid:107) 2 as well.
Communication topology.
Figure 4 provides a graph example to illustrate the decentralized topology and its communication pattern that is used in all decentralized methods listed in the paper.
B A DAPT - WHILE / THEN - COMMUNICATE STRUCTURE AND ITS LIMITATION B. DA DAM AND ADAPT - WHILE - COMMUNICATE STRUCTURE
The DAdam algorithm (Nazari et al., 2019) is listed in Algorithm 2.
B. F IXED POINT CONDITION FOR DA DAM
To see the ﬁxed point condition for DAdam, we assume the iterate x ( t ) i in DAdam converges to some ﬁxed point x ∞ i as k → ∞ for any i ∈ [ n ] .
In order to remove the inﬂuence of the gradient noise, we let g ( t ) i be the full-batch gradient ∇ f i ( x ( t ) i ) =
E [ ∇ F ( x ( t ) i ; ξ i )] for any i ∈ [ n ] .
Under this setting, it is easy to verify that
[Equation]
If we deﬁne H i := diag( (cid:112) ∇ f i ( x ∞ i ) (cid:12) ∇ f i ( x ∞ i ) ) to be a positive diagonal matrix, then the highlighted step in DAdam will become (when t → ∞ )
[Equation]
[ n ] .
(28) which is the ﬁxed-point condition listed in (5).
If we further let ¯ x ∞ := 1 n (cid:80) ni =1 x ∞ i , the following relation can be achieved by (28) and Assumption A.3:
[Equation]
The last relation is the one listed in (6).
B. P ROOF OF P ROPOSITION
Before we prove Proposition 1, we need to introduce several notations:
• x ∞ = col { x ∞ 1 , · · · , x ∞ n } ∈ R nd • ¯ x ∞ = col { ¯ x ∞ , · · · , ¯ x ∞ } ∈ R nd where ¯ x ∞ = 1 n (cid:80) ni
=1 x ∞ i .
• x (cid:63) =
col { x (cid:63) , · · · , x (cid:63) } ∈ R nd where x (cid:63) is the optimal solution to problem (1) when the cost function is convex • ∇ f ( x ∞ ) =
col {∇ f 1 ( x ∞ 1 ) , · · · , ∇ f n ( x ∞ n ) } ∈ R nd • ( H + (cid:15)I ) − 1 =
diag
( H 1 + (cid:15)I ) − 1 , · · · , ( H n + (cid:15)I ) − 1 }
With these notations, we can rewrite the ﬁxed-point condition (28) into a compact manner:
[Equation]
To examine (cid:107) x ∞ − x (cid:63) (cid:107) , we will evaluate (cid:107) x ∞ − ¯ x ∞ (cid:107) and (cid:107) ¯ x ∞ − x (cid:63) (cid:107) respectively.
Upper bound of (cid:107) ¯ x ∞ − x (cid:63) (cid:107) .
Left-multiplying 1 n 1 T to both sides of recursion (30), we achieve n
[Equation]
With the above fact, we have
[Equation]
where the last inequality holds because each f i ( x ) is µ -strongly convex and
L -smooth
[Equation]
and the deﬁnition of constant B as follows:
[Equation]
Inequality (32) leads to
[Equation]
Upper bound of (cid:107) x ∞ − ¯ x ∞ (cid:107) .
Subtracting ¯ x ∞ from both sides of (30), we have
[Equation]
With inequality (43) in (Yuan et al., 2021), we have
[Equation]
I
[Equation]
With (35) and (36), and noticing (cid:107) ( H + (cid:15)I ) − 1 (cid:107) =
max i {(cid:107)
( H i + (cid:15)I ) − 1 (cid:107) 2 } ≤ 1 (cid:15) , we achieve (cid:107)
[Equation]
(cid:107)
[Equation]
γ (cid:107)∇ f (
[Equation]
(cid:107) (
[Equation]
where the last inequality holds because of Assumption A.4.
Upper bound of lim t →∞ 1 n (cid:80) ni =1 (cid:107) x
( t ) i − x (cid:63) (cid:107) 2 .
With (34) and (37), we have
[Equation]
If we ignore the inﬂuence of µ and L , we ﬁnally achieve
[Equation]
Lower bound of lim t →∞
[Equation]
n (cid:80)
[Equation] [Equation]
x
[Equation]
t ) i − x
[Equation]
(cid:107) 2 .
Now we show that the terms depending on B 2 in the upper bound (39) are necessary and cannot be removed by an improved analysis.
In fact, there exists strongly convex and smooth functions such that x ( t )
i generated by DAdam satisﬁes
[Equation]
To prove the above conclusion, we consider minimizing problem of the form (1) with f
[Equation] [Equation]
where x, y i ∈ R .
Apparently, the solution to problem (1) is given by x (cid:63) = 1 n (cid:80) ni
=
y i .
Next we let x
∞
i be the limiting point of DAdam (Algorithm 2) in node i .
With notations in Sec.
B.2, we have
[Equation] [Equation]
Under the above setting, the limiting point of DAdam (30) reduces to
[Equation]
in which W ∈ R n × n is the weight matrix.
By left-multiplying 1 n 1 T to both sides, we have
[Equation]
On the other hand, it is easy to verify that
[Equation] [Equation]
With (46) and (47), we have
[Equation]
(cid:107) (46) =
(cid:107)n n (cid:88) i =
[Equation]
f i (
[Equation]
∞ i ) |
[Equation]
f i (
[Equation]
∞ i ) |
[Equation]
(cid:15)
[Equation]
n n (cid:88)
i = ∇ f i (¯ x
[Equation]
) (cid:107) = (cid:107) 1 n n (cid:88)
i = ∇ f i ( x ∞ i ) |∇
f i ( x ∞ i ) |
[Equation]
(cid:15)
[Equation]
n n
[Equation]
i
[Equation] [Equation]
n
[Equation]
i
[Equation] [Equation]
n
[Equation]
i
[Equation] [Equation]
Algorithm
Initialize x (0) i arbitrarily; let β 1 , β 2 ∈ (0 , 1) ; let m (0) i
=
ˆ
[Equation]
= v (0)
[Equation]
= ˆ v (0) i = 0 ; set γ properly;
For k = 0 , 1 , 2 , ..., T − 1 , every node i do
Sample g ( t ) i
[Equation]
; ξ ( t ) i ) ;
[Equation]
n
[Equation]
n
[Equation]
i =
[Equation]
f
[Equation] [Equation]
n (cid:88) i =
( x
[Equation]
i − ¯ x
[Equation]
) (cid:107) = (cid:107)
[Equation]
Finally
[Equation]
we have
[Equation]
where equality (a) holds because
[Equation]
Inequality (49) implies that the term B 2 cannot be removed from bound (39) with any improved analysis for our constructed problem.
B. QG-DA DAM AND ADAPT - THEN - COMMUNICATE STRUCTURE
The QG-DAdam algorithm (Lin et al., 2021) is listed in Algorithm 3.
B. F IXED POINT CONDITION FOR QG-DA DAM
We let x ( t ) i → x ∞ i as t → ∞ for any i ∈ [ n ] .
We also assume each node i can access the full-batch gradient ∇ f i ( x ) in order to remove the inﬂuence of gradient noise.
Under such setting, we can verify in QG-DAdam that
[Equation]
Substituting m ∞ i and v ∞ i to the highlighted step in QG-DAdam, we can achieve the ﬁxed-point condition as follows
[Equation]
where H i = diag { (cid:112) ∇ f i ( x ∞ i ) (cid:12) ∇ f i ( x ∞ i ) } .
If we further let ¯ x ∞ := 1 n (cid:80) ni =1 x ∞ i
[Equation]
the following relation can be achieved by (52) and Assumption A.3:
[Equation]
The last relation is the one listed in (10).
B. P ROOF OF P ROPOSITION
The proof of Proposition 2 is similar to that of Proposition 1.
With notations deﬁned in Sec.
B.3, we can rewrite (52) into the following compact manner:
[Equation]
Upper bound of (cid:107) ¯ x ∞ − x (cid:63) (cid:107) .
Left-multiplying 1 n 1 T to both sides of recursion (30), we achieve
[Equation]
[Equation]
Using a similar argument to that in (32) and (34), we have
[Equation]
[Equation]
in which the constant bias term B is deﬁned as
[Equation]
Upper bound of (cid:107) x ∞ − ¯ x ∞ (cid:107) .
Subtracting ¯ x ∞ from both sides of (54), we have
[Equation]
[Equation]
Using similar arguments as in (36) and (37), and facts that (cid:107) ( √ 1 − β 2 H
+ (cid:15)I ) − 1 (cid:107) =
max i {(cid:107) ( √ 1 − β 2 H i + (cid:15)I ) − 1 (cid:107) 2 } ≤ 1 (cid:15) and (cid:107)W(cid:107) ≤ 1 , we have
[Equation] [Equation]
Upper bound of lim t →∞ 1 n (cid:80) ni =1 (cid:107) x
( t ) i − x
[Equation]
(cid:107) 2 .
Using similar arguments to those in (38) and (39), we achieve
[Equation]
Lower bound of lim t →∞ 1 n (cid:80)
[Equation] [Equation]
−
[Equation]
With the same constructed problem as in (41), we can easily establish
[Equation] [Equation]
following the arguments in (42) – (50), where B is deﬁned in (57).
B.
GT-DA DAM
Fixed-point condition.
If we deﬁne ¯ H = 1 n (cid:80) ni =1 H i , the ﬁxed-point condition of GT-DAdam (i.e.,
Algorithm 2 in (Chen et al., 2021a)) is as follows:
[Equation]
It is observed that the ﬁxed-point of GT-DAdam satisﬁes the optimality condition of problem (1).
Proposition 4 Under the same conditions as in Proposition 1, the distance between GT-DAdam ﬁxed point and the global solution x (cid:63) can be characterized as follows:
[Equation]
Proof.
With notations deﬁned in Sec.
B.3, the ﬁxed-point condition in (62) can be rewritten as
[Equation]
(65) where ¯ H = diag { ¯ H, · · · , ¯ H } ∈ R nd × nd .
Bound of (cid:107) ¯ x ∞ − x (cid:63) (cid:107) .
Left-multiplying 1 n 1 T to both sides of recursion (65), we achieve
[Equation] [Equation]
With the above fact, we have
[Equation]
where the last inequality holds since each f i ( x ) is µ -strongly convex and L -smooth.
We thus have
[Equation]
Bound of (cid:107) x ∞ − ¯ x ∞ (cid:107) .
Subtracting ¯ x ∞ from both sides of (65), we have
[Equation]
With inequality (43) in (Yuan et al., 2021), we have
[Equation]
With (69) and (70), and noting that (cid:107) ( ¯ H + (cid:15)I ) − 1 (cid:107) 2 ≤ 1 /(cid:15) , we achieve
[Equation]
Proof of Proposition 4.
With (68) and (71), we have
[Equation]
The above inequality implies that
[Equation]
Remark 3 Different from Propositions 1 and 2, it is observed from Proposition 4 that the distance between the GT-DAdam ﬁxed point and the optimal solution will diminish to zero as learning rate γ → 0 .
However, GT-DAdam will incur two rounds of neighborhood communication per iteration, which is much more expensive than DAdam and QG-DAdam.
C F IXED - POINT CONDITION FOR DAG-A DAM C. F IXED - POINT CONDITION
When t → ∞ , we let x ( t ) i → x ∞ i for any i ∈ [ n ] in DAG-Adam.
We further assume each agent can access the real gradient ∇ f i ( x ) instead of the stochastic gradient ∇ F ( x ; ξ i ) .
Under this setting, we can verify that
[Equation]
74) and m ∞
i
=
ˆ g ∞ i , v ∞ i
= ˆ g ∞ i (cid:12) ˆ g ∞ i .
By letting ˆ H i = diag { (cid:112) v ∞ i } , it can be derived from the adptive step in DAG-Adam that
[Equation]
In other words, DAG-Adam can converge to the stationary solution if x ∞ 1 = · · · = x ∞ n = x ∞ in the limit.
Compared to DAdam and QG-DAdam in which the discrepancy between H i ’s has made the limiting ﬁxed point deviate from the desired solution, the communicate-then-adapt structure in
DAG-Adam make it immune to the difference between H i .
C. P ROOF OF P ROPOSITION 3.
With notations deﬁned in Sec.
B.3, the ﬁxed-point condition in (76) can be rewritten as
[Equation] [Equation]
Bound of (cid:107) ¯ x ∞ − x (cid:63) (cid:107) .
Left-multiplying 1 n 1 T to both sides of recursion (79)
, we achieve n n (cid:88)
i
[Equation] [Equation]
i (
[Equation]
[Table]
Using similar arguments to (67), we achieve
[Equation]
Bound of (cid:107) x ∞ − ¯ x ∞ (cid:107) .
Subtracting ¯ x ∞ from both sides of (79), we have
[Equation]
With inequality (43) in (Yuan et al., 2021), we have
[Equation]
With (82) and (83), we achieve
[Equation]
[Equation]
Proof of Proposition 3.
With (81) and (84), we have
[Equation]
The above inequality implies that
[Equation]
C. C OMPARISON IN LIMITING POINT DISTANCES TO GLOBAL SOLUTION
Table 5 lists the results we establish in Propositions 1 – 4.
When ν = 1 in the DAG-DAdam method, which is a valid choice for convex settings in empirical studies, the comparison between different decentralized adaptive methods in the limiting point distance to the global solution x (cid:63) can be listed as follows:
[Equation]
This conclusion matches with the numerical experiment in Fig.
C. E XPERIMENT SETTING FOR LOGISTIC REGRESSION ( I . E ., F IG 1)
In this experiment, we consider the binary logistic regression problem:
[Equation] [Equation]
where h i,k ∈ R
d is the k -th feature vector at agent i and γ i,k ∈ { +1 , − 1 } is the corresponding label.
In the experiment, we set λ = 0 .
25 , n = 16 , and all computing nodes are organized into the (undirected) ring topology.
The weighted matrix is simply generated through the average rule so that is satiﬁes Assumption A.3.
Also, we choose d = 25 as the dimension of feature vector and generate
K = 50 data for each agent locally.
The corresponding γ is generated through a pre-set random x o vector.
We ﬁrst compute the linear score
h Ti,k x o , then determine the γ by the probability based on the sigmoid value.
For all algorithms, we choose the constant learning rate γ = 0 .
01 , β 1 = 0 .
9 , β 2 = 0 .
One exception is that we choose β 1 = 0 .
1 for QG-DAdam algorithm to achieve better convergence performance.
Besides the learning curve we have shown in Fig.
1, we also plots the v ( t ) i of various algorithms at the last iteration in Fig.
To characterize the difference between different agents, we use the box plot to show the difference of v ( t ) i of each entry among all agents.
From the ﬁgure, it is clear that the v ( t ) i in DAG-Adam algorithm is almost zero when the algorithm converged, which is much narrower and lower than others methods.
[Figure] [Figure]
Fig.
6 provides complementary experimental results to Fig.
We added a new algorithm named
DAdam-ReParam which reparameterizes DAdam to obtain a communicate-then-adapt structure.
In addition, the convergence with both constant and decaying learning rates are depicted in the left and right plot in Fig. 6, respectively.
The It is observed that DAdam-ReParam converges closely to
DAdam, and both approaches converge to a less accurate solution than DAG-Adam.
D M ORE DISCUSSION ON THE HYPER - PARAMETER ν
This section discusses the role of hyper-parameter ν in DAG-Adam, and how it inﬂuences the convergence performance.
To this end, we ﬁrst discuss whether ν is needed in the vanilla decentralized
SGD (or DSGD for short) algorithm and DAdam.
Next, we will study the inﬂuence of ν in the more complicated DAG-Adam algorithm.
D. I NFLUENCE OF ν ON DSGD
Without loss of generality, we consider the DSGD method with a full-batch gradient.
In this setting, the augmented gradient of DSGD (13) is given by
[Equation]
With notations deﬁned in Appendix A and the introduction of the hyper-parameter ν as in DAGAdam, we can rewrite the above recursion into a more compact form
[Equation]
The above recursion implies that the hyper-parameter ν will replace the original weight matrix W with (cid:102) W , whose eigenvalues are derived as
[Equation]
where eig k ( W ) stands for the k -th eigenvalue of matrix W .
Equation (91) implies all eigenvalues of (cid:102) W are more closer to value 1 for any ν ∈ [0 , 1) .
Since a larger spectral gap, i.e. 1 − λ 2 ( (cid:102) W ) , will reduce the mixing time of gossip average and hence speed up the DSGD convergence, the optimal choice for DSGD is ν = 1 ; other ν values will not improve the convergence of the vanilla DSGD.
We validate the above discussion with deep learning experiments over CIFAR-10 dataset.
Fig.illustrates the inﬂuence of ν on vanilla DSGD.
It is observed that ν = 1 enables DSGD to converge fastest and achieve the best validation accuracy, which is consistent with the above conclusions.
80 Epochs
T r a i n
L o ss v=1.0e+00 v=1.0ev=1.0e-02 v=1.0e80 Epochs 0.
V a li d a t i o n
A cc u r a c y v=1.0e+00 v=1.0ev=1.0e-02 v=1.0e-
D. I NFLUENCE OF ν ON DA DAM
Since DAdam is an immediate variant of DSGD, we expect the inﬂuence of ν on DAdam will be similar to that on DSGD.
To validate it, we impose a hyper-parameter ν to DAdam recursion:
[Equation]
[Equation] [Equation]
Fig.
8 illustrates the inﬂuence of ν on DAdam.
Similar to DSGD, it is observed that DAdam with ν = 1 (i.e., the vanilla DAdam with no introduction of ν ) has the best convergence performance.
[Figure]
D. I NFLUENCE OF ν ON DAG-A DAM
Without loss of generality, we consider DAG-Adam with dimension d = 1 .
In this scenario, DAGAdam can be rewritten as
[Equation] [Equation]
where W ∈ R n × n is the weight matrix and H ( t )
= diag {√ v ( t ) } ∈ R n × n is a diagonal matrix.
Subtracting (99) by β 1 times (99) with one iteration shift, we have
[Equation]
Rearranging the terms in the above recursion, we obtain
[Equation]
In the following, we will ﬁrst study a simple example that can shed lights on the inﬂuence of ν on
DAG-Adam, and next we will discuss the more generalized scenario.
An illustrating example.
We consider a trivial scenario in which H ( t ) = αI , which amounts to the case where all nodes maintain constant v ( t ) i across i and t in the local buffer.
As we have shown in the convex experiment in Fig.
5, this trivial scenario becomes valid in the asymptotic stage of
DAG-Adam in which v ( t ) i converges to the same value v for any i ∈ [ n ] .
Substituting H ( t ) = αI into (101), we have
[Equation]
From the expression of (cid:102) W DAG , it is observed that the optimal value for ν to minimize the mixing time of the weight matrix is α + (cid:15) 1 − β 1 , which is not 1 as in the vanilla DSGD algorithm.
In addition, it is worth noting that (cid:102) W DAG cannot outperform W in terms of mixing time for any choice of ν .
This trivial example illustrates the necessity to have the hyper-parameter ν in DAG-Adam.
The introduction of ν is to compensate the negative inﬂuence of the communicate-then-adapt structure in DAG-Adam which incorporates both the gradient and the consensus constraint into the adaptive momentum buffer.
The weight matrix (cid:102) W DAG can be quite inefﬁcient to aggregate information if no ν exists to speed up partial averaging.
Generalized scenario.
Unlike the above trivial scenario, it is very challenging to determine a proper value for ν in the general scenarios.
To see it, we ignore the last term in (101) to achieve
[Equation]
which is a reasonable approximation to (101) since β 2 is typically set as 0 .
999 and v ( t ) (and hence
H ( t ) ) will not change dramatically.
The ﬁrst term in (103) mixes x ( t ) i with weight matrix
[Equation]
If we let Λ := ν (1 −
[Equation] [Equation]
H
[Equation]
t )
+
[Equation]
) − 1 , it holds that
[Equation] [Equation]
The left-hand side of (105) is the similarity transformation of (cid:102) W DAG .
Hence, the eigenvalues of (cid:102) W DAG equal to ones of the right-hand side of (105).
Unfortunately, it is difﬁcult to determine the eigenvalues of Λ 1 / 2 W Λ 1 / 2 , not to mention the optimal value for ν .
However, it is obvious that ν = 1 is not necessarily optimal for DAG-Adam.
This motivates us to tune ν in experiments to boost the convergence performance of DAG-Adam.
In fact, the results in Table 1 (with CIFARdataset), Table 3 (with ImageNet dataset), and Table 4 (with SQuAD dataset) corroborate that ν
=is not the optimal choice for DAG-Adam.
How to ﬁnd the optimal ν automatically is the main direction of our future work.
E C ONVERGENCE A NALYSIS E. C ONVERGENCE PROOF OF DAG-A DAM
As we motivated in the Section 3, we consider the augmented loss function
[Equation]
It is easy to verify that
[Equation]
which is exactly the same format as our decentralized augmented gradient in equation 20.
Proof idea.
Therefore, we can view our DAG-Adam over multiple agents as the standard Adam for virtual single agent with a long stacked vector.
This observation leads to the following convergence analysis idea.
First, we use the similar proof technique as Zaheer et al.
(2018) did for the standard Adam to show the convergence of the decentralized augmented gradient.
Next, based on that, we establish the consensus lemma to show that the models between all agents are close.
Last, combining two, we ﬁnally arrive the convergence of centralized iterate, (cid:107)∇ f (¯ x ( t ) )
(cid:107) as the common decentralized algorithm theorem stated.
It is common in the proof of adaptive gradient methods, for example Zaheer et al.
; Chen et al.
(2021a; 2018b); Reddi et al.
(2019; 2020), to assume the gradient ∇ f i ( x ( t ) i ) and/or stochastic gradient ∇ F i ( x ( t ) i ; ξ i ) have bounded (cid:96) 2 or (cid:96) ∞ norms at all iterations t , i.e., (cid:107)∇ f i ( x ( t ) i ))
(cid:107) ≤ G and (cid:107)∇ F i ( x ( t ) i ; ξ i ) (cid:107) ≤ G .
Hence, it is nature to see we need the similar bounded gradient assumption on the augmented gradient.
However, notice our augmented gradient has a ( I − W ) x term.
So we need to further assume that (cid:107) x
( t ) (cid:107) ≤ C .
This is a relatively strong assumption, but it can be strictly achieved by simply taking a projection step.
Also, in the practice, we always observe that the terms g ( t ) converges to a very small value without any requirement on the boundary on the iterate, as shown in Fig.
In this case, we can show that
[Equation] [Equation]
Same as the paper Zaheer et al.
(2018) did, we show the case that there is no-momentum for the sake of simplicity, i.e., β 1 = 0 , which is typically referred as RMSprop.
Lemma 1 (Convergence lemma of augmented gradient) Suppose the parameters γ, ν and βsatisfy the conditions that γ ≤ (cid:15) − 2 ν 2 L , ν < min { (cid:15)/ 2 , κγ } , ( κ is any positive constant) and 1 − β 2 ≤ (cid:15) 2 16( G + κC ) 2 .
Under the Assumption A.1 - A.4 and bounded augmented gradient, the augmented gradient generated using DAG-Adam has the following bound
[Equation]
where L ∗ is the minimum value of the augmented loss function (106).
Proof.
To shorten the notation, we denote the stochastic augmented gradient as
[Equation]
Utilizing the L -smooth property and noting that the augmented loss function L has a new Lipschitz constant L (cid:48)
=
L + νγ , we have
[Equation]
where the last equality we add and subtract the inner product with ( √ β 2 H ( t − 1) + (cid:15) ) − 1 (cid:98) L ( x ) .
The main reason to do this is observing that H ( t ) is correlated with the stochastic gradient ∇ (cid:98) L ( x ) .
Taking the conditional expectation E t of L ( x ( t +1) ) , more precisely, taking the expectation given all previous recursion information { x ( j ) , v ( j ) } tj =1 , we obtain
[Equation]
[Equation]
Here we can remove the gradient noise by conditional expectation since the weighted diagonal matrix √ β 2 H ( t − 1) is deterministic given the all information before iteration t and the zero gradientnoise assumption A.2.
To bound the term E t (cid:107) ( H ( t ) + (cid:15) ) − 1 ∇ x
[Equation]
L ( x ) (cid:107) 2 , we notice that any diagonal element of weighting matrix satisﬁes that
[Equation]
[Equation]
Apply the inequality (25), we establish that
[Equation]
[Equation]
where the second inequality is because that √ β 2 H ( t − 1) ≤ H ( t ) .
Next, we bound the cross terms:
[Equation]
where the last inequality is Holder’s inequality
[Equation]
Using the same methods as in (Zaheer et al., 2018,
Appendix A)
[Equation]
we can show that
[Equation]
Plugging back, we establish
[Equation]
Substituting (114) and (117) into (112), we obtain
[Equation]
Next
[Equation]
we bound the weighted norm of stochastic noisy gradient:
[Equation]
where the ﬁrst equality is because of independent and zero-mean gradient noise assumption and the ﬁrst inequality exploited the equation 25.
Plugging back and rearranging the terms
[Equation]
we get
[Equation] [Equation]
To make the coefﬁcient in the left-hand side of inequality positive
[Equation]
we require that
[Equation]
Hence
[Equation]
we conclude that by taking the telescoping sum:
[Equation]
where we use the fact that L ∗ ≤ L ( x ) for any x .
[Equation]
Above lemma shows that that local augmented gradient will converge to the stationary point within the constant factor of O ( σ 2 ) for constant learning rate γ , which is the same as stated in Theoremof Zaheer et al.
Next, we evaluate that how close the model between each agent.
Lemma 2 (Convergence of consensus value)
Under the same condition as Lemma 1, it holds that
[Equation]
where the ρ is the second largest eigenvalue of combination matrix W in magnitude, which is strictly less than 1.
Proof : Since W is a symmetric doubly stochastic matrix
[Equation]
there is an eigenvalue decomposition form
W = Q Λ Q T , where Q is some orthogonal matrix.
The diagonal matrix Λ has the value within the range (0 , 1] and contains one and only one entry that is 1 due to Assumption A.3.
Therefore, we can establish that
[Equation]
where the ﬁrst equality holds because ( I − W ) 1 = 0 .
Now
[Equation]
we are read to establish the bound for the consensus error from the augmented gradient:
[Equation] [Equation]
where the last inequality utilizes the Jensen’s inequality (cid:107) a (cid:107) 2 = (cid:107) a + b +( − b ) (cid:107) 2 ≤ 2 (cid:107) a + b (cid:107) 2 +2 (cid:107) b (cid:107)and by setting a = νγ ( I − W )
x ( t ) and b = ∇ f ( x ( t ) ) .
Using the conclusion from Lemma 1 and bounded gradient assumption, we can immediately establish that
[Equation]
which gives the desired bound that the consensus error has the order of O (cid:16)
[Equation]
(cid:4)
This bounds for the consensus error has several easy interpretable conclusions: To have the smaller consensus error, we can either 1) use a better connectivity of the graph (i.e., ρ is closer to 0), or 2) set a smaller learning rate γ , or 3) choose a larger ν .
More accurately, the consensus error is determined by the ratio between γ and ν , which is straight-forward to understand since the communication term ( I − W ) x is scaled by ν/γ .
With above two lemma, we are ready to present the convergence theorem.
Proof of theorem 1 .
The ﬁrst half of statement has already been shown in the lemma 1.
We just to show the convergence for the centralized iterate.
To connect the gradient over the centralized iterate ¯ x with the stacked gradient over the stacked iterate ¯ x , we have
[Equation]
where step (a) follows the fact that W = [ w ij ] is a doubly stochastic matrix and step (b) and (c) are due to Jensen’s inequality and step(d) utilized the Lipschitz property.
Therefor, combining with the conclusions in lemma 1 and lemma 2, we have
[Equation]
which gives us the desired result
[Equation]
E. C ONVERGENCE PROOF SKETCH FOR THE VARIATION OF DAG-A DAM ALGORITHM
Similar as adapt-while-communicate versus adapt-then-communicate format, our proposed DAGAdam algorithm also can have another style:
[Equation]
Note there are two W in the augmented gradient step, but we only need one round of communication in practice.
To see that, we can re-write into this form
[Equation]
That is, instead of communicating the parameters directly, we communicate the parameters with one
SGD step forward.
To establish the convergence analysis of this algorithm, we need to ﬁnd the gradient (134) as a standard gradient of another augmented loss function.
As long as as we ﬁnd that loss function, the rest of proof will be similar as we did in Theorem 1.
Fortunately, it has been stated in Yuan et al.
(2021) paper for the momentum algorithms.
Here, we give a modiﬁcation for the Adam algorithm.
First, we need to further assume that W is a positive deﬁnite matrix.
It is easy to achieve that since if the original W is not, we can use ˜ W
= ( I + W ) / 2 instead.
Suppose W has eigenvalue decomposition form W = Q T Λ Q , where Q is an orthogonal matrix.
Then, we deﬁne W α =
Q T Λ α Q , which is also positive deﬁnite matrix for any α > 0 .
So we have W = W 12 W 12 .
Now we introduce the algorithm into a transformed domain by multiplying W − 12 on the both sides:
[Equation]
We further introduce
[Equation]
The algorithm (130)-(133)is equivalently to
[Equation]
where the notation ∇ ( · ) f ( · ) means taking the gradient with respect to the whole argument of f and note that
[Equation]
t ) W − 12 + (cid:15) (cid:17)
[Equation]
Remember we can always retrieve x ( t ) by x ( t )
=
W 12 s ( t ) .
Now, we are prepared for introduce this new augmented loss function
[Equation]
It is not hard to verify that
[Equation]
This means that (140) - (143) can be viewed as Adam algorithm applied on this augmented loss function L W ( s ) .
There is one difference that U ( t ) is no longer diagonal matrix.
However, recall that we mainly use it in weighted norm, and the useful inequality (25) can be extended for arbitrary positive matrix
[Equation]
) where a and b are the smallest and largest eigenvalue of matrix A .
To see that, suppose A = Q T Λ Q , we have
[Equation]
[Equation]
Therefore, we can establish the similar conclusion as Theorem 1.
F H ETEROGENEOUS DATA DISTRIBUTION GENERATION
This subsection provides the detailed heterogeneous data generation for CIFAR-10 training dataset.
To simulate the heterogeneous data, the disjoint non-i.i.d.
training data over agents are generated through the Dirichlet distribution.
For each data class, a categorical vector q = ( q 1 , · · · , q n ) , where (cid:80) i q i = 1 , q i ≥ 0 , is used to illustrate the training data distribution in a network of size n .
It is drawn from a symmetric Dirichlet distribution Dir( α ) , where α stands for the degree of the data heterogeneity (Lin et al., 2021; Yurochkin et al., 2019).
The training data for a particular class tends to concentrate in a single node as α → 0 , i.e. becoming more heterogenous while the homogeneous data is formed as α → ∞ .
After the sampling procedure, the data is reallocated to ensure the same amount of training data in each network node.
Fig.
9 visualizes the heterogeneous data distribution for the CIFAR-10 experiments shown in Table 1 for both network size (4 and 8).
The radius of each circle stands for the proportion of the samples for the corresponding data class in the node.
Clearly, as α → 0 , the data for each class tends to be allocated in a single node.
When α → ∞ , all the data is distributed evenly.
Fig.
10 illustrates the training loss and validation accuracy curves of different algorithms when α = 10 .
It is observed in the left plot that PAdam and DAG-Adam converges with much lower training losses and higher validation accuracy than the others due to their insensitivity to data heterogeneity.
G M ORE EXPERIMENTAL SETTING AND BERT E XPERIMENTAL R ESULTS
More Implementation details.
We also follow DDP’s design to enable computation and communication overlap.
Each server contains 8 V100 GPUs in our cluster and is treated as one node.
The inter-node network fabrics are 25 Gbps TCP as default, which is a common distributed training platform setting.
The exponential topologies (Assran et al.
(2019)) are utilized as default.
Training Loss.
Fig.
11 shows the iteration-wise training loss curves of aforementioned algorithms separately.
The effect of using dynamic ν .
Besides with ﬁne-tuning a proper ν constant for DAG-Adam, we tried to use a dynamic ν scheduler to validate its performance.
Compared to the common learning rate scheduler, we use dynamic ν schedulers that provide a relatively small ν value at ﬁrst and grow to 1 at the end of training.
Both linear schedulers and cosine schedulers were examined.
Similar as learning rate scheduler, we also use a warm-up strategy, i.e., to ﬁx a static ν for a certain iterations then use dynamic ν schedulers afterwards.
We provide some preliminary results in Table 6.
Unfortunately, it did not provide better performance compared with the result of using constant ν .
We leave it as future works to design a better strategy to select ν adaptively without introducing more hyper-parameters.
3 Node
Airplane
Automobile
Bird
Cat
Deer
Dog
Frog
Horse
Ship
Truck
D a t a
C l a ss
Network size 4, α =       7 Node
Airplane
Automobile
Bird
Cat
Deer
Dog
Frog
Horse
Ship
Truck
D a t a
C l a ss
Network size 8, α =   3 Node
Airplane
Automobile
Bird
Cat
Deer
Dog
Frog
Horse
Ship
Truck
D a t a
C l a ss
Network size 4, α =       7 Node
Airplane
Automobile
Bird
Cat
Deer
Dog
Frog
Horse
Ship
Truck
D a t a
C l a ss
Network size 8, α =   3 Node
Airplane
Automobile
Bird
Cat
Deer
Dog
Frog
Horse
Ship
Truck
D a t a
C l a ss
Network size 4, α = 0.
7 Node
Airplane
Automobile
Bird
Cat
Deer
Dog
Frog
Horse
Ship
Truck
D a t a
C l a ss
Network size 8, α = 0.
[Figure]
T r a i n
L o ss
[Figure] [Figure]
V a li d a t i o n
A cc u r a c y
PAdam DAdam
QG-DAdam GT-DAdam
DAG-Adam
[Figure] [Table]