Original: DADAM: A CONSENSUS - BASED DISTRIBUTED ADAPTIVE GRADIENT METHOD FOR O NLINE O PTIMIZATION
Revised: DADAM: A CONSENSUS - BASED DISTRIBUTED ADAPTIVE GRADIENT METHOD FOR O NLINE O PTIMIZATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Online and stochastic optimization methods such as SGD, ADAGRAD and
Revised: Online and stochastic optimization methods such as SGD, ADAGRAD and

Original: ADAM are key algorithms in solving large-scale machine learning problems including deep learning.
Revised: ADAM are key algorithms in solving large-scale machine learning problems including deep learning.

Original: A number of schemes that are based on communications of nodes with a central server have been recently proposed in the literature to parallelize them.
Revised: A number of schemes that are based on communications of nodes with a central server have been recently proposed in the literature to parallelize them.

Original: A bottleneck of such centralized algorithms lies on the high communication cost incurred by the central node.
Revised: A bottleneck of such centralized algorithms lies on the high communication cost incurred by the central node.

Original: In this paper, we present a new consensus-based distributed adaptive moment estimation method (DADAM) for online optimization over a decentralized network that enables data parallelization, as well as decentralized computation.
Revised: In this paper, we present a new consensus-based distributed adaptive moment estimation method (DADAM) for online optimization over a decentralized network that enables data parallelization, as well as decentralized computation.

Original: Such a framework note only can be extremely useful for learning agents with access to only local data in a communication constrained environment, but as shown in this work also outperform centralized adaptive algorithms such as ADAM for certain realistic classes of loss functions.
Revised: Such a framework not only can be extremely useful for learning agents with access to only local data in a communication constrained environment, but as shown in this work also outperform centralized adaptive algorithms such as ADAM for certain realistic classes of loss functions.

Original: We analyze the convergence properties of the proposed algorithm and provide a dynamic regret bound on the convergence rate of adaptive moment estimation methods in both stochastic and deterministic settings.
Revised: We analyze the convergence properties of the proposed algorithm and provide a dynamic regret bound on the convergence rate of adaptive moment estimation methods in both stochastic and deterministic settings.

Original: Empirical results demonstrate that DADAM works well in practice and compares favorably to competing online optimization methods.
Revised: Empirical results demonstrate that DADAM works well in practice and compares favorably to competing online optimization methods.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Online optimizationor learning is a fundamental procedure for solving a wide range of machine learning problems Shalev-Shwartz et al.
Revised: Online optimization is a fundamental procedure for solving a wide range of machine learning problems Shalev-Shwartz et al.

Original: ; Hazan et al.
Revised: ; Hazan et al.

Original: It can be formulated as a repeated game between a learner (algorithm) and an adversary.
Revised: It can be formulated as a repeated game between a learner (algorithm) and an adversary.

Original: The learner receives a streaming data sequence, sequentially selects actions, and the adversary reveals the convex or nonconvex losses to the learner.
Revised: The learner receives a streaming data sequence, sequentially selects actions, and the adversary reveals the convex or nonconvex losses to the learner.

Original: A standard performance metric for an online algorithm is regret , which measures the performance of the algorithm versus a static benchmark Zinkevich (2003); Hazan et al.
Revised: A standard performance metric for an online algorithm is regret , which measures the performance of the algorithm versus a static benchmark Zinkevich (2003); Hazan et al.

Original: For example, the benchmark could be an optimal point of the online average of  loss ( local cost) functions, had the learner known all the losses in advance.
Revised: For example, the benchmark could be an optimal point of the online average of the loss (local cost) function, had the learner known all the losses in advance.

Original: In a broad sense, if the benchmark is a ﬁxed sequence, the regret is called static .
Revised: In a broad sense, if the benchmark is a ﬁxed sequence, the regret is called static .

Original: Recent work on online optimization has investigated the notion of dynamic regret Zinkevich (2003);
Revised: Recent work on online optimization has investigated the notion of dynamic regret Zinkevich (2003);

Original: Hall & Willett (2015); Besbes et al. (2015).
Revised: Hall & Willett (2015); Besbes et al. (2015).

Original: The dynamic regret can take the form of the cumulative difference between the instantaneous loss and the minimum loss.
Revised: Dynamic regret can take the form of the cumulative difference between the instantaneous loss and the minimum loss.

Original: For convex functions, previous studies have shown that the dynamic regret of online gradient-based methods can be upper bounded by O ( √ TD T ) , where D T is a measure of regularity of the comparator sequence or the function sequence Zinkevich (2003); Hall & Willett (2015).
Revised: For convex functions, previous studies have shown that the dynamic regret of online gradient-based methods can be upper bounded by O ( √ TD T ) , where D T is a measure of regularity of the comparator sequence or the function sequence Zinkevich (2003); Hall & Willett (2015).

Original: This bound can be improved to O ( D T ) Mokhtari et al.
Revised: This bound can be improved to O ( D T ) Mokhtari et al.

Original: ; Zhang et al.
Revised: ; Zhang et al.

Original: (2017) when all the cost functions are strongly convex and smooth.
Revised: (2017) when all the cost functions are strongly convex and smooth.

Original: Decentralized nonlinear programming has received a great deal of interest in diverse scientiﬁc and engineering ﬁelds Tsitsiklis et al.
Revised: Decentralized nonlinear programming has received a great deal of interest in diverse scientiﬁc and engineering ﬁelds Tsitsiklis et al.

Original: ; Li et al.
Revised: ; Li et al.

Original: ; Rabbat & Nowak (2004); Lesser et al.
Revised: ; Rabbat & Nowak (2004); Lesser et al.

Original: The key problem involves optimizing a cost function f ( x ) = 1 n ∑ ni = 1 f i ( x ) , where x ∈ R p and each f i is only known to the individual agent i in a connected network of n agents.
Revised: The key problem involves optimizing a cost function f ( x ) = 1 n ∑ ni = 1 f i ( x ) , where x ∈ R p and each f i is only known to the individual agent i in a connected network of n agents.

Original: The agents collaborate by successively sharing information with other agents located in their neighborhood with the goal of jointly converging to the network-wide optimal argument Nedic & Ozdaglar (2009).
Revised: The agents collaborate by successively sharing information with other agents located in their neighborhood with the goal of jointly converging to the network-wide optimal argument Nedic & Ozdaglar (2009).

Original: Compared to optimization procedures involving a fusion center that collects data and performs the computation, decentralized nonlinear programming enjoys the advantage of scalability to network sizes, robustness to dynamic topologies, and privacy preservation in data-sensitive applications Yuan et al.
Revised: Compared to optimization procedures involving a fusion center that collects data and performs the computation, decentralized nonlinear programming enjoys the advantage of scalability to network sizes, robustness to dynamic topologies, and privacy preservation in data-sensitive applications Yuan et al.

Original: ; Shi et al.
Revised: ; Shi et al.

Original: ; Jiang et al.
Revised: ; Jiang et al.

Original: (2017); Lian et al.
Revised: (2017); Lian et al.

Original: Appropriately choosing the learning rate that scale coordinates of the gradient and the way of updating them are crucial issues driving the performance of ﬁrst Duchi et al.
Revised: Appropriately choosing the learning rate that scale coordinates of the gradient and the way of updating them are crucial issues driving the performance of ﬁrst Duchi et al.

Original: ; Kingma & Ba (2014) and second order optimization proceduresTarzanagh et al. ; Peyghami & Tarzanagh (2015).
Revised: ; Kingma & Ba (2014) and second order optimization procedures Peyghami & Tarzanagh (2015).

Original: Indeed, the understanding that adaptation of the learning rate is advantageous, particularly on a per parameter basis dynamically, led to the development of a family of widely-used adaptive gradient methods including ADAGRAD Duchi et al.
Revised: Indeed, the understanding that adaptation of the learning rate is advantageous, particularly on a per parameter basis dynamically, led to the development of a family of widely-used adaptive gradient methods including ADAGRAD Duchi et al.

Original: (2011), RMSPROP Tieleman & Hinton, and ADAM Kingma & Ba (2014).
Revised: (2011), RMSPROP Tieleman & Hinton, and ADAM Kingma & Ba (2014).

Original: The ADAM optimizer computes adaptive learning rates for different parameters from estimates of ﬁrst and second moments of the gradients and performs a local optimization.
Revised: The ADAM optimizer computes adaptive learning rates for different parameters from estimates of ﬁrst and second moments of the gradients and performs a local optimization.

Original: Numerical results show that ADAM can achieve signiﬁcantly better performance compared to ADAGRAD, RMSPROP and other gradient descent procedures when the gradients are sparse, or in general small in magnitude.
Revised: Numerical results show that ADAM can achieve signiﬁcantly better performance compared to ADAGRAD, RMSPROP and other gradient descent procedures when the gradients are sparse, or in general small in magnitude.

Original: However, its performance has been observed to deteriorate in settings where the loss functions are non-convex and gradients are dense.
Revised: However, its performance has been observed to deteriorate in settings where the loss functions are nonconvex and gradients are dense.

Original: Further, there is currently a gap in the theoretical understanding of these methods, especially in the non-convex and stochastic settingReddi et al. Ward et al.
Revised: Further, there is currently a gap in the theoretical understanding of these methods, especially in the nonconvex and stochastic setting Ward et al.

Original: ; Zeng & Yin (2018).
Revised: ; Zeng & Yin (2018).

Original: In this paper, we develop and analyze a new consensus-based distributed adaptive moment estimation (DADAM) method that incorporates decentalized optimization and uses a variant of the adaptive moment estimation method Duchi et al.
Revised: In this paper, we develop and analyze a new consensus-based distributed adaptive moment estimation (DADAM) method that incorporates decentralized optimization and uses a variant of the adaptive moment estimation methods Duchi et al.

Original: ; Kingma & Ba (2014); Tieleman & Hinton; McMahan & Streeter (2010).
Revised: ; Kingma & Ba (2014); Tieleman & Hinton; McMahan & Streeter (2010).

Original: The proposed method is suitable for large scale machine learning problems such as deep learning, since it enables both data parallelization and decentralized computation.
Revised: Existing distributed stochastic and adaptive gradient methods for deep learning are mostly designed for central network topology Dean et al. ; Li et al.. The main bottleneck of such a topology lies on the communication overload on the central node, since all nodes need to concurrently communicate with it.

Original: Numerical results show the efﬁciency and effectiveness of the method in comparison with existing methods in the literature.
Revised: Hence, performance can be signiﬁcantly degraded when network bandwidth is limited. These considerations motivate us to study an adaptive algorithm for network topologies, where all nodes can only communicate with their neighbors and none of the nodes is designated as “central” Therefore, the proposed method is suitable for large scale machine learning problems,

Revised: since

Revised: it enables both data parallelization and decentralized computation.

Original: Next, we brieﬂy summarize the main technical contributions of the work.
Revised: Next we brieﬂy summarize the main technical contributions of the work.

Original: - Our ﬁrst main result (Theorem 4) provides guarantees of DADAM for constrained convex minimization problems deﬁned over a closed convex set X .
Revised: - Our ﬁrst main result (Theorem 4) provides guarantees of DADAM for constrained convex minimization problems deﬁned over a closed convex set X .

Original: We provide the convergence bound in terms of dynamic regret and show that when the data features are sparse and have bounded gradients, our algorithm’s regret bound can be considerably better than the ones provided by standard mirror descent and gradient descent methods Nedic & Ozdaglar (2009); Hall & Willett (2015); Besbes et al.
Revised: We provide the convergence bound in terms of dynamic regret and show that when the data features are sparse and have bounded gradients, our algorithm’s regret bound can be considerably better than the ones provided by standard mirror descent and gradient descent methods Nedic & Ozdaglar (2009); Hall & Willett (2015); Besbes et al.

Original: It is worth mentioning that the regret bounds provided for adaptive gradient methods Duchi et al.
Revised: It is worth mentioning that the regret bounds provided for adaptive gradient methods Duchi et al.

Original: (2011) are static and our results generalize them to dynamic settings. -
Revised: (2011) are static and our results generalize them to dynamic settings. -

Original: In Theorem 5, we give a new static regret analysis for distributed gradient-based algorithms for nonconvex minimization problems computed over a network of agents.
Revised: In Theorem 5, we give a new local regret analysis for distributed gradient-based algorithms for nonconvex minimization problems computed over a network of agents.

Original: Speciﬁcally, we prove that under certain regularity conditions, DADAM can achieve a  regret bound of order  O ( 1/ T ) for nonconvex distributed optimization.
Revised: Speciﬁcally, we prove that under certain regularity conditions, DADAM can achieve a local regret bound of order ˜ O ( 1 T ) for nonconvex distributed optimization.

Original: To the best of our knowledge, rigorous extensions of existing adaptive gradient methodsDuchi et al. ; Kingma & Ba (2014); Tieleman & Hinton to the (distributed) nonconvex setting considered in this work do not seem to be available.
Revised: To the best of our knowledge, rigorous extensions of existing adaptive gradient methods to the (distributed) nonconvex setting considered in this work do not seem to be available.

Original: - In this paper, we also present regret analysis for distributed stochastic optimization problems computed over a network of agents.
Revised: In this paper, we also present regret analysis for distributed stochastic optimization problems computed over a network of agents.

Original: Theorems 7 and 9 provide regret bounds of DADAM for minimization problem equation 3 with stochastic gradients and indicate that the result of Theorem 4 and 5 hold true in expectation.
Revised: Theorems 7 and 9 provide regret bounds of DADAM for minimization problem equation 3 with stochastic gradients and indicate that the result of Theorems 4 and 5 hold true in expectation.

Original: Hence, DADAM can be used in decentralized environments where the agents have access to noisy data. - DADAM uses the projection map Π X [ x ] . If Π X ( x )  is the identity operator for all x and t , then DADAM corresponds to a decentralized version of adaptive gradient methods Duchi et al.
Revised: Further, in Corollary 10 we show that DADAM can achieve a local regret bound of order O ( ξ 2 √ nT + 1 T ) for nonconvex distributed stochastic optimization where ξ is the gradient noise. Hence, DADAM outperforms centralized adaptive algorithms such as ADAM for certain realistic classes of loss functions.

Original: ; Kingma & Ba (2014); Tieleman & Hinton.

Original: Further, if the mixing matrix W is the identity, Theorems 5, 7 and 9 provide novel regret bounds for adaptive gradient methods for nonconvex and stochastic settings.

Original: The remainder of the paper is organized as follows.
Revised: The remainder of the paper is organized as follows.

Original: Section 2 gives a detailed description of
Revised: Section 2 gives a detailed description of

Original: DADAM while Section 3 establishes its theoretical results.
Revised: DADAM, while Section 3 establishes its theoretical results.

Original: Section 4 explains a network correction technique for our proposed algorithm.
Revised: Section 4 explains a network correction technique for our proposed algorithm.

Original: Section 5 illustrates the proposed framework on a number of synthetic and real data sets.
Revised: Section 5 illustrates the proposed framework on a number of synthetic and real data sets.

Original: Finally, Section 6 concludes the paper.
Revised: Finally, Section 6 concludes the paper.

Original: The detailed proofs of the main results established are delegated to the Appendix.
Revised: The detailed proofs of the main results established are delegated to the Appendix.

Original: Notation.
Revised: 1. M ATHEMATICAL P RELIMINARIES AND N OTATIONS .

Original: Throughout the paper, R p denotes the real coordinate space of p dimensions.
Revised: Throughout the paper, R p denotes the real coordinate space of p dimensions.

Original: For any pair of vectors x , y ∈ R p , (cid:104) x , y (cid:105) indicates the standard Euclidean inner product.
Revised: For any pair of vectors x , y ∈ R p , (cid:104) x , y (cid:105) indicates the standard Euclidean inner product.

Original: We denote the (cid:96)  norm by (cid:107) X (cid:107) 1 =
Revised: We denote the (cid:96) 1 norm by (cid:107) X (cid:107) 1 =

Original: ∑ ij | x ij | , the inﬁnity norm by (cid:107) X (cid:107) ∞ =
Revised: ∑ ij | x ij | , the inﬁnity norm by (cid:107) X (cid:107) ∞ =

Original: max ij | x ij | , and the Euclidean norm by (cid:107) X (cid:107) =
Revised: max ij | x ij | , and the Euclidean norm by (cid:107) X (cid:107) =

Original: (cid:113)
Revised: (cid:113)

Original: ∑ ij | x ij | 2 .
Revised: ∑ ij | x ij | 2 .

Original: The above norms reduce to the vector norms if X is a vector.
Revised: The above norms reduce to the vector norms if X is a vector.

Original: The diameter of the set Xwith respect to these norms is givenrespectively by
Revised: The diameter of the set X is given by

Original: [Equation]
Revised: [Equation]

Original: Let S p + be the set of all positive deﬁnite p × p matrices.
Revised: Let S p + be the set of all positive deﬁnite p × p matrices.

Original: Π X , A [ x ] denotes the Euclidean projection of a vector x onto X for A ∈ S p + :
Revised: Π X , A [ x ] denotes the Euclidean projection of a vector x onto X for A ∈ S p + :

Original: [Equation]
Revised: [Equation]

Original: The subscript t is often used to denote the time step while y i , t , d stands for the d -th element of y i , t .
Revised: The subscript t is often used to denote the time step while y i , t , d stands for the d -th element of y i [Equation]

Original:  Further, y i , 1: t , d ∈ R t is given by
Revised: t . Further, y i , 1: t , d ∈ R t is given by

Original: [Equation]
Revised: [Equation]

Original: We let g i , t to denote the gradient of f at x i , t .
Revised: We let g i , t to denote the gradient of f at x i , t .

Original: The i -th largest singular  of matrix X is denoted by σ i ( X ) .
Revised: The i -th largest singular value of matrix X is denoted by σ i ( X ) .

Original: We denote the element in the i th row and j th column of matrix X by [ X ] ij .
Revised: We denote the element in the i -th row and j -th column of matrix X by [ X ] ij .

Original: In several theorems, we consider a connected undirected graph G =
Revised: In several theorems, we consider a connected undirected graph G =

Original: ( V , E ) with nodes V = { 1 , 2 ,..., n } and edges E .
Revised: ( V , E ) with nodes V = { 1 , 2 ,..., n } and edges E .

Original: The matrix W ∈ R p × p is often used to denote a symmetric mixing matrix Shi et al.
Revised: The matrix W ∈ R p × p is often used to denote a symmetric mixing matrix Shi et al.

Original: (2015) of graph G .
Revised: (2015) of graph G .

Original: Indeed, for any agent i , we assign a positive weight [ W ] ij for the information received from agent j (cid:54)
Revised: Indeed, for any agent i , we assign a positive weight [ W ] ij for the information received from agent j (cid:54)

Original: =
Revised: =

Original: i so that [ W ] ij > 0 if and only if ( i , j ) ∈ E and 
Revised: i so that [ W ] ij > 0 if and only if ( i , j ) ∈ E and [Equation]

Original: [Equation]
Revised: [Equation]

Revised: The role of W is the similar as that in decentralized gradient descent methods Yuan et al.

Revised: and average consensus Xiao et al.

Revised: It has a few common choices, which can signiﬁcantly affect performance.

Revised: Throughout this paper, we consider Metropolis constant edge weight matrix Xiao &

Revised: Boyd (2004); Xiao et al.

Revised: (2007) deﬁne as follows

Revised: [Equation]

Revised: for some small positive ε > 0.

Revised: When ˆ W is chosen according this scheme, W =

Revised: I + ˆ W 2 is found to be very efﬁcient Xiao & Boyd (2004).

Revised: Also, the above doubly stochastic matrix implies uniqueness of σ 1 ( W ) = 1 and warrants that other singular value of W are strictly less than one in magnitude.

Original: The Hadamard (entrywise) and Kronecker product are denoted by (cid:12) and ⊗ , respectively.
Revised: The Hadamard (entrywise) and Kronecker product are denoted by (cid:12) and ⊗ , respectively.

Original: Finally, the expectation operator is denoted by E .
Revised: Finally, the expectation operator is denoted by E .

Original:  P ROBLEM F ORMULATION AND A LGORITHM
Revised:  P ROBLEM F ORMULATION AND A LGORITHM

Original: In this section, we propose a new optimization method for online optimization that employs data parallelization and decentralized computation over a network of agents.
Revised: In this section, we propose a new optimization method for online optimization that employs data parallelization and decentralized computation over a network of agents.

Original: In our new structure, given a connected undirected graph G =
Revised: In our new structure, given a connected undirected graph G =

Original: ( V , E )
Revised: ( V , E )

Original: , we let each node i ∈ V at time t ∈ [ T ] hold its own measurement and training data m i , and set f i , t ( x ) 
Revised: , we let each node i ∈ V at time t ∈ { 1 ,..., T } hold its own measurement and training data m i , and set f i , t ( x ) =

Original: = 1 m i ∑ m i j = 1 f ji , t ( x ) .
Revised: 1 m i ∑ m i j = 1 f j i , t ( x ) .

Original: We also let each agent i hold a local copy of the global variable x at time t ∈ [ T ] , which is denoted by x i , t ∈ R p .
Revised: We also let each agent i hold a local copy of the global variable x at time t ∈ { 1 ,..., T } , which is denoted by x i , t ∈ R p .

Original: With this setup, we present the following decentralized adaptive gradient method for solving the minimization problemminimize
Revised: With this setup, we present the following decentralized adaptive gradient method for solving the minimization problem

Original: [Equation]
Revised: [Equation]

Original: where f i , t : X → R is a continuously differentiable mapping on the convex set X .
Revised: where f i , t : X → R is a continuously differentiable mapping on the convex set X .

Original: It is known that it is impossible to achieve a sub-linear dynamic regret bound Zhang et al.
Revised: It is known that it is impossible to achieve a sub-linear dynamic regret bound Zhang et al.

Original: [Equation] ), because of the arbitrary ﬂuctuations in problem equation 3.
Revised: (2017), because of the arbitrary ﬂuctuations in problem equation 3.

Original: We want to characterize the hardness of the problem via a complexly measure that captures the pattern of the minimizer sequence { x ∗ t } Tt = 1 , where x ∗ t = argmin x ∈ X f t ( x ) .
Revised: We want to characterize the hardness of the problem via a complexly measure that captures the pattern of the minimizer sequence { x ∗ t } Tt = 1 , where x ∗ t = argmin x ∈ X f t ( x ) .

Original: Subsequently, we would like to provide a regret bound in terms of
Revised: Subsequently, we would like to provide a regret bound in terms of

Original: [Equation]
Revised: [Equation]

Revised: for

Original: [Equation] 
Revised: [Equation] [Equation]

Original: which represents the variations in { x ∗ t } Tt = 1 .
Revised: which represents the variations in { x ∗ t } Tt = 1 .

Original: DADAM uses a new distributed adaptive gradient method in which a group of n agents seeking to solve a sequential version of problem equation 3.
Revised: DADAM uses a new distributed adaptive gradient method in which a group of n agents seeking to solve a sequential version of problem equation 3.

Original: Here, we assume that each component function f i , t : X → R becomes only available to agent i ∈ V , after having made its decision at time t ∈ { 1 ,..., T } .
Revised: Here, we assume that each component function f i , t : X → R becomes only available to agent i ∈ V , after having made its decision at time t ∈ { 1 ,..., T } .

Original: In the t -th, the i -th agent chooses a point x
Revised: In the t -th, the i -th agent chooses a point x

Original: i , t corresponding to what it considers the network as a whole should have selected.
Revised: i , t corresponding to what it considers the network as a whole should have selected.

Original: After committing to this choice, the agent has access to a cost function f i , t : X → R and the network cost is then given by f t ( x ) = 1 n ∑ ni = 1 f
Revised: After committing to this choice, the agent has access to a cost function f i , t : X → R and the network cost is then given by f t ( x ) = 1 n ∑ ni = 1 f

Original: i , t ( x ) .
Revised: i , t ( x ) .

Original: Note that this function is not known to any of the agents and is not available at any single location.
Revised: Note that this function is not known to any of the agents and is not available at any single location.

Original: The procedure of our proposed method is outlined in Algorithm 1.
Revised: The procedure of our proposed method is outlined in Algorithm 1.

Original: It is worth mentioning that DADAM computes adaptive learning rates from estimates of both ﬁrst and second moments of the gradients similar to Duchi et al.{+moment estimate β 1 , β 2 , β 3 ∈ [ 0 , 1 ) , and a mixing matrix W satisfying equation 2; 1 for all i ∈ V , initialize moment vectors m i , 
 = υ i , 
 =+}
Revised: input : x 1 ∈ X , a positive sequence { α t } Tt = 1 , exponential decay rates for the {+moment estimate β 1 , β 2 , β 3 ∈ [ 0 , 1 ) , and a mixing matrix W satisfying equation 2; 1 for all i ∈ V , initialize moment vectors m i , 
 = υ i , 
 =+}

Original: {+ˆ υ i , 
+}
Revised: {+ˆ υ i , 
+}

Revised: = 0 and x i , 1 = x 1 ; 2 for i ∈ V do for t ← 1 to T do g i , t = ∇ f i , t ( x i , t ) ; m i , t = β 1 m i , t −

Revised: [Equation]

Revised: [Equation]

Revised: 1 ) g i , t ; υ i , t =

Revised: β 2 υ i , t

Revised: [Equation]

Revised: 1 +( 1 − β 2 ) g i , t (cid:12) g i , t ; ˆ υ i , t

Revised: [Equation]

Revised: i , t

Revised: [Equation]

Revised: 3 ) max ( ˆ

Original: ; Tieleman & Hinton; Kingma &
Revised: [Equation] i , t [Equation] [Equation]

Original: Ba (2014).
Revised: i , t ) ; [Equation] [Equation] ; output: resulting parameter ¯ x = 1 n ∑ ni = 1 x i , T +

Revised: It is worth mentioning that DADAM computes adaptive learning rates from estimates of both ﬁrst and second moments of the gradients similar to Duchi et al.

Original:  However, DADAM uses a smaller learning rate in comparison to ADAM Kingma & Ba (2014) and yet incorporates the intuition of slowly decaying the effect of previous gradients on the learning rate.
Revised: ; Tieleman & Hinton; Kingma & Ba (2014); Reddi et al. However, DADAM uses a larger learning rate in comparison to AMSGrad Reddi et al. and yet incorporates the intuition of slowly decaying the effect of previous gradients on the learning rate.

Original: The key difference of DADAM with ADAM  is that it maintains the maximum of all second moment estimates of the gradient vectors until time step t and uses
Revised: The key difference of DADAM with ADAM and AMSGrad is that it maintains the maximum of all second moment estimates of the gradient vectors until time step t and uses

Original: [Equation]
Revised: [Equation]

Original: for normalizing the running average of the gradient instead of υ i , t in ADAM.
Revised: for normalizing the running average of the gradient instead of υ i , t in ADAM and max ( ˆ υ i , t − 1 , υ i , t ) in AMSGrad.

Original: The learning rate ˆ υ i , t is an important component of the DADAM framework, since it enables us to develop a convergent adaptive method for solving equation 3. Note that in general, ADAM is not necessarily convergent even without operator Π X ,. Reddi et al. Note also that DADAM is initialized at x i , t = 0 to keep the presentation of the convergence analysis clear.
Revised: The learning rate ˆ υ i , t is an important component of the DADAM framework, since it enables us to develop a convergent adaptive method similar to AMSGrad while maintaining the efﬁciency of ADAM.

Original: In general, any initialization can be selected for implementation purposes.

Original: Next, we provide the notion of regret which measures the performance of DADAM against a sequence of local minimizers.
Revised: Next, we provide the notion of regret which measures the performance of DADAM against a sequence of local minimizers.

Original: In the framework of online convex optimization, the performance of algorithms is assdessed by regret that measures how competitive the algorithm is with respect to the best ﬁxed solution MateosN´unez & Cort´es (2014); Hazan et al.; Shahrampour et al.
Revised: In the framework of online convex optimization, the performance of algorithms is assessed by regret that measures how competitive the algorithm is with respect to the best ﬁxed solution Mateos-N´unez & Cort´es (2014); Hazan et al.

Original: However, the notion of regret fails to illustrate the performance of online algorithms in a dynamic setting.
Revised: However, the notion of regret fails to illustrate the performance of online algorithms in a dynamic setting.

Original: To overcome this issue, we consider a more stringent metric—dynamic regret Hall & Willett (2015);
Revised: To overcome this issue, we consider a more stringent metric–dynamic regret Hall & Willett (2015);

Original: Besbes et al.
Revised: Besbes et al.

Original: ; Zinkevich (2003), in which the cumulative loss of the learner is compared against the minimizer sequence { x ∗ t } Tt = 1 , i.e.,
Revised: ; Zinkevich (2003), in which the cumulative loss of the learner is compared against the minimizer sequence { x ∗ t } Tt = 1 , i.e.,

Original: [Equation]
Revised: Reg CT : = 1 n n ∑ i = T ∑ t = 1 f t ( x i , t ) −

Original:  where x ∗ t = argmin x ∈ X f t ( x ) .
Revised: T ∑ t = 1 f t ( x ∗ t ) , where x ∗ t = argmin x ∈ X f t ( x ) .

Original: On the other hand, in the framework of nonconvex optimization, it is usual to state convergence guarantees of an algorithm towards an ε -approximate stationary point- that is, there exist some iterates x  t for which (cid:107) ∇ f t ( x  t ) (cid:107) ≤ ε .
Revised: On the other hand, in the framework of nonconvex optimization, it is usual to state convergence guarantees of an algorithm towards an ε -approximate stationary point–that is, there exist some iterates x i , t for which (cid:107) ∇ f t ( x i , t ) (cid:107) ≤ ε .

Original: Inspired by Hazan et al.
Revised: Inspired by Hazan et al.

Original: (2017), we next provide the deﬁnition of projected gradient and introduce localstatic regret , a new notion of regret which quantiﬁes the moving average of gradients over network.
Revised: (2017), we next provide the deﬁnition of projected gradient and introduce local regret , a new notion of regret which quantiﬁes the moving average of gradients over network.

Original: Deﬁnition 1.
Revised: Deﬁnition 1.

Original: Let f i : X → R be a differentiable function on a closed convex set X ⊆ R n .
Revised: (Local Regret) . Assume f i : X → R is a differentiable function on a closed convex set X ⊆ R p .

Original: Let α > 0 . We deﬁne G X ( x , f i , α ) :
Revised: Given a step size α > 0 , we deﬁne G X ( x , f i , α ) :

Original: X → R n , the projected gradient of f i at x, by
Revised: X → R p the projected gradient of f i at x, by

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: Now, let ¯ f i , t
Revised: Then, the local regret of an online algorithm is given by

Original: ( x i , t )
Revised: [Equation]

Original:  = 1 t ∑ ts = 1 f i , s ( x i , t ) be the aggregate losswith gradient ∇ ¯ f i , t
Revised: where ¯ f i , t ( x i , t ) = 1 t ∑ ts = 1 f i , s ( x i , t ) is an aggregate loss

Original: ( x i , t ) = 1 t ∑ ts = 1 ∇ f i , s ( x i , t ) . We analyze the convergence of DADAM as applied tothe nonconvex minimization problem equation 3 using  the following local dynamic regret:
Revised: We analyze the convergence of DADAM as applied to minimization problem equation 3 using regrets Reg CT and Reg NT . It is worth to mention that DADAM is initialized at x i , t = 0 to keep the presentation of the convergence analysis clear.

Original: [Equation] where G X is given by equation 5.
Revised: In general, any initialization can be selected for implementation purposes.

Original:  C ONVERGENCE A NALYSIS
Revised:  C ONVERGENCE A NALYSIS

Original: In this section, our aim is to establish convergence properties of DADAM under the following assumptions:
Revised: In this section, our aim is to establish convergence properties of DADAM under the following assumptions:

Original: Assumption 2.
Revised: Assumption 2.

Original: For all i ∈ V and t ∈ { 1 ,..., T } , the function f i , t ( x ) is continuously differentiable over X , and has Lipschitz continuous gradient on this set, i.e. there exists a constant ρ ≥ 

Revised: For all i ∈ V and t ∈ { 1 ,..., T } , the function f i , t ( x ) is continuously differentiable over X , and has Lipschitz continuous gradient on this set, i.e. there exists a constant ρ ≥

Original: so that
Revised: so that

Original: [Equation]

Original: which implies

Original: [Equation]
Revised: [Equation]

Original: Further, f i , t ( . ) is Lipschitz continuous on X with a uniform constant G ≥ 0 , i.e.,
Revised: Further, f i , t ( · ) is Lipschitz continuous on X with a uniform constant L ≥ 0 , i.e.,

Original: [Equation]
Revised: [Equation]

Original: There are many cost functions f i , t that satisfy this type of Lipschitz condition.
Revised: There are many cost functions f i , t that satisfy this type of Lipschitz condition.

Original: For example, it holds for any convex function on a compact set X , or for any polyhedral function on an arbitrary domain
Revised: For example, it holds for any convex function on a compact set X , or for any polyhedral function on an arbitrary domain

Original: Duchi et al.
Revised: Duchi et al.

Original: Assumption 3.
Revised: Assumption 3.

Original: For all i ∈ V and t ∈ { 1 ,..., T } , the stochastic gradient g = ∇ f i , t ( x i , t )  satisﬁes
Revised: For all i ∈ V and t ∈ { 1 ,..., T } , the stochastic gradient, as notation we use bold letters g = ∇ f i , t ( x i , t ) , satisﬁes

Original: [Equation]
Revised: [Equation]

Original: where F t is the ξ -ﬁeld containing all information prior to the outset of round t + 1 .
Revised: where F t is the ξ -ﬁeld containing all information prior to the outset of round t + 1 .

Original:  C ONVEX C ASE
Revised: 3. C ONVEX C ASE

Original: Next, we focus on the case where for all i ∈ V and t ∈ { 1 ,..., T } , the agent i at time t has access to the exact gradient g i , t = ∇ f i , t ( x i , t ) .
Revised: Next, we focus on the case where for all i ∈ V and t ∈ { 1 ,..., T } , the agent i at time t has access to the exact gradient g i , t = ∇ f i , t ( x i , t ) .

Original: The following results apply to convex problems, as well as their stochastic variants in a dynamic environment.
Revised: The following results apply to convex problems, as well as their stochastic variants in a dynamic environment.

Original: Theorem 4.
Revised: Theorem 4.

Original: Suppose that the parameters ν 1 , ν 2 ∈ [ 0 , 1 ) satisfy η = ν 1 √ ν 2 < 1 .
Revised: Suppose that the parameters β 1 , β 2 ∈ [ 0 , 1 ) satisfy η = β 1 √ β 2 < 1 .

Original: Let ν 1 , t = ν 1 λ t − 1 , λ ∈ ( 0 , 1 ) and (cid:107) ∇ f i , t ( x ) (cid:107) ∞ ≤ G ∞ for all t ∈ { 1 ,..., T } .
Revised: Let β 1 , t = β 1 λ t − 1 , λ ∈ ( 0 , 1 ) and (cid:107) ∇ f i , t ( x ) (cid:107) ∞ ≤ G ∞ for all t ∈ { 1 ,..., T } .

Original: Then, using a step-size α t = α √ t for the sequence x i , t generated by Algorithm 1, we have
Revised: Then, using a step-size α t = α √ t for the sequence x i , t generated by Algorithm 1, we have

Original: [Equation]
Revised: [Equation]

Original: Next, we analyze the stochastic convex setting and extend the result of Theorem 4 to the noisy case where agents have access to stochastic gradients of the objective function equation 3.
Revised: Next, we analyze the stochastic convex setting and extend the result of Theorem 4 to the noisy case where agents have access to stochastic gradients of the objective function equation 3.

Original: Theorem 5.
Revised: Theorem 5.

Original: Suppose that Assumption 3 holds.
Revised: Suppose that Assumption 3 holds.

Original: Further, the parameters ν 1 , ν 2 ∈ [ 0 , 1 ) satisfy η = ν√ ν 2 < 1 .
Revised: Further, the parameters β 1 , β 2 ∈ [ 0 , 1 ) satisfy η = β 1 √ β 2 < 1 .

Original: Let ν 1 , t = ν 1 λ t − 1 , λ ∈ ( 0 , 1 ) .
Revised: Let β 1 , t = β 1 λ t − 1 , λ ∈ ( 0 , 1 ) .

Original: Then, using a step-size α t = α √ t for the sequence x i , t generated by Algorithm 1, we have
Revised: Then, using a step-size α t = α √ t for the sequence x i , t generated by Algorithm 1, we have

Original: [Equation]
Revised: [Equation]

Original: Remark 6.
Revised: Remark 6.

Original: The dynamic regret bounds provided in Theorems 4 and 5 areworse-case optimal, and for many classes of loss functions they are better than existing bounds for both centralized and decentralized settings Zinkevich (2003);
Revised: The dynamic regret bounds provided in Theorems 4 and 5 are better than existing bounds for both centralized and decentralized settings Zinkevich (2003);

Original: Hall & Willett (2015); Besbes et al. (2015); Shahrampour & Jadbabaie (2018) since
Revised: Hall & Willett (2015); Besbes et al. (2015); Shahrampour & Jadbabaie (2018) since

Original: [Equation]
Revised: [Equation]

Original: and
Revised: It is easy to show that the regret of DADAM is bounded by O ( G ∞ √ T ) as well.

Original: [Equation]
Revised: Indeed, the term ∑ Tt = 1 | g i , t , d | / √ t in the proof of Lemma 2 can be bounded by O ( G ∞ √ T ) instead of O ( √ 1 + log T || g 1: T , d || ) . Hence, the regret of DADAM is upper bounded by minimum of O ( G ∞ √ T ) and the bound presented in Theorems 4 and 5, and thus the worst case dependence on T is √ T rather than √ T log T.

Original: 3. N ONCONVEX C ASE
Revised: 3. N ONCONVEX C ASE

Original: In this section, we provide convergence guarantees for DADAM for the nonconvex minimization problem equation 3 deﬁned over a closed convex set Xand show that its regret bound is O ( 1 T ) .
Revised: In this section, we provide convergence guarantees for DADAM for the nonconvex minimization problem equation 3 deﬁned over a closed convex set X .

Original: To do so, we use the projection map Π X instead of Π X , √ diag ( ˆ v i , t ) for updating parameters x i , t  (see, Algorithm 1 for details).
Revised: To do so, we use the projection map Π X instead of Π X , √ diag ( ˆ v i , t ) for updating parameters x i , t for t ∈ { 1 ,..., T } , i ∈ V (see, Algorithm 1 for details).

Revised: Next, we derive an upper and lower bound for the second moment of gradient, υ i , t .

Original: Also, we assume that there exist two positive numbers υ and ¯ 
Revised: Assume that there exists a positive number ¯ υ such that

Original: υ such that min d ∈{ 1 ,..., p } g i , t , d ≥ υ and max d ∈{ 1 ,..., p } g i , t , d ≤ ¯ υ .
Revised: [Equation]

Original: Now, using the update  of υ i , t as υ i , t =( 1 − ν 2 )
Revised: Now, using the update rule of υ i , t , we have υ i , t =

Original: ∑ tl = 1 ν t − l 2 g i , l and since min d ∈{ 1 ,..., p } g i , t , d ≥ υ , we have

Original: [Equation]
Revised: [Equation]

Original: and
Revised: 1 − β

Original: [Equation]
Revised: [Equation]

Original: The update rules ˜

Original: υ
Revised: ∑ tl

Original: i , t and ˆ υ i , t along with equation 8 and equation 9 lead to
Revised: [Equation] 1 β t − l 2 g 2 i , l . This together with equationimply that

Original: [Equation]
Revised: [Equation]

Original:  and 
Revised: On the other hand, since υ i , t , d is non-decreasing and g i , t , d is required to be element-wise non-zero, we have

Original: [Equation]
Revised: [Equation]

Original: Therefore, the estimates of second moments of the gradients satisfy
Revised: for some constant υ . Now, using the update rule of ˆ υ i , t along with equation 10 and equation 9, we have

Original: [Equation]
Revised: [Equation]

Original: where υ and ¯ υ are two positive constants.

Original: Theorem 7.
Revised: Theorem 7.

Original: Suppose that Assumption 2 holds.
Revised: Suppose that Assumption 2 holds.

Original: Further, the parameters ν 1 , ν 2 ∈ [ 0 , 1 ) satisfy η = ν 1 √ ν 2 < 1 .
Revised: Further, the parameters β 1 , β 2 ∈ [ 0 , 1 ) satisfy η = β 1 √ β 2 < 1 .

Original: Choose a positive sequence { α t } Tt = 1 such that 0 < α t ≤ ( 2 − ν 1 ) υ 2 ρ ¯ υ with α t < ( 2 − ν 1 ) υ 2 ρ ¯ υ for at least one t. Then, for the sequence x i , t generated by Algorithm 1, we have
Revised: Choose the positive sequence { α t } Tt = 1 such that 0 < α t ≤ ( 2 − β 1 ) υ 2 ρ ¯ υ with α t < ( 2 − β 1 ) υ 2 ρ ¯ υ for at least one t. Then, for the sequence x i , t generated by Algorithm 1, we have

Original: [Equation]
Revised: [Equation]

Original:  where ϑ t =
Revised: 12) where ϑ t =

Original: ∑ Tt = 

Revised: ∑ Tt =

Original: [ ( 2 − ν 1 ) α t 2 ¯ υ − ρα 2 t 2 υ 2 ] .
Revised: [ ( 2 − β 1 ) α t 2 ¯ υ − ρα 2 t 2 υ 2 ] .

Original: The following corollary shows that DADAM using a certain step-size leads to a near optimal regret bound for nonconvex functions.
Revised: The following corollary shows that DADAM using a certain step-size leads to a near optimal regret bound for nonconvex functions.

Original: Corollary 8.
Revised: Corollary 8.

Original: Suppose that in the DADAM algorithm the step-size is given by α t 
Revised: Under the same conditions of Theorem 7, using the step-sizes α t =

Original: = ( 2 − ν 1 ) υ 2 2 ρ ¯ υ  for all t ∈ { 1 ,..., T } . Then, we have
Revised: ( 2 − β 1 ) υ 2 2 ρ ¯ υ and β 1 , t = β 1 λ t − 1 , λ ∈ ( 0 , 1 ) for all t ∈ { 1 ,..., T } , we have

Original: [Equation]
Revised: [Equation]

Original: To complete the analysis of our algorithm in the nonconvex setting, we provide the regret bound for
Revised: To complete the analysis of our algorithm in the nonconvex setting, we provide the regret bound for

Original: DADAM, when stochastic gradients are accessible to the learner.
Revised: DADAM, when stochastic gradients are accessible to the learner.

Original: Theorem 9.
Revised: Theorem 9.

Original: Suppose that Assumption 2 holds.
Revised: Suppose that Assumption 2 holds.

Original: Further, the parameters ν 1 , ν 2 ∈ [ 0 , 1 ) satisfy η = ν 1 √ ν 2 < 1 .
Revised: Further, the parameters β 1 , β 2 ∈ [ 0 , 1 ) satisfy η = β 1 √ β 2 < 1 .

Original: Choose a positive sequence { α t } Tt = 1 such that 0 < α t ≤ ( 2 − ν 1 ) υ 2 ρ ¯ υ with α t < ( 2 − ν 1 ) υ 2 ρ ¯ υ for at least one t. Then, for the sequence x i , t generated by Algorithm 1, we have
Revised: Choose the positive sequence { α t } Tt = 1 such that 0 < α t ≤ ( 2 − β 1 ) υ 2 ρ ¯ υ with α t < ( 2 − β 1 ) υ 2 ρ ¯ υ for at least one t. Then, for the sequence x i , t generated by Algorithm 1, we have

Original: [Equation]
Revised: [Equation]

Original: 13) where ϑ t =
Revised: where ϑ t =

Original: ∑ Tt = 

Revised: ∑ T t =

Original: [ ( 2 − ν 1 ) α t 2 ¯ υ [Equation] υ [Equation]
Revised: [ ( 2 − β 1 ) α t 2 ¯ υ − ρα 2 t 2 υ 2 ] .

Original: 3.2. W HEN DOES DADAM O UTPERFORM ADAM?
Revised: 3.2. W HEN DOES DADAM O UTPERFORM ADAM?

Original: We next theoretically justify the potential advantage of the proposed decentralized algorithm
Revised: We next theoretically justify the potential advantage of the proposed decentralized algorithm

Original: DADAM over centralized adaptive moment estimation methods such as ADAM .
Revised: DADAM over centralized adaptive moment estimation methods such as ADAM .

Original: More speciﬁcally, the following corollary shows that when T is sufﬁciently large, the 1 T term will be dominated by the 1 √ nT term which leads to a 1 √ nT convergence rate.
Revised: More speciﬁcally, the following corollary shows that when T is sufﬁciently large, the 1 T term will be dominated by the 1 √ nT term which leads to a 1 √ nT convergence rate.

Original: That is, DADAM achieves linear speed-up with respect to the number of agents.

Original: Corollary 10.
Revised: Corollary 10.

Original: Suppose that Assumptions 2 and 3 hold.
Revised: Suppose that Assumptions 2 and 3 hold.

Original: Moreover, the parameters ν 1 , ν 2 ∈ [ 0 , 1 ) satisfy η = ν 1 √ ν 2 < 1 and ν 1 , t = ν 1 λ t − 1 , λ ∈ ( 0 , 1 ) .
Revised: Moreover, the parameters β 1 , β 2 ∈ [ 0 , 1 ) satisfy η = β 1 √ β 2 < 1 and β 1 , t = β 1 λ t − 1 , λ ∈ ( 0 , 1 ) .

Original: Choose the step-size sequence as α t = α √ nT with α = ( 2 − 
Revised: Choose the step-size sequence as α t = α √ nT with α = ( 2 − β 1 )

Original: [Equation]
Revised: υ 2 ρ ¯ υ .

Original: Then, for the sequence x i , t generated by Algorithm 1, we haveE [Equation] Reg
Revised: Then, for the sequence x i , t generated by Algorithm 1, we have

Original: [Equation]
Revised: [Equation]

Original: if the total number of time steps T is sufﬁciently large; in particular,
Revised: if the total number of time steps T is sufﬁciently large; in particular,

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Revised: Let ε -approximation solution of equation 3 be deﬁned by T − 1 E (cid:2)

Revised: Reg NT (cid:3) ≤ ε .

Revised: Corollary 10 indicates that the total computational complexity of DADAM to achieve an ε -approximation solution is bounded by O ( 1 ε 2 ) .

Original:  A N E XTENSION OF DADAM WITH A C ORRECTED U PDATE R ULE
Revised:  A N E XTENSION OF DADAM WITH A C ORRECTED U PDATE R ULE

Original: Compared to classical centralized algorithms, decentralized algorithms encounter more restrictive assumptions and typically worse convergence rates.
Revised: Compared to classical centralized algorithms, decentralized algorithms encounter more restrictive assumptions and typically worse convergence rates.

Original: In order to improve the convergence rate, Shi et al. (2015) introduced a corrected decentralized gradient method called EXTRA, which has a linear rate of convergence if the objective function is strongly convex.
Revised: In order to improve the convergence rate, Shi et al. (2015) introduced a corrected decentralized gradient method called EXTRA, which has a linear rate of convergence if the objective function is strongly convex.

Original: Similar to EXTRA, we provide next a corrected update rule for i = 1 ,..., n , given by
Revised: Similar to EXTRA, we provide next a corrected update rule for i ∈ V , given by

Original: [Equation]
Revised: [Equation]

Original: where ˆ W = I + W 

Revised: where ˆ W = I + W

Original: and x i , t + 1 is generated by Algorithm 1.
Revised: and x i , t + 1 is generated by Algorithm 1.

Original: Our numerical results on some test problems show the efﬁciency and effectiveness of the corrected
Revised: Our numerical results on some test problems show the efﬁciency and effectiveness of the corrected

Original: DADAM (C-DADAM) in practice.
Revised: DADAM (C-DADAM) in practice.

Original: Note that a C-DADAM update is a DADAM update with a cumulative correction term.
Revised: Note that a C-DADAM update is a DADAM update with a cumulative correction term.

Original: The summation in equation 16 is necessary, since each individual term ∑ nj = 

Revised: The summation in equation 17 is necessary, since each individual term ∑ nj =

Original: [ W − ˆ W ] ij x j , s is asymptotically vanishing and the terms must work cumulatively.
Revised: [ W − ˆ W ] ij x j , s is asymptotically vanishing and the terms must work cumulatively.

Original:  N UMERICAL RESULTS
Revised:  N UMERICAL RESULTS

Original: In this section, we present numerical results for Algorithm 1 (DADAM), on both synthetic and real data sets.
Revised: In this section, we present numerical results for Algorithm 1 (DADAM), on both synthetic and real data sets.

Original: The employed data sets and the code will be made available on github upon acceptance of this manuscript.
Revised: The employed data sets and the code will be made available on github upon acceptance of this manuscript.

Original: The results are organized in the following sub-sections.
Revised: The results are organized in the following sub-sections.

Original: In Section 5.1, we investigate the efﬁcency of adaptive decentralized updating rule in static and deterministic settings.
Revised: In Section 5.1, we investigate the efﬁciency of adaptive decentralized updating rule in static and deterministic settings.

Original: We present numerical results on a sparse binary classiﬁcation problem, comparing the performance of DADAM to that of standard decentralized gradient descent (DGD) Nedic & Ozdaglar (2009);
Revised: We present numerical results on a sparse binary classiﬁcation problem, comparing the performance of DADAM to that of standard decentralized gradient descent (DGD) Nedic & Ozdaglar (2009);

Original: Yuan et al.
Revised: Yuan et al.

Original: and its corrected version EXTRA Shi et al.
Revised: and its corrected version EXTRA Shi et al.

Original: In Section 5.2, we demonstrate the efﬁcacy of DADAM in comparison with a decentralized ADAM (Algorithm 1 with ν 3 = 1) and the recently proposed distributed federated averaging SGD (FedAvg) algorithm on benchmark datasets such as MNIST and CIFAR-10.
Revised: In Section 5.2, we demonstrate the efﬁcacy of DADAM in comparison with a decentralized ADAM (Algorithm 1 with β 3 = 1) and the recently proposed distributed federated averaging SGD (FedAvg) algorithm on benchmark datasets such as MNIST and CIFAR-10.

Original: All algorithms have been implemented in a Mac machine equipped with a 1.8 GHz Intel Core i processor and 8 GB 1600 MHz DDR3 of memory.
Revised: All algorithms have been implemented in a Mac machine equipped with a 1.8 GHz Intel Core i processor and 8 GB 1600 MHz DDR3 of memory.

Original: 5. B INARY CLASSIFICATION BASED ON SYNTHETIC DATA IN THE STATIC ENVIRONMENT
Revised: 5. B INARY CLASSIFICATION BASED ON SYNTHETIC DATA IN THE STATIC ENVIRONMENT

Original: Consider the following online distributed learning setting: at each time t , m i randomly generated data points are given to every agent i in the form of ( y t , i , j , x t , i , j ) , where y t , i , j ∈ {± 1 } is a binary label and x t , i , j ∈ R p is a feature vector for j = 1 ,..., m i .
Revised: Consider the following online distributed learning setting: at each time t , m i randomly generated data points are given to every agent i in the form of ( y t , i , j , x t , i , j ) , where y t , i , j ∈ {± 1 } is a binary label and x t , i , j ∈ R p is a feature vector for j = 1 ,..., m i .

Original: Our goal is to train a classiﬁer θ ∈ R p such that for an arbitrary new feature vector
Revised: Our goal is to train a classiﬁer θ ∈ R p such that for an arbitrary new feature vector

Original: ˆ x it assigns ˆ y = sign ( (cid:104) θ , ˆ x (cid:105) ) .
Revised: ˆ x it assigns ˆ y = sign ( (cid:104) θ , ˆ x (cid:105) ) .

Original: We design decentralized convex logistic and nonconvex sigmoid loss functions as follows:
Revised: We design decentralized convex logistic and nonconvex sigmoid loss functions as follows:

Original: [Equation]
Revised: [Equation]

Original: [Figure]

Original:  where m i is the size of the training data on node i .
Revised: (18) where m i is the size of the training data on node i .

Original: Note that f 1 i , t ( θ ) is smooth, Lipschitz and nonconvex, while f 2 i , t ( θ ) is convex.
Revised: Note that f 1 i , t ( θ ) is smooth, Lipschitz and nonconvex, while f 2 i , t ( θ ) is convex.

Original: For X , we consider the (cid:96) 1 ball X (cid:96) 1 = { θ ∈ R p : (cid:107) θ (cid:107) 1 ≤ r } , when a sparse classiﬁer is preferred.
Revised: For X , we consider the (cid:96) 1 ball X (cid:96) 1 = { θ ∈ R p : (cid:107) θ (cid:107) 1 ≤ r } , when a sparse classiﬁer is preferred.

Original: We show results for the decentralized problem solved by DADAM, C-DADAM, DGD and EXTRA over a medium-scale network.
Revised: We show results for the decentralized problem solved by DADAM, C-DADAM, DGD and EXTRA over a medium-scale network.

Original: We have implemented EXTRA and DGD with their default settings.
Revised: We have implemented EXTRA and DGD with their default settings.

Original: The connected network is randomly generated with n = 50 agents and connectivity ratio r = 0 .
Revised: The connected network is randomly generated with n = 50 agents and connectivity ratio r = 0 .

Original: Each agent holds 10 samples, i.e., m i = 10 , ∀ i  .
Revised: Each agent holds 10 samples, i.e., m i = 10 , ∀ i ∈ V .

Original: The agents shall collaboratively obtain p coefﬁcients via loss functions equation 17.
Revised: The agents shall collaboratively obtain p coefﬁcients via loss functions equation 18.

Original: All samples are randomly generated, and the reference classiﬁer θ ∗ is pre-computed using a centralized method.
Revised: All samples are randomly generated, and the reference classiﬁer θ ∗ is pre-computed using a centralized method.

Original: As it is easy to implement in practice, we use the Metropolis constant edge weight matrix W Shi et al.
Revised: As it is easy to implement in practice, we use the Metropolis constant edge weight matrix W Shi et al.

Original: and we use ˆ W
Revised: and we use ˆ W

Original: =
Revised: =

Original: I + W 2 .
Revised: I + W 2 .

Original: We have tested the DADAM and C-DADAM with ν 1 = ν 2 = ν 3 = 0 .
Revised: We have tested the DADAM and C-DADAM with β 1 = β 2 = β 3 = 0 .

Original: The numerical results are illustrated in Figure 1.
Revised: The numerical results are illustrated in Figure 1.

Original: It can be seen that DADAM and C-DADAM outperform DGD and EXTRA, showing almost linear convergence in the convex setting to the reference logistic classiﬁer x ∗ .
Revised: It can be seen that DADAM and C-DADAM outperform DGD and EXTRA, showing almost linear convergence in the convex setting to the reference logistic classiﬁer x ∗ .

Original: 5. CIFAR AND MNIST EXPERIMENTS
Revised: 5. CIFAR AND MNIST EXPERIMENTS

Original: Next, we present the experimental results using the CIFAR-10 image recognition dataset and the MNIST digit recognition task.
Revised: Next, we present the experimental results using the CIFAR-10 image recognition dataset and the MNIST digit recognition task.

Original: The model architecture for training CNN to classify CIFAR10 dataset was taken from the TensorFlow tutorial 2 .
Revised: The model architecture for training CNN to classify CIFAR10 dataset was taken from the TensorFlow tutorial 2 .

Original: Also, the model for training a simple multilayer percepton (MLP) on the MNIST dataset was taken from https://github.com/keras-team/keras/blob/master .
Revised: Also, the model for training a simple multilayer percepton (MLP) on the MNIST dataset was taken from https://github.com/keras-team/keras/blob/master .

Original: We compare the accuracy of DADAM with that of the decentralized ADAM (Algorithm 1 with ν 3 = 1 ) and the Federated averaging SGD (FedAvg) algorithm McMahan et al.
Revised: We compare the accuracy of DADAM with that of the decentralized ADAM (Algorithm 1 with β 3 = 1 ) and the Federated averaging SGD (FedAvg) algorithm McMahan et al.

Original: (2016) which also performs data parallelization without decentralized computation.
Revised: (2016) which also performs data parallelization without decentralized computation.

Original: The connected network is gener- ated with n = 5 agents and connectivity ratio r = 0 .
Revised: The connected network is generated with n = 5 agents and connectivity ratio r = 0 .

Original: The parameters for DADAM and ADAM are selected in a way similar to the previous experiments.
Revised: The parameters for DADAM and ADAM are selected in a way similar to the previous experiments.

Original: In our implementation, we use same number of agents and choose E = C
Revised: In our implementation, we use same number of agents and choose E = C

Original: = 1 as the parameters in the FedAvg algorithm since it is close to a connected topology scenario as considered in the DADAM and ADAM.
Revised: = 1 as the parameters in the FedAvg algorithm since it is close to a connected topology scenario as considered in the DADAM and ADAM.

Original: It can be easily seen from Figure 2 that DADAM and the decentralized ADAM ( ν 3 = 1) outperform FedAvg.
Revised: It can be easily seen from Figure 2 that DADAM and the decentralized ADAM ( β 3 = 1) outperform FedAvg.

Original: Further,
Revised: Further,

Original: DADAM can achieve high accuracy in comparison with the decentralized ADAM and FedAvg. 
Revised: DADAM can achieve high accuracy in comparison with the decentralized ADAM and FedAvg. [Figure]

Original: [Figure]
Revised: [Figure]

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: A decentralized adaptive moment estimation method DADAM was proposed for the distributed learning of deep networks based on adaptive moment of ﬁrst and second moment of estimations.
Revised: A decentralized adaptive moment estimation method DADAM was proposed for the distributed learning of deep networks based on adaptive moment of ﬁrst and second moment of estimations.

Original: Convergence properties of the proposed algorithm were established for convex (and dynamic) and nonconvex functions in both stochastic and deterministic settings.
Revised: Convergence properties of the proposed algorithm were established for convex (and dynamic) and nonconvex functions in both stochastic and deterministic settings.

Original: Numerical results on some datasets show the efﬁciency and effectiveness of the new proposed method in practice.
Revised: Numerical results on some datasets show the efﬁciency and effectiveness of the new proposed method in practice.

Original: A CKNOWLEDGMENTS

Original: Use unnumbered third level headings for the acknowledgments.

Original: 6. P ROPERTIES OF DADAM
Revised: 6. P ROPERTIES OF DADAM

Original: Lemma 11.
Revised: Lemma 11.

Original: Beck & Teboulle (2003) Let X be a nonempty closed convex set in R p .
Revised: Beck & Teboulle (2003) Let X be a nonempty closed convex set in R p .

Original: Then, for all d ∈ X , we have
Revised: Then, for any d ∈ X , we have

Original: [Equation]
Revised: [Equation]

Original: wherex
Revised: where

Original: [Equation]
Revised: [Equation]

Original: = argmin

Original: [Equation]

Original: Lemma 12.
Revised: Lemma 12.

Original: McMahan & Streeter (2010)
Revised: McMahan & Streeter (2010)

Original: For any A ∈ S p + and convex feasible set C ⊂ R p , suppose a 1 = Π C , A [ b 1 ] , a 2 = Π C ,
Revised: For any A ∈ S p + and convex feasible set C ⊂ R p , suppose a 1 = Π C , A [ b 1 ] , a 2 = Π C ,

Original: A [ b 2 ] , we have
Revised: A [ b 2 ] , we have

Original: [Equation]
Revised: [Equation]

Original: (cid:107) .

Original: Lemma 13.
Revised: Lemma 13.

Original: For all i ∈ V if ν 1 , ν 2 ∈ [ 0 , 1 ) satisfy η = ν 1 √ ν 2 < 1 , then we have
Revised: For all i ∈ V if β 1 , β 2 ∈ [ 0 , 1 ) satisfy η = β 1 √ β 2 < 1 , then we have

Original: [Equation]
Revised: [Equation]

Original: where α t = α √ t for all t ∈ { 1 ,..., T } .
Revised: where α t = α √ t for all t ∈ { 1 ,..., T } .

Original: Proof.
Revised: Proof.

Original: Using the update rules in Algorithm [Equation] we have
Revised: Using the update rules in Algorithm 1, we have

Original: [Equation]
Revised: [Equation]

Original: where (i) follows from the fact that the update rules of m T and υ T can be written as m T
Revised: where (i) follows from the fact that the update rules of m T and υ T can be written as m T

Original: = ( 1 − ν 1 )
Revised: = ( 1 − β 1 )

Original: ∑ Tl = 1 ν T − lg l and υ T =
Revised: ∑ Tl = 1 β T − lg l and υ T =

Original: ( 1 − ν 2 )
Revised: ( 1 − β 2 )

Original: ∑ Tl = 1 ν T − lg 2 l , respectively.
Revised: ∑ Tl = 1 β T − lg 2 l , respectively.

Original: (ii) follows from Cauchy-Schwarz inequality and the fact that 0 ≤ ν 1 < 1.
Revised: (ii) follows from Cauchy-Schwarz inequality and the fact that 0 ≤ β 1 < 1.

Original: Inequality (iii) follows since ∑ Tl = 1 ν T − l≤ 1 1 − ν 1 .
Revised: Inequality (iii) follows since ∑ T l = 1 β T − l≤ 1 1 − β 1 .

Original: Hence, we have
Revised: Hence, we have

Original: [Equation]
Revised: [Equation]

Original: where inequality (i) follows since ∑ Tl =
Revised: where inequality (i) follows since ∑ Tl =

Original: t η l − t ≤ 1 ( 1 − η ) .
Revised: t η l − t ≤ 1 ( 1 − η ) .

Original: Inequality (ii) follows from Cauchy-Schwarz inequality.
Revised: Inequality (ii) follows from Cauchy-Schwarz inequality.

Original: The inequality (iii) follows since 
Revised: The inequality (iii) follows since T

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Next, we provide an upper bound on the deviation of the local estimates at each iteration from their consensual value.
Revised: Next, we provide an upper bound on the deviation of the local estimates at each iteration from their consensual value.

Original: A similar result has been proven in Shahrampour & Jadbabaie (2017) for online decentralized mirror descent; however, the following lemma extends that of Shahrampour &
Revised: A similar result has been proven in Shahrampour & Jadbabaie (2017) for online decentralized mirror descent; however, the following lemma extends that of Shahrampour &

Original: Jadbabaie (2017) to the online adaptive setting and takes into account the sparsity of gradient vector.
Revised: Jadbabaie (2017) to the online adaptive setting and takes into account the sparsity of gradient vector.

Original: Lemma 14.
Revised: Lemma 14.

Original: ( Network Error with Sparse Data)
Revised: ( Network Error with Sparse Data)

Original: If ν 1 , ν 2 ∈ [ 0 , 1 ) satisfy η = ν 1 √ ν 2 < 1 , then the sequence x i , t generated by Algorithm 1 satisﬁes
Revised: If β 1 , β 2 ∈ [ 0 , 1 ) satisfy η = β 1 √ β 2 < 1 , then the sequence x i , t generated by Algorithm 1 satisﬁes

Original: [Equation]
Revised: [Equation]

Original: where ˆ V i , t = diag ( ˆ υ i , t ) and ¯ x t = 1 n ∑ ni = 1 x i , t .
Revised: where ˆ V i , t = diag ( ˆ υ i , t ) and ¯ x t = 1 n ∑ ni = 1 x i , t .

Original: Proof.
Revised: Proof.

Original: Let e i , t : = x i , t + 1 − ∑ nj = 1 [ W ] ij x j , t , where W satisﬁes equation 2.
Revised: Let e i , t : = x i , t + 1 − ∑ nj = 1 [ W ] ij x j , t , where W satisﬁes equation 2.

Original: Using the update rule of x i , t +in Algorithm 1, we have
Revised: Using the update rule of x i , t +in Algorithm 1, we have

Original: [Equation]
Revised: [Equation]

Original: where the ﬁrst inequality follows from Lemma 12.
Revised: where the ﬁrst inequality follows from Lemma 12.

Original: Further, from the deﬁnition of e i , t , we have
Revised: Further, from the deﬁnition of e i , t , we have

Original: [Equation]
Revised: [Equation]

Original: Now, from equation 2,equation 20 and we have:
Revised: Now, from equation 2,equation 21 and we have:

Original: [Equation]
Revised: [Equation]

Original: where ¯ e t = 1 n ∑ ni = 1 e i , t .
Revised: where ¯ e t = 1 n ∑ ni = 1 e i , t .

Original: Hence,
Revised: Hence,

Original: [Equation]
Revised: [Equation]

Original: It follows from equation 20 that
Revised: It follows from equation 21 that

Original: [Equation]
Revised: [Equation]

Revised: Now, using equation 23 and equation

Original: [Equation]
Revised: [Equation]

Original: Now, using equation 22 and equation 21, we have 
Revised: we have [Equation] Now

Original: [Equation]
Revised: [Equation]

Original: Now, taking the Euclidean norm of equation 23 and summing over t ∈ { 1 ,..., T } , one has:
Revised: taking the Euclidean norm of equation 24 and summing over t ∈ { 1 ,..., T } [Equation] one has:

Original: [Equation]
Revised: [Equation]

Original: where inequality (i) is obtained{+follows from (cid:107) ∑ ni = 
 a i (cid:107) 2 ≤ n ∑ ni = 1 (cid:107) a i (cid:107) 2 , step (ii) follows+} from the following property of mixing matrix W Horn et al.
Revised: where step (i) {+follows from (cid:107) ∑ ni = 
 a i (cid:107) 2 ≤ n ∑ ni = 1 (cid:107) a i (cid:107) 2 , step (ii) follows+} from the following property of mixing matrix W Horn et al.

Original: [Equation]
Revised: [Equation]

Original: inequality (ii) follows from equation 19. Finally, the inequality (iii) follows from Lemma 13.
Revised: step (iii) follows from ∑ Tt = 1 t σ t 2 ( W ) < 1 ( 1 − σ 2 ( W )) 2 , step (iv) follows from equation 20 and step (v) follows from Lemma 13.

Original: Now, summing the inequality equation 24 over i ∈ V and using 
Revised: Now, summing the inequality equation 25 over i ∈ V and using [Equation]

Original: [Equation]
Revised: [Equation]

Original: we complete the proof
Revised: we complete the proof.

Original: [Equation]

Original: Lemma 15.
Revised: Lemma 15.

Original: For the sequence x i , t generated by Algorithm [Equation] we have
Revised: For the sequence x i , t generated by Algorithm 1, we have

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: From the left side of equation 27, we have
Revised: From the left side of equation [Equation] we have

Original: [Equation]
Revised: [Equation]

Original: By construction of equation
Revised: By construction of equation

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: where the last inequality holds due to equation1.
Revised: where the last inequality holds due to equation

Original:  Summing equation 29 over t ∈ { 1 ,..., T } , the ﬁrst term telescopes, while the second is handled with equation 30.
Revised: [Equation] Summing equation 30 over t ∈ { 1 ,..., T } , the ﬁrst term telescopes, while the second is handled with equation 31.

Original: Hence,
Revised: Hence,

Original: [Equation]
Revised: [Equation]

Original: where the last inequality follows since from deﬁnition of ˆ υ i [Equation] t 
Revised: where the last inequality follows since from deﬁnition of ˆ υ i , t , we have

Original: [Equation]
Revised: [Equation]

Original: we have [Equation] Now, summing equation 31 over i ∈ V and using the inequality ∑ ni = 1 (cid:112)ˆ υ
Revised: Now, summing equation 32 over i ∈ V and using the inequality ∑ ni = 1 (cid:112)

Original:  i , t ≤ √ n √ ˆ υ t , the claim in equation 27 follows.
Revised: ˆ υ i , t ≤ √ n √ ˆ υ t , the claim in equation 28 follows.

Original: Lemma 16.
Revised: Lemma 16.

Original: Suppose that the parameters ν 1 , ν 2 ∈ [ 0 , 1 ) satisfy η = ν 1 √ ν 2 < 1 .
Revised: Suppose that the parameters β 1 , β 2 ∈ [ 0 , 1 ) satisfy η = β 1 √ β 2 < 1 .

Original: Let ν 1 , t = ν 1 λ t − 1 , λ ∈ ( 0 , 1 ) and (cid:107) ∇ f i , t ( x ) (cid:107) ∞ ≤ G ∞ for all t ∈ { 1 ,..., T } .
Revised: Let β 1 , t = β 1 λ t − 1 , λ ∈ ( 0 , 1 ) and (cid:107) ∇ f i , t ( x ) (cid:107) ∞ ≤ G ∞ for all t ∈ { 1 ,..., T } .

Original: Then, using a step-size α t = α √ t for the sequence x i , t generated by Algorithm 1, we have
Revised: Then, using a step-size α t = α √ t for the sequence x i , t generated by Algorithm 1, we have

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: From convexity of f i
Revised: From convexity of f i

Original: [Equation]
Revised: [Equation]

Original: t ( · )
Revised: t ( · )

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: Individual terms in equation 33 can be bounded in the following way.
Revised: Individual terms in equation 34 can be bounded in the following way.

Original: From the Young’s inequality for products 
Revised: From the Young’s inequality for products 3 , we have

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: Note also that:

Original: [Equation]
Revised: [Equation]

Original: In addition, we have
Revised: Using Cauchy-Schwarz inequality [Equation] equation 19 and equation

Original: [Equation]
Revised: [Equation]

Original: n
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: n [Equation]
Revised: Using equation 35 and equation 36, we have

Original: [Equation]
Revised: [Equation]

Original: n
Revised: In addition

Original: [Equation]
Revised: [Equation]

Original: n n
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: n
Revised: where equation 38 follows from Cauchy-Schwarz inequality

Original: [Equation]
Revised: [Equation]

Original: where equation 35 and equation 36 utilize Cauchy-Schwarz inequality. Inequality equation 37 follows from equation
Revised: Now

Original: [Equation]
Revised: [Equation]

Original: Lemma 14 and the following inequality:
Revised: using equation 38 we obtain

Original: [Equation]
Revised: [Equation]

Original: hence
Revised: where equation 39 utilize Cauchy-Schwarz inequality and equation 40 follows from equation 2, Lemma 14 and equation

Original: [Equation]
Revised: [Equation]

Original: from Cauchy-Schwarz inequality
Revised: To bound I 1 , using the update rule of ˆ m i

Original: [Equation]
Revised: [Equation]

Original: equation 18 and equation 26 we get
Revised: t in Algorithm

Original: [Equation]
Revised: [Equation]

Original: To bound I 1 , using the update rule of m i , t in Algorithm 1, we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: Now, by rearranging the above equality [Equation] one has:
Revised: Now, by rearranging the above equality, we obtain:

Original: [Equation]
Revised: [Equation]

Original: where ( II 1 ) follows from Lemma 11.
Revised: where ( II 1 ) follows from Lemma 11.

Original: The term ( II 2 ) is obtained by induction as follows: using the assumption, (cid:107) g
Revised: The term ( II 2 ) is obtained by induction as follows: using the assumption, (cid:107) g

Original: i , t (cid:107) ∞
Revised: i , t (cid:107) ∞

Original: ≤ G ∞ ; now, using the update rule of m i , t in Algorithm1, we have
Revised: ≤ G ∞ ; now, using the update rule of m i , t in Algorithm

Original: [Equation]
Revised: [Equation]

Revised: we have

Original: [Equation]
Revised: [Equation]

Original:  where (cid:107) m i , t − 1 (cid:107) ∞ ≤ G ∞ by induction hypothesis.
Revised: ) where (cid:107) m i , t − 1 (cid:107) ∞ ≤ G ∞ by induction hypothesis.

Original: Note that from our assumption, we have ν 1 , t = ν 1 λ t − 1 , λ ∈ ( 0 , 1 ) and ν 1 , t ≤ ν 1 .
Revised: Note that from our assumption [Equation] we have β 1 , t = β 1 λ t − 1 , λ ∈ ( 0 , 1 ) and β 1 , t ≤ β 1 .

Original: Substituting this into equation39, we get
Revised: Substituting this into equation

Original: [Equation]
Revised: [Equation]

Original: where the last line comes from equation
Revised: we get

Original: [Equation]
Revised: [Equation]

Original: Plugging equation 
Revised: where the last line comes from equation 1.

Original: [Equation] equation 37, equation 38 and equation 41 into equation [Equation] we obtain
Revised: Plugging equation 37, equation 40, and equation 43 into equation 34, we obtain

Original: [Equation]
Revised: [Equation]

Original: Now, since ν 1 , t = ν 1 λ t − 1 , λ ∈ ( 0 , 1 ) and ν 1 , t ≤ ν 1 , we have
Revised: Now, since β 1 , t = β 1 λ t − 1 , λ ∈ ( 0 , 1 ) and β 1 , t ≤ β 1 , we have

Original: [Equation]
Revised: [Equation]

Original: Finally, using equation 42 and inequality ∑ ni = 1 (cid:112) ˆ υ
Revised: [Equation]

Original: i , T , d ≤ √ n (cid:112) ˆ υ t , d , we obtain the desired result.
Revised: Finally, using Lemma 15 and inequality equation 44, we obtain the desired result.

Original: 6.1. P ROOF OF T HEOREM
Revised: 6.1. P ROOF OF T HEOREM

Original: Proof.
Revised: Proof.

Original: From convexity of f i , t ( · ) it follows that
Revised: From convexity of f i , t ( · ) it follows that

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: Summing over t ∈ { 1 ,..., T } and i ∈ V , and applying Lemma 16 and equation 37 gives the desired result.
Revised: Summing over t ∈ { 1 ,..., T } and i ∈ V [Equation] and applying Lemma 16 and equation 40 gives the desired result.

Original: 6.1. P ROOF OF T HEOREM
Revised: 6.1. P ROOF OF T HEOREM

Original: Proof.
Revised: Proof.

Original: We just need to rework the proof of Lemma 16 and Theorem 4 using stochastic gradients by tracking the changes.
Revised: We just need to rework the proof of Lemma 16 and Theorem 4 using stochastic gradients by tracking the changes.

Original: Indeed, using stochastic gradient at the beginning of Lemma
Revised: Indeed, using stochastic gradient at the beginning of Lemma

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: Further
Revised: Further

Original: [Equation]
Revised: [Equation]

Original: if we replace any bound involving G ∞ which is an upper bound on the exact gradient with the norm of stochastic gradient
Revised: if we replace any bound involving G ∞ which is an upper bound on the exact gradient with the norm of stochastic gradient

Original: [Equation]
Revised: [Equation]

Original: we obtain
Revised: we obtain

Original: [Equation]
Revised: [Equation]

Original: Note that using Assumption
Revised: Note that using Assumption

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: and

Original: [Equation]

Original: Hence taking expectation from equation [Equation] using the above inequality and equation [Equation]
Revised: Hence taking expectation from equation 46, using the above inequality and equation 42, we have

Original: we have

Original: [Equation]
Revised: [Equation]

Original: According to the result in Lemma 15 and equation
Revised: According to the result in Lemma 15 and equation

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: Finally, using equation 43 we complete the proof.
Revised: Finally, using equation 45 we complete the proof.

Original: Next, we establish a series of lemmas used in the proof of Theorem 7 and 9.
Revised: Next, we establish a series of lemmas used in the proof of Theorem 7 and 9.

Original: Lemma 17.
Revised: Lemma 17.

Original: Attouch et al.
Revised: Attouch et al.

Original: Let G X be the projected gradient deﬁed in equation 5.
Revised: Let G X be the projected gradient deﬁed in equation 5.

Original: Then,
Revised: Then,

Original: G X ( ϖ , f i , α ) = 

Revised: G X ( ϖ , f i , α ) =

Original: if and only if ϖ is a critical point of equation 3.
Revised: if and only if ϖ is a critical point of equation 3.

Original: Next, we show that the projection map G X in Deﬁnition 1 is Lipschitz continuous.
Revised: Next, we show that the projection map G X in Deﬁnition 1 is Lipschitz continuous.

Original: Lemma 18.
Revised: Lemma 18.

Original: Suppose the second moment ˆ υ satisﬁes equation 10.
Revised: Suppose the second moment ˆ υ satisﬁes equation 11.

Original: Let x + 1 , i and x + 2 , i be given in
Revised: Let x + 1 , i and x + 2 , i be given in

Original: Deﬁnition 1 with m i √
Revised: Deﬁnition 1 with m i √

Original: ˆ υ i replaced by m 1 i √
Revised: ˆ υ i replaced by m 1 i √

Original: ˆ υ 1 i and m 2 i √
Revised: ˆ υ 1 i and m 2 i √

Original: ˆ υ 2 i , respectively.
Revised: ˆ υ 2 i , respectively.

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: where G 1 X and G 2 X are the projection maps corresponding to x + 1 , i and x + 2 , i , respectively.
Revised: where G 1 X and G 2 X are the projection maps corresponding to x + 1 , i and x + 2 , i , respectively.

Original: Proof.
Revised: Proof.

Original: Consider the optimality condition of equation 6, for any u ∈ X .
Revised: Consider the optimality condition of equation 6, for any u ∈ X .

Original: For each i ∈ V , observe that
Revised: For each i ∈ V , observe that

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: Taking u = x + [Equation] i in equation [Equation]
Revised: Taking u = x + 2 , i in equation 47, we have

Original: we have

Original: [Equation]
Revised: [Equation]

Original: Likewise [Equation] setting u = x + [Equation]
Revised: Likewise, setting u = x + 1 , i in equation

Original: i in equation

Original: [Equation]
Revised: [Equation]

Original: we get
Revised: we get

Original: [Equation]
Revised: [Equation]

Original:  Now, using equation 47, equation 48 and equation 10, we obtain 
Revised: [Equation] Now, using equation 49, equation 50 and equation 11, we obtain [Equation]

Original: [Equation]
Revised: [Equation]

Original: Without loss of generality, assumes that (cid:113) ˆ υ 1 i ≤ (cid:113) ˆ
Revised: Without loss of generality, assumes that (cid:113) ˆ υ 1 i ≤ (cid:113) ˆ

Original: υ 2 i .
Revised: υ 2 i .

Original: Then, using equation 5, we have
Revised: Then, using equation 5, we have

Original: [Equation]
Revised: [Equation]

Original: where (i) follows from equation 10 and (ii) follows from equation 49.
Revised: where (i) follows from equation 11 and (ii) follows from equation [Equation]

Original: Lemma 19.
Revised: Lemma 19.

Original: Let ν 1 , ν 2 ∈ [ 0 , 1 ) satisfy η = ν 1 √ ν 2 < 1 .
Revised: Let β 1 , β 2 ∈ [ 0 , 1 ) satisfy η = β 1 √ β 2 < 1 .

Original: Then, for the sequence x i , t generated by Algorithm 1, we have
Revised: Then, for the sequence x i , t generated by Algorithm 1, we have

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: In the proof of Lemma 14 we showed that
Revised: In the proof of Lemma 14 we showed that

Original: [Equation]
Revised: [Equation]

Original: Now, using the above equality, we have
Revised: Now, using the above equality, we have

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Also, using Lemma 13, we get
Revised: Also, using Lemma [Equation] we get

Original: [Equation]
Revised: [Equation]

Original: The above inequality together with the update rule of x i , t + 1 , imply that
Revised: The above inequality together with the update rule of x i , t + 1 , imply that

Original: [Equation]
Revised: [Equation]

Original: where the ﬁrst inequality follows from the nonexpansiveness property of the Euclidean projection 4 .
Revised: where the ﬁrst inequality follows from the nonexpansiveness property of the Euclidean projection [Equation]

Original: Substituting equation 25 and equation 52 into equation
Revised: Substituting equation 26 and equation 54 into equation

Original: [Equation]
Revised: [Equation]

Original: we get
Revised: we get

Original: [Equation]
Revised: [Equation]

Original: Summing the above inequality over i ∈ V
Revised: Summing the above inequality over i ∈ V

Original: [Equation]
Revised: [Equation]

Original: we conclude that
Revised: we conclude that

Original: [Equation]
Revised: [Equation]

Original: Lemma 20.
Revised: Lemma 20.

Original: For the sequence x i , t generated by Algorithm 1, we have
Revised: For the sequence x i , t generated by Algorithm 1, we have

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: A quick look at optimality condition of equation 6 veriﬁes that
Revised: A quick look at optimality condition of equation 6 veriﬁes that

Original: [Equation]
Revised: [Equation]

Original: Substituting z
Revised: Substituting z

Original: = x i , t into the above inequality
Revised: = x i , t into the above inequality

Original: [Equation]
Revised: [Equation]

Original: we get
Revised: we get

Original: [Equation]
Revised: [Equation]

Original: Now using equation
Revised: Now using equation

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: where the ﬁrst equality follows from the update rule of m i , t .
Revised: where the ﬁrst equality follows from the update rule of m i , t .

Original: The last inequality is valid since CauchySchwarz inequality and inequality (cid:112)
Revised: The last inequality is valid since CauchySchwarz inequality and inequality (cid:112)

Original: ˆ υ
Revised: ˆ υ

Original: i , t − 1 ν 1 , t ≤ (cid:112)
Revised: i , t − 1 β 1 , t ≤ (cid:112)

Original: ˆ υ
Revised: ˆ υ

Original: i , t ν 1 .
Revised: i , t β 1 .

Original: The claim then follows after using Deﬁnition
Revised: The claim then follows after using Deﬁnition

Original: [Equation]
Revised: [Equation]

Original: 6.1. P ROOF OF T HEOREM
Revised: 6.1. P ROOF OF T HEOREM

Original: Proof.
Revised: Proof.

Original: Let ¯ f i 
Revised: Let ¯ f i , t

Original: [Equation]

Original: t ( x i , t ) = 1 t ∑ ts = 1 f i , s ( x i , t ) .
Revised: ( x i , t ) = 1 t ∑ ts = 1 f i , s ( x i , t ) .

Original: Using Assumption 2, Taylor’s expansion and Deﬁnition 1, we have
Revised: Using Assumption 2, Taylor’s expansion and Deﬁnition 1, we have

Original: [Equation]
Revised: [Equation]

Original: The above inequality together with Lemma 20, equation 10 and ν 1 , t ≤ ν 1 , imply that
Revised: The above inequality together with Lemma 20, equation 11 and β 1 , t ≤ β 1 , imply that

Original: [Equation]
Revised: [Equation]

Original: Let ∆ i

Original: [Equation]

Original: t =

Original: ¯ f i

Original: [Equation] t ( x i [Equation] t ) − ¯ f i , t ( x ∗ t ) denotes the instantaneous loss at round t .
Revised: Let ∆ i , t = ¯ f i , t ( x i , t ) − ¯ f i , t ( x ∗ t ) denotes the instantaneous loss at round t .

Original: We have
Revised: We have

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Note that ( I ) can be bounded as follows:
Revised: Note that ( I ) can be bounded as follows:

Original: [Equation]
Revised: [Equation]

Original: where the ﬁrst inequality is due to the optimality of x ∗ t and the second inequality follows from equation 53.
Revised: where the ﬁrst inequality is due to the optimality of x ∗ t and the second inequality follows from equation 55.

Original: Now, combining equation 55 and equation 54 gives
Revised: Now, combining equation 57 and equation 56 gives

Original: [Equation]
Revised: [Equation]

Original: By rearranging the above inequality
Revised: By rearranging the above inequality

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: Using the deﬁnition of ∆ i
Revised: Using the deﬁnition of ∆ i

Original: [Equation]
Revised: [Equation]

Original: t +
Revised: t +

Original: [Equation]
Revised: [Equation]

Original: we get 1 t
Revised: we get 1 t

Original: + 1 ( f i , t + 1 ( x i , t + 1 ) − f i [Equation]
Revised: + 1 ( f i , t + 1 ( x i , t + 1 ) − f i , t + 1 ( x ∗ t + 1 ))

Original: t + 1 ( x ∗ t + 1 )) − ∆ i , t + 1 = − ( t t + 1 )( ¯ f i , t ( x i , t + 1 ) − ¯ f i , t ( x ∗ t + 1 )) .
Revised: − ∆ i , t + 1 = − ( t t + 1 )( ¯ f i , t ( x i , t + 1 ) − ¯ f i , t ( x ∗ t + 1 )) .

Original: Hence, using equation 56 and simplifying terms
Revised: Hence, using equation 58 and simplifying terms

Original: [Equation]
Revised: [Equation]

Original: we obtain
Revised: we obtain

Original: [Equation]
Revised: [Equation]

Original: Observe that:
Revised: Note that:

Original: [Equation]
Revised: [Equation]

Original: where the ﬁrst equality uses
Revised: where the ﬁrst equality uses

Original: [Equation]
Revised: [Equation]

Original: The ﬁrst inequality follows from Assumption 2.
Revised: The ﬁrst inequality follows from Assumption 2.

Original: The second inequality follows from Lemma 19 and equation
Revised: The second inequality follows from Lemma 19 and equation

Original: [Equation]
Revised: [Equation]

Original: Summing equation 57 over i ∈ V , t ∈ { 1 ,..., T } and using equation
Revised: Summing equation 59 over i ∈ V , t ∈ { 1 ,..., T } and using equation

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: Note that ∑ Tt = 1 [ ( 2 − ν 1 ) α t 2 ¯ υ − ρα 2 t 2 υ 2 ] > 0.
Revised: Note that ∑ Tt = 1 [ ( 2 − β 1 ) α t 2 ¯ υ − ρα 2 t 2 υ 2 ] > 0.

Original: Therefore, dividing both sides of the equation 60 by ∑ Tt = 1 [ ( 2 − ν 1 ) α t 2 ¯ υ − ρα 2 t 2 υ 2 ] , we obtain equation 13.
Revised: Therefore, dividing both sides of the equation 62 by ∑ Tt = 1 [ ( 2 − β 1 ) α t 2 ¯ υ − ρα 2 t 2 υ 2 ] , we obtain equation 14.

Original: 6.1. P ROOF OF C OROLLARY
Revised: 6.1. P ROOF OF C OROLLARY

Original: Proof.
Revised: Proof.

Original: With the constant step-sizes α t = ( 2 − ν 1 ) υ 2 2 ρ ¯ υ for all t ∈ { 1 ,..., T } , we have
Revised: With the constant step-sizes α t = ( 2 − β 1 ) υ 2 2 ρ ¯ υ for all t ∈ { 1 ,..., T } , we have

Original: [Equation]
Revised: [Equation]

Original: t
Revised: t

Original: [Equation]
Revised: [Equation]

Original: Therefore, using the above equality and equation 11 together with α t = ( 2 − ν 1 ) υ 2 

Revised: Therefore, using the above equality and equation 12 together with α t = ( 2 − β 1 ) υ 2

Original: ρ ¯ υ for all t ∈ { 1 ,..., T } , we obtain
Revised: ρ ¯ υ for all t ∈ { 1 ,..., T } , we obtain

Original: [Equation]
Revised: [Equation]

Original: 61) and
Revised: 63) and

Original: [Equation]
Revised: [Equation]

Original: Combine the results in equation 61 and equation
Revised: Combine the results in equation 63 and equation

Original: [Equation]
Revised: [Equation]

Original: we have equation 12.
Revised: we have equation 13.

Original: 6.1. P ROOF OF T HEOREM
Revised: 6.1. P ROOF OF T HEOREM

Original: Proof.
Revised: Proof.

Original: Let
Revised: Let

Original: [Equation]
Revised: [Equation]

Original: where ∇ ¯ f i , t ( x i , t ) = 1 t ∑ ts = 1 ∇ f i , s ( x i , t ) denotes the minibatch stochastic gradient on node i .
Revised: where ∇ ¯ f i , t ( x i , t ) = 1 t ∑ ts = 1 ∇ f i , s ( x i , t ) denotes the minibatch stochastic gradient on node i .

Original: Since f i , t is ρ -smooth
Revised: Since f i , t is ρ -smooth

Original: [Equation]
Revised: [Equation]

Original: it follows that ¯ f i , t is also ρ -smooth.
Revised: it follows that ¯ f i , t is also ρ -smooth.

Original: Hence for every i ∈ V and t ∈ { 1 ,..., T } , we have
Revised: Hence for every i ∈ V and t ∈ { 1 ,..., T } , we have

Original: [Equation]
Revised: [Equation]

Original: where G X ( x i , t , ¯ f i , t , α t ) denotes the projected stochastic gradient on node i .
Revised: where G X ( x i , t , ¯ f i , t , α t ) denotes the projected stochastic gradient on node i .

Original: The above inequality together with Lemma 20, equation 10 and ν 1 , t ≤ ν 1 , imply that
Revised: The above inequality together with Lemma 20, equation 11 and β 1 , t ≤ β 1 , imply that

Original: [Equation]
Revised: [Equation]

Original: where the second inequality follows from the Cauchy-Schwartz inequality.
Revised: where the second inequality follows from the Cauchy-Schwartz inequality.

Original: The last inequality follows from the fact that ν 1 , t = ν 1 λ t − 1 , λ ∈ ( 0 , 1 ) and Lemma 18 if we set G 1 X ( x i , ¯ f i , α ) =
Revised: The last inequality follows from the fact that β 1 , t = β 1 λ t − 1 , λ ∈ ( 0 , 1 ) and Lemma 18 if we set G 1 X ( x i , ¯ f i , α ) =

Original: G X ( x i , t , ¯ f i , t , α t ) , and G 2 X ( x i , ¯ f i , α ) =
Revised: G X ( x i , t , ¯ f i , t , α t ) , and G 2 X ( x i , ¯ f i , α ) =

Original: G X ( x i , t , ¯ f i , t , α t ) .
Revised: G X ( x i , t , ¯ f i , t , α t ) .

Original: Let ∆ i , t = ¯ f i , t ( x i , t ) − ¯ f i , t ( x ∗ t ) denotes the instantaneous stochastic loss at time t .
Revised: Let ∆ i , t = ¯ f i , t ( x i , t ) − ¯ f i , t ( x ∗ t ) denotes the instantaneous stochastic loss at time t .

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Revised: i

Revised: [Equation]

Revised: [Equation]

Original: Observe that ( I ) can be bounded as follows:
Revised: Observe that ( I ) can be bounded as follows:

Original: [Equation]
Revised: [Equation]

Original: where the ﬁrst inequality is due to x ∗ t + 1 ∈ X and the optimality of x ∗ t and the second inequality is due to equation 63 and the fact that 0 ≤ ν 1 , r < 1.
Revised: where the ﬁrst inequality is due to x ∗ t + 1 ∈ X and the optimality of x ∗ t and the second inequality is due to equation 65 and the fact that 0 ≤ β 1 , r < 1.

Original: Thus, using equation 64 and equation
Revised: Thus, using equation 66 and equation

Original: [Equation]
Revised: [Equation]

Original: we get ∆ i , t + 1 ≤ 
Revised: we get ∆ i , t + 1 ≤ t t +

Original: [Equation]
Revised: [Equation]

Original: By rearranging the above inequality, we have t 
Revised: By rearranging the above inequality, we have t t + 1 [

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Revised: Now, using deﬁnition of ∆ i , t + 1 , 1 t + 1 ( f i , t + 1 ( x i , t + 1 ) − f i , t +

Original: [Equation]
Revised: [Equation]

Original: Now, using deﬁnition of ∆ i , t + 1 , 1 t + 1 ( f i , t + 1 ( x i , t + 1 ) − f i , t + 1 ( x ∗ t + 1 )) − ∆ i , t + 1 = − ( t t + 1 )( ¯ f i , t ( x i , t + 1 ) − ¯ f i , t ( x ∗ t + 1 )) , together with equation 66, we obtain [
Revised: x ∗ t + [Equation] ) − ∆ i , t + 1 = − ( t t + 1 )( ¯ f i , t ( x i , t + 1 ) − ¯ f i , t ( x ∗ t + 1 )) , together with equation 68, we obtain [

Original: [Equation]
Revised: [Equation]

Original: α t
Revised: t

Original: [Equation]
Revised: [Equation]

Original: Note 
Revised: Note that from

Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: we
Revised: we

Revised: [Equation]

Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: which implies that
Revised: which implies that

Original: [Equation]
Revised: [Equation]

Original: The above inequality together with Cauchy-Schwarz for expextations
Revised: The above inequality together with Cauchy-Schwarz for expextations

Original: [Equation]
Revised: [Equation]

Original: imply that
Revised: imply that

Original: [Equation]
Revised: [Equation]

Original: [Equation] Now, using equation 68 and equation 59, summing equation 67 over i ∈ V and t ∈ { 1 ,..., T } [Equation] and taking expectation
Revised: Now, using equation 70 and equation 61, summing equation 69 over i ∈ V and t ∈ { 1 ,..., T } , and taking expectation, we obtain

Original: [Equation]
Revised: [Equation]

Original: we obtain [Equation] Note that ∑ Tt = 1 [ ( 2 − 3 ν 1 ) α t 2 ¯ υ − 
Revised: Note that ∑ Tt = 1 [ ( 2 − 3 β 1 ) α t 2 ¯ υ − ρα 2 t 2 υ 2 ] > 0 , and

Original: [Equation]
Revised: [Equation]

Original: ] >

Original: [Equation]
Revised: [Equation]

Original: and [Equation] [Equation] Therefore, dividing both sides of the equation 71 by ∑ Tt = 1 [ ( 2 − 3 ν 1 ) α t 2 ¯ υ 
Revised: Therefore, dividing both sides of the equation 74 by ∑ Tt = 1 [ ( 2 − 3 β 1 ) α t 2 ¯ υ −

Original: [Equation]
Revised: [Equation]

Original:  ] , using equation 70, we complete the proof.
Revised: 2 ] , using equation 72, we complete the proof.

Original: 6.1. P ROOF OF C OROLLARY
Revised: 6.1. P ROOF OF C OROLLARY

Original: Proof.
Revised: Proof.

Original: Recall equation{+Let ϑ t = ∑ Tt = 
+}
Revised: {+Let ϑ t = ∑ Tt = 
+}

Original: [Equation] equation 67 and equation 68. These expressions lead to the following inequality:
Revised: [ ( 2 − β 1 ) α t 2 ¯

Original: [Equation]
Revised: [Equation]

Original: We ﬁrst show ϑ t =

Original: [-∑ Tt = 
 [ ( 2-] − ν 1 )
Revised: [-∑ Tt = 
 [ ( 2-] − ρα 2 t 2 υ 2 ] .

Original: α t 2 ¯ υ − ρα 2 t 2 υ 2 ] is approximately constant when equation 15b is satisﬁed.

Original: Observe that 
Revised: We claim that if

Original: [Equation]
Revised: [Equation]

Original: which mean that
Revised: then ϑ t ≥ 1 / 2.

Original: [Equation] equivalently [Equation]
Revised: One can easily seen that T must satisfy ( 2 − β 1 )

Original: The requirements on α t are given by making each of the last two terms smaller than 12 :
Revised: α t 2 ¯ υ − ρα 2 t 2 υ 2 ≥ 12 T which results in

Original: [Equation]
Revised: [Equation]

Original: By selecting α t = 1 √ nT , we have
Revised: By rearranging the above inequality, we obtain

Original: [Equation]
Revised: [Equation]

Original: ϑ t ≥ 1  2 will be satisﬁed. We proceed by substituting α t = α √ nT , ν 1  ,  t =
Revised: Now, if A 1 , A 2 ≤ 12 , then α t ≤ υ 2 ( 2 − β 1 ) 2 ρ ¯ υ , and α t ≥ 2 ¯ υ T ( 2 − β 1 ) , respectively. These bounds together with our assumption on α t give equation 73. Now, let equation 73 holds.

Original: ν 1 λ t − 1 , λ ∈ ( 0 , 1 ) and  ϑ t ≥ 1into equation 71 and using Lemma 19, yields
Revised: Using equation [Equation] and since ϑ t ≥ 1 / [Equation]

Revised: we have

Original: [Equation]
Revised: [Equation]

Original:  and 
Revised: where the last inequality follows from equation 60 and equation 70. We proceed to bound RHS of equation 74. By substituting α t = α √ nT and β 1 , t = β 1 λ t − 1 , λ ∈ ( 0 , 1 ) into the last two terms of equation 74, we have

Original: [Equation]
Revised: [Equation]

Original: also
Revised: and

Original: [Equation]
Revised: [Equation]

Original: Given equation 15a, the equation 73 and equation 74 are bounded by equation 72.
Revised: Further, using equation 74 and Lemma [Equation]

Original: Moreover
Revised: we get

Original: [Equation]
Revised: [Equation]

Original: Tn
Revised: Tn

Original: [Equation]
Revised: [Equation]

Original: where the ﬁrst inequality follows from Lemma 19.
Revised: [Equation] Similarly, using equation 74 and Lemma [Equation] we have

Original: The second inequality is due to Cauchy-Schwarz inequality.
Revised: [Equation]

Original: The last inequality holds because ∑ Tt = 1 1 t 2 ≤ 2 − 1 T ≤ 2.
Revised: where the second inequality follows because ∑ Tt = 2 1 t 2 < 1.

Revised: Using equation 16a, the equation 76, equation 77, equation 78 are bounded by equation 75.

Original: This completes the proof.
Revised: This completes the proof.

Revised: 6. S ENSITIVITY OF DADAM TO ITS PARAMETERS

Revised: Next, we examine the sensitivity of DADAM on the parameters related to the network connection and update of the moment estimate.

Revised: As it is clear from Figure 3 the convergence of average accuracy value happens faster for sparser networks (higher second largest singular value).

Revised: This is similar to the trend observed for FedAvg algorithm while reducing parameter C which makes the agent interaction matrix sparser.

Revised: This is also expected as discussed in Theorems 4 and 5.

Revised: Note that, with the availability of a central parameter server (as in federated averaging), sparser topology may be useful for a faster convergence, however, topology density is important for a distributed learning scheme with decentralized computation.

Revised: We also empirically evaluate the effect of the β 3 in Algorithm 1.

Revised: We consider a range of hyperparameter choices, i.e. β 3 ∈ { 0 , 0 .

Revised: From Figure 4 it can be easily seen that DADAM performs equal or better than AMSGrad ( β 3 = 0), regardless of hyper-parameter setting for β 1 and β 2 .