Original: DADAM: A CONSENSUS - BASED DISTRIBUTED ADAPTIVE GRADIENT METHOD FOR O NLINE O PTIMIZATION
Revised: DADAM: A CONSENSUS - BASED DISTRIBUTED ADAPTIVE GRADIENT METHOD FOR O NLINE O PTIMIZATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Online and stochastic optimization methods such as SGD, ADAGRAD and
Revised: Online and stochastic optimization methods such as SGD, ADAGRAD and

Original: ADAM are key algorithms in solving large-scale machine learning problems including deep learning.
Revised: ADAM are key algorithms in solving large-scale machine learning problems including deep learning.

Original: A number of schemes that are based on communications of nodes with a central server have been recently proposed in the literature to parallelize them.
Revised: A number of schemes that are based on communications of nodes with a central server have been recently proposed in the literature to parallelize them.

Original: A bottleneck of such centralized algorithms lies on the high communication cost incurred by the central node.
Revised: A bottleneck of such centralized algorithms lies on the high communication cost incurred by the central node.

Original: In this paper, we present a new consensus-based distributed adaptive moment estimation method (DADAM) for online optimization over a decentralized network that enables data parallelization, as well as decentralized computation.
Revised: In this paper, we present a new consensus-based distributed adaptive moment estimation method (DADAM) for online optimization over a decentralized network that enables data parallelization, as well as decentralized computation.

Original: Such a framework not only can be extremely useful for learning agents with access to only local data in a communication constrained environment, but as shown in this work also outperform centralized adaptive algorithms such as ADAM for certain realistic classes of loss functions.
Revised: Such a framework not only can be extremely useful for learning agents with access to only local data in a communication constrained environment, but as shown in this work also outperform centralized adaptive algorithms such as ADAM for certain realistic classes of loss functions.

Original: We analyze the convergence properties of the proposed algorithm and provide a dynamic regret bound on the convergence rate of adaptive moment estimation methods in both stochastic and deterministic settings.
Revised: We analyze the convergence properties of the proposed algorithm and provide a dynamic regret bound on the convergence rate of adaptive moment estimation methods in both stochastic and deterministic settings.

Original: Empirical results demonstrate that DADAM works well in practice and compares favorably to competing online optimization methods.
Revised: Empirical results demonstrate that DADAM works well in practice and compares favorably to competing online optimization methods.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Online optimization is a fundamental procedure for solving a wide range of machine learning problems Shalev-Shwartz et al.
Revised: Online optimization is a fundamental procedure for solving a wide range of machine learning problems Shalev-Shwartz et al.

Original: ; Hazan et al.
Revised: ; Hazan et al.

Original: It can be formulated as a repeated game between a learner (algorithm) and an adversary.
Revised: It can be formulated as a repeated game between a learner (algorithm) and an adversary.

Original: The learner receives a streaming data sequence, sequentially selects actions, and the adversary reveals the convex or nonconvex losses to the learner.
Revised: The learner receives a streaming data sequence, sequentially selects actions, and the adversary reveals the convex or nonconvex losses to the learner.

Original: A standard performance metric for an online algorithm is regret , which measures the performance of the algorithm versus a static benchmark Zinkevich (2003); Hazan et al.
Revised: A standard performance metric for an online algorithm is regret , which measures the performance of the algorithm versus a static benchmark Zinkevich (2003); Hazan et al.

Original: For example, the benchmark could be an optimal point of the online average of the loss (local cost) function, had the learner known all the losses in advance.
Revised: For example, the benchmark could be an optimal point of the online average of the loss (local cost) function, had the learner known all the losses in advance.

Original: In a broad sense, if the benchmark is a ﬁxed sequence, the regret is called static .
Revised: In a broad sense, if the benchmark is a ﬁxed sequence, the regret is called static .

Original: Recent work on online optimization has investigated the notion of dynamic regret Zinkevich (2003);
Revised: Recent work on online optimization has investigated the notion of dynamic regret Zinkevich (2003);

Original: Hall & Willett (2015); Besbes et al. (2015).
Revised: Hall & Willett (2015); Besbes et al. (2015).

Original: Dynamic regret can take the form of the cumulative difference between the instantaneous loss and the minimum loss.
Revised: Dynamic regret can take the form of the cumulative difference between the instantaneous loss and the minimum loss.

Original: For convex functions, previous studies have shown that the dynamic regret of online gradient-based methods can be upper bounded by O ( √ TD T ) , where D T is a measure of regularity of the comparator sequence or the function sequence Zinkevich (2003); Hall & Willett (2015).
Revised: For convex functions, previous studies have shown that the dynamic regret of online gradient-based methods can be upper bounded by O ( √ TD T ) , where D T is a measure of regularity of the comparator sequence or the function sequence Zinkevich (2003); Hall & Willett (2015).

Original: This bound can be improved to
Revised: This bound can be improved to

Original: O ( D T ) Mokhtari et al.
Revised: O ( D T ) Mokhtari et al.

Original: ; Zhang et al.
Revised: ; Zhang et al.

Original: (2017) when all the cost functions are strongly convex and smooth.
Revised: (2017) when all the cost functions are strongly convex and smooth.

Original: Decentralized nonlinear programming has received a great deal of interest in diverse scientiﬁc and engineering ﬁelds Tsitsiklis et al.
Revised: Decentralized nonlinear programming has received a great deal of interest in diverse scientiﬁc and engineering ﬁelds Tsitsiklis et al.

Original: ; Li et al.
Revised: ; Li et al.

Original: ; Rabbat & Nowak (2004); Lesser et al.
Revised: ; Rabbat & Nowak (2004); Lesser et al.

Original: The key problem involves optimizing a cost function f ( x ) = 1 n ∑ ni = 1 f i ( x ) , where x ∈ R p and each f i is only known to the individual agent i in a connected network of n agents.
Revised: The key problem involves optimizing a cost function f ( x ) = 1 n ∑ ni = 1 f i ( x ) , where x ∈ R p and each f i is only known to the individual agent i in a connected network of n agents.

Original: The agents collaborate by successively sharing information with other agents located in their neighborhood with the goal of jointly converging to the network-wide optimal argument Nedic & Ozdaglar (2009).
Revised: The agents collaborate by successively sharing information with other agents located in their neighborhood with the goal of jointly converging to the network-wide optimal argument Nedic & Ozdaglar (2009).

Original: Compared to optimization procedures involving a fusion center that collects data and performs the computation, decentralized nonlinear programming enjoys the advantage of scalability to network sizes, robustness to dynamic topologies, and privacy preservation in data-sensitive applications Yuan et al.
Revised: Compared to optimization procedures involving a fusion center that collects data and performs the computation, decentralized nonlinear programming enjoys the advantage of scalability to network sizes, robustness to dynamic topologies, and privacy preservation in data-sensitive applications Yuan et al.

Original: ; Shi et al.
Revised: ; Shi et al.

Original: ; Jiang et al.
Revised: ; Jiang et al.

Original: (2017); Lian et al.
Revised: (2017); Lian et al.

Original: Appropriately choosing the learning rate that scale coordinates of the gradient and the way of updating them are crucial issues driving the performance of ﬁrst Duchi et al.
Revised: Appropriately choosing the learning rate that scale coordinates of the gradient and the way of updating them are crucial issues driving the performance of ﬁrst Duchi et al.

Original: ; Kingma & Ba (2014) and second order optimization procedures Peyghami & Tarzanagh (2015).
Revised: ; Kingma & Ba (2014) and second order optimization procedures Peyghami & Tarzanagh (2015).

Original: Indeed, the understanding that adaptation of the learning rate is advantageous, particularly on a per parameter basis dynamically, led to the development of a family of widely-used adaptive gradient methods including
Revised: Indeed, the understanding that adaptation of the learning rate is advantageous, particularly on a per parameter basis dynamically, led to the development of a family of widely-used adaptive gradient methods including

Original: ADAGRAD Duchi et al.
Revised: ADAGRAD Duchi et al.

Original: (2011), RMSPROP Tieleman & Hinton, and ADAM Kingma & Ba (2014).
Revised: (2011), RMSPROP Tieleman & Hinton, and ADAM Kingma & Ba (2014).

Original: The ADAM optimizer computes adaptive learning rates for different parameters from estimates of ﬁrst and second moments of the gradients and performs a local optimization.
Revised: The ADAM optimizer computes adaptive learning rates for different parameters from estimates of ﬁrst and second moments of the gradients and performs a local optimization.

Original: Numerical results show that ADAM can achieve signiﬁcantly better performance compared to ADAGRAD, RMSPROP and other gradient descent procedures when the gradients are sparse, or in general small in magnitude.
Revised: Numerical results show that ADAM can achieve signiﬁcantly better performance compared to ADAGRAD, RMSPROP and other gradient descent procedures when the gradients are sparse, or in general small in magnitude.

Original: However, its performance has been observed to deteriorate in settings where the loss functions are nonconvex and gradients are dense.
Revised: However, its performance has been observed to deteriorate in settings where the loss functions are nonconvex and gradients are dense.

Original: Further, there is currently a gap in the theoretical understanding of these methods, especially in the nonconvex and stochastic setting Ward et al.
Revised: Further, there is currently a gap in the theoretical understanding of these methods, especially in the nonconvex and stochastic setting Ward et al.

Original: ; Zeng & Yin (2018).
Revised: ; Zeng & Yin (2018).

Original: 1. C ONTENTS AND C ONTRIBUTIONS
Revised: 1. C ONTENTS AND C ONTRIBUTIONS

Original: In this paper, we develop and analyze a new consensus-based distributed adaptive moment estimation (DADAM) method that incorporates decentralized optimization and uses a variant of the adaptive moment estimation methods Duchi et al.
Revised: In this paper, we develop and analyze a new consensus-based distributed adaptive moment estimation (DADAM) method that incorporates decentralized optimization and uses a variant of the adaptive moment estimation methods Duchi et al.

Original: ; Kingma & Ba (2014); Tieleman & Hinton; McMahan & Streeter (2010).
Revised: ; Kingma & Ba (2014); Tieleman & Hinton; McMahan & Streeter (2010).

Original: Existing distributed stochastic and adaptive gradient methods for deep learning are mostly designed for central network topology Dean et al.
Revised: Existing distributed stochastic and adaptive gradient methods for deep learning are mostly designed for central network topology Dean et al.

Original: ; Li et al..
Revised: ; Li et al..

Original: The main bottleneck of such a topology lies on the communication overload on the central node, since all nodes need to concurrently communicate with it.
Revised: The main bottleneck of such a topology lies on the communication overload on the central node, since all nodes need to concurrently communicate with it.

Original: Hence, performance can be signiﬁcantly degraded when network bandwidth is limited.
Revised: Hence, performance can be signiﬁcantly degraded when network bandwidth is limited.

Original: These considerations motivate us to study an adaptive algorithm for network topologies, where all nodes can only communicate with their neighbors and none of the nodes is
Revised: These considerations motivate us to study an adaptive algorithm for network topologies, where all nodes can only communicate with their neighbors and none of the nodes is

Original: designated as “central”
Revised: designated as “central”

Original: Therefore, the proposed method is suitable for large scale machine learning problems,
Revised: Therefore, the proposed method is suitable for large scale machine learning problems,

Original: since
Revised: since

Original: it enables both data parallelization and decentralized computation.
Revised: it enables both data parallelization and decentralized computation

Original: Next we brieﬂy summarize the main technical contributions of the work.
Revised: Next we brieﬂy summarize the main technical contributions of the work.

Original: - Our ﬁrst main result (Theorem 4) provides guarantees of DADAM for constrained convex minimization problems deﬁned over a closed convex set X .
Revised: - Our ﬁrst main result (Theorem 4) provides guarantees of DADAM for constrained convex minimization problems deﬁned over a closed convex set X .

Original: We provide the convergence bound in terms of dynamic regret and show that when the data features are sparse and have bounded gradients, our algorithm’s regret bound can be considerably better than the ones provided by standard mirror descent and gradient descent methods Nedic & Ozdaglar (2009); Hall & Willett (2015); Besbes et al.
Revised: We provide the convergence bound in terms of dynamic regret and show that when the data features are sparse and have bounded gradients, our algorithm’s regret bound can be considerably better than the ones provided by standard mirror descent and gradient descent methods Nedic & Ozdaglar (2009); Hall & Willett (2015); Besbes et al.

Original: It is worth mentioning that the regret bounds provided for adaptive gradient methods Duchi et al.
Revised: It is worth mentioning that the regret bounds provided for adaptive gradient methods Duchi et al.

Original: (2011) are static and our results generalize them to dynamic settings. -
Revised: (2011) are static and our results generalize them to dynamic settings. -

Original: In Theorem 5, we give a new local regret analysis for distributed gradient-based algorithms for nonconvex minimization problems computed over a network of agents.
Revised: In Theorem 5, we give a new local regret analysis for distributed gradient-based algorithms for nonconvex minimization problems computed over a network of agents.

Original: Speciﬁcally, we prove that under certain regularity conditions, DADAM can achieve a local regret bound of order ˜ O
Revised: Speciﬁcally, we prove that under certain regularity conditions, DADAM can achieve a local regret bound of order ˜ O

Original: ( 1 T ) for nonconvex distributed optimization.
Revised: ( 1 T ) for nonconvex distributed optimization.

Original: To the best of our knowledge, rigorous extensions of existing adaptive gradient methods to the (distributed) nonconvex setting considered in this work do not seem to be available.
Revised: To the best of our knowledge, rigorous extensions of existing adaptive gradient methods to the (distributed) nonconvex setting considered in this work do not seem to be available.

Original: In this paper, we also present regret analysis for distributed stochastic optimization problems computed over a network of agents.
Revised: In this paper, we also present regret analysis for distributed stochastic optimization problems computed over a network of agents.

Original: Theorems 7 and 9 provide regret bounds of DADAM for minimization problem equation 3 with stochastic gradients and indicate that the result of Theorems 4 and 5 holds true in expectation.
Revised: Theorems 7 and 9 provide regret bounds of DADAM for minimization problem equation 3 with stochastic gradients and indicate that the result of Theorems 4 and 5 holds true in expectation.

Original: Further, in Corollary 10 we show that DADAM can achieve a local regret bound of order O ( ξ 2 √ nT + 1 T ) for nonconvex distributed stochastic optimization where ξ is  the gradient noise.
Revised: Further, in Corollary 10 we show that DADAM can achieve a local regret bound of order O ( ξ 2 √ nT + 1 T ) for nonconvex distributed stochastic optimization where ξ is an upper bound on the variance of the stochastic gradient.

Original: Hence, DADAM outperforms centralized adaptive algorithms such as ADAM for certain realistic classes of loss functions when T is sufﬁciently large.
Revised: Hence, DADAM outperforms centralized adaptive algorithms such as ADAM for certain realistic classes of loss functions when T is sufﬁciently large.

Original: The remainder of the paper is organized as follows.
Revised: The remainder of the paper is organized as follows.

Original: Section 2 gives a detailed description of
Revised: Section 2 gives a detailed description of

Original: DADAM, while Section 3 establishes its theoretical results.
Revised: DADAM, while Section 3 establishes its theoretical results.

Original: Section 4 explains a network correction technique for our proposed algorithm.
Revised: Section 4 explains a network correction technique for our proposed algorithm.

Original: Section 5 illustrates the proposed framework on a number of synthetic and real data sets.
Revised: Section 5 illustrates the proposed framework on a number of synthetic and real data sets.

Original: Finally, Section 6 concludes the paper.
Revised: Finally, Section 6 concludes the paper.

Original: The detailed proofs of the main results established are delegated to the Appendix.
Revised: The detailed proofs of the main results established are delegated to the Appendix.

Original: 1. M ATHEMATICAL P RELIMINARIES AND N OTATIONS .
Revised: 1. M ATHEMATICAL P RELIMINARIES AND N OTATIONS .

Original: Throughout the paper, R p denotes the real coordinate space of p dimensions.
Revised: Throughout the paper, R p denotes the real coordinate space of p dimensions.

Original: For any pair of vectors x , y ∈ R p , (cid:104) x , y (cid:105) indicates the standard Euclidean inner product.
Revised: For any pair of vectors x , y ∈ R p , (cid:104) x , y (cid:105) indicates the standard Euclidean inner product.

Original: We denote the (cid:96) 1 norm by (cid:107) X (cid:107) 1 =
Revised: We denote the (cid:96) 1 norm by (cid:107) X (cid:107) 1 =

Original: ∑ ij | x ij | , the inﬁnity norm by (cid:107) X (cid:107) ∞ =
Revised: ∑ ij | x ij | , the inﬁnity norm by (cid:107) X (cid:107) ∞ =

Original: max ij | x ij | , and the Euclidean norm by (cid:107) X (cid:107) =
Revised: max ij | x ij | , and the Euclidean norm by (cid:107) X (cid:107) =

Original: (cid:113)
Revised: (cid:113)

Original: ∑ ij | x ij | 2 .
Revised: ∑ ij | x ij | 2 .

Original: The above norms reduce to the vector norms if X is a vector.
Revised: The above norms reduce to the vector norms if X is a vector.

Original: The diameter of the set X is given by
Revised: The diameter of the set X is given by

Original: [Equation]
Revised: [Equation]

Original: Let S p + be the set of all positive deﬁnite p × p matrices.
Revised: Let S p + be the set of all positive deﬁnite p × p matrices.

Original: Π X , A [ x ] denotes the Euclidean projection of a vector x onto X for A ∈ S p + :
Revised: Π X , A [ x ] denotes the Euclidean projection of a vector x onto X for A ∈ S p + :

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: The subscript t is often used to denote the time step while y i , t , d stands for the d -th element of y i , t .
Revised: The subscript t is often used to denote the time step while y i , t , d stands for the d -th element of y i , t .

Original: Further, y i , 1: t , d ∈ R t is given by
Revised: Further, y i , 1: t , d ∈ R t is given by

Original: [Equation]
Revised: [Equation]

Original: We let g i , t to denote the gradient of f at x i , t .
Revised: We let g i , t to denote the gradient of f at x i , t .

Original: The i -th largest singular value of matrix X is denoted by σ i ( X ) .
Revised: The i -th largest singular value of matrix X is denoted by σ i ( X ) .

Original: We denote the element in the i -th row and j -th column of matrix X by [ X ] ij .
Revised: We denote the element in the i -th row and j -th column of matrix X by [ X ] ij .

Original: In several theorems, we consider a connected undirected graph G =
Revised: In several theorems, we consider a connected undirected graph G =

Original: ( V , E ) with nodes V = { 1 , 2 ,..., n } and edges E .
Revised: ( V , E ) with nodes V = { 1 , 2 ,..., n } and edges E .

Original: The matrix W ∈ R n × n is often used to denote a symmetric mixing matrix Shi et al.
Revised: The matrix W ∈ R n × n is often used to denote a symmetric mixing matrix Shi et al.

Original: (2015) of graph G .
Revised: (2015) of graph G .

Original: Indeed, for any agent i , we assign a positive weight [ W ] ij for the information received from agent j (cid:54)
Revised: Indeed, for any agent i , we assign a positive weight [ W ] ij for the information received from agent j (cid:54)

Original: =
Revised: =

Original: i so that [ W ] ij > 0 if and only if ( i , j ) ∈ E and
Revised: i so that [ W ] ij > 0 if and only if ( i , j ) ∈ E and

Original: [Equation]
Revised: [Equation]

Original: The role of W is the similar as that in decentralized gradient descent methods Yuan et al.
Revised: The role of W is the similar as that in decentralized gradient descent methods Yuan et al.

Original: and average consensus Xiao et al.
Revised: and average consensus Xiao et al.

Original: It has a few common choices, which can signiﬁcantly affect performance (see, Shi et al.
Revised: It has a few common choices, which can signiﬁcantly affect performance (see, Shi et al.

Original: Throughout this paper, we consider Metropolis constant edge weight matrix Xiao & Boyd (2004);
Revised: Throughout this paper, we consider Metropolis constant edge weight matrix Xiao & Boyd (2004);

Original: Xiao et al.
Revised: Xiao et al.

Original: (2007) deﬁned as follows
Revised: (2007) deﬁned as follows

Original: [Equation]
Revised: [Equation]

Original: for some small positive ε > 0.
Revised: for some small positive ε > 0.

Original: When ˆ W is chosen according this scheme, W =
Revised: When ˆ W is chosen according this scheme, W =

Original: I + ˆ Wis found to be very efﬁcient.
Revised: I + ˆ Wis found to be very efﬁcient.

Original: Note, the above doubly stochastic matrix implies uniqueness of σ 1 ( W ) = 1 and warrants that other singular value of W are strictly less than one in magnitude.
Revised: Note, the above doubly stochastic matrix implies uniqueness of σ 1 ( W ) = 1 and warrants that other singular value of W are strictly less than one in magnitude.

Original: The Hadamard (entrywise) and Kronecker product are denoted by (cid:12) and ⊗ , respectively.
Revised: The Hadamard (entrywise) and Kronecker product are denoted by (cid:12) and ⊗ , respectively.

Original: Finally, the expectation operator is denoted by E .
Revised: Finally, the expectation operator is denoted by E .

Original:  P ROBLEM F ORMULATION AND A LGORITHM
Revised:  P ROBLEM F ORMULATION AND A LGORITHM

Original: In this section, we propose a new optimization method for online optimization that employs data parallelization and decentralized computation over a network of agents.
Revised: In this section, we propose a new optimization method for online optimization that employs data parallelization and decentralized computation over a network of agents.

Original: In our new structure, given a connected undirected graph G =
Revised: In our new structure, given a connected undirected graph G =

Original: ( V , E )
Revised: ( V , E )

Original: , we let each node i ∈ V at time t ∈ { 1 ,..., T } hold its own measurement and training data m i , and set f i , t ( x ) =
Revised: , we let each node i ∈ V at time t ∈ { 1 ,..., T } hold its own measurement and training data m i , and set f i , t ( x ) =

Original: 1 m i ∑ m i j = 1 f ji , t ( x ) .
Revised: 1 m i ∑ m i j = 1 f ji , t ( x ) .

Original: We also let each agent i hold a local copy of the global variable x at time t ∈ { 1 ,..., T } , which is denoted by x i , t ∈ R p .
Revised: We also let each agent i hold a local copy of the global variable x at time t ∈ { 1 ,..., T } , which is denoted by x i , t ∈ R p .

Original: With this setup, we present a distributed adaptive gradient method for solving the minimization problem
Revised: With this setup, we present a distributed adaptive gradient method for solving the minimization problem

Original: [Equation]
Revised: [Equation]

Original: where f i , t : X → R is a continuously differentiable mapping on the convex set X .
Revised: where f i , t : X → R is a continuously differentiable mapping on the convex set X .

Original: It is known that it is impossible to achieve a sub-linear dynamic regret bound Zhang et al.
Revised: It is known that it is impossible to achieve a sub-linear dynamic regret bound Zhang et al.

Original: (2017), because of the arbitrary ﬂuctuations in problem equation 3.
Revised: (2017), because of the arbitrary ﬂuctuations in problem equation 3.

Original: We want to characterize the hardness of the problem via a complexly measure that captures the pattern of the minimizer sequence { x ∗ t } Tt = 1 , where x ∗ t = argmin x ∈ X f t ( x ) .
Revised: We want to characterize the hardness of the problem via a complexly measure that captures the pattern of the minimizer sequence { x ∗ t } Tt = 1 , where x ∗ t = argmin x ∈ X f t ( x ) .

Original: Subsequently, we would like to provide a regret bound in terms of
Revised: Subsequently, we would like to provide a regret bound in terms of

Original: [Equation]
Revised: [Equation]

Original: which represents the variations in { x ∗ t } Tt = 1 .
Revised: which represents the variations in { x ∗ t } Tt = 1 .

Original: DADAM uses a new distributed adaptive gradient method in which a group of n agents seeking to solve a sequential version of problem equation 3.
Revised: DADAM uses a new distributed adaptive gradient method in which a group of n agents seeking to solve a sequential version of problem equation 3.

Original: Here, we assume that each component function f i , t : X → R becomes only available to agent i ∈ V , after having made its decision at time t ∈ { 1 ,..., T } .
Revised: Here, we assume that each component function f i , t : X → R becomes only available to agent i ∈ V , after having made its decision at time t ∈ { 1 ,..., T } .

Original: In the t -th, the i -th agent chooses a point x
Revised: In the t -th, the i -th agent chooses a point x

Original: i , t corresponding to what it considers the network as a whole should have selected.
Revised: i , t corresponding to what it considers the network as a whole should have selected.

Original: After committing to this choice, the agent has access to a cost function f i , t : X → R and the network cost is then given by f t ( x ) = 1 n ∑ ni = 1 f
Revised: After committing to this choice, the agent has access to a cost function f i , t : X → R and the network cost is then given by f t ( x ) = 1 n ∑ ni = 1 f

Original: i , t ( x ) .
Revised: i , t ( x ) .

Original: Note that this function is not known to any of the agents and is not available at any single location.
Revised: Note that this function is not known to any of the agents and is not available at any single location.

Original: The procedure of our proposed method is outlined in Algorithm 1.
Revised: The procedure of our proposed method is outlined in Algorithm 1.

Original: input : x 1 ∈ X , a positive sequence { α t } Tt = 1 , exponential decay rates for the moment estimate β 1 , β 2 , β 3 ∈ [ 0 , 1 ) , and a mixing matrix W satisfying equation 2; 1 for all i ∈ V , initialize moment vectors
Revised: input : x 1 ∈ X , a positive sequence { α t } Tt = 1 , exponential decay rates for the moment estimate β 1 , β 2 , β 3 ∈ [ 0 , 1 ) , and a mixing matrix W satisfying equation 2; 1 for all i ∈ V , initialize moment vectors

Original: m i , 

Revised: m i ,

Original: = υ i , 

Revised: = υ i ,

Original: =
Revised: =

Original: ˆ υ i , 

Revised: ˆ υ i ,

Original: = 0 and x i , 1 = x 1 ;2 for i ∈ V dofor t ← 1 to T do 
Revised: = 0 and x i , 1 = x 1 ; do 1 to T do [Equation] [Equation]

Original: [Equation] 
Revised: [Equation] [Equation] [Equation] [Equation]

Original: output: resulting parameter ¯ x = 1 n ∑ ni = 1 x i , T +
Revised: output : resulting parameter ¯ x = 1 n ∑ ni = 1 x i , T +

Original: It is worth mentioning that DADAM computes adaptive learning rates from estimates of both ﬁrst and second moments of the gradients similar to Duchi et al.
Revised: It is worth mentioning that DADAM computes adaptive learning rates from estimates of both ﬁrst and second moments of the gradients similar to Duchi et al.

Original: ; Tieleman & Hinton; Kingma & Ba (2014); Reddi et al.
Revised: ; Tieleman & Hinton; Kingma & Ba (2014); Reddi et al.

Original: However, DADAM uses a larger learning rate in comparison to
Revised: However, DADAM uses a larger learning rate in comparison to

Original: AMSGrad Reddi et al.
Revised: AMSGrad Reddi et al.

Original: and yet incorporates the intuition of slowly decaying the effect of previous gradients on the learning rate.
Revised: and yet incorporates the intuition of slowly decaying the effect of previous gradients on the learning rate.

Original: The key difference of DADAM with ADAM and AMSGrad is that it maintains the maximum of all second moment estimates of the gradient vectors until time step t and uses
Revised: The key difference of DADAM with ADAM and AMSGrad is that it maintains the maximum of all second moment estimates of the gradient vectors until time step t and uses

Original: [Equation]
Revised: [Equation]

Original: for normalizing the running average of the gradient instead of υ i , t in ADAM and max ( ˆ υ i , t − 1 , υ i , t ) in AMSGrad.
Revised: for normalizing the running average of the gradient instead of υ i , t in ADAM and max ( ˆ υ i , t − 1 , υ i , t ) in AMSGrad.

Original: The learning rate ˆ υ i , t is an important component of the DADAM framework, since it enables us to develop a convergent adaptive method similar to AMSGrad while maintaining the efﬁciency of ADAM.
Revised: The learning rate ˆ υ i , t is an important component of the DADAM framework, since it enables us to develop a convergent adaptive method similar to AMSGrad while maintaining the efﬁciency of ADAM.

Original: Next, we provide the notion of regret which measures the performance of DADAM against a sequence of local minimizers.
Revised: Next, we provide the notion of regret which measures the performance of DADAM against a sequence of local minimizers.

Original: In the framework of online convex optimization, the performance of algorithms is assessed by regret that measures how competitive the algorithm is with respect to the best ﬁxed solution Mateos-N´unez & Cort´es (2014); Hazan et al.
Revised: In the framework of online convex optimization, the performance of algorithms is assessed by regret that measures how competitive the algorithm is with respect to the best ﬁxed solution Mateos-N´unez & Cort´es (2014); Hazan et al.

Original: However, the notion of regret fails to illustrate the performance of online algorithms in a dynamic setting.
Revised: However, the notion of regret fails to illustrate the performance of online algorithms in a dynamic setting.

Original: To overcome this issue, we consider a more stringent metric–dynamic regret Hall & Willett (2015);
Revised: To overcome this issue, we consider a more stringent metric–dynamic regret Hall & Willett (2015);

Original: Besbes et al.
Revised: Besbes et al.

Original: ; Zinkevich (2003), in which the cumulative loss of the learner is compared against the minimizer sequence { x ∗ t } Tt = 1 , i.e.,
Revised: ; Zinkevich (2003), in which the cumulative loss of the learner is compared against the minimizer sequence { x ∗ t } Tt = 1 , i.e.,

Original: Reg CT : = 1 n n ∑ i =
Revised: Reg CT : = 1 n n ∑ i =

Original: T ∑ t = 1 f i , t ( x i , t ) −
Revised: T ∑ t = 1 f i , t ( x i , t ) −

Original: T ∑ t = 1 f t ( x ∗ t ) , where x ∗ t = argmin x ∈ X f t ( x ) .
Revised: T ∑ t = 1 f t ( x ∗ t ) , where x ∗ t = argmin x ∈ X f t ( x ) .

Original: On the other hand, in the framework of nonconvex optimization, it is usual to state convergence guarantees of an algorithm towards an ε -approximate stationary point–that is, there exist some iterates x i , t for which (cid:107) ∇ f t ( x i , t ) (cid:107) ≤ ε .
Revised: On the other hand, in the framework of nonconvex optimization, it is usual to state convergence guarantees of an algorithm towards an ε -approximate stationary point–that is, there exist some iterates x i , t for which (cid:107) ∇ f t ( x i , t ) (cid:107) ≤ ε .

Original: Inspired by Hazan et al.
Revised: Inspired by Hazan et al.

Original: (2017), we next provide the deﬁnition of projected gradient and introduce local regret , a new notion of regret which quantiﬁes the moving average of gradients over network.
Revised: (2017), we next provide the deﬁnition of projected gradient and introduce local regret , a new notion of regret which quantiﬁes the moving average of gradients over network.

Original: Deﬁnition 1.
Revised: Deﬁnition 1.

Original: (Local Regret) .
Revised: (Local Regret) .

Original: Assume f
Revised: Assume f

Original: i : X → R is a differentiable function on a closed convex set X ⊆ R p .
Revised: i : X → R is a differentiable function on a closed convex set X ⊆ R p .

Original: Given a step-size α > 0 , we deﬁne G X ( x , f i , α ) :
Revised: Given a step-size α > 0 , we deﬁne G X ( x , f i , α ) :

Original: X → R p the projected gradient of f i at x, by
Revised: X → R p the projected gradient of f i at x, by

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: Then, the local regret of an online algorithm is given by
Revised: Then, the local regret of an online algorithm is given by

Original: [Equation]
Revised: [Equation]

Original: where ¯ f i , t ( x i , t ) = 1 t ∑ ts = 1 f i , s ( x i , t ) is an aggregate loss.
Revised: where ¯ f i , t ( x i , t ) = 1 t ∑ ts = 1 f i , s ( x i , t ) is an aggregate loss.

Original: We analyze the convergence of DADAM as applied to minimization problem equation 3 using regrets Reg CT and Reg NT .
Revised: We analyze the convergence of DADAM as applied to minimization problem equation 3 using regrets Reg CT and Reg NT .

Original: It is worth to mention that DADAM is initialized at x i , t = 0 to keep the presentation of the convergence analysis clear.
Revised: It is worth to mention that DADAM is initialized at x i , t = 0 to keep the presentation of the convergence analysis clear.

Original: In general, any initialization can be selected for implementation purposes.
Revised: In general, any initialization can be selected for implementation purposes.

Original:  C ONVERGENCE A NALYSIS
Revised:  C ONVERGENCE A NALYSIS

Original: In this section, our aim is to establish convergence properties of DADAM under the following assumptions:
Revised: In this section, our aim is to establish convergence properties of DADAM under the following assumptions:

Original: Assumption 2.
Revised: Assumption 2.

Original: For all i ∈ V and t ∈ { 1 ,..., T } , the function f i , t ( x ) is continuously differentiable over X , and has Lipschitz continuous gradient on this set, i.e. there exists a constant ρ ≥ 

Revised: For all i ∈ V and t ∈ { 1 ,..., T } , the function f i , t ( x ) is continuously differentiable over X , and has Lipschitz continuous gradient on this set, i.e. there exists a constant ρ ≥

Original: so that
Revised: so that

Original: [Equation]
Revised: [Equation]

Original: Further, f i , t ( · ) is Lipschitz continuous on X with a uniform constant L ≥ 0 , i.e.,
Revised: Further, f i , t ( · ) is Lipschitz continuous on X with a uniform constant L ≥ 0 , i.e.,

Original: [Equation]
Revised: [Equation]

Original: There are many cost functions f i , t that satisfy this type of Lipschitz condition.
Revised: There are many cost functions f i , t that satisfy this type of Lipschitz condition.

Original: For example, it holds for any convex function on a compact set X , or for any polyhedral function on an arbitrary domain
Revised: For example, it holds for any convex function on a compact set X , or for any polyhedral function on an arbitrary domain

Original: Duchi et al.
Revised: Duchi et al.

Original: Assumption 3.
Revised: Assumption 3.

Original: For all i ∈ V and t ∈ { 1 ,..., T } , the stochastic gradient, as notation we use bold letters g = ∇ f i , t ( x i , t ) , satisﬁes
Revised: For all i ∈ V and t ∈ { 1 ,..., T } , the stochastic gradient, as notation we use bold letters g = ∇ f i , t ( x i , t ) , satisﬁes

Original: [Equation]
Revised: [Equation]

Original: where F t is the ξ -ﬁeld containing all information prior to the outset of round t + 1 .
Revised: where F t is the ξ -ﬁeld containing all information prior to the outset of round t + 1 .

Original: 3. C ONVEX C ASE
Revised: 3. C ONVEX C ASE

Original: Next, we focus on the case where for all i ∈ V and t ∈ { 1 ,..., T } , the agent i at time t has access to the exact gradient g i , t = ∇ f i , t ( x i , t ) .
Revised: Next, we focus on the case where for all i ∈ V and t ∈ { 1 ,..., T } , the agent i at time t has access to the exact gradient g i , t = ∇ f i , t ( x i , t ) .

Original: The following results apply to convex problems, as well as their stochastic variants in a dynamic environment.
Revised: The following results apply to convex problems, as well as their stochastic variants in a dynamic environment.

Original: Theorem 4.
Revised: Theorem 4.

Original: Suppose that the parameters β 1 , β 2 ∈ [ 0 , 1 ) satisfy η = β 1 √ β 2 < 1 .
Revised: Suppose that the parameters β 1 , β 2 ∈ [ 0 , 1 ) satisfy η = β 1 √ β 2 < 1 .

Original: Let β 1 , t = β 1 λ t − 1 , λ ∈ ( 0 , 1 ) and (cid:107) ∇ f i , t ( x ) (cid:107) ∞ ≤ G ∞ for all t ∈ { 1 ,..., T } .
Revised: Let β 1 , t = β 1 λ t − 1 , λ ∈ ( 0 , 1 ) and (cid:107) ∇ f i , t ( x ) (cid:107) ∞ ≤ G ∞ for all t ∈ { 1 ,..., T } .

Original: Then, using a step-size α t = α √ t for the sequence x i , t generated by Algorithm 1, we have
Revised: Then, using a step-size α t = α √ t for the sequence x i , t generated by Algorithm 1, we have

Original: [Equation]
Revised: [Equation]

Original: Next, we analyze the stochastic convex setting and extend the result of Theorem 4 to the noisy case where agents have access to stochastic gradients of the objective function equation 3.
Revised: Next, we analyze the stochastic convex setting and extend the result of Theorem 4 to the noisy case where agents have access to stochastic gradients of the objective function equation 3.

Original: Theorem 5.
Revised: Theorem 5.

Original: Suppose that Assumption 3 holds.
Revised: Suppose that Assumption 3 holds.

Original: Further, the parameters β 1 , β 2 ∈ [ 0 , 1 ) satisfy η = β 1 √ β 2 < 1 .
Revised: Further, the parameters β 1 , β 2 ∈ [ 0 , 1 ) satisfy η = β 1 √ β 2 < 1 .

Original: Let β 1 , t = β 1 λ t − 1 , λ ∈ ( 0 , 1 ) .
Revised: Let β 1 , t = β 1 λ t − 1 , λ ∈ ( 0 , 1 ) .

Original: Then, using a step-size α t = α √ t for the sequence x i , t generated by Algorithm 1, we have
Revised: Then, using a step-size α t = α √ t for the sequence x i , t generated by Algorithm 1, we have

Original: [Equation]
Revised: [Equation]

Original: Remark 6.
Revised: Remark 6.

Original: Theorems 4 and 5 show that the regret bound of DADAM can be considerably better than the ones provided by standard mirror descent and gradient descent methods for both centralized and decentralized settings Zinkevich (2003); Hall & Willett (2015);
Revised: Theorems 4 and 5 show that the regret bound of DADAM can be considerably better than the ones provided by standard mirror descent and gradient descent methods for both centralized and decentralized settings Zinkevich (2003); Hall & Willett (2015);

Original: Besbes et al.
Revised: Besbes et al.

Original: ; Shahrampour & Jadbabaie (2018)
Revised: ; Shahrampour & Jadbabaie (2018)

Original: because
Revised: because

Original: [Equation]
Revised: [Equation]

Original: It is easy to show that the regret of DADAM is bounded by O ( G ∞ D T √ T ) where D T = max d ∈{ 1 ,..., p } D T , d .
Revised: It is easy to show that the regret of DADAM is bounded by O ( G ∞ D T √ T ) where D T = max d ∈{ 1 ,..., p } D T , d .

Original: Indeed, the term ∑ Tt = 1 |
Revised: Indeed, the term ∑ Tt = 1 |

Original: g i , t , d | / √ t in the proof of Lemma 13 can be bounded by
Revised: g i , t , d | / √ t in the proof of Lemma 13 can be bounded by

Original: O ( G ∞ √ T ) instead of O ( G ∞ √ T log T ) .
Revised: O ( G ∞ √ T ) instead of O ( G ∞ √ T log T ) .

Original: Hence, the regret of DADAM is upper bounded by minimum of O ( G ∞ D T √ T ) and the bound presented in Theorems 4 and 5, and thus the worst case dependence on T is √ T rather than √ T log T. Note also that in the static setting, i.e. D T = 0 , the regret of
Revised: Hence, the regret of DADAM is upper bounded by minimum of O ( G ∞ D T √ T ) and the bound presented in Theorems 4 and 5, and thus the worst case dependence on T is √ T rather than √ T log T. Note also that in the static setting, i.e. D T = 0 , the regret of

Original: DADAM is upper bounded by O ( G ∞ √ T ) .
Revised: DADAM is upper bounded by O ( G ∞ √ T ) .

Original: 3. N ONCONVEX C ASE
Revised: 3. N ONCONVEX C ASE

Original: In this section, we provide convergence guarantees for DADAM for the nonconvex minimization problem equation 3 deﬁned over a closed convex set X .
Revised: In this section, we provide convergence guarantees for DADAM for the nonconvex minimization problem equation 3 deﬁned over a closed convex set X .

Original: To do so, we use the projection map Π X instead of Π X , √ diag ( ˆ v i , t ) for updating parameters x i , t for t ∈ { 1 ,..., T } , i ∈ V (see, Algorithm 1 for details).
Revised: To do so, we use the projection map Π X instead of Π X , √ diag ( ˆ v i , t ) for updating parameters x i , t for t ∈ { 1 ,..., T } , i ∈ V (see, Algorithm 1 for details).

Original: Next, we derive an upper and lower bound for the second moment of gradient, υ i , t .
Revised: Next, we derive an upper and lower bound for the second moment of gradient, υ i , t .

Original: Assume that there exists a positive number ¯ υ such that
Revised: Assume that there exists a positive number ¯ υ such that

Original: [Equation]
Revised: [Equation]

Original: Now, using the update rule of υ i , t , we have υ
Revised: Now, using the update rule of υ i , t , we have υ

Original: i , t =
Revised: i , t =

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: 1 − β
Revised: β

Original: [Equation]
Revised: [Equation]

Original: ∑ tl
Revised: ∑ tl

Original: [Equation]
Revised: [Equation]

Original: 1 β t − l 2 g 2 i , l .
Revised: 1 β t − l 2 g 2 i , l .

Original: This together with equationimply that
Revised: This together with equationimply that

Original: [Equation]
Revised: [Equation]

Original: On the other hand, since υ i , t , d is non-decreasing and g i , t , d is required to be element-wise non-zero, we have
Revised: On the other hand, since υ i , t , d is non-decreasing and g i , t , d is required to be element-wise non-zero, we have

Original: [Equation]
Revised: [Equation]

Original: for some constant υ .
Revised: for some constant υ .

Original: Now, using the update rule of ˆ υ i , t along with equation 10 and equation 9, we have
Revised: Now, using the update rule of ˆ υ i , t along with equation 10 and equation 9, we have

Original: [Equation]
Revised: [Equation]

Original: Theorem 7.
Revised: Theorem 7.

Original: Suppose that Assumption 2 holds.
Revised: Suppose that Assumption 2 holds.

Original: Further, the parameters β 1 , β 2 ∈ [ 0 , 1 ) satisfy η = β 1 √ β 2 < 1 .
Revised: Further, the parameters β 1 , β 2 ∈ [ 0 , 1 ) satisfy η = β 1 √ β 2 < 1 .

Original: Choose the positive sequence { α t } Tt = 1 such that 0 < α t ≤ ( 2 − β 1 )
Revised: Choose the positive sequence { α t } Tt = 1 such that 0 < α t ≤ ( 2 − β 1 )

Original: υ 2 ρ ¯ υ with α t < ( 2 − β1 )
Revised: υ 2 ρ ¯ υ with α t < ( 2 − β

Revised: [Equation]

Original: υ 2 ρ ¯ υ for at least one t. Then, for the sequence x i , t generated by Algorithm 1, we have
Revised: 2 ρ ¯ [Equation] for at least one t. Then, for the sequence x i , t generated by Algorithm 1, we have

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where ϑ t =
Revised: where ϑ t =

Original: ∑ Tt = 

Revised: ∑ Tt =

Original: [ ( 2 − β 1 ) α t 2 ¯ υ − ρα 2 t 2 υ 2 ] .
Revised: [ ( 2 − β 1 ) α t 2 ¯ υ − [Equation] 2 t 2 υ 2 ] .

Original: The following corollary shows that DADAM using a certain step-size leads to a near optimal regret bound for nonconvex functions.
Revised: The following corollary shows that DADAM using a certain step-size leads to a near optimal regret bound for nonconvex functions.

Original: Corollary 8.
Revised: Corollary 8.

Original: Under the same conditions of Theorem 7, using the step-sizes α t =
Revised: Under the same conditions of Theorem 7, using the step-sizes α t =

Original: ( 2 − β 1 ) υ 2 2 ρ ¯ υ and β 1 , t = β 1 λ t − 1 , λ ∈ ( 0 , 1 ) for all t ∈ { 1 ,..., T } , we have
Revised: ( 2 − β 1 ) υ 2 2 ρ ¯ υ and β 1 , t = β 1 λ t − 1 , λ ∈ ( 0 , 1 ) for all t ∈ { 1 ,..., T } , we have

Original: [Equation]
Revised: [Equation]

Original: To complete the analysis of our algorithm in the nonconvex setting, we provide the regret bound for
Revised: To complete the analysis of our algorithm in the nonconvex setting, we provide the regret bound for

Original: DADAM, when stochastic gradients are accessible to the learner.
Revised: DADAM, when stochastic gradients are accessible to the learner.

Original: Theorem 9.
Revised: Theorem 9.

Original: Suppose that Assumption 2 holds.
Revised: Suppose that Assumption 2 holds.

Original: Further, the parameters β 1 , β 2 ∈ [ 0 , 1 ) satisfy η = β 1 √ β 2 < 1 .
Revised: Further, the parameters β 1 , β 2 ∈ [ 0 , 1 ) satisfy η = β 1 √ β 2 < 1 .

Original: Choose the positive sequence { α t } Tt = 1 such that 0 < α t ≤ ( 2 − β 1 )
Revised: Choose the positive sequence { α t } Tt = 1 such that 0 < α t ≤ ( 2 − β 1 )

Original: υ 2 ρ ¯ υ with α t < ( 2 − β 1 )
Revised: υ 2 ρ ¯ υ with α t < ( 2 − β 1 )

Original: υ 2 ρ ¯ υ for at least one t.
Revised: υ 2 ρ ¯ υ for at least one t.

Original: Then, for the sequence x i , t generated by Algorithm 1, we have
Revised: Then, for the sequence x i , t generated by Algorithm 1, we have

Original: [Equation]
Revised: [Equation]

Original: where ϑ t =
Revised: where ϑ t =

Original: ∑ T t = 

Revised: ∑ T t =

Original: [ ( 2 − β 1 ) α t 2 ¯ υ − ρα 2 t 2 υ 2 ] .
Revised: [ ( 2 − β 1 ) α t 2 ¯ υ − ρα 2 t 2 υ 2 ] .

Original: 3.2. W HEN DOES DADAM O UTPERFORM ADAM?
Revised: 3.2. W HEN DOES DADAM O UTPERFORM ADAM?

Original: We next theoretically justify the potential advantage of the proposed decentralized algorithm
Revised: We next theoretically justify the potential advantage of the proposed decentralized algorithm

Original: DADAM over centralized adaptive moment estimation methods such as ADAM .
Revised: DADAM over centralized adaptive moment estimation methods such as ADAM .

Original: More speciﬁcally, the following corollary shows that when T is sufﬁciently large, the 1 T term will be dominated by the 1 √ nT term which leads to a 1 √ nT convergence rate.
Revised: More speciﬁcally, the following corollary shows that when T is sufﬁciently large, the 1 T term will be dominated by the 1 √ nT term which leads to a 1 √ nT convergence rate.

Original: Corollary 10.
Revised: Corollary 10.

Original: Suppose that Assumptions 2 and 3 hold.
Revised: Suppose that Assumptions 2 and 3 hold.

Original: Moreover, the parameters β 1 , β 2 ∈ [ 0 , 1 ) satisfy η = β 1 √ β 2 < 1 and β 1 , t = β 1 λ t − 1 , λ ∈ ( 0 , 1 ) .
Revised: Moreover, the parameters β 1 , β 2 ∈ [ 0 , 1 ) satisfy η = β 1 √ β 2 < 1 and β 1 , t = β 1 λ t − 1 , λ ∈ ( 0 , 1 ) .

Original: Choose the step-size sequence as α t = α √ nT with α = ( 2 − β 1 )
Revised: Choose the step-size sequence as α t = α √ nT with α = ( 2 − β 1 )

Original: υ 2 ρ ¯ υ .
Revised: υ 2 ρ ¯ υ .

Original: Then, for the sequence x i , t generated by Algorithm 1, we have
Revised: Then, for the sequence x i , t generated by Algorithm 1, we have

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: if the total number of time steps T satisﬁes
Revised: if the total number of time steps T satisﬁes

Original: [Equation]
Revised: [Equation]

Original: 16a
Revised: 16a

Original: [Equation]
Revised: [Equation]

Original: 16b
Revised: 16b

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: Let ε -approximation solution of equation 3 be deﬁned by E
Revised: Let ε -approximation solution of equation 3 be deﬁned by E

Original: [ Reg NT ] T ≤ ε .
Revised: [ Reg NT ] T ≤ ε .

Original: Corollary 10 indicates that the total computational complexity of DADAM to achieve an ε -approximation solution is bounded by O ( 1 ε 2 ) .
Revised: Corollary 10 indicates that the total computational complexity of DADAM to achieve an ε -approximation solution is bounded by O ( 1 ε 2 ) .

Original:  A N E XTENSION OF DADAM WITH A C ORRECTED U PDATE R ULE
Revised:  A N E XTENSION OF DADAM WITH A C ORRECTED U PDATE R ULE

Original: Compared to classical centralized algorithms, decentralized algorithms encounter more restrictive assumptions and typically worse convergence rates.
Revised: Compared to classical centralized algorithms, decentralized algorithms encounter more restrictive assumptions and typically worse convergence rates.

Original: In order to improve the convergence rate, Shi et al. (2015) introduced a corrected decentralized gradient method called EXTRA, which has a linear rate of convergence if the objective function is strongly convex.
Revised: In order to improve the convergence rate, Shi et al. (2015) introduced a corrected decentralized gradient method called EXTRA, which has a linear rate of convergence if the objective function is strongly convex.

Original: Similar to EXTRA, we provide next a corrected update rule for i ∈ V , given by
Revised: Similar to EXTRA, we provide next a corrected update rule for i ∈ V , given by

Original: [Equation]
Revised: [Equation]

Original: ) where x i , t + 1 is generated by Algorithm 1.
Revised: where x i , t + 1 is generated by Algorithm 1.

Original: Our numerical results on some test problems show the efﬁciency and effectiveness of the corrected
Revised: Our numerical results on some test problems show the efﬁciency and effectiveness of the corrected

Original: DADAM (C-DADAM) in practice.
Revised: DADAM (C-DADAM) in practice.

Original: Note that a C-DADAM update is a DADAM update with a cumulative correction term.
Revised: Note that a C-DADAM update is a DADAM update with a cumulative correction term.

Original: The summation in equation 17 is necessary, since each individual term ∑ nj = 

Revised: The summation in equation 17 is necessary, since each individual term ∑ nj =

Original: [ W − ˆ W ] ij x j , s is asymptotically vanishing and the terms must work cumulatively.
Revised: [ W − ˆ W ] ij x j , s is asymptotically vanishing and the terms must work cumulatively.

Original:  N UMERICAL RESULTS
Revised:  N UMERICAL RESULTS

Original: In this section, we present numerical results for Algorithm 1 (DADAM), on both synthetic and real data sets.
Revised: In this section, we present numerical results for Algorithm 1 (DADAM), on both synthetic and real data sets.

Original: The employed data sets and the code will be made available on github upon acceptance of this manuscript.
Revised: The employed data sets and the code will be made available on github upon acceptance of this manuscript.

Original: The results are organized in the following sub-sections.
Revised: The results are organized in the following sub-sections.

Original: In Section 5.1, we investigate the efﬁciency of adaptive decentralized updating rule in static and deterministic settings.
Revised: In Section 5.1, we investigate the efﬁciency of adaptive decentralized updating rule in static and deterministic settings.

Original: We present numerical results on a sparse binary classiﬁcation problem, comparing the performance of DADAM to that of standard decentralized gradient descent (DGD) Nedic & Ozdaglar (2009);
Revised: We present numerical results on a sparse binary classiﬁcation problem, comparing the performance of DADAM to that of standard decentralized gradient descent (DGD) Nedic & Ozdaglar (2009);

Original: Yuan et al.
Revised: Yuan et al.

Original: and its corrected version EXTRA Shi et al.
Revised: and its corrected version EXTRA Shi et al.

Original: In Section 5.2, we demonstrate the efﬁcacy of DADAM in comparison with a decentralized ADAM (Algorithm 1 with β 3 = 1) and the recently proposed distributed federated averaging SGD (FedAvg) algorithm on benchmark datasets such as MNIST and CIFAR-10.
Revised: In Section 5.2, we demonstrate the efﬁcacy of DADAM in comparison with a decentralized ADAM (Algorithm 1 with β 3 = 1) and the recently proposed distributed federated averaging SGD (FedAvg) algorithm on benchmark datasets such as MNIST and CIFAR-10.

Original: All algorithms have been implemented in a Mac machine equipped with a 1.8 GHz Intel Core i processor and 8 GB 1600 MHz DDR3 of memory.
Revised: All algorithms have been implemented in a Mac machine equipped with a 1.8 GHz Intel Core i processor and 8 GB 1600 MHz DDR3 of memory.

Original: 5. B INARY CLASSIFICATION BASED ON SYNTHETIC DATA IN THE STATIC ENVIRONMENT
Revised: 5. B INARY CLASSIFICATION BASED ON SYNTHETIC DATA IN THE STATIC ENVIRONMENT

Original: Consider the following online distributed learning setting: at each time t , m i randomly generated data points are given to every agent i in the form of ( y t , i , j , x t , i , j ) , where y t , i , j ∈ {± 1 } is a binary label and x t , i , j ∈ R p is a feature vector for j = 1 ,..., m i .
Revised: Consider the following online distributed learning setting: at each time t , m i randomly generated data points are given to every agent i in the form of ( y t , i , j , x t , i , j ) , where y t , i , j ∈ {± 1 } is a binary label and x t , i , j ∈ R p is a feature vector for j = 1 ,..., m i .

Original: Our goal is to train a classiﬁer θ ∈ R p such that for an arbitrary new feature vector
Revised: Our goal is to train a classiﬁer θ ∈ R p such that for an arbitrary new feature vector

Original: ˆ x it assigns ˆ y = sign ( (cid:104) θ , ˆ x (cid:105) ) .
Revised: ˆ x it assigns ˆ y = sign ( (cid:104) θ , ˆ x (cid:105) ) .

Original: We design decentralized convex logistic and nonconvex sigmoid loss functions as follows:
Revised: We design decentralized convex logistic and nonconvex sigmoid loss functions as follows:

Original: [Equation]
Revised: [Equation]

Original: (18) where m i is the size of the training data on node i .
Revised: (18) where m i is the size of the training data on node i .

Original: Note that f 1 i , t ( θ ) is smooth, Lipschitz and nonconvex, while f 2 i , t ( θ ) is convex.
Revised: Note that f 1 i , t ( θ ) is smooth, Lipschitz and nonconvex, while f 2 i , t ( θ ) is convex.

Original: For X , we consider the (cid:96) 1 ball X (cid:96) 

Revised: For X , we consider the (cid:96) 1 ball X (cid:96)

Original: = {
Revised: = {

Original: [Equation]
Revised: [Equation]

Original: ∈ R p : (cid:107) θ (cid:107) 1 ≤ r } , when a sparse classiﬁer is preferred.
Revised: ∈ R p : (cid:107) θ (cid:107) 1 ≤ r } , when a sparse classiﬁer is preferred.

Original: We show results for the decentralized problem solved by DADAM, C-DADAM, DGD and EXTRA over a medium-scale network.
Revised: We show results for the decentralized problem solved by DADAM, C-DADAM, DGD and EXTRA over a medium-scale network.

Original: We have implemented EXTRA and DGD with their default settings.
Revised: We have implemented EXTRA and DGD with their default settings.

Original: The connected network is randomly generated with n = 50 agents and connectivity ratio r = 0 .
Revised: The connected network is randomly generated with n = 50 agents and connectivity ratio r = 0 .

Original: Each agent holds 10 samples, i.e., m i = 10 , ∀ i ∈ V .
Revised: Each agent holds 10 samples, i.e., m i = 10 , ∀ i ∈ V .

Original: The agents shall collaboratively obtain p coefﬁcients via loss functions equation 18.
Revised: The agents shall collaboratively obtain p coefﬁcients via loss functions equation 18.

Original: All samples are randomly generated, and the reference classiﬁer θ ∗ is pre-computed using a centralized method.
Revised: All samples are randomly generated, and the reference classiﬁer θ ∗ is pre-computed using a centralized method.

Original: As it is easy to implement in practice, we use the Metropolis constant edge weight matrix W Shi et al.
Revised: As it is easy to implement in practice, we use the Metropolis constant edge weight matrix W Shi et al.

Original: (2015) in our experimental evaluation.
Revised: (2015) in our experimental evaluation.

Original: We have tested the DADAM and C-DADAM with β 1 = β 2 = β 3 = 0 .
Revised: We have tested the DADAM and C-DADAM with β 1 = β 2 = β 3 = 0 .

Original: The numerical results are illustrated in Figure 1.
Revised: The numerical results are illustrated in Figure 1.

Original: It can be seen that DADAM and C-DADAM outperform DGD and EXTRA, showing almost linear convergence in the convex setting to the reference logistic classiﬁer x ∗ .
Revised: It can be seen that DADAM and C-DADAM outperform DGD and EXTRA, showing almost linear convergence in the convex setting to the reference logistic classiﬁer x ∗ .

Original: 5. CIFAR AND MNIST EXPERIMENTS
Revised: 5. CIFAR AND MNIST EXPERIMENTS

Original: Next, we present the experimental results using the CIFAR-10 image recognition dataset and the MNIST digit recognition task.
Revised: Next, we present the experimental results using the CIFAR-10 image recognition dataset and the MNIST digit recognition task.

Original: The model architecture for training CNN to classify CIFAR10 dataset was taken from the TensorFlow tutorial 2 .
Revised: The model architecture for training CNN to classify CIFAR10 dataset was taken from the TensorFlow tutorial 2 .

Original: Also, the model for training a simple multilayer percepton (MLP) on the MNIST dataset was taken from https://github.com/keras-team/keras/blob/master .
Revised: Also, the model for training a simple multilayer percepton (MLP) on the MNIST dataset was taken from https://github.com/keras-team/keras/blob/master .

Original: We compare the accuracy of DADAM with that of the decentralized ADAM (Algorithm 1 with β 3 = 1 ) and the Federated Averaging (FedAvg) algorithm McMahan et al.
Revised: We compare the accuracy of DADAM with that of the decentralized ADAM (Algorithm 1 with β 3 = 1 ) and the Federated Averaging (FedAvg) algorithm McMahan et al.

Original: (2016) which also performs data parallelization without decentralized computation.
Revised: (2016) which also performs data parallelization without decentralized computation.

Original: The connected network is generated with n = 5 agents and connectivity ratio r = 0 .
Revised: The connected network is generated with n = 5 agents and connectivity ratio r = 0 .

Original: The parameters for DADAM and ADAM are selected in a way similar to the previous experiments.
Revised: The parameters for DADAM and ADAM are selected in a way similar to the previous experiments.

Original: In our implementation, we use same number of agents and choose E = C
Revised: In our implementation, we use same number of agents and choose E = C

Original: = 1 as the parameters in the FedAvg algorithm since it is close to a connected topology scenario as considered in the DADAM and ADAM.
Revised: = 1 as the parameters in the FedAvg algorithm since it is close to a connected topology scenario as considered in the DADAM and ADAM.

Original: It can be easily seen from Figure 2 that
Revised: It can be easily seen from Figure 2 that

Original: DADAM and the decentralized ADAM ( β 3 = 1) outperform FedAvg.
Revised: DADAM and the decentralized ADAM ( β 3 = 1) outperform FedAvg.

Original: Further, DADAM can achieve high accuracy in comparison with the decentralized ADAM and FedAvg.
Revised: Further, DADAM can achieve high accuracy in comparison with the decentralized ADAM and FedAvg.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: A decentralized adaptive moment estimation method DADAM was proposed for the distributed learning of deep networks based on adaptive moment of ﬁrst and second moment of estimations.
Revised: A decentralized adaptive moment estimation method DADAM was proposed for the distributed learning of deep networks based on adaptive moment of ﬁrst and second moment of estimations.

Original: Convergence properties of the proposed algorithm were established for convex (and dynamic) and nonconvex functions in both stochastic and deterministic settings.
Revised: Convergence properties of the proposed algorithm were established for convex (and dynamic) and nonconvex functions in both stochastic and deterministic settings.

Original: Numerical results on some datasets show the efﬁciency and effectiveness of the new proposed method in practice.
Revised: Numerical results on some datasets show the efﬁciency and effectiveness of the new proposed method in practice.

Original: 6. P ROPERTIES OF DADAM
Revised: 6. P ROPERTIES OF DADAM

Original: Lemma 11.
Revised: Lemma 11.

Original: Beck & Teboulle (2003) Let X be a nonempty closed convex set in R p .
Revised: Beck & Teboulle (2003) Let X be a nonempty closed convex set in R p .

Original: Then, for any d ∈ X , we have
Revised: Then, for any d ∈ X , we have

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: Lemma 12.
Revised: Lemma 12.

Original: McMahan & Streeter (2010)
Revised: McMahan & Streeter (2010)

Original: For any A ∈ S p + and convex feasible set C ⊂ R p , suppose a 1 = Π C , A [ b 1 ] , a 2 = Π C ,
Revised: For any A ∈ S p + and convex feasible set C ⊂ R p , suppose a 1 = Π C , A [ b 1 ] , a 2 = Π C ,

Original: A [ b 2 ] , we have
Revised: A [ b 2 ] , we have

Original: [Equation]
Revised: [Equation]

Original: Lemma 13.
Revised: Lemma 13.

Original: For all i ∈ V if β 1 , β 2 ∈ [ 0 , 1 ) satisfy η = β 1 √ β 2 < 1 , then we have
Revised: For all i ∈ V if β 1 , β 2 ∈ [ 0 , 1 ) satisfy η = β 1 √ β 2 < 1 , then we have

Original: [Equation]
Revised: [Equation]

Original: where α t = α √ t for all t ∈ { 1 ,..., T } .
Revised: where α t = α √ t for all t ∈ { 1 ,..., T } .

Original: Proof.
Revised: Proof.

Original: Using the update rules in Algorithm 1, we have
Revised: Using the update rules in Algorithm 1, we have

Original: [Equation]
Revised: [Equation]

Original: where (i) follows from the fact that the update rules of m T and υ T can be written as m T
Revised: where (i) follows from the fact that the update rules of m T and υ T can be written as m T

Original: = ( 1 − β 1 )
Revised: = ( 1 − β 1 )

Original: ∑ Tl = 1 β T − lg l and υ T =
Revised: ∑ Tl = 1 β T − lg l and υ T =

Original: ( 1 − β 2 )
Revised: ( 1 − β 2 )

Original: ∑ Tl = 1 β T − lg 2 l , respectively.
Revised: ∑ Tl = 1 β T − lg 2 l , respectively.

Original: (ii) follows from Cauchy-Schwarz inequality and the fact that 0 ≤ β 1 < 1.
Revised: (ii) follows from Cauchy-Schwarz inequality and the fact that 0 ≤ β 1 < 1.

Original: Inequality (iii) follows since ∑ T l = 1 β T − l≤ 1 1 − β 1 .
Revised: Inequality (iii) follows since ∑ T l = 1 β T − l≤ 1 1 − β 1 .

Original: Hence, we have
Revised: Hence, we have

Original: [Equation]
Revised: [Equation]

Original: where inequality (i) follows since ∑ Tl =
Revised: where inequality (i) follows since ∑ Tl =

Original: t η l − t ≤ 1 ( 1 − η ) .
Revised: t η l − t ≤ 1 ( 1 − η ) .

Original: Inequality (ii) follows from Cauchy-Schwarz inequality.
Revised: Inequality (ii) follows from Cauchy-Schwarz inequality.

Original: The inequality (iii) follows since
Revised: The inequality (iii) follows since

Original: T
Revised: T

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Next, we provide an upper bound on the deviation of the local estimates at each iteration from their consensual value.
Revised: Next, we provide an upper bound on the deviation of the local estimates at each iteration from their consensual value.

Original: A similar result has been proven in Shahrampour & Jadbabaie (2017) for online decentralized mirror descent; however, the following lemma extends that of Shahrampour &
Revised: A similar result has been proven in Shahrampour & Jadbabaie (2017) for online decentralized mirror descent; however, the following lemma extends that of Shahrampour &

Original: Jadbabaie (2017) to the online adaptive setting and takes into account the sparsity of gradient vector.
Revised: Jadbabaie (2017) to the online adaptive setting and takes into account the sparsity of gradient vector.

Original: Lemma 14.
Revised: Lemma 14.

Original: ( Network Error with Sparse Data)
Revised: ( Network Error with Sparse Data)

Original: If β 1 , β 2 ∈ [ 0 , 1 ) satisfy η = β 1 √ β 2 < 1 , then the sequence x i , t generated by Algorithm 1 satisﬁes
Revised: If β 1 , β 2 ∈ [ 0 , 1 ) satisfy η = β 1 √ β 2 < 1 , then the sequence x i , t generated by Algorithm 1 satisﬁes

Original: [Equation]
Revised: [Equation]

Original: where ˆ V i , t = diag ( ˆ υ i , t ) and ¯ x t = 1 n ∑ ni = 1 x i , t .
Revised: where ˆ V i , t = diag ( ˆ υ i , t ) and ¯ x t = 1 n ∑ ni = 1 x i , t .

Original: Proof.
Revised: Proof.

Original: Let e i , t : = x i , t + 1 − ∑ nj = 1 [ W ] ij x j , t , where W satisﬁes equation 2.
Revised: Let e i , t : = x i , t + 1 − ∑ nj = 1 [ W ] ij x j , t , where W satisﬁes equation 2.

Original: Using the update rule of x i , t +in Algorithm 1, we have
Revised: Using the update rule of x i , t +in Algorithm 1, we have

Original: [Equation]
Revised: [Equation]

Original: where the ﬁrst inequality follows from Lemma 12.
Revised: where the ﬁrst inequality follows from Lemma 12.

Original: Further, from the deﬁnition of e i , t , we have
Revised: Further, from the deﬁnition of e i , t , we have

Original: [Equation]
Revised: [Equation]

Original: Now, from equation 2,equation 21 and  we have:
Revised: Now, from equation 2 and equation 21, we have

Original: [Equation]
Revised: [Equation]

Original: where ¯ e t = 1 n ∑ ni = 1 e i , t .
Revised: where ¯ e t = 1 n ∑ ni = 1 e i , t .

Original: Hence,
Revised: Hence,

Original: [Equation]
Revised: [Equation]

Original: It follows from equation 21 that
Revised: It follows from equation 21 that

Original: [Equation]
Revised: [Equation]

Original: Now, using equation 23 and equation
Revised: Now, using equation 23 and equation

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: Now

Original: [Equation]
Revised: [Equation]

Original:  taking the Euclidean norm of equation 24 and summing over t ∈ { 1 ,..., T } [Equation] one has:
Revised: Now, taking the Euclidean norm of equation 24 and summing over t ∈ { 1 ,..., T } , one has:

Original: [Equation]
Revised: [Equation]

Original: where step (i) follows from (cid:107) ∑ ni = 

Revised: where step (i) follows from (cid:107) ∑ ni =

Original: a i (cid:107) 2 ≤ n ∑ ni = 1 (cid:107) a i (cid:107) 2 , step (ii) follows from the following property of mixing matrix W Horn et al.
Revised: a i (cid:107) 2 ≤ n ∑ ni = 1 (cid:107) a i (cid:107) 2 , step (ii) follows from the following property of mixing matrix W Horn et al.

Original: [Equation]
Revised: [Equation]

Original: step (iii) follows from ∑ Tt = 1 t σ t 2 ( W ) < 1 ( 1 − σ 2 ( W )) 2 , step (iv) follows from equation 20 and step (v) follows from Lemma 13.
Revised: step (iii) follows from ∑ Tt = 1 t σ t 2 ( W ) < 1 ( 1 − σ 2 ( W )) 2 , step (iv) follows from equation 20 and step (v) follows from Lemma 13.

Original: Now, summing theinequality equation 25 over i ∈ V and using
Revised: Now, summing the equation 25 over i ∈ V and using

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: we complete the proof.
Revised: we complete the proof.

Original: Lemma 15.
Revised: Lemma 15.

Original: For the sequence x i , t generated by Algorithm 1, we have
Revised: For the sequence x i , t generated by Algorithm 1, we have

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: From the left side of equation
Revised: From the left side of equation

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: By construction of equation
Revised: By construction of equation

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: where the last inequality holds due to equation
Revised: where the last inequality holds due to equation

Original: [Equation]
Revised: [Equation]

Original: Summing equation 30 over t ∈ { 1 ,..., T } , the ﬁrst term telescopes, while the second is handled with equation 31.
Revised: Summing equation 30 over t ∈ { 1 ,..., T } , the ﬁrst term telescopes, while the second is handled with equation 31.

Original: Hence,
Revised: Hence,

Original: [Equation]
Revised: [Equation]

Original: where the last inequality follows since from deﬁnition of ˆ υ i , t , we have
Revised: where the last inequality follows since from deﬁnition of ˆ υ i , t , we have

Original: [Equation]
Revised: [Equation]

Original: Now, summing equation 32 over i ∈ V and using the inequality ∑ ni = 1 (cid:112)
Revised: Now, summing equation 32 over i ∈ V and using the inequality ∑ ni = 1 (cid:112)

Original: ˆ υ i , t ≤ √ n √ ˆ υ t , the claim in equation 28 follows.
Revised: ˆ υ i , t ≤ √ n √ ˆ υ t , the claim in equation 28 follows.

Original: Lemma 16.
Revised: Lemma 16.

Original: Suppose that the parameters β 1 , β 2 ∈ [ 0 , 1 ) satisfy η = β 1 √ β 2 < 1 .
Revised: Suppose that the parameters β 1 , β 2 ∈ [ 0 , 1 ) satisfy η = β 1 √ β 2 < 1 .

Original: Let β 1 , t = β 1 λ t − 1 , λ ∈ ( 0 , 1 ) and (cid:107) ∇ f i , t ( x ) (cid:107) ∞ ≤ G ∞ for all t ∈ { 1 ,..., T } .
Revised: Let β 1 , t = β 1 λ t − 1 , λ ∈ ( 0 , 1 ) and (cid:107) ∇ f i , t ( x ) (cid:107) ∞ ≤ G ∞ for all t ∈ { 1 ,..., T } .

Original: Then, using a step-size α t = α √ t for the sequence x i , t generated by Algorithm 1, we have
Revised: Then, using a step-size α t = α √ t for the sequence x i , t generated by Algorithm 1, we have

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: From convexity of f i
Revised: From convexity of f i

Original: [Equation]
Revised: [Equation]

Original: t ( · )
Revised: t ( · )

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: Individual terms in equation 34 can be bounded in the following way.
Revised: Individual terms in equation 34 can be bounded in the following way.

Original: From the Young’s inequality for products 3 , we have
Revised: From the Young’s inequality for products 3 , we have

Original: [Equation]
Revised: [Equation]

Original: Note also that:
Revised: Note also that:

Original: [Equation]
Revised: [Equation]

Original: Using Cauchy-Schwarz inequality
Revised: Using Cauchy-Schwarz inequality

Original: [Equation]
Revised: [Equation]

Original: equation 19 and equation
Revised: equation 19 and equation

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: Using equation 35 and equation 36, we have
Revised: Using equation 35 and equation 36, we have

Original: [Equation]
Revised: [Equation]

Original: In addition
Revised: In addition

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: where equation 38 follows from Cauchy-Schwarz inequality
Revised: where equation 38 follows from Cauchy-Schwarz inequality

Original: [Equation]
Revised: [Equation]

Original: Now
Revised: Now

Original: [Equation]
Revised: [Equation]

Original: using equation 38 we obtain
Revised: using equation 38 we obtain

Original: [Equation]
Revised: [Equation]

Original: where equation 39 utilize Cauchy-Schwarz inequality and equation 40 follows from equation 2,
Revised: where equation 39 utilize Cauchy-Schwarz inequality and equation 40 follows from equation 2,

Original: Lemma 14 and equation
Revised: Lemma 14 and equation

Original: [Equation]
Revised: [Equation]

Original: To bound I 1 , using the update rule of ˆ m i
Revised: To bound I 1 , using the update rule of ˆ m i

Original: [Equation]
Revised: [Equation]

Original: t in Algorithm
Revised: t in Algorithm

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: Now, by rearranging the above equality, we obtain:
Revised: Now, by rearranging the above equality, we obtain:

Original: [Equation]
Revised: [Equation]

Original: where ( II 1 ) follows from Lemma 11.
Revised: where ( II 1 ) follows from Lemma 11.

Original: The term ( II 2 ) is obtained by induction as follows: using the assumption, (cid:107) g
Revised: The term ( II 2 ) is obtained by induction as follows: using the assumption, (cid:107) g

Original: i , t (cid:107) ∞
Revised: i , t (cid:107) ∞

Original: ≤ G ∞ ; now, using the update rule of m i , t in Algorithm
Revised: ≤ G ∞ ; now, using the update rule of m i , t in Algorithm

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: ) where (cid:107) m i , t − 1 (cid:107) ∞ ≤ G ∞ by induction hypothesis.
Revised: where (cid:107) m i , t − 1 (cid:107) ∞ ≤ G ∞ by induction hypothesis.

Original: Note that from our assumption
Revised: Note that from our assumption

Original: [Equation]
Revised: [Equation]

Original: we have β 1 , t = β 1 λ t − 1 , λ ∈ ( 0 , 1 ) and β 1 , t ≤ β 1 .
Revised: we have β 1 , t = β 1 λ t − 1 , λ ∈ ( 0 , 1 ) and β 1 , t ≤ β 1 .

Original: Substituting this into equation
Revised: Substituting this into equation

Original: [Equation]
Revised: [Equation]

Original: we get
Revised: we get

Original: [Equation]
Revised: [Equation]

Original: where the last line comes from equation 1.
Revised: where the last line comes from equation 1.

Original: Plugging equation 37, equation 40, and equation 43 into equation 34, we obtain
Revised: Plugging equation 37, equation 40, and equation 43 into equation 34, we obtain

Original: [Equation]
Revised: [Equation]

Original: Now, since β 1 , t = β 1 λ t − 1 , λ ∈ ( 0 , 1 ) and β 1 , t ≤ β 1 , we have
Revised: Now, since β 1 , t = β 1 λ t − 1 , λ ∈ ( 0 , 1 ) and β 1 , t ≤ β 1 , we have

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Finally, using Lemma 15 andinequality equation 44, we obtain the desired result.
Revised: Finally, using Lemma 15 and equation 44, we obtain the desired result.

Original: 6.1. P ROOF OF T HEOREM
Revised: 6.1. P ROOF OF T HEOREM

Original: Proof.
Revised: Proof.

Original: From convexity of f i , t ( · ) it follows that
Revised: From convexity of f i , t ( · ) it follows that

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: Summing over t ∈ { 1 ,..., T } and i ∈ V
Revised: Summing over t ∈ { 1 ,..., T } and i ∈ V

Original: [Equation]
Revised: [Equation]

Original: and applying Lemma 16 and equation 40 gives the desired result.
Revised: and applying Lemma 16 and equation 40 gives the desired result.

Original: 6.1. P ROOF OF T HEOREM
Revised: 6.1. P ROOF OF T HEOREM

Original: Proof.
Revised: Proof.

Original: We just need to rework the proof of Lemma 16 and Theorem 4 using stochastic gradients by tracking the changes.
Revised: We just need to rework the proof of Lemma 16 and Theorem 4 using stochastic gradients by tracking the changes.

Original: Indeed, using stochastic gradient at the beginning of Lemma
Revised: Indeed, using stochastic gradient at the beginning of Lemma

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: Further
Revised: Further

Original: [Equation]
Revised: [Equation]

Original: if we replace any bound involving G ∞ which is an upper bound on the exact gradient with the norm of stochastic gradient
Revised: if we replace any bound involving G ∞ which is an upper bound on the exact gradient with the norm of stochastic gradient

Original: [Equation]
Revised: [Equation]

Original: we obtain
Revised: we obtain

Original: [Equation]
Revised: [Equation]

Original: Note that using Assumption
Revised: Note that using Assumption

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: Hence taking expectation from equation 46, using the above inequality and equation 42, we have
Revised: Hence taking expectation from equation 46, using the above inequality and equation 42, we have

Original: [Equation]
Revised: [Equation]

Original: According to the result in Lemma 15 and equation
Revised: According to the result in Lemma 15 and equation

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: Finally, using equation 45 we complete the proof.
Revised: Finally, using equation 45 we complete the proof.

Original: Next, we establish a series of lemmas used in the proof of Theorem 7 and 9.
Revised: Next, we establish a series of lemmas used in the proof of Theorems 7 and 9.

Original: Lemma 17.
Revised: Lemma 17.

Original: Attouch et al.
Revised: Attouch et al.

Original: Let G X be the projected gradient deﬁed in equation 5.
Revised: Let G X be the projected gradient deﬁed in equation 5.

Original: Then,
Revised: Then,

Original: G X ( ϖ , f i , α ) = 

Revised: G X ( ϖ , f i , α ) =

Original: if and only if ϖ is a critical point of equation 3.
Revised: if and only if ϖ is a critical point of equation 3.

Original: Next, we show that the projection map G X in Deﬁnition 1 is Lipschitz continuous.
Revised: Next, we show that the projection map G X in Deﬁnition 1 is Lipschitz continuous.

Original: Lemma 18.
Revised: Lemma 18.

Original: Suppose the second moment ˆ υ satisﬁes equation 11.
Revised: Suppose the second moment ˆ υ satisﬁes equation 11.

Original: Let x + 1 , i and x + 2 , i be given in
Revised: Let x + 1 , i and x + 2 , i be given in

Original: Deﬁnition 1 with m i √
Revised: Deﬁnition 1 with m i √

Original: ˆ υ i replaced by m 1 i √
Revised: ˆ υ i replaced by m 1 i √

Original: ˆ υ 1 i and m 2 i √
Revised: ˆ υ 1 i and m 2 i √

Original: ˆ υ 2 i , respectively.
Revised: ˆ υ 2 i , respectively.

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: where G 1 X and G 2 X are the projection maps corresponding to x + 1 , i and x + 2 , i , respectively.
Revised: where G 1 X and G 2 X are the projection maps corresponding to x + 1 , i and x + 2 , i , respectively.

Original: Proof.
Revised: Proof.

Original: Consider the optimality condition of equation 6, for any u ∈ X .
Revised: Consider the optimality condition of equation 6, for any u ∈ X .

Original: For each i ∈ V , observe that
Revised: For each i ∈ V , observe that

Original: [Equation]
Revised: [Equation]

Original: Taking u = x + 2 , i in equation 47, we have
Revised: Taking u = x + 2 , i in equation 47, we have

Original: [Equation]
Revised: [Equation]

Original: Likewise, setting u = x + 1 , i in equation
Revised: Likewise, setting u = x + 1 , i in equation

Original: [Equation]
Revised: [Equation]

Original: we get
Revised: we get

Original: [Equation]
Revised: [Equation]

Revised: Now, using equation

Original: [Equation]
Revised: [Equation]

Original: Now, using equation 49, equation 50 and equation 11, we obtain
Revised: equation 50 and equation 11, we obtain

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Without loss of generality, assumes that (cid:113) ˆ υ 1 i ≤ (cid:113) ˆ
Revised: Without loss of generality, assumes that (cid:113) ˆ υ 1 i ≤ (cid:113) ˆ

Original: υ 2 i .
Revised: υ 2 i .

Original: Then, using equation 5, we have
Revised: Then, using equation 5, we have

Original: [Equation]
Revised: [Equation]

Original: where (i) follows from equation 11 and (ii) follows from equation
Revised: where (i) follows from equation 11 and (ii) follows from equation

Original: [Equation]
Revised: [Equation]

Original: Lemma 19.
Revised: Lemma 19.

Original: Let β 1 , β 2 ∈ [ 0 , 1 ) satisfy η = β 1 √ β 2 < 1 .
Revised: Let β 1 , β 2 ∈ [ 0 , 1 ) satisfy η = β 1 √ β 2 < 1 .

Original: Then, for the sequence x i , t generated by
Revised: Then, for the sequence x i , t generated by

Original: Algorithm 1, we have
Revised: Algorithm 1, we have

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: In the proof of Lemma 14 we showed that
Revised: In the proof of Lemma 14 we showed that

Original: [Equation]
Revised: [Equation]

Original: Now, using the above equality, we have
Revised: Now, using the above equality, we have

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Also, using Lemma
Revised: Also, using Lemma

Original: [Equation]
Revised: [Equation]

Original: we get
Revised: we get

Original: [Equation]
Revised: [Equation]

Original: The above inequality together with the update rule of x i , t + 1 , imply that
Revised: The above inequality together with the update rule of x i , t + 1 , imply that

Original: [Equation]
Revised: [Equation]

Original: where the ﬁrst inequality follows from the nonexpansiveness property of the Euclidean projection
Revised: where the ﬁrst inequality follows from the nonexpansiveness property of the Euclidean projection

Original: [Equation]
Revised: [Equation]

Original: Substituting equation 26 and equation 54 into equation
Revised: Substituting equation 26 and equation 54 into equation

Original: [Equation]
Revised: [Equation]

Original: we get
Revised: we get

Original: [Equation]
Revised: [Equation]

Original: Summing the above inequality over i ∈ V
Revised: Summing the above inequality over i ∈ V

Original: [Equation]
Revised: [Equation]

Original: we conclude that
Revised: we conclude that

Original: [Equation]
Revised: [Equation]

Original: Lemma 20.
Revised: Lemma 20.

Original: For the sequence x i , t generated by Algorithm 1, we have
Revised: For the sequence x i , t generated by Algorithm 1, we have

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: A quick look at optimality condition of equation 6 veriﬁes that
Revised: A quick look at optimality condition of equation 6 veriﬁes that

Original: [Equation]
Revised: [Equation]

Original: Substituting z
Revised: Substituting z

Original: = x i , t into the above inequality
Revised: = x i , t into the above inequality

Original: [Equation]
Revised: [Equation]

Original: we get
Revised: we get

Original: [Equation]
Revised: [Equation]

Original: Now using equation
Revised: Now using equation

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: where the ﬁrst equality follows from the update rule of m i , t .
Revised: where the ﬁrst equality follows from the update rule of m i , t .

Original: The last inequality is valid since CauchySchwarz inequality and inequality (cid:112)
Revised: The last inequality is valid since CauchySchwarz inequality and inequality (cid:112)

Original: ˆ υ
Revised: ˆ υ

Original: i , t − 1 β 1 , t ≤ (cid:112)
Revised: i , t − 1 β 1 , t ≤ (cid:112)

Original: ˆ υ
Revised: ˆ υ

Original: i , t β 1 .
Revised: i , t β 1 .

Original: The claim then follows after using Deﬁnition
Revised: The claim then follows after using Deﬁnition

Original: [Equation]
Revised: [Equation]

Original: 6.1. P ROOF OF T HEOREM
Revised: 6.1. P ROOF OF T HEOREM

Original: Proof.
Revised: Proof.

Original: Let ¯ f i , t
Revised: Let ¯ f i , t

Original: ( x i , t )
Revised: ( x i , t )

Original: = 1 t ∑ ts = 1 f i , s ( x i , t ) .
Revised: = 1 t ∑ ts = 1 f i , s ( x i , t ) .

Original: Using Assumption 2, Taylor’s expansion and Deﬁnition 1, we have
Revised: Using Assumption 2, Taylor’s expansion and Deﬁnition 1, we have

Original: [Equation]
Revised: [Equation]

Original: The above inequality together with Lemma 20, equation 11 and β 1 , t ≤ β 1 , imply that
Revised: The above inequality together with Lemma 20, equation 11 and β 1 , t ≤ β 1 , imply that

Original: [Equation]
Revised: [Equation]

Original: Let ∆ i , t = ¯ f i , t ( x i , t ) − ¯ f i , t ( x ∗ t ) denotes the instantaneous loss at round t .
Revised: Let ∆ i , t = ¯ f i , t ( x i , t ) − ¯ f i , t ( x ∗ t ) denotes the instantaneous loss at round t .

Original: We have
Revised: We have

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Note that ( I ) can be bounded as follows:
Revised: Note that ( I ) can be bounded as follows:

Original: [Equation]
Revised: [Equation]

Original: where the ﬁrst inequality is due to the optimality of x ∗ t and the second inequality follows from equation 55.
Revised: where the ﬁrst inequality is due to the optimality of x ∗ t and the second inequality follows from equation 55.

Original: Now, combining equation 57 and equation 56 gives
Revised: Now, combining equation 57 and equation 56 gives

Original: [Equation]
Revised: [Equation]

Original: By rearranging the above inequality
Revised: By rearranging the above inequality

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: Using the deﬁnition of ∆ i
Revised: Using the deﬁnition of ∆ i

Original: [Equation]
Revised: [Equation]

Original: t +
Revised: t +

Original: [Equation]
Revised: [Equation]

Original: we get 1 t
Revised: we get 1 t

Original: + 1 ( f i , t + 1 ( x i , t + 1 ) − f i , t + 1 ( x ∗ t + 1 ))
Revised: + 1 ( f i , t + 1 ( x i , t + 1 ) − f i , t + 1 ( x ∗ t + 1 ))

Original: − ∆ i , t + 1 = − ( t t + 1 )( ¯ f i , t ( x i , t + 1 ) − ¯ f i , t ( x ∗ t + 1 )) .
Revised: − ∆ i , t + 1 = − ( t t + 1 )( ¯ f i , t ( x i , t + 1 ) − ¯ f i , t ( x ∗ t + 1 )) .

Original: Hence, using equation 58 and simplifying terms
Revised: Hence, using equation 58 and simplifying terms

Original: [Equation]
Revised: [Equation]

Original: we obtain
Revised: we obtain

Original: [Equation]
Revised: [Equation]

Original: Note that:
Revised: Note that:

Original: [Equation]
Revised: [Equation]

Original: where the ﬁrst equality uses
Revised: where the ﬁrst equality uses

Original: [Equation]
Revised: [Equation]

Original: The ﬁrst inequality follows from Assumption 2.
Revised: The ﬁrst inequality follows from Assumption 2.

Original: The second inequality follows from Lemma 19 and equation
Revised: The second inequality follows from Lemma 19 and equation

Original: [Equation]
Revised: [Equation]

Original: Summing equation 59 over i ∈ V , t ∈ { 1 ,..., T } and using equation
Revised: Summing equation 59 over i ∈ V , t ∈ { 1 ,..., T } and using equation

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: Note that ∑ Tt = 1 [ ( 2 − β 1 ) α t 2 ¯ υ − ρα 2 t 2 υ 2 ] > 0.
Revised: Note that ∑ Tt = 1 [ ( 2 − β 1 ) α t 2 ¯ υ − ρα 2 t 2 υ 2 ] > 0.

Original: Therefore, dividing both sides of the equation 62 by ∑ Tt = 1 [ ( 2 − β 1 ) α t 2 ¯ υ − ρα 2 t 2 υ 2 ] , we obtain equation 14.
Revised: Therefore, dividing both sides of the equation 63 by ∑ Tt = 1 [ ( 2 − β 1 ) α t 2 ¯ υ − ρα 2 t 2 υ 2 ] , we obtain equation 14.

Original: 6.1. P ROOF OF C OROLLARY
Revised: 6.1. P ROOF OF C OROLLARY

Original: Proof.
Revised: Proof.

Original: With the constant step-sizes α t = ( 2 − β 1 ) υ 2 2 ρ ¯ υ for all t ∈ { 1 ,..., T } , we have
Revised: With the constant step-sizes α t = ( 2 − β 1 ) υ 2 2 ρ ¯ υ for all t ∈ { 1 ,..., T } , we have

Original: [Equation]
Revised: [Equation]

Original: t
Revised: t

Original: [Equation]
Revised: [Equation]

Original: Therefore, using the above equality and equation 12 together with α t = ( 2 − β 1 ) υ 2 

Revised: Therefore, using the above equality and equation 12 together with α t = ( 2 − β 1 ) υ 2

Original: ρ ¯ υ for all t ∈ { 1 ,..., T } , we obtain
Revised: ρ ¯ υ for all t ∈ { 1 ,..., T } , we obtain

Original: [Equation]
Revised: [Equation]

Original: 63) and
Revised: 64) and

Original: [Equation]
Revised: [Equation]

Original: Combine the results in equation 63 and equation
Revised: Combine the results in equation 64 and equation

Original: [Equation]
Revised: [Equation]

Original: we have equation 13.
Revised: we have equation 13.

Original: 6.1. P ROOF OF T HEOREM
Revised: 6.1. P ROOF OF T HEOREM

Original: Proof.
Revised: Proof.

Original: Let
Revised: Let

Original: [Equation]
Revised: [Equation]

Original: where ∇ ¯ f i , t ( x i , t ) = 1 t ∑ ts = 1 ∇ f i , s ( x i , t ) denotes the minibatch stochastic gradient on node i .
Revised: where ∇ ¯ f i , t ( x i , t ) = 1 t ∑ ts = 1 ∇ f i , s ( x i , t ) denotes the minibatch stochastic gradient on node i .

Original: Since f i , t is ρ -smooth
Revised: Since f i , t is ρ -smooth

Original: [Equation]
Revised: [Equation]

Original: it follows that ¯ f i , t is also ρ -smooth.
Revised: it follows that ¯ f i , t is also ρ -smooth.

Original: Hence for every i ∈ V and t ∈ { 1 ,..., T } , we have
Revised: Hence for every i ∈ V and t ∈ { 1 ,..., T } , we have

Original: [Equation]
Revised: [Equation]

Original: where G X ( x i , t , ¯ f i , t , α t ) denotes the projected stochastic gradient on node i .
Revised: where G X ( x i , t , ¯ f i , t , α t ) denotes the projected stochastic gradient on node i .

Original: The above inequality together with Lemma 20, equation 11 and β 1 , t ≤ β 1 , imply that
Revised: The above inequality together with Lemma 20, equation 11 and β 1 , t ≤ β 1 , imply that

Original: [Equation]
Revised: [Equation]

Original: where the second inequality follows from the Cauchy-Schwartz inequality.
Revised: where the second inequality follows from the Cauchy-Schwartz inequality.

Original: The last inequality follows from the fact that β 1 , t = β 1 λ t − 1 , λ ∈ ( 0 , 1 ) and Lemma 18 if we set G 1 X ( x i , ¯ f i , α ) =
Revised: The last inequality follows from the fact that β 1 , t = β 1 λ t − 1 , λ ∈ ( 0 , 1 ) and Lemma 18 if we set G 1 X ( x i , ¯ f i , α ) =

Original: G X ( x i , t , ¯ f i , t , α t ) , and G 2 X ( x i , ¯ f i , α ) =
Revised: G X ( x i , t , ¯ f i , t , α t ) , and G 2 X ( x i , ¯ f i , α ) =

Original: G X ( x i , t , ¯ f i , t , α t ) .
Revised: G X ( x i , t , ¯ f i , t , α t ) .

Original: Let ∆ i , t = ¯ f i , t ( x i , t ) − ¯ f i , t ( x ∗ t ) denotes the instantaneous stochastic loss at time t .
Revised: Let ∆ i , t = ¯ f i , t ( x i , t ) − ¯ f i , t ( x ∗ t ) denotes the instantaneous stochastic loss at time t .

Original: We have [Equation] i 
Revised: We have ∆ i , t

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Observe that ( I ) can be bounded as follows:
Revised: Observe that ( I ) can be bounded as follows:

Original: [Equation]
Revised: [Equation]

Original: where the ﬁrst inequality is due to x ∗ t + 1 ∈ X and the optimality of x ∗ t and the second inequality is due to equation 65 and the fact that 0 ≤ β 1 , r < 1.
Revised: where the ﬁrst inequality is due to x ∗ t + 1 ∈ X and the optimality of x ∗ t and the second inequality is due to equation 66 and the fact that 0 ≤ β 1 , r < 1.

Original: Thus, using equation 66 and equation
Revised: Thus, using equation 67 and equation

Original: [Equation]
Revised: [Equation]

Original: we get ∆ i , t + 1 ≤t t +
Revised: we get ∆ i , t + 1 ≤

Original: [Equation]
Revised: [Equation]

Original: By rearranging the above inequality, we have t t + 1 [
Revised: By rearranging the above inequality

Original: [Equation]
Revised: [Equation]

Revised: we have t

Original: [Equation]
Revised: [Equation]

Original: Now, using deﬁnition of ∆ i , t + 1 ,1 t + 1 ( f i , t + 1 ( x i , t + 1 ) − f i , t +
Revised: Now, using deﬁnition of ∆ i , t + 1 ,

Original: [Equation]
Revised: [Equation]

Original: x ∗ t + 
Revised: + 1 ( f i ,

Original: [Equation]
Revised: [Equation]

Original: ) − ∆ i , t + 1 = − ( t t + 1 )( ¯ f i , t ( x i , t + 1 ) − ¯ f i , t ( x ∗ t + 1 )) , together with equation 68, we obtain [

Original: [Equation]
Revised: [Equation]

Original:  t 
Revised: ) − ¯ f i , t ( x ∗ t + 1 )) , together with equation 69, we obtain

Original: [Equation]
Revised: [Equation]

Original: Note that from
Revised: Note that from

Original: [Table]
Revised: [Table]

Revised: , we have

Revised: [Table]

Revised: (cid:2) (cid:104)

Revised: [Equation]

Revised: [Table]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: we
Revised: [Table] [Equation] [Table] (cid:105) (cid:3)

Original: [Equation]
Revised: [Table]

Revised: 0 and

Original: [Table]
Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Original: [Equation]
Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Original: which implies that
Revised: which implies that

Original: [Equation]
Revised: [Equation]

Original: The above inequality together with Cauchy-Schwarz for expextations
Revised: The above inequality together with Cauchy-Schwarz for expextations, imply that

Original: [Equation]
Revised: [Equation]

Original: imply that
Revised: Now [Equation] using equation 71 and equation

Original: [Equation]
Revised: [Equation]

Original: Now, using equation 70 and equation 61, summing equation 69 over i ∈ V and t ∈ { 1 ,..., T } , and taking expectation, we obtain
Revised: summing equation 70 over i ∈ V and t ∈ { 1 ,..., T } [Equation] and taking expectation

Original: [Equation]
Revised: [Equation]

Original:  Note that ∑ Tt = 1 [ ( 2 − 3 β 1 ) α t 2 ¯ υ− ρα 2 t 2 υ 2 ] > 0 , and
Revised: we obtain [Equation] [Equation] Note that ∑ Tt = 1 [ ( 2 − 3 β 1 ) α t 2 ¯ υ

Original: [Equation]
Revised: [Equation]

Revised: t

Original: [Equation]
Revised: [Equation]

Original: Therefore, dividing both sides of the equation 74 by ∑ Tt = 1 [ ( 2 − 3 β 1 ) α t 2 ¯ υ −
Revised: 2 ] > 0 , and

Original: [Equation]
Revised: [Equation]

Original: {+Therefore, dividing both sides of the equation 75 by ∑ Tt = 
+}
Revised: {+Therefore, dividing both sides of the equation 75 by ∑ Tt = 
+}

Revised: [ ( 2 − 3 β 1 )

Original:  2 ] , using equation 72, we complete the proof.
Revised: α t 2 ¯ υ − ρα 2 t 2 υ 2 ] , using equation 73, we complete the proof.

Original: 6.1. P ROOF OF C OROLLARY
Revised: 6.1. P ROOF OF C OROLLARY

Original: Proof.
Revised: Proof.

Original: Let ϑ t =
Revised: Let ϑ t =

Original: ∑ Tt = 

Revised: ∑ Tt =

Original: [ ( 2 − β 1 ) α t 2 ¯ [Equation] − ρα 2 t 2 υ 2 ] .
Revised: [ ( 2 − β 1 ) α t 2 ¯ υ − ρα 2 t 2 υ 2 ] .

Original: We claim that if
Revised: We claim that if

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: then ϑ t ≥ 1 / 2.
Revised: then ϑ t ≥ 1 / 2.

Original: One can easily seen that T must satisfy ( 2 − β 1 )
Revised: One can easily seen that T must satisfy ( 2 − β 1 )

Original: α t 2 ¯ υ − ρα 2 t 2 υ 2 ≥ 12 T which results in
Revised: α t 2 ¯ υ − ρα 2 t 2 υ 2 ≥ 12 T which results in

Original: [Equation]
Revised: [Equation]

Original: By rearranging the above inequality, we obtain
Revised: By rearranging the above inequality, we obtain

Original: [Equation]
Revised: [Equation]

Original: Now, if A 1 , A 2 ≤ 12 , then α t ≤ υ 2 ( 2 − β 1 ) 2 ρ ¯ υ , and α t ≥ 2 ¯ υ T ( 2 − β 1 ) , respectively.
Revised: Now, if A 1 , A 2 ≤ 12 , then α t ≤ υ 2 ( 2 − β 1 ) 2 ρ ¯ υ , and α t ≥ 2 ¯ υ T ( 2 − β 1 ) , respectively.

Original: These bounds together with our assumption on α t give equation 73.
Revised: These bounds together with our assumption on α t give equation 74.

Original: Now, let equation 73 holds.
Revised: Now, let equation 74 holds.

Original: Using equation
Revised: Using equation

Original: [Equation]
Revised: [Equation]

Original: and since ϑ t ≥ 1 /
Revised: and since ϑ t ≥ 1 /

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: where the last inequality follows from equation 60 and equation 70.
Revised: where the last inequality follows from equation 60, equation 61 and equation 71.

Original: We proceed to bound RHS of equation 74.
Revised: We proceed to bound RHS of equation 75.

Original: By substituting α t = α √ nT and β 1 , t = β 1 λ t − 1 , λ ∈ ( 0 , 1 ) into the last two terms of equation 74, we have
Revised: By substituting α t = α √ nT and β 1 , t = β 1 λ t − 1 , λ ∈ ( 0 , 1 ) into the last two terms of equation 75, we have

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Further, using equation 74 and Lemma
Revised: Further, using equation 75 and Lemma

Original: [Equation]
Revised: [Equation]

Original: we get
Revised: we get

Original: [Equation]
Revised: [Equation]

Original: Tn

Original: [Equation]

Original: [Equation]

Original: Similarly, using equation 74 and Lemma
Revised: Similarly, using equation 75 and Lemma

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: where the second inequality  follows because ∑ Tt = 2 1 t 2 < 1.
Revised: where the second inequality is due to Cauchy-Schwarz inequality and the last inequality follows because ∑ Tt = 2 1 t 2 ≤ 1.

Original: Using equation 16a, equation76, equation 77,and equation 78  are bounded by equation 75.
Revised: Using equation 16a, equation 77, equation 78 and equation 79 are bounded by equation 76.

Original: This completes the proof.
Revised: This completes the proof.

Original: 6. S ENSITIVITY OF DADAM TO ITS PARAMETERS
Revised: 6. S ENSITIVITY OF DADAM TO ITS PARAMETERS

Original: Next, we examine the sensitivity of DADAM on the parameters related to the network connection and update of the moment estimate.
Revised: Next, we examine the sensitivity of DADAM on the parameters related to the network connection and update of the moment estimate.

Original: As it is clear from Figure 3 the convergence of training accuracy value happens faster for sparser networks (higher second largest singular value).
Revised: As it is clear from Figure 3 the convergence of training accuracy value happens faster for sparser networks (higher second largest singular value).

Original: This is similar to the trend observed for FedAvg algorithm while reducing parameter C which makes the agent interaction matrix sparser.
Revised: This is similar to the trend observed for FedAvg algorithm while reducing parameter C which makes the agent interaction matrix sparser.

Original: This is also expected as discussed in Theorems 4 and 5.
Revised: This is also expected as discussed in Theorems 4 and 5.

Original: Note that with the availability of a central parameter server (as in FedAvg algorithm), sparser topology may be useful for a faster convergence, however, topology density is important for a distributed learning scheme with decentralized computation.
Revised: Note that with the availability of a central parameter server (as in FedAvg algorithm), sparser topology may be useful for a faster convergence, however, topology density is important for a distributed learning scheme with decentralized computation.

Original: We also empirically evaluate the effect of the β 3 in Algorithm 1.
Revised: We also empirically evaluate the effect of the β 3 in Algorithm 1.

Original: We consider a range of hyperparameter choices, i.e. β 3 ∈ { 0 , 0 .
Revised: We consider a range of hyperparameter choices, i.e. β 3 ∈ { 0 , 0 .

Original: From Figure 4 it can be easily seen that DADAM per- forms equal or better than AMSGrad ( β 3 = 0), regardless of the hyper-parameter setting forβ 1 and β 2 .
Revised: From Figure 4 it can be easily seen that DADAM per- forms equal or better than AMSGrad ( β 3 = 0), regardless of the hyper-parameter setting for