Original: S TOCHASTIC G RADIENT P USH FOR D ISTRIBUTED
Revised: S TOCHASTIC G RADIENT P USH FOR D ISTRIBUTED

Original: D EEP L EARNING
Revised: D EEP L EARNING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Large mini-batch parallel SGD is commonly used for distributed training of deep networks.
Revised: Large mini-batch parallel SGD is commonly used for distributed training of deep networks.

Original: Approaches that use tightly-coupled exact distributed averaging based on AllReduce are sensitive to slow nodes and high-latency communication.
Revised: Approaches that use tightly-coupled exact distributed averaging based on AllReduce are sensitive to slow nodes and high-latency communication.

Original: In this work we show the applicability of Stochastic Gradient Push (SGP) for distributed training.
Revised: In this work we show the applicability of Stochastic Gradient Push (SGP) for distributed training.

Original: SGP uses a gossip algorithm called PushSum for approximate distributed averaging, allowing for much more loosely coupled communications which can be beneﬁcial in high-latency or high-variability scenarios.
Revised: SGP uses a gossip algorithm called PushSum for approximate distributed averaging, allowing for much more loosely coupled communications which can be beneﬁcial in high-latency or high-variability scenarios.

Original: The tradeoff is that approximate distributed averaging injects additional noise in the gradient which can affect the train and test accuracies.
Revised: The tradeoff is that approximate distributed averaging injects additional noise in the gradient which can affect the train and test accuracies.

Original: We prove that SGP converges to a stationary point of smooth, non-convex objective functions.
Revised: We prove that SGP converges to a stationary point of smooth, non-convex objective functions.

Original: Furthermore, we validate empirically the potential of SGP.
Revised: Furthermore, we validate empirically the potential of SGP.

Original: For example, using 32 nodes with
Revised: For example, using 32 nodes with

Original: GPUs per node to train ResNet-50 on ImageNet, where nodes communicate over 10Gbps Ethernet, SGP completes 90 epochs in around 1.6 hours while AllReduce
Revised: GPUs per node to train ResNet-50 on ImageNet, where nodes communicate over 10Gbps Ethernet, SGP completes 90 epochs in around 1.6 hours while AllReduce

Original: SGD takes over 5 hours, and the top-1 validation accuracy of SGP remains within 1.2% of that obtained using AllReduce SGD.
Revised: SGD takes over 5 hours, and the top-1 validation accuracy of SGP remains within 1.2% of that obtained using AllReduce SGD.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Deep Neural Networks (DNNs) are the state-of-the art machine learning approach in many application areas, including image recognition (
Revised: Deep Neural Networks (DNNs) are the state-of-the art machine learning approach in many application areas, including image recognition (

Original: He et al., 2016) and natural language processing (Vaswani et al., 2017).
Revised: He et al., 2016) and natural language processing (Vaswani et al., 2017).

Original: Stochastic Gradient Descent (SGD) is the current workhorse for training neural networks.
Revised: Stochastic Gradient Descent (SGD) is the current workhorse for training neural networks.

Original: The algorithm optimizes the network parameters, x , to minimize a loss function, f ( · ) , through gradient descent, where the loss function’s gradients are approximated using a subset of training examples (a mini-batch).
Revised: The algorithm optimizes the network parameters, x , to minimize a loss function, f ( · ) , through gradient descent, where the loss function’s gradients are approximated using a subset of training examples (a mini-batch).

Original: DNNs often require large amounts of training data and trainable parameters, necessitating non-trivial computational requirements (Wu et al., 2016; Mahajan et al., 2018).
Revised: DNNs often require large amounts of training data and trainable parameters, necessitating non-trivial computational requirements (Wu et al., 2016; Mahajan et al., 2018).

Original: There is a need for efﬁcient methods to train DNNs in large-scale computing environments.
Revised: There is a need for efﬁcient methods to train DNNs in large-scale computing environments.

Original: A parallel version of SGD is usually adopted for large-scale, distributed training (Goyal et al., 2017;
Revised: A parallel version of SGD is usually adopted for large-scale, distributed training (Goyal et al., 2017;

Original: Li et al., 2014).
Revised: Li et al., 2014).

Original: Worker nodes compute local mini-batch gradients of the loss function on different subsets of the data, and then calculate an exact inter-node average gradient using either the A LL R EDUCE communication primitive, in synchronous implementations (Goyal et al., 2017), or using a central parameter server, in asynchronous implementations (Dean et al., 2012).
Revised: Worker nodes compute local mini-batch gradients of the loss function on different subsets of the data, and then calculate an exact inter-node average gradient using either the A LL R EDUCE communication primitive, in synchronous implementations (Goyal et al., 2017), or using a central parameter server, in asynchronous implementations (Dean et al., 2012).

Original: Using a parameter server to aggregate gradients introduces a potential bottleneck and a central point of failure (Lian et al., 2017).
Revised: Using a parameter server to aggregate gradients introduces a potential bottleneck and a central point of failure (Lian et al., 2017).

Original: The A LL R EDUCE primitive computes the exact average gradient at all workers in a decentralized manner, avoiding issues associated with centralized communication and computation.
Revised: The A LL R EDUCE primitive computes the exact average gradient at all workers in a decentralized manner, avoiding issues associated with centralized communication and computation.

Original: However, exact averaging algorithms like A LL R EDUCE are not robust in high-latency or highvariability platforms, e.g. , where the network bandwidth may be a signiﬁcant bottleneck, because they involve tightly-coupled, blocking communication ( i.e. , the call does not return until all nodes have ﬁnished aggregating).
Revised: However, exact averaging algorithms like A LL R EDUCE are not robust in high-latency or highvariability platforms, e.g. , where the network bandwidth may be a signiﬁcant bottleneck, because they involve tightly-coupled, blocking communication ( i.e. , the call does not return until all nodes have ﬁnished aggregating).

Original: Moreover, aggregating gradients across all the nodes in the network can introduce non-trivial computational overhead when there are many nodes, or when the gradients themselves are large.
Revised: Moreover, aggregating gradients across all the nodes in the network can introduce non-trivial computational overhead when there are many nodes, or when the gradients themselves are large.

Original: This issue motivates the investigation of a decentralized and inexact version of SGD to reduce the overhead associated with distributed training.
Revised: This issue motivates the investigation of a decentralized and inexact version of SGD to reduce the overhead associated with distributed training.

Original: There have been numerous decentralized optimization algorithms proposed and studied in the control-systems literature that leverage consensus-based approaches to aggregate information; see the recent survey Nedi´c et al.
Revised: There have been numerous decentralized optimization algorithms proposed and studied in the control-systems literature that leverage consensus-based approaches to aggregate information; see the recent survey Nedi´c et al.

Original: and references therein.
Revised: and references therein.

Original: Rather than exactly aggregating gradi- ents (as with A LL R EDUCE ), this line of work uses less-coupled message passing algorithms which compute inexact distributed averages.
Revised: Rather than exactly aggregating gradi- ents (as with A LL R EDUCE ), this line of work uses less-coupled message passing algorithms which compute inexact distributed averages.

Original: Most previous work in this area has focused on theoretical convergence analysis assuming convex objectives.
Revised: Most previous work in this area has focused on theoretical convergence analysis assuming convex objectives.

Original: Recent work has begun to investigate their applicability to large-scale training of DNNs (Lian et al., 2017; Jiang et al., 2017).
Revised: Recent work has begun to investigate their applicability to large-scale training of DNNs (Lian et al., 2017; Jiang et al., 2017).

Original: However, these papers study methods based on communication patterns which are static (the same at every iteration) and symmetric (if i sends to j , then i must also receive from j before proceeding).
Revised: However, these papers study methods based on communication patterns which are static (the same at every iteration) and symmetric (if i sends to j , then i must also receive from j before proceeding).

Original: Such methods inherently require blocking and communication overhead.
Revised: Such methods inherently require blocking and communication overhead.

Original: State-of-the-art consensus optimization methods build on the P USH S UM algorithm for approximate distributed averaging (Kempe et al., 2003; Nedi´c et al., 2018), which allows for non-blocking, time-varying, and directed (asymmetric) communication.
Revised: State-of-the-art consensus optimization methods build on the P USH S UM algorithm for approximate distributed averaging (Kempe et al., 2003; Nedi´c et al., 2018), which allows for non-blocking, time-varying, and directed (asymmetric) communication.

Original: Since SGD already uses stochastic mini-batches, the hope is that an inexact average mini-batch will be as useful as the exact one if the averaging error is sufﬁciently small relative to the variability in the stochastic gradient.
Revised: Since SGD already uses stochastic mini-batches, the hope is that an inexact average mini-batch will be as useful as the exact one if the averaging error is sufﬁciently small relative to the variability in the stochastic gradient.

Original: This paper studies the use of Stochastic Gradient Push (SGP), an algorithm blending SGD and
Revised: This paper studies the use of Stochastic Gradient Push (SGP), an algorithm blending SGD and

Original: P USH S UM , for distributed training of deep neural networks.
Revised: P USH S UM , for distributed training of deep neural networks.

Original: We provide a theoretical analysis of
Revised: We provide a theoretical analysis of

Original: SGP, showing it converges for smooth non-convex objectives.
Revised: SGP, showing it converges for smooth non-convex objectives.

Original: We also evaluate SGP experimentally, training ResNets on ImageNet using up to 32 nodes, each with 8 GPUs ( i.e. , 256 GPUs in total).
Revised: We also evaluate SGP experimentally, training ResNets on ImageNet using up to 32 nodes, each with 8 GPUs ( i.e. , 256 GPUs in total).

Original: Our main contributions are summarized as follows:
Revised: Our main contributions are summarized as follows:

Original: • We provide the ﬁrst convergence analysis for Stochastic Gradient Push when the objective function is smooth and non-convex.
Revised: • We provide the ﬁrst convergence analysis for Stochastic Gradient Push when the objective function is smooth and non-convex.

Original: We show that, for an appropriate choice of the step size, SGP converges to a stationary point at a rate of O (cid:16) 1 / √ nK (cid:17) , where n is the number of nodes and K is the number of iterations.
Revised: We show that, for an appropriate choice of the step size, SGP converges to a stationary point at a rate of O (cid:16) 1 / √ nK (cid:17) , where n is the number of nodes and K is the number of iterations.

Original: • In a high-latency scenario, where nodes communicate over 10Gbps Ethernet, SGP runs up to 3 × faster than A LL R EDUCE SGD and exhibits 88.6% scaling efﬁciency over the range from 4–32 nodes.
Revised: • In a high-latency scenario, where nodes communicate over 10Gbps Ethernet, SGP runs up to 3 × faster than A LL R EDUCE SGD and exhibits 88.6% scaling efﬁciency over the range from 4–32 nodes.

Original: • The top-1 validation accuracy of SGP matches that of A LL R EDUCE SGD for up to 8 nodes (64 GPUs), and remains within 1.2% of A LL R EDUCE SGD for larger networks.
Revised: • The top-1 validation accuracy of SGP matches that of A LL R EDUCE SGD for up to 8 nodes (64 GPUs), and remains within 1.2% of A LL R EDUCE SGD for larger networks.

Original: • In a low-latency scenario, where nodes communicate over a 100Gbps InﬁniBand network supporting GPUDirect, SGP is on par with A LL R EDUCE SGD in terms of running time, and SGP exhibits 92.4% scaling efﬁciency.
Revised: • In a low-latency scenario, where nodes communicate over a 100Gbps InﬁniBand network supporting GPUDirect, SGP is on par with A LL R EDUCE SGD in terms of running time, and SGP exhibits 92.4% scaling efﬁciency.

Original: • In comparison to other  decentralized consensus-based approaches that require symmetric messaging, SGP runs faster and it produces models with better validation accuracy.
Revised: • In comparison to other synchronous decentralized consensus-based approaches that require symmetric messaging, SGP runs faster and it produces models with better validation accuracy.

Original:  P RELIMINARIES
Revised:  P RELIMINARIES

Original: Problem formulation.
Revised: Problem formulation.

Original: We consider the setting where a network of n nodes cooperates to solve the stochastic consensus optimization problem
Revised: We consider the setting where a network of n nodes cooperates to solve the stochastic consensus optimization problem

Original: [Equation]
Revised: [Equation]

Original: Each node has local data following a distribution D i , and the nodes wish to cooperate to ﬁnd the parameters x of a DNN that minimizes the average loss with respect to their data, where F i is the loss function at node i .
Revised: Each node has local data following a distribution D i , and the nodes wish to cooperate to ﬁnd the parameters x of a DNN that minimizes the average loss with respect to their data, where F i is the loss function at node i .

Original: Moreover, the goal codiﬁed in the constraints is for the nodes to reach agreement ( i.e. , consensus) on the solution they report.
Revised: Moreover, the goal codiﬁed in the constraints is for the nodes to reach agreement ( i.e. , consensus) on the solution they report.

Original: We assume that nodes can locally evaluate stochastic gradients ∇ F ( x i ; ξ i ) , ξ i ∼ D i , but they must communicate to access information about the objective functions at other nodes.
Revised: We assume that nodes can locally evaluate stochastic gradients ∇ F ( x i ; ξ i ) , ξ i ∼ D i , but they must communicate to access information about the objective functions at other nodes.

Original: Distributed averaging.
Revised: Distributed averaging.

Original: The problem described above encompasses distributed training based on data parallelism.
Revised: The problem described above encompasses distributed training based on data parallelism.

Original: There a canonical approach is large mini-batch parallel stochastic gradient descent: for an overall mini-batch of size nb , each node computes a local stochastic mini-batch gradient using b samples, and then the nodes use the A LL R EDUCE communication primitive to compute the average gradient at every node.
Revised: There a canonical approach is large mini-batch parallel stochastic gradient descent: for an overall mini-batch of size nb , each node computes a local stochastic mini-batch gradient using b samples, and then the nodes use the A LL R EDUCE communication primitive to compute the average gradient at every node.

Original: Let f i ( x i ) =
Revised: Let f i ( x i ) =

Original: E ξ i ∼ D
Revised: E ξ i ∼ D

Original: i F i ( x i ; ξ i ) denote the objective at node i , and let f ( x ) = 1 n (cid:80) ni
Revised: i F i ( x i ; ξ i ) denote the objective at node i , and let f ( x ) = 1 n (cid:80) ni

Original: =

Revised: =

Original: f i ( x ) denote the overall objective.
Revised: f i ( x ) denote the overall objective.

Original: Since ∇ f ( x ) = 1 n (cid:80) ni =1 ∇ f i ( x ) , averaging gradients via A LL R EDUCE provides an exact stochastic gradient of f .
Revised: Since ∇ f ( x ) = 1 n (cid:80) ni =1 ∇ f i ( x ) , averaging gradients via A LL R EDUCE provides an exact stochastic gradient of f .

Original: Typical implementations of
Revised: Typical implementations of

Original: A LL R EDUCE have each node send and receive 2 n − 1 n B bytes, where B is the size (in bytes) of the tensor being reduced, and involve 2 log 2 ( n ) communication steps (Rabenseifner, 2004).
Revised: A LL R EDUCE have each node send and receive 2 n − 1 n B bytes, where B is the size (in bytes) of the tensor being reduced, and involve 2 log 2 ( n ) communication steps (Rabenseifner, 2004).

Original: Moreover,
Revised: Moreover,

Original: A LL R EDUCE is a blocking primitive, meaning that no node will proceed with local computations until the primitive returns.
Revised: A LL R EDUCE is a blocking primitive, meaning that no node will proceed with local computations until the primitive returns.

Original: Approximate distributed averaging.
Revised: Approximate distributed averaging.

Original: In this work we explore the alternative approach of using a gossip algorithm for approximate distributed averaging—speciﬁcally, the P USH S UM algorithm.
Revised: In this work we explore the alternative approach of using a gossip algorithm for approximate distributed averaging—speciﬁcally, the P USH S UM algorithm.

Original: Gossip algorithms typically use linear iterations for averaging.
Revised: Gossip algorithms typically use linear iterations for averaging.

Original: For example, let y (0) i ∈
Revised: For example, let y (0) i ∈

Original: R n be a vector at node i , and consider the goal of computing the average vector 1 n (cid:80) ni =1 y (0)
Revised: R n be a vector at node i , and consider the goal of computing the average vector 1 n (cid:80) ni =1 y (0)

Original: i at all nodes.
Revised: i at all nodes.

Original: Stack the initial vectors into a matrix Y (0) ∈
Revised: Stack the initial vectors into a matrix Y (0) ∈

Original: R n × d with one row per node.
Revised: R n × d with one row per node.

Original: Typical gossip iterations have the form Y ( k +1)
Revised: Typical gossip iterations have the form Y ( k +1)

Original: =
Revised: =

Original: P ( k ) Y ( k ) where P ( k ) ∈ R n × n is referred to as the mixing matrix.
Revised: P ( k ) Y ( k ) where P ( k ) ∈ R n × n is referred to as the mixing matrix.

Original: This corresponds to the update y ( k +1)
Revised: This corresponds to the update y ( k +1)

Original: i=
Revised: i

Original: (cid:80) nj =1 p ( k ) i,j y ( k ) j at node i .
Revised: [Equation]

Original:  To implement this update, node i only needs to receive messages from other nodes j for which p ( k ) i,j (cid:54) = 0 , so it will be appealing to use sparse P ( k ) to reduce communications.
Revised: at node i . To implement this update, node i only needs to receive messages from other nodes j for which p ( k ) i,j (cid:54) = 0 , so it will be appealing to use sparse P ( k ) to reduce communications.

Original: Drawing inspiration from the theory of Markov chains (Seneta, 1981), the mixing matrices P ( k ) are designed to be column stochastic.
Revised: Drawing inspiration from the theory of Markov chains (Seneta, 1981), the mixing matrices P ( k ) are designed to be column stochastic.

Original: Then, under mild conditions ( e.g. , ensuring that information from every node eventually reaches all other nodes) one can show that lim K →∞ (cid:81)
Revised: Then, under mild conditions ( e.g. , ensuring that information from every node eventually reaches all other nodes) one can show that lim K →∞ (cid:81)

Original: Kk =
Revised: Kk =

Original: P
Revised: P

Original: k )
Revised: k )

Original: =
Revised: =

Original: π 1 (cid:62) , where π is the ergodic limit of the chain and 1 is a vector with all entries equal to 1 .
Revised: π 1 (cid:62) , where π is the ergodic limit of the chain and 1 is a vector with all entries equal to 1 .

Original: Consequently, the gossip iterations converge to a limit Y ( ∞ ) =
Revised: Consequently, the gossip iterations converge to a limit Y ( ∞ ) =

Original: π (cid:0) 1 (cid:62) Y (0) (cid:1) ; i.e. , the value at node i converges to y ( ∞ )
Revised: π (cid:0) 1 (cid:62) Y (0) (cid:1) ; i.e. , the value at node i converges to y ( ∞ )

Original: i =
Revised: i =

Original: π i (cid:80) nj =1 y (0) j .
Revised: π i (cid:80) nj =1 y (0) j .

Original: When the matrices P ( k ) are symmetric, it is straightforward to design the algorithm so
Revised: When the matrices P ( k ) are symmetric, it is straightforward to design the algorithm so

Original: that π i = 1 /n for all i by making P ( k ) doubly stochastic.
Revised: that π i = 1 /n for all i by making P ( k ) doubly stochastic.

Original: However, symmetric P ( k ) has strong practical ramiﬁcations, such as requiring care in the implementation to avoid deadlocks.
Revised: However, symmetric P ( k ) has strong practical ramiﬁcations, such as requiring care in the implementation to avoid deadlocks.

Original: The P USH S UM algorithm only requires that P ( k ) be column-stochastic, and not necessarily symmetric (so node i may send to node j , but not necessarily vice versa).
Revised: The P USH S UM algorithm only requires that P ( k ) be column-stochastic, and not necessarily symmetric (so node i may send to node j , but not necessarily vice versa).

Original: Instead, one additional scalar parameter w ( k ) i is maintained at each node.
Revised: Instead, one additional scalar parameter w ( k ) i is maintained at each node.

Original: The parameter is initialized to w (0) i
Revised: The parameter is initialized to w (0) i

Original: = 1 for all i , and updated using the same linear iteration, w ( k +1)
Revised: = 1 for all i , and updated using the same linear iteration, w ( k +1)

Original: =
Revised: =

Original: P ( k ) w ( k ) .
Revised: P ( k ) w ( k ) .

Original: Consequently, the parameter converges to w ( ∞ ) =
Revised: Consequently, the parameter converges to w ( ∞ ) =

Original: π ( 1 (cid:62) w (0) ) , or w ( ∞ ) i =
Revised: π ( 1 (cid:62) w (0) ) , or w ( ∞ ) i =

Original: π i n at node i .
Revised: π i n at node i .

Original: Thus each node can recover the average of the initial vectors by computing the de-biased ratio y ( ∞ ) i /w ( ∞ ) i .
Revised: Thus each node can recover the average of the initial vectors by computing the de-biased ratio y ( ∞ ) i /w ( ∞ ) i .

Original: In practice, we stop after a ﬁnite number of gossip iterations K and compute y ( K )
Revised: In practice, we stop after a ﬁnite number of gossip iterations K and compute y ( K )

Original: i /w ( K ) i .
Revised: i /w ( K ) i .

Original: The distance of the de-biased ratio to the exact average can be quantiﬁed in terms of properties of the matrices { P ( k ) } K − 1 k =0 .
Revised: The distance of the de-biased ratio to the exact average can be quantiﬁed in terms of properties of the matrices { P ( k ) } K − 1 k =0 .

Original: Let
Revised: Let

Original: N out i ( k ) =
Revised: N out i ( k ) =

Original: { j : p ( k ) j,i > 0 } and N in i ( k ) =
Revised: { j : p ( k ) j,i > 0 } and N in i ( k ) =

Original: { j : p ( k ) i,j > 0 } denote the sets of nodes that i transmits to and receives from, respectively, at iteration k .
Revised: { j : p ( k ) i,j > 0 } denote the sets of nodes that i transmits to and receives from, respectively, at iteration k .

Original: If we use B bytes to represent the vector y ( k ) i , then node i sends and receives (cid:12)(cid:12) N out i ( k ) (cid:12)(cid:12) B and (cid:12)(cid:12) N in i ( k ) (cid:12)(cid:12) B bytes, respectively, per iteration.
Revised: If we use B bytes to represent the vector y ( k ) i , then node i sends and receives (cid:12)(cid:12) N out i ( k ) (cid:12)(cid:12) B and (cid:12)(cid:12) N in i ( k ) (cid:12)(cid:12) B bytes, respectively, per iteration.

Original: In our experiments we use graph sequences with (cid:12)(cid:12) N out i ( k ) (cid:12)(cid:12) =
Revised: In our experiments we use graph sequences with (cid:12)(cid:12) N out i ( k ) (cid:12)(cid:12) =

Original: (cid:12)(cid:12) N in i ( k ) (cid:12)(cid:12)
Revised: (cid:12)(cid:12) N in i ( k ) (cid:12)(cid:12)

Original: = 1 or 2 , and ﬁnd that approximate averaging is both fast and still facilitates training.
Revised: = 1 or 2 , and ﬁnd that approximate averaging is both fast and still facilitates training.

Original:  S TOCHASTIC G RADIENT P USH
Revised:  S TOCHASTIC G RADIENT P USH

Original: Algorithm description.
Revised: Algorithm description.

Original: The stochastic gradient push (SGP) method for solving equation 1 is obtained by interleaving one local stochastic gradient descent update at each node with one iteration of P USH S UM .
Revised: The stochastic gradient push (SGP) method for solving equation 1 is obtained by interleaving one local stochastic gradient descent update at each node with one iteration of P USH S UM .

Original: Each node maintains three variables: the model parameters x
Revised: Each node maintains three variables: the model parameters x

Original: ( k ) i at node i ,
Revised: ( k ) i at node i ,

Original: the scalar
Revised: the scalar

Original: P USH S UM weight w ( k ) i , and the de-biased parameters z ( k )
Revised: P USH S UM weight w ( k ) i , and the de-biased parameters z ( k )

Original: i = (cid:0) w ( k ) i (cid:1) − 1 x ( k ) i .
Revised: i = (cid:0) w ( k ) i (cid:1) − 1 x ( k ) i .

Original: The initial x (0) i and z (0) i can be initialized to any arbitrary value as long as x (0)
Revised: The initial x (0) i and z (0) i can be initialized to any arbitrary value as long as x (0)

Original: i =
Revised: i =

Original: z (0) i .
Revised: z (0) i .

Original: Pseudocode is shown in Alg. 1.
Revised: Pseudocode is shown in Alg. 1.

Original: Each node performs a local SGD step (lines 2–4) followed by one step of P USH S UM for approximate distributed averaging (lines 5–8).
Revised: Each node performs a local SGD step (lines 2–4) followed by one step of P USH S UM for approximate distributed averaging (lines 5–8).

Original: Note that the gradients are evaluated at the de-biased parameters z ( k ) i in line 3, and they are then used to update x ( k ) i , the P USH S UM numerator, in line 4.
Revised: Note that the gradients are evaluated at the de-biased parameters z ( k ) i in line 3, and they are then used to update x ( k ) i , the P USH S UM numerator, in line 4.

Original: All communication takes place in line 5, and each message contains two parts, the P USH S UM numerator and denominator.
Revised: All communication takes place in line 5, and each message contains two parts, the P USH S UM numerator and denominator.

Original: In particular, node i controls the values p ( k ) j,i used to weight the values in messages it sends.Algorithm
Revised: In particular, node i controls the values p ( k ) j,i used to weight the values in messages it sends.

Original: Require:
Revised: Require:

Original: Initialize γ > 0 , x (0)
Revised: Initialize γ > 0 , x (0)

Original: i =
Revised: i =

Original: z (0) i ∈ R d and w (0) i
Revised: z (0) i ∈ R d and w (0) i

Original: = 1 for all nodes i ∈ { 1 , 2 , . . .
Revised: = 1 for all nodes i ∈ { 1 , 2 , . . .

Original: , n } 1: for k = 0 , 1 , 2 , · · · , K do at node i 2:
Revised: , n } 1: for k = 0 , 1 , 2 , · · · , K do at node i 2:

Original: Sample new mini-batch ξ ( k ) i ∼ D i from local distribution 3:
Revised: Sample new mini-batch ξ ( k ) i ∼ D i from local distribution 3:

Original: Compute a local stochastic mini-batch gradient at z ( k ) i : ∇ F i ( z ( k ) i ; ξ ( k ) i ) 4: x ( k + 12 ) i =
Revised: Compute a local stochastic mini-batch gradient at z ( k ) i : ∇ F i ( z ( k ) i ; ξ ( k ) i ) 4: x ( k + 12 ) i =

Original: x ( k ) i − γ ∇ F i ( z ( k ) i ; ξ ( k ) i ) 5:
Revised: x ( k ) i − γ ∇ F i ( z ( k ) i ; ξ ( k ) i ) 5:

Original: Send (cid:0) p ( k ) j,i x ( k + 12 ) i , p ( k ) j,i w ( k ) i (cid:1) to out-neighbors j ∈ N out
Revised: Send (cid:0) p ( k ) j,i x ( k + 12 ) i , p ( k ) j,i w ( k ) i (cid:1) to out-neighbors j ∈ N out

Original: i ( k ) ; receive (cid:0) p ( k ) i,j x ( k + 12 ) j , p ( k ) i,j w ( k ) j (cid:1) from in-neighbors j ∈ N in i ( k ) 6: x ( k +1)
Revised: i ( k ) ; receive (cid:0) p ( k ) i,j x ( k + 12 ) j , p ( k ) i,j w ( k ) j (cid:1) from in-neighbors j ∈ N in i ( k ) 6: x ( k +1)

Original: i =
Revised: i =

Original: (cid:80) j ∈N in i ( k ) p ( k ) i,j x ( k + 12 ) j 7: w ( k +1)
Revised: (cid:80) j ∈N in i ( k ) p ( k ) i,j x ( k + 12 ) j 7: w ( k +1)

Original: i =
Revised: i =

Original: (cid:80) j ∈N in i ( k ) p ( k ) i,j w ( k ) j 8: z
Revised: (cid:80) j ∈N in i ( k ) p ( k ) i,j w ( k ) j 8: z

Original: ( k +1)
Revised: ( k +1)

Original: i =
Revised: i =

Original: x ( k +1)
Revised: x ( k +1)

Original: i /w ( k +1)
Revised: i /w ( k +1)

Original: i 9: end for
Revised: i 9: end for

Original: We are mainly interested in the case where the mixing matrices P ( k ) are sparse in order to have low communication overhead.
Revised: We are mainly interested in the case where the mixing matrices P ( k ) are sparse in order to have low communication overhead.

Original: However, we point out that when the nodes’ initial values are identical, x (0) i
Revised: However, we point out that when the nodes’ initial values are identical, x (0) i

Original: = x (0) j for all i, j ∈ [ n ] , and every entry of P ( k ) is equal to 1 /n , then SGP is mathematically equivalent to parallel SGD using A LL R EDUCE .
Revised: = x (0) j for all i, j ∈ [ n ] , and every entry of P ( k ) is equal to 1 /n , then SGP is mathematically equivalent to parallel SGD using A LL R EDUCE .

Revised: Please refer to appendix A for pratical implementation details, including how we design mixing matrices P ( k ) .

Original: Theoretical guarantees.
Revised: Theoretical guarantees.

Original: SGP was ﬁrst proposed and analyzed in (Nedi ´ c & Olshevsky, 2016) assuming
Revised: SGP was ﬁrst proposed and analyzed in (Nedi´c & Olshevsky, 2016) assuming

Original: the local objectives f i ( x ) are strongly convex.
Revised: the local objectives f i ( x ) are strongly convex.

Original: Here we provide convergence results in the more general setting of smooth, non-convex objectives.
Revised: Here we provide convergence results in the more general setting of smooth, non-convex objectives.

Original: We make the following three assumptions:
Revised: We make the following three assumptions:

Original: ( L -smooth)
Revised: ( L -smooth)

Original: There exists a constant L > 0 such that (cid:107)∇ f i ( x ) − ∇ f i ( y ) (cid:107) ≤ L (cid:107) x − y (cid:107) , or equivalently
Revised: There exists a constant L > 0 such that (cid:107)∇ f i ( x ) − [Equation] f i ( y ) (cid:107) ≤ L (cid:107) x − y (cid:107) , or equivalently

Original: [Equation]
Revised: [Equation]

Original: Note that this assumption implies that function f ( x ) is also L-smooth.
Revised: Note that this assumption implies that function f ( x ) is also L-smooth.

Original: (Bounded variance)
Revised: (Bounded variance)

Original: There exist ﬁnite positive constants σ 2 and ζ  such that
Revised: There exist ﬁnite positive constants σ 2 and ζ 2 such that

Original: [Equation]
Revised: [Equation]

Original: Thus σ 2 bounds the variance of stochastic gradients at each node, and ζ 2 quantiﬁes the similarity of data distributions at different nodes.
Revised: Thus σ 2 bounds the variance of stochastic gradients at each node, and ζ 2 quantiﬁes the similarity of data distributions at different nodes.

Original: (Mixing connectivity)
Revised: (Mixing connectivity)

Original: To each mixing matrix P ( k ) we can associate a graph with vertex set { 1 , . . . , n } and edge set E ( k ) 
Revised: To each mixing matrix P ( k ) we can associate a graph with vertex set { 1 , . . . , n } and edge set E ( k ) =

Original: = { ( i, j ): p ( k ) i,j > 0 } ; i.e. , with edges ( i, j ) from j to i if i receives a message from j at iteration k .
Revised: { ( i, j ): p ( k ) i,j > 0 } ; i.e. , with edges ( i, j ) from j to i if i receives a message from j at iteration k .

Original: Assume that the graph with edge set (cid:83) ( l +1) B − 1 k = lB
Revised: Assume that the graph with edge set (cid:83) ( l +1) B − 1 k = lB

Original: E ( k ) is strongly connected and has diameter at most ∆ for every l ≥ 0 .
Revised: E ( k ) is strongly connected and has diameter at most ∆ for every l ≥ 0 .

Original: To simplify the discussion, we assume that every column of the mixing matrices P ( k ) has at most D non-zero entries.
Revised: To simplify the discussion, we assume that every column of the mixing matrices P ( k ) has at most D non-zero entries.

Original: Let x ( k ) =
Revised: Let x ( k ) =

Original: 1 n (cid:80) ni
Revised: 1 n (cid:80) ni

Original: =1 x ( k ) i .
Revised: =1 x ( k ) i .

Original: Under similar assumptions, Lian et al.
Revised: Under similar assumptions, Lian et al.

Original: (2017) deﬁne that a decentralized algorithm for solving equation 1 converges if, for any (cid:15) > 0 , it eventually satisﬁes
Revised: (2017) deﬁne that a decentralized algorithm for solving equation 1 converges if, for any (cid:15) > 0 , it eventually satisﬁes

Original: [Equation]
Revised: [Equation]

Original: Our ﬁrst result shows that SGP converges in this sense.
Revised: Our ﬁrst result shows that SGP converges in this sense.

Original: Theorem 1.
Revised: Theorem 1.

Original: Suppose that Assumptions 1–3 hold, and run SGP for K iterations with step-size γ = (cid:112) n/K .
Revised: Suppose that Assumptions 1–3 hold, and run SGP for K iterations with step-size γ = (cid:112) n/K .

Original: Let f ∗ =
Revised: Let f ∗ =

Original: min x f ( x ) and assume that f ∗ > −∞ .
Revised: min x f ( x ) and assume that f ∗ > −∞ .

Original: There exist constants C > 0 and q ∈ (0 , 1) which depend on B , n , and ∆ such that if the total number of iterations satisﬁesK ≥
Revised: There exist constants C > 0 and q ∈ (0 , 1) which depend on B , n , and ∆ such that if the total number of iterations satisﬁes

Original: [Equation]
Revised: [Equation]

Original:  where P 1 = 4( σ 2 + 3 ζ 2 ) n + 
Revised: ) where P 1 = 4( σ 2 + 3 ζ 2 ) n + (cid:80) [Equation] =1 (cid:107) x i

Original: [Equation]
Revised: [Equation]

Original:  and
Revised: 2 n and

Original: [Equation]
Revised: [Equation]

Original: , then
Revised: , then

Original: [Equation]
Revised: [Equation]

Original: nK .
Revised: nK .

Original: The proof is given in Appendix C, where we also provide precise expressions for the constants C and q .
Revised: The proof is given in Appendix C, where we also provide precise expressions for the constants C and q .

Original: The proof of Theorem 1 builds on an approach developed in Lian et al.
Revised: The proof of Theorem 1 builds on an approach developed in Lian et al.

Original: Theorem 1 shows that, for a given number of nodes n , by running a sufﬁciently large number of iterations K (roughly speaking, Ω( n ) , which is reasonable for distributed training of DNNs) and choosing the step-size γ as prescribed, then the criterion equation 5 is satisﬁed with a number of iterations K = Ω(1 /n(cid:15) 2 ) .
Revised: Theorem 1 shows that, for a given number of nodes n , by running a sufﬁciently large number of iterations K (roughly speaking, Ω( n ) , which is reasonable for distributed training of DNNs) and choosing the step-size γ as prescribed, then the criterion equation 5 is satisﬁed with a number of iterations K = Ω(1 /n(cid:15) 2 ) .

Original: That is, we achieve a linear speedup in the number of nodes.
Revised: That is, we achieve a linear speedup in the number of nodes.

Original: Theorem 1 shows that the average of the nodes parameters, x ( k ) , converges, but it doesn’t directly say anything about the parameters at each node.
Revised: Theorem 1 shows that the average of the nodes parameters, x ( k ) , converges, but it doesn’t directly say anything about the parameters at each node.

Original: In fact, we can show a stronger result.
Revised: In fact, we can show a stronger result.

Original: Theorem 2.
Revised: Theorem 2.

Original: Under the same assumptions as in Theorem 1,
Revised: Under the same assumptions as in Theorem 1,

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Revised: [Equation]

Revised: k

Original: [Equation]
Revised: [Equation]

Original: The proof is also given in Appendix C.
Revised: The proof is also given in Appendix C.

Original: This result shows that as K grows, the de-biased variables z ( k ) i converge to the node-wise average x ( k ) , and hence the de-biased variables at each node also converge to a stationary point.
Revised: This result shows that as K grows, the de-biased variables z ( k ) i converge to the node-wise average x ( k ) , and hence the de-biased variables at each node also converge to a stationary point.

Original: Note that for ﬁxed n and large K , the 1 / √ nK term will dominate the other factors.
Revised: Note that for ﬁxed n and large K , the 1 / √ nK term will dominate the other factors.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: A variety of approaches have been proposed to accelerate distributed training of DNNs, including quantizing gradients (Alistarh et al., 2007; Wen et al., 2007) and performing multiple local SGD steps at each node before averaging (McMahan et al., 2017).
Revised: A variety of approaches have been proposed to accelerate distributed training of DNNs, including quantizing gradients (Alistarh et al., 2007; Wen et al., 2007) and performing multiple local SGD steps at each node before averaging (McMahan et al., 2017).

Original: These approaches are complementary to the tradeoff we consider in this paper, between exact and approximate distributed averaging.
Revised: These approaches are complementary to the tradeoff we consider in this paper, between exact and approximate distributed averaging.

Original: Similar to using P USH S UM for averaging,  quantizing gradients and performing multiple local SGD steps before averagingboth can also be seen as injecting additional noise into SGD, leading to a trade off between training faster (by reducing communication overhead) and potentially obtaining a less accurate result.
Revised: Similar to using P USH S UM for averaging, both quantizing gradients and performing multiple local SGD steps before averaging can also be seen as injecting additional noise into SGD, leading to a trade off between training faster (by reducing communication overhead) and potentially obtaining a less accurate result.

Original: Combining these approaches (quantized, inexact, and infrequent averaging) is an interesting direction for future work.
Revised: Combining these approaches (quantized, inexact, and infrequent averaging) is an interesting direction for future work.

Original: For the remainder of this section we review related work applying consensus-based approaches to large-scale training of DNNs.
Revised: For the remainder of this section we review related work applying consensus-based approaches to large-scale training of DNNs.

Original: Blot et al.
Revised: Blot et al.

Original: (2016) report initial experimental results on small-scale experiments with an SGP-like algorithm.
Revised: (2016) report initial experimental results on small-scale experiments with an SGP-like algorithm.

Original: Jin et al.
Revised: Jin et al.

Original: (2016) make a theoretical connection between
Revised: (2016) make a theoretical connection between

Original: P USH S UM -based methods and Elastic Averaged SGD (Zhang et al., 2015).
Revised: P USH S UM -based methods and Elastic Averaging SGD (Zhang et al., 2015).

Original: Relatively to those previous works, we provide the ﬁrst convergence analysis for a P USH S UM -based method in the smooth non-convex case.
Revised: Relative to those previous works, we provide the ﬁrst convergence analysis for a P USH S UM -based method in the smooth non-convex case.

Original: Lian et al.
Revised: Lian et al.

Original: (2017) and Jiang et al.
Revised: (2017) and Jiang et al.

Original: (2017) study synchronous consensus-based versions of SGD.
Revised: (2017) study synchronous consensus-based versions of SGD.

Original: However, unlike P USH S UM , those methods involve symmetric message passing (if i sends to j at iteration k , then j also sends to i before both nodes update) which is inherently blocking.
Revised: However, unlike P USH S UM , those methods involve symmetric message passing (if i sends to j at iteration k , then j also sends to i before both nodes update) which is inherently blocking.

Original: Consequently, these methods are more sensitive to high-latency communication settings, and each node generally must communicate more per iteration, in comparison to P USH S UM -based SGP where communication may be directed ( i can send to j without needing a response from i ).
Revised: Consequently, these methods are more sensitive to high-latency communication settings, and each node generally must communicate more per iteration, in comparison to P USH S UM -based SGP where communication may be directed ( i can send to j without needing a response from i ).

Original: The decentralized parallel SGD (D-PSGD) method proposed in Lian et al.
Revised: The decentralized parallel SGD (D-PSGD) method proposed in Lian et al.

Original: (2017) produces iterates whose node-wise average, x ( k ) , is shown to converge in the sense of equation 5.
Revised: (2017) produces iterates whose node-wise average, x ( k ) , is shown to converge in the sense of equation 5.

Original: Our proof of Theorem 1, showing the convergence of SGP in the same sense, adapts some ideas from their analysis and also goes beyond to show that, since the values at each node converge to the average, the individual values at each node also converge to a stationary point.
Revised: Our proof of Theorem 1, showing the convergence of SGP in the same sense, adapts some ideas from their analysis and also goes beyond to show that, since the values at each node converge to the average, the individual values at each node also converge to a stationary point.

Original: We compare SGP with D-PSGD experimentally in Section 5 below and ﬁnd that although the two methods ﬁnd solutions of comparable accuracy,
Revised: We compare SGP with D-PSGD experimentally in Section 5 below and ﬁnd that although the two methods ﬁnd solutions of comparable accuracy,

Original: SGP is consistently faster in both high-latency and low-latency scenarios.
Revised: SGP is consistently faster.

Original: Jin et al.
Revised: Jin et al.

Original: and Lian et al.
Revised: and Lian et al.

Original: (2018) study asynchronous consensus-based methods for training DNNs.
Revised: (2018) study asynchronous consensus-based methods for training DNNs.

Original: Lian et al.
Revised: Lian et al.

Original: (2018) analyzes an asynchronous version of D-PSGD and proves that its nodewise averages also converge to a stationary point.
Revised: (2018) analyzes an asynchronous version of D-PSGD and proves that its node-wise averages also converge to a stationary point.

Original: Asynchrony introduces an additional source of noise, and in general,the models returned by these asynchronous methods generalize worse than their synchronous counterparts.
Revised: In general, these contributions focusing on asynchrony can be seen as orthogonal to the use of a P USH S UM based protocol for consensus averaging.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: Next, we compare SGP with A LL R EDUCE SGD, and D-PSGD (Lian et al., 2017), an approximate distributed averaging baseline relying on doubly-stochastic gossip.
Revised: Next, we compare SGP with A LL R EDUCE SGD, and D-PSGD (Lian et al., 2017), an approximate distributed averaging baseline relying on doubly-stochastic gossip.

Original: We run experiments on a large-scale distributed computing environment using up to 256 GPUs.
Revised: We run experiments on a large-scale distributed computing environment using up to 256 GPUs.

Original: Our results show that when communication is the bottleneck, SGP is faster than both SGD and D-PSGD.
Revised: Our results show that when communication is the bottleneck, SGP is faster than both SGD and D-PSGD.

Original: SGP also outperforms
Revised: SGP also outperforms

Original: D-PSGD in terms of validation accuracy, while achieving a slightly worse accuracy compared to
Revised: D-PSGD in terms of validation accuracy, while achieving a slightly worse accuracy compared to

Original: SGD when using a large number of compute nodes.
Revised: SGD when using a large number of compute nodes.

Original: Our results also highlight that, in a setting where communication is efﬁcient ( e.g. , over InﬁniBand), doing exact averaging through A LL R EDUCE SGD remains a competitive approach.
Revised: Our results also highlight that, in a setting where communication is efﬁcient ( e.g. , over InﬁniBand), doing exact averaging through A LL R EDUCE SGD remains a competitive approach.

Original: We run experiments on 32 DGX-

Revised: We run experiments on 32 DGX-

Original: GPU servers in a high-performance computing cluster.
Revised: GPU servers in a high-performance computing cluster.

Original: Each server contains eight NVIDIA Volta V100 GPUs.
Revised: Each server contains 8 NVIDIA Volta-V100 GPUs.

Original: We consider two communication scenarios: in the high-latency scenario the nodes communicate over a 10 Gbit/s Ethernet network, and in the low-latency scenario the nodes communicate over 100 Gbit/s InﬁniBand which supports GPUDirect
Revised: We consider two communication scenarios: in the high-latency scenario the nodes communicate over a 10 Gbit/s Ethernet network, and in the lowlatency scenario the nodes communicate over 100 Gbit/s InﬁniBand, which supports GPUDirect

Original: RDMA communications.
Revised: RDMA communications.

Original: To investigate how each algorithm scales, we run experiments with 4, 8, 16, and 32 nodes ( i.e. , 32, 64, 128, and 256 GPUs).
Revised: To investigate how each algorithm scales, we run experiments with 4, 8, 16, and 32 nodes ( i.e. , 32, 64, 128, and 256 GPUs).

Original: We adopt the 1000-way ImageNet classiﬁcation task (Russakovsky et al., 2015) as our experimental benchmark.
Revised: We adopt the 1000-way ImageNet classiﬁcation task (Russakovsky et al., 2015) as our experimental benchmark.

Original: We train a ResNet-

Revised: We train a ResNet-

Original: (He et al., 2016) following the experimental protocol of Goyal et al.
Revised: (He et al., 2016) following the experimental protocol of Goyal et al.

Original: (2017), using the same hyperparameters with the exception of the learning rate schedule in the 32 node experiment for SGP and D-PSGD.
Revised: (2017), using the same hyperparameters with the exception of the learning rate schedule in the 32 node experiment for SGP and D-PSGD.

Original: In the experiments, we also modify SGP to use Nesterov momentum.
Revised: In the experiments, we also modify SGP to use Nesterov momentum.

Original: In our default implementation of SGP, each node sends and receives to one other node at each iteration, and this destination changes from one iteration to the next.
Revised: In our default implementation of SGP, each node sends and receives to one other node at each iteration, and this destination changes from one iteration to the next.

Original: Please refer to A.3 for more implementation details.
Revised: Please refer to appendix A for more information about our implementation, including how we design/implement the sequence of mixing matrices P ( k ) .

Original: All algorithms are implemented in PyTorch v0.5 (Paszke et al.).
Revised: All algorithms are implemented in PyTorch v0.5 (Paszke et al.).

Original: To leverage the highly efﬁcient
Revised: To leverage the highly efﬁcient

Original: NVLink interconnect within each server, we treat each DGX-1 as one node in all of our experiments.
Revised: NVLink interconnect within each server, we treat each DGX-1 as one node in all of our experiments.

Original: In our implementation of SGP, each node computes a local mini-batch in parallel using all eight GPUs using a local A LL R EDUCE , which is efﬁciently implemented via the NVIDIA Collective Communications Library.
Revised: In our implementation of SGP, each node computes a local mini-batch in parallel using all eight GPUs using a local A LL R EDUCE , which is efﬁciently implemented via the NVIDIA Collective Communications Library.

Original: Then inter-node averaging is accomplished using P USH S UM either over Ethernet or InﬁniBand.
Revised: Then inter-node averaging is accomplished using P USH S UM either over Ethernet or InﬁniBand.

Original: In the low-latency experiments, we leverage GPUDirect to directly send/receive messages between GPUs on different nodes and avoid transferring the model back to host memory.
Revised: In the low-latency experiments, we leverage GPUDirect to directly send/receive messages between GPUs on different nodes and avoid transferring the model back to host memory.

Original: In the high-latency experiments this is not possible, so the model is transferred to host memory after the local A LL R EDUCE , and then P USH S UM messages are sent over Ethernet.
Revised: In the high-latency experiments this is not possible, so the model is transferred to host memory after the local A LL R EDUCE , and then P USH S UM messages are sent over Ethernet.

Original: 5. E VALUATION ON H IGH -L ATENCY I NTERCONNECT
Revised: 5. E VALUATION ON H IGH -L ATENCY I NTERCONNECT

Original: We consider the high-latency scenario where nodes communicate over 10Gbit/s Ethernet.
Revised: We consider the high-latency scenario where nodes communicate over 10Gbit/s Ethernet.

Original: With a local mini-batch size of 256 samples per node (32 samples per GPU), a single Volta DGX-1 server can perform roughly 4 .
Revised: With a local mini-batch size of 256 samples per node (32 samples per GPU), a single Volta DGX-1 server can perform roughly 4 .

Original: 384 mini
Revised: 384 mini

Original: -batches per second.
Revised: -batches per second.

Original: Since the ResNet-50 model size is roughly 100MBytes, transmitting one copy of the model per iteration requires 3 . 5 Gbit/s.
Revised: Since the ResNet-50 model size is roughly 100MBytes, transmitting one copy of the model per iteration requires 3 . 5 Gbit/s.

Original: Thus in the highlatency scenario the problem, if a single 10 Gbit/s link must carry the trafﬁc between more than two pairs of nodes, then communication clearly becomes a bottleneck. 
Revised: Thus in the highlatency scenario the problem, if a single 10 Gbit/s link must carry the trafﬁc between more than two pairs of nodes, then communication clearly becomes a bottleneck. [Figure] Comparison with synchronous approaches.

Original:  Figure 1 (a) shows the validation curves when training on 4 and 32 nodes (additional training and validation curves for all the training runs can be found in B.1).For any number of nodes used in
Revised: We ﬁrst compare SGP with other synchronous and decentralized approaches. Figure 1 (a) shows the validation curves when training on 4 and 32 nodes (additional training and validation curves for all the training runs can be found in B.1).

Original: [Figure]
Revised: Note that when we increase the number of nodes n , we also decrease the total number of iterations K to K/n following Theorem 1 (see Figure B.3).

Original:  our experiments, we observe that SGP consistently outperforms D-PSGD and A LL R EDUCE SGD in terms of total training time in this scenario.
Revised: For any number of nodes used in our experiments, we observe that SGP consistently outperforms D-PSGD and A LL R EDUCE SGD in terms of total training time in this scenario.

Original: In particular for 32 nodes, SGP training time takes less than 1 .
Revised: In particular for 32 nodes, SGP training time takes less than 1 .

Original: 6 hours while D-PSGD and A LL R EDUCE SGD require roughly 2 . 6 and 5 . 1 hours.
Revised: 6 hours while D-PSGD and A LL R EDUCE SGD require roughly 2 . 6 and 5 . 1 hours.

Original: Appendix B.2 provides experimental evidence that all nodes converge to models with  similar training and validation accuracy when using SGP.
Revised: Appendix B.2 provides experimental evidence that all nodes converge to models with a similar training and validation accuracy when using SGP.

Original: Figure 1 (b) shows the average time per iteration for the different training runs.
Revised: Figure 1 (b) shows the average time per iteration for the different training runs.

Original: As we increase the number of nodes, the average iteration time stays almost constant for SGP and D-PSGD, while we observe a signiﬁcant time-increase in the case of A LL R EDUCE SGD, resulting in an overall slower training time.
Revised: As we increase the number of nodes, the average iteration time stays almost constant for SGP and D-PSGD, while we observe a signiﬁcant time-increase in the case of A LL R EDUCE SGD, resulting in an overall slower training time.

Original: Moreover, although D-SGD and SGP both exhibit strong scaling, SGP is roughly 200ms faster per iteration, supporting the claim that it involves less communication overhead.
Revised: Moreover, although D-PSGD and SGP both exhibit strong scaling, SGP is roughly 200ms faster per iteration, supporting the claim that it involves less communication overhead.

Original: Figure 1 (c) reports the best validation accuracy for the different training runs.
Revised: Figure 1 (c) reports the best validation accuracy for the different training runs.

Original: While they all start around the same value, the accuracy of D-PSGD and SGP decreases as we increase the number of nodes.
Revised: While they all start around the same value, the accuracy of D-PSGD and SGP decreases as we increase the number of nodes.

Original: In the case of SGP, we see its performance decrease by 1 . 2% relative to SGD on 32 nodes.
Revised: In the case of SGP, we see its performance decrease by 1 . 2% relative to SGD on 32 nodes.

Original: We hypothesize that this decrease is due to the noise introduced by approximate distributed averaging.
Revised: We hypothesize that this decrease is due to the noise introduced by approximate distributed averaging.

Original: We will see below than changing the connectivity between the nodes can ameliorate this issue.
Revised: We will see below than changing the connectivity between the nodes can ameliorate this issue.

Original: We also note that the SGP validation accuracy is better than D-PSGD for larger networks.
Revised: We also note that the SGP validation accuracy is better than D-PSGD for larger networks.

Original: 5. E VALUATION ON A “L OW L ATENCY ” I NTERCONNECT
Revised: Comparison with asynchronous approach.

Original: [Figure]
Revised: The results in Tables 1 and 2 provide a comparison between the aforementioned synchronous methods and AD-PSGD (Lian et al., 2018), a state-ofart asynchronous method. AD-PSGD is an asynchronous implementation of the doubly-stochastic method D-PSGD, which relies on doubly-stochastic averaging. All methods are trained for exactly 90 epochs, therefore, the time-per-iteration is a direct reﬂection of the total training time. Training using AD-PSGD does not degrade the accuracy (relative to D-PSGD), and provides substantial speedups in training time.

Original: Validation
Revised: Relative to SGP, the AD-PSGD method runs slightly faster at the expense of lower validation accuracy (except in the 32 nodes case).

Original: [Figure]
Revised: In general, we emphasize that this asynchronous line of work is orthogonal, and that by combining the two approaches (leveraging the P USH S UM protocol in an asynchronous manner), one can expect to further speed up SGP. We leave this as a promising line of investigation for future work. 5. E VALUATION ON A “L OW L ATENCY ” I NTERCONNECT

Original: We now investigate the behavior of SGP and A LL R EDUCE SGD over InﬁniBand 100Gbit/s. We follow the same experimental protocol as in the Ethernet 10Gbit/s case. 
Revised: We now investigate the behavior of SGP and A LL R EDUCE SGD over InﬁniBand 100Gbit/s, following the same experimental protocol as in the Ethernet 10Gbit/s case. In this scenario which is not

Original: On this low-latency interconnect, SGD and SGP obtain similar timing and differ at most by 21 ms per iteration (Figure 2 (b)
Revised: [Table]

Original: [Figure]
Revised: [Table]

Original: for 4 nodes). This conﬁrms that for model sizes on the scale of ResNet50, communication is not a bottleneck in this setting.
Revised: communication bound for a Resnet-50 model, we do not expect SGP to outperform A LL R EDUCE

Original:  SGP does outperform A LL R EDUCE 
Revised: SGD. Our goal is to illustrate that SGP is not signiﬁcantly slower than A LL R EDUCE SGD. On this low-latency interconnect

Original:  SGD when using 32 nodes by a small 13 ms margin (Figure 2 (a) and (b)).
Revised: [Table] SGD and SGP obtain similar timing and differ at most by 21 ms per iteration (Figure 2 (b) for 4 nodes).

Original: Overall SGP trains a ResNet-50 on ImageNet in 1 .
Revised: In particular, using 32 nodes, SGP trains a ResNet-50 on ImageNet in 1 .

Original: 16 hour and SGD in 1 . 20 hour.
Revised: 16 hours and SGD in 1 . 20 hours.

Original: Time differences of this scale could be impacted by other factors such as data loading.
Revised: SGD, however, exhibits better validation accuracy for large networks. Communication on InﬁniBand is not a bottleneck for models the size of ResNet50. These results therefore conﬁrm that SGP beneﬁts are more prominent in high-latency/lowbandwidth communication-bound scenarios. Although timing are similar, SGP still shows better scaling in term of sample throughput than A LL R EDUCE

Original: We show the computation and communication time during training in Appendix B.3.
Revised: SGD (see Figure B.6) For experiments running at this speed (less than 0.31 seconds per iteration), timing could be impacted by other factors such as data loading. To better isolate the effects of data-loading, we run additional experiments on 32, 64, and 128 GPUs where we ﬁrst copied the data locally on every node; see Appendix B.3 for more details.

Original: We observe that  SGP computation and communication time is close to constant as  the number of nodes increases, while SGD increases slightly. Validation accuracy follows a similar trend as for
Revised: In that setting, the time-per-iteration of SGP remains approximately constant as we increase the number of nodes in the network, while the time for

Original: Ethernet 10Gbit/s (Figure 2 (c)).
Revised: AllReduceSGD increases with more nodes.

Original: 5. I MPACT OF G RAPH T OPOLOGY
Revised: 5. I MPACT OF G RAPH T OPOLOGY

Original: Next we investigate the impact of the communication graph topology on the SGP validation performance using Ethernet 10Gbit/s. In the limit of a fully-connected communication graph, SGD and
Revised: Next we investigate the impact of the communication graph topology on the SGP validation performance using Ethernet 10Gbit/s. In the limit of a fully-connected communication graph, SGD and

Original: SGP are strictly equivalent (see section 3).
Revised: SGP are strictly equivalent (see section 3).

Original: By increasing the number of neighbors in the graph, we expect the accuracy of SGP to improve (approximate averages are more accurate) but the communication time required for training will increase.
Revised: By increasing the number of neighbors in the graph, we expect the accuracy of SGP to improve (approximate averages are more accurate) but the communication time required for training will increase.

Original: In ﬁgure 3, we compare the training and validation accuracy for SGP using a communication graph with 1-neighbor and 2-neighbors with D-PSGD and SGD on 32 nodes.
Revised: In Figure 3, we compare the training and validation accuracies of SGP using a communication graph with 1-neighbor and 2-neighbors with D-PSGD and SGD on 32 nodes.

Original: By increasing the number of neighbors to two, SGP achieves better training/validation accuracy (from 74 .
Revised: By increasing the number of neighbors to two, SGP achieves better training/validation accuracy (from 74 .

Original: to 75 .
Revised: to 75 .

Original: and gets closer to ﬁnal validation achieves by SGD ( 77 . 0 / 76 . 2 ).
Revised: and gets closer to ﬁnal validation achieves by SGD ( 77 . 0 / 76 . 2 ).

Original: Increasing the number of neighbors alsoto increases the communication, hence the overall training time.
Revised: Increasing the number of neighbors also increases the communication, hence the overall training time.

Original: SGP with 2 neighbors completes training in 2 .
Revised: SGP with 2 neighbors completes training in 2 .

Original: 1 hours and its average time per iteration increases by 27% relative to SGP with one neighbor.
Revised: 1 hours and its average time per iteration increases by 27% relative to SGP with one neighbor.

Original: Nevertheless, SGP 2-neighbors is still faster than SGD and D-PSGD, while achieving better accuracy than SGP 1-neighbor. 
Revised: Nevertheless, SGP 2-neighbors is still faster than SGD and D-PSGD, while achieving better accuracy than SGP 1-neighbor. [Figure] [Figure]

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: DNN training often necessistates non-trivial computational requirements leveaging distributed computing resources.
Revised: DNN training often necessistates non-trivial computational requirements leveaging distributed computing resources.

Original: Traditional parallel versions of SGD use exact averaging algorithms to parallelize the computation between nodes, and induce additional parallelization overhead as the model and network sizes grow.
Revised: Traditional parallel versions of SGD use exact averaging algorithms to parallelize the computation between nodes, and induce additional parallelization overhead as the model and network sizes grow.

Original: This paper proposes the use of Stochastic Gradient Push for distributed deep learning.
Revised: This paper proposes the use of Stochastic Gradient Push for distributed deep learning.

Original: The proposed method computes in-exact averages at each iteartion in order to improve scaling efﬁciency and reduce the dependency on the underlying network topology.
Revised: The proposed method computes in-exact averages at each iteartion in order to improve scaling efﬁciency and reduce the dependency on the underlying network topology.

Original: SGP converges to a stationary point at an O (cid:16) 1 / √ nK (cid:17) rate in the smooth and non-convex case, and proveably achieves a linear speedup (in iterations) with respect to the number of nodes.
Revised: SGP converges to a stationary point at an O (cid:16) 1 / √ nK (cid:17) rate in the smooth and non-convex case, and proveably achieves a linear speedup (in iterations) with respect to the number of nodes.

Original: Empirical results show that SGP can be up to 3 × times faster than traditional A LL R EDUCE SGD over high-latency interconnect, matches the top-1 validation accuray up to 8 nodes ( 64 GPUs), and remains within 1 . 2%  top-1 validation accuracy for larger-networks.
Revised: Empirical results show that SGP can be up to 3 × times faster than traditional A LL R EDUCE SGD over high-latency interconnect, matches the top-1 validation accuracy up to 8 nodes ( 64 GPUs), and remains within 1 . 2% of the top-1 validation accuracy for larger-networks

Original: A I MPLEMENTATION D ETAILS
Revised: A I MPLEMENTATION D ETAILS

Original: [Figure]
Revised: [Figure]

Original: A. C OMMUNICATION T OPOLOGY
Revised: A. C OMMUNICATION T OPOLOGY

Original: Directed exponential graph.
Revised: Directed exponential graph.

Original: For the SGP experiments we use a time-varying directed graph to represent the inter-node connectivity.
Revised: For the SGP experiments we use a time-varying directed graph to represent the inter-node connectivity.

Original: Thinking of the nodes as being ordered sequentially, according to their rank, 0 , . .
Revised: Thinking of the nodes as being ordered sequentially, according to their rank, 0 , . .

Original: n − 1 , 1 each node periodically communicates with peers that are 2 0 , 2 1 , . . .
Revised: n − 1 , 1 each node periodically communicates with peers that are 2 0 , 2 1 , . . .

Original: , 2  log 2 ( n − 1)  hops away.
Revised: , 2 (cid:98) log 2 ( n − 1) (cid:99) hops away.

Original: Fig.
Revised: Fig.

Original: 4 shows an example of a directed 8-node exponential graph.
Revised: A.1 shows an example of a directed 8-node exponential graph.

Original: Node 0 ’s 2 0 -hop neighbour is node 1 , node 0 ’s 2 1 -hop neighbour is node 2 , and node 0 ’s 2 2 -hop neighbour is node 4 .
Revised: Node 0 ’s 2 0 -hop neighbour is node 1 , node 0 ’s 2 1 -hop neighbour is node 2 , and node 0 ’s 2 2 -hop neighbour is node 4 .

Original: In the one-peer-per-node experiments, each node cycles through these peers, transmitting, only, to a single peer from this list at each iteration.
Revised: In the one-peer-per-node experiments, each node cycles through these peers, transmitting, only, to a single peer from this list at each iteration.

Original: E.g., at iteration k , all nodes transmit messages to their 2 0 -hop neighbours, at iteration k + 1 all nodes transmit messages to their 2 

Revised: E.g., at iteration k , all nodes transmit messages to their 2 0 -hop neighbours, at iteration k + 1 all nodes transmit messages to their 2

Original: -hop neighbours, an so on, eventually returning to the beginning of the list before cycling through the peers again.
Revised: -hop neighbours, an so on, eventually returning to the beginning of the list before cycling through the peers again.

Original: This  ensures that each nodealso only  receives a single message at each iteration.
Revised: This procedure ensures that each node only sends and receives a single message at each iteration.

Original: By using full-duplex communication, sending and receiving can happen in parallel.
Revised: By using full-duplex communication, sending and receiving can happen in parallel.

Original: In the two-peer-per-node experiments, each node cycles through the same set of peers, transmitting to two peers from the list at each iteration.
Revised: In the two-peer-per-node experiments, each node cycles through the same set of peers, transmitting to two peers from the list at each iteration.

Original: E.g., at iteration k , all nodes transmit messages to their 2 0 -hop and 2 1 -hop neighbours, at iteration k + 1 all nodes transmit messages to their 2 1hop and 2 2 neighbours, an so on, eventually returning to the beginning of the list before cycling through the peers again.
Revised: E.g., at iteration k , all nodes transmit messages to their 2 0 -hop and 2 1 -hop neighbours, at iteration k + 1 all nodes transmit messages to their 2 1hop and 2 2 neighbours, an so on, eventually returning to the beginning of the list before cycling through the peers again.

Original: Similarly, at each iteration, each node also receives, in a full-duplex manner, two messages from some peers that are unknown to the receiving node ahead of time.
Revised: Similarly, at each iteration, each node also receives, in a full-duplex manner, two messages from some peers that are unknown to the receiving node ahead of time.

Original: Thereby performing the send and receive operations in parallel.
Revised: Thereby performing the send and receive operations in parallel.

Revised: Deﬁnition of P ( k ) .

Revised: Based on the description above, in the one-peer-per-node experiments, each node sends to one neighbor at every iteration, and so each column of P ( k ) has exactly two nonzero entries, both of which are equal to 1 / 2 .

Revised: The diagonal entries p ( k ) i,i = 1 / 2 for all i and k .

Revised: At time step k , each node sends to a neighbor that is 2 k mod (cid:98) log 2 ( n − 1) (cid:99) hops away.

Original: {+Thus, with h k = 
+}
Revised: {+Thus, with h k = 
+}

Original: Undirected exponential graph.
Revised: k mod (cid:98) log 2 ( n − 1) (cid:99) , we get that [Equation] Note that, with this design, in fact each node sends to one peer and receives from one peer at every iteration, so the communication load is balanced across the network. In the two-peer-per-node experiments, the deﬁnition is similar, but now there will be three non-zero entries in each column of P ( k ) , all of which will be equal to 1 / 3 ; these are the diagonal, and the entries corresponding to the two neighbors to which the node sends at that iteration.

Revised: In addition, each node will send two messages and receive two messages at every iteration, so the communication load is again balanced across the network.

Original:  For the D-PSGD experiments we use a time-varying undirected bipartite exponential graph to represent the inter-node connectivity.
Revised: Undirected exponential graph. For the D-PSGD experiments we use a time-varying undirected bipartite exponential graph to represent the inter-node connectivity.

Original: Odd-numbered nodes send a message to a peer that are 2 1 − 1 , 2 2 − 1 , . ..
Revised: Odd-numbered nodes send messages to peers that are 2 1 − 1 , 2 2 − 1 , . .

Original:  , 2  log 2 ( n − 1)  − 1hops away (even-numbered nodes), and wait to a receive a message back in return.
Revised: . , 2 (cid:98) log 2 ( n − 1) (cid:99) − 1 (even-numbered nodes), and wait to a receive a message back in return.

Original: Each odd-numbered node cycles through the peers in the list in a similar fashion to the one-peer-per-node SGP experiments.
Revised: Each odd-numbered node cycles through the peers in the list in a similar fashion to the one-peer-per-node SGP experiments.

Original: Even-numbered nodes wait to receive a message from some peer (unknown to the receiving node ahead of time), and send a message back in return.
Revised: Even-numbered nodes wait to receive a message from some peer (unknown to the receiving node ahead of time), and send a message back in return.

Original: We adopt these graphs to be consistent with the experimental setup used in Lian et al.
Revised: We adopt these graphs to be consistent with the experimental setup used in Lian et al.

Original: (2017) and
Revised: (2017) and

Original: Lian et al.
Revised: Lian et al.

Original: Note also that these graphs are all regular, in that all nodes have the same number of in-coming and out-going connections. 
Revised: Note also that these graphs are all regular, in that all nodes have the same number of in-coming and out-going connections. Decentralized averaging errors.

Original: A. S TOCHASTIC G RADIENT P USH 
Revised: To further motivate our choice of using the directed exponential graph with SGP, let us forget about optimization for a moment and focus on the problem of distributed averaging, described in Section 2, using the P USH S UM algorithm. Recall that each node i starts with a vector y (0) i , and the goal of the agents is to compute the average y = 1 n (cid:80) i y (0) i . Then, since y ( k +1) i = (cid:80) nj =1 p ( k ) i,j y ( k ) j , after k steps we have [Equation]

Revised: where Y ( k ) is a n × d matrix with y ( k ) i as its i th row.

Revised: Let P ( k − 1:0) =

Revised: P ( k − 1) P ( k − 2) · · · P (1) P (0) .

Revised: The worst-case rate of convergence can be related to the second-largest singular value of P ( k − 1:0) Nedi´c et al.

Revised: In particular, after k iterations we have

Revised: [Equation]

Revised: where λ 2 ( P ( k − 1:0) ) denotes the second largest singular value of P ( k − 1:0) .

Revised: For the scheme proposed above, cycling deterministically through neighbors in the directed exponential graph, one can verify that after k = (cid:98) log 2 ( n − 1) (cid:99) iterations, we have λ 2 ( P ( k − 1:0) )

Revised: = 0 , so all nodes exactly have the average.

Revised: Intuitively, this happens because the directed exponential graph has excellent mixing properties: from any starting node in the network, one can get to any other node in at most log 2 ( n ) hops.

Revised: For n = 32 nodes, after 5 iterations averaging has converged using this strategy.

Revised: In comparison, if one were to cycle through edges of the complete graph (where every node is connected to every other node), then for n = 32 , after 5 consecutive iterations one would have still have λ 2 ( P ( k − 1:0) )

Revised: ≈ 0 .

Revised: 6 ; i.e., nodes could be much further from the average (and hence, much less well-synchronized).

Revised: Similarly, one could consider designing the matrices P ( k ) in a stochastic manner, where each node randomly samples one neighbor to send to at every iteration.

Revised: If each node samples a destination uniformly from its set of neighbors in the directed exponential graph, then E λ 2 ( P ( k − 1:0) ) ≈ 0 . 4 , and if each node randomly selected a destination uniformly among all other nodes in the network (i.e., randomly from neighbors in the complete graph), then E λ 2 ( P ( k − 1:0) )

Revised: ≈ 0 .

Revised: Thus, random schemes are still not as effective at quickly averaging as deterministically cycling through neighbors in the directed exponential graph.

Revised: Moreover, with randomized schemes, we are no longer guaranteed that each node receives the same number of messages at every iteration, so the communication load will not be balanced as in the deterministic scheme.

Revised: The above discussion focused only on approximate distributed averaging, which is a key step within decentralized optimization.

Revised: When averaging occurs less quickly, this also impacts optimization.

Revised: Speciﬁcally, since nodes are less well-synchronized (i.e., further from a consensus), each node will be evaluating its local mini-batch gradient at a different point in parameter space.

Revised: Averaging these points (rather than updates based on mini-batch gradients evaluated at the same point) can be seen as injecting additional noise into the optimization process, and in our experience this can lead to worse performance in terms of train and generalization errors.

Revised: A. S TOCHASTIC G RADIENT P USH

Original: In all of our experiments, we minimize the number of ﬂoating-point operations performed in each iteration, k , by using the mixing weights
Revised: In all of our experiments, we minimize the number of ﬂoating-point operations performed in each iteration, k , by using the mixing weights

Original: [Equation]
Revised: [Equation]

Original: for all i, j = 1 , 2 , . . .
Revised: for all i, j = 1 , 2 , . . .

Original: , n .
Revised: , n .

Original: In words, each node assigns mixing weights uniformly to all of its outneighbors in each iteration.
Revised: In words, each node assigns mixing weights uniformly to all of its outneighbors in each iteration.

Original: Recalling our convention that each node is an in- and out-neighbor of itself, it is easy to see that this choice of mixing-weight satisﬁes the column-stochasticity property.
Revised: Recalling our convention that each node is an in- and out-neighbor of itself, it is easy to see that this choice of mixing-weight satisﬁes the column-stochasticity property.

Original: It may very well be that there is a different choice of mixing-weights that lead to better spectral properties of the gossip algorithm; however we leave this exploration for future work.
Revised: It may very well be that there is a different choice of mixing-weights that lead to better spectral properties of the gossip algorithm; however we leave this exploration for future work.

Original: We denote node i ’s uniform mixing weights at time t by p ( k ) i — dropping the other subscript, which identiﬁes the receiving node.
Revised: We denote node i ’s uniform mixing weights at time t by p ( k ) i — dropping the other subscript, which identiﬁes the receiving node.

Original: To maximize the utility of the resources available on each server, each node (occupying a single server exclusively) runs two threads, a gossip thread and a computation thread.
Revised: To maximize the utility of the resources available on each server, each node (occupying a single server exclusively) runs two threads, a gossip thread and a computation thread.

Original: The computation thread executes the main logic used to train the local model on the GPUs available to the node, while the communication thread is used for inter-node network I/O. In particular, the communication thread is used to gossip messages between nodes.
Revised: The computation thread executes the main logic used to train the local model on the GPUs available to the node, while the communication thread is used for inter-node network I/O. In particular, the communication thread is used to gossip messages between nodes.

Original: When using Ethernet-based communication, the nodes communicate their parameter tensors over CPUs.
Revised: When using Ethernet-based communication, the nodes communicate their parameter tensors over CPUs.

Original: When using IniﬁniBand-based communication, the nodes communicate their parameter tensors using GPUDirect RDMA, thereby avoiding superﬂuous device to pinned-memory transfers of the model parameters.
Revised: When using IniﬁniBand-based communication, the nodes communicate their parameter tensors using GPUDirect RDMA, thereby avoiding superﬂuous device to pinned-memory transfers of the model parameters.

Original: Each node initializes its model on one of its GPUs, and initializes its scalar push-sum weight to 1 .
Revised: Each node initializes its model on one of its GPUs, and initializes its scalar push-sum weight to 1 .

Original: At the start of training, each node also allocates a send - and a receive - communication-buffer in pinned memory on the CPU (or equivalently on a GPU in the case of GPUDirect RDMA communication).
Revised: At the start of training, each node also allocates a send - and a receive - communication-buffer in pinned memory on the CPU (or equivalently on a GPU in the case of GPUDirect RDMA communication).

Original: In each iteration, the communication thread waits for the send-buffer to  ﬁlled by the computation thread; transmits the message in the send-buffer to its out-neighbours; and then aggregates any newly-received messages into the receive-buffer.
Revised: In each iteration, the communication thread waits for the send-buffer to be ﬁlled by the computation thread; transmits the message in the send-buffer to its out-neighbours; and then aggregates any newly-received messages into the receive-buffer.

Original: In each iteration, the computation thread blocks to retrieve the aggregated messages in the receivebuffer; directly adds the received parameters to its own model parameters; and directly adds the received push-sum weights to its own push-sum weight.
Revised: In each iteration, the computation thread blocks to retrieve the aggregated messages in the receivebuffer; directly adds the received parameters to its own model parameters; and directly adds the received push-sum weights to its own push-sum weight.

Original: The computation thread then converts the model parameters to the de-biased estimate by dividing by the push-sum weight; executes a forwardbackward pass of the de-biased model in order to compute a stochastic mini-batch gradient; converts the model parameters back to the biased estimate by multiplying by the push-sum weight; and applies the newly-computed stochastic gradients to the biased model.
Revised: The computation thread then converts the model parameters to the de-biased estimate by dividing by the push-sum weight; executes a forwardbackward pass of the de-biased model in order to compute a stochastic mini-batch gradient; converts the model parameters back to the biased estimate by multiplying by the push-sum weight; and applies the newly-computed stochastic gradients to the biased model.

Original: The updated model parameters are then multiplied by the mixing weight, p ( k ) i , and asynchronously copied back into the send-buffer for use by the communication thread.
Revised: The updated model parameters are then multiplied by the mixing weight, p ( k ) i , and asynchronously copied back into the send-buffer for use by the communication thread.

Original: The push-sum weight is also multiplied by the same mixing weight and concatenated into the send-buffer.
Revised: The push-sum weight is also multiplied by the same mixing weight and concatenated into the send-buffer.

Original: In short, gossip is performed on the biased model parameters (push-sum numerators); stochastic gradients are computed using the de-biased model parameters; stochastic gradients are applied back to the biased model parameters; and then the biased-model and the push-sum weight are multiplied by the same uniform mixing-weight and copied back into the send-buffer.
Revised: In short, gossip is performed on the biased model parameters (push-sum numerators); stochastic gradients are computed using the de-biased model parameters; stochastic gradients are applied back to the biased model parameters; and then the biased-model and the push-sum weight are multiplied by the same uniform mixing-weight and copied back into the send-buffer.

Original: A. H YPERPARAMETERS
Revised: A. H YPERPARAMETERS

Original: When we “apply the stochastic gradients” to the biased model parameters, we actually carry out an
Revised: When we “apply the stochastic gradients” to the biased model parameters, we actually carry out an

Original: SGD step with nesterov momentum.
Revised: SGD step with nesterov momentum.

Original: For the 32 , 64 , and 128 GPU experiments we use the same exact learning-rate, schedule, momentum, and weight decay as those suggested in (Goyal et al., 2017) for SGD.
Revised: For the 32 , 64 , and 128 GPU experiments we use the same exact learning-rate, schedule, momentum, and weight decay as those suggested in (Goyal et al., 2017) for SGD.

Original: In particular, we use a reference learning-rate of 0 . 1 with respect to a 256 sample batch, and scale this linearly with the batch-size; we decay the learning-rate by a factor of 10 at epochs 30 , 60 , 80 ; we use a nesterov momentum parameter of 0 . 9 , and we use weight decay 0 . 0001 .
Revised: In particular, we use a reference learning-rate of 0 . 1 with respect to a 256 sample batch, and scale this linearly with the batch-size; we decay the learning-rate by a factor of 10 at epochs 30 , 60 , 80 ; we use a nesterov momentum parameter of 0 . 9 , and we use weight decay 0 . 0001 .

Original: For the 256 GPU experiments, we decay the learning-rate by a factor of 10 at epochs 40 , 70 , 85 , and we use a reference learning-rate of 0 . 0375 .
Revised: For the 256 GPU experiments, we decay the learning-rate by a factor of 10 at epochs 40 , 70 , 85 , and we use a reference learning-rate of 0 . 0375 .

Original: In the 256 GPU experiment with two peers-per-node, we revert to the original learning-rate and schedule.
Revised: In the 256 GPU experiment with two peers-per-node, we revert to the original learning-rate and schedule.

Original: Require:
Revised: Require:

Original: Initialize γ > 0 , m ∈ (0 , 1) , x (0)
Revised: Initialize γ > 0 , m ∈ (0 , 1) , x (0)

Original: i =
Revised: i =

Original: z (0) i ∈ R d and w (0) i
Revised: z (0) i ∈ R d and w (0) i

Original: = 1 for all nodes i ∈ [ n ] 1: for k = 0 , 1 , 2 , · · · , K do at node i 2:
Revised: = 1 for all nodes i ∈ [ n ] 1: for k = 0 , 1 , 2 , · · · , K do at node i 2:

Original: Sample new mini-batch ξ ( k ) i ∼ D i from local distribution 3:
Revised: Sample new mini-batch ξ ( k ) i ∼ D i from local distribution 3:

Original: Compute a local stochastic mini-batch gradient at z ( k ) i : ∇ F i ( z ( k ) i ; ξ ( k ) i ) 4: u ( k +1) i
Revised: Compute a local stochastic mini-batch gradient at z ( k ) i : ∇ F i ( z ( k ) i ; ξ ( k ) i ) 4: u ( k +1) i

Original: = m u ( k ) i + ∇ F i
Revised: = m u ( k ) i + ∇ F i

Original: ( z ( k ) i ; ξ ( k ) i ) 5:
Revised: ( z ( k ) i ; ξ ( k ) i ) 5:

Original: x ( k + 12 ) i =
Revised: x ( k + 12 ) i =

Original: x ( k ) i − γ ( m u ( k +1)
Revised: x ( k ) i − γ ( m u ( k +1)

Original: i + ∇ F i
Revised: i + ∇ F i

Original: ( z ( k ) i ; ξ ( k ) i )) 6:
Revised: ( z ( k ) i ; ξ ( k ) i )) 6:

Original: Send (cid:0) p ( k ) j,i x ( k + 12 ) i , p ( k ) j,i w ( k ) i (cid:1) to out-neighbors j ∈ N out
Revised: Send (cid:0) p ( k ) j,i x ( k + 12 ) i , p ( k ) j,i w ( k ) i (cid:1) to out-neighbors j ∈ N out

Original: i ( k ) ; receive (cid:0) p ( k ) i,j x ( k + 12 ) j , p ( k ) i,j w ( k ) j (cid:1) from in-neighbors j ∈ N in i ( k ) 7: x ( k +1)
Revised: i ( k ) ; receive (cid:0) p ( k ) i,j x ( k + 12 ) j , p ( k ) i,j w ( k ) j (cid:1) from in-neighbors j ∈ N in i ( k ) 7: x ( k +1)

Original: i =
Revised: i =

Original: (cid:80) j ∈N in i ( k ) p ( k ) i,j x ( k + 12 ) j 8: w ( k +1)
Revised: (cid:80) j ∈N in i ( k ) p ( k ) i,j x ( k + 12 ) j 8: w ( k +1)

Original: i =
Revised: i =

Original: (cid:80) j ∈N in i ( k ) p ( k ) i,j w ( k ) j 9:
Revised: (cid:80) j ∈N in i ( k ) p ( k ) i,j w ( k ) j 9:

Original: z ( k +1) i
Revised: z ( k +1) i

Original: =
Revised: =

Original: x ( k +1)
Revised: x ( k +1)

Original: i /w ( k +1)
Revised: i /w ( k +1)

Original: i 10: end for
Revised: i 10: end for

Original: B E XTRA E XPERIMENTS B. A DDITIONAL T RAINING C URVES
Revised: B E XTRA E XPERIMENTS B. A DDITIONAL T RAINING C URVES

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: Figure5 show the train and validation curve for the different runs performed on Ethernet 10Gbit/s.
Revised: FigureB.1 show the train and validation curve for the different runs performed on Ethernet 10Gbit/s.

Original: Figure 6 show the train and validation curve for the different runs performed on InﬁniBand 100Gbit/s
Revised: Figure B.2 show the train and validation curve for the different runs performed on InﬁniBand 100Gbit/s.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Revised: Figure B.3 reports the training and validation accuracy of SGP when using a high-latency interconnect.

Revised: As we scale up the number of nodes n , we scale down the total number of iterations K to K/n following Theorem 1.

Revised: In particular, 32-node runs involves 8 times fewer global iterations than 4-node runs.

Revised: We additionally report the total number of iterations and the ﬁnal performances in Table 3.

Revised: While we reduce the total number iterations by a factor of 8 when going from 4 tonodes, the validation accuracy and training accuracy of the 32 node runs remain within 1 . 7% and 2 .

Revised: 6% , respecively, of the validation and training accuracy achieved by the 4-node runs (and remains within the 1 . 2% of A LL R EDUCE SGD accuracies).

Revised: [Table]

Original: B. D ISCREPANCY ACROSS DIFFERENT NODES
Revised: B. D ISCREPANCY ACROSS DIFFERENT NODES

Original: Here, we investigate the performance variability across nodes during training for SGP.
Revised: Here, we investigate the performance variability across nodes during training for SGP.

Original: In ﬁgure 7, we report the minimum, maximum and mean error across the different nodes for training and validation.
Revised: In ﬁgure B.4, we report the minimum, maximum and mean error across the different nodes for training and vali- [Figure]

Original:  In an initial training phase, we observe that nodes have different validation errors; their local copies of the Resnet-50 model diverge.
Revised: dation. In an initial training phase, we observe that nodes have different validation errors; their local copies of the Resnet-50 model diverge.

Original: As we decrease the learning, the variability between the different nodes diminish and the nodes eventually converging to similar errors.
Revised: As we decrease the learning, the variability between the different nodes diminish and the nodes eventually converging to similar errors.

Original: This suggests that all models ultimately represent the same function, achieving consensus.
Revised: This suggests that all models ultimately represent the same function, achieving consensus [Figure]

Original: B. T IMINGOF A R ES N ET -50 T RAINED ON I NFINI B AND 
Revised: B. T IMING ON I NFINI B AND WITH LOCAL DATA COPY

Revised: To better isolate the effects of data-loading

Original: [Figure]
Revised: [Figure]

Original: of a ResNet-50 model on a cluster using InﬁniBand interconnect.
Revised: we ran experiments on 32,

Original: SGP computation and communication time remains almost constant as we increase the number of nodes.
Revised: [Figure] and 128 GPUs

Original: SGD on the other hand shows a slight increases in its per-iteration time. On InﬁniBand interconnect, SGD and SGP obtain similar timing and differs at most by 21 ms per iteration. In addition, SGP outperforms A LL R EDUCE SGD when using 32 nodes by a small 13 ms second margin Time differences of this scale could be impacted by other factors such as data loading. We therefore speciﬁcaly track the computation and communication time and the data loading time during training in ﬁgure 8.
Revised: [Figure]

Original: We observe  that  SGP computation and communication time is close to constant as we grows the number of nodes. SGD on theother hand sees a slight increases. Data loading timeremains similar for both algorithms.
Revised: where we ﬁrst copied the data locally on every node. In that setting, we observe in Figure B.5 that the time-per-iteration of SGP remains approximately constant as we increase the number of nodes in the network, while the time for A LL R EDUCE SGD increases

Original: [Figure]
Revised: [Figure]

Revised: B. SGP S CALING A NALYSIS

Original: [Figure]
Revised: [Figure]

Revised: B

Revised: 6 highlights SGP input images throughput as we scale up the number of cluster node on both Ethernet 10Gbit/s and Inﬁniband 100Gbit/s. exhibits 88.6% scaling efﬁciency on

Revised: 10Gbit/s and 92.4% on InﬁniBand and stay close to the ideal scaling in both cases.

Revised: In addition

Revised: Figure (c) shows that SGP exhibit better scaling as we increase the network size and is more robust

Original: C P ROOFS OF T HEORETICAL G UARANTEES
Revised: C P ROOFS OF T HEORETICAL G UARANTEES

Original: Our convergence rate analysis is divided into three main parts.
Revised: Our convergence rate analysis is divided into three main parts.

Original: In the ﬁrst one (subsection C.1) we present upper bounds for three important expressions that appear in our computations.
Revised: In the ﬁrst one (subsection C.1) we present upper bounds for three important expressions that appear in our computations.

Original: In subsection
Revised: In subsection

Original: C.2 we focus on proving the important for our analysis Lemma 8 based on which we later build the proofs of our main Theorems.
Revised: C.2 we focus on proving the important for our analysis Lemma 8 based on which we later build the proofs of our main Theorems.

Original: Finally in the third part (subsection C.3)
Revised: Finally in the third part (subsection C.3)

Original: we provide the proofs for
Revised: we provide the proofs for

Original: Theorems 1 and 2.
Revised: Theorems 1 and 2.

Original: Preliminary results.
Revised: Preliminary results.

Original: In our analysis two preliminary results are extensively used.
Revised: In our analysis two preliminary results are extensively used.

Original: We state them here for future reference.
Revised: We state them here for future reference.

Original: • Let a, b ∈ R .
Revised: • Let a, b ∈ R .

Original: Since ( a − b ) 2 ≥ 0 , it holds that
Revised: Since ( a − b ) 2 ≥ 0 , it holds that

Original: [Equation]
Revised: [Equation]

Original: Thus, (cid:107) x (cid:107) (cid:107) y (cid:107) ≤
Revised: Thus, (cid:107) x (cid:107) (cid:107) y (cid:107) ≤

Original: ( (cid:107) x (cid:107) 

Revised: ( (cid:107) x (cid:107)

Original: + (cid:107) y (cid:107) 2 ) / 2 .
Revised: + (cid:107) y (cid:107) 2 ) / 2 .

Original: • Let r ∈
Revised: • Let r ∈

Original:  then from the summation of geometric sequence and for any K ≤ ∞ it holds that
Revised: (0 , 1) then from the summation of geometric sequence and for any K ≤ ∞ it holds that

Original: [Equation]
Revised: [Equation]

Original: Matrix Representation.
Revised: Matrix Representation.

Original: The presentation of stochastic gradient push (Algorithm 1) was done from node i’s perspective for all i ∈ [ n ] .
Revised: The presentation of stochastic gradient push (Algorithm 1) was done from node i’s perspective for all i ∈ [ n ] .

Original: Note however, that the update rule of SGP at the k th iteration can be viewed from a global viewpoint.
Revised: Note however, that the update rule of SGP at the k th iteration can be viewed from a global viewpoint.

Original: To see this let us deﬁne the following matrices (concatenation of the values of all nodes at the k th iteration):
Revised: To see this let us deﬁne the following matrices (concatenation of the values of all nodes at the k th iteration):

Original: [Equation]
Revised: [Equation]

Original: Using the above matrices, the 6 th step of SGP (Algorithm 1) can be expressed as follows 2 :
Revised: Using the above matrices, the 6 th step of SGP (Algorithm 1) can be expressed as follows 2 :

Original: [Equation]
Revised: [Equation]

Original: where [ P ( k ) ]
Revised: where [ P ( k ) ]

Original: T is the transpose of matrix P k with entries:
Revised: T is the transpose of matrix P k with entries:

Original: [Equation]
Revised: [Equation]

Original: Recall that we also have x
Revised: Recall that we also have x

Original: [Equation]
Revised: [Equation]

Original: Bound for the mixing matrices.
Revised: Bound for the mixing matrices.

Original: Next we state a known result from the control literature studying consensus-based optimization which allows us to bound the distance between the de-biased parameters at each node and the node-wise average.
Revised: Next we state a known result from the control literature studying consensus-based optimization which allows us to bound the distance between the de-biased parameters at each node and the node-wise average.

Original: Recall that we have assumed that the sequence of mixing matrices P ( k ) are B -strongly connected.
Revised: Recall that we have assumed that the sequence of mixing matrices P ( k ) are B -strongly connected.

Original: A directed graph is called strongly connected if every pair of vertices is connected with a directed path ( i.e. , following the direction of edges), and the B -strongly connected assumption is that the graph with edge set (cid:83) ( l +1) B − 1 k = lB
Revised: A directed graph is called strongly connected if every pair of vertices is connected with a directed path ( i.e. , following the direction of edges), and the B -strongly connected assumption is that the graph with edge set (cid:83) ( l +1) B − 1 k = lB

Original: E ( k ) is strongly connected, for every l ≥ 0 .
Revised: E ( k ) is strongly connected, for every l ≥ 0 .

Original: We have also assumed that for all k ≥ 0 , each column of P ( k ) has D non-zero entries, and the diameter of the graph with edge set (cid:83) ( l +1)
Revised: We have also assumed that for all k ≥ 0 , each column of P ( k ) has D non-zero entries, and the diameter of the graph with edge set (cid:83) ( l +1)

Original: B − 1 k = lB
Revised: B − 1 k = lB

Original: E ( k ) has diameter at most ∆ .
Revised: E ( k ) has diameter at most ∆ .

Original: Based on these assumptions, after ∆ B consecutive iterations, the product
Revised: Based on these assumptions, after ∆ B consecutive iterations, the product

Original: [Equation]
Revised: [Equation]

Original: has no non-zero entries.
Revised: has no non -zero entries.

Original: Moreover, every entry of A ( k ) is at least D − ∆ B .
Revised: Moreover, every entry of A ( k ) is at least D − ∆ B .

Original: Lemma 3.
Revised: Lemma 3.

Original: Suppose that Assumption 3 (mixing connectivity) holds.
Revised: Suppose that Assumption 3 (mixing connectivity) holds.

Original: Let λ = 1 − nD − ∆ B and let q
Revised: Let λ = 1 − nD − ∆ B and let q

Original: = λ 1 / (∆ B +1) .
Revised: = λ 1 / (∆ B +1) .

Original: Then there exists a constant
Revised: Then there exists a constant

Original: [Equation]
Revised: [Equation]

Original: where d is the dimension of x ( k ) ,
Revised: where d is the dimension of x ( k ) ,

Original: z i ( k ) , and x i (0) , such that, for all i = 1 , 2 , . . . , n and k ≥ 0 ,
Revised: z i ( k ) , and x i (0) , such that, for all i = 1 , 2 , . . . , n and k ≥ 0 ,

Original: [Equation]
Revised: [Equation]

Original: This particular lemma follows after a small adaptation to Theorem 1 in Assran & Rabbat (2018) and its proof is based on Wolfowitz (1963).
Revised: This particular lemma follows after a small adaptation to Theorem 1 in Assran & Rabbat (2018) and its proof is based on Wolfowitz (1963).

Original: Similar bounds appear in a variety of other papers, including
Revised: Similar bounds appear in a variety of other papers, including

Original: Nedi ´ c & Olshevsky (2016).
Revised: Nedi´c & Olshevsky (2016).

Original: C. I MPORTANT U PPER B OUNDS
Revised: C. I MPORTANT U PPER B OUNDS

Original: Lemma 4 (Bound of stochastic gradient) .
Revised: Lemma 4 (Bound of stochastic gradient) .

Original: We have the following inequality under Assumptionsand 2:
Revised: We have the following inequality under Assumptionsand 2:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Revised: E

Original: [Equation]
Revised: [Equation]

Revised: ≤

Revised: [Equation]

Revised: [Equation]

Revised: (cid:13)(cid:13)(cid:13)

Revised: [Equation]

Revised: +

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: x

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: ≤

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: −

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Original: Lemma 5.
Revised: Lemma 5.

Original: Let Assumptions 1-3 hold.
Revised: Let Assumptions 1-3 hold.

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Revised: ≤ (cid:18)

Revised: [Equation]

Revised: [Equation]

Revised: (cid:19)

Revised: [Equation]

Revised: (cid:18)

Revised: [Equation]

Revised: [Equation]

Revised: [Equation]

Revised: [Equation]

Revised: (cid:19)

Revised: [Equation]

Revised: + (cid:18)

Revised: [Equation]

Revised: (cid:19) k (cid:88)

Revised: [Equation]

Revised: [Equation]

Revised: k

Revised: [Equation]

Revised: ( j )

Revised: [Equation]

Revised: + (cid:18)

Revised: [Equation]

Revised: (cid:19)

Revised: [Equation]

Revised: (cid:88)

Revised: [Equation]

Revised: =

Revised: [Equation]

Revised: (cid:13)(cid:13)(cid:13)

Original: Proof.
Revised: [Equation] [Equation] ( j ) ) (cid:13)(cid:13)(cid:13)+ (cid:18)

Revised: [Equation]

Original: Q
Revised: k

Original: [Equation]
Revised: [Equation]

Original: k ) i
Revised: 1 −

Original: [Equation]
Revised: [Equation]

Original:  
Revised: (cid:19) (cid:13)(cid:13)(cid:13) x i (0) (cid:13)(cid:13)(cid:13) 2 .

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Revised: s =

Revised: q k − s (cid:13)(cid:13)(cid:13) ∇ f i ( z ( s ) i ) (cid:13)(cid:13)(cid:13) (cid:124) (cid:123)(cid:122) (cid:125) c   (13)

Original: Thus, using the above expressions of a , b and c we have that Q ( k ) i ≤ E [Equation] ac ) .
Revised: Thus, using the above expressions of a , b and c we have that Q ( k ) i ≤ E ( a 2 + b 2 + c 2 +2 ab +2 bc +2 ac ) .

Original: Let us now obtain bounds for all of these quantities:
Revised: Let us now obtain bounds for all of these quantities:

Original: [Equation]
Revised: [Equation]

Original: [Table]

Original: [Equation]
Revised: [Equation]

Original: The expression b 1 is bounded as follows:

Original: [Equation]

Original: [Equation]

Original: [Equation]

Original: (cid:13)(cid:13)(cid:13)
Revised: (cid:13)(cid:13)(cid:13)

Original: [Equation]
Revised: [Equation]

Original: z ( j ) i

Original: [Equation]

Original: ( j ) i )

Original: [Equation]

Original: i
Revised: k

Original: [Equation]
Revised: [Equation]

Original: z ( j ) i ) (cid:13)(cid:13)(cid:13)

Original: [Equation]

Original: [Equation]

Original: (cid:13)(cid:13)(cid:13)
Revised: (cid:13)(cid:13)(cid:13)

Original: [Equation]
Revised: [Equation]

Original: [Equation] (cid:13)(cid:13)(cid:13)
Revised: j )

Original: [Equation]
Revised: [Equation]

Original:  i 
Revised: ξ ( j ) i ) − ∇ f

Original: [Equation]
Revised: [Equation]

Original: j ) i )

Original: [Equation]
Revised: [Equation]

Original: i ( z ( j ) i ) (cid:13)(cid:13)(cid:13)k
Revised: ( j ) i ) (cid:13)(cid:13)(cid:13) (cid:13)(cid:13)(cid:13) ∇ F

Original: [Equation]
Revised: [Equation]

Original:  s [Equation] [Equation]
Revised: ( z ( s )

Original: k
Revised: i

Original: [Equation]
Revised: [Equation]

Original:  s [Equation]
Revised: ( s ) i )

Original: [Equation]
Revised: [Equation]

Original: i
Revised: f

Original: [Equation]

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: (  s ) i ) (cid:13)(cid:13)(cid:13)k
Revised: ( z ( s ) i ) (cid:13)(cid:13)(cid:13)

Original: [Equation]
Revised: [Equation]

Original: j =
Revised: (cid:123)(cid:122) (cid:125)

Original: [Equation]
Revised: [Equation]

Original: k
Revised: j

Original: [Equation] 
Revised: [Equation] [Equation]

Revised: (cid:13)(cid:13)(cid:13)

Original: [Equation]
Revised: [Equation]

Original: j
Revised: f

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original:  (cid:13)(cid:13)(cid:13)
Revised: ( j ) i ) (cid:13)(cid:13)(cid:13)

Original: [Equation]
Revised: [Table]

Revised: The expression b 1 is bounded as follows:

Original: [Equation]
Revised: [Equation]

Original: Thus,
Revised: Thus,

Original: [Equation]
Revised: [Equation]

Original: 15) where in the ﬁrst inequality above we use the fact that for q ∈ (0 , 1) , we have q k < 11 − q , ∀ k > 0 .
Revised: where in the ﬁrst inequality above we use the fact that for q ∈ (0 , 1) , we have q k < 11 − q , ∀ k > 0 .

Original: By identical construction we have
Revised: By identical construction we have

Original: [Equation]
Revised: [Equation]

Original: Now let us bound the products 2 ab , 2 ac and 2 bc .
Revised: Now let us bound the products 2 ab , 2 ac and 2 bc .

Original: [Equation]
Revised: [Equation]

Original: By similar procedure,
Revised: By similar procedure,

Original: [Equation]
Revised: [Equation]

Original: Finally,
Revised: Finally,

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: By combining all of the above bounds together we obtain:
Revised: By combining all of the above bounds together we obtain:

Original: [Equation]
Revised: [Equation]

Original: After grouping terms together and using the upper bound of Lemma 4, we obtain
Revised: After grouping terms together and using the upper bound of Lemma 4, we obtain

Original: [Equation]
Revised: [Equation]

Original: ( k )
Revised: ( k )

Original: [Equation]
Revised: [Equation]

Original: This completes the proof.
Revised: This completes the proof.

Original: Having found a bound for the quantity Q ( k ) i , let us know present a lemma for bounding the quantity (cid:80) K − 1 k =
Revised: Having found a bound for the quantity Q ( k ) i , let us know present a lemma for bounding the quantity (cid:80) K − 1 k =

Original: M ( k ) where K > 1 is a constant and M ( k ) is the average of Q ( k ) i across all nodes i ∈
Revised: M ( k ) where K > 1 is a constant and M ( k ) is the average of Q ( k ) i across all nodes i ∈

Original: [ n ] .
Revised: [ n ] .

Original: That is, M ( k ) = 1 n (cid:80) ni =1 Q ( k ) i .
Revised: That is, M ( k ) = 1 n (cid:80) ni =1 Q ( k ) i .

Original: Lemma 6.
Revised: Lemma 6.

Original: Let Assumptions 1-3 hold and let us deﬁne D 2 = 1 − γ 2 12 L 2 C 2 (1 − q ) 2 − γ 3 L 2 C 2 (1 − q ) 2 .
Revised: Let Assumptions 1-3 hold and let us deﬁne D 2 = 1 − γ 2 12 L 2 C 2 (1 − q ) 2 − γ 3 L 2 C 2 (1 − q ) 2 .

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Using the bound for Q ( k ) i let us ﬁrst bound its average across all nodes M ( k )
Revised: Using the bound for Q ( k ) i let us ﬁrst bound its average across all nodes M ( k )

Original: [Equation]
Revised: [Equation]

Original: At this point note that for any λ ∈ (0 , 1) , non-negative integer K ∈ N , and non-negative sequence { β ( j ) } kj =0 , it holds that
Revised: At this point note that for any λ ∈ (0 , 1) , non-negative integer K ∈ N , and non-negative sequence { β ( j ) } kj =0 , it holds that

Original: [Equation]
Revised: [Equation]

Original: Similarly,
Revised: Similarly,

Original: [Equation]
Revised: [Equation]

Original: Now by summing from k = 0 to K − 1 and using the bounds of (23) and (24) we obtain:
Revised: Now by summing from k = 0 to K − 1 and using the bounds of (23) and (24) we obtain:

Original: [Equation]
Revised: [Equation]

Original: )(cid:19) (cid:80) ni =1 (cid:13)(cid:13) x i (0) (cid:13)(cid:13)n + (cid:18) γ 2 12 C(1 − q ) 2 + γ 3 C1 − q(cid:19) K − 1 (cid:88) k
Revised: )(cid:19) (cid:80) ni =1 (cid:13)(cid:13) x i (0) (cid:13)(cid:13)n + (cid:18) γ 2 12 C(1 − q ) 2 + γ 3 C1 − q(cid:19) K − 1 (cid:88) k

Original: =
Revised: =

Original: E (cid:13)(cid:13)(cid:13) ∇ f ( x ( k ) )
Revised: E (cid:13)(cid:13)(cid:13) ∇ f ( x ( k ) )

Original: (cid:13)(cid:13)(cid:13)+ (cid:18) γ 2 12 L 2 C(1 − q ) 2 + γ 3 L 2 C1 − q(cid:19) K − 1 (cid:88) k
Revised: (cid:13)(cid:13)(cid:13)+ (cid:18) γ 2 12 L 2 C(1 − q ) 2 + γ 3 L 2 C1 − q(cid:19) K − 1 (cid:88) k

Original: =
Revised: =

Original: M ( k ) .
Revised: M ( k ) .

Original: By rearranging: (cid:18)
Revised: By rearranging: (cid:18)

Original: [Equation]
Revised: [Equation]

Original: (cid:19)
Revised: (cid:19)

Original: [Equation]
Revised: [Equation]

Original: M
Revised: M

Original: [Equation]
Revised: [Equation]

Original: (cid:18)
Revised: (cid:18)

Original: [Equation]
Revised: [Equation]

Original: 4 C
Revised: 4 C

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: q )
Revised: q )

Original: [Equation]
Revised: [Equation]

Original: (cid:19) σ
Revised: (cid:19) σ

Original: [Equation]
Revised: [Equation]

Original: K
Revised: K

Original: [Equation]
Revised: [Equation]

Original: (cid:18)
Revised: (cid:18)

Original: [Equation]
Revised: [Equation]

Original: C(1 − q )(cid:19) σ+ (cid:18)
Revised: C(1 − q )(cid:19) σ+ (cid:18)

Original: γ 2 12 C(1 − q )(cid:19) ζ 2 K + (cid:18) γ 3 C(1 − q )(cid:19) ζ+
Revised: γ 2 12 C(1 − q )(cid:19) ζ 2 K + (cid:18) γ 3 C(1 − q )(cid:19) ζ+

Original: (cid:18) C1 − q 2 + γ 2 C(1 − q )(cid:19) (cid:80) ni
Revised: (cid:18) C1 − q 2 + γ 2 C(1 − q )(cid:19) (cid:80) ni

Original: =1 (cid:13)(cid:13) x
Revised: =1 (cid:13)(cid:13) x

Original: i (0) (cid:13)(cid:13)n + (cid:18) γ 2 12 C(1 − q ) 2 + γ 3 C1 − q(cid:19) K − 1 (cid:88) k
Revised: i (0) (cid:13)(cid:13)n + (cid:18) γ 2 12 C(1 − q ) 2 + γ 3 C1 − q(cid:19) K − 1 (cid:88) k

Original: =
Revised: =

Original: E (cid:13)(cid:13)(cid:13) ∇
Revised: E (cid:13)(cid:13)(cid:13) ∇

Original: [Equation]
Revised: [Equation]

Original: Note that since q ∈ (0 , 1) it holds that
Revised: Note that since q ∈ (0 , 1) it holds that

Original: [Equation]
Revised: [Equation]

Original: Thus,
Revised: Thus,

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: (cid:19)
Revised: (cid:19)

Original: [Equation]
Revised: [Equation]

Original: γ
Revised: γ

Original: C
Revised: C

Original: [Equation]
Revised: [Equation]

Original: (1 − q )
Revised: (1 − q )

Original: [Equation]
Revised: [Equation]

Original: (cid:19) σ
Revised: (cid:19) σ

Original: [Equation]
Revised: [Equation]

Original: + (cid:18) γ
Revised: + (cid:18) γ

Original: [Equation]
Revised: [Equation]

Original: 12 C
Revised: 12 C

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: q )
Revised: q )

Original: [Equation]
Revised: [Equation]

Original: (cid:19) ζ
Revised: (cid:19) ζ

Original: [Equation]
Revised: [Equation]

Original: K + (cid:18) γ 3 C(1 − q )(cid:19) ζ+ (cid:18)
Revised: K + (cid:18) γ 3 C(1 − q )(cid:19) ζ+ (cid:18)

Original: C(1 − q ) 2 + γ 2 C(1 − q )(cid:19) (cid:80) ni
Revised: C(1 − q ) 2 + γ 2 C(1 − q )(cid:19) (cid:80) ni

Original: =1 (cid:13)(cid:13) x
Revised: =1 (cid:13)(cid:13) x

Original: i (0) (cid:13)(cid:13)n + (cid:18) γ
Revised: i (0) (cid:13)(cid:13)n + (cid:18) γ

Original: [Equation]
Revised: [Equation]

Original: 12 C
Revised: 12 C

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: γ 3 C
Revised: γ 3 C

Original: [Equation]
Revised: [Equation]

Original: (1 − q )
Revised: (1 − q )

Original: [Equation]
Revised: [Equation]

Original: (cid:19) K − 1 (cid:88) k
Revised: (cid:19) K − 1 (cid:88) k

Original: =
Revised: =

Original: E (cid:13)(cid:13)(cid:13) ∇ f ( x ( k ) )
Revised: E (cid:13)(cid:13)(cid:13) ∇ f ( x ( k ) )

Original: (cid:13)(cid:13)(cid:13)
Revised: (cid:13)(cid:13)(cid:13)

Original: Dividing both sides with D 2 = 1 − γ 2 12 L 2 C(1 − q ) 2 − γ 3 L 2 C(1 − q ) 2 completes the proof.
Revised: Dividing both sides with D 2 = 1 − γ 2 12 L 2 C(1 − q ) 2 − γ 3 L 2 C(1 − q ) 2 completes the proof.

Original: C.
Revised: C.

Original: T OWARDS THE PROOF OF THE MAIN T HEOREMS
Revised: T OWARDS THE PROOF OF THE MAIN T HEOREMS

Original: The goal of this section is the presentation of Lemma 8.
Revised: The goal of this section is the presentation of Lemma 8.

Original: It is the main lemma of our convergence analysis and based on which we build the proofs of Theorems 1 and 2.
Revised: It is the main lemma of our convergence analysis and based on which we build the proofs of Theorems 1 and 2.

Original: Let us ﬁrst state a preliminary lemma that simpliﬁes some of the expressions that involve expectations with respect to the random variable ξ ( t ) i .
Revised: Let us ﬁrst state a preliminary lemma that simpliﬁes some of the expressions that involve expectations with respect to the random variable ξ ( t ) i .

Original: Lemma 7.
Revised: Lemma 7.

Original: Under the deﬁnition of our problem and the Assumptions 1-3 we have that:
Revised: Under the deﬁnition of our problem and the Assumptions 1-3 we have that:

Original: (i)
Revised: (i)

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: (ii)
Revised: (ii)

Original: [Table]
Revised: [Table]

Original: Proof.
Revised: Proof.

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: n
Revised: n

Original: [Equation]
Revised: [Equation]

Original:  [Table] [Table]
Revised: [Equation] [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Equation]
Revised: [Equation]

Original: Before present the proof of next lemma let us deﬁne the conditional expectation E [ ·|F k ] =
Revised: Before present the proof of next lemma let us deﬁne the conditional expectation E [ ·|F k ] =

Original: E ξ ( k ) i ∼D i [ · ] =
Revised: E ξ ( k ) i ∼D i [ · ] =

Original: E ξ ( k ) i [ · ] .
Revised: E ξ ( k ) i [ · ] .

Original: The expectation in this expression is with respect to the random choice ξ ( k ) i for node i ∈ [ n ] at the k th iteration.
Revised: The expectation in this expression is with respect to the random choice ξ ( k ) i for node i ∈ [ n ] at the k th iteration.

Original: In other words, F k denotes all the information generated by the stochastic gradient-push algorithm by time t , i.e., all the x ( k ) i , z ( k ) i , w ( k ) i , y ( k ) i , ∇ F i ( z ( k ) i ; ξ ( k ) i ) for k = 1 , . . .
Revised: In other words, F k denotes all the information generated by the stochastic gradient-push algorithm by time t , i.e., all the x ( k ) i , z ( k ) i , w ( k ) i , y ( k ) i , ∇ F i ( z ( k ) i ; ξ ( k ) i ) for k = 1 , . . .

Original: , t .
Revised: , t .

Original: In addition, we should highlight that the choices of random variables ξ ki ∼ D i , ξ kj ∼ D j at the step t of the algorithm, are independent for any two nodes i (cid:54) =
Revised: In addition, we should highlight that the choices of random variables ξ ki ∼ D i , ξ kj ∼ D j at the step t of the algorithm, are independent for any two nodes i (cid:54) =

Original: j ∈ [ n ] .
Revised: j ∈ [ n ] .

Original: This is also true in the case that the two nodes follow the same distribution D = D i =
Revised: This is also true in the case that the two nodes follow the same distribution D = D i =

Original: D j .
Revised: D j .

Original: Lemma 8.
Revised: Lemma 8.

Original: Let Assumptions 1-3 hold and let
Revised: Let Assumptions 1-3 hold and let

Original: [Equation]
Revised: [Equation]

Original: Here C > 0 and q ∈ (0 , 1) are the two non-negative constants deﬁned in Lemma 3.
Revised: Here C > 0 and q ∈ (0 , 1) are the two non-negative constants deﬁned in Lemma 3.

Original: Let { X k } ∞ k
Revised: Let { X k } ∞ k

Original: = be the random sequence produced by (9) (Matrix representation of Algorithm 1).
Revised: = be the random sequence produced by (9) (Matrix representation of Algorithm 1).

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: [Equation]
Revised: [Equation]

Original: Taking expectations of both sides with respect to F k :
Revised: Taking expectations of both sides with respect to F k :

Original: [Equation]
Revised: [Equation]

Original: |F k (cid:35)
Revised: |F k (cid:35)

Original: Lemma 7[ i ] = f (cid:18) X ( k ) 1 n n (cid:19) − γ (cid:28) ∇ f (cid:18)
Revised: Lemma 7[ i ] = f (cid:18) X ( k ) 1 n n (cid:19) − γ (cid:28) ∇ f (cid:18)

Original: X
Revised: X

Original: ( k ) 1 n n (cid:19) , ∇ F ( Z ( k ) )
Revised: ( k ) 1 n n (cid:19) , ∇ F ( Z ( k ) )

Original: 1 n n (cid:29) + Lγ2 E  
Revised: 1 n n (cid:29) + Lγ2 E  

Original: (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:80) ni
Revised: (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:80) ni

Original: =1 ∇ F i ( z ( k ) i ; ξ ( k ) i ) − (cid:80) ni =1 ∇ f i ( z ( k ) i ) n (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) |F
Revised: =1 ∇ F i ( z ( k ) i ; ξ ( k ) i ) − (cid:80) ni =1 ∇ f i ( z ( k ) i ) n (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) |F

Original: k   + Lγ

Revised: k   + Lγ

Original: E [ (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:80) ni
Revised: E [ (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:80) ni

Original: =1 ∇ f i ( z ( k ) i ) n (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
Revised: =1 ∇ f i ( z ( k ) i ) n (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)

Original: |F k ]
Revised: |F k ]

Original: Lemma 7[ ii ] ≤ f (cid:18) X ( k ) 1 n n (cid:19) − γ (cid:28) ∇ f (cid:18)
Revised: Lemma 7[ ii ] ≤ f (cid:18) X ( k ) 1 n n (cid:19) − γ (cid:28) ∇ f (cid:18)

Original: X
Revised: X

Original: ( k ) 1 n n (cid:19) , ∇ F ( Z ( k ) )
Revised: ( k ) 1 n n (cid:19) , ∇ F ( Z ( k ) )

Original: 1 n n (cid:29) + Lγ 2 σ 2 n +
Revised: 1 n n (cid:29) + Lγ 2 σ 2 n +

Original: Lγ2 E  
Revised: Lγ2 E  

Original: (cid:13)(cid:13) (cid:13)(cid:13)(cid:13) (cid:80) n i =1 ∇
Revised: (cid:13)(cid:13) (cid:13)(cid:13)(cid:13) (cid:80) n i =1 ∇

Original: f
Revised: f

Original: i
Revised: i

Original: z ( k ) i ) n (cid:13)(cid:13) (cid:13)(cid:13)(cid:13)
Revised: z ( k ) i ) n (cid:13)(cid:13) (cid:13)(cid:13)(cid:13)

Original: |F k   =
Revised: |F k   =

Original: f (cid:18) X ( k )
Revised: f (cid:18) X ( k )

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where in the last step above we simply expand the inner product.
Revised: where in the last step above we simply expand the inner product.

Original: Taking expectations again and using the tower property, we get
Revised: Taking expectations again and using the tower property, we get

Original: [Table]
Revised: [Table]

Revised: [Equation]

Original:  [Equation]
Revised: [Table] [Equation] [Table] [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table][Equation] [Table] [Equation]
Revised: [Table]

Original: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: (cid:35)
Revised: (cid:35)

Original: [Equation]
Revised: [Equation]

Original: (cid:13)(cid:13)(cid:13)(cid:13)
Revised: (cid:13)(cid:13)(cid:13)(cid:13)

Original: [Equation]
Revised: [Equation]

Original: (cid:13)(cid:13)(cid:13)(cid:13)
Revised: (cid:13)(cid:13)(cid:13)(cid:13)

Original: [Equation]
Revised: [Equation]

Original: n (cid:13)(cid:13)(cid:13)(cid:13)
Revised: n (cid:13)(cid:13)(cid:13)(cid:13)

Original: [Equation]
Revised: [Equation]

Original: (cid:35)
Revised: (cid:35)

Original: [Equation]
Revised: [Equation]

Original: n (29)
Revised: n (29)

Original: Let us now focus on ﬁnd an upper bound for the quantity
Revised: Let us now focus on ﬁnd an upper bound for the quantity

Original: E (cid:20)(cid:13)(cid:13)(cid:13) ∇ f (cid:16) X ( k ) 1 n n (cid:17) − ∇ F ( Z ( k ) )
Revised: E (cid:20)(cid:13)(cid:13)(cid:13) ∇ f (cid:16) X ( k ) 1 n n (cid:17) − ∇ F ( Z ( k ) )

Original: 1 n n (cid:13)(cid:13)(cid:13) 2 (cid:21) .
Revised: 1 n n (cid:13)(cid:13)(cid:13) 2 (cid:21) .

Original: E (cid:34)(cid:13)(cid:13)(cid:13)(cid:13) ∇ f (cid:18) X ( k ) 1 n n (cid:19)
Revised: E (cid:34)(cid:13)(cid:13)(cid:13)(cid:13) ∇ f (cid:18) X ( k ) 1 n n (cid:19)

Original: − ∇ F ( Z ( k ) )
Revised: − ∇ F ( Z ( k ) )

Original: 1 n n (cid:13)(cid:13)(cid:13)(cid:13) 2 (cid:35) =
Revised: 1 n n (cid:13)(cid:13)(cid:13)(cid:13) 2 (cid:35) =

Original: E   (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) ∇ f (cid:32)(cid:80) ni
Revised: E   (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) ∇ f (cid:32)(cid:80) ni

Original: =1 x ( k ) i n (cid:33) − (cid:80) ni
Revised: =1 x ( k ) i n (cid:33) − (cid:80) ni

Original: =1 ∇ f i ( z ( k ) i ) n (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) 2   =
Revised: =1 ∇ f i ( z ( k ) i ) n (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) 2   =

Original: E   (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) 
Revised: E   (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) 1 n n

Original: [Table]
Revised: [Table]

Revised: i ∇ f i

Revised: [Table]

Original: n
Revised: ni

Original: [Table]
Revised: [Table]

Original: i ∇ f i
Revised: x

Original: [Table]
Revised: [Table]

Original: n (cid:33) − (cid:80) 
Revised: n (cid:33) − (cid:80) ni

Original: [Table]
Revised: [Table]

Original:  f [Table] ( z
Revised: ∇ f i ( z

Original: [Table]
Revised: [Table]

Original: n (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
Revised: n (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)

Original: [Table]
Revised: [Table]

Original: 
Revised: 

Original:  =
Revised:  =

Original: E   (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:80) ni ∇ f
Revised: E   (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:80) ni ∇ f

Original: i
Revised: i

Original: [Table]
Revised: [Table]

Original: n
Revised: n

Original: [Table]
Revised: [Table]

Original: 
Revised: 

Original: [Table]
Revised: [Table]

Revised: E

Revised: [Table]

Revised: (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)

Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: Thus we have that:
Revised: Thus we have that:

Original: [Equation]
Revised: [Equation]

Original: By rearranging:
Revised: By rearranging:

Original: [Equation]
Revised: [Equation]

Original: Let us now sum from k = 0 to k
Revised: Let us now sum from k = 0 to k

Original: = K − 1 :
Revised: = K − 1 :

Original: [Equation]
Revised: [Equation]

Original: For the last inequality above, recall that with f ∗ we deﬁne the optimal solution of our problem.
Revised: For the last inequality above, recall that with f ∗ we deﬁne the optimal solution of our problem.

Original: Using the bound for the expression (cid:80) K − 1 k =

Revised: Using the bound for the expression (cid:80) K − 1 k =

Original: M k from Lemma 6 we obtain:
Revised: M k from Lemma 6 we obtain:

Original: [Equation]
Revised: [Equation]

Original: n
Revised: n

Original: [Equation]
Revised: [Equation]

Original: By rearranging and dividing all terms by γK we obtain:
Revised: By rearranging and dividing all terms by γK we obtain:

Original: [Equation]
Revised: [Equation]

Original: ni =
Revised: ni =

Revised: [Equation]

Revised: (cid:13)(cid:13)

Original: [Equation]
Revised: [Equation]

Original:  (0) (cid:13)(cid:13)
Revised: i (0) (cid:13)(cid:13)

Original: [Equation]
Revised: [Equation]

Original: n .
Revised: n .

Original: By deﬁning D 1 = (cid:104)
Revised: By deﬁning D 1 = (cid:104)

Original: [Equation]
Revised: [Equation]

Original: (cid:17)(cid:105) the proof is complete.
Revised: (cid:17)(cid:105) the proof is complete.

Original: C. P ROOFS OF M AIN T HEOREMS
Revised: C. P ROOFS OF M AIN T HEOREMS

Original: Having present all of the above Lemmas we are now ready to provide the proofs of main Theoremsand 2.
Revised: Having present all of the above Lemmas we are now ready to provide the proofs of main Theoremsand 2.

Original: C.3. P ROOF OF T HEOREM
Revised: C.3. P ROOF OF T HEOREM

Original: Let γ ≤ (1 − q )60 L 2 C 2 .
Revised: Let γ ≤ min (cid:26) [Equation]

Original: Then since q ∈ (0 , 1) , it holds that γ < 1 and as a result:
Revised: Then:

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: By substituting the above bounds into the result of Lemma 8 and by removing the second term of left hand side we obtain:
Revised: By substituting the above bounds into the result of Lemma 8 and by removing the second term of left hand side we obtain:

Original: [Equation]
Revised: [Equation]

Original: n
Revised: n

Original: [Equation]
Revised: [Equation]

Original: Let us now substitute in the above expression γ = (cid:112) nK .
Revised: Let us now substitute in the above expression γ = (cid:112) nK .

Original: This can be done due to the lower bound (see equation 6) on the total number of iterations K where guarantees that (cid:112) nK ≤ (1 − q ) 4 60 L 2 C 2 .
Revised: This can be done due to the lower bound (see equation 6) on the total number of iterations K where guarantees that (cid:112) nK ≤ min

Original: [Equation]
Revised: [Equation]

Original: Using again the assumption on the lower bound (6) of the total number of iterations K
Revised: Using again the assumption on the lower bound (6) of the total number of iterations K

Original: [Equation]
Revised: [Equation]

Original: the last two terms of the above expression are bounded by the ﬁrst term
Revised: the last two terms of the above expression are bounded by the ﬁrst term.

Original: [Equation]

Original: Thus,
Revised: Thus,

Original: [Equation]
Revised: [Equation]

Original: C.3. P ROOF OF T HEOREM
Revised: C.3. P ROOF OF T HEOREM

Original: Proof.
Revised: Proof.

Original: From Lemma 6 we have that:
Revised: From Lemma 6 we have that:

Original: [Equation]
Revised: [Equation]

Original: Using the assumptions of Theorem 1 and stepsize γ = (cid:112) nK :
Revised: Using the assumptions of Theorem 1 and stepsize γ = (cid:112) nK :

Original: [Equation]
Revised: [Equation]

Original: where the
Revised: where the

Original: Big
Revised: Big

Original: O notation swallows all constants of our setting (cid:16) 
Revised: O notation swallows all constants of our setting (cid:16) n [Table] =

Original: [Table]
Revised: [Table]

Revised: x

Revised: [Table]

Revised: [Table]

Revised: and f

Revised: [Table]

Original:  Now using the above upper bound equation 38 and result of Theorem 1 we obtain:
Revised: x [Table] − f ∗ (cid:17) Now using the above upper bound equation 38 and result of Theorem 1 we obtain:

Original: [Equation]
Revised: [Equation]

Original: where again the
Revised: where again the

Original: Big
Revised: Big

Original: O notation swallows all constants of our setting (cid:16) n, L,
Revised: O notation swallows all constants of our setting (cid:16) n, L,

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: and f ( x
Revised: and f ( x

Original: [Equation]
Revised: [Equation]

Original: − f ∗ (cid:17) .
Revised: − f ∗ (cid:17) .