Original: A2BCD : Asynchronous Acceleration with Optimal Complexity
Revised: A2BCD : Asynchronous Acceleration with Optimal Complexity

Original: Anonymous authors

Original: Paper under double-blind review

Original: Abstract
Revised: Abstract

Original: In this paper, we propose the A synchronous A ccelerated Nonuniform Randomized
Revised: In this paper, we propose the A synchronous A ccelerated Nonuniform Randomized

Original: B lock C oordinate D escent algorithm ( A2BCD ).
Revised: B lock C oordinate D escent algorithm ( A2BCD ).

Original: We prove A2BCD converges linearly to a solution of the convex minimization problem at the same rate as NU_ACDM , so long as the maximum delay is not too large.
Revised: We prove A2BCD converges linearly to a solution of the convex minimization problem at the same rate as NU_ACDM , so long as the maximum delay is not too large.

Original: This is the ﬁrst asynchronous
Revised: This is the ﬁrst asynchronous

Original: Nesterov-accelerated algorithm that attains any provable speedup.
Revised: Nesterov-accelerated algorithm that attains any provable speedup.

Original: Moreover, we then prove that these algorithms both have optimal complexity.
Revised: Moreover, we then prove that these algorithms both have optimal complexity.

Original: Asynchronous algorithms complete much faster iterations, and A2BCD has optimal complexity.
Revised: Asynchronous algorithms complete much faster iterations, and A2BCD has optimal complexity.

Original: Hence we observe in experiments that A2BCD is the top-performing coordinate descent algorithm, converging up to 4 − 5 × faster than NU_ACDM on some data sets in terms of wall-clock time.
Revised: Hence we observe in experiments that A2BCD is the top-performing coordinate descent algorithm, converging up to 4 − 5 × faster than NU_ACDM on some data sets in terms of wall-clock time.

Original: To motivate our theory and proof techniques, we also derive and analyze a continuous-time analogue of our algorithm and prove it converges at the same rate.
Revised: To motivate our theory and proof techniques, we also derive and analyze a continuous-time analogue of our algorithm and prove it converges at the same rate.

Original:  Introduction
Revised:  Introduction

Original: In this paper, we propose and prove the convergence of the A synchronous A ccelerated Nonuniform Randomized B lock C oordinate D escent algorithm ( A2BCD ), the ﬁrst asynchronous Nesterovaccelerated algorithm that achieves optimal complexity.
Revised: In this paper we propose and prove the convergence of the A synchronous A ccelerated Nonuniform Randomized B lock C oordinate D escent algorithm ( A2BCD ), the ﬁrst asynchronous Nesterovaccelerated algorithm that achieves optimal complexity.

Original: No previous attempts have been able to prove a speedup for asynchronous Nesterov acceleration.
Revised: No previous attempts have been able to prove a speedup for asynchronous Nesterov acceleration.

Original: We aim to ﬁnd the minimizer x ∗ of the unconstrained minimization problem:
Revised: We aim to ﬁnd the minimizer x ∗ of the unconstrained minimization problem:

Original: [Equation]
Revised: [Equation]

Original: (1.1) where f is σ -strongly convex for σ > 0 with L -Lipschitz gradient ∇ f = ( ∇ 1 f, . .
Revised: where f is σ -strongly convex for σ > 0 with L -Lipschitz gradient ∇ f = ( ∇ 1 f, . .

Original: , ∇ n f ).
Revised: , ∇ n f ).

Original: x ∈ R d is composed of coordinate blocks x (1) , . . .
Revised: x ∈ R d is composed of coordinate blocks x (1) , . . .

Original: , x ( n ) .
Revised: , x ( n ) .

Original: The coordinate blocks of the gradient ∇ i f are assumed
Revised: The coordinate blocks of the gradient ∇ i f are assumed

Original: L i -Lipschitz with respect to the i th block.
Revised: L i -Lipschitz with respect to the i th block.

Original: That is, ∀ x, 
Revised: That is, ∀ x, h ∈ R d :

Original: [Equation]
Revised: [Equation]

Original: where P i is the projection onto the i th block of R d .
Revised: where P i is the projection onto the i th block of R d .

Original: Let ¯ L (cid:44) 1 n P ni =1 L
Revised: Let ¯ L (cid:44) 1 n P ni =1 L

Original: i be the average block
Revised: i be the average block

Original: Lipschitz constant.
Revised: Lipschitz constant.

Original: These conditions on f are assumed throughout this whole paper.
Revised: These conditions on f are assumed throughout this whole paper.

Original: Our algorithm can also be applied to non-strongly convex objectives ( σ = 0) or non-smooth objectives using the black box reduction techniques proposed in Allen-Zhu & Hazan (2016).
Revised: Our algorithm can also be applied to non-strongly convex objectives ( σ = 0) or non-smooth objectives using the black box reduction techniques proposed in Allen-Zhu & Hazan (2016).

Original: Hence we consider only the coordinate smooth, strongly-convex case.
Revised: Hence we consider only the coordinate smooth, strongly-convex case.

Original: Our algorithm can also be applied to the convex regularized ERM problem via the standard dual transformation (see for instance Lin et al.
Revised: Our algorithm can also be applied to the convex regularized ERM problem via the standard dual transformation (see for instance Lin et al.

Original: [Equation]
Revised: [Equation]

Original: Hence A2BCD can be used as an asynchronous Nesterov-accelerated ﬁnite-sum algorithm.
Revised: Hence A2BCD can be used as an asynchronous Nesterov-accelerated ﬁnite-sum algorithm.

Original: Coordinate descent methods, in which a chosen coordinate block i k is updated at every iteration, are a popular way to solve equation 1.1.
Revised: Coordinate descent methods, in which a chosen coordinate block i k is updated at every iteration, are a popular way to solve equation 1.1.

Original: Randomized block coordinate descent ( RBCD , Nesterov (2012)) updates a uniformly randomly chosen coordinate block
Revised: Randomized block coordinate descent ( RBCD , Nesterov (2012)) updates a uniformly randomly chosen coordinate block

Original: i k with a gradient-descent-like step: x k +1 = x k − (1 /L i k ) ∇ i
Revised: i k with a gradient-descent-like step: x k +1 = x k − (1 /L i k ) ∇ i

Original: k f ( x k ).
Revised: k f ( x k ).

Original: The complexity K ( (cid:15) ) of an algorithm is deﬁned as the number of iterations required to decrease the error E ( f ( x k ) − f ( x ∗ )) to less than (cid:15) ( f ( x 0 ) − f ( x ∗ )).
Revised: The complexity K ( (cid:15) ) of an algorithm is deﬁned as the number of iterations required to decrease the error E ( f ( x k ) − f ( x ∗ )) to less than (cid:15) ( f ( x 0 ) − f ( x ∗ )).

Original: Randomized coordinate descent has a complexity of K ( (cid:15) )
Revised: Randomized coordinate descent has a complexity of K ( (cid:15) )

Original: =
Revised: =

Original: O ( n (¯ L/σ ) ln(1 /(cid:15) )).
Revised: O ( n (¯ L/σ ) ln(1 /(cid:15) )).

Original: Using a series of averaging and extrapolation steps, accelerated RBCD Nesterov (2012) improves
Revised: Using a series of averaging and extrapolation steps, accelerated RBCD Nesterov (2012) improves

Original: RBCD ’s iteration complexity K ( (cid:15) ) to O ( n q ¯ L/σ ln(1 /(cid:15) )), which leads to much faster convergence when ¯ Lσ is large.
Revised: RBCD ’s iteration complexity K ( (cid:15) ) to O ( n q ¯ L/σ ln (1 /(cid:15) )), which leads to much faster convergence when ¯ Lσ is large.

Original: This rate is optimal when all L i are equal Lan & Zhou (2015).
Revised: This rate is optimal when all L i are equal Lan & Zhou (2015).

Original: Finally, using a special probability distribution for the random block index i k , the non-uniform accelerated coordinate descent method Allen-Zhu et al.
Revised: Finally, using a special probability distribution for the random block index i k , the non-uniform accelerated coordinate descent method Allen-Zhu et al.

Original: (2015) ( NU_ACDM ) can further decrease the complexity to O
Revised: (2015) ( NU_ACDM ) can further decrease the complexity to O

Original: ( P ni =1 p L
Revised: ( P ni =1 p L

Original: i /σ ln(1 /(cid:15) )), which can be up to √ n times faster than accelerated RBCD , since some
Revised: i /σ ln(1 /(cid:15) )), which can be up to √ n times faster than accelerated RBCD , since some

Original: L i can be signiﬁcantly smaller than L .
Revised: L i can be signiﬁcantly smaller than L .

Original: NU_ACDM is the current state-of-the-art coordinate descent algorithm for solving equation 1.1.
Revised: NU_ACDM is the current state-of-the-art coordinate descent algorithm for solving equation 1.1.

Original: Our A2BCD algorithm generalizes NU_ACDM to the asynchronous-parallel case.
Revised: Our A2BCD algorithm generalizes NU_ACDM to the asynchronous-parallel case.

Original: We solve equation 1.
Revised: We solve equation 1.

Original: with a collection of p computing nodes that continually read a shared-access solution vector y into local memory then compute a block gradient ∇ i f , which is used to update shared solution vectors ( x, y, v ).
Revised: with a collection of p computing nodes that continually read a shared-access solution vector y into local memory then compute a block gradient ∇ i f , which is used to update shared solution vectors ( x, y, v ).

Original: Proving convergence in the asynchronous case requires extensive new technical machinery.
Revised: Proving convergence in the asynchronous case requires extensive new technical machinery.

Original: A traditional synchronous-parallel implementation is organized into rounds of computation: Every computing node must complete an update in order for the next iteration to begin.
Revised: A traditional synchronous-parallel implementation is organized into rounds of computation: Every computing node must complete an update in order for the next iteration to begin.

Original: However, this synchronization process can be extremely costly, since the lateness of a single node can halt the entire system.
Revised: However, this synchronization process can be extremely costly, since the lateness of a single node can halt the entire system.

Original: This becomes increasingly problematic with scale, as diﬀerences in node computing speeds, load balancing, random network delays, and bandwidth constraints mean that a synchronous-parallel solver may spend more time waiting than computing a solution.
Revised: This becomes increasingly problematic with scale, as diﬀerences in node computing speeds, load balancing, random network delays, and bandwidth constraints mean that a synchronous-parallel solver may spend more time waiting than computing a solution.

Original: Computing nodes in an asynchronous solver do not wait for others to complete and share their updates before starting the next iteration.
Revised: Computing nodes in an asynchronous solver do not wait for others to complete and share their updates before starting the next iteration.

Original: They simply continue to update the solution vectors with the most recent information available, without any central coordination.
Revised: They simply continue to update the solution vectors with the most recent information available, without any central coordination.

Original: This eliminates costly idle time, meaning that asynchronous algorithms can be much faster than traditional ones, since they have much faster iterations.
Revised: This eliminates costly idle time, meaning that asynchronous algorithms can be much faster than traditional ones, since they have much faster iterations.

Original: For instance, random network delays cause asynchronous algorithms to complete iterations Ω(ln( p )) time faster than synchronous algorithms at scale.
Revised: For instance, random network delays cause asynchronous algorithms to complete iterations Ω(ln( p )) time faster than synchronous algorithms at scale.

Original: This and other factors that inﬂuence the speed of iterations are discussed in Hannah & Yin (2017a).
Revised: This and other factors that inﬂuence the speed of iterations are discussed in Hannah & Yin (2017a).

Original: However, since many iterations may occur between the time that a node reads the solution vector, and the time that its computed update is applied, eﬀectively the solution vector is being updated with outdated information.
Revised: However, since many iterations may occur between the time that a node reads the solution vector, and the time that its computed update is applied, eﬀectively the solution vector is being updated with outdated information.

Original: At iteration k , the block gradient ∇ i k f is computed at a delayed iterate ˆ y k deﬁned as 1 :
Revised: At iteration k , the block gradient ∇ i k f is computed at a delayed iterate ˆ y k deﬁned as 1 :

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: [Equation]

Original: ) for delay parameters j ( k, 1) , . . .,
Revised: for delay parameters j ( k, 1) , . . .

Original:  j ( k, n ) ∈ N .
Revised: , j ( k, n ) ∈ N .

Original: Here j ( k, i ) denotes how many iterations out of date coordinate block i is at iteration k .
Revised: Here j ( k, i ) denotes how many iterations out of date coordinate block i is at iteration k .

Original: Diﬀerent blocks may be out of date by diﬀerent amounts, which is known as an inconsistent read .
Revised: Diﬀerent blocks may be out of date by diﬀerent amounts, which is known as an inconsistent read .

Original: We assume 2 that j ( k, i ) ≤ τ for some constant τ < ∞ .
Revised: We assume 2 that j ( k, i ) ≤ τ for some constant τ < ∞ .

Original: Asynchronous algorithms were proposed in Chazan & Miranker (1969) to solve linear systems.
Revised: Asynchronous algorithms were proposed in Chazan & Miranker (1969) to solve linear systems.

Original: General convergence results and theory were developed later in Bertsekas (1983);
Revised: General convergence results and theory were developed later in Bertsekas (1983);

Original: Bertsekas &
Revised: Bertsekas &

Original: Tsitsiklis (1997); Tseng et al.
Revised: Tsitsiklis (1997); Tseng et al.

Original: Luo & Tseng (1992; 1993); Tseng (1991) for partially and totally asynchronous systems, with essentially-cyclic block sequence i k .
Revised: Luo & Tseng (1992; 1993); Tseng (1991) for partially and totally asynchronous systems, with essentially-cyclic block sequence i k .

Original: More recently, there has been renewed interest in asynchronous algorithms with random block coordinate updates.
Revised: More recently, there has been renewed interest in asynchronous algorithms with random block coordinate updates.

Original: Linear and sublinear convergence results were proven for asynchronous RBCD Liu & Wright (2015);
Revised: Linear and sublinear convergence results were proven for asynchronous RBCD Liu & Wright (2015);

Original: Liu et al.
Revised: Liu et al.

Original: ; Avron et al.
Revised: ; Avron et al.

Original: (2014), and similar was proven for asynchronous SGD Recht et al.
Revised: (2014), and similar was proven for asynchronous SGD Recht et al.

Original: (2011), and variance reduction algorithms
Revised: (2011), and variance reduction algorithms

Original: Reddi et al.
Revised: Reddi et al.

Original: ; Leblond et al.
Revised: ; Leblond et al.

Original: ; Mania et al.
Revised: ; Mania et al.

Original: Huo & Huang (2016), and primal-dual algorithms Combettes & Eckstein (2018).
Revised: Huo & Huang (2016), and primal-dual algorithms Combettes & Eckstein (2018).

Original: There is also a rich body of work on asynchronous SGD.
Revised: There is also a rich body of work on asynchronous SGD.

Original: In the distributed setting, Zhou et al.
Revised: In the distributed setting, Zhou et al.

Original: (2018) showed global convergence for stochastic variationally coherent problems even when the delays grow at a polynomial rate.
Revised: (2018) showed global convergence for stochastic variationally coherent problems even when the delays grow at a polynomial rate.

Original: In Lian et al.
Revised: In Lian et al.

Original: (2018), an asynchronous decentralized SGD was proposed with the same optimal sublinear convergence rate as SGD and linear speedup with respect to the number of workers.
Revised: (2018), an asynchronous decentralized SGD was proposed with the same optimal sublinear convergence rate as SGD and linear speedup with respect to the number of workers.

Original: In Liu et al.
Revised: In Liu et al.

Original: (2018), authors obtained an asymptotic rate of convergence for asynchronous momentum SGD on streaming PCA, which provides insight into the tradeoﬀ between asynchrony and momentum.
Revised: (2018), authors obtained an asymptotic rate of convergence for asynchronous momentum SGD on streaming PCA, which provides insight into the tradeoﬀ between asynchrony and momentum.

Original: In Dutta et al.
Revised: In Dutta et al.

Original: , authors prove convergence results for asynchronous SGD that highlight the tradeoﬀ between faster iterations and iteration complexity.
Revised: , authors prove convergence results for asynchronous SGD that highlight the tradeoﬀ between faster iterations and iteration complexity.

Original: Further related work is discussed in Section 4.
Revised: Further related work is discussed in Section 4.

Original: 1. Summary of Contributions
Revised: 1. Summary of Contributions

Original: In this paper, we prove that A2BCD attains NU_ACDM ’s state-of-the-art iteration complexity to highest order for solving equation 1.1, so long as delays are not too large (see Section 2).
Revised: In this paper, we prove that A2BCD attains NU_ACDM ’s state-of-the-art iteration complexity to highest order for solving equation 1.1, so long as delays are not too large (see Section 2).

Original: The proof is very diﬀerent from that of Allen-Zhu et al.
Revised: The proof is very diﬀerent from that of Allen-Zhu et al.

Original: and involves signiﬁcant technical innovations and complexity related to the analysis of asynchronicity.
Revised: and involves signiﬁcant technical innovations and complexity related to the analysis of asynchronicity.

Original: We also prove that A2BCD (and hence NU_ACDM ) has optimal complexity to within a constant factor over a fairly general class of randomized block coordinate descent algorithms (see Section 2.1).
Revised: We also prove that A2BCD (and hence NU_ACDM ) has optimal complexity to within a constant factor over a fairly general class of randomized block coordinate descent algorithms (see Section 2.1).

Original: This extends results in Lan & Zhou (2015) to asynchronous algorithms with L i not all equal.
Revised: This extends results in Lan & Zhou (2015) to asynchronous algorithms with L i not all equal.

Original: Since asynchronous algorithms complete faster iterations, and A2BCD has optimal complexity, we expect
Revised: Since asynchronous algorithms complete faster iterations, and A2BCD has optimal complexity, we expect

Original: A2BCD to be faster than all existing coordinate descent algorithms.
Revised: A2BCD to be faster than all existing coordinate descent algorithms.

Original: We conﬁrm with numerical experiments that A2BCD is the current fastest coordinate descent algorithm (see Section 5).
Revised: We conﬁrm with numerical experiments that A2BCD is the current fastest coordinate descent algorithm (see Section 5).

Original: We are only aware of one previous and one contemporaneous attempt at proving convergence results for asynchronous Nesterov-accelerated algorithms.
Revised: We are only aware of one previous and one contemporaneous attempt at proving convergence results for asynchronous Nesterov-accelerated algorithms.

Original: However, the ﬁrst is not accelerated and relies on extreme assumptions, and the second obtains no speedup.
Revised: However, the ﬁrst is not accelerated and relies on extreme assumptions, and the second obtains no speedup.

Original: Therefore, we claim that our results are the ﬁrst-ever analysis of asynchronous Nesterov-accelerated algorithms that attains a speedup.
Revised: Therefore, we claim that our results are the ﬁrst-ever analysis of asynchronous Nesterov-accelerated algorithms that attains a speedup.

Original: Moreover, our speedup is optimal for delays not too large 3 .
Revised: Moreover, our speedup is optimal for delays not too large 3 .

Original: The work of Meng et al.
Revised: The work of Meng et al.

Original: claims to obtain square-root speedup for an asynchronous accelerated SVRG.
Revised: claims to obtain square-root speedup for an asynchronous accelerated SVRG.

Original: In the case where all component functions have the same Lipschitz constant L , the complexity they obtain reduces to ( n + κ ) ln(1 /(cid:15) ) for κ = O (cid:0) τn 2 (cid:1) (Corollary 4.4).
Revised: In the case where all component functions have the same Lipschitz constant L , the complexity they obtain reduces to ( n + κ ) ln(1 /(cid:15) ) for κ = O (cid:0) τn 2 (cid:1) (Corollary 4.4).

Original: Hence authors do not even obtain accelerated rates.
Revised: Hence authors do not even obtain accelerated rates.

Original: Their convergence condition is τ < 1 4∆ 1 / 8 for sparsity parameter ∆.
Revised: Their convergence condition is τ < 1 4∆ 1 / 8 for sparsity parameter ∆.

Original: Since the dimension d satisﬁes d ≥ 1∆ , they require d ≥ 2 16 τ 8 .
Revised: Since the dimension d satisﬁes d ≥ 1∆ , they require d ≥ 2 16 τ 8 .

Original: So τ = 20 requires dimension d > 10 15 .
Revised: So τ = 20 requires dimension d > 10 15 .

Original: In a contemporaneous preprint, authors in Fang et al.
Revised: In a contemporaneous preprint, authors in Fang et al.

Original: (2018) skillfully devised accelerated schemes for asynchronous coordinate descent and SVRG using momentum compensation techniques.
Revised: (2018) skillfully devised accelerated schemes for asynchronous coordinate descent and SVRG using momentum compensation techniques.

Original: Although their complexity results have the improved √ κ dependence on the condition number, they do not prove any speedup.
Revised: Although their complexity results have the improved √ κ dependence on the condition number, they do not prove any speedup.

Original: Their complexity is τ times larger than the serial complexity.
Revised: Their complexity is τ times larger than the serial complexity.

Original: Since τ is necessarily greater than p , their results imply that adding more computing nodes will increase running time.
Revised: Since τ is necessarily greater than p , their results imply that adding more computing nodes will increase running time.

Original: The authors claim that they can extend their results to linear speedup for asynchronous, accelerated SVRG under sparsity assumptions.
Revised: The authors claim that they can extend their results to linear speedup for asynchronous, accelerated SVRG under sparsity assumptions.

Original: And while we think this is quite likely, they have not yet provided proof.
Revised: And while we think this is quite likely, they have not yet provided proof.

Original: We also derive a second-order ordinary diﬀerential equation (ODE), which is the continuous-time limit of A2BCD (see Section 3).
Revised: We also derive a second-order ordinary diﬀerential equation (ODE), which is the continuous-time limit of A2BCD (see Section 3).

Original: This extends the ODE found in Su et al.
Revised: This extends the ODE found in Su et al.

Original: (2014) to an asynchronous accelerated algorithm minimizing a strongly convex function.
Revised: (2014) to an asynchronous accelerated algorithm minimizing a strongly convex function.

Original: We prove this ODE linearly converges to a solution with the same rate as A2BCD ’s, without needing to resort to the restarting techniques.
Revised: We prove this ODE linearly converges to a solution with the same rate as A2BCD ’s, without needing to resort to the restarting techniques.

Original: The ODE analysis motivates and clariﬁes the our proof strategy of the main result.
Revised: The ODE analysis motivates and clariﬁes the our proof strategy of the main result.

Original:  Main results
Revised:  Main results

Original: We should consider functions f where it is eﬃcient to calculate blocks of the gradient, so that coordinate-wise parallelization is eﬃcient.
Revised: We should consider functions f where it is eﬃcient to calculate blocks of the gradient, so that coordinate-wise parallelization is eﬃcient.

Original: That is, the function should be “coordinate friendly”
Revised: That is, the function should be “coordinate friendly”

Original: Peng et al.
Revised: Peng et al.

Original: (2016b).
Revised: (2016b).

Original: This is a very wide class that includes regularized linear regression, logistic regression, etc.
Revised: This is a very wide class that includes regularized linear regression, logistic regression, etc.

Original: The L 2 -regularized empirical risk minimization problem is not coordinate friendly in general, however the equivalent dual problem is, and hence can be solved eﬃciently by A2BCD (see
Revised: The L 2 -regularized empirical risk minimization problem is not coordinate friendly in general, however the equivalent dual problem is, and hence can be solved eﬃciently by A2BCD (see

Original: Lin et al.
Revised: Lin et al.

Original: (2014), and Section 5).
Revised: (2014), and Section 5).

Original: To calculate the k + 1’th iteration of the algorithm from iteration k , we use only one block of the gradient ∇ i k f .
Revised: To calculate the k + 1’th iteration of the algorithm from iteration k , we use only one block of the gradient ∇ i k f .

Original: We assume that the delays j ( k, i ) are independent of the block sequence i k , but otherwise arbitrary (This is a standard assumption found in the vast majority of papers, but can be relaxed Sun et al.
Revised: We assume that the delays j ( k, i ) are independent of the block sequence i k , but otherwise arbitrary (This is a standard assumption found in the vast majority of papers, but can be relaxed Sun et al.

Original: ; Leblond et al.
Revised: ; Leblond et al.

Original: ; Cannelli et al.
Revised: ; Cannelli et al.

Original: Deﬁnition 1.
Revised: Deﬁnition 1.

Original: Asynchronous Accelerated Randomized Block Coordinate Descent ( A2BCD ).
Revised: Asynchronous Accelerated Randomized Block Coordinate Descent ( A2BCD ).

Original: Let f be σ -strongly convex, and let its gradient ∇ f be L -Lipschitz with block coordinate Lipschitz parameters
Revised: Let f be σ -strongly convex, and let its gradient ∇ f be L -Lipschitz with block coordinate Lipschitz parameters

Original: L i as in equation 1.2.
Revised: L i as in equation 1.2.

Original: We deﬁne the condition number κ = L/σ , and let L = min i L i .
Revised: We deﬁne the condition number κ = L/σ , and let L = min i L i .

Original: Using these parameters, we sample i k in an independent and identically distributed (IID) fashion according to
Revised: Using these parameters, we sample i k in an independent and identically distributed (IID) fashion according to

Original: [Equation]
Revised: [Equation]

Original: Let τ be the maximum asynchronous delay.
Revised: Let τ be the maximum asynchronous delay.

Original: We deﬁne the dimensionless asynchronicity parameter ψ , which is proportional to τ , and quantiﬁes how strongly asynchronicity will aﬀect convergence:
Revised: We deﬁne the dimensionless asynchronicity parameter ψ , which is proportional to τ , and quantiﬁes how strongly asynchronicity will aﬀect convergence:

Original: [Equation]
Revised: [Equation]

Original: We use the above system parameters and ψ to deﬁne the coeﬃcients α, β , and γ via eqs.
Revised: We use the above system parameters and ψ to deﬁne the coeﬃcients α, β , and γ via eqs.

Original: (2.3) to (2.5).
Revised: (2.3) to (2.5).

Original: Hence A2BCD algorithm is deﬁned via the iterations: eqs.
Revised: Hence A2BCD algorithm is deﬁned via the iterations: eqs.

Original: (2.6) to (2.8).
Revised: (2.6) to (2.8).

Original: [Equation]
Revised: [Equation]

Original: See Section A for a discussion of why it is practical and natural to have the gradient ∇ i k f
Revised: See Section A for a discussion of why it is practical and natural to have the gradient ∇ i k f

Original: (ˆ y k ) to be outdated, while the actual variables x k , y k , v k can be eﬃciently kept up to date.
Revised: (ˆ y k ) to be outdated, while the actual variables x k , y k , v k can be eﬃciently kept up to date.

Original: Essentially it is because most of the computation lies in computing ∇ i k f
Revised: Essentially it is because most of the computation lies in computing ∇ i k f

Original: (ˆ y k ).
Revised: (ˆ y k ).

Original: After this is computed, x k , y k , v k can be updated more-or-less atomically with minimal overhead, meaning that they will always be up to date.
Revised: After this is computed, x k , y k , v k can be updated more-or-less atomically with minimal overhead, meaning that they will always be up to date.

Original: However our main results still hold for more general asynchronicity.
Revised: However our main results still hold for more general asynchronicity.

Original: A natural quantity to consider in asynchronous convergence analysis is the asynchronicity error , a powerful tool for analyzing asynchronous algorithms used in several recent works Peng et al.
Revised: A natural quantity to consider in asynchronous convergence analysis is the asynchronicity error , a powerful tool for analyzing asynchronous algorithms used in several recent works Peng et al.

Original: (2016a)
Revised: (2016a)

Original: ; Hannah & Yin (2017b); Sun et al.
Revised: ; Hannah & Yin (2017b); Sun et al.

Original: ; Hannah & Yin (2017a).
Revised: ; Hannah & Yin (2017a).

Original: We adapt it and use a weighted sum of the history of the algorithm with decreasing weight as you go further back in time.
Revised: We adapt it and use a weighted sum of the history of the algorithm with decreasing weight as you go further back in time.

Original: Deﬁnition 2.
Revised: Deﬁnition 2.

Original: Asynchronicity error.
Revised: Asynchronicity error.

Original: Using the above parameters, we deﬁne:
Revised: Using the above parameters, we deﬁne:

Original: [Equation]
Revised: [Equation]

Original: Here we deﬁne y k
Revised: Here we deﬁne y k

Original: = y 

Revised: = y

Original: for all k < 0.
Revised: for all k < 0.

Original: The determination of the coeﬃcients c i is in general a very involved process of trial and error, intuition, and balancing competing requirements.
Revised: The determination of the coeﬃcients c i is in general a very involved process of trial and error, intuition, and balancing competing requirements.

Original: The algorithm doesn’t depend on the coeﬃcients, however; they are only an analytical tool.
Revised: The algorithm doesn’t depend on the coeﬃcients, however; they are only an analytical tool.

Original: We deﬁne E k
Revised: We deﬁne E k

Original: [ X ] as the expectation of X conditioned on ( x 0 , . . .
Revised: [ X ] as the expectation of X conditioned on ( x 0 , . . .

Original: , x k ), ( y 0 , . . .
Revised: , x k ), ( y 0 , . . .

Original: , y k ), ( v 0 , . . , v k ), and ( i 0 , . .
Revised: , y k ), ( v 0 , . . , v k ), and ( i 0 , . .

Original: , i k − 1 ).
Revised: , i k − 1 ).

Original: To simplify notation 4 , we assume that the minimizer x ∗ = 0, and that f ( x ∗ ) = 0 with no loss in generality.
Revised: To simplify notation 4 , we assume that the minimizer x ∗ = 0, and that f ( x ∗ ) = 0 with no loss in generality.

Original: We deﬁne the Lyapunov function :
Revised: We deﬁne the Lyapunov function :

Original: [Equation]
Revised: [Equation]

Original: We now present this paper’s ﬁrst main contribution.
Revised: We now present this paper’s ﬁrst main contribution.

Original: Theorem 1.
Revised: Theorem 1.

Original: Let f be σ -strongly convex with a gradient ∇ f
Revised: Let f be σ -strongly convex with a gradient ∇ f

Original: that is L -Lipschitz with block Lipschitz constants { L i } ni =1 .
Revised: that is L -Lipschitz with block Lipschitz constants { L i } ni =1 .

Original: Let ψ deﬁned in equation 2.2 satisfy ψ ≤ 37 (i.e. τ ≤ 121 S 1 / 2 L 1 / 2 L − 3 / 4 κ − 1 / 4 ).
Revised: Let ψ deﬁned in equation 2.2 satisfy ψ ≤ 37 (i.e. τ ≤ 121 S 1 / 2 L 1 / 2 L − 3 / 4 κ − 1 / 4 ).

Original: Then for A2BCD we have:
Revised: Then for A2BCD we have:

Original: [Equation]
Revised: [Equation]

Original: To obtain E [ ρ k ] ≤ (cid:15)ρ 0 , it takes K A2BCD ( (cid:15) ) iterations for:
Revised: To obtain E [ ρ k ] ≤ (cid:15)ρ 0 , it takes K A2BCD ( (cid:15) ) iterations for:

Original: [Equation]
Revised: [Equation]

Original: where O ( · ) is asymptotic with respect to σ − 1 / 2 S → ∞ , and uniformly bounded.
Revised: where O ( · ) is asymptotic with respect to σ − 1 / 2 S → ∞ , and uniformly bounded.

Original: This result is proven in Section B. A stronger result for L i ≡ L can be proven, but this adds to the complexity of the proof; see Section E for a discussion.
Revised: This result is proven in Section B. A stronger result for L i ≡ L can be proven, but this adds to the complexity of the proof; see Section E for a discussion.

Original: In practice, asynchronous algorithms are far more resilient to delays than the theory predicts.
Revised: In practice, asynchronous algorithms are far more resilient to delays than the theory predicts.

Original: τ can be much larger without negatively aﬀecting the convergence rate and complexity.
Revised: τ can be much larger without negatively aﬀecting the convergence rate and complexity.

Original: This is perhaps because we are limited to a worst-case analysis, which is not representative of the average-case performance.
Revised: This is perhaps because we are limited to a worst-case analysis, which is not representative of the average-case performance.

Original: Allen-Zhu et al.
Revised: Allen-Zhu et al.

Original: (Theorem 5.1) shows a linear convergence rate of 1 − 2 / (cid:0) 1 + 2 σ − 1 / 2 S (cid:1) for NU_ACDM , which leads to the corresponding iteration complexity of K NU_ACDM ( (cid:15) )
Revised: (Theorem 5.1) shows a linear convergence rate of 1 − 2 / (cid:0) 1 + 2 σ − 1 / 2 S (cid:1) for NU_ACDM , which leads to the corresponding iteration complexity of K NU_ACDM ( (cid:15) )

Original: = (cid:0) σ − 1 / 2 S + O (1) (cid:1) ln(1 /(cid:15) ).
Revised: = (cid:0) σ − 1 / 2 S + O (1) (cid:1) ln(1 /(cid:15) ).

Original: Hence, we have:
Revised: Hence, we have:

Original: [Equation]
Revised: [Equation]

Original: When 0 ≤ ψ (cid:28) 1, or equivalently, when τ (cid:28) S 1 / 2 L 1 / 2 L − 3 / 4 κ − 1 / 4 , the complexity of A2BCD asymptotically matches that of NU_ACDM .
Revised: When 0 ≤ ψ (cid:28) 1, or equivalently, when τ (cid:28) S 1 / 2 L 1 / 2 L − 3 / 4 κ − 1 / 4 , the complexity of A2BCD asymptotically matches that of NU_ACDM .

Original: Hence A2BCD combines state-of-the-art complexity with the faster iterations and superior scaling that asynchronous iterations allow.
Revised: Hence A2BCD combines state-of-the-art complexity with the faster iterations and superior scaling that asynchronous iterations allow.

Original: We now present some special cases of the conditions on the maximum delay τ required for good complexity.
Revised: We now present some special cases of the conditions on the maximum delay τ required for good complexity.

Original: Corollary 3.
Revised: Corollary 3.

Original: Let the conditions of Theorem 1 hold.
Revised: Let the conditions of Theorem 1 hold.

Original: If all coordinate-wise Lipschitz constants L i are equal (i.e. L i =
Revised: If all coordinate-wise Lipschitz constants L i are equal (i.e. L i =

Original: L 1 , ∀ i ), then we have K A2BCD ( (cid:15) ) ∼ K NU_ACDM ( (cid:15) ) when τ (cid:28) n 1 / 2 κ − 1 / 

Revised: L 1 , ∀ i ), then we have K A2BCD ( (cid:15) ) ∼ K NU_ACDM ( (cid:15) ) when τ (cid:28) n 1 / 2 κ − 1 /

Original: ( L 1 /L ) 3 / 4 .
Revised: ( L 1 /L ) 3 / 4 .

Original: If we further assume all coordinate-wise Lipschitz constants L i equal L .
Revised: If we further assume all coordinate-wise Lipschitz constants L i equal L .

Original: Then K A2BCD ( (cid:15) ) ∼ K NU_ACDM ( (cid:15) ) =
Revised: Then K A2BCD ( (cid:15) ) ∼ K NU_ACDM ( (cid:15) ) =

Original: K ACDM ( (cid:15) ) , when τ (cid:28) n 1 / 2 κ − 1 / 4 .
Revised: K ACDM ( (cid:15) ) , when τ (cid:28) n 1 / 2 κ − 1 / 4 .

Original: Remark 1.
Revised: Remark 1.

Original: Reduction to synchronous case.
Revised: Reduction to synchronous case.

Original: Notice that when τ = 0, we have ψ = 0,
Revised: Notice that when τ = 0, we have ψ = 0,

Original: c i ≡and hence
Revised: c i ≡and hence

Original: A k ≡ 0.
Revised: A k ≡ 0.

Original: Thus A2BCD becomes equivalent to NU_ACDM , the Lyapunov function 5 ρ k becomes equivalent to one found in Allen-Zhu et al.
Revised: Thus A2BCD becomes equivalent to NU_ACDM , the Lyapunov function 5 ρ k becomes equivalent to one found in Allen-Zhu et al.

Original: (2015)(pg. 9), and Theorem 1 yields the same complexity.
Revised: (2015)(pg. 9), and Theorem 1 yields the same complexity.

Original: The maximum delay τ will be a function τ ( p ) of p , number of computing nodes.
Revised: The maximum delay τ will be a function τ ( p ) of p , number of computing nodes.

Original: Clearly τ ≥ p , and experimentally it has been observed that τ = O ( p )
Revised: Clearly τ ≥ p , and experimentally it has been observed that τ = O ( p )

Original: Leblond et al.
Revised: Leblond et al.

Original: Let gradient complexity
Revised: Let gradient complexity

Original: K ( (cid:15), τ ) be the number of gradients required for an asynchronous algorithm with maximum delay τ to attain suboptimality (cid:15) .
Revised: K ( (cid:15), τ ) be the number of gradients required for an asynchronous algorithm with maximum delay τ to attain suboptimality (cid:15) .

Original: τ (1) = 0, since with only 1 computing node there can be no delay.
Revised: τ (1) = 0, since with only 1 computing node there can be no delay.

Original: This corresponds to the serial complexity.
Revised: This corresponds to the serial complexity.

Original: We say that an asynchronous algorithm attains a complexity speedup if pK ( (cid:15),τ (0))
Revised: We say that an asynchronous algorithm attains a complexity speedup if pK ( (cid:15),τ (0))

Original: K ( (cid:15),τ ( p ) is increasing in p .
Revised: K ( (cid:15),τ ( p ) is increasing in p .

Original: We say it attains linear complexity speedup if pK ( (cid:15),τ (0))
Revised: We say it attains linear complexity speedup if pK ( (cid:15),τ (0))

Original: K ( (cid:15),τ ( p ) =
Revised: K ( (cid:15),τ ( p ) =

Original: Ω( p ).
Revised: Ω( p ).

Original: In Theorem 1, we obtain a linear complexity speedup (for p not too large), whereas no other prior attempt can attain even a complexity speedup with Nesterov acceleration.
Revised: In Theorem 1, we obtain a linear complexity speedup (for p not too large), whereas no other prior attempt can attain even a complexity speedup with Nesterov acceleration.

Original: In the ideal scenario where the rate at which gradients are calculated increases linearly with p , algorithms that have linear complexity speedup will have a linear decrease in wall-clock time.
Revised: In the ideal scenario where the rate at which gradients are calculated increases linearly with p , algorithms that have linear complexity speedup will have a linear decrease in wall-clock time.

Original: However in practice, when the number of computing nodes is suﬃciently large, the rate at which gradients are calculated will no longer be linear.
Revised: However in practice, when the number of computing nodes is suﬃciently large, the rate at which gradients are calculated will no longer be linear.

Original: This is due to many parallel overhead factors including too many nodes sharing the same memory read/write bandwidth, and network bandwidth.
Revised: This is due to many parallel overhead factors including too many nodes sharing the same memory read/write bandwidth, and network bandwidth.

Original: However we note that even with these issues, we obtain much faster convergence than the synchronous counterpart experimentally.
Revised: However we note that even with these issues, we obtain much faster convergence than the synchronous counterpart experimentally.

Original: 2. Optimality
Revised: 2. Optimality

Original: NU_ACDM and hence A2BCD are in fact optimal in some sense.
Revised: NU_ACDM and hence A2BCD are in fact optimal in some sense.

Original: That is, among a fairly wide class of coordinate descent algorithms A , they have the best-possible worst-case complexity to highest order.
Revised: That is, among a fairly wide class of coordinate descent algorithms A , they have the best-possible worst-case complexity to highest order.

Original: We extend the work in Lan & Zhou (2015) to encompass algorithms are asynchronous and have unequal L i .
Revised: We extend the work in Lan & Zhou (2015) to encompass algorithms are asynchronous and have unequal L i .

Original: For a subset S ∈ R d , we let IC( S ) (inconsistent read) denote the set of vectors v whose components are a combination of components of vectors in the set S .
Revised: For a subset S ∈ R d , we let IC( S ) (inconsistent read) denote the set of vectors v whose components are a combination of components of vectors in the set S .

Original: That is, v = ( v 1 , 1 , v 2 , 2 , . .
Revised: That is, v = ( v 1 , 1 , v 2 , 2 , . .

Original: . , v d,d ) for some vectors v 1 , v 2 , . .
Revised: . , v d,d ) for some vectors v 1 , v 2 , . .

Original: . , v d ∈ S .
Revised: . , v d ∈ S .

Original: Here v i,j denotes the j th component of vector v i .
Revised: Here v i,j denotes the j th component of vector v i .

Original: Deﬁnition 4.
Revised: Deﬁnition 4.

Original: Asynchronous Randomized Incremental Algorithms.
Revised: Asynchronous Randomized Incremental Algorithms.

Original: Consider the unconstrained minimization problem equation 1.1 for function f satisfying the conditions stated in Section 1.
Revised: Consider the unconstrained minimization problem equation 1.1 for function f satisfying the conditions stated in Section 1.

Original: We deﬁne the class A as algorithms G on this problem such that:
Revised: We deﬁne the class A as algorithms G on this problem such that:

Original: For each parameter set ( σ, L 1 , . . .
Revised: For each parameter set ( σ, L 1 , . . .

Original: , L n , n ), G has an associated IID random variable
Revised: , L n , n ), G has an associated IID random variable

Original: i k
Revised: i k

Original: with some ﬁxed distribution P [ i k ] = p i for P ni =1 p i = 1. 2.
Revised: with some ﬁxed distribution P [ i k ] = p i for P ni =1 p i = 1. 2.

Original: The iterates of A satisfy: x k +1 ∈ span { IC( X k ) , ∇ i 0 f
Revised: The iterates of A satisfy: x k +1 ∈ span { IC( X k ) , ∇ i 0 f

Original: (IC( X 0 )) , ∇ i 1 f (IC( X 1 )) , . .
Revised: (IC( X 0 )) , ∇ i 1 f (IC( X 1 )) , . .

Original: . , ∇ i k f (IC( X k )) }
Revised: . , ∇ i k f (IC( X k )) }

Original: This is a rather general class: x k +1 can be constructed from any inconsistent reading of past iterates
Revised: This is a rather general class: x k +1 can be constructed from any inconsistent reading of past iterates

Original: IC( X k ), and any past gradient of an inconsistent read ∇ i j f (IC( X j )).
Revised: IC( X k ), and any past gradient of an inconsistent read ∇ i j f (IC( X j )).

Original: Theorem 2.
Revised: Theorem 2.

Original: For any algorithm G ∈
Revised: For any algorithm G ∈

Original: A that solves eq.
Revised: A that solves eq.

Original: (1.1) , and parameter set ( σ, L 1 , . . .
Revised: (1.1) , and parameter set ( σ, L 1 , . . .

Original: , L n , n ) , there is a dimension d , a corresponding function f on R d , and a starting point x 0 , such that
Revised: , L n , n ) , there is a dimension d , a corresponding function f on R d , and a starting point x 0 , such that

Original: [Equation]
Revised: [Equation]

Original: k Hence A has a complexity lower bound: {+K ( (cid:15) ) ≥ 14 (1 + o (1)) (cid:0) P nj =
+}
Revised: Hence A has a complexity lower bound: {+K ( (cid:15) ) ≥ 14 (1 + o (1)) (cid:0) P nj =
+}

Original: [Equation]
Revised: p L i /σ + 2 n (cid:1) ln(1 / 2 (cid:15) )

Original: Our proof in Section D follows very similar lines to Lan & Zhou (2015);
Revised: Our proof in Section D follows very similar lines to Lan & Zhou (2015);

Original: Nesterov (2013).
Revised: Nesterov (2013).

Original:  ODE Analysis
Revised:  ODE Analysis

Original: In this section we present and analyze an ODE which is the continuous-time limit of A2BCD .
Revised: In this section we present and analyze an ODE which is the continuous-time limit of A2BCD .

Original: This
Revised: This

Original: ODE is a strongly convex, and asynchronous version of the ODE found in Su et al.
Revised: ODE is a strongly convex, and asynchronous version of the ODE found in Su et al.

Original: For simplicity, assume L i =
Revised: For simplicity, assume L i =

Original: L, ∀ i .
Revised: L, ∀ i .

Original: We rescale (I.e. we replace f ( x ) with 1 σ f .)
Revised: We rescale (I.e. we replace f ( x ) with 1 σ f .)

Original: f so that σ = 1, and hence κ = L/σ = L .
Revised: f so that σ = 1, and hence κ = L/σ = L .

Original: Taking the discrete limit of synchronous A2BCD (i.e. accelerated RBCD ), we can derive the following ODE 6 (see Section equation C.1):
Revised: Taking the discrete limit of synchronous A2BCD (i.e. accelerated RBCD ), we can derive the following ODE 6 (see Section equation C.1):

Original: [Equation]
Revised: [Equation]

Original: We deﬁne the parameter η (cid:44) nκ 1 / 2 , and the energy: E ( t ) =e n − [Equation] [Equation]
Revised: We deﬁne the parameter η (cid:44) nκ 1 / 2 , and the energy: E ( t ) =

Original: f

Original: [Equation] Y ) + 14 (cid:13)(cid:13)
Revised: e n − 1 κ − 1 / 2 t ( f ( Y ) + 14 (cid:13)(cid:13)

Original: Y + η ˙ Y
Revised: Y + η ˙ Y

Original: (cid:13)(cid:13) 2 ).
Revised: (cid:13)(cid:13) 2 ).

Original: This is very similar to the Lyapunov function discussed in equation 2.11, with 14 (cid:13)(cid:13) Y ( t ) + η ˙ Y ( t ) (cid:13)(cid:13) 2 fulﬁlling the role of k v k k 2 , and A k = 

Revised: This is very similar to the Lyapunov function discussed in equation 2.11, with 14 (cid:13)(cid:13) Y ( t ) + η ˙ Y ( t ) (cid:13)(cid:13) 2 fulﬁlling the role of k v k k 2 , and A k =

Original: (since there is no delay yet).
Revised: (since there is no delay yet).

Original: Much like the traditional analysis in the proof of Theorem 1, we can derive a linear convergence result with a similar rate.
Revised: Much like the traditional analysis in the proof of Theorem 1, we can derive a linear convergence result with a similar rate.

Original: See Section C.2.
Revised: See Section C.2.

Original: Lemma 5.
Revised: Lemma 5.

Original: If Y satisﬁes equation 3.1, the energy satisﬁes E 0 ( t ) ≤ 0 , E ( t ) ≤ E (0) , and hence:
Revised: If Y satisﬁes equation 3.1, the energy satisﬁes E 0 ( t ) ≤ 0 , E ( t ) ≤ E (0) , and hence:

Original: [Equation]
Revised: [Equation]

Original: We may also analyze an asynchronous version of equation 3.1 to motivate the proof of our main theorem.
Revised: We may also analyze an asynchronous version of equation 3.1 to motivate the proof of our main theorem.

Original: Here ˆ Y ( t ) is a delayed version of Y ( t ) with the delay bounded by τ .
Revised: Here ˆ Y ( t ) is a delayed version of Y ( t ) with the delay bounded by τ .

Original: [Equation]
Revised: [Equation]

Original: Unfortunately, this energy satisﬁes (see Section equation C.4, equation C.7):
Revised: Unfortunately, this energy satisﬁes (see Section equation C.4, equation C.7):

Original: [Equation]
Revised: [Equation]

Original: Hence this energy E ( t ) may not be decreasing in general.
Revised: Hence this energy E ( t ) may not be decreasing in general.

Original: But, we may add a continuous-time asynchronicity error (see Sun et al.
Revised: But, we may add a continuous-time asynchronicity error (see Sun et al.

Original: (2017)), much like in Deﬁnition 2, to create a decreasing energy.
Revised: (2017)), much like in Deﬁnition 2, to create a decreasing energy.

Original: Let c 0 ≥ 0 and r > 0 be arbitrary constants that will be set later.
Revised: Let c 0 ≥ 0 and r > 0 be arbitrary constants that will be set later.

Original: Deﬁne:
Revised: Deﬁne:

Original: [Equation]
Revised: [Equation]

Original: Lemma 6.
Revised: Lemma 6.

Original: When rτ ≤ 12 , the asynchronicity error A ( t ) satisﬁes:
Revised: When rτ ≤ 12 , the asynchronicity error A ( t ) satisﬁes:

Original: [Equation]
Revised: [Equation]

Original: See Section C.3 for the proof.
Revised: See Section C.3 for the proof.

Original: Adding this error to the Lyapunov function serves a similar purpose in the continuous-time case as in the proof of Theorem 1 (see Lemma 11).
Revised: Adding this error to the Lyapunov function serves a similar purpose in the continuous-time case as in the proof of Theorem 1 (see Lemma 11).

Original: It allows us to negate 12 τ − 1 c 0 units of D ( t ) for the cost of creating c 0 units of (cid:13)(cid:13) ˙
Revised: It allows us to negate 12 τ − 1 c 0 units of D ( t ) for the cost of creating c 0 units of (cid:13)(cid:13) ˙

Original: Y ( t ) (cid:13)(cid:13) 2 .
Revised: Y ( t ) (cid:13)(cid:13) 2 .

Original: This restores monotonicity.
Revised: This restores monotonicity.

Original: Theorem 3.
Revised: Theorem 3.

Original: Let c 0 = 6 κ 2 η − 1 τ 2 , and r = η − 1 .
Revised: Let c 0 = 6 κ 2 η − 1 τ 2 , and r = η − 1 .

Original: If τ ≤ 1 √ 48 nκ − 1 / 2 then we have:
Revised: If τ ≤ 1 √ 48 nκ − 1 / 2 then we have:

Original: [Equation]
Revised: [Equation]

Original: Hence f ( Y ( t )) convergence linearly to f ( x ∗ ) with rate O (cid:0) exp (cid:0) − t/ ( nκ 1 / 2 ) (cid:1)(cid:1)
Revised: Hence f ( Y ( t )) convergence linearly to f ( x ∗ ) with rate O (cid:0) exp (cid:0) − t/ ( nκ 1 / 2 ) (cid:1)(cid:1)

Original: Notice how this convergence condition is similar to Corollary 3, but a little looser.
Revised: Notice how this convergence condition is similar to Corollary 3, but a little looser.

Original: The convergence condition in Theorem 1 can actually be improved to approximately match this (see Section E).
Revised: The convergence condition in Theorem 1 can actually be improved to approximately match this (see Section E).

Original: Proof.
Revised: Proof.

Original: [Equation]
Revised: [Equation]

Original: The preceding should hopefully elucidate the logic and general strategy of the proof of Theorem 1.
Revised: The preceding should hopefully elucidate the logic and general strategy of the proof of Theorem 1.

Original:  Related work
Revised:  Related work

Original: We now discuss related work that was not addressed in Section 1.
Revised: We now discuss related work that was not addressed in Section 1.

Original: Nesterov acceleration is a method for improving an algorithm’s iteration complexity’s dependence the condition number κ .
Revised: Nesterov acceleration is a method for improving an algorithm’s iteration complexity’s dependence the condition number κ .

Original: Nesterov-accelerated methods have been proposed and discovered in many settings Nesterov (1983);
Revised: Nesterov-accelerated methods have been proposed and discovered in many settings Nesterov (1983);

Original: Tseng (2008); Nesterov (2012); Lin et al.
Revised: Tseng (2008); Nesterov (2012); Lin et al.

Original: ; Lu & Xiao (2014); Shalev-Shwartz & Zhang (2016);
Revised: ; Lu & Xiao (2014); Shalev-Shwartz & Zhang (2016);

Original: Allen-Zhu (2017), including for coordinate descent algorithms (algorithms that use 1 gradient block ∇ i f or minimize with respect to 1 coordinate block per iteration), and incremental algorithms (algorithms for ﬁnite
Revised: Allen-Zhu (2017), including for coordinate descent algorithms (algorithms that use 1 gradient block ∇ i f or minimize with respect to 1 coordinate block per iteration), and incremental algorithms (algorithms for ﬁnite

Original: sum problems 1 n P ni =

Revised: sum problems 1 n P ni =

Original: f i ( x ) that use 1 function gradient ∇ f i ( x ) per iteration).
Revised: f i ( x ) that use 1 function gradient ∇ f i ( x ) per iteration).

Original: Such algorithms can often be augmented to solve composite minimization problems (minimization for objective of the form f ( x ) + g ( x ), especially for nonsomooth g ), or include constraints.
Revised: Such algorithms can often be augmented to solve composite minimization problems (minimization for objective of the form f ( x ) + g ( x ), especially for nonsomooth g ), or include constraints.

Original: In Peng et al.
Revised: In Peng et al.

Original: (2016a), authors proposed and analyzed an asynchronous ﬁxed-point algorithm called
Revised: (2016a), authors proposed and analyzed an asynchronous ﬁxed-point algorithm called

Original: ARock, that takes proximal algorithms, forward-backward, ADMM, etc.
Revised: ARock, that takes proximal algorithms, forward-backward, ADMM, etc.

Original: as special cases.
Revised: as special cases.

Original: Work has also been done on asynchronous algorithms for ﬁnite sums in the operator setting Davis (2016);
Revised: Work has also been done on asynchronous algorithms for ﬁnite sums in the operator setting Davis (2016);

Original: Johnstone & Eckstein (2018).
Revised: Johnstone & Eckstein (2018).

Original: In Hannah & Yin (2017b); Sun et al.
Revised: In Hannah & Yin (2017b); Sun et al.

Original: ; Peng et al.
Revised: ; Peng et al.

Original: (2016c);
Revised: (2016c);

Original: Cannelli et al.
Revised: Cannelli et al.

Original: (2017) showed that many of the assumptions used in prior work (such as bounded delay τ < ∞ ) were unrealistic and unnecessary in general.
Revised: (2017) showed that many of the assumptions used in prior work (such as bounded delay τ < ∞ ) were unrealistic and unnecessary in general.

Original: In Hannah & Yin (2017a)
Revised: In Hannah & Yin (2017a)

Original: the authors showed that asynchronous iterations will complete far more iterations per second, and that a wide class of asynchronous algorithms, including asynchronous RBCD , have the same iteration complexity as their synchronous counterparts.
Revised: the authors showed that asynchronous iterations will complete far more iterations per second, and that a wide class of asynchronous algorithms, including asynchronous RBCD , have the same iteration complexity as their synchronous counterparts.

Original: Hence certain asynchronous algorithms can be expected to signiﬁcantly outperform traditional ones.
Revised: Hence certain asynchronous algorithms can be expected to signiﬁcantly outperform traditional ones.

Original: In Xiao et al.
Revised: In Xiao et al.

Original: (2017) authors propose a novel asynchronous catalyst-accelerated Lin et al.
Revised: (2017) authors propose a novel asynchronous catalyst-accelerated Lin et al.

Original: primal-dual algorithmic framework to solve regularized ERM problems.
Revised: primal-dual algorithmic framework to solve regularized ERM problems.

Original: They structure the parallel updates so that the data that an update depends on is up to date (though the rest of the data may not be).
Revised: They structure the parallel updates so that the data that an update depends on is up to date (though the rest of the data may not be).

Original: However catalyst acceleration incurs a log( κ ) penalty over Nesterov acceleration in general.
Revised: However catalyst acceleration incurs a log( κ ) penalty over Nesterov acceleration in general.

Original: In Allen-Zhu (2017), the author argues that the inner iterations of catalyst acceleration are hard to tune, making it less practical than Nesterov acceleration.
Revised: In Allen-Zhu (2017), the author argues that the inner iterations of catalyst acceleration are hard to tune, making it less practical than Nesterov acceleration.

Original:  Numerical experiments
Revised:  Numerical experiments

Original: To investigate the performance of A2BCD , we solve the ridge regression problem.
Revised: To investigate the performance of A2BCD , we solve the ridge regression problem.

Original: Consider the following primal and corresponding dual objective (see for instance Lin et al.
Revised: Consider the following primal and corresponding dual objective (see for instance Lin et al.

Original: [Equation]
Revised: [Equation]

Original: where A ∈ R d × n is a matrix of n samples and d features, and l is a label vector.
Revised: where A ∈ R d × n is a matrix of n samples and d features, and l is a label vector.

Original: We let A =
Revised: We let A =

Original: [ A 1 , . .
Revised: [ A 1 , . .

Original: A m ] where A i are the column blocks of A .
Revised: A m ] where A i are the column blocks of A .

Original: We compare A2BCD (which is asynchronous accelerated), synchronous NU_ACDM (which is synchronous accelerated), and asynchronous RBCD (which is asynchronous non-accelerated).
Revised: We compare A2BCD (which is asynchronous accelerated), synchronous NU_ACDM (which is synchronous accelerated), and asynchronous RBCD (which is asynchronous non-accelerated).

Original: Nodes randomly select a coordinate block according to equation 2.1, calculate the corresponding block gradient, and use it to apply an update to the shared solution vectors.
Revised: Nodes randomly select a coordinate block according to equation 2.1, calculate the corresponding block gradient, and use it to apply an update to the shared solution vectors.

Original: synchronous NU_ACDM is implemented in a batch fashion, with batch size p (1 block per computing node).
Revised: synchronous NU_ACDM is implemented in a batch fashion, with batch size p (1 block per computing node).

Original: Nodes in synchronous NU_ACDM implementation must wait until all nodes apply their computed gradients before they can start the next iteration, but the asynchronous algorithms simply compute with the most up-to-date information available.
Revised: Nodes in synchronous NU_ACDM implementation must wait until all nodes apply their computed gradients before they can start the next iteration, but the asynchronous algorithms simply compute with the most up-to-date information available.

Original: We use the datasets w1a (47272 samples, 300 features), wxa which combines the data from from w1a to w8a (293201 samples, 300 features), and aloi (108000 samples, 128 features) from LIBSVM
Revised: We use the datasets w1a (47272 samples, 300 features), wxa which combines the data from from w1a to w8a (293201 samples, 300 features), and aloi (108000 samples, 128 features) from LIBSVM

Original: Chang & Lin (2011).
Revised: Chang & Lin (2011).

Original: The algorithm is implemented in a multi-threaded fashion using C++11 and
Revised: The algorithm is implemented in a multi-threaded fashion using C++11 and

Original: GNU Scientiﬁc Library with a shared memory architecture.
Revised: GNU Scientiﬁc Library with a shared memory architecture.

Original: We use 40 threads on two 2.5GHz 10-core Intel Xeon E5-2670v2 processors.
Revised: We use 40 threads on two 2.5GHz 10-core Intel Xeon E5-2670v2 processors.

Original: See Section A.1 for a discussion of parameter tuning and estimation.
Revised: See Section A.1 for a discussion of parameter tuning and estimation.

Original: The parameters for each algorithm are tuned to give the fastest performance, so that a fair comparison is possible.
Revised: The parameters for each algorithm are tuned to give the fastest performance, so that a fair comparison is possible.

Original: A critical ingredient in the eﬃcient implementation of A2BCD and NU_ACDM for this problem is the eﬃcient update scheme discussed in Lee & Sidford (2013b;a).
Revised: A critical ingredient in the eﬃcient implementation of A2BCD and NU_ACDM for this problem is the eﬃcient update scheme discussed in Lee & Sidford (2013b;a).

Original: In linear regression applications such as this, it is essential to be able to eﬃciently maintain or recover Ay .
Revised: In linear regression applications such as this, it is essential to be able to eﬃciently maintain or recover Ay .

Original: This is because calculating block gradients requires the vector A Ti Ay , and without an eﬃcient way to recover Ay , block gradient evaluations are essentially 50% as expensive as full-gradient calculations.
Revised: This is because calculating block gradients requires the vector A Ti Ay , and without an eﬃcient way to recover Ay , block gradient evaluations are essentially 50% as expensive as full-gradient calculations.

Original: Unfortunately, every accelerated iteration results in dense updates to y k because of the averaging step in equation 2.6.
Revised: Unfortunately, every accelerated iteration results in dense updates to y k because of the averaging step in equation 2.6.

Original: Hence Ay must be recalculated from scratch.
Revised: Hence Ay must be recalculated from scratch.

Original: However Lee & Sidford (2013a) introduces a linear transformation that allows for an equivalent iteration that results in sparse updates to new iteration variables p and q .
Revised: However Lee & Sidford (2013a) introduces a linear transformation that allows for an equivalent iteration that results in sparse updates to new iteration variables p and q .

Original: The original purpose of this transformation was to ensure that the averaging steps (e.g. equation 2.6) do not dominate the computational cost for sparse problems.
Revised: The original purpose of this transformation was to ensure that the averaging steps (e.g. equation 2.6) do not dominate the computational cost for sparse problems.

Original: However we ﬁnd a more important secondary use which applies to both sparse and dense problems.
Revised: However we ﬁnd a more important secondary use which applies to both sparse and dense problems.

Original: Since the updates to p and q are sparse coordinate-block updates, the vectors Ap , and Aq can be eﬃciently maintained, and therefore block gradients can be eﬃciently calculated.
Revised: Since the updates to p and q are sparse coordinate-block updates, the vectors Ap , and Aq can be eﬃciently maintained, and therefore block gradients can be eﬃciently calculated.

Original: The speciﬁcs of this eﬃcient implementation are discussed in Section A.2.
Revised: The speciﬁcs of this eﬃcient implementation are discussed in Section A.2.

Original: In Table 5, we plot the sub-optimality vs. time for decreasing values of λ , which corresponds to increasingly large condition numbers κ .
Revised: In Table 5, we plot the sub-optimality vs. time for decreasing values of λ , which corresponds to increasingly large condition numbers κ .

Original: When κ is small, acceleration doesn’t result in a signiﬁcantly better convergence rate, and hence A2BCD and async-RBCD both outperform sync- NU_ACDM since they complete faster iterations at similar complexity.
Revised: When κ is small, acceleration doesn’t result in a signiﬁcantly better convergence rate, and hence A2BCD and async-RBCD both outperform sync- NU_ACDM since they complete faster iterations at similar complexity.

Original: Acceleration for low κ has unnecessary overhead, which means async-RBCD can be quite competitive.
Revised: Acceleration for low κ has unnecessary overhead, which means async-RBCD can be quite competitive.

Original: When κ becomes large, async-RBCD is no longer competitive, since it has a poor convergence rate.
Revised: When κ becomes large, async-RBCD is no longer competitive, since it has a poor convergence rate.

Original: We observe that A2BCD and sync- NU_ACDM have essentially the same convergence rate, but A2BCD is up to 4 − 5 × faster than sync- NU_ACDM because it completes much faster iterations.
Revised: We observe that A2BCD and sync- NU_ACDM have essentially the same convergence rate, but A2BCD is up to 4 − 5 × faster than sync- NU_ACDM because it completes much faster iterations.

Original: We observe this advantage despite the fact that we are in an ideal environment for synchronous computation:
Revised: We observe this advantage despite the fact that we are in an ideal environment for synchronous computation:

Original: A small, homogeneous, high-bandwidth, low-latency cluster.
Revised: A small, homogeneous, high-bandwidth, low-latency cluster.

Original: In large-scale heterogeneous systems with greater synchronization overhead, bandwidth constraints, and latency, we expect A2BCD ’s advantage to be much larger.  
Revised: In large-scale heterogeneous systems with greater synchronization overhead, bandwidth constraints, and latency, we expect A2BCD ’s advantage to be much larger.  Acknowledgement

Revised: The authors would like to thank the reviewers for their helpful comments.

Revised: The research presented in this paper was supported in part by AFOSR MURI FA9550-18-10502, NSF DMS- and

Revised: ONR N0001417121.

Original: A Efficient Implementation
Revised: A Efficient Implementation

Original: An eﬃcient implementation will have coordinate blocks of size greater than 1.
Revised: An eﬃcient implementation will have coordinate blocks of size greater than 1.

Original: This to ensure the eﬃciency of linear algebra subroutines.
Revised: This to ensure the eﬃciency of linear algebra subroutines.

Original: Especially because of this, the bulk of the computation for each iteration is computing ∇ i k f
Revised: Especially because of this, the bulk of the computation for each iteration is computing ∇ i k f

Original: (ˆ y k ), and not the averaging steps.
Revised: (ˆ y k ), and not the averaging steps.

Original: Hence the computing nodes only need a local copy of y k in order to do the bulk of an iteration’s computation.
Revised: Hence the computing nodes only need a local copy of y k in order to do the bulk of an iteration’s computation.

Original: Given this gradient ∇ i k f (ˆ y k ), updating y k and v k is extremely fast ( x k can simply be eliminated).
Revised: Given this gradient ∇ i k f (ˆ y k ), updating y k and v k is extremely fast ( x k can simply be eliminated).

Original: Hence it is natural to simply store y k and v k centrally, and update them when the delayed gradients ∇ i k f
Revised: Hence it is natural to simply store y k and v k centrally, and update them when the delayed gradients ∇ i k f

Original: (ˆ y k ).
Revised: (ˆ y k ).

Original: Given the above, a write mutex over ( y, v ) has minuscule overhead (which we conﬁrm with experiments), and makes the labeling of iterates unambiguous.
Revised: Given the above, a write mutex over ( y, v ) has minuscule overhead (which we conﬁrm with experiments), and makes the labeling of iterates unambiguous.

Original: This also ensures that v k and y k are always up to date when ( y, v ) are being updated.
Revised: This also ensures that v k and y k are always up to date when ( y, v ) are being updated.

Original: Whereas the gradient ∇ i k f (ˆ y k ) may at the same time be out of date, since it has been calculated with an outdated version of y k .
Revised: Whereas the gradient ∇ i k f (ˆ y k ) may at the same time be out of date, since it has been calculated with an outdated version of y k .

Original: However a write mutex is not necessary in practice, and does not appear to aﬀect convergence rates or computation time.
Revised: However a write mutex is not necessary in practice, and does not appear to aﬀect convergence rates or computation time.

Original: Also it is possible to prove convergence under more general asynchronicity.
Revised: Also it is possible to prove convergence under more general asynchronicity.

Original: A. Parameter selection and tuning
Revised: A. Parameter selection and tuning

Original: When deﬁning the coeﬃcients, σ may be underestimated, and L, L 1 , . .
Revised: When deﬁning the coeﬃcients, σ may be underestimated, and L, L 1 , . .

Original: , L n may be overestimated if exact values are unavailable.
Revised: , L n may be overestimated if exact values are unavailable.

Original: Notice that x k can be eliminated from the above iteration, and the block gradient ∇ i k f (ˆ y k ) only needs to be calculated once per iteration.
Revised: Notice that x k can be eliminated from the above iteration, and the block gradient ∇ i k f (ˆ y k ) only needs to be calculated once per iteration.

Original: A larger (or overestimated) maximum delay τ will cause a larger asynchronicity parameter ψ , which leads to more conservative step sizes to compensate.
Revised: A larger (or overestimated) maximum delay τ will cause a larger asynchronicity parameter ψ , which leads to more conservative step sizes to compensate.

Original: To estimate ψ , one can ﬁrst performed a dry run with all coeﬃcient set to 0 to estimate τ .
Revised: To estimate ψ , one can ﬁrst performed a dry run with all coeﬃcient set to 0 to estimate τ .

Original: All function parameters can be calculated exactly for this problem in terms of the data matrix and λ .
Revised: All function parameters can be calculated exactly for this problem in terms of the data matrix and λ .

Original: We can then use these parameters and this tau to calculate ψ .
Revised: We can then use these parameters and this tau to calculate ψ .

Original: ψ and τ merely change the parameters, and do not change execution patterns of the processors.
Revised: ψ and τ merely change the parameters, and do not change execution patterns of the processors.

Original: Hence their parameter speciﬁcation doesn’t aﬀect the observed delay.
Revised: Hence their parameter speciﬁcation doesn’t aﬀect the observed delay.

Original: Through simple tuning though, we found that ψ = 0 .
Revised: Through simple tuning though, we found that ψ = 0 .

Original: 25 resulted in good performance.
Revised: 25 resulted in good performance.

Original: In tuning for general problems, there are theoretical reasons why it is diﬃcult to attain acceleration without some prior knowledge of σ , the strong convexity modulus Arjevani (2017).
Revised: In tuning for general problems, there are theoretical reasons why it is diﬃcult to attain acceleration without some prior knowledge of σ , the strong convexity modulus Arjevani (2017).

Original: Ideally σ is pre-speciﬁed for instance in a regularization term.
Revised: Ideally σ is pre-speciﬁed for instance in a regularization term.

Original: If the Lipschitz constants L i cannot be calculated directly (which is rarely the case for the classic dual problem of empirical risk minimization objectives), the line-search method discussed in Roux et al.
Revised: If the Lipschitz constants L i cannot be calculated directly (which is rarely the case for the classic dual problem of empirical risk minimization objectives), the line-search method discussed in Roux et al.

Original: (2012) Section 4 can be used.
Revised: (2012) Section 4 can be used.

Original: A. Sparse update formulation
Revised: A. Sparse update formulation

Original: As mentioned in Section 5, authors in Lee & Sidford (2013a) proposed a linear transformation of an accelerated RBCD scheme that results in sparse coordinate updates.
Revised: As mentioned in Section 5, authors in Lee & Sidford (2013a) proposed a linear transformation of an accelerated RBCD scheme that results in sparse coordinate updates.

Original: Our proposed algorithm can be given a similar eﬃcient implementation.
Revised: Our proposed algorithm can be given a similar eﬃcient implementation.

Original: We may eliminate x k from A2BCD , and derive the equivalent iteration below:
Revised: We may eliminate x k from A2BCD , and derive the equivalent iteration below:

Original: [Equation]
Revised: [Equation]

Original: where C and Q k are deﬁned in the obvious way.
Revised: where C and Q k are deﬁned in the obvious way.

Original: Hence we deﬁne auxiliary variables p k , q k deﬁned via:
Revised: Hence we deﬁne auxiliary variables p k , q k deﬁned via:

Original: [Equation]
Revised: [Equation]

Original: These clearly follow the iteration:
Revised: These clearly follow the iteration:

Original: [Equation]
Revised: [Equation]

Original: Since the vector Q k is sparse, we can evolve variables p k , and q k in a sparse manner, and recover the original iteration variables at the end of the algorithm via A.1.
Revised: Since the vector Q k is sparse, we can evolve variables p k , and q k in a sparse manner, and recover the original iteration variables at the end of the algorithm via A.1.

Original: The gradient of the dual function is given by:
Revised: The gradient of the dual function is given by:

Original: [Equation]
Revised: [Equation]

Original: As mentioned before, it is necessary to maintain or recover
Revised: As mentioned before, it is necessary to maintain or recover

Original: Ay k to calculate block gradients.
Revised: Ay k to calculate block gradients.

Original: Since
Revised: Since

Original: Ay k can be recovered via the linear relation in equation A.1, and the gradient is an aﬃne function, we maintain the auxiliary vectors
Revised: Ay k can be recovered via the linear relation in equation A.1, and the gradient is an aﬃne function, we maintain the auxiliary vectors

Original: Ap k and Aq k instead.
Revised: Ap k and Aq k instead.

Original: Hence we propose the following eﬃcient implementation in Algorithm 1.
Revised: Hence we propose the following eﬃcient implementation in Algorithm 1.

Original: We used this to generate the results in Table 5.
Revised: We used this to generate the results in Table 5.

Original: We also note also that it can improve performance to periodically recover v k and y k , reset the values of p k , q k , and C to v k , y k , and I respectively, and restarting the scheme (which can be done cheaply in time O ( d )).
Revised: We also note also that it can improve performance to periodically recover v k and y k , reset the values of p k , q k , and C to v k , y k , and I respectively, and restarting the scheme (which can be done cheaply in time O ( d )).

Original: We let B ∈ R 2 × 2 represent C k , and b represent B − 1 .
Revised: We let B ∈ R 2 × 2 represent C k , and b represent B − 1 .

Original: ⊗ is the Kronecker product.
Revised: ⊗ is the Kronecker product.

Original: Each computing node has local outdated versions of p, q, Ap, Aq which we denote ˆ p, ˆ q, ˆ Ap, ˆ Aq respectively.
Revised: Each computing node has local outdated versions of p, q, Ap, Aq which we denote ˆ p, ˆ q, ˆ Ap, ˆ Aq respectively.

Original: We also ﬁnd it convenient to deﬁne:
Revised: We also ﬁnd it convenient to deﬁne:

Original: [Equation]
Revised: [Equation]

Original: [Equation] 1: Inputs:Function parameters A , λ , L , { L i } ni =1 , n , d .
Revised: 1: Inputs:

Original: Delay τ (obtained in dry run). Starting vectors y , v .
Revised: L

Original: 2: Shared data: Solution vectors p , q ; auxiliary vectors Ap , Aq ; sparsifying matrix B 3: Node local data: Solutionvectors ˆ p, ˆ q, auxiliary vectors ˆ Ap, ˆ Aq , sparsifyingmatrix ˆ B .
Revised: 2: Shared data B 3: Node local data Solution p q auxiliary vectors ˆ Ap ˆ Aq , sparsifying

Original:  4: Calculate parameters ψ, α, β, h via 1.
Revised: ˆ B . 4: Calculate parameters ψ, α, β, h via 1.

Original: Set k = 0.
Revised: Set k = 0.

Original: 5: Initializations: p ← y , q ← v , Ap ←
Revised: 5: Initializations: p ← y , q ← v , Ap ←

Original: Ay , Aq ← Av , B ← I .
Revised: Ay , Aq ← Av , B ← I .

Original: 6: while not converged, each computing node asynchronous do 7:
Revised: 6: while not converged, each computing node asynchronous do 7:

Original: Randomly select block i via equation 2.1.
Revised: Randomly select block i via equation 2.1.

Original: Read shared data into local memory: ˆ p ← p , ˆ q ← q , ˆ Ap ← Ap , ˆ Aq ← Aq , ˆ B ← B .
Revised: Read shared data into local memory: ˆ p ← p , ˆ q ← q , ˆ Ap ← Ap , ˆ Aq ← Aq , ˆ B ← B .

Original: Compute block gradient: ∇ i f
Revised: Compute block gradient: ∇ i f

Original: (ˆ y )
Revised: (ˆ y )

Original: = 1 nλ (cid:16) 1 n A Ti (cid:16) ˆ B 1 , 1 ˆ Ap + ˆ B 1 , 2 ˆ Aq (cid:17) + λ (cid:16) ˆ B 1 , 1 ˆ p + ˆ B 1 , 2 ˆ q (cid:17)(cid:17) 10:
Revised: = 1 nλ (cid:16) 1 n A Ti (cid:16) ˆ B 1 , 1 ˆ Ap + ˆ B 1 , 2 ˆ Aq (cid:17) + λ (cid:16) ˆ B 1 , 1 ˆ p + ˆ B 1 , 2 ˆ q (cid:17)(cid:17) 10:

Original: Compute quantity g i
Revised: Compute quantity g i

Original: =
Revised: =

Original: A Ti ∇ i f (ˆ y )
Revised: A Ti ∇ i f (ˆ y )

Original: Shared memory updates: 11:
Revised: Shared memory updates: 11:

Original: Update B ← (cid:20) 1 − αβ αβ 1 − β β (cid:21) × B , calculate inverse b ← B − 1 .
Revised: Update B ← (cid:20) 1 − αβ αβ 1 − β β (cid:21) × B , calculate inverse b ← B − 1 .

Original: 12: (cid:20) pq (cid:21) − =
Revised: 12: (cid:20) pq (cid:21) − =

Original: b (cid:20) D k
Revised: b (cid:20) D k

Original: D k(cid:21)
Revised: D k(cid:21)

Original: ⊗ ∇ i f
Revised: ⊗ ∇ i f

Original: (ˆ y ) , (cid:20) ApAq (cid:21) − =
Revised: (ˆ y ) , (cid:20) ApAq (cid:21) − =

Original: b (cid:20) D k
Revised: b (cid:20) D k

Original: D k(cid:21) ⊗ g i 13:
Revised: D k(cid:21) ⊗ g i 13:

Original: Increase iteration count:
Revised: Increase iteration count:

Original: k ← k +14: end while 15: Recover original iteration variables: (cid:20) yv (cid:21)
Revised: k ← k +14: end while 15: Recover original iteration variables: (cid:20) yv (cid:21)

Original: [Equation]
Revised: [Equation]

Original: B Proof of the main result
Revised: B Proof of the main result

Original: We ﬁrst recall a couple of inequalities for convex functions.
Revised: We ﬁrst recall a couple of inequalities for convex functions.

Original: Lemma 7.
Revised: Lemma 7.

Original: Let f be σ -strongly convex with L -Lipschitz gradient.
Revised: Let f be σ -strongly convex with L -Lipschitz gradient.

Original: Then we have:
Revised: Then we have:

Original: [Equation]
Revised: [Equation]

Original: We also ﬁnd it convenient to deﬁne the norm:
Revised: We also ﬁnd it convenient to deﬁne the norm:

Original: [Equation]
Revised: [Equation]

Original: B. Starting point
Revised: B. Starting point

Original: First notice that using the deﬁnition equation 2.8 of v k +1 we have:
Revised: First notice that using the deﬁnition equation 2.8 of v k +1 we have:

Original: [Equation]
Revised: [Equation]

Original: We have the following general identity:
Revised: We have the following general identity:

Original: [Equation]
Revised: [Equation]

Original: It can also easily be veriﬁed from equation 2.6 that we have:
Revised: It can also easily be veriﬁed from equation 2.6 that we have:

Original: [Equation]
Revised: [Equation]

Original: Using equation B
Revised: Using equation B

Original: [Equation]
Revised: [Equation]

Original: 5 on term
Revised: 5 on term

Original: A
Revised: A

Original: [Equation]
Revised: [Equation]

Original: equation B
Revised: equation B

Original: [Equation]
Revised: [Equation]

Original: 6 on term B
Revised: 6 on term B

Original: [Equation]
Revised: [Equation]

Original: and recalling the deﬁnition equation B
Revised: and recalling the deﬁnition equation B

Original: [Equation]
Revised: [Equation]

Original: 3 on term C
Revised: 3 on term C

Original: [Equation]
Revised: [Equation]

Original: we have from equation B
Revised: we have from equation B

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: This inequality is our starting point.
Revised: This inequality is our starting point.

Original: We analyze the terms on the second line in the next section.
Revised: We analyze the terms on the second line in the next section.

Original: B. The Cross Term
Revised: B. The Cross Term

Original: To analyze these terms, we need a small lemma.
Revised: To analyze these terms, we need a small lemma.

Original: This lemma is fundamental in allowing us to deal with asynchronicity.
Revised: This lemma is fundamental in allowing us to deal with asynchronicity.

Original: Lemma 8.
Revised: Lemma 8.

Original: Let χ, A > 0 .
Revised: Let χ, A > 0 .

Original: Let the delay be bounded by τ .
Revised: Let the delay be bounded by τ .

Original: Then:
Revised: Then:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: See Hannah & Yin (2017a).
Revised: See Hannah & Yin (2017a).

Original: Lemma 9.
Revised: Lemma 9.

Original: We have:
Revised: We have:

Original: [Equation]
Revised: [Equation]

Original: The terms in bold in equation B.8 and equation B.9 are a result of the asynchronicity, and are identically 0 in its absence.
Revised: The terms in bold in equation B.8 and equation B.9 are a result of the asynchronicity, and are identically 0 in its absence.

Original: Proof.
Revised: Proof.

Original: Our strategy is to separately analyze terms that appear in the traditional analysis of Nesterov (2012)
Revised: Our strategy is to separately analyze terms that appear in the traditional analysis of Nesterov (2012)

Original: [Equation]
Revised: [Equation]

Original: and the terms that result from asynchronicity.
Revised: and the terms that result from asynchronicity.

Original: We ﬁrst prove equation B.8:
Revised: We ﬁrst prove equation B.8:

Original: [Equation]
Revised: [Equation]

Original: equation B.10 follows from strong convexity (equation B.2 with x = y k and y = x ∗ )
Revised: equation B.10 follows from strong convexity (equation B.2 with x = y k and y = x ∗ )

Original: [Equation]
Revised: [Equation]

Original: and the fact that ∇ f is L -Lipschitz.
Revised: and the fact that ∇ f is L -Lipschitz.

Original: The term due to asynchronicity becomes:
Revised: The term due to asynchronicity becomes:

Original: [Equation]
Revised: [Equation]

Original: using Lemma 8 with χ = κψ − 1 , A = k y k k .
Revised: using Lemma 8 with χ = κψ − 1 , A = k y k k .

Original: Combining this with equation B.10 completes the proof of equation B.
Revised: Combining this with equation B.10 completes the proof of equation B.

Original: [Equation]
Revised: [Equation]

Original: We now prove equation B.9:
Revised: We now prove equation B.9:

Original: [Equation]
Revised: [Equation]

Original: Here the last line follows from Lemma 8 with χ = κψ − 1 β −
Revised: Here the last line follows from Lemma 8 with χ = κψ − 1 β −

Original: [Equation]
Revised: [Equation]

Revised: −

Revised: [Equation]

Revised: −

Revised: [Equation]

Revised: [Equation]

Revised: =

Revised: [Equation]

Original: k − y k .
Revised: − y k .

Original: We can complete the proof using the following identity that can be easily obtained from equation 2.6:
Revised: We can complete the proof using the following identity that can be easily obtained from equation 2.6:

Original: [Equation]
Revised: [Equation]

Original: B. Function-value term
Revised: B. Function-value term

Original: Much like Nesterov (2012), we need a f ( x k ) term in the Lyapunov function (see the middle of page 357).
Revised: Much like Nesterov (2012), we need a f ( x k ) term in the Lyapunov function (see the middle of page 357).

Original: However we additionally need to consider asynchronicity when analyzing the growth of this term.
Revised: However we additionally need to consider asynchronicity when analyzing the growth of this term.

Original: Again terms due to asynchronicity are emboldened.
Revised: Again terms due to asynchronicity are emboldened.

Original: Lemma 10.
Revised: Lemma 10.

Original: We have:
Revised: We have:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: From the deﬁnition equation 2.7 of x k +1 , we can see that x k +1 − y k is supported on block i k .
Revised: From the deﬁnition equation 2.7 of x k +1 , we can see that x k +1 − y k is supported on block i k .

Original: Since each gradient block ∇ i f is L i Lipschitz with respect to changes to block i , we can use equation B.1 to obtain:
Revised: Since each gradient block ∇ i f is L i Lipschitz with respect to changes to block i , we can use equation B.1 to obtain:

Original: [Equation]
Revised: [Equation]

Original: (from equation 2.7) =
Revised: from equation [Equation] [Equation] =

Original: [Equation]
Revised: [Equation]

Original: Here the last line followed from the deﬁnition equation B.3 of the norm k·k ∗ 1 / 2 .
Revised: Here the last line followed from the deﬁnition equation B.3 of the norm k·k ∗ 1 / 2 .

Original: We now analyze the middle term:
Revised: We now analyze the middle term:

Original: [Equation]
Revised: [Equation]

Original: (L ≤ L i , ∀ i and equation B.3) ≤ L − 
Revised: (L ≤ L i , ∀ i and equation B.3) ≤ L − 1 / [Equation] k∇ f ( y [Equation]

Revised: f

Original: [Equation]
Revised: [Equation]

Revised: f

Revised: [Equation]

Original: Lipschitz
Revised: Lipschitz

Original: [Equation]
Revised: [Equation]

Original: We then apply Lemma 8 to this with χ = 2 h − 1 σ 1 / 2 L 1 / 4 κψ − 1 , A =
Revised: We then apply Lemma 8 to this with χ = 2 h − 1 σ 1 / 2 L 1 / 4 κψ − 1 , A =

Original: k∇ f (ˆ y k ) k ∗ to yield: −
Revised: k∇ f (ˆ y k ) k ∗ to yield: −

Original: [Equation]
Revised: [Equation]

Original: Finally to complete the proof, we combine equation B.11, with equation B.12.
Revised: Finally to complete the proof, we combine equation B.11, with equation B.12.

Original: B. Asynchronicity error
Revised: B. Asynchronicity error

Original: The previous inequalities produced diﬀerence terms of the form k y k
Revised: The previous inequalities produced diﬀerence terms of the form k y k

Original: +1 − j − y k − j k 2 .
Revised: +1 − j − y k − j k 2 .

Original: The following lemma shows how these errors can be incorporated into a Lyapunov function.
Revised: The following lemma shows how these errors can be incorporated into a Lyapunov function.

Original: Lemma 11.
Revised: Lemma 11.

Original: Let 0 < r < 1 and consider the asynchronicity error and corresponding coeﬃcients:
Revised: Let 0 < r < 1 and consider the asynchronicity error and corresponding coeﬃcients:

Original: [Equation]
Revised: [Equation]

Original: This sum satisﬁes:
Revised: This sum satisﬁes:

Original: [Equation]
Revised: [Equation]

Original: Remark 2.
Revised: Remark 2.

Original: Interpretation.
Revised: Interpretation.

Original: This result means that an asynchronicity error term A k can negate a series of diﬀerence terms − P ∞ j =

Revised: This result means that an asynchronicity error term A k can negate a series of diﬀerence terms − P ∞ j =

Original: s
Revised: s

Original: j k y k
Revised: j k y k

Original: +1 − j − y k − j k 2 at the cost of producing an additional error c 1 E k k y k
Revised: +1 − j − y k − j k 2 at the cost of producing an additional error c 1 E k k y k

Original: +1 − y k k 2 , while maintaining a convergence rate of r .
Revised: +1 − y k k 2 , while maintaining a convergence rate of r .

Original: This essentially converts diﬀerence terms, which are hard to deal with, into a k y k
Revised: This essentially converts diﬀerence terms, which are hard to deal with, into a k y k

Original: +1 − y k k 2 term which can be negated by other terms in the Lyapunov function.
Revised: +1 − y k k 2 term which can be negated by other terms in the Lyapunov function.

Original: The proof is straightforward.
Revised: The proof is straightforward.

Original: Proof.
Revised: Proof.

Original: [Equation]
Revised: [Equation]

Original: Noting the following completes the proof:
Revised: Noting the following completes the proof:

Original: [Equation]
Revised: [Equation]

Original: Given that A k allows us to negate diﬀerence terms, we now analyze the cost c 1 E
Revised: Given that A k allows us to negate diﬀerence terms, we now analyze the cost c 1 E

Original: k k y k
Revised: k k y k

Original: +1 − y k k 2 of this negation.
Revised: +1 − y k k 2 of this negation.

Original: Lemma 12.
Revised: Lemma 12.

Original: We have:
Revised: We have:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: [Equation]
Revised: [Equation]

Original: Here equation B.13 following from equation 2.8, the deﬁnition of v k +1 .
Revised: Here equation B.13 following from equation 2.8, the deﬁnition of v k +1 .

Original: equation B.14 follows from the inequality k x + y k 2 ≤ 2 k x k 2 + 2 k y k 2 .
Revised: equation B.14 follows from the inequality k x + y k 2 ≤ 2 k x k 2 + 2 k y k 2 .

Original: The rest is simple algebraic manipulation.
Revised: The rest is simple algebraic manipulation.

Original: [Equation]
Revised: [Equation]

Original: Finally, to complete the proof, we prove L 1 / 2 σ − 1 / 

Revised: Finally, to complete the proof, we prove L 1 / 2 σ − 1 /

Original: α + h (1 − α ) ≤ 1.
Revised: α + h (1 − α ) ≤ 1.

Original: [Equation]
Revised: [Equation]

Original: L
Revised: (deﬁnitions of h and α : equation 2.3, and equation 2.5) = 1 −

Original: [Equation]
Revised: [Equation]

Original: (deﬁnitions of h and α : equation 2.3, and equation 2.5) = 1 − 1 2 σ [-1 / 2 L − 1 / 2 ψ + σ 1 / 2 S − 1 (cid:
-]
Revised: 2 σ [-1 / 2 L − 1 / 2 ψ + σ 1 / 2 S − 1 (cid:
-][Equation]

Original: ) L
Revised: 2 L

Original: [Equation]
Revised: [Equation]

Original: 2 σ−
Revised: σ

Original: [Equation]
Revised: [Equation]

Revised: S

Revised: [Equation]

Revised: (cid:16) L

Original: / 2 (cid:17)
Revised: [Equation] σ [Equation] (cid:17)

Original: [Equation]
Revised: [Equation]

Original: Rearranging the deﬁnition of ψ , we have:
Revised: Rearranging the deﬁnition of ψ , we have:

Original: [Equation]
Revised: [Equation]

Original: Using this on equation B.15, we have:
Revised: Using this on equation B.15, we have:

Original: [Equation]
Revised: [Equation]

Original: This completes the proof.
Revised: This completes the proof.

Original: B. Master inequality
Revised: B. Master inequality

Original: We are ﬁnally in a position to bring together all the all the previous results together into a master inequality for the Lyapunov function ρ k (deﬁned in equation 2.11).
Revised: We are ﬁnally in a position to bring together all the all the previous results together into a master inequality for the Lyapunov function ρ k (deﬁned in equation 2.11).

Original: After this lemma is proven, we will prove that the right hand size is negative, which will imply that ρ k linearly converges to 0 with rate β .
Revised: After this lemma is proven, we will prove that the right hand size is negative, which will imply that ρ k linearly converges to 0 with rate β .

Original: Lemma 13.
Revised: Lemma 13.

Original: Master inequality.
Revised: Master inequality.

Original: We have:
Revised: We have:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: [Equation]
Revised: [Equation]

Original: We now collect and organize the similar terms of this inequality.
Revised: We now collect and organize the similar terms of this inequality

Original: [Equation]
Revised: [Equation]

Original: Now ﬁnally, we add the function-value and asynchronicity terms to our analysis.
Revised: Now ﬁnally, we add the function-value and asynchronicity terms to our analysis.

Original: We use Lemmais with r = 1 − σ 1 / 2 S − 1 , and
Revised: We use Lemmais with r = 1 − σ 1 / 2 S − 1 , and

Original: [Equation]
Revised: [Equation]

Original: Notice that this choice of s i will recover the coeﬃcient formula given in equation 2.9.
Revised: Notice that this choice of s i will recover the coeﬃcient formula given in equation 2.9.

Original: Hence we have:
Revised: Hence we have:

Original: [Equation]
Revised: [Equation]

Original: (Lemmas 11 and 12) + 
Revised: (Lemmas 11 and 12) + c

Original: [Equation]
Revised: [Equation]

Original: Notice A k ( r − β ) ≤ 0.
Revised: Notice A k ( r − β ) ≤ 0.

Original: Finally, combining equation B.17 and equation B.19 completes the proof.
Revised: Finally, combining equation B.17 and equation B.19 completes the proof.

Original: In the next section, we will prove that every coeﬃcient on the right hand side of equation B.16 isor less, which will complete the proof of Theorem 1.
Revised: In the next section, we will prove that every coeﬃcient on the right hand side of equation B.16 isor less, which will complete the proof of Theorem 1.

Original: B. Proof of main theorem
Revised: B. Proof of main theorem

Original: Lemma 14.
Revised: Lemma 14.

Original: The coeﬃcients of k y k k 2 , f
Revised: The coeﬃcients of k y k k 2 , f

Original: ( y k ), and P τj =1 k y k
Revised: ( y k ), and P τj =1 k y k

Original: +1 − j − y k − j k 2 in Lemma 13 are non-positive.
Revised: +1 − j − y k − j k 2 in Lemma 13 are non-positive.

Original: Proof.
Revised: Proof.

Original: The coeﬃcient 1 − (1 − ψ ) σ 1 / 2 S − 1 − β of k y k k 2 is identically 

Revised: The coeﬃcient 1 − (1 − ψ ) σ 1 / 2 S − 1 − β of k y k k 2 is identically

Original: via the deﬁnition equation 2.
Revised: via the deﬁnition equation 2.

Original: of β .
Revised: of β .

Original: The coeﬃcient c − 2 σ − 1 / 2 S − 1 (cid:0) βα − 1 (1 − α ) + 1 (cid:1) of f ( y k ) is identically 0 via the deﬁnition equation 2.12 of c .
Revised: The coeﬃcient c − 2 σ − 1 / 2 S − 1 (cid:0) βα − 1 (1 − α ) + 1 (cid:1) of f ( y k ) is identically 0 via the deﬁnition equation 2.12 of c .

Original: First notice from the deﬁnition equation 2.12 of c :
Revised: First notice from the deﬁnition equation 2.12 of c :

Original: [Equation]
Revised: [Equation]

Original: (deﬁnitions of
Revised: (deﬁnitions of

Original: [Equation]
Revised: [Equation]

Original: B.
Revised: B.21)

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: B.22)

Original: Here the last line followed since ψ ≤ 12 and σ 1 / 2 S − 1 ≤ 1.
Revised: Here the last line followed since ψ ≤ 12 and σ 1 / 2 S − 1 ≤ 1.

Original: We now analyze the coeﬃcient of
Revised: We now analyze the coeﬃcient of

Original: P τj =1 k y k
Revised: P τj =1 k y k

Original: +1 − j − y k − j k 2 .
Revised: +1 − j − y k − j k 2 .

Original: [Equation]
Revised: [Equation]

Original: (deﬁnition equation B.18 of s ) ≤
Revised: (deﬁnition equation B.18 of s ) ≤

Original: Lemma 15.
Revised: Lemma 15.

Original: The coeﬃcient β (cid:0) 
Revised: The coeﬃcient β (cid:0) 2 σ

Original: [Equation]
Revised: [Equation]

Original:  of f ( x k ) in Lemma [-
-] is non-positive.
Revised: [Equation] [Equation] c (cid:1) of f ( x k ) in Lemma [-
-]13 is non-positive.

Original: Proof.
Revised: Proof.

Original: [Equation]
Revised: [Equation]

Original: Lemma 16.
Revised: Lemma 16.

Original: The coeﬃcient S − 1 (cid:16) σ − 1 + 2L 
Revised: The coeﬃcient S − 1 (cid:16) σ − 1 + 2L − 1 c 1 −

Original: [Equation]
Revised: [Equation]

Revised: ch

Original: [Equation]
Revised: [Equation]

Original: 2 −
Revised: 2 −

Original: [Equation]
Revised: [Equation]

Original:  + [Equation] −
Revised: 1 + 12 σ

Original: [Equation]
Revised: [Equation]

Original: ψ
Revised: ψ

Original: [Equation]
Revised: [Equation]

Original: of k∇ f
Revised: of k∇ f

Original: (ˆ y k ) k [Equation] in Lemma 13 is non-positive.
Revised: (ˆ y k ) k 2 ∗ in Lemma 13 is non-positive.

Original: Proof.
Revised: Proof.

Original: We ﬁrst need to bound c 1 .
Revised: We ﬁrst need to bound c 1 .

Original: (equation B.18 and equation 2.9) c
Revised: (equation B.18 and equation 2.9) c

Original: 1 = s
Revised: 1 = s

Original: [Equation]
Revised: [Equation]

Original: equation B.18 ≤ 6 S − 1 L 1 / 2 κ 3 / 2 ψ −1 τ
Revised: equation B.18 ≤ 6 S − 1 L 1 / 2 κ 3 / 2 ψ −

Original: [Equation]
Revised: [Equation]

Original: j

Original: [Equation] (cid:16) [Equation] It can be easily veriﬁed that if x ≤ 12 and y ≥ 0, then (1 − x ) − y ≤ exp(2 xy ).
Revised: It can be easily veriﬁed that if x ≤ 12 and y ≥ 0, then (1 − x ) − y ≤ exp(2 xy ).

Original: Using this fact with x = σ 1 / 2 S − 1 and y =
Revised: Using this fact with x = σ 1 / 2 S − 1 and y =

Original: τ [Equation] we have:
Revised: τ , we have:

Original: [Equation]
Revised: [Equation]

Original:  ψ ≤ 3 /7 and hence τσ 1 / 
Revised: since ψ ≤ 3 / and hence τσ 1 / 2 S

Original: [Equation]
Revised: [Equation]

Original: −

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: We now analyze the coeﬃcient of k∇ f
Revised: We now analyze the coeﬃcient of k∇ f

Original: (ˆ y k ) k 2 ∗
Revised: (ˆ y k ) k 2 ∗

Original: [Table]
Revised: [Table]

Original: [Equation] 
Revised: [Equation] [Table] [Equation] [Table]

Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: Lemma 17.
Revised: Lemma 17.

Original: The coeﬃcient β
Revised: The coeﬃcient β

Original: [Equation]
Revised: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: of k v k −
Revised: of k v k −

Original: y k k 2 in 13 is nonpositive.
Revised: y k k 2 in 13 is nonpositive.

Original: Proof.
Revised: Proof.

Original: [Equation]
Revised: [Equation]

Original: Here the last inequality follows since β ≤ 1 and α ≤ σ 1 / 2 S − 1 .
Revised: Here the last inequality follows since β ≤ 1 and α ≤ σ 1 / 2 S − 1 .

Original: We now rearrange the deﬁnition of ψ to yield the identity:
Revised: We now rearrange the deﬁnition of ψ to yield the identity:

Original: [Equation]
Revised: [Equation]

Original: Using this, we have:
Revised: Using this, we have:

Original: [Equation]
Revised: [Equation]

Original: Here the last line followed since L ≤ L , ψ ≤ 37 , and τ ≥ 1.
Revised: Here the last line followed since L ≤ L , ψ ≤ 37 , and τ ≥ 1.

Original: Hence the proof is complete.
Revised: Hence the proof is complete.

Original: Proof of Theorem 1.
Revised: Proof of Theorem 1.

Original: Using the master inequality 13 in combination with the previous Lemmas 14, 15,
Revised: Using the master inequality 13 in combination with the previous Lemmas 14, 15,

Original: [Equation]
Revised: [Equation]

Original: and 17, we have:
Revised: and 17, we have:

Original: [Equation]
Revised: [Equation]

Original: When we have:
Revised: When we have:

Original: [Equation]
Revised: [Equation]

Original: then the Lyapunov function ρ k has decreased below (cid:15)ρ 0 in expectation.
Revised: then the Lyapunov function ρ k has decreased below (cid:15)ρ 0 in expectation.

Original: Hence the complexity K ( (cid:15) ) satisﬁes:
Revised: Hence the complexity K ( (cid:15) ) satisﬁes:

Original: [Equation]
Revised: [Equation]

Original: Now it can be shown that for 0 < x ≤ 12 , we have:
Revised: Now it can be shown that for 0 < x ≤ 12 , we have:

Original: [Equation]
Revised: [Equation]

Original: Since n ≥
Revised: Since n ≥

Original: [Equation]
Revised: [Equation]

Original: we have σ 1 / 2 S − 1 ≤ 12 .
Revised: we have σ 1 / 2 S − 1 ≤ 12 .

Original: Hence:
Revised: Hence:

Original: [Equation]
Revised: [Equation]

Original: An expression for K NU_ACDM ((cid:15) )
Revised: An expression for K NU_ACDM (

Original: [Equation]
Revised: [Equation]

Original:  the complexity of NU_ACDM follows by similar reasoning.
Revised: ), the complexity of NU_ACDM follows by similar reasoning.

Original: [Equation]
Revised: [Equation]

Original: Finally we have:
Revised: Finally we have:

Original: [Equation]
Revised: [Equation]

Original: which completes the proof.
Revised: which completes the proof.

Original: C Ordinary Differential Equation Analysis C. Derivation of ODE for synchronous A2BCD
Revised: C Ordinary Differential Equation Analysis C. Derivation of ODE for synchronous A2BCD

Original: If we take expectations with respect to E k
Revised: If we take expectations with respect to E k

Original: [Equation]
Revised: [Equation]

Original: then synchronous (no delay) A2BCD becomes:
Revised: then synchronous (no delay) A2BCD becomes:

Original: [Equation]
Revised: [Equation]

Original: We ﬁnd it convenient to deﬁne η = nκ 1 / 2 .
Revised: We ﬁnd it convenient to deﬁne η = nκ 1 / 2 .

Original: Inspired by this
Revised: Inspired by this

Original: [Equation]
Revised: [Equation]

Original: we consider the following iteration:
Revised: we consider the following iteration:

Original: [Equation]
Revised: [Equation]

Original: for coeﬃcients:
Revised: for coeﬃcients:

Original: [Equation]

Original: β

Original: [Equation]
Revised: [Equation]

Original: s is a discretization scale parameter that will be sent to 0 to obtain an ODE analogue of synchronous
Revised: s is a discretization scale parameter that will be sent to 0 to obtain an ODE analogue of synchronous

Original: A2BCD .
Revised: A2BCD .

Original: We ﬁrst use equation B.6 to eliminate v k from from equation C.3.
Revised: We ﬁrst use equation B.6 to eliminate v k from from equation C.3.

Original: [Equation]
Revised: [Equation]

Original: We now eliminate x k using equation C.1:
Revised: We now eliminate x k using equation C.1:

Original: [Equation]
Revised: [Equation]

Original: Now to derive an ODE
Revised: Now to derive an ODE

Original: [Equation]
Revised: [Equation]

Original: we let y k
Revised: we let y k

Original: = Y (cid:0) ks 1 / 2 (cid:1) .
Revised: = Y (cid:0) ks 1 / 2 (cid:1) .

Original: Then ∇ f ( y k − 1 ) =
Revised: Then ∇ f ( y k − 1 ) =

Original: ∇ f ( y k )
Revised: ∇ f ( y k )

Original: + O (cid:0) s 1 / 2 (cid:1) .
Revised: + O (cid:0) s 1 / 2 (cid:1) .

Original: Hence the above becomes:
Revised: Hence the above becomes:

Original: [Equation]
Revised: [Equation]

Original: We now look at some of the terms in this equation to ﬁnd the highest-order dependence on s 
Revised: We now look at some of the terms in this equation to ﬁnd the highest-order dependence on s .

Original: [Equation]
Revised: [Equation]

Original: We also have:
Revised: We also have:

Original: [Equation]
Revised: [Equation]

Original: Hence using these facts on equation C.
Revised: Hence using these facts on equation C.

Original: [Equation]
Revised: [Equation]

Original: we have:
Revised: we have:

Original: [Equation]
Revised: [Equation]

Original: Taking the limit as s →
Revised: Taking the limit as s →

Original: [Equation]
Revised: [Equation]

Original: we obtain the ODE:
Revised: we obtain the ODE:

Original: [Equation]
Revised: [Equation]

Original: C. Convergence proof for synchronous ODE
Revised: C. Convergence proof for synchronous ODE

Original: [Equation]
Revised: [Equation]

Original: strong

Original: [Equation]

Original: equation B [Equation] [Equation] Hence we have E 0 ( t ) ≤ 0.
Revised: Hence we have E 0 ( t ) ≤ 0.

Original: Therefore E ( t ) ≤ E (0).
Revised: Therefore E ( t ) ≤ E (0).

Original: That is:
Revised: That is:

Original: [Equation]
Revised: [Equation]

Original: which implies:
Revised: which implies:

Original: [Equation]
Revised: [Equation]

Original: C. Asynchronicity error lemma
Revised: C. Asynchronicity error lemma

Original: This result is the continuous-time analogue of Lemma 11.
Revised: This result is the continuous-time analogue of Lemma 11.

Original: First notice that c (0) =
Revised: First notice that c (0) =

Original: c 0 and c ( τ ) = 0.
Revised: c 0 and c ( τ ) = 0.

Original: We also have:
Revised: We also have:

Original: [Equation]
Revised: [Equation]

Original: Hence using c ( τ ) = 0:
Revised: Hence using c ( τ ) = 0:

Original: [Equation]
Revised: [Equation]

Original: Now when x ≤
Revised: Now when x ≤

Original: [Equation]
Revised: [Equation]

Original: we have e − x
Revised: we have e − x

Original: 1 − e − x ≥ 12 x − 1 .
Revised: 1 − e − x ≥ 12 x − 1 .

Original: Hence when rτ ≤
Revised: Hence when rτ ≤

Original: [Equation]
Revised: [Equation]

Original: we have:
Revised: we have:

Original: [Equation]
Revised: [Equation]

Original: and the result easily follows.
Revised: and the result easily follows.

Original: C. Convergence analysis for the asynchronous ODE
Revised: C. Convergence analysis for the asynchronous ODE

Original: We consider the same energy as in the synchronous case (that is, the ODE in equation 3.1).
Revised: We consider the same energy as in the synchronous case (that is, the ODE in equation 3.1).

Original: Similar to before, we have:
Revised: Similar to before, we have:

Original: [Equation]
Revised: [Equation]

Original: where the ﬁnal equality follows from the proof in Section C.2.
Revised: where the ﬁnal equality follows from the proof in Section C.2.

Original: Hence
Revised: Hence

Original: [Equation]
Revised: [Equation]

Original: Now we present an inequality that is similar to equation 8.
Revised: Now we present an inequality that is similar to equation 8.

Original: Lemma 18.
Revised: Lemma 18.

Original: Let A, χ > 0 .
Revised: Let A, χ > 0 .

Original: Then:
Revised: Then:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Since ˆ Y ( t ) is a delayed version of Y ( t ), we have: ˆ Y ( t ) =
Revised: Since ˆ Y ( t ) is a delayed version of Y ( t ), we have: ˆ Y ( t ) =

Original: Y ( t − j ( t ))
Revised: Y ( t − j ( t ))

Original: for some function j ( t )
Revised: for some function j ( t )

Original: ≥(though this can be easily generalized to an inconsistent read).
Revised: ≥(though this can be easily generalized to an inconsistent read).

Original: Recall that for χ > 0, we have ab ≤ 12 (cid:0) χa 2 + χ − 1 b 2 (cid:1) .
Revised: Recall that for χ > 0, we have ab ≤ 12 (cid:0) χa 2 + χ − 1 b 2 (cid:1) .

Original: Hence
Revised: Hence

Original: [Equation]
Revised: [Equation]

Original: Holder’s inequality
Revised: (Holder’s inequality

Original: [Equation]
Revised: [Equation]

Original: We use this lemma twice on k Y k
Revised: We use this lemma twice on k Y k

Original: (cid:13)(cid:13)(cid:13)
Revised: (cid:13)(cid:13)(cid:13)

Original: ˆ Y − Y (cid:13)(cid:13)(cid:13) and (cid:13)(cid:13) ˙
Revised: ˆ Y − Y (cid:13)(cid:13)(cid:13) and (cid:13)(cid:13) ˙

Original: Y (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
Revised: Y (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)

Original: ˆ Y − Y (cid:13)(cid:13)(cid:13) in equation C.7 with χ = 2 L, A = k Y k and χ = 4 Lη − 1 , A = (cid:13)(cid:13) ˙
Revised: ˆ Y − Y (cid:13)(cid:13)(cid:13) in equation C.7 with χ = 2 L, A = k Y k and χ = 4 Lη − 1 , A = (cid:13)(cid:13) ˙

Original: Y (cid:13)(cid:13) respectively, to yield:e [Equation] =
Revised: Y (cid:13)(cid:13) respectively, to yield:

Original: [Equation]
Revised: [Equation]

Original: The proof of convergence is completed in Section 3.
Revised: The proof of convergence is completed in Section 3.

Original: D Optimality proof
Revised: D Optimality proof

Original: For parameter set σ, L 1 , . . .
Revised: For parameter set σ, L 1 , . . .

Original: , L n , n , we construct a block-separable function f on the space R bn (separated into n blocks of size b ), which will imply this lower bound.
Revised: , L n , n , we construct a block-separable function f on the space R bn (separated into n blocks of size b ), which will imply this lower bound.

Original: Deﬁne κ i =
Revised: Deﬁne κ i =

Original: L i /σ .
Revised: L i /σ .

Original: We deﬁne the matrix A i ∈ R b × b via:
Revised: We deﬁne the matrix A i ∈ R b × b via:

Original: [Equation]
Revised: [Equation]

Original: , for 
Revised: , for θ

Original: [Equation]
Revised: [Equation]

Revised: κ

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Hence we deﬁne f i on R b via:
Revised: Hence we deﬁne f i on R b via:

Original: [Equation]
Revised: [Equation]

Original: which is clearly σ -strongly convex and L i -Lipschitz on R b .
Revised: which is clearly σ -strongly convex and L i -Lipschitz on R b .

Original: From Lemma 8 of Lan & Zhou (2015), we know that this function has unique minimizer
Revised: From Lemma 8 of Lan & Zhou (2015), we know that this function has unique minimizer

Original: [Equation]
Revised: [Equation]

Original: Finally, we deﬁne f via:
Revised: Finally, we deﬁne f via:

Original: [Equation]
Revised: [Equation]

Original: Now let e ( i, j ) be the j th unit vector of the i th block of size b in R bn .
Revised: Now let e ( i, j ) be the j th unit vector of the i th block of size b in R bn .

Original: For I 1 , . . .
Revised: For I 1 , . . .

Original: , I n ∈ N , we deﬁne the subspaces
Revised: , I n ∈ N , we deﬁne the subspaces

Original: V i

Original: [Equation]
Revised: [Equation]

Original: I
Revised: I

Original: [Equation]

Original: [Equation]

Original: i,

Original: [Equation]

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: I
Revised: i

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: V ( I 1 , . .
Revised: V ( I 1 , . .

Original: , I n ) is the subspace with the ﬁrst I 1 components of block 1 nonzero, the ﬁrst I 2 components of block 2 nonzero, etc.
Revised: , I n ) is the subspace with the ﬁrst I 1 components of block 1 nonzero, the ﬁrst I 2 components of block 2 nonzero, etc.

Original: First notice that IC( V ( I 1 , . .
Revised: First notice that IC( V ( I 1 , . .

Original: , I n ))
Revised: , I n ))

Original: =
Revised: =

Original: V ( I 1 , . .
Revised: V ( I 1 , . .

Original: , I n ).
Revised: , I n ).

Original: Also, clearly, we have:
Revised: Also, clearly, we have:

Original: [Equation]
Revised: [Equation]

Original: ∇ i f is supported on the i th block, hence why all the other indices are 0.
Revised: ∇ i f is supported on the i th block, hence why all the other indices are 0.

Original: The patten of nonzeros in
Revised: The patten of nonzeros in

Original: A means that the gradient will have at most 1 more nonzero on the i th block (see Nesterov (2013)).
Revised: A means that the gradient will have at most 1 more nonzero on the i th block (see Nesterov (2013)).

Original: Let the initial point x 0 belong to V (cid:0) ¯ I 1 , . .
Revised: Let the initial point x 0 belong to V (cid:0) ¯ I 1 , . .

Original: , ¯ I n (cid:1) .
Revised: , ¯ I n (cid:1) .

Original: Let I K,i be the number of times we have had i k
Revised: Let I K,i be the number of times we have had i k

Original: = i for k = 0 , . . .
Revised: = i for k = 0 , . . .

Original: , K − 1.
Revised: , K − 1.

Original: By induction on condition 2 of Deﬁnition 4 using equation D.1, we have:
Revised: By induction on condition 2 of Deﬁnition 4 using equation D.1, we have:

Original: [Equation]
Revised: [Equation]

Original: Hence if x 0 , ( i ) ∈
Revised: Hence if x 0 , ( i ) ∈

Original: V i (0) and k ≤ b , then
Revised: V i (0) and k ≤ b , then

Original: [Equation]
Revised: [Equation]

Original: Therefore for all i , we have:
Revised: Therefore for all i , we have:

Original: [Equation]
Revised: [Equation]

Original: To evaluate this expectation, we note:
Revised: To evaluate this expectation, we note:

Original: [Equation]
Revised: [Equation]

Original: Hence
Revised: Hence

Original: [Equation]
Revised: [Equation]

Original: For any i , we may select the starting iterate x 0 by deﬁning its block j = 1 , . . . , n via:
Revised: For any i , we may select the starting iterate x 0 by deﬁning its block j = 1 , . . . , n via:

Original: [Equation]
Revised: [Equation]

Original: For such a choice of x 0 , we have
Revised: For such a choice of x 0 , we have

Original: [Equation]
Revised: [Equation]

Original: Hence for this choice of x 0 :
Revised: Hence for this choice of x 0 :

Original: [Equation]
Revised: [Equation]

Original: Now notice:
Revised: Now notice:

Original: [Equation]
Revised: [Equation]

Original: Hence
Revised: Hence

Original: [Equation]
Revised: [Equation]

Original: Now if we let b = 2 k , then we have:
Revised: Now if we let b = 2 k , then we have:

Original: [Equation]
Revised: [Equation]

Original: Now let us take the minimum of the right-hand side over the parameters p i , subject to P ni =1 p i = 1.
Revised: Now let us take the minimum of the right-hand side over the parameters p i , subject to P ni =1 p i = 1.

Original: The solution to this minimization is clearly:
Revised: The solution to this minimization is clearly:

Original: [Equation]
Revised: [Equation]

Original: Hence
Revised: Hence

Original: [Equation]
Revised: [Equation]

Original: Hence the complexity I ( (cid:15) ) satisﬁes:
Revised: Hence the complexity I ( (cid:15) ) satisﬁes:

Original: [Equation]
Revised: [Equation]

Original: E Extensions
Revised: E Extensions

Original: As mentioned
Revised: As mentioned

Original: [Equation]
Revised: [Equation]

Original: a stronger result than Theorem 1 is possible.
Revised: a stronger result than Theorem 1 is possible.

Original: In the case when L i =
Revised: In the case when L i =

Original: L for all
Revised: L for all

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: we can consider a slight modiﬁcation of the coeﬃcients:
Revised: we can consider a slight modiﬁcation of the coeﬃcients:

Original: [Equation]
Revised: [Equation]

Original: for the asynchronicity parameter:
Revised: for the asynchronicity parameter:

Original: [Equation]
Revised: [Equation]

Original: This leads to complexity:
Revised: This leads to complexity:

Original: [Equation]
Revised: [Equation]

Original: Here there is no restriction on ψ as in Theorem 1, and hence there is no restriction on τ .
Revised: Here there is no restriction on ψ as in Theorem 1, and hence there is no restriction on τ .

Original: Assuming ψ ≤ 1 gives optimal complexity to within a constant factor.
Revised: Assuming ψ ≤ 1 gives optimal complexity to within a constant factor.

Original: Notice then that the resulting condition of τ
Revised: Notice then that the resulting condition of τ

Original: [Equation]
Revised: [Equation]

Original: now essentially matches the one in Theorem 3 in Section 3.
Revised: now essentially matches the one in Theorem 3 in Section 3.

Original: While this result is stronger, it increases the complexity of the proof substantially.
Revised: While this result is stronger, it increases the complexity of the proof substantially.

Original: So in the interests of space and simplicity, we do not prove this stronger result.
Revised: So in the interests of space and simplicity, we do not prove this stronger result.