Original: T OWARD U NDERSTANDING THE I MPACT OF S TALENESS IN D ISTRIBUTED M ACHINE L EARNING
Revised: T OWARD U NDERSTANDING THE I MPACT OF S TALENESS IN D ISTRIBUTED M ACHINE L EARNING

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Most distributed machine learning (ML) systems store a copy of the model parameters locally on each machine to minimize network communication.
Revised: Most distributed machine learning (ML) systems store a copy of the model parameters locally on each machine to minimize network communication.

Original: In practice, in order to reduce synchronization waiting time, these copies of the model are not necessarily updated in lock-step, and can become stale .
Revised: In practice, in order to reduce synchronization waiting time, these copies of the model are not necessarily updated in lock-step, and can become stale .

Original: Despite much development in large-scale ML, the effect of staleness on the learning efﬁciency is inconclusive, mainly because it is challenging to control or monitor the staleness in complex distributed environments.
Revised: Despite much development in large-scale ML, the effect of staleness on the learning efﬁciency is inconclusive, mainly because it is challenging to control or monitor the staleness in complex distributed environments.

Original: In this work, we study the convergence behaviors of a wide array of ML models and algorithms under delayed updates.
Revised: In this work, we study the convergence behaviors of a wide array of ML models and algorithms under delayed updates.

Original: Our extensive experiments reveal the rich diversity of the effects of staleness on the convergence of ML algorithms and offer insights into seemingly contradictory reports in the literature.
Revised: Our extensive experiments reveal the rich diversity of the effects of staleness on the convergence of ML algorithms and offer insights into seemingly contradictory reports in the literature.

Original: The empirical ﬁndings also inspire a new convergence analysis of SGD in non-convex optimization under staleness, matching the best-known convergence rate of O (1 / √ T ) .
Revised: The empirical ﬁndings also inspire a new convergence analysis of SGD in non-convex optimization under staleness, matching the best-known convergence rate of O (1 / √ T ) .

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: With the advent of big data and complex models, there is a growing body of works on scaling machine learning under synchronous and non-synchronous 1 distributed execution (Dean et al., 2012; Goyal et al., 2017; Li et al., 2014a).
Revised: With the advent of big data and complex models, there is a growing body of works on scaling machine learning under synchronous and non-synchronous 1 distributed execution (Dean et al., 2012; Goyal et al., 2017; Li et al., 2014a).

Original: These works, however, point to seemingly contradictory conclusions on whether non-synchronous execution outperforms synchronous counterparts in terms of absolute convergence, which is measured by the wall clock time to reach the desired model quality.
Revised: These works, however, point to seemingly contradictory conclusions on whether non-synchronous execution outperforms synchronous counterparts in terms of absolute convergence, which is measured by the wall clock time to reach the desired model quality.

Original: For deep neural networks, Chilimbi et al.
Revised: For deep neural networks, Chilimbi et al.

Original: (2014); Dean et al.
Revised: (2014); Dean et al.

Original: (2012) show that fully asynchronous systems achieve high scalability and model quality, but others argue that synchronous training converges faster (Chen et al., 2016; Cui et al., 2016).
Revised: (2012) show that fully asynchronous systems achieve high scalability and model quality, but others argue that synchronous training converges faster (Chen et al., 2016; Cui et al., 2016).

Original: The disagreement goes beyond deep learning models: Ho et al.
Revised: The disagreement goes beyond deep learning models: Ho et al.

Original: ; Zhang & Kwok (2014); Langford et al.
Revised: ; Zhang & Kwok (2014); Langford et al.

Original: ; Lian et al.
Revised: ; Lian et al.

Original: ; Recht et al.
Revised: ; Recht et al.

Original: (2011) empirically and theoretically show that many algorithms scale effectively under non-synchronous settings, but McMahan & Streeter (2014); Mitliagkas et al.
Revised: (2011) empirically and theoretically show that many algorithms scale effectively under non-synchronous settings, but McMahan & Streeter (2014); Mitliagkas et al.

Original: ; Hadjis et al.
Revised: ; Hadjis et al.

Original: (2016) demonstrate signiﬁcant penalties from asynchrony.
Revised: (2016) demonstrate signiﬁcant penalties from asynchrony.

Original: The crux of the disagreement lies in the trade-off between two factors contributing to the absolute convergence: statistical efﬁciency and system throughput .
Revised: The crux of the disagreement lies in the trade-off between two factors contributing to the absolute convergence: statistical efﬁciency and system throughput .

Original: Statistical efﬁciency measures convergence per algorithmic step (e.g., a mini-batch), while system throughput captures the performance of the underlying implementation and hardware.
Revised: Statistical efﬁciency measures convergence per algorithmic step (e.g., a mini-batch), while system throughput captures the performance of the underlying implementation and hardware.

Original: Non-synchronous execution can improve system throughput due to lower synchronization overheads, which is well understood (Ho et al., 2013;
Revised: Non-synchronous execution can improve system throughput due to lower synchronization overheads, which is well understood (Ho et al., 2013;

Original: Chen et al., 2016; Cui et al., 2014; Chilimbi et al., 2014).
Revised: Chen et al., 2016; Cui et al., 2014; Chilimbi et al., 2014; Dai et al., 2015).

Original: However, by allowing various workers to use stale versions of the model that do not always reﬂect the latest updates, non-synchronous systems can exhibit lower statistical efﬁciency (Chen et al., 2016; Cui et al., 2016).
Revised: However, by allowing various workers to use stale versions of the model that do not always reﬂect the latest updates, non-synchronous systems can exhibit lower statistical efﬁciency (Chen et al., 2016; Cui et al., 2016).

Original: How statistical efﬁciency and system throughput trade off in distributed systems, however, is far from clear.
Revised: How statistical efﬁciency and system throughput trade off in distributed systems, however, is far from clear.

Original: The difﬁculties in understanding the trade-off arise because statistical efﬁciency and system throughput are coupled during execution in distributed environments.
Revised: The difﬁculties in understanding the trade-off arise because statistical efﬁciency and system throughput are coupled during execution in distributed environments.

Original: Non-synchronous executions are in general non-deterministic, which can be difﬁcult to proﬁle.
Revised: Non-synchronous executions are in general non-deterministic, which can be difﬁcult to proﬁle.

Original: Furthermore, large-scale experiments are sensitive to the underlying hardware and software artifacts, which confounds the comparison between studies.
Revised: Furthermore, large-scale experiments are sensitive to the underlying hardware and software artifacts, which confounds the comparison between studies.

Original: Even when they are controlled, innocuous change in the system conﬁgurations such as adding more machines or sharing resources with other workloads can inadvertently alter the underlying staleness levels experienced by ML algorithms, masking the true effects of staleness.
Revised: Even when they are controlled, innocuous change in the system conﬁgurations such as adding more machines or sharing resources with other workloads can inadvertently alter the underlying staleness levels experienced by ML algorithms, masking the true effects of staleness.

Original: Understanding the impact of staleness on ML convergence independently from the underlying distributed systems is a crucial step towards decoupling statistical efﬁciency from the system complexity.
Revised: Understanding the impact of staleness on ML convergence independently from the underlying distributed systems is a crucial step towards decoupling statistical efﬁciency from the system complexity.

Original: The gleaned insights can also guide distributed ML system development, potentially using different synchronization for different problems.
Revised: The gleaned insights can also guide distributed ML system development, potentially using different synchronization for different problems.

Original: In particular, we are interested in the following aspects: Do
Revised: In particular, we are interested in the following aspects: Do

Original: ML algorithms converge under staleness?
Revised: ML algorithms converge under staleness?

Original: To what extent does staleness impact the convergence?
Revised: To what extent does staleness impact the convergence?

Original: By resorting to simulation study, we side step the challenges faced in distributed execution.
Revised: By resorting to simulation study, we side step the challenges faced in distributed execution.

Original: We study the impact of staleness on a diverse set of models: Convolutional Neural Networks (CNNs), recurrent neural networks (RNNs), Deep Neural Networks (DNNs), multi-class Logistic Regression (MLR),
Revised: We study the impact of staleness on a diverse set of models: Convolutional Neural Networks (CNNs), recurrent neural networks (RNNs), Deep Neural Networks (DNNs), multi-class Logistic Regression (MLR),

Original: Matrix Factorization (MF), Latent Dirichlet Allocation (LDA), and Variational Autoencoders (VAEs).
Revised: Matrix Factorization (MF), Latent Dirichlet Allocation (LDA), and Variational Autoencoders (VAEs).

Original: They are addressed by 7 algorithms, spanning across optimization, sampling, and blackbox variational inference.
Revised: They are addressed by 7 algorithms, spanning across optimization, sampling, and blackbox variational inference.

Original: Our ﬁndings suggest that while some algorithms are more robust to staleness, no ML method is immune to the negative impact of staleness.
Revised: Our ﬁndings suggest that while some algorithms are more robust to staleness, no ML method is immune to the negative impact of staleness.

Original: We ﬁnd that all investigated algorithms reach the target model quality under moderate levels of staleness, but the convergence can progress very slowly or fail under high staleness levels.
Revised: We ﬁnd that all investigated algorithms reach the target model quality under moderate levels of staleness, but the convergence can progress very slowly or fail under high staleness levels.

Original: The effects of staleness are also problem dependent.
Revised: The effects of staleness are also problem dependent.

Original: For
Revised: For

Original: CNNs, DNNs, and RNNs, the staleness slows down deeper models more than shallower counterparts.
Revised: CNNs, DNNs, and RNNs, the staleness slows down deeper models more than shallower counterparts.

Original: For MLR, a convex objective, staleness has minimal effect.
Revised: For MLR, a convex objective, staleness has minimal effect.

Original: Different algorithms respond to staleness very differently.
Revised: Different algorithms respond to staleness very differently.

Original: For example, high staleness levels incur more statistical penalty for Momentum methods than stochastic gradient descent (SGD) and Adagrad (Duchi et al., 2011).
Revised: For example, high staleness levels incur more statistical penalty for Momentum methods than stochastic gradient descent (SGD) and Adagrad (Duchi et al., 2011).

Original: Separately, Gibbs sampling for LDA is highly resistant to staleness up to a certain level, beyond which it does not converge to a ﬁxed point.
Revised: Separately, Gibbs sampling for LDA is highly resistant to staleness up to a certain level, beyond which it does not converge to a ﬁxed point.

Original: Overall, it appears that staleness is a key governing parameter of ML convergence.
Revised: Overall, it appears that staleness is a key governing parameter of ML convergence.

Original: To gain deeper insights, for gradient-based methods we further introduce gradient coherence along the optimization path, and show that gradient coherence is a possible explanation for an algorithm’s sensitivity to staleness.
Revised: To gain deeper insights, for gradient-based methods we further introduce gradient coherence along the optimization path, and show that gradient coherence is a possible explanation for an algorithm’s sensitivity to staleness.

Original: In particular, our theoretical result establishes the O (1 / √ T ) convergence rate of the asynchronous SGD in nonconvex optimization by exploiting gradient coherence, matching the rate of best-known results (Lian et al., 2015).
Revised: In particular, our theoretical result establishes the O (1 / √ T ) convergence rate of the asynchronous SGD in nonconvex optimization by exploiting gradient coherence, matching the rate of best-known results (Lian et al., 2015).

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Staleness is reported to help absolute convergence for distributed deep learning in Chilimbi et al.
Revised: Staleness is reported to help absolute convergence for distributed deep learning in Chilimbi et al.

Original: (2014); Dean et al.
Revised: (2014); Dean et al.

Original:  and has minimal impact on convergence (Mitliagkas et al., 2016; 
Revised: Xing et al. and has minimal impact on convergence (Mitliagkas et al., 2016; Hadjis et al., 2016; Lian et al., 2015;

Original: Hadjis et al., 2016; Lian et al., 2015).
Revised: Dai et al., 2013; Zhou et al., 2018; 2016).

Original: But Chen et al.
Revised: But Chen et al.

Original: ; Cui et al.
Revised: ; Cui et al.

Original: (2016) show signiﬁcant negative effects of staleness.
Revised: (2016) show signiﬁcant negative effects of staleness.

Original: LDA training is generally insensitive to staleness (Smola & Narayanamurthy, 2010; Yuan et al., 2015; Wei et al., 2015;
Revised: LDA training is generally insensitive to staleness (Smola & Narayanamurthy, 2010; Yuan et al., 2015; Wei et al., 2015;

Original: Ho et al., 2013), and so is MF training (Yun et al., 2013;
Revised: Ho et al., 2013), and so is MF training (Yun et al., 2013;

Original: Low et al., 2012; Cui et al., 2014; Zhang & Kwok, 2014).
Revised: Low et al., 2012; Cui et al., 2014; Zhang & Kwok, 2014).

Original: However, none of their evaluations quantiﬁes the level of staleness in the systems.
Revised: However, none of their evaluations quantiﬁes the level of staleness in the systems.

Original: By explicitly controlling the staleness, we decouple the distributed execution, which is hard to control, from ML convergence outcomes.
Revised: By explicitly controlling the staleness, we decouple the distributed execution, which is hard to control, from ML convergence outcomes.

Original: We focus on algorithms that are commonly used in large-scale optimization (Goyal et al., 2017;
Revised: We focus on algorithms that are commonly used in large-scale optimization (Goyal et al., 2017;

Original: Chen et al., 2016; Dean et al., 2012), instead of methods speciﬁcally designed to minimize synchronization (Neiswanger et al., 2013; Scott et al., 2016; Jordan et al., 2013).
Revised: Chen et al., 2016; Dean et al., 2012), instead of methods speciﬁcally designed to minimize synchronization (Neiswanger et al., 2013; Scott et al., 2016; Jordan et al., 2013).

Original: Non-synchronous execution has theoretical underpinning (Li et al., 2014b; Ho et al., 2013; Zhang & Kwok, 2014; Lian et al., 2015;
Revised: Non-synchronous execution has theoretical underpinning (Li et al., 2014b; Ho et al., 2013; Zhang & Kwok, 2014; Lian et al., 2015;

Original: Recht et al., 2011).
Revised: Recht et al., 2011).

Original: Here we study algorithms that do not necessarily satisfy assumptions in their analyses.
Revised: Here we study algorithms that do not necessarily satisfy assumptions in their analyses.

Original:  M ETHODS
Revised:  M ETHODS

Original: We study six ML models and focus on algorithms that lend itself to data parallelism, which a primary approach for distributed ML.
Revised: We study six ML models and focus on algorithms that lend itself to data parallelism, which a primary approach for distributed ML.

Original: Our algorithms span optimization, sampling, and black box variational inference.
Revised: Our algorithms span optimization, sampling, and black box variational inference.

Original: Table 1 summarizes the studied models and algorithms.
Revised: Table 1 summarizes the studied models and algorithms.

Original: Simulation Model.
Revised: Simulation Model.

Original: Each update generated by worker p needs to be propagated to both worker p ’s model cache and other worker’s model cache.
Revised: Each update generated by worker p needs to be propagated to both worker p ’s model cache and other worker’s model cache.

Original: We apply a uniformly random delay model to these updates that are in transit.
Revised: We apply a uniformly random delay model to these updates that are in transit.

Original: Speciﬁcally, let u t p be the update generated at iteration t by worker p .
Revised: Speciﬁcally, let u tp be the update generated at iteration t by worker p .

Original: For each worker p (cid:48) (including p itself), our delay model applies a delay r tp,p (cid:48) ∼ Categorical (0 , 1 , .., s ) , where s is the maximum delay and Categorical () is the categorical distribution placing equal weights on each integer 2 .
Revised: For each worker p (cid:48) (including p itself), our delay model applies a delay r tp,p (cid:48) ∼ Categorical (0 , 1 , .., s ) , where s is the maximum delay and Categorical () is the categorical distribution placing equal weights on each integer 2 .

Original: Under this delay model, update u tp shall arrive at worker p (cid:48) at the start of iteration t
Revised: Under this delay model, update u tp shall arrive at worker p (cid:48) at the start of iteration t

Original: + 1 + r tp,p (cid:48) .
Revised: + 1 + r tp,p (cid:48) .

Original: The average delay under this model is 12 s + 1 .
Revised: The average delay under this model is 12 s + 1 .

Original: Notice that for one worker with s =
Revised: Notice that for one worker with s =

Original: we reduce to the sequential setting.
Revised: we reduce to the sequential setting.

Original: Since model caches on each worker are symmetric, we use the ﬁrst worker’s model to evaluate the model quality.
Revised: Since model caches on each worker are symmetric, we use the ﬁrst worker’s model to evaluate the model quality.

Original: Finally, we are most interested in measuring convergence against the logical time, and wall clock time is in general immaterial as the simulation on a single machine is not optimized for performance
Revised: Finally, we are most interested in measuring convergence against the logical time [Table] and wall clock time is in general immaterial as the simulation on a single machine is not optimized for performance

Original: [Table]
Revised: [Table]

Original: 3. M ODELS AND A LGORITHMS
Revised: 3. M ODELS AND A LGORITHMS

Revised: [Table]

Original: [Table]
Revised: [Table]

Original: Convolutional Neural Networks (CNNs) have been a strong focus of large-scale training, both under synchronous (Goyal et al., 2017;
Revised: Convolutional Neural Networks (CNNs) have been a strong focus of large-scale training, both under synchronous (Goyal et al., 2017;

Original: Cui et al., 2016; Coates et al., 2013) and non-synchronous (Chilimbi et al., 2014;
Revised: Cui et al., 2016; Coates et al., 2013) and non-synchronous (Chilimbi et al., 2014;

Original: Dean et al., 2012; Chen et al., 2016; Hadjis et al., 2016) training.
Revised: Dean et al., 2012; Chen et al., 2016; Hadjis et al., 2016) training.

Original: We consider residual networks with 6 n + 2 weight layers (He et al., 2016).
Revised: We consider residual networks with 6 n + 2 weight layers (He et al., 2016).

Original: The networks consist of 3 groups of n residual blocks, with 16, 32, and 64 feature maps in each group, respectively, followed by a global pooling layer and a softmax layer.
Revised: The networks consist of 3 groups of n residual blocks, with 16, 32, and 64 feature maps in each group, respectively, followed by a global pooling layer and a softmax layer.

Original: The residual blocks have the same construction as in (He et al., 2016).
Revised: The residual blocks have the same construction as in (He et al., 2016).

Original: We measure the model quality using test accuracy.
Revised: We measure the model quality using test accuracy.

Original: For simplicity, we omit data augmentation in our experiments.
Revised: For simplicity, we omit data augmentation in our experiments.

Original: Deep Neural Networks (DNNs) are neural networks composed of fully connected layers.
Revised: Deep Neural Networks (DNNs) are neural networks composed of fully connected layers.

Original: Our DNNs have 1 to 6 hidden layers, with 256 neurons in each layer, followed by a softmax layer.
Revised: Our DNNs have 1 to 6 hidden layers, with 256 neurons in each layer, followed by a softmax layer.

Original: We use rectiﬁed linear units (ReLU) for nonlinearity after each hidden layer (Nair & Hinton, 2010).
Revised: We use rectiﬁed linear units (ReLU) for nonlinearity after each hidden layer (Nair & Hinton, 2010).

Original: Multiclass Logistic Regression (MLR) is the special case of DNN with 0 hidden layers.
Revised: Multiclass Logistic Regression (MLR) is the special case of DNN with 0 hidden layers.

Original: We measure the model quality using test accuracy.
Revised: We measure the model quality using test accuracy.

Original: Matrix factorization (MF) is commonly used in recommender systems and have been implemented at scale (Yun et al., 2013;
Revised: Matrix factorization (MF) is commonly used in recommender systems and have been implemented at scale (Yun et al., 2013;

Original: Low et al., 2012; Cui et al., 2014; Zhang & Kwok, 2014; Kim et al., 2016;
Revised: Low et al., 2012; Cui et al., 2014; Zhang & Kwok, 2014; Kim et al., 2016;

Original: Ho et al., 2013; Kumar et al., 2014).
Revised: Ho et al., 2013; Kumar et al., 2014).

Original: Let D ∈ R M × N be a partially ﬁlled matrix, MF factorizes D into two factor matrices L ∈ R M × r and R ∈ R N × r ( r (cid:28) min( M, N ) is the user-deﬁned rank).
Revised: Let D ∈ R M × N be a partially ﬁlled matrix, MF factorizes D into two factor matrices L ∈ R M × r and R ∈ R N × r ( r (cid:28) min( M, N ) is the user-deﬁned rank).

Original: The (cid:96) 2 -penalized optimization problem is: min L,R 
Revised: The (cid:96) 2 -penalized optimization problem is: min L,R 1 | D obs | (cid:110)(cid:80) ( i, j ) ∈ D obs || D ij − (cid:80)

Original: [Equation]{+Kk = 1 L ik R kj || 
 + λ ( || L+}
Revised: {+Kk = 1 L ik R kj || 
 + λ ( || L+}

Original: | R || 2 F ) (cid:111) where || · || F is the
Revised: || 2 F + || R || 2 F ) (cid:111) where || · || F is the

Original: Frobenius norm and λ is the regularization parameter.
Revised: Frobenius norm and λ is the regularization parameter.

Original: We partition observations D to workers while treating L, R as shared model parameters.
Revised: We partition observations D to workers while treating L, R as shared model parameters.

Original: We optimize MF via SGD, and measure model quality by training loss deﬁned by the objective function above.
Revised: We optimize MF via SGD, and measure model quality by training loss deﬁned by the objective function above.

Original: Latent Dirichlet Allocation (LDA) is an unsupervised method to uncover hidden semantics (“topics”) from a group of documents, each represented as a bag of tokens.
Revised: Latent Dirichlet Allocation (LDA) is an unsupervised method to uncover hidden semantics (“topics”) from a group of documents, each represented as a bag of tokens.

Original: LDA has been scaled under non-synchronous execution (Ahmed et al., 2012;
Revised: LDA has been scaled under non-synchronous execution (Ahmed et al., 2012;

Original: Low et al., 2012; Yuan et al., 2015) with great success.
Revised: Low et al., 2012; Yuan et al., 2015) with great success.

Original: Further details are provided in Appendix.
Revised: Further details are provided in Appendix.

Original: Variational Autoencoder (VAE) is commonly optimized by black box variational inference, which can be considered as a hybrid of optimization and sampling methods.
Revised: Variational Autoencoder (VAE) is commonly optimized by black box variational inference, which can be considered as a hybrid of optimization and sampling methods.

Original: The inputs to VAE training include two sources of stochasticity: the data sampling x and samples of random variable (cid:15) .
Revised: The inputs to VAE training include two sources of stochasticity: the data sampling x and samples of random variable (cid:15) .

Original: We measure the model quality by test loss.
Revised: We measure the model quality by test loss.

Original: We use DNNs with 1 ∼ 3 layers as the encoders and decoders in VAE, in which each layer has 256 units furnished with rectiﬁed linear function for non-linearity.
Revised: We use DNNs with 1 ∼ 3 layers as the encoders and decoders in VAE, in which each layer has 256 units furnished with rectiﬁed linear function for non-linearity.

Original: The model quality is measured by the training objective value, assuming continuous input x and isotropic Gaussian prior p ( z ) ∼ N (0 , I ) .
Revised: The model quality is measured by the training objective value, assuming continuous input x and isotropic Gaussian prior p ( z ) ∼ N (0 , I ) .

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We use batch size 32 for CNNs, DNNs, MLR, and VAEs 34 .
Revised: We use batch size 32 for CNNs, DNNs, MLR, and VAEs 34 .

Original: For MF, we use batch size ofsamples, which is 2.5% of the MovieLens dataset (1M samples).
Revised: For MF, we use batch size ofsamples, which is 2.5% of the MovieLens dataset (1M samples).

Original: We study staleness up to s = 50 on 8 workers, which means model caches can miss updates up to 8.75 data passes.
Revised: We study staleness up to s = 50 on 8 workers, which means model caches can miss updates up to 8.75 data passes.

Original: For LDA we use D 10 P as the batch size, where D is the number of documents and P is the number of workers.
Revised: For LDA we use D 10 P as the batch size, where D is the number of documents and P is the number of workers.

Original: We study staleness up to s = 20 , which means model caches can miss updates up to 2 data passes.
Revised: We study staleness up to s = 20 , which means model caches can miss updates up to 2 data passes.

Original: We measure time in terms of the amount of work performed, such as the number of batches processed.[Figure] r [Figure]
Revised: We measure time in terms of the amount of work performed, such as the number of batches processed.

Original: f

Original: [Figure]
Revised: [Figure]

Original: t

Original: [Figure]

Original: h

Original: [Figure]

Original: u r a

Original: [Figure]

Original: y

Original: [Figure]
Revised: [Figure]

Original: r m a li z e d N u m b e r o f B a t c h e s t o R e a c
Revised: A

Original: h% T e s t

Original: A cc u r a c y (b)

Original: [Figure]
Revised: [Figure]

Original: t c

Original: [Figure]

Original: t

Original: [Figure]

Original: R e a c

Original: [Figure]

Original: T e s t

Original: [Figure]

Original: u r a c 
Revised: a c y

Original: [Figure]
Revised: [Figure]

Original: Convergence Slowdown.
Revised: Convergence Slowdown.

Original: Perhaps the most prominent effect of staleness on ML algorithms is the slowdown in convergence, evident throughout the experiments.
Revised: Perhaps the most prominent effect of staleness on ML algorithms is the slowdown in convergence, evident throughout the experiments.

Original: Fig.
Revised: Fig.

Original: 1 shows the number of batches needed to reach the desired model quality for CNNs and DNNs/MLR with varying network depths and different staleness ( s = 0 , ..., 16 ).
Revised: 1 shows the number of batches needed to reach the desired model quality for CNNs and DNNs/MLR with varying network depths and different staleness ( s = 0 , ..., 16 ).

Original: Fig.
Revised: Fig.

Original: 1(b)(d) show that convergence under higher level of staleness requires more batches to be processed in order to reach the same model quality.
Revised: 1(b)(d) show that convergence under higher level of staleness requires more batches to be processed in order to reach the same model quality.

Original: This additional work can potentially be quite substantial, such as in Fig. 1(d) where it takes up to 6x more batches compared with settings without staleness ( s = 0 ).
Revised: This additional work can potentially be quite substantial, such as in Fig. 1(d) where it takes up to 6x more batches compared with settings without staleness ( s = 0 ).

Original: It is also worth pointing out that while there can be a substantial slowdown in convergence, the optimization still reaches desirable models under most cases in our experiments.
Revised: It is also worth pointing out that while there can be a substantial slowdown in convergence, the optimization still reaches desirable models under most cases in our experiments.

Original: When staleness is geometrically distributed (Fig. 4(c)), we observe similar patterns of convergence slowdown.
Revised: When staleness is geometrically distributed (Fig. 4(c)), we observe similar patterns of convergence slowdown.

Original: We are not aware of any prior work reporting slowdown as high as observed here.
Revised: We are not aware of any prior work reporting slowdown as high as observed here.

Original: This ﬁnding has important ramiﬁcations for distributed ML.
Revised: This ﬁnding has important ramiﬁcations for distributed ML.

Original: Usually, the moderate amount of workload increases due to parallelization errors can be compensated by the additional computation resources and higher system throughput in the distributed execution.
Revised: Usually, the moderate amount of workload increases due to parallelization errors can be compensated by the additional computation resources and higher system throughput in the distributed execution.

Original: However, it may be difﬁcult to justify spending large amount of resources for a distributed implementation if the statistical penalty is too high, which should be avoided (e.g., by staleness minimization system designs or synchronous execution).
Revised: However, it may be difﬁcult to justify spending large amount of resources for a distributed implementation if the statistical penalty is too high, which should be avoided (e.g., by staleness minimization system designs or synchronous execution).

Original: Model Complexity.
Revised: Model Complexity.

Original: Fig.
Revised: Fig.

Original: 1 also reveals that the impact of staleness can depend on ML parameters, such as the depths of the networks.
Revised: 1 also reveals that the impact of staleness can depend on ML parameters, such as the depths of the networks.

Original: Overall we observe that staleness impacts deeper networks more schemes, generalizes to other numbers of workers (
Revised: Overall we observe that staleness impacts deeper networks more than shallower ones.

Original:  Appendix) 5 .
Revised: This holds true for SGD, Adam, Momentum, RMSProp, Adagrad (Fig. 1), and other optimization schemes, and generalizes to other numbers of workers (see Appendix) 5 .

Original: This is perhaps not surprising, given the fact that deeper models pose more optimization challenges even under the sequential settings (Glorot & Bengio, 2010; He et al., 2016), though we point out that existing literature does not explicitly consider model complexity as a factor in distributed ML (Lian et al., 2015; Goyal et al., 2017).
Revised: This is perhaps not surprising, given the fact that deeper models pose more optimization challenges even under the sequential settings (Glorot & Bengio, 2010; He et al., 2016), though we point out that existing literature does not explicitly consider model complexity as a factor in distributed ML (Lian et al., 2015; Goyal et al., 2017).

Original: Our results suggest that the staleness level acceptable in distributed training can depend strongly on the complexity of the model.
Revised: Our results suggest that the staleness level acceptable in distributed training can depend strongly on the complexity of the model.

Original: For sufﬁciently complex models it may be more advantageous to eliminate staleness altogether and use synchronous training.
Revised: For sufﬁciently complex models it may be more advantageous to eliminate staleness altogether and use synchronous training.

Original: Algorithms’ Sensitivity to Staleness.
Revised: Algorithms’ Sensitivity to Staleness.

Original: Staleness has uneven impacts on different SGD variants.
Revised: Staleness has uneven impacts on different SGD variants.

Original: Fig.
Revised: Fig.

Original: 2 shows the amount of work (measured in the number of batches) to reach the desired model quality for ﬁve SGD variants.
Revised: 2 shows the amount of work (measured in the number of batches) to reach the desired model quality for ﬁve SGD variants.

Original: Fig.
Revised: Fig.

Original: 2(d)(e)(f) reveals that while staleness generally increases the number of batches needed to reach the target test accuracy, the increase can be higher for certain algorithms, such as Momentum.
Revised: 2(d)(e)(f) reveals that while staleness generally increases the number of batches needed to reach the target test accuracy, the increase can be higher for certain algorithms, such as Momentum.

Original: On the other hand, Adagrad appear to be robust to staleness 6 .
Revised: On the other hand, Adagrad appear to be robust to staleness 6 .

Original: Our ﬁnding is consistent with the fact that, to our knowledge, all existing successful cases applying non-synchronous training to deep neural networks use SGD (Dean et al., 2012; Chilimbi et al., 2014).
Revised: Our ﬁnding is consistent with the fact that, to our knowledge, all existing successful cases applying non-synchronous training to deep neural networks use SGD (Dean et al., 2012; Chilimbi et al., 2014).

Original: In contrast, works reporting subpar performance from non-synchronous training often use momentum, such as RMSProp with momentum (Chen et al., 2016) and momentum (Cui et al., 2016).
Revised: In contrast, works reporting subpar performance from non-synchronous training often use momentum, such as RMSProp with momentum (Chen et al., 2016) and momentum (Cui et al., 2016).

Original: Our results suggest that these different outcomes may be partly driven by the choice of optimization algorithms, leading to the seemingly contradictory reports of whether non-synchronous execution is advantageous over synchronous ones.
Revised: Our results suggest that these different outcomes may be partly driven by the choice of optimization algorithms, leading to the seemingly contradictory reports of whether non-synchronous execution is advantageous over synchronous ones.

Original: Effects of More Workers.
Revised: Effects of More Workers.

Original: The impact of staleness is ampliﬁed by the number of workers.
Revised: The impact of staleness is ampliﬁed by the number of workers.

Original: In the case of MF, Fig. 3(b) shows that the convergence slowdown in terms of the number of batches (normalized by the convergence for s = 0 ) on 8 workers is more than twice of the slowdown on 4 workers.
Revised: In the case of MF, Fig. 3(b) shows that the convergence slowdown in terms of the number of batches (normalized by the convergence for s = 0 ) on 8 workers is more than twice of the slowdown on 4 workers.

Original: For example, in Fig.
Revised: For example, in Fig.

Original: 3(b) the slowdown at s = 15 is ∼ 3.4, but the slowdown at the same staleness level on 8 workers is ∼ 8.2.
Revised: 3(b) the slowdown at s = 15 is ∼ 3.4, but the slowdown at the same staleness level on 8 workers is ∼ 8.2.

Original: Similar observations can be made for CNNs (Fig. 3).
Revised: Similar observations can be made for CNNs (Fig. 3).

Original: This can be explained by the fact that additional workers ampliﬁes the effect of staleness by (1) generating updates that will be subject to delays, and (2) missing updates from other workers that are subject to delays.
Revised: This can be explained by the fact that additional workers ampliﬁes the effect of staleness by (1) generating updates that will be subject to delays, and (2) missing updates from other workers that are subject to delays.

Original: LDA.
Revised: LDA.

Original: Fig.
Revised: Fig.

Original: 3(c)(d) show the convergence curves of LDA with different staleness levels for two settings varying on the number of workers and topics.
Revised: 3(c)(d) show the convergence curves of LDA with different staleness levels for two settings varying on the number of workers and topics.

Original: Unlike the convergence curves for SGD-based algorithms (see Appendix), the convergence curves of Gibbs sampling are highly smooth, even under high staleness and a large number of workers.
Revised: Unlike the convergence curves for SGD-based algorithms (see Appendix), the convergence curves of Gibbs sampling are highly smooth, even under high staleness and a large number of workers.

Original: This can be attributed to the structure of log likelihood objective function (Grifﬁths & Steyvers, 2004).
Revised: This can be attributed to the structure of log likelihood objective function (Grifﬁths & Steyvers, 2004).

Original: Since in each sampling step we only update the count statistics based on a portion of the corpus, the objective value will generally change smoothly.
Revised: Since in each sampling step we only update the count statistics based on a portion of the corpus, the objective value will generally change smoothly.

Original: Staleness levels under a certain threshold ( s ≤ 10 ) lead to convergence, following indistinguishable log likelihood trajectories, regardless of the number of topics ( K = 10 , 100 ) or the number of workers (2–16 workers, see Appendix).
Revised: Staleness levels under a certain threshold ( s ≤ 10 ) lead to convergence, following indistinguishable log likelihood trajectories, regardless of the number of topics ( K = 10 , 100 ) or the number of workers (2–16 workers, see Appendix).

Original: Also, there is very minimal variance in those trajectories.
Revised: Also, there is very minimal variance in those trajectories.

Original: However, for staleness beyond a certain level ( s ≥ 15 ), Gibbs sampling does not converge to a ﬁxed point.
Revised: However, for staleness beyond a certain level ( s ≥ 15 ), Gibbs sampling does not converge to a ﬁxed point.

Original: The convergence trajectories are distinct and are sensitive to the number of topics and the number of workers.
Revised: The convergence trajectories are distinct and are sensitive to the number of topics and the number of workers.

Original: There appears to be a “phase transition” at a certain staleness level that creates two distinct phases of convergence behaviors 7 .
Revised: There appears to be a “phase transition” at a certain staleness level that creates two distinct phases of convergence behaviors 7 .

Original: We believe this is the ﬁrst report of a staleness-induced failure case for LDA Gibbs sampling.
Revised: We believe this is the ﬁrst report of a staleness-induced failure case for LDA Gibbs sampling.

Original: VAE In Fig.
Revised: VAE In Fig.

Original: 3(e)(f), VAEs exhibit a much higher sensitivity to staleness compared with DNNs (Fig. 1(e)(f)).
Revised: 3(e)(f), VAEs exhibit a much higher sensitivity to staleness compared with DNNs (Fig. 1(e)(f)).

Original: This is the case even considering that VAE with depth 3 has 6 weight layers, which has a comparable number of model parameters and network architecture to DNNs with 6 layers.
Revised: This is the case even considering that VAE with depth 3 has 6 weight layers, which has a comparable number of model parameters and network architecture to DNNs with 6 layers.

Original: We hypothesize that this is caused by the additional source of stochasticity from the sampling procedure, in addition to the data sampling process 
Revised: We hypothesize that this is caused by the additional source of stochasticity from the sampling procedure, in addition to the data sampling process [Figure]

Original: N u m b e r o f B a t
Revised: N u m b e r o f B a t

Original: c h e s t
Revised: c h e s t

Original: o R e a c h [Figure] T 
Revised: o R e a c h 71 % T e s t

Original: [Figure]

Original: s t

Original: A cc u r a c y
Revised: A cc u r a c y

Original: N o r m a li
Revised: N o r m a li

Original: z e d
Revised: z e d

Original: N u m b e r o f B a t
Revised: N u m b e r o f B a t

Original: c h e s t
Revised: c h e s t

Original: o R e a c
Revised: o R e a c

Original: h% T e s t
Revised: h% T e s t

Original: A cc u ra c y
Revised: A cc u r

Original: [Figure]
Revised: [Figure]

Revised: c y

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: N m a c h

Original: [Figure]
Revised: [Figure]

Original: c
Revised: c

Original: [Figure]
Revised: [Figure]

Original:  t 
Revised: [Figure] h e s t o R e a c h T e s t L o

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: s

Original: [Figure]
Revised: [Figure]

Revised: s

Original: [Figure]
Revised: [Figure]

Revised: s

Original: [Figure]
Revised: [Figure]

Original: = s= [-s= s= s=
-]
Revised: s= [-s= s= s=
-]s [Figure]

Original: N o r m a li z e d N u m
Revised: N o r m a li z e d N u m

Original: B a t c h e s t
Revised: B a t c h e s t

Original: o R e a c h
Revised: o R e a c h

Original: T e s t L o ss
Revised: T e s t L o ss

Original: VAE (1 worker, Adam)
Revised: VAE (1 worker, Adam)

Original:  G RADIENT C OHERENCE AND C ONVERGENCE OF A SYNCHRONOUS SGD
Revised:  G RADIENT C OHERENCE AND C ONVERGENCE OF A SYNCHRONOUS SGD

Original: We now provide theoretical insight into the effect of staleness on the observed convergence slowdown.
Revised: We now provide theoretical insight into the effect of staleness on the observed convergence slowdown.

Original: We focus on the challenging asynchronous SGD (Async-SGD) case, which characterizes the neural network models, among others.
Revised: We focus on the challenging asynchronous SGD (Async-SGD) case, which characterizes the neural network models, among others.

Original: Consider the following nonconvex optimization problem
Revised: Consider the following nonconvex optimization problem

Original: [Equation]
Revised: [Equation]

Original: where f i corresponds to the loss on the i -th data sample, and the objective function is assumed to satisfy the following standard conditions:
Revised: where f i corresponds to the loss on the i -th data sample, and the objective function is assumed to satisfy the following standard conditions:

Original: Assumption 1.
Revised: Assumption 1.

Original: The objective function F in the problem (P) satisﬁes:
Revised: The objective function F in the problem (P) satisﬁes:

Original: Function F is continuously differentiable and bounded below, i.e., inf x ∈ R d F ( x ) >
Revised: Function F is continuously differentiable and bounded below, i.e., inf x ∈ R d F ( x ) >

Original: −∞ ; 2.
Revised: −∞ ; 2.

Original: The gradient of F is L -Lipschitz continuous.
Revised: The gradient of F is L -Lipschitz continuous.

Original: Notice that we allow F to be nonconvex.
Revised: Notice that we allow F to be nonconvex.

Original: We apply the Async-SGD to solve the problem (P).
Revised: We apply the Async-SGD to solve the problem (P).

Original: Let ξ ( k ) be the mini-batch of data indices sampled from { 1 , . .
Revised: Let ξ ( k ) be the mini-batch of data indices sampled from { 1 , . .

Original: . , n } uniformly at random by the algorithm at iteration k , and | ξ ( k ) | is the mini-batch size.
Revised: . , n } uniformly at random by the algorithm at iteration k , and | ξ ( k ) | is the mini-batch size.

Original: Denote mini-batch gradient as ∇ f ξ ( k ) ( x k ) :
Revised: Denote mini-batch gradient as ∇ f ξ ( k ) ( x k ) :

Original: = (cid:80) i ∈ ξ ( k ) ∇ f i ( x k ) .
Revised: = (cid:80) i ∈ ξ ( k ) ∇ f i ( x k ) .

Original: Then, the update rule of Async-SGD can be written as
Revised: Then, the update rule of Async-SGD can be written as

Original: [Equation]
Revised: [Equation]

Original: where η k corresponds to the stepsize, τ k denotes the delayed clock and the maximum staleness is assumed to be bounded by s .
Revised: where η k corresponds to the stepsize, τ k denotes the delayed clock and the maximum staleness is assumed to be bounded by s .

Original: This implies that k − s + 1 ≤ τ k ≤ k .
Revised: This implies that k − s + 1 ≤ τ k ≤ k .

Original: The optimization dynamics of Async-SGD is complex due to the nonconvexity and the uncertainty of the delayed updates.
Revised: The optimization dynamics of Async-SGD is complex due to the nonconvexity and the uncertainty of the delayed updates.

Original: Interestingly, we ﬁnd that the following notion of gradient coherence provides insights toward understanding the convergence property of Async-SGD.
Revised: Interestingly, we ﬁnd that the following notion of gradient coherence provides insights toward understanding the convergence property of Async-SGD.

Original: Deﬁnition 1 (Gradient coherence) .
Revised: Deﬁnition 1 (Gradient coherence) .

Original: The gradient coherence at iteration k is deﬁned as
Revised: The gradient coherence at iteration k is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: Parameter µ k captures the minimum coherence between the current gradient ∇ F ( x k ) and the gradients along the past s iterations 8 .
Revised: Parameter µ k captures the minimum coherence between the current gradient ∇ F ( x k ) and the gradients along the past s iterations 8 .

Original: Intuitively, if µ k is positive, then the direction of the current gradient is well aligned to those of the past gradients.
Revised: Intuitively, if µ k is positive, then the direction of the current gradient is well aligned to those of the past gradients.

Original: In this case, the convergence property induced by using delayed stochastic gradients is close to that induced by using synchronous stochastic gradients.
Revised: In this case, the convergence property induced by using delayed stochastic gradients is close to that induced by using synchronous stochastic gradients.

Original: Note that Deﬁnition 1 only requires the gradients to be positively correlated over a small number of iterations s, which is often very small (e.g. <10 in our experiments).
Revised: Note that Deﬁnition 1 only requires the gradients to be positively correlated over a small number of iterations s, which is often very small (e.g. <10 in our experiments).

Original: Therefore, Deﬁnition 1 is not a global requirement on optimization path.
Revised: Therefore, Deﬁnition 1 is not a global requirement on optimization path.

Original: Even though neural network’s loss function is non-convex, recent studies showed strong evidences that SGD in practical neural network training encourage positive gradient coherence (Li et al., 2017;
Revised: Even though neural network’s loss function is non-convex, recent studies showed strong evidences that SGD in practical neural network training encourage positive gradient coherence (Li et al., 2017;

Original: Lorch, 2016).
Revised: Lorch, 2016).

Original: This is consistent with the ﬁndings that the loss surface of shallow networks and deep networks with skip connections are dominated by large, ﬂat, nearly convex attractors around the critical points (Li et al., 2017; Keskar et al., 2016), implying that the degree of non-convexity is mild around critical points.
Revised: This is consistent with the ﬁndings that the loss surface of shallow networks and deep networks with skip connections are dominated by large, ﬂat, nearly convex attractors around the critical points (Li et al., 2017; Keskar et al., 2016), implying that the degree of non-convexity is mild around critical points.

Original: We show in the sequel that µ k > 0 through most of the optimization path, especially when the staleness is minimized in practice by system optimization (Fig. 4).
Revised: We show in the sequel that µ k > 0 through most of the optimization path, especially when the staleness is minimized in practice by system optimization (Fig. 4).

Original: Our theory can be readily adapted to account for a limited amount of negative µ k (see Appendix), but our primary interest is to provide a quantity that is (1) easy to compute empirically during the course of optimization 9 , and (2) informative for the impact of staleness and can potentially be used to control synchronization levels.
Revised: Our theory can be readily adapted to account for a limited amount of negative µ k (see Appendix), but our primary interest is to provide a quantity that is (1) easy to compute empirically during the course of optimization 9 , and (2) informative for the impact of staleness and can potentially be used to control synchronization levels.

Original: We now characterize the convergence property of Async-SGD.
Revised: We now characterize the convergence property of Async-SGD.

Original: Theorem 1.
Revised: Theorem 1.

Original: Let Assumption 1 hold.
Revised: Let Assumption 1 hold.

Original: Suppose for some µ > 0 , the gradient coherence satisﬁes µ k ≥ µ for all k and the variance of the stochastic gradients is bounded by σ 2 > 0 .
Revised: Suppose for some µ > 0 , the gradient coherence satisﬁes µ k ≥ µ for all k and the variance of the stochastic gradients is bounded by σ 2 > 0 .

Original: Choose stepsize η k = µ sL √ k .
Revised: Choose stepsize η k = µ sL √ k .

Original: Then, the iterates generated by the Async-SGD satisfy
Revised: Then, the iterates generated by the Async-SGD satisfy

Original: [Equation] [Equation]
Revised: [Equation] [Figure]

Original: We refer readers to Appendix for the the proof.
Revised: We refer readers to Appendix for the the proof.

Original: Theorem 1 characterizes several theoretical aspects of
Revised: Theorem 1 characterizes several theoretical aspects of

Original: Async-SGD.
Revised: Async-SGD.

Original: First, the choice of the stepsize η k = µ sL √ k is adapted to both the maximum staleness and the gradient coherence.
Revised: First, the choice of the stepsize η k = µ sL √ k is adapted to both the maximum staleness and the gradient coherence.

Original: Intuitively, if the system encounters a larger staleness, then a smaller stepsize should be used to compensate the negative effect.
Revised: Intuitively, if the system encounters a larger staleness, then a smaller stepsize should be used to compensate the negative effect.

Original: On the other hand, the stepsize can be accordingly enlarged if the gradient coherence along the iterates turns out to be high.
Revised: On the other hand, the stepsize can be accordingly enlarged if the gradient coherence along the iterates turns out to be high.

Original: In this case, the direction of the gradient barely changes along the past several iterations, and a more aggressive stepsize can be adopted.
Revised: In this case, the direction of the gradient barely changes along the past several iterations, and a more aggressive stepsize can be adopted.

Original: In summary, the choice of stepsize should trade-off between the effects caused by both the staleness and the gradient coherence.
Revised: In summary, the choice of stepsize should trade-off between the effects caused by both the staleness and the gradient coherence.

Original: [Figure]
Revised: [Figure]

Original: [Figure] Furthermore, Theorem 1 shows that the minimum gradient norm decays at the rate O ( log T √ T ) , implying that the Async-SGD converges to a stationary point provided a positive gradient coherence, which we observe empirically in the sequel.
Revised: Furthermore, Theorem 1 shows that the minimum gradient norm decays at the rate O ( log T √ T ) , implying that the Async-SGD converges to a stationary point provided a positive gradient coherence, which we observe empirically in the sequel.

Original: On the other hand, the bound in Eq.
Revised: On the other hand, the bound in Eq.

Original: (1) captures the trade-off between the maximum staleness s and the gradient coherence µ .
Revised: (1) captures the trade-off between the maximum staleness s and the gradient coherence µ .

Original: Speciﬁcally, minimizing the right hand side of Eq.
Revised: Specifically, minimizing the right hand side of Eq.

Original: (1) with regard to the maximum staleness s yields the optimal choice s ∗ = σµ 
Revised: (1) with regard to the maximum staleness s yields the optimal choice s ∗ = σµ (cid:113) log

Original: [Equation]
Revised: [Equation]

Revised: F

Original: [Figure]
Revised: [Equation]

Original:  larger staleness is allowed if the gradients remain to be highly coherent along the past iterates.
Revised: x )) , i.e., a larger staleness is allowed if the gradients remain to be highly coherent along the past iterates.

Original: Empirical Observations.
Revised: Empirical Observations.

Original: Theorem 1 suggests that more coherent gradients along the optimization paths can be advantageous under non-synchronous execution.
Revised: Theorem 1 suggests that more coherent gradients along the optimization paths can be advantageous under non-synchronous execution.

Original: Fig.
Revised: Fig.

Original: 4 shows the cosine similarity sim ( a , b ) :=
Revised: 4 shows the cosine similarity sim ( a , b ) : = a ·

Original: [Equation]
Revised: [Table]

Original: between gradients along the convergence path for CNNs and DNNs 10 .
Revised: between gradients along the convergence path for CNNs and DNNs 10 .

Original: We observe the followings: (1) Cosine similarity improves over the course of convergence (Fig. 4(a)(b)).
Revised: We observe the followings: (1) Cosine similarity improves over the course of convergence (Fig. 4(a)(b)).

Original: Except the highest staleness during the early phase of convergence, cosine similarity remains positive 11 .
Revised: Except the highest staleness during the early phase of convergence, cosine similarity remains positive 11 .

Original: In practice the staleness experienced during run time can be limited to small staleness (Dai et al., 2015), which minimizes the likelihood of negative gradient coherence during the early phase.
Revised: In practice the staleness experienced during run time can be limited to small staleness (Dai et al., 2015), which minimizes the likelihood of negative gradient coherence during the early phase.

Original: (2) Fig.
Revised: (2) Fig.

Original: 5 shows that cosine similarity decreases with increasing CNN model complexity.
Revised: 5 shows that cosine similarity decreases with increasing CNN model complexity.

Original: Theoremimplies that lower gradient coherence ampliﬁes the effect of staleness s through the factor sµ 2 in
Revised: Theoremimplies that lower gradient coherence ampliﬁes the effect of staleness s through the factor sµ 2 in

Original: Eq.
Revised: Eq.

Original: This is consistent with the convergence difﬁculty encountered in deeper models (Fig. 1).
Revised: This is consistent with the convergence difﬁculty encountered in deeper models (Fig. 1).

Original:  D ISCUSSION AND C ONCLUSION
Revised:  D ISCUSSION AND C ONCLUSION

Original: In this work, we study the convergence behaviors under delayed updates for a wide array of models and algorithms.
Revised: In this work, we study the convergence behaviors under delayed updates for a wide array of models and algorithms.

Original: Our extensive experiments reveal that staleness appears to be a key governing parameter in learning.
Revised: Our extensive experiments reveal that staleness appears to be a key governing parameter in learning.

Original: Overall staleness slows down the convergence, and under high staleness levels the convergence can progress very slowly or fail.
Revised: Overall staleness slows down the convergence, and under high staleness levels the convergence can progress very slowly or fail.

Original: The effects of staleness are highly problem dependent, inﬂuenced by model complexity, choice of the algorithms, the number of workers, and the model itself, among others.
Revised: The effects of staleness are highly problem dependent, inﬂuenced by model complexity, choice of the algorithms, the number of workers, and the model itself, among others.

Original: Our empirical ﬁndings inspire new analyses of non-convex optimization under asynchrony based on gradient coherence, matching the existing rate of O (1 / √ T ) .
Revised: Our empirical ﬁndings inspire new analyses of non-convex optimization under asynchrony based on gradient coherence, matching the existing rate of O (1 / √ T ) .

Original: Our ﬁndings have clear implications for distributed ML.
Revised: Our ﬁndings have clear implications for distributed ML.

Original: To achieve actual speed-up in absolute convergence, any distributed ML system needs to overcome the slowdown from staleness, and carefully trade off between system throughput gains and statistical penalties.
Revised: To achieve actual speed-up in absolute convergence, any distributed ML system needs to overcome the slowdown from staleness, and carefully trade off between system throughput gains and statistical penalties.

Original: Many ML methods indeed demonstrate certain robustness against low staleness, which should offer opportunities for system optimization.
Revised: Many ML methods indeed demonstrate certain robustness against low staleness, which should offer opportunities for system optimization.

Original: Our results support the broader observation that existing successful nonsynchronous systems generally keep staleness low and use algorithms efﬁcient under staleness.
Revised: Our results support the broader observation that existing successful nonsynchronous systems generally keep staleness low and use algorithms efﬁcient under staleness

Original: A A PPENDIX A. P ROOF OF T HEOREM
Revised: A A PPENDIX A. P ROOF OF T HEOREM

Original: Theorem 2.
Revised: Theorem 2.

Original: Let Assumption 1 hold.
Revised: Let Assumption 1 hold.

Original: Suppose the gradient coherence µ k is lower bounded by some µ > 0 for all k and the variance of the stochastic gradients is upper bounded by some σ 2 > 0 .
Revised: Suppose the gradient coherence µ k is lower bounded by some µ > 0 for all k and the variance of the stochastic gradients is upper bounded by some σ 2 > 0 .

Original: Choose stepsize η k = µ sL √ k .
Revised: Choose stepsize η k = µ sL √ k .

Original: Then, the iterates generated by the Async-SGD satisfy
Revised: Then, the iterates generated by the Async-SGD satisfy

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: By the L -Lipschitz property of ∇ F
Revised: By the L -Lipschitz property of ∇ F

Original: [Equation]
Revised: [Equation]

Original: we obtain that for all k
Revised: we obtain that for all k

Original: [Equation]
Revised: [Equation]

Original: Taking expectation on both sides of the above inequality and note that the variance of the stochastic gradient is bounded by σ
Revised: Taking expectation on both sides of the above inequality and note that the variance of the stochastic gradient is bounded by σ

Original: [Equation]
Revised: [Equation]

Original: we further obtain that
Revised: we further obtain that

Original: [Equation]
Revised: [Equation]

Original: Telescoping the above inequality over k from 0 to T yields that
Revised: Telescoping the above inequality over k from 0 to T yields that

Original: [Equation]
Revised: [Equation]

Original: Rearranging the above inequality and note that F ( x k +1 ) > inf x F ( x ) >
Revised: Rearranging the above inequality and note that F ( x k +1 ) > inf x F ( x ) >

Original: −∞
Revised: −∞

Original: [Equation]
Revised: [Equation]

Original: we further obtain that
Revised: we further obtain that

Original: [Equation]
Revised: [Equation]

Original: Note that the choice of stepsize guarantees that η k µ − Lsη 2 k> 0 for all k .
Revised: Note that the choice of stepsize guarantees that η k µ − Lsη 2 k> 0 for all k .

Original: Thus
Revised: Thus

Original: [Equation]
Revised: [Equation]

Original: we conclude that
Revised: we conclude that

Original: [Equation]
Revised: [Equation]

Original: where the last inequality uses the fact that η k µ − Lsη 2 k> η k µ 2 .
Revised: where the last inequality uses the fact that η k µ − Lsη 2 k> η k µ 2 .

Original: Substituting the stepsize η k = µ sL √ k into the above inequality and simplifying, we ﬁnally obtain that
Revised: Substituting the stepsize η k = [Equation] sL √ k into the above inequality and simplifying, we ﬁnally obtain that

Original: [Equation]
Revised: [Equation]

Original: A. H ANDLING N EGATIVE G RADIENT C OHERENCE IN T HEOREM
Revised: A. H ANDLING N EGATIVE G RADIENT C OHERENCE IN T HEOREM

Original: Our assumption of positive gradient coherence (GC) is motivated by strong empirical evidence that GC is largely positive (Fig.
Revised: Our assumption of positive gradient coherence (GC) is motivated by strong empirical evidence that GC is largely positive (Fig.

Original: 4(a)(b) in the main text).
Revised: 4(a)(b) in the main text).

Original: Contrary to conventional wisdom, GC generally improves when approaching convergence for both SGD and Adam.
Revised: Contrary to conventional wisdom, GC generally improves when approaching convergence for both SGD and Adam.

Original: Furthermore, in practice, the effective staleness for any given iteration generally concentrates in low staleness for the non-stragglers (Dai et al., 2015).
Revised: Furthermore, in practice, the effective staleness for any given iteration generally concentrates in low staleness for the non-stragglers (Dai et al., 2015).

Original: When some µ k are negative at some iterations, in eq.
Revised: When some µ k are negative at some iterations, in eq.

Original: 11 in the Appendix we can move the negative terms in (cid:80)
Revised: 11 in the Appendix we can move the negative terms in (cid:80)

Original: k
Revised: k

Original: η k µ k to the right hand side and yield a higher upper bound (i.e., slower convergence).
Revised: η k µ k to the right hand side and yield a higher upper bound (i.e., slower convergence).

Original: This is also consistent with empirical observations that higher staleness lowers GC and slows convergence.
Revised: This is also consistent with empirical observations that higher staleness lowers GC and slows convergence.

Original: A. E XPONENTIAL DELAY DISTRIBUTION .
Revised: A. E XPONENTIAL DELAY DISTRIBUTION .

Original: We consider delays drawn from geometric distribution (GD), which is the discrete version of exponential distribution.
Revised: We consider delays drawn from geometric distribution (GD), which is the discrete version of exponential distribution.

Original: For each iterate we randomly select a worker to be the straggler with large mean delay ( p = 0 . 1 ), while all other non-straggler workers have small delays.
Revised: For each iterate we randomly select a worker to be the straggler with large mean delay ( p = 0 . 1 ), while all other non-straggler workers have small delays.

Original: The non-straggler delay is drawn from GD with p chosen to achieve the same mean delay as in the uniform case (after factoring in straggler) in the main text.
Revised: The non-straggler delay is drawn from GD with p chosen to achieve the same mean delay as in the uniform case (after factoring in straggler) in the main text.

Original: The delay is drawn per worker for each iteration, and thus a straggler’s outgoing updates to all workers suffer the same delay.
Revised: The delay is drawn per worker for each iteration, and thus a straggler’s outgoing updates to all workers suffer the same delay.

Original: Fig. 4(c) in the main text shows the convergence speed under the corresponding staleness s with the same mean delay (though s is not a parameter in GD).
Revised: Fig. 4(c) in the main text shows the convergence speed under the corresponding staleness s with the same mean delay (though s is not a parameter in GD).

Original: It exhibits trends analogous to Fig. 1(b) in the main text: staleness slows convergence substantially and overall impacts deeper networks more.
Revised: It exhibits trends analogous to Fig. 1(b) in the main text: staleness slows convergence substantially and overall impacts deeper networks more.

Original: A. A DDITIONAL R ESULTS FOR DNN S
Revised: A. A DDITIONAL R ESULTS FOR DNN S

Original: We present additional results for DNNs.
Revised: We present additional results for DNNs.

Original: Fig.
Revised: Fig.

Original: 6 shows the number of batches, normalized by s = 0 , to reach convergence using 1 hidden layer and 1 worker under varying staleness levels and batch sizes.
Revised: 6 shows the number of batches, normalized by s = 0 , to reach convergence using 1 hidden layer and 1 worker under varying staleness levels and batch sizes.

Original: Overall, the effect of batch size is relatively small except in high staleness regime ( s = 32 ).
Revised: Overall, the effect of batch size is relatively small except in high staleness regime ( s = 32 ).

Original: Fig.
Revised: Fig.

Original: 7 shows the number of batches to reach convergence, normalized by s = 0 case, for 5 variants of
Revised: 7 shows the number of batches to reach convergence, normalized by s = 0 case, for 5 variants of

Original: SGD using 1 worker.
Revised: SGD using 1 worker.

Original: The results are in line with the analyses in the main text: staleness generally leads to larger slow down for deeper networks than shallower ones.
Revised: The results are in line with the analyses in the main text: staleness generally leads to larger slow down for deeper networks than shallower ones.

Original: SGD and Adagrad are more robust to staleness than Adam, RMSProp, and SGD with momentum.
Revised: SGD and Adagrad are more robust to staleness than Adam, RMSProp, and SGD with momentum.

Original: In particular, RMSProp exhibit high variance in batches to convergence (not shown in the normalized plot) and thus does not exhibit consistent trend.
Revised: In particular, RMSProp exhibit high variance in batches to convergence (not shown in the normalized plot) and thus does not exhibit consistent trend.

Original: Fig.
Revised: Fig.

Original: 8 shows the number of batches to convergence under Adam and SGD on 1, 8, 16 simulated workers, respectively normalized by staleness 0’s values.
Revised: 8 shows the number of batches to convergence under Adam and SGD on 1, 8, 16 simulated workers, respectively normalized by staleness 0’s values.

Original: The results are consistent with the observations and analyses in the main text, namely, that having more workers ampliﬁes the effect of staleness.
Revised: The results are consistent with the observations and analyses in the main text, namely, that having more workers ampliﬁes the effect of staleness.

Original: We can also observe that SGDS is more robust to staleness than Adam, and shallower networks are less impacted by staleness.
Revised: We can also observe that SGDS is more robust to staleness than Adam, and shallower networks are less impacted by staleness.

Original: In particular, note that staleness sometimes accelerates convergence, such as in Fig. 8(d).
Revised: In particular, note that staleness sometimes accelerates convergence, such as in Fig. 8(d).

Original: This is due to the implicit momentum created by staleness (Mitliagkas et al., 2016).
Revised: This is due to the implicit momentum created by staleness (Mitliagkas et al., 2016).

Original: A. LDA AND A DDITIONAL R ESULTS FOR LDA
Revised: A. LDA AND A DDITIONAL R ESULTS FOR LDA

Original: In LDA each token w ij ( j -th token in the i -th document) is assigned with a latent topic z ij from totally K topics.
Revised: In LDA each token w ij ( j -th token in the i -th document) is assigned with a latent topic z ij from totally K topics.

Original: We use Gibbs sampling to infer the topic assignments z ij .
Revised: We use Gibbs sampling to infer the topic assignments z ij .

Original: The Gibbs sampling step involves three sets of parameters, known as sufﬁcient statistics: (1) document-topic vector θ i ∈ R K where θ ik the number of topic assignments within document i to topic k = 1 ...K ; (2) word-topic
Revised: The Gibbs sampling step involves three sets of parameters, known as sufﬁcient statistics: (1) document-topic vector θ i ∈ R K where θ ik the number of topic assignments within document i to topic k = 1 ...K ; (2) word-topic

Original: [Table]
Revised: [Table]

Original: [Table][Figure] [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: batchs= s= s= s=8 s=16 s=32 0.
Revised: batchs= s= s= s=8 s=16 s=32 0.

Original: N o r m a li
Revised: N o r m a li

Original: z e d
Revised: z e d

Original: N u m
Revised: N u m

Original: B a t c h e s t
Revised: B a t c h e s t

Original: o R e a c h 95 % T e s t
Revised: o R e a c h 95 % T e s t

Original: A cc u r a c y vector φ w ∈ R K where φ wk is the number of topic assignments to topic k = 1 , ..., K for word (vocabulary) w across all documents; (3) ˜
Revised: A cc u r a c y vector φ w ∈ R K where φ wk is the number of topic assignments to topic k = 1 , ..., K for word (vocabulary) w across all documents; (3) ˜

Original: φ ∈ R K where ˜
Revised: φ ∈ R K where ˜

Original: φ k = (cid:80) Ww =1 φ wk is the number of tokens in the corpus assigned to topic k .
Revised: φ k = (cid:80) Ww =1 φ wk is the number of tokens in the corpus assigned to topic k .

Original: The corpus ( w ij , z ij ) is partitioned to workers, while φ w and ˜ φ are shared model parameters.
Revised: The corpus ( w ij , z ij ) is partitioned to workers, while φ w and ˜ φ are shared model parameters.

Original: We measure the model quality using log likelihood.
Revised: We measure the model quality using log likelihood.

Original: We present additional results of LDA under different numbers of workers and topics in Fig.
Revised: We present additional results of LDA under different numbers of workers and topics in Fig.

Original: 9 and
Revised: 9 and

Original: Fig.
Revised: Fig.

Original: These panels extends Fig.
Revised: These panels extends Fig.

Original: 3(c)(d) in the main text.
Revised: 3(c)(d) in the main text.

Original: See the main text for experimental setup and analyses and experimental setup.
Revised: See the main text for experimental setup and analyses and experimental setup.

Original: A. A DDITIONAL R ESULTS FOR MF
Revised: A. A DDITIONAL R ESULTS FOR MF

Original: We show the convergence curves for MF under different numbers of workers and staleness levels in
Revised: We show the convergence curves for MF under different numbers of workers and staleness levels in

Original: Fig.
Revised: Fig.

Original: It is evident that higher staleness leads to a higher variance in convergence.
Revised: It is evident that higher staleness leads to a higher variance in convergence.

Original: Furthermore, the number of workers also affects variance, given the same staleness level.
Revised: Furthermore, the number of workers also affects variance, given the same staleness level.

Original: For example, MF withworkers incurs very low standard deviation up to staleness 20.
Revised: For example, MF withworkers incurs very low standard deviation up to staleness 20.

Original: In contrast, MF with 8 workers already exhibits a large variance at staleness 15.
Revised: In contrast, MF with 8 workers already exhibits a large variance at staleness 15.

Original: The ampliﬁcation of staleness from increasing number of
Revised: The ampliﬁcation of staleness from increasing number of

Original: [Figure]
Revised: [Figure]

Original:  li 
Revised: a li z e d N u m B a t c h e s t o R e a c h

Original: [Figure]
Revised: [Figure]

Original: e d
Revised: e s

Original: N u

Original: [Figure]
Revised: [Figure]

Original: a t c h e
Revised: A cc u

Original: [Figure]
Revised: [Figure]

Original: R e
Revised: a c y

Original: [Figure]
Revised: [Figure]

Original: c h
Revised: r m a li z e d N

Original: [Figure]
Revised: [Figure]

Original: c y
Revised: m

Original: [Figure]
Revised: [Figure]

Revised: t

Revised: [Figure]

Revised: t

Original: reduces to MLR, which is convex.
Revised: [Figure] a [Figure] to MLR, which is convex.

Original: The numbers are averaged over 5 randomized runs.
Revised: The numbers are averaged over 5 randomized runs.

Original: We omit the result whenever convergence is not achieved within the experiment horizon ( batches), such as SGD with momentum at depth 6 and s = 32 .
Revised: We omit the result whenever convergence is not achieved within the experiment horizon ( batches), such as SGD with momentum at depth 6 and s = 32 .

Original: workers is consistent with the discussion in the main text.
Revised: workers is consistent with the discussion in the main text.

Original: See the main text for experimental setup and analyses.
Revised: See the main text for experimental setup and analyses.

Original: A. A DDITIONAL R ESULTS FOR VAE S
Revised: A. A DDITIONAL R ESULTS FOR VAE S

Original: Fig.
Revised: Fig.

Original: 12 shows the number of batches to reach test loss 130 by Variational Autoencoders (VAEs) onworker, under staleness 0 to 16 and 4 SGD variants.
Revised: 12 shows the number of batches to reach test loss 130 by Variational Autoencoders (VAEs) onworker, under staleness 0 to 16 and 4 SGD variants.

Original: We consider VAEs with depth 1, 2, and 3 (the number of layers in the encoder and decoder networks).
Revised: We consider VAEs with depth 1, 2, and 3 (the number of layers in the encoder and decoder networks).

Original: The number of batches are normalized by s = 0 for each VAE depth, respectively.
Revised: The number of batches are normalized by s = 0 for each VAE depth, respectively.

Original: See the main text for analyses.
Revised: See the main text for analyses.

Original: [Table]
Revised: [Table]

Original: [Figure] 
Revised: [Figure] [Figure] [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: N o r m a li
Revised: N o r m a li

Original: z e d
Revised: z e d

Original: N u m
Revised: N u m

Original: B a t c h e s t
Revised: B a t c h e s t

Original: o R e a c h 92 % T e s t
Revised: o R e a c h 92 % T e s t

Original: A cc u r a c y
Revised: A cc u r a c y

Original: A. R ECURRENT N EURAL N ETWORKS
Revised: A. R ECURRENT N EURAL N ETWORKS

Original: Recurrent Neural Networks (RNNs) are widely used in recent natural language processing tasks.
Revised: Recurrent Neural Networks (RNNs) are widely used in recent natural language processing tasks.

Original: We consider long short-term memory (LSTM) (Hochreiter & Schmidhuber, 1997) applied to the language modeling task, using a subset of Penn Treebank dataset (PTB) (Marcus et al., 1993) containingwords.
Revised: We consider long short-term memory (LSTM) (Hochreiter & Schmidhuber, 1997) applied to the language modeling task, using a subset of Penn Treebank dataset (PTB) (Marcus et al., 1993) containingwords.

Original: The dataset is pre-processed by standard de-capitalization and tokenization.
Revised: The dataset is pre-processed by standard de-capitalization and tokenization.

Original: We evaluate the impact of staleness for LSTM with 1 to 4 layers, with 256 neurons in each layer.
Revised: We evaluate the impact of staleness for LSTM with 1 to 4 layers, with 256 neurons in each layer.

Original: The maximum length for each sentence is 25.
Revised: The maximum length for each sentence is 25.

Original: Note that 4 layer LSTM is about 4x more model parameters than the 1 layer LSTM, which is the same ratio between ResNet32 and Resnet 8.
Revised: Note that 4 layer LSTM is about 4x more model parameters than the 1 layer LSTM, which is the same ratio between ResNet32 and Resnet 8.

Original: We use batch sizesimilar to other experiments.
Revised: We use batch sizesimilar to other experiments.

Original: We consider staleness s = 0 , 4 , 8 , 16 on 8 workers.
Revised: We consider staleness s = 0 , 4 , 8 , 16 on 8 workers.

Original: The model quality is measured in perplexity.
Revised: The model quality is measured in perplexity.

Original: Fig.
Revised: Fig.

Original: 13 shows the number of batches needed to reach the desired model quality for RNNs with varying network depths.
Revised: 13 shows the number of batches needed to reach the desired model quality for RNNs with varying network depths.

Original: We again observe that staleness impacts deeper network variants more than shallower counterparts, which is consistent with our observation in CNNs and DNNs.
Revised: We again observe that staleness impacts deeper network variants more than shallower counterparts, which is consistent with our observation in CNNs and DNNs.

Original: [Figure]
Revised: [Figure]

Original: Fig.
Revised: Fig.

Original: 14 shows the number of batches needed to reach the desired model quality for RNNs with on
Revised: 14 shows the number of batches needed to reach the desired model quality for RNNs with on

Original: SGD variants: SGD, Adam, Momentum, and RMSProp.
Revised: SGD variants: SGD, Adam, Momentum, and RMSProp.

Original: Similar to the discussion in the main text, different algorithms respond to staleness differently, with SGD and Adam more robust to staleness than Momentum and RMSProp.
Revised: Similar to the discussion in the main text, different algorithms respond to staleness differently, with SGD and Adam more robust to staleness than Momentum and RMSProp.

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure] [Figure]
Revised: [Figure] s= s= s= s= s=16 0.

Original: N o r m a li
Revised: N o r m a li

Original: z e d
Revised: z e d

Revised: N u m

Revised: B a t c h e s t

Revised: o R e a c

Revised: h T e s t L o ss

Revised: Depth 1 Depth

Revised: Depth(

Revised: [Figure]

Revised: SGD s= s= s= s=

Original: {+s=
+}
Revised: {+s=
+}

Revised: N o r m a li

Revised: z e d

Revised: N u m

Revised: B a t c h e s t

Revised: o R e a c

Revised: h T e s t L o ss (b)

Revised: Adam s= s= s= s=

Original: {+s=
+}
Revised: {+s=
+}

Revised: N o r m a li

Revised: z e d

Original: N u m
Revised: N u m

Original: [Figure]
Revised: B a t c h e s t o R e a c h T e s t L o ss (c) SGD with Momentum s= s= s= s= s=16 0.

Original: a t c h e s t o R e a c
Revised: N o r m a li z e d N u m

Original:  hT e s tL o ss
Revised: B a t c h e s t

Original: [Figure]

Original: t [Figure] a c 
Revised: o R e a c h T e s

Original: [Figure]
Revised: [Figure]

Original: ss (c)
Revised: ss (

Original: SGD with Momentum

Original: [Figure]
Revised: [Figure]

Original: Adagrad
Revised: Adagrad