Paper:
TOWARD UNDERSTANDING THE IMPACT OF STALENESS IN DISTRIBUTED MACHINE LEARNING
Most distributed machine learning (ML) systems store a copy of the model parameters locally on each machine to minimize network communication. In practice, in order to reduce synchronization waiting time, these copies of the model are not necessarily updated in lock-step, and can become stale. Despite much development in large-scale ML, the effect of staleness on the learning efficiency is inconclusive, mainly because it is challenging to control or monitor the staleness in complex distributed environments. In this work, we study the convergence behaviors of a wide array of ML models and algorithms under delayed updates. Our extensive experiments reveal the rich diversity of the effects of staleness on the convergence of ML algorithms and offer insights into seemingly contradictory reports in the literature. The empirical findings also inspire a new convergence analysis of SGD in non-convex optimization under staleness, matching the best-known convergence rate of O(1/ √ T ).

Review:
This paper presents and empirical and theoretical study of the convergence of asynchronous stochastic gradient descent training if there are delays due to the asynchronous part of it. The paper can be neatly split in two parts: a simulation study and a theoretical analysis.
The simulation study compares, under fixed hyperparameters, the behavior of distributed training under different simulated levels of delay on different problems and different model architectures. Overall the results are very interesting, but the simulation could have been more thorough. Specifically, the same hyperparameter values were used across batch sizes and across different values of the distributed delay. Some algorithms failed to converge under some settings and others experienced dramatic slowdowns, but without careful study of hyperparameters it's hard to tell whether these behaviors are normal or outliers. Also it would have been interesting to see a recurrent architecture there, as I've heard much anecdotal evidence about the robustness of RNNs and LSTMs to asynchronous training. I strongly advise the authors to redo the experiments with some hyperparameter tuning for different levels of staleness to make these results more believable.
The theoretical analysis identifies a quantity called gradient coherence and proves that a learning rate based on the coherence can lead to an optimal convergence rate even under asynchronous training. The proof is correct (I checked the major steps but not all details), and it's sufficiently different from the analysis of hogwild algorithms to be of independent interest. The paper also shows the empirical behavior of the gradient coherence statistic during model training; interestingly this seems to also explain the heuristic commonly believed that to make asynchronous training work one needs to slowly anneal the number of workers (coherence is much worse in the earlier than later phases of training). This quantity is interesting also because it's somewhat independent of the variance of the stochastic gradient across minibatches (it's the time variance, in a way), and further analysis might also show interesting results.