Paper:
Deep learning with Elastic Averaging SGD
We study the problem of stochastic optimization for deep learning in the parallel computing environment under communication constraints. A new algorithm is proposed in this setting where the communication and coordination of work among concurrent processes (local workers), is based on an elastic force which links the parameters they compute with a center variable stored by the parameter server (master). The algorithm enables the local workers to perform more exploration, i.e. the algorithm allows the local variables to fluctuate further from the center variable by reducing the amount of communication between local workers and the master. We empirically demonstrate that in the deep learning setting, due to the existence of many local optima, allowing more exploration can lead to the improved performance. We propose synchronous and asynchronous variants of the new algorithm. We provide the stability analysis of the asynchronous variant in the round-robin scheme and compare it with the more common parallelized method ADMM. We show that the stability of EASGD is guaranteed when a simple stability condition is satisfied, which is not the case for ADMM. We additionally propose the momentum-based version of our algorithm that can be applied in both synchronous and asynchronous settings. Asynchronous variant of the algorithm is applied to train convolutional neural networks for image classification on the CIFAR and ImageNet datasets. Experiments demonstrate that the new algorithm accelerates the training of deep architectures compared to DOWNPOUR and other common baseline approaches and furthermore is very communication efficient.

Review:
This paper studies general-purpose training algorithms for deep learning and proposes a family of algorithms called elastic averaging SGD. The idea is novel and the paper is of very high quality.The paper focuses on training large-scale deep learning models under communication constraints. This problem is difficult since there are many local optima in non-convex problems like in deep learning. The optimization problem is formulated as a global variable consensus problem such that local workers would not fall into different local optima, and then its gradient update rules are reinterpreted using the elastic forces between local and global parameters. This is why the proposed family of algorithms is named elastic averaging SGD (EASGD). There are four algorithms in the family: a synchronous version, an asynchronous version, and their momentum versions. They are demonstrated promising theoretically and experimentally.In the theoretical analysis, the authors present a vivid illustrative example in which the popular ADMM can be unstable and its stability condition is unknown, but the stability condition of EASGD is quite simple. The writing style here is very impressive and friendly. All theorems are in the supplementary material and there is only a small part of theoretical analysis in the main paper, but the organization made me feel quite smooth (though I did not carefully go through the supplementary material). I have some minor questions here: What is the most important reason for the superior stability of EASGD? Is it because the maps of ADMM are asymmetric while the maps of EASGD are symmetric? Is it possible to find another illustrative example such that on this specific example EASGD is more unstable than ADMM?In their experiments, the proposed algorithms were used to train convolutional neural networks and they outperformed state-of-the-art SGD algorithms on CIFAR and ImageNet. The proposed algorithms achieved the goal such that the larger the communication periods are, the better EASGD would be than other SGD as shown in Figure 2. Figures 3 and 4 also showed that the speedup of EASGD would be better than the baselines when there were more workers. As further experimental results, we have implemented similar gradient update rules adapted to our clusters (according to the arxiv version) and they successfully improved our baselines. The resulted models have been or will be launched online and one fifth of the world's population would benefit from these improved models. So I believe that this is a significant paper.