Paper:
Sparsified SGD with Memory
Huge scale machine learning problems are nowadays tackled by distributed optimization algorithms, i.e. algorithms that leverage the compute power of many devices for training. The communication overhead is a key bottleneck that hinders perfect scalability. Various recent works proposed to use quantization or sparsification techniques to reduce the amount of data that needs to be communicated, for instance by only sending the most significant entries of the stochastic gradient (top-k sparsification). Whilst such schemes showed very promising performance in practice, they have eluded theoretical analysis so far. In this work we analyze Stochastic Gradient Descent (SGD) with k-sparsification or compression (for instance top-k or random-k) and show that this scheme converges at the same rate as vanilla SGD when equipped with error compensation (keeping track of accumulated errors in memory). That is, communication can be reduced by a factor of the dimension of the problem (sometimes even more) whilst still converging at the same rate. We present numerical experiments to illustrate the theoretical findings and the good scalability for distributed applications.
compression can hurt the performance, unless it is implemented in a clever way: 1Bit-SGD [33,37] combines gradient quantization with an error compensation technique, which is a memory or feedback mechanism. We in this work leverage this key mechanism but apply it within the more general setting of SGD. We will now sketch how the algorithm uses feedback to correct for errors accumulated in previous iterations. Roughly speaking, the method keeps track of a memory vector m which contains the sum of the information that has been suppressed thus far, i.e. m t+1 := m t + g t − comp(g t ), and injects this information back in the next iteration, by transmitting comp(m t+1 + g t+1 ) instead of only comp(g t+1 ). Note that updates of this kind are not unbiased (even if comp(g t+1 ) would be) and there is also no control over the delay after which the single coordinates are applied. These are some (technical) reasons why there exists no theoretical analysis of this scheme up to now.
In this paper we give a concise convergence rate analysis for SGD with memory and k-compression operators 2 , such as (but not limited to) top-k sparsification. Our analysis also supports ultra-sparsification operators for which k < 1, i.e. where less than one coordinate of the stochastic gradient is applied on average in (1). We not only provide the first convergence result of this method, but the result also shows that the method converges at the same rate as vanilla SGD.

Review:
This work analyzes a variant of SGD designed to incur lower communication costs in distributed systems. This variant of SGD combines sparsification with memory. At each iteration, the algorithm sparsifies the stochastic gradient by sending only the k (out of d) largest coordinates (in absolute value) or a random k coordinates to other nodes in the system. The portion that was not transmitted is added to the next stochastic gradient, and their sum is sparsified in the same manner. The authors show that without memory, the convergence of this method could be adversely affected, but that by maintaining this memory, the convergence rate resembles that of SGD, modulo a d/k factor. The main theory of the paper uses perturbed iterate analysis to analyze sparsified SGD with memory. The authors also give detailed experiments showing that selecting the top k coordinates outperforms random k coordinates, and comparing the method favorably to QSGD.  Quality: The submission seems to be theoretically sound. Many of the techniques are straightforward adapatations of techniques from perturbed iterate analysis. The primary theoretical novelty is Lemma 3.2 which uses the fact that sparsification can be seen as a kind of contraction operator (Definition 2.1) and analytic estimates to analyze the norm of the memory vector over time. Moreover, the experimental results show (as prior works have shown) the efficacy of the strategy.  I have only two main comments about this part. The first is that some discussion of how this analysis might carry over to the convex or even non-convex setting would be illuminating. Seeing where and how the theory breaks could help spur the community to fix these issues. The second has to do with Figure 3. The authors correctly note that sparsified SGD with memory can be much more sparse than QSGD. The authors try 2, 4, and 8 bit QSGD (as the authors of the original QSGD paper do), but they only display 4 and 8 bit QSGD for one experiment and 2 and 4 for the other. It is not immediately clear why this omission occurs.  Clarity: The paper is well-written throughout. The underlying problem is well-motivated in the introduction, and a very useful technical motivation is given in Section 2.2. Comparing the variance blow-up reduction to that of mini-batch SGD was particularly insightful. The theory is organized well, and the experiments setups are detailed.  A few comments that might help the readability throughout. First, more clearly differentiating between sparsification and quantization may be helpful. For instance, in line 29 you refer to a quantization function as sparse(), even though quantizations need not be sparse. Second, in the appendix every equation is numbered. It might be more helpful to the reader to only number the important ones. Intermediate steps need not be numbered.  Originality: The theoretical results in the work are to the best of my knowledge novel. They are very similar in scope to results concerning the use of 1-bit SGD with memory, but apply to sparsification instead. The authors do a good job of citing other work that has analyzed SGD with memory empirically and differentiate themselves by presenting their theoretical work.  While the authors do a good job of citing all the related work, some of the related work section could be fleshed out or improved. For instance, lines 51-52 state that "Sparsification methods reduce the number of non-zero entries in the stochastic gradient," which while true, ignores all nuance in different methods. Short general explanations of what some of the methods do would be useful, especially for QSGD which is later compared directly to SGD with memory. Last, some of the phrasing in the related work section is based on opinion or conjecture instead of fact. For instance, line 55 says that sparsification methods "can only work" when memory is incorporated. While I agree that vanilla sparsification methods can use memory to their benefit, to suggest they are never effective without them or that there are no other possible methods is not necessarily correct.  Significance: The theory is useful for confirming what empirical studies have seen in practice. The theory is also significant for its simplicity and conciseness. In terms of scope, it is similar to other results (such as 1-bit SGD with memory), and only extends to strongly convex functions. Nevertheless, its mathematical elegance is worth highlighting. Its experimental results comparing SGD with memory to QSGD are somewhat significant as well, though due to the number of variants of QSGD and parameters to be set, a more comprehensive analysis might be warranted.  All in all, I believe that the paper deserves acceptance to NIPS. It is well-written, mathematically concise, and gives a step forward in understanding how the use of memory can help sparsification/quantiztion techniques.