Paper:
LAG: Lazily Aggregated Gradient for Communication-Efficient Distributed Learning
This paper presents a new class of gradient methods for distributed machine learning that adaptively skip the gradient calculations to learn with reduced communication and computation. Simple rules are designed to detect slowly-varying gradients and, therefore, trigger the reuse of outdated gradients. The resultant gradient-based algorithms are termed Lazily Aggregated Gradient -justifying our acronym LAG used henceforth. Theoretically, the merits of this contribution are: i) the convergence rate is the same as batch gradient descent in stronglyconvex, convex, and nonconvex cases; and, ii) if the distributed datasets are heterogeneous (quantified by certain measurable constants), the communication rounds needed to achieve a targeted accuracy are reduced thanks to the adaptive reuse of lagged gradients. Numerical experiments on both synthetic and real data corroborate a significant communication reduction compared to alternatives.

Review:
This paper explores the question of minimizing the communication among workers while solving an optimization problem in a distributed fashion. In particular, the authors argue that most of the existing work in this direction focused on minimizing the amount of data during each message exchange between the works. In contrast, the authors focus on reducing the number of such exchanges (or communication rounds) during the optimization procedures. The authors argue that reducing the number of rounds is more beneficial as it does not degrade the rate of convergence for a wide class of objective functions.  The authors propose a simple approach to reduce the communication: a worker sends the current values of the gradient (based on its data) only if this gradient is significantly far from the previous gradient supplied by this worker, leading to the name lazily aggregated gradient (LAG). The authors propose two variants of this approach depending on whether the parameter server or the individual worker decides when to send the current estimate of the gradient. Under natural assumptions of the objective functions, such as smoothness, convexity, or strong-convexity, the authors show that their approach results into the convergence rate similar to that of the conventional gradient descent method where each worker sends the current estimate of the gradient (based on its data) during each iteration. Using a detailed experimental evaluation, the authors show that the LAG approach significantly outperforms some of the previous approaches to reduce the communication in terms of both the convergence rate and the total amount of communication among the workers.   The paper proposes a simple yet effective method to reduce the overall communication load for a distributed optimization problem. The proposed method has favorable performance guarantees and has been shown work well with both synthetic and real datasets.   The authors argue that the proposed solution can be combined with other approaches to reduce communication, such as quantized gradients and second-order methods. Can authors comment on how the theoretical guarantees might be affected by bringing in these additional ideas?