Original: O N D ISTRIBUTED A DAPTIVE O PTIMIZATION WITH G RADIENT C OMPRESSION
Revised: O N D ISTRIBUTED A DAPTIVE O PTIMIZATION WITH G RADIENT C OMPRESSION

Original: A BSTRACT
Revised: A BSTRACT

Original: We study C OMP -AMS, a distributed optimization framework based on gradient averaging and adaptive AMSGrad algorithm.
Revised: We study C OMP -AMS, a distributed optimization framework based on gradient averaging and adaptive AMSGrad algorithm.

Original: Gradient compression with error feedback is applied to reduce the communication cost in the gradient transmission process.
Revised: Gradient compression with error feedback is applied to reduce the communication cost in the gradient transmission process.

Original: Our convergence analysis of C OMP -AMS shows that such compressed gradient averaging strategy yields same convergence rate as standard AMSGrad, and also exhibits linear speedup effect w.r.t.
Revised: Our convergence analysis of C OMP -AMS shows that such compressed gradient averaging strategy yields same convergence rate as standard AMSGrad, and also exhibits linear speedup effect w.r.t.

Original: the number of local workers.
Revised: the number of local workers.

Original: Compared with recently proposed protocols on distributed adaptive methods, C OMPAMS is simple and convenient.
Revised: Compared with recently proposed protocols on distributed adaptive methods, C OMPAMS is simple and convenient.

Original: Numerical experiments are conducted to justify the theoretical ﬁndings, and demonstrate that the proposed method can achieve same test accuracy as full-gradient AMSGrad with substantial communication savings.
Revised: Numerical experiments are conducted to justify the theoretical ﬁndings, and demonstrate that the proposed method can achieve same test accuracy as full-gradient AMSGrad with substantial communication savings.

Original: With its simplicity and efﬁciency, C OMP -AMS can serve as a useful distributed training framework for adaptive  methods.
Revised: With its simplicity and efﬁciency, C OMP -AMS can serve as a useful distributed training framework for adaptive gradient methods.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Deep neural network has achieved the state-of-the-art learning performance on numerous AI applications, e g., computer vision and natural language processing (Graves et al., 2013; Goodfellow et al., 2014;
Revised: Deep neural network has achieved the state-of-the-art learning performance on numerous AI applications, e.g., computer vision and natural language processing (Graves et al., 2013; Goodfellow et al., 2014;

Original: He et al., 2016; Young et al., 2018; Zhang et al., 2018), reinforcement learning (Mnih et al., 2013; Levine et al., 2016; Silver et al., 2017), recommendation systems (Covington et al., 2016), computational advertising (Zhao et al., 2019; Xu et al., 2021; Zhao et al., 2022) etc.
Revised: He et al., 2016; Young et al., 2018; Zhang et al., 2018), reinforcement learning (Mnih et al., 2013; Levine et al., 2016; Silver et al., 2017), recommendation systems (Covington et al., 2016), computational advertising (Zhao et al., 2019; Xu et al., 2021; Zhao et al., 2022) etc.

Original: With the increasing size of data and growing complexity of deep neural networks, standard single-machine training procedures encounter at least two major challenges:
Revised: With the increasing size of data and growing complexity of deep neural networks, standard single-machine training procedures encounter at least two major challenges:

Original: • Due to the limited computing power of a single-machine, processing the massive number of data samples takes a long time—training is too slow.
Revised: • Due to the limited computing power of a single-machine, processing the massive number of data samples takes a long time—training is too slow.

Original: Many real-world applications can not afford spending days or even weeks on training.
Revised: Many real-world applications cannot afford spending days or even weeks on training.

Original: • In many scenarios, data are stored in multiple servers, possibly at different locations, due to the storage constraints (massive user behavior data, Internet images, etc.) or privacy reasons (Chang et al., 2018).
Revised: • In many scenarios, data are stored on multiple servers, possibly at different locations, due to the storage constraints (massive user behavior data, Internet images, etc.) or privacy reasons (Chang et al., 2018).

Original: Hence, transmitting data among servers might be costly.
Revised: Hence, transmitting data among servers might be costly.

Original: Distributed learning framework has been commonly used to tackle the above two issues.
Revised: Distributed learning framework has been commonly used to tackle the above two issues.

Original: Consider the distributed optimization task where n workers jointly solve the following optimization problem:
Revised: Consider the distributed optimization task where n workers jointly solve the following optimization problem

Original: [Equation]
Revised: [Equation]

Original: where the non-convex function f
Revised: where the non-convex function f

Original: i represents the average loss over the local data samples for worker i ∈ [ n ] , and θ ∈ R d the global model parameter.
Revised: i represents the average loss over the local data samples for worker i ∈ [ n ] , and θ ∈ R d the global model parameter.

Original: X i is the data distribution on each local node.
Revised: X i is the data distribution on each local node.

Original: In the classical centralized distributed setting, in each iteration the central server uniformly randomly assigns the data to n local workers ( X i ’s are the same), at which the gradients of the model are computed in parallel.
Revised: In the classical centralized distributed setting, in each iteration the central server uniformly randomly assigns the data to n local workers ( X i ’s are the same), at which the gradients of the model are computed in parallel.

Original: Then the central server aggregates the local gradients, updates the global model (e.g., by stochastic gradient descent (SGD)), and transmits back the updated model to the local nodes for subsequent gradient computation.
Revised: Then the central server aggregates the local gradients, updates the global model (e.g., by stochastic gradient descent (SGD)), and transmits back the updated model to the local nodes for subsequent gradient computation.

Original: The scenario where X i ’s are different gives rise to the recently proposed Federated Learning (FL) (McMahan et al., 2017) framework, which will not be the major focus of this work.
Revised: The scenario where X i ’s are different gives rise to the recently proposed Federated Learning (FL) (McMahan et al., 2017) framework, which will not be the major focus of this work.

Original: As we can see, distributed training naturally solves aforementioned issues: 1) We use n computing nodes to train the model, so the time per training epoch can be largely reduced; 2) There is no need to transmit the local data to central server.
Revised: As we can see, distributed training naturally solves aforementioned issues: 1) We use n computing nodes to train the model, so the time per training epoch can be largely reduced; 2) There is no need to transmit the local data to central server.

Original: Besides, distributed training also provides stronger error tolerance since the training process could continue even one local machine breaks down.
Revised: Besides, distributed training also provides stronger error tolerance since the training process could continue even one local machine breaks down.

Original: As a result of these advantages, there has been a surge of study and applications on distributed systems (Nedic & Ozdaglar, 2009; Boyd et al., 2011; Duchi et al., 2012;
Revised: As a result of these advantages, there has been a surge of study and applications on distributed systems (Nedic & Ozdaglar, 2009; Boyd et al., 2011; Duchi et al., 2012;

Original: Goyal et al., 2017; Hong et al., 2017; Koloskova et al., 2019; Lu et al., 2019).
Revised: Goyal et al., 2017; Hong et al., 2017; Koloskova et al., 2019; Lu et al., 2019).

Original: Gradient compression.
Revised: Gradient compression.

Original: Among many optimization strategies, SGD is still the most popular prototype in distributed training for its simplicity and effectiveness (Chilimbi et al., 2014; Agarwal et al., 2018; Mikami et al., 2018).
Revised: Among many optimization strategies, SGD is still the most popular prototype in distributed training for its simplicity and effectiveness (Chilimbi et al., 2014; Agarwal et al., 2018; Mikami et al., 2018).

Original: Yet, when the deep learning model is very large, the communication between local nodes and central server could be expensive, and the burdensome gradient transmission would slow down the whole training system.
Revised: Yet, when the deep learning model is very large, the communication between local nodes and central server could be expensive, and the burdensome gradient transmission would slow down the whole training system.

Original: Thus, reducing the communication cost in distributed
Revised: Thus, reducing the communication cost in distributed

Original: SGD has become an active topic, and an important ingredient of large-scale distributed systems (e.g., Seide et al.
Revised: SGD has become an active topic, and an important ingredient of large-scale distributed systems (e.g., Seide et al.

Original: Solutions based on quantization, sparsiﬁcation and other compression techniques of the local gradients are proposed, e.g., Aji & Heaﬁeld (2017);
Revised: Solutions based on quantization, sparsiﬁcation and other compression techniques of the local gradients are proposed, e.g., Aji & Heaﬁeld (2017);

Original: Alistarh et al.
Revised: Alistarh et al.

Original: Sa et al.
Revised: Sa et al.

Original: Wen et al.
Revised: Wen et al.

Original: ; Bernstein et al.
Revised: ; Bernstein et al.

Original: ; Stich et al.
Revised: ; Stich et al.

Original: ; Wangni et al.
Revised: ; Wangni et al.

Original: Ivkin et al.
Revised: Ivkin et al.

Original: ; Yang et al.
Revised: ; Yang et al.

Original: ; Haddadpour et al.
Revised: ; Haddadpour et al.

Original: However, it has been observed both theoretically and empirically (Stich et al., 2018; Ajalloeian & Stich, 2020), that directly updating with the compressed gradients usually brings non-negligible performance downgrade in terms of convergence speed and accuracy.
Revised: However, it has been observed both theoretically and empirically (Stich et al., 2018; Ajalloeian & Stich, 2020), that directly updating with the compressed gradients usually brings non-negligible performance downgrade in terms of convergence speed and accuracy.

Original: To tackle this problem, studies (e.g., Stich et al.
Revised: To tackle this problem, studies (e.g., Stich et al.

Original: ; Karimireddy et al.
Revised: ; Karimireddy et al.

Original: (2019)) show that the technique of error feedback can to a large extent remedy the issue of such gradient compression, achieving same convergence rate as full-gradient SGD.
Revised: (2019)) show that the technique of error feedback can to a large extent remedy the issue of such gradient compression, achieving same convergence rate as full-gradient SGD.

Original: Adaptive optimization.
Revised: Adaptive optimization.

Original: In recent years, adaptive optimization algorithms (e.g., AdaGrad (Duchi et al., 2010), Adam (Kingma & Ba, 2015) and AMSGrad (Reddi et al., 2018)) have become popular because of their superior empirical performance.
Revised: In recent years, adaptive optimization algorithms (e.g., AdaGrad (Duchi et al., 2010), Adam (Kingma & Ba, 2015) and AMSGrad (Reddi et al., 2018)) have become popular because of their superior empirical performance.

Original: These methods use different implicit learning rates for different coordinates that keep changing adaptively throughout the training process, based on the learning trajectory.
Revised: These methods use different implicit learning rates for different coordinates that keep changing adaptively throughout the training process, based on the learning trajectory.

Original: In many cases, adaptive methods have been shown to converge faster than SGD, sometimes with better generalization as well.
Revised: In many cases, adaptive methods have been shown to converge faster than SGD, sometimes with better generalization as well.

Original: However, the body of literature that extends adaptive methods to distributed training is still fairly limited.
Revised: However, the body of literature that extends adaptive methods to distributed training is still fairly limited.

Original: In particular, even the simple gradient averaging approach, though appearing standard, has not been analyzed for adaptive optimization algorithms.
Revised: In particular, even the simple gradient averaging approach, though appearing standard, has not been analyzed for adaptive optimization algorithms.

Original: Given that distributed SGD with compressed gradient averaging can match the performance of standard SGD, one natural question is: is it also true for adaptive methods?
Revised: Given that distributed SGD with compressed gradient averaging can match the performance of standard SGD, one natural question is: is it also true for adaptive methods?

Original: In this work, we ﬁll this gap formally, by analyzing C OMP -AMS, a distributed adaptive optimization framework using the gradient averaging protocol, with communication-efﬁcient gradient compression.
Revised: In this work, we ﬁll this gap formally, by analyzing C OMP -AMS, a distributed adaptive optimization framework using the gradient averaging protocol, with communication-efﬁcient gradient compression.

Original: Our method has been implemented in PaddlePaddle platform ( https://www.paddlepaddle.org.cn ).
Revised: Our method has been implemented in PaddlePaddle platform ( https://www.paddlepaddle.org.cn ).

Original: Our contributions.
Revised: Our contributions.

Original: We study a simple algorithm design leveraging the adaptivity of AMSGrad and the computational virtue of local gradient compression :
Revised: We study a simple algorithm design leveraging the adaptivity of AMSGrad and the computational virtue of local gradient compression :

Original: • We consider C OMP -AMS, a synchronous distributed adaptive optimization framework based on global averaging with gradient compression, which is efﬁcient in both communication and memory as no local moment estimation is needed.
Revised: • We consider C OMP -AMS, a synchronous distributed adaptive optimization framework based on global averaging with gradient compression, which is efﬁcient in both communication and memory as no local moment estimation is needed.

Original: We consider the BlockSign and Top- k compressors, coupled with the error-feedback technique to compensate for the bias implied by the compression step for fast convergence.
Revised: We consider the BlockSign and Top- k compressors, coupled with the error-feedback technique to compensate for the bias implied by the compression step for fast convergence.

Original: • We provide the convergence analysis of distributed C OMP -AMS (with n workers) in smooth non-convex optimization.
Revised: • We provide the convergence analysis of distributed C OMP -AMS (with n workers) in smooth non-convex optimization.

Original: In the special case of n = 1 (single machine), similar to SGD, gradient compression with error feedback in adaptive method achieves same convergence rate
Revised: In the special case of n = 1 (single machine), similar to SGD, gradient compression with error feedback in adaptive method achieves same convergence rate

Original: O ( 1 √ T ) as the standard full-gradient counterpart.
Revised: O ( 1 √ T ) as the standard full-gradient counterpart.

Original: Also, we show that with a properly chosen learning rate, C OMP -AMS achieves O ( 1 √ nT ) convergence, implying a linear speedup in terms of the number of local workers to attain a stationary point.
Revised: Also, we show that with a properly chosen learning rate, C OMP -AMS achieves O ( 1 √ nT ) convergence, implying a linear speedup in terms of the number of local workers to attain a stationary point.

Original: • Experiments are conducted on various training tasks on image classiﬁcation and sentiment analysis to validate our theoretical ﬁndings on the linear speedup effect.
Revised: • Experiments are conducted on various training tasks on image classiﬁcation and sentiment analysis to validate our theoretical ﬁndings on the linear speedup effect.

Original: Our results show that C OMP -AMS has comparable performance with other distributed adaptive methods, and approaches the accuracy of full-precision AMSGrad with substantially reduced communication cost.
Revised: Our results show that C OMP -AMS has comparable performance with other distributed adaptive methods, and approaches the accuracy of full-precision AMSGrad with substantially reduced communication cost.

Original: Thus, it can serve as a convenient distributed training strategy in practice.
Revised: Thus, it can serve as a convenient distributed training strategy in practice.

Original:  R ELATED W ORK 2. D ISTRIBUTED SGD WITH C OMPRESSED G RADIENTS
Revised:  R ELATED W ORK 2. D ISTRIBUTED SGD WITH C OMPRESSED G RADIENTS

Original: Quantization.
Revised: Quantization.

Original: To reduce the expensive communication in large-scale distributed SGD training systems, extensive works have considered various compression techniques applied to the gradient transaction procedure.
Revised: To reduce the expensive communication in large-scale distributed SGD training systems, extensive works have considered various compression techniques applied to the gradient transaction procedure.

Original: The ﬁrst strategy is quantization.
Revised: The ﬁrst strategy is quantization.

Original: Dettmers (2016) condenses 32-bit ﬂoating numbers into 8-bits when representing the gradients.
Revised: Dettmers (2016) condenses 32-bit ﬂoating numbers into 8-bits when representing the gradients.

Original: Seide et al.
Revised: Seide et al.

Original: ; Bernstein et al.
Revised: ; Bernstein et al.

Original: (2018; 2019); Karimireddy et al.
Revised: (2018; 2019); Karimireddy et al.

Original: (2019) use the extreme 1-bit information (sign) of the gradients, combined with tricks like momentum, majority vote and memory.
Revised: (2019) use the extreme 1-bit information (sign) of the gradients, combined with tricks like momentum, majority vote and memory.

Original: Other quantization-based methods include QSGD (Alistarh et al., 2017; Zhang et al., 2017;
Revised: Other quantization-based methods include QSGD (Alistarh et al., 2017; Zhang et al., 2017;

Original: Wu et al., 2018) and LPC-SVRG (Yu et al., 2019b), leveraging unbiased stochastic quantization.
Revised: Wu et al., 2018) and LPC-SVRG (Yu et al., 2019b), leveraging unbiased stochastic quantization.

Original: Quantization has been successfully applied to industrial-level applications, e.g., Xu et al.
Revised: Quantization has been successfully applied to industrial-level applications, e.g., Xu et al.

Original: The saving in communication of quantization methods is moderate: for example, 8-bit quantization reduces the cost to 25% (compared with 32-bit full-precision).
Revised: The saving in communication of quantization methods is moderate: for example, 8-bit quantization reduces the cost to 25% (compared with 32-bit full-precision).

Original: Even in the extreme 1-bit case, the largest compression ratio is around 1 / 32 ≈ 3 .
Revised: Even in the extreme 1-bit case, the largest compression ratio is around 1 / 32 ≈ 3 .

Original: Sparsiﬁcation.
Revised: Sparsiﬁcation.

Original: Gradient sparsiﬁcation is another popular solution which may provide higher compression rate.
Revised: Gradient sparsiﬁcation is another popular solution which may provide higher compression rate.

Original: Instead of commuting the full gradient, each local worker only passes a few coordinates to the central server and zeros out the others.
Revised: Instead of commuting the full gradient, each local worker only passes a few coordinates to the central server and zeros out the others.

Original: Thus, we can more freely choose higher compression ratio (e.g., 1%, 0.1%), still achieving impressive performance in many applications (Lin et al., 2018).
Revised: Thus, we can more freely choose higher compression ratio (e.g., 1%, 0.1%), still achieving impressive performance in many applications (Lin et al., 2018).

Original: Stochastic sparsiﬁcation methods, including uniform and magnitude based sampling (Wangni et al., 2018), select coordinates based on some sampling probability, yielding unbiased gradient compressors with proper scaling.
Revised: Stochastic sparsiﬁcation methods, including uniform and magnitude based sampling (Wangni et al., 2018), select coordinates based on some sampling probability, yielding unbiased gradient compressors with proper scaling.

Original: Deterministic methods are simpler, e.g., Random- k , Top- k
Revised: Deterministic methods are simpler, e.g., Random- k , Top- k

Original: (Stich et al., 2018; Shi et al., 2019) (selecting k elements with largest magnitude), Deep Gradient Compression (Lin et al., 2018), but usually lead to biased gradient estimation.
Revised: (Stich et al., 2018; Shi et al., 2019) (selecting k elements with largest magnitude), Deep Gradient Compression (Lin et al., 2018), but usually lead to biased gradient estimation.

Original: More applications and analysis of compressed distributed SGD can be found in Alistarh et al.
Revised: More applications and analysis of compressed distributed SGD can be found in Alistarh et al.

Original: ; Jiang & Agrawal (2018); Jiang et al.
Revised: ; Jiang & Agrawal (2018); Jiang et al.

Original: ; Shen et al.
Revised: ; Shen et al.

Original: ; Basu et al.
Revised: ; Basu et al.

Original: (2019), among others.
Revised: (2019), among others.

Original: Error Feedback (EF).
Revised: Error Feedback (EF).

Original: Biased gradient estimation, which is a consequence of many aforementioned methods (e.g., signSGD, Top- k ), undermines the model training, both theoretically and empirically, with slower convergence and worse generalization (Ajalloeian & Stich, 2020; Beznosikov et al., 2020).
Revised: Biased gradient estimation, which is a consequence of many aforementioned methods (e.g., signSGD, Top- k ), undermines the model training, both theoretically and empirically, with slower convergence and worse generalization (Ajalloeian & Stich, 2020; Beznosikov et al., 2020).

Original: The technique of error feedback is able to “correct for the bias” and ﬁx the convergence issues.
Revised: The technique of error feedback is able to “correct for the bias” and ﬁx the convergence issues.

Original: In this procedure, the difference between the true stochastic gradient and the compressed one is accumulated locally, which is then added back to the local gradients in later iterations.
Revised: In this procedure, the difference between the true stochastic gradient and the compressed one is accumulated locally, which is then added back to the local gradients in later iterations.

Original: Stich et al.
Revised: Stich et al.

Original: ; Karimireddy et al.
Revised: ; Karimireddy et al.

Original: (2019) prove the O ( 1 T ) and O ( 1 √ T ) convergence rate of EF-SGD in strongly convex and non-convex setting respectively, matching the rates of vanilla SGD (Nemirovski et al., 2009; Ghadimi & Lan, 2013).
Revised: (2019) prove the O ( 1 T ) and O ( 1 √ T ) convergence rate of EF-SGD in strongly convex and non-convex setting respectively, matching the rates of vanilla SGD (Nemirovski et al., 2009; Ghadimi & Lan, 2013).

Original: More recent works on the convergence rate of SGD with error feedback include Stich & Karimireddy (2019); Zheng et al.
Revised: More recent works on the convergence rate of SGD with error feedback include Stich & Karimireddy (2019); Zheng et al.

Original: ; Richtárik et al.
Revised: ; Richtárik et al.

Original: (2021), etc.
Revised: (2021), etc.

Original: 2. A DAPTIVE O PTIMIZATION
Revised: 2. A DAPTIVE O PTIMIZATION

Original: Algorithm al., 2018)
Revised: Algorithm al., 2018)

Original: 2 t 7: ˆ v t =
Revised: 2 t 7: ˆ v t =

Original: max(ˆ v t − 1 , v t ) 8: θ t +1 =
Revised: max(ˆ v t − 1 , v t ) 8: θ t +1 =

Original: θ t − η t m t √ ˆ v t + (cid:15) 9: end for
Revised: θ t − η t m t √ ˆ v t + (cid:15) 9: end for

Original: In each SGD update, all the coordinates share the same learning rate, which is either constant or decreasing through the iterations.
Revised: In each SGD update, all the coordinates share the same learning rate, which is either constant or decreasing through the iterations.

Original: Adaptive optimization methods cast different learning rates on each dimension.
Revised: Adaptive optimization methods cast different learning rates on each dimension.

Original: For instance, AdaGrad, developed in Duchi et al.
Revised: For instance, AdaGrad, developed in Duchi et al.

Original: (2010), divides the gradient elementwise by (cid:113)(cid:80)
Revised: (2010), divides the gradient elementwise by (cid:113)(cid:80)

Original: Tt =1 g 2 t ∈ R d , where g t ∈ R
Revised: Tt =1 g 2 t ∈ R d , where g t ∈ R

Original: d is the gradient vector at time t and d is the model dimensionality.
Revised: d is the gradient vector at time t and d is the model dimensionality.

Original: Thus, it intrinsically assigns different learning rates to different coordinates throughout the training— elements with smaller previous gradient magnitudes tend to move a larger step via larger learning rate.
Revised: Thus, it intrinsically assigns different learning rates to different coordinates throughout the training— elements with smaller previous gradient magnitudes tend to move a larger step via larger learning rate.

Original: Other adaptive methods include AdaDelta (Zeiler, 2012) and Adam (Kingma & Ba, 2015), which introduce momentum and moving average of second moment estimation into AdaGrad hence leading to better performances.
Revised: Other adaptive methods include AdaDelta (Zeiler, 2012) and Adam (Kingma & Ba, 2015), which introduce momentum and moving average of second moment estimation into AdaGrad hence leading to better performances.

Original: AMSGrad (Reddi et al., 2018)
Revised: AMSGrad (Reddi et al., 2018)

Original: (Algorithm 1, which is the prototype in our paper), ﬁxes the potential convergence issue of Adam.
Revised: (Algorithm 1, which is the prototype in our paper), ﬁxes the potential convergence issue of Adam.

Original: Wang et al.
Revised: Wang et al.

Original: (2021) and Zhou et al.
Revised: (2021) and Zhou et al.

Original: (2020) improve the convergence and generalization of AMSGrad through optimistic acceleration and differential privacy.
Revised: (2020) improve the convergence and generalization of AMSGrad through optimistic acceleration and differential privacy.

Original: Adaptive optimization methods have been widely used in training deep learning models in language, computer vision and advertising applications, e.g., Choi et al.
Revised: Adaptive optimization methods have been widely used in training deep learning models in language, computer vision and advertising applications, e.g., Choi et al.

Original: ; You et al.
Revised: ; You et al.

Original: ; Zhang et al.
Revised: ; Zhang et al.

Original: (2021); Zhao et al.
Revised: (2021); Zhao et al.

Original: In distributed setting, Nazari et al.
Revised: In distributed setting, Nazari et al.

Original: ; Chen et al.
Revised: ; Chen et al.

Original: (2021b) studied decentralized adaptive methods, but communication efﬁciency was not considered.
Revised: (2021b) study decentralized adaptive methods, but communication efﬁciency was not considered.

Original: Mostly relevant to our work, Chen et al.
Revised: Mostly relevant to our work, Chen et al.

Original: (2021a) proposed a distributed training algorithm based on Adam, which requires every local node to store a local estimation of the moments of the gradient.
Revised: (2021a) proposes a distributed training algorithm based on Adam, which requires every local node to store a local estimation of the moments of the gradient.

Original: Thus, one has to keep extra two more tensors of the model size on each local worker, which may be less feasible in terms of memory particularly with large models.
Revised: Thus, one has to keep extra two more tensors of the model size on each local worker, which may be less feasible in terms of memory particularly with large models.

Original: More recently, Tang et al.
Revised: More recently, Tang et al.

Original: (2021) proposes an Adam pre-conditioned momentum SGD method.
Revised: (2021) proposes an Adam pre-conditioned momentum SGD method.

Original: In addition, in federated learning, Chen et al.
Revised: Chen et al.

Original: ; Karimireddy et al.
Revised: ; Karimireddy et al.

Revised: Reddi et al.

Original: (2020) proposes local adaptive  methods, which can be further accelerated via layer-wise adaptivity (Karimi et al., 2021).
Revised: (2021) proposed local/global adaptive FL methods, which can be further accelerated via layer-wise adaptivity (Karimi et al., 2021).

Original:  C OMMUNICATION -E FFICIENT A DAPTIVE O PTIMIZATION 3. G RADIENT C OMPRESSORS
Revised:  C OMMUNICATION -E FFICIENT A DAPTIVE O PTIMIZATION 3. G RADIENT C OMPRESSORS

Original: In this paper, we mainly consider deterministic q -deviate compressors deﬁned as below.
Revised: In this paper, we mainly consider deterministic q -deviate compressors deﬁned as below.

Original: Assumption 1.
Revised: Assumption 1.

Original: The gradient compressor C : R d (cid:55)→ R d is q -deviate: for ∀ x ∈ R d , ∃ 0 ≤ q <such that (cid:107)C ( x ) − x (cid:107) ≤ q (cid:107) x (cid:107) .
Revised: The gradient compressor C : R d (cid:55)→ R d is q -deviate: for ∀ x ∈ R d , ∃ 0 ≤ q <such that (cid:107)C ( x ) − x (cid:107) ≤ q (cid:107) x (cid:107) .

Original: Larger q indicates heavier compression, while smaller q implies better approximation of the true gradient.
Revised: Larger q indicates heavier compression, while smaller q implies better approximation of the true gradient.

Original: q = 0 implies C ( x ) = x , i.e., no compression.
Revised: q = 0 implies C ( x ) = x , i.e., no compression.

Original: In the following, we give two popular and efﬁcient q -deviate compressors that will be adopted in this paper.
Revised: In the following, we give two popular and efﬁcient q -deviate compressors that will be adopted in this paper.

Original: Deﬁnition 1 (Top- k ) .
Revised: Deﬁnition 1 (Top- k ) .

Original: For x ∈ R d , denote S as the size- k set of i ∈ [ d ] with largest k magnitude | x i | .
Revised: For x ∈ R d , denote S as the size- k set of i ∈ [ d ] with largest k magnitude | x i | .

Original: The Top- k compressor is deﬁned as C ( x ) i
Revised: The Top- k compressor is deﬁned as C ( x ) i

Original: = x i , if i ∈ S ; C ( x ) i = 0 otherwise.
Revised: = x i , if i ∈ S ; C ( x ) i = 0 otherwise.

Original: Deﬁnition 2 (Block-Sign) .
Revised: Deﬁnition 2 (Block-Sign) .

Original: For x ∈ R d , deﬁne M blocks indexed by B i , i = 1 , ..., M , with d i := |B i | .
Revised: For x ∈ R d , deﬁne M blocks indexed by B i , i = 1 , ..., M , with d i := |B i | .

Original: The Block-Sign compressor is deﬁned as C ( x ) =
Revised: The Block-Sign compressor is deﬁned as C ( x ) =

Original: [ sign ( x B 1 ) (cid:107) x B 1 (cid:107) 1 d, ..., sign ( x B M ) (cid:107) x B M (cid:107) 1 d M ] , where x B i is the sub-vector of x at indices B i .
Revised: [ sign ( x B 1 ) (cid:107) x B 1 (cid:107) 1 d, ..., sign ( x B M ) (cid:107) x B M (cid:107) 1 d M ] , where x B i is the sub-vector of x at indices B i .

Original: Remark 1.
Revised: Remark 1.

Original: It is well-known (Stich et al., 2018) that for Top- k , q 2 = 1 − kd .
Revised: It is well-known (Stich et al., 2018) that for Top- k , q 2 = 1 − kd .

Original: For Block-Sign , by Cauchy-Schwartz inequality we have q 2 = 1 − min
Revised: For Block-Sign , by Cauchy-Schwartz inequality we have q 2 = 1 − min

Original: i ∈
Revised: i ∈

Original: [ M ] 1 d i where M and d i are deﬁned in
Revised: [ M ] 1 d i where M and d i are deﬁned in

Original: Deﬁnition 2 (Zheng et al., 2019).
Revised: Deﬁnition 2 (Zheng et al., 2019).

Original: The intuition behind Top- k is that, it has been observed empirically that when training many deep models, most gradients are typically very small, and gradients with large magnitude contain most information.
Revised: The intuition behind Top- k is that, it has been observed empirically that when training many deep models, most gradients are typically very small, and gradients with large magnitude contain most information.

Original: The Block-Sign compressor is a simple extension of the 1-bit SIGN compressor (Seide et al., 2014; Bernstein et al., 2018), adapted to different gradient magnitude in different blocks, which, for neural nets, are usually set as the distinct network layers.
Revised: The Block-Sign compressor is a simple extension of the 1-bit SIGN compressor (Seide et al., 2014; Bernstein et al., 2018), adapted to different gradient magnitude in different blocks, which, for neural nets, are usually set as the distinct network layers.

Original: The scaling factor in Deﬁnitionis to preserve the (possibly very different) gradient magnitude in each layer.
Revised: The scaling factor in Deﬁnitionis to preserve the (possibly very different) gradient magnitude in each layer.

Original: In principle, Top- k would perform the best when the gradient is effectively sparse, while Block-Sign compressor is favorable by nature when most gradients have similar magnitude within each layer.
Revised: In principle, Top- k would perform the best when the gradient is effectively sparse, while Block-Sign compressor is favorable by nature when most gradients have similar magnitude within each layer.

Original: 3. C OMP -AMS: D ISTRIBUTED A DAPTIVE T RAINING BY G RADIENT A GGREGATION
Revised: 3. C OMP -AMS: D ISTRIBUTED A DAPTIVE T RAINING BY G RADIENT A GGREGATION

Original: We present in Algorithm 2 the proposed communication-efﬁcient distributed adaptive method in this paper, C OMP -AMS.
Revised: We present in Algorithm 2 the proposed communication-efﬁcient distributed adaptive method in this paper, C OMP -AMS.

Original: This framework can be regarded as an analogue to the standard synchronous distributed SGD: in each iteration, each local worker transmits to the central server the compressed stochastic gradient computed using local data.
Revised: This framework can be regarded as an analogue to the standard synchronous distributed SGD: in each iteration, each local worker transmits to the central server the compressed stochastic gradient computed using local data.

Original: Then the central server takes the average of local gradients, and performs an AMSGrad update.
Revised: Then the central server takes the average of local gradients, and performs an AMSGrad update.

Original: In Algorithm 2, line 7-8 depict the error feedback operation at local nodes.
Revised: In Algorithm 2, line 7-8 depict the error feedback operation at local nodes.

Original: e t,i is the accumulated error from gradient compression on the i -th worker up to time t − 1 .
Revised: e t,i is the accumulated error from gradient compression on the i -th worker up to time t − 1 .

Original: This residual is added back to g t,i to get the “corrected” gradient.
Revised: This residual is added back to g t,i to get the “corrected” gradient.

Original: In Sectionand Section 5, we will show that error feedback, similar to the case of SGD, also brings good convergence behavior under gradient compression in distributed AMSGrad.
Revised: In Sectionand Section 5, we will show that error feedback, similar to the case of SGD, also brings good convergence behavior under gradient compression in distributed AMSGrad.

Original: 1: Input : parameters β 1 , β 2 , (cid:15) , learning rate η t 2: Initialize : central server parameter θ 1 ∈ R d ⊆ R d ; e 1 ,i = 

Revised: 1: Input : parameters β 1 , β 2 , (cid:15) , learning rate η t 2: Initialize : central server parameter θ 1 ∈ R d ⊆ R d ; e 1 ,i =

Original: the error accumulator for each worker; m 0 = 0 , v 0 = 0 , ˆ v 0 =3:
Revised: the error accumulator for each worker; m 0 = 0 , v 0 = 0 , ˆ v 0 =3:

Original: for t = 1 , . . .
Revised: for t = 1 , . . .

Original: , T do 4: parallel for worker i ∈
Revised: , T do 4: parallel for worker i ∈

Original: [ n ] do : 5:
Revised: [ n ] do : 5:

Original: Receive model parameter θ t from central server 6:
Revised: Receive model parameter θ t from central server 6:

Original: Compute stochastic gradient g t,i at θ t 7:
Revised: Compute stochastic gradient g t,i at θ t 7:

Original: Compute the compressed gradient ˜ g t,i =
Revised: Compute the compressed gradient ˜ g t,i =

Original: C ( g t,i + e t,i ) 8:
Revised: C ( g t,i + e t,i ) 8:

Original: Update the error e t +1 ,i =
Revised: Update the error e t +1 ,i =

Original: e t,i + g t,i − ˜
Revised: e t,i + g t,i − ˜

Original: g t,i 9:
Revised: g t,i 9:

Original: Send ˜ g t,i back to central server 10: end parallel 11:
Revised: Send ˜ g t,i back to central server 10: end parallel 11:

Original: Central server do: 12: ¯ g t = 1 n (cid:80) ni =

Revised: Central server do: 12: ¯ g t = 1 n (cid:80) ni =

Original: ˜ g t,i 13: m t = β 1 m t − 1 + (1 − β 1 )¯ g t 14: v t = β 2 v t − 1 + (1 − β 2 )
Revised: ˜ g t,i 13: m t = β 1 m t − 1 + (1 − β 1 )¯ g t 14: v t = β 2 v t − 1 + (1 − β 2 )

Original: ¯ g 2 t 15: ˆ v t = max( v t , ˆ v t − 1 ) 16:
Revised: ¯ g 2 t 15: ˆ v t = max( v t , ˆ v t − 1 ) 16:

Original: Update the global model θ t +

Revised: Update the global model θ t +

Original: =
Revised: =

Original: θ t − η t m t √ ˆ v t + (cid:15) 17: end for
Revised: θ t − η t m t √ ˆ v t + (cid:15) 17: end for

Original: Comparison with related methods.
Revised: Comparison with related methods.

Original: Our method can be regarded as a compressed version of Reddi et al.

Original: (2021) with one local step.

Original: Next, we discuss the differences between C OMP -AMS and two recently proposed methods also trying to solve the distributed adaptive optimization problem.
Revised: Next, we discuss the differences between C OMP -AMS and two recently proposed methods also trying to solve compressed distributed adaptive optimization.

Original: • Comparison with Chen et al.
Revised: • Comparison with Chen et al.

Original: 2021a).
Revised: (2021a).

Original: Chen et al.
Revised: Chen et al.

Original: (2021a) develops a quantized variant of Adam (Kingma & Ba, 2015), called “QAdam”
Revised: (2021a) develops a quantized variant of Adam (Kingma & Ba, 2015), called “QAdam”.

Original: In this method, each worker keeps a local copy of the moment estimates, commonly noted m and v , and compresses and transmits the ratio mv as a whole to the server.
Revised: In this method, each worker keeps a local copy of the moment estimates, commonly noted m and v , and compresses and transmits the ratio mv as a whole to the server.

Original: Thus, that method is very much like the compressed distributed SGD, with the exception that the ratio mv plays the role of the gradient vector g communication-wise.
Revised: Thus, that method is very much like the compressed distributed SGD, with the exception that the ratio mv plays the role of the gradient vector g communication-wise.

Original: Thus, two local moment estimators are additionally required, which have same size as the deep learning model.
Revised: Thus, two local moment estimators are additionally required, which have same size as the deep learning model.

Original: In our C OMP -AMS, the moment estimates m and v are kept and updated only at the central server, thus not introducing any extra variable (tensor) on local nodes during training (except for the error accumulator).
Revised: In our C OMP -AMS, the moment estimates m and v are kept and updated only at the central server, thus not introducing any extra variable (tensor) on local nodes during training (except for the error accumulator).

Original: Hence, C OMP -AMS is not only effective in communication reduction, but also efﬁcient in terms of memory (space), which is feasible when training large-scale learners like BERT and CTR prediction models, e g., Devlin et al.
Revised: Hence, C OMP -AMS is not only effective in communication reduction, but also efﬁcient in terms of memory (space), which is feasible when training large-scale learners like BERT and CTR prediction models, e.g., Devlin et al.

Original: ; Xu et al.
Revised: ; Xu et al.

Original: (2021), to lower the hardware consumptionhappening in practice.
Revised: (2021), to lower the hardware consumption in practice.

Revised: Additionally, the convergence rate in Chen et al.

Revised: (2021a) does not improve linearly with n , while we prove the linear speedup effect of C OMP -AMS.

Original: Comparison with Tang et al.
Revised: Comparison with Tang et al.

Original: The recent work (Tang et al., 2021) proposes “1BitAdam”.
Revised: The recent work (Tang et al., 2021) proposes “1BitAdam”.

Original: They ﬁrst run some warm-up training steps using standard Adam, and then store the second moment moving average v .
Revised: They ﬁrst run some warm-up training steps using standard Adam, and then store the second moment moving average v .

Original: Then, distributed Adam training starts with v frozen.
Revised: Then, distributed Adam training starts with v frozen.

Original: Thus, 1BitAdam is actually more like a distributed momentum SGD with some pre-conditioned coordinate-wise learning rates.
Revised: Thus, 1BitAdam is actually more like a distributed momentum SGD with some pre-conditioned coordinate-wise learning rates.

Original: The number of warm-up steps also needs to be carefully tuned, otherwise bad pre-conditioning may hurt the learning performance.
Revised: The number of warm-up steps also needs to be carefully tuned, otherwise bad pre-conditioning may hurt the learning performance.

Original: Our C OMP -AMS is simpler, as no pre-training is needed.
Revised: Our C OMP -AMS is simpler, as no pre-training is needed.

Original: Also, 1BitAdam requires extra tensors for m locally, while C OMP -AMS does not need additional local memory.
Revised: Also, 1BitAdam requires extra tensors for m locally, while C OMP -AMS does not need additional local memory.

Original:  C ONVERGENCE A NALYSIS
Revised:  C ONVERGENCE A NALYSIS

Original: For the convergence analysis of C OMP -AMS we will make following additional assumptions.
Revised: For the convergence analysis of C OMP -AMS we will make following additional assumptions.

Original: Assumption 2.
Revised: Assumption 2.

Original: (Smoothness)
Revised: (Smoothness)

Original: For ∀ i ∈ [ n ] , f i is L-smooth: (cid:107)∇ f i ( θ ) − ∇ f i ( ϑ ) (cid:107) ≤ L (cid:107) θ − ϑ (cid:107) .
Revised: For ∀ i ∈ [ n ] , f i is L-smooth: (cid:107)∇ f i ( θ ) − ∇ f i ( ϑ ) (cid:107) ≤ L (cid:107) θ − ϑ (cid:107) .

Original: Assumption 3.
Revised: Assumption 3.

Original: (Unbiased and bounded stochastic gradient)
Revised: (Unbiased and bounded stochastic gradient)

Original: For ∀ t > 0 , ∀ i ∈ [ n ] , the stochastic gradient is unbiased and uniformly bounded: E [ g t,i ]
Revised: For ∀ t > 0 , ∀ i ∈ [ n ] , the stochastic gradient is unbiased and uniformly bounded: E [ g t,i ]

Original: = ∇ f i ( θ t ) and (cid:107) g t,i (cid:107) ≤ G .
Revised: = ∇ f i ( θ t ) and (cid:107) g t,i (cid:107) ≤ G .

Original: Assumption 4.
Revised: Assumption 4.

Original: (Bounded variance)
Revised: (Bounded variance)

Original: For ∀ t > 0 , ∀ i ∈ [ n ] : (i) the local variance of the stochastic gradient is bounded: E [ (cid:107) g t,i − ∇ f i ( θ t ) (cid:107) 2 ] < σ 2 ; (ii) the global variance is bounded byn (cid:80) ni
Revised: For ∀ t > 0 , ∀ i ∈ [ n ] : (i) the local variance of the stochastic gradient is bounded: E [ (cid:107) g t,i − ∇ f i ( θ t ) (cid:107) 2 ] < σ 2 ; (ii) the global variance is bounded byn (cid:80) ni

Original: =1 (cid:107)∇ f i ( θ t ) − ∇ f ( θ t ) (cid:107) 2 ≤ σ 2 g .
Revised: =1 (cid:107)∇ f i ( θ t ) − ∇ f ( θ t ) (cid:107) 2 ≤ σ 2 g .

Original: In Assumption 3, the uniform bound on the stochastic gradient is common in the convergence analysis of adaptive methods, e.g., Reddi et al.
Revised: In Assumption 3, the uniform bound on the stochastic gradient is common in the convergence analysis of adaptive methods, e.g., Reddi et al.

Original: ; Zhou et al.
Revised: ; Zhou et al.

Original: ; Chen et al.
Revised: ; Chen et al.

Original: The global variance bound σ 2 g in Assumption 4 characterizes the difference among local objective functions, which, is mainly caused by different local data distribution X i in (1).
Revised: The global variance bound σ 2 g in Assumption 4 characterizes the difference among local objective functions, which, is mainly caused by different local data distribution X i in (1).

Original: In classical distributed setting where all the workers can access the same dataset and local data are assigned randomly, σ 2 g ≡ 0 .
Revised: In classical distributed setting where all the workers can access the same dataset and local data are assigned randomly, σ 2 g ≡ 0 .

Original: While typical federated learning (FL) setting with σ 2 g > 0 is not the focus of this present paper, we consider the global variance in our analysis to shed some light on the impact of non-i.i.d.
Revised: While typical federated learning (FL) setting with σ 2 g > 0 is not the focus of this present paper, we consider the global variance in our analysis to shed some light on the impact of non-i.i.d.

Original: data distribution in the federated setting for broader interest and future investigation.
Revised: data distribution in the federated setting for broader interest and future investigation.

Original: We derive the following general convergence rate of C OMP -AMS in the distributed setting.
Revised: We derive the following general convergence rate of C OMP -AMS in the distributed setting.

Original: The proof is deferred to Appendix B.
Revised: The proof is deferred to Appendix B.

Original: Theorem 1.
Revised: Theorem 1.

Original: Denote C 

Revised: Denote C

Original: = (cid:113) 4(1+{+4(
 ++} q [-2 ) 
-]
Revised: = (cid:113) {+4(
 ++} q [-2 ) 
-][Equation]

Original:  (1 − q 2 ) 2 G 2 + (cid:15) , C 1 = β 1 1 − β 1 + 2 q 1 − q 2 , θ ∗ = arg min f ( θ ) deﬁned as (1).
Revised: 3 (1 − q [Equation] 2 G 2 + (cid:15) , C 1 = β 1 1 − β 1 + 2 q 1 − q 2 , θ ∗ = arg min f ( θ ) deﬁned as (1).

Original: Under Assumption 1 to 4, with η t = η ≤
Revised: Under Assumptions 1 to 4, with η t = η ≤

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original:  , Algorithm 2 satisﬁes
Revised: C 1 } , Algorithm 2 satisﬁes

Original: [Equation]
Revised: [Equation]

Original: The LHS of Theorem 1 is the expected squared norm of the gradient from a uniformly chosen iterate t ∈ [ T ] , which is a common convergence measure in non-convex optimization.
Revised: The LHS of Theorem 1 is the expected squared norm of the gradient from a uniformly chosen iterate t ∈ [ T ] , which is a common convergence measure in non-convex optimization.

Original: From Theorem 1, we see that the more compression we apply to the gradient vectors (i.e., larger q ), the larger the gradient magnitude is, i.e., the slower the algorithm converges.
Revised: From Theorem 1, we see that the more compression we apply to the gradient vectors (i.e., larger q ), the larger the gradient magnitude is, i.e., the slower the algorithm converges.

Original: This is intuitive as heavier compression loses more gradient information which would slower down the learner to ﬁnd a good solution.
Revised: This is intuitive as heavier compression loses more gradient information which would slower down the learner to ﬁnd a good solution.

Original: Note that, C OMP -AMS with n = 1 naturally reduces to the single-machine (sequential) AMSGrad (Algorithm 1) with compressed gradients instead of full-precision ones.
Revised: Note that, C OMP -AMS with n = 1 naturally reduces to the single-machine (sequential) AMSGrad (Algorithm 1) with compressed gradients instead of full-precision ones.

Original: Karimireddy et al.
Revised: Karimireddy et al.

Original: (2019) speciﬁcally analyzed this case for SGD, showing that compressed single-machine SGD with error feedback has the same convergence rate as vanilla SGD using full gradients.
Revised: (2019) speciﬁcally analyzed this case for SGD, showing that compressed single-machine SGD with error feedback has the same convergence rate as vanilla SGD using full gradients.

Original: In alignment with the conclusion in Karimireddy et al.
Revised: In alignment with the conclusion in Karimireddy et al.

Original: (2019), for adaptive AMSGrad, we have a similar result.
Revised: (2019), for adaptive AMSGrad, we have a similar result.

Original: Corollary 1.
Revised: Corollary 1.

Original: When n = 1 , under Assumption 1 to  4, setting the stepsize as η = min {
Revised: When n = 1 , under Assumption 1 to Assumption 4, setting the stepsize as η = min {

Original: [Equation]
Revised: [Equation]

Original: , Algorithm 2 satisﬁes:
Revised: , Algorithm 2 satisﬁes

Original: [Equation]
Revised: [Equation]

Original: Corollary 1 states that with error feedback, single machine AMSGrad with biased compressed gradients can also match the convergence rate
Revised: Corollary 1 states that with error feedback, single machine AMSGrad with biased compressed gradients can also match the convergence rate

Original: O ( 1 √ T + dT ) of standard AMSGrad
Revised: O ( 1 √ T + dT ) of standard AMSGrad

Original: (Zhou et al., 2018) in non-convex optimization.
Revised: (Zhou et al., 2018) in non-convex optimization.

Original: It also achieves the same rate O ( 1 √ T ) of vanilla SGD (Karimireddy et al., 2019) when T is sufﬁciently large.
Revised: It also achieves the same rate O ( 1 √ T ) of vanilla SGD (Karimireddy et al., 2019) when T is sufﬁciently large.

Original: In other words, EF also ﬁxes the convergence issue of using compressed gradients in AMSGrad.
Revised: In other words, error feedback also ﬁxes the convergence issue of using compressed gradients in AMSGrad.

Original: Linear Speedup.
Revised: Linear Speedup.

Original: In Theorem 1, the convergence rate is derived assuming constant learning rate.
Revised: In Theorem 1, the convergence rate is derived assuming constant learning rate.

Original: By carefully choosing a decreasing learning rate dependent on the number of workers, we have the following simpliﬁed statement.
Revised: By carefully choosing a decreasing learning rate dependent on the number of workers, we have the following simpliﬁed statement.

Original: Corollary 2.
Revised: Corollary 2.

Original: Under the same setting as Theorem 1, set η = min {
Revised: Under the same setting as Theorem 1, set η = min {

Original: [Equation]
Revised: [Equation]

Original: The
Revised: The

Original: C OMP -AMS iterates admit
Revised: C OMP -AMS iterates admit

Original: [Equation]
Revised: [Equation]

Original: In Corollary 2, we see that the global variance σ 2 g appears in the O ( 1 T ) term, which says that it asymptotically has no impact on the convergence.
Revised: In Corollary 2, we see that the global variance σ 2 g appears in the O ( 1 T ) term, which says that it asymptotically has no impact on the convergence.

Original: This matches the result of momentum SGD (Yu et al., 2019a).
Revised: This matches the result of momentum SGD (Yu et al., 2019a).

Original: When T ≥ O ( n 3 ) is sufﬁciently large, the third term in (2) vanishes, and the convergence rate becomes O ( 1 √ nT ) .
Revised: When T ≥ O ( n 3 ) is sufﬁciently large, the third term in (2) vanishes, and the convergence rate becomes O ( 1 √ nT ) .

Original: Therefore, to reach a O ( δ ) stationary point, one worker ( n = 1 ) needs T
Revised: Therefore, to reach a O ( δ ) stationary point, one worker ( n = 1 ) needs T

Original: = O ( 1 δ 2 ) iterations, while distributed training with n workers requires only T
Revised: = O ( 1 δ 2 ) iterations, while distributed training with n workers requires only T

Original: = O ( 1 nδ 2 ) iterations, which is n times faster than single machine training.
Revised: = O ( 1 nδ 2 ) iterations, which is n times faster than single machine training.

Original: That is, C OMP -AMS has a linear speedup in terms of the number of the local workers.
Revised: That is, C OMP -AMS has a linear speedup in terms of the number of the local workers.

Original: Such acceleration effect has also been reported for compressed SGD (Jiang & Agrawal, 2018; Zheng et al., 2019) and momentum SGD (Yu et al., 2019a) with error feedback.
Revised: Such acceleration effect has also been reported for compressed SGD (Jiang & Agrawal, 2018; Zheng et al., 2019) and momentum SGD (Yu et al., 2019a) with error feedback.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we provide numerical results on several real-world datasets.
Revised: In this section, we provide numerical results on several common datasets.

Original: Our main objective is to validate the theoretical results, and demonstrate that the proposed C OMP -AMS can approach the learning performance of full-precision AMSGrad with signiﬁcantly reduced communication costs.
Revised: Our main objective is to validate the theoretical results, and demonstrate that the proposed C OMP -AMS can approach the learning performance of full-precision AMSGrad with signiﬁcantly reduced communication costs.

Original: 5. D ATASETS , M ODELS AND M ETHODS
Revised: 5. D ATASETS , M ODELS AND M ETHODS

Original: Our experiments are conducted on various image and text datasets.
Revised: Our experiments are conducted on various image and text datasets.

Original: The MNIST (LeCun et al., 1998) contains 60000 training samples of 28 × 18 gray-scale hand-written digits from 10 classes, and 10000 test samples.
Revised: The MNIST (LeCun et al., 1998) contains 60000 training samples of 28 × 18 gray-scale hand-written digits from 10 classes, and 10000 test samples.

Original: We train MNIST with a Convolutional Neural Network (CNN), which has two convolutional layers followed by two fully connected layers with ReLu activation.
Revised: We train MNIST with a Convolutional Neural Network (CNN), which has two convolutional layers followed by two fully connected layers with ReLu activation.

Original: Dropout is applied after the max-pooled convolutional layer with rate 0.5.
Revised: Dropout is applied after the max-pooled convolutional layer with rate 0.5.

Original: The CIFAR-10 dataset (Krizhevsky & Hinton, 2009) consists of 50000 32 × 32 RGB natural images from 10 classes for training and 10000 images for testing, which is trained by LeNet-5 (LeCun et al., 1998).
Revised: The CIFAR-10 dataset (Krizhevsky & Hinton, 2009) consists of 50000 32 × 32 RGB natural images from 10 classes for training and 10000 images for testing, which is trained by LeNet-5 (LeCun et al., 1998).

Original: Moreover, we also implement ResNet-18 (He et al., 2016) on this dataset.
Revised: Moreover, we also implement ResNet-18 (He et al., 2016) on this dataset.

Original: The IMDB movie review (Maas et al., 2011) is a popular binary classiﬁcation dataset for sentiment analysis.
Revised: The IMDB movie review (Maas et al., 2011) is a popular binary classiﬁcation dataset for sentiment analysis.

Original: Each movie review is tokenized by top-2000 most frequently appeared words and transformed into integer vectors, which is of maximal length 500.
Revised: Each movie review is tokenized by top-2000 most frequently appeared words and transformed into integer vectors, which is of maximal length 500.

Original: We train a Long-Short Term Memory (LSTM) network with a 32-dimensional embedding layer and 64 LSTM cells, followed by two fully connected layers before output.
Revised: We train a Long-Short Term Memory (LSTM) network with a 32-dimensional embedding layer and 64 LSTM cells, followed by two fully connected layers before output.

Original: Cross-entropy loss is used for all the tasks.
Revised: Cross-entropy loss is used for all the tasks.

Original: Following the classical distributed training setting, in each training iteration, data sample are uniformly randomly assigned to the workers.
Revised: Following the classical distributed training setting, in each training iteration, data sample are uniformly randomly assigned to the workers.

Original: We compare C OMP -AMS with full-precision distributed AMSGrad, QAdam (Chen et al., 2021a) and 1BitAdam (Tang et al., 2021).
Revised: We compare C OMP -AMS with full-precision distributed AMSGrad, QAdam (Chen et al., 2021a) and 1BitAdam (Tang et al., 2021).

Original: For C OMP -AMS, Top- k picks top 1% gradient coordinates (i.e., sparsity 0.01).
Revised: For C OMP -AMS, Top- k picks top 1% gradient coordinates (i.e., sparsity 0.01).

Original: QAdam and 1BitAdam both use 1-bit quantization to achieve high compression.
Revised: QAdam and 1BitAdam both use 1-bit quantization to achieve high compression.

Original: For MNIST and CIFAR-10, the local batch size on each worker is set to be 32.
Revised: For MNIST and CIFAR-10, the local batch size on each worker is set to be 32.

Original: For IMDB, the local batch size is 16.
Revised: For IMDB, the local batch size is 16.

Original: The hyper-parameters in C OMP -AMS are set as default β 1 = 0 .
Revised: The hyper-parameters in C OMP -AMS are set as default β 1 = 0 .

Original: 9 , β 2 = 0 .
Revised: 9 , β 2 = 0 .

Original: 999 and (cid:15) =
Revised: 999 and (cid:15) =

Original: 10 − 8 , which are also used for QAdam and 1BitAdam.
Revised: 10 − 8 , which are also used for QAdam and 1BitAdam.

Original: For 1BitAdam, the epochs for warm-up training is set to be 1 / 20 of the total epochs.
Revised: For 1BitAdam, the epochs for warm-up training is set to be 1 / 20 of the total epochs.

Original: For all methods, we tune the initial learning rate over a ﬁne grid (see Appendix A) and report the best results averaged over three independent runs.
Revised: For all methods, we tune the initial learning rate over a ﬁne grid (see Appendix A) and report the best results averaged over three independent runs.

Original: Our experiments are performed on a GPU cluster with NVIDIA Tesla V100 cards.
Revised: Our experiments are performed on a GPU cluster with NVIDIA Tesla V100 cards.

Original: 5. G ENERAL E VALUATION AND C OMMUNICATION E FFICIENCY
Revised: 5. G ENERAL E VALUATION AND C OMMUNICATION E FFICIENCY

Original: The training loss and test accuracy on MNIST + CNN, CIFAR-10 + LeNet and IMDB + LSTM are reported in Figure 1.
Revised: The training loss and test accuracy on MNIST + CNN, CIFAR-10 + LeNet and IMDB + LSTM are reported in Figure 1.

Original: We provide more results on larger ResNet-18 model in Appendix A. On
Revised: We provide more results on larger ResNet-18 model in Appendix A. On

Original: CIFAR-10, we deploy a popular decreasing learning rate schedule, where the step size η is divided by 10 at the 40-th and 80-th epoch, respectively.
Revised: CIFAR-10, we deploy a popular decreasing learning rate schedule, where the step size η is divided by 10 at the 40-th and 80-th epoch, respectively.

Original: We observe:
Revised: We observe:

Original: • On MNIST, all the methods can approach the training loss and test accuracy of fullprecision AMSGrad.
Revised: • On MNIST, all the methods can approach the training loss and test accuracy of fullprecision AMSGrad.

Original: The 1BitAdam seems slightly better, but the gap is very small.
Revised: The 1BitAdam seems slightly better, but the gap is very small.

Original: On CIFAR-10, C OMP -AMS with Block-Sign performs the best and matches AMSGrad in terms of test accuracy.
Revised: On CIFAR-10, C OMP -AMS with Block-Sign performs the best and matches AMSGrad in terms of test accuracy.

Original: • On IMDB, C OMP -AMS with Top- k has both the fastest convergence and best generalization compared with other compressed methods.
Revised: • On IMDB, C OMP -AMS with Top- k has both the fastest convergence and best generalization compared with other compressed methods.

Original: This is because the IMDB text data is more sparse (with many padded zeros), where Top- k is expected to work better than sign.
Revised: This is because the IMDB text data is more sparse (with many padded zeros), where Top- k is expected to work better than sign.

Original: The 1BitAdam converges slowly.
Revised: The 1BitAdam converges slowly.

Original: We believe one possible reason is that 1BitAdam is quite sensitive to the quality of the warm-up training.
Revised: We believe one possible reason is that 1BitAdam is quite sensitive to the quality of the warm-up training.

Original: For sparse text data, the estimation of second moment v is more unstable, making the strategy of freezing v by warm-up less effective.
Revised: For sparse text data, the estimation of second moment v is more unstable, making the strategy of freezing v by warm-up less effective.

Original: [Figure]
Revised: [Figure]

Original: Communication Efﬁciency.
Revised: Communication Efﬁciency.

Original: In Figure 2, we plot the training loss and test accuracy against the number of bits transmitted to the central server during the distributed training process, where we assume that the full-precision gradient is represented using 32 bits per ﬂoating number.
Revised: In Figure 2, we plot the training loss and test accuracy against the number of bits transmitted to the central server during the distributed training process, where we assume that the full-precision gradient is represented using 32 bits per ﬂoating number.

Original: As we can see, C OMP -AMS- Top- 0 .
Revised: As we can see, C OMP -AMS- Top- 0 .

Original: 01 achieves around 100x communication reduction, to attain similar accuracy as the full-precision distributed AMSGrad.
Revised: 01 achieves around 100x communication reduction, to attain similar accuracy as the full-precision distributed AMSGrad.

Original: The saving of Block-Sign is around 30x
Revised: The saving of Block-Sign is around 30x

Original: [Figure]
Revised: [Figure]

Original: but it gives slightly higher accuracy than Top- 0 . 01 on MNIST and CIFAR-10.
Revised: but it gives slightly higher accuracy than Top- 0 . 01 on MNIST and CIFAR-10.

Original: In all cases, C OMPAMS can substantially reduce the communication cost compared with full-precision distributed
Revised: In all cases, C OMPAMS can substantially reduce the communication cost compared with full-precision distributed

Original: AMSGrad, without losing accuracy
Revised: AMSGrad, without losing accuracy

Original: [Figure]
Revised: [Figure]

Original: QAdam1BitAdam
Revised: QAdam1BitAdam

Original: [Figure]
Revised: [Figure]

Original: T r a i n
Revised: T r a i n

Original: Lo ss
Revised: Lo ss

Original: MNIST n = 2 n =n = 8 n =
Revised: MNIST n = 2 n =n = 8 n =

Original: Iterations 0.
Revised: Iterations 0.

Original: [Figure]
Revised: [Figure]

Original: T r a i n
Revised: T r a i n

Original: Lo ss
Revised: Lo ss

Original: CIFAR n = 2 n =
Revised: CIFAR n = 2 n =

Original: [Figure]
Revised: [Figure]

Original: n = 8 n =
Revised: n = 8 n =

Original: Corollary 2 reveals the linear speedup of C OMP -AMS in distributed training.
Revised: Corollary 2 reveals the linear speedup of C OMP -AMS in distributed training.

Original: In Figure 3, we present the training loss on MNIST and CIFAR-10 against the number of iterations, with varying number of workers n .
Revised: In Figure 3, we present the training loss on MNIST and CIFAR-10 against the number of iterations, with varying number of workers n .

Original: We use C OMP -AMS with Block-Sign on MNIST, and Top- k -0.01 on CIFAR.
Revised: We use C OMP -AMS with Block-Sign on MNIST, and Top- k -0.01 on CIFAR.

Original: As suggested by the theory, we use 5 × 10 − 4 √ n as the learning rate.
Revised: As suggested by the theory, we use 5 × 10 − 4 √ n as the learning rate.

Original: From Figure 3, we see the number of iterations to achieve a certain loss exhibits a strong linear relationship with n —it (approximately) decreases by half whenever we double n , which justiﬁes the linear speedup of C OMP -AMS.
Revised: From Figure 3, we see the number of iterations to achieve a certain loss exhibits a strong linear relationship with n —it (approximately) decreases by half whenever we double n , which justiﬁes the linear speedup of C OMP -AMS.

Original: 5. D ISCUSSION
Revised: 5. D ISCUSSION

Original: We provide a brief summary of our empirical observations.
Revised: We provide a brief summary of our empirical observations.

Original: The proposed C OMP -AMS is able to match the learning performance of full-gradient AMSGrad in all the experiments.
Revised: The proposed C OMP -AMS is able to match the learning performance of full-gradient AMSGrad in all the experiments.

Original: In particular, for data/model involving some sparsity structure, C OMP -AMS with the Top- k compressor could be more effective.
Revised: In particular, for data/model involving some sparsity structure, C OMP -AMS with the Top- k compressor could be more effective.

Original: Also, our results reveal that 1BitAdam might be quite sensitive to the pre-conditioning quality, while C OMP -AMS can be more easily tuned and implemented in practice.
Revised: Also, our results reveal that 1BitAdam might be quite sensitive to the pre-conditioning quality, while C OMP -AMS can be more easily tuned and implemented in practice.

Original: We would like to emphasize that, the primary goal of the experiments is to show that C OMP -AMS is able to match the performance of full-precision AMSGrad, but not to argue that C OMP -AMS is always better than the other algorithms.
Revised: We would like to emphasize that, the primary goal of the experiments is to show that C OMP -AMS is able to match the performance of full-precision AMSGrad, but not to argue that C OMP -AMS is always better than the other algorithms.

Original: Since different methods use different underlying optimization algorithms (e.g., AMSGrad, Adam, momentum SGD), comparing C OMP -AMS with other distributed training methods would be largely determined by the comparison among these optimization protocols, which is typically data and task dependent.
Revised: Since different methods use different underlying optimization algorithms (e.g., AMSGrad, Adam, momentum SGD), comparing C OMP -AMS with other distributed training methods would be largely determined by the comparison among these optimization protocols, which is typically data and task dependent.

Original: Our results says that: whenever one wants to use AMSGrad to train a deep neural network, she/he can simply employ the distributed
Revised: Our results says that: whenever one wants to use AMSGrad to train a deep neural network, she/he can simply employ the distributed

Original: C OMP -AMS scheme to gain a linear speedup in training time with learning performance as good as the full-precision training, taking little communication cost and memory consumption.
Revised: C OMP -AMS scheme to gain a linear speedup in training time with learning performance as good as the full-precision training, taking little communication cost and memory consumption.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we study the simple, convenient, yet unexplored gradient averaging strategy for distributed adaptive optimization called C OMP -AMS.
Revised: In this paper, we study the simple, convenient, yet unexplored gradient averaging strategy for distributed adaptive optimization called C OMP -AMS.

Original: Top-
Revised: Top-

Original: k and Block-Sign compressor are incorporated for communication efﬁciency, whose biases are compensated by the error feedback strategy.
Revised: k and Block-Sign compressor are incorporated for communication efﬁciency, whose biases are compensated by the error feedback strategy.

Original: We develop the convergence rate of C OMP -AMS, and show that same as the case of SGD, for AMSGrad, compressed gradient averaging with error feedback matches the convergence of full-gradient
Revised: We develop the convergence rate of C OMP -AMS, and show that same as the case of SGD, for AMSGrad, compressed gradient averaging with error feedback matches the convergence of full-gradient

Original: AMSGrad, and linear speedup can be obtained in the distributed training.
Revised: AMSGrad, and linear speedup can be obtained in the distributed training.

Original: Numerical experiments are conducted to justify the theoretical ﬁndings, and demonstrate that C OMP -AMS provides comparable performance with other distributed adaptive methods, and achieves similar accuracy as fullprecision AMSGrad with signiﬁcantly reduced communication overhead.
Revised: Numerical experiments are conducted to justify the theoretical ﬁndings, and demonstrate that C OMP -AMS provides comparable performance with other distributed adaptive methods, and achieves similar accuracy as fullprecision AMSGrad with signiﬁcantly reduced communication overhead.

Original: Given the simple architecture and hardware (memory) efﬁciency, we expect C OMP -AMS to serve as a useful and convenient distributed adaptive optimization framework in practice.
Revised: Given the simple architecture and hardware (memory) efﬁciency, we expect C OMP -AMS to serve as a useful and convenient distributed adaptive optimization framework in practice.

Original: A T UNING D ETAILS AND M ORE R ESULTS ON R ES N ET - 
Revised: A T UNING D ETAILS AND M ORE R ESULTS ON R ES N ET -

Original: The search grids of the learning rate of each method can be found in Table 1.
Revised: The search grids of the learning rate of each method can be found in Table 1.

Original: Empirically, DistAMS, C OMP -AMS and 1BitAdam has similar optimal learning rate, while QAdam usually needs larger step size to reach its best performance.
Revised: Empirically, DistAMS, C OMP -AMS and 1BitAdam has similar optimal learning rate, while QAdam usually needs larger step size to reach its best performance.

Original: [Table]
Revised: [Table]

Original: We provide more experimental results on CIFAR-10 dataset, trained with ResNet-18 (He et al., 2016).
Revised: We provide more experimental results on CIFAR-10 dataset, trained with ResNet-18 (He et al., 2016).

Original: For reference
Revised: For reference

Original: [Table]
Revised: [Table]

Original: we also present the result of distributed SGD.
Revised: we also present the result of distributed SGD.

Original: As we can see from Figure
Revised: As we can see from Figure

Original: [Table]
Revised: [Table]

Original: again C OMP -AMS can achieve similar accuracy as AMSGrad, and the Top- k compressor gives the best accuracy, with substantial communication reduction.
Revised: again C OMP -AMS can achieve similar accuracy as AMSGrad, and the Top- k compressor gives the best accuracy, with substantial communication reduction.

Original: Note that distributed SGD converges faster than adaptive methods, but the generalization error is slightly worse.
Revised: Note that distributed SGD converges faster than adaptive methods, but the generalization error is slightly worse.

Original: This experiment again conﬁrms that C OMP -AMS can serve as a simple and convenient distributed adaptive training framework with fast convergence, reduced communication and little performance drop
Revised: This experiment again conﬁrms that C OMP -AMS can serve as a simple and convenient distributed adaptive training framework with fast convergence, reduced communication and little performance drop

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: ss
Revised: ss

Original: [Figure]
Revised: [Figure]

Original: B P ROOF OF C ONVERGENCE R ESULTSB. P ROOF OF T HEOREM
Revised: B P ROOF OF C ONVERGENCE R ESULTS

Revised: In this section, we provide the proof of our main result. B. P ROOF OF T HEOREM

Original: Theorem.
Revised: Theorem.

Original: Denote C 

Revised: Denote C

Original: = (cid:113) 4(1+ q 2 ) 

Revised: = (cid:113) 4(1+ q 2 )

Original: (1 − q 2 ) 2 G 2 + (cid:15) , C 1 = β 1 1 − β 1 + 2 q 1 − q 2 .
Revised: (1 − q 2 ) 2 G 2 + (cid:15) , C 1 = β 1 1 − β 1 + 2 q 1 − q 2 .

Original: Under Assumption 1 to Assumption 4, with η t = η ≤ (cid:15) 3 C 0 √ 2 L max { 2 L,C 2 } , for any T > 0 , C OMP -AMS satisﬁes
Revised: Under Assumption 1 to Assumption 4, with η t = η ≤ (cid:15) 3 C 0 √ 2 L max { 2 L,C 2 } , for any T > 0 , C OMP -AMS satisﬁes

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: We ﬁrst clarify some notations.
Revised: We ﬁrst clarify some notations.

Original: At time t , let the full-precision gradient of the j -th worker be g t,j , the error accumulator be e t,j , and the compressed gradient be ˜ g t,j = C ( g t,j + e t,j ) .
Revised: At time t , let the full-precision gradient of the i -th worker be g t,i , the error accumulator be e t,i , and the compressed gradient be ˜ g t,i = C ( g t,i + e t,i ) .

Revised: Slightly different from the notations in the algorithm, we denote ¯ g t = 1 n (cid:80) ni

Original: Denote ¯ g t = 1 n (cid:80) n j =1 g t,j , ˜ g t = 1 n (cid:80) n j =1{+ni =
+} ˜ g t,j and ¯ e t = 1 n (cid:80) n j =1 e t,j .
Revised: =1 g t,i , ˜ g t = 1 n (cid:80) {+ni =
+} ˜ g t,i and ¯ e t = 1 n (cid:80) ni =1 e t,i .

Original: The second moment computed by the compressed gradients is denoted as v t = β 2 v t − 1 + (1 − β 2 )˜
Revised: The second moment computed by the compressed gradients is denoted as v t = β 2 v t − 1 + (1 − β 2 )˜

Original: g 2 t , and ˆ v t = max { ˆ v t − 1 , v t } .
Revised: g 2 t , and ˆ v t = max { ˆ v t − 1 , v t } .

Original: Also, the ﬁrst order moving average sequence
Revised: Also, the ﬁrst order moving average sequence

Original: [Equation]
Revised: [Equation]

Original: = β
Revised: and m (cid:48)

Original: [Equation]
Revised: [Equation]

Original: t −
Revised: = β 1 m (cid:48)

Original: [Equation]
Revised: [Equation]

Original: 1 − β
Revised: t

Original: [Equation]
Revised: [Equation]

Original: and m (cid:48) t = β 1 m (cid:48) t − 1 + (1 − β 1 )¯ g t .
Revised: where m (cid:48) t represents the ﬁrst moment moving average sequence using the uncompressed stochastic gradients.

Original: By construction we have m (cid:48) t
Revised: By construction we have m (cid:48) t

Original: = (1 − β 1 ) (cid:80) ti =1 β t − [-i 
 ¯-] g i .
Revised: = (1 − β 1 ) (cid:80) tτ = 1 β t − [-i 
 ¯-]τ¯ g τ .

Original: Denote the following auxiliary sequences,
Revised: Our proof will use the following auxiliary sequences,

Original: [Equation]
Revised: [Equation]

Original: Then, 
Revised: Then, we can write the evolution of θ (cid:48) t as

Original: [Equation]
Revised: [Equation]

Original: where (a) uses the fact that for every j ∈ [ n ] , ˜ g t,j + e t +1 ,j = g t,j + e t,j , and e t, 1 = 0 at initialization.
Revised: where (a) uses the fact that for every i ∈ [ n ] , ˜ g t,i + e t +1 ,i = g t,i + e t,i , and e t, 1 = 0 at initialization.

Original: Further deﬁne the virtual iterates:
Revised: Further deﬁne the virtual iterates:

Original: [Equation]
Revised: [Equation]

Original: which follows the recurrence:
Revised: which follows the recurrence:

Original: [Equation]
Revised: [Equation]

Original: When summing over t = 1 , ..., T , the difference sequence D t satisﬁes the bounds of Lemma 5.
Revised: When summing over t = 1 , ..., T , the difference sequence D t satisﬁes the bounds of Lemma 5.

Original: By  Assumption 2 we have
Revised: By the smoothness Assumption [Equation] we have

Original: [Equation]
Revised: [Equation]

Original: Taking expectation w
Revised: Taking expectation w.r

Original: [Equation]
Revised: [Equation]

Original: r
Revised: t. the randomness at time t

Original: [Equation]
Revised: [Equation]

Original: t. the randomness at time t , we obtainE [ f
Revised: we obtain

Original: [Equation]
Revised: [Equation]

Original: x
Revised: In the following

Original: [Equation]
Revised: [Equation]

Revised: we bound the terms separately.

Original: Bounding term I.
Revised: Bounding term I.

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: where we use Assumption
Revised: where we use Assumption

Original: [Equation]
Revised: [Equation]

Original: Lemma 4 and the fact that l 2 norm is no larger than l 1 norm.
Revised: Lemma 4 and the fact that l 2 norm is no larger than l 1 norm.

Original: Bounding term II.
Revised: Bounding term II.

Original: It holds that 
Revised: By the deﬁnition of E t [Equation] we know that (cid:107)E t (cid:107) ≤ (1 − β 1 ) (cid:80) tτ =

Revised: 1 β t − τ(cid:107)

Revised: ¯ e t (cid:107) ≤ 2 q 1 − q 2 G .

Revised: Then we have

Original: [Equation]
Revised: [Equation]

Original: where C 1 := β 1 1 − β 1 + 2 q 1 − q 2 .
Revised: where C 1 := β 1 1 − β 1 + 2 q 1 − q 2 .

Original: The second inequality is because of smoothness of f ( θ ) , and the last inequality is due to Lemma 2, Assumption 3 and the property of norms.
Revised: The second inequality is because of smoothness of f ( θ ) , and the last inequality is due to Lemma 2, Assumption 3 and the property of norms.

Original: Bounding term III.
Revised: Bounding term III.

Original: This term can be bounded as follows:
Revised: This term can be bounded as follows:

Original: [Equation]
Revised: [Equation]

Original: where (a) follows from ∇ f ( θ t )
Revised: where (a) follows from ∇ f ( θ t )

Original: = 1 n (cid:80) nj =1 ∇ f j (
Revised: = 1 n (cid:80) ni =1 ∇ f i

Original: [Equation]
Revised: [Equation]

Original: ) and Assumption 4 that g t,j is unbiased of ∇ f j ( θ t ) and has bounded variance σ 2 .
Revised: ) and Assumption 4 that g t,i is unbiased of ∇ f i ( θ t ) and has bounded variance σ 2 .

Original: Bounding term IV.
Revised: Bounding term IV.

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: where (a) is due to Young’s inequality and (b) is based on Assumption
Revised: where (a) is due to Young’s inequality and (b) is based on Assumption

Original: [Equation]
Revised: [Equation]

Original: Regarding the second term in (7)
Revised: Regarding the second term in (7)

Original: [Equation]
Revised: [Equation]

Original: by Lemma 3 and Lemma [Equation] summing over t = 1 , ...
Revised: by Lemma 3 and Lemma 1, summing over t = 1 , ..., T we have

Original: [Equation]

Original: T we have

Original: T
Revised: T

Original: [Equation]
Revised: [Equation]

Original: m (cid:48)
Revised: m (cid:48)

Original: [Equation]
Revised: [Equation]

Original: ] ≤ T (cid:88)
Revised: −

Original: [Equation]
Revised: [Equation]

Original: =
Revised: E t (cid:107) 2 ] ≤

Original: [Equation]
Revised: T (cid:88) t =

Original: ≤ Tη
Revised: η 3 L

Original: [Equation]
Revised: [Equation]

Original: Lσnρ (1 −
Revised: (cid:104)

Original: [Equation][Equation]
Revised: [Equation]

Original: +

Original: [Equation]
Revised: [Equation]

Original: (cid:88)
Revised: (cid:48)

Original: [Equation]
Revised: [Equation]

Original: (cid:107)∇
Revised: [ (cid:107)E

Original: [Equation]
Revised: [Equation]

Original: ( θ
Revised: (cid:107)

Original: [Equation]
Revised: [Equation]

Original: ) (cid:107) 2 ] = Tη 3 LC 2 σnρ(cid:15) + 
Revised: ] (cid:105) ≤ Tη 3 β 21 Lσnρ (1 − β 1 ) 2 (cid:15) + η 3 β 21 L ρ (

Original: [Equation]
Revised: [Equation]

Original: g ρ ( [Equation] [Equation] (cid:15)
Revised: β 1 ) 2 (cid:15)

Original:  +  η 
Revised: T (cid:88) t = E [ (cid:107)∇ f ( θ t ) (cid:107) 2 ] + 4 Tη 3 q 2 L ρ (1 − q 2 ) 2 (cid:15) ( σ 2 + σ 2 g ) + 4 η 3 q 2 L ρ (1 − q 2 ) 2 (cid:15) T (cid:88) t =

Original: [Equation]
Revised: E [ (cid:107)∇ f ( θ t ) (cid:107) 2 ] =

Original:  LC [Equation] ρ(cid:15)
Revised: Tη 3 LC 2 σnρ(cid:15)

Original: [Equation]
Revised: + 4 Tη 3 q Lσ 2 g ρ (1 − q 2 ) 2 (cid:15) + η 3 LCρ(cid:15)

Original:  (cid:88) t =
Revised: T (cid:88) t =

Original: E [ (cid:107)∇ f ( θ

Original: [Equation] ) (cid:107) [Equation] ] , (8) with C 2 := β 21 (1 − β 1 ) 2 + 4 q 2 (1 − q 2 ) 2 .
Revised: E [ (cid:107)∇ f ( θ t ) (cid:107) 2 ] , (8) with C 2 := β 21 (1 − β 1 ) 2 + 4 q 2 (1 − q 2 ) 2 .

Original: Now integrating (4), (5), (6), (7) and (8) into (3), taking the telescoping summation over t = 1 , ..., T , we obtain
Revised: Now integrating (4), (5), (6), (7) and (8) into (3), taking the telescoping summation over t = 1 , ..., T , we obtain

Original: E [ f ( x T +1 ) − f ( x 1 )] ≤ ( − η C 0 + η 2 L (cid:15) + ηρ 2 (cid:15) + η 3 LC 2 ρ(cid:15) )
Revised: E [ f ( x T +1 ) − f ( x 1 )] ≤ ( − η C 0 + η 2 L (cid:15) + ηρ 2 (cid:15) + η 3 LC 2 ρ(cid:15) )

Original: T (cid:88) t =
Revised: T (cid:88) t =

Original: E [ (cid:107)∇
Revised: E [ (cid:107)∇

Original: [Equation]
Revised: [Equation]

Original: with
Revised: with

Original: [Equation]
Revised: [Equation]

Original: and choosing
Revised: and choosing

Original: [Equation]
Revised: [Equation]

Original: , we obtain
Revised: , we further arrive at

Original: [Equation]
Revised: [Equation]

Original: where thelast inequality follows from Lemma 5.
Revised: where the inequality follows from Lemma 5.

Original: Re-arranging terms
Revised: Re-arranging terms

Original: [Equation]
Revised: [Equation]

Original: we get that
Revised: we get that

Original: T

Original: T

Original: [Equation]
Revised: [Equation]

Original: , where C 0 = (cid:113) 4(1+ q 2 )3 (1 − q 2 {+(cid:15) 
 + (1 + C 1+} )2 G 2 + (cid:15) , C 1 = β 1 1 − β 1 + 2 q 1 − q 2 .{+d T(cid:15) (cid:17) , where C 
+}
Revised: 2 ) 2 {+(cid:15) 
 + (1 + C 1+} ) G 2 d T √ (cid:15) + η (1 + 2 C 1 ) C 1 LG 2 {+d T(cid:15) (cid:17) , where C 
+}

Original: {+= (cid:113) 4(1+ q 2 ) 
 (1 − q 2 ) 2 G 2 + (cid:15) , C 1 = β 1 1 − β 1 + 2 q 1 − q 2 .+} The last inequality is because θ (cid:48) 1 = θ 1 , θ ∗ := arg min θ f ( θ ) and the fact that C 2 ≤ C 21 .
Revised: {+= (cid:113) 4(1+ q 2 ) 
 (1 − q 2 ) 2 G 2 + (cid:15) , C 1 = β 1 1 − β 1 + 2 q 1 − q 2 .+} The last inequality is because x 1 = θ 1 , θ ∗ := arg min θ f ( θ ) and the fact that C 2 ≤ C 21 .

Original: This completes the proof.
Revised: This completes the proof.

Original: B. I NTERMEDIATE L EMMATA
Revised: B. I NTERMEDIATE L EMMATA

Revised: The lemmas used in the proof of Theorem 1 are given as below.

Original: Lemma 1.
Revised: Lemma 1.

Original: Under Assumption 1 to Assumption 4 we have:
Revised: Under Assumption 1 to Assumption 4 we have:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Revised: For the ﬁrst part, it is easy to see that by Assumption 3,

Revised: [Equation]

Original: Firstly, the expected squared norm of average stochastic gradient can be bounded by
Revised: For the second claim, the expected squared norm of average stochastic gradient can be bounded by

Original: [Equation]
Revised: [Equation]

Original: where we use Assumption 4 that g t,i is unbiased with bounded variance.
Revised: where we use Assumption 4 that g t,i is unbiased with bounded variance.

Original: Let ¯ g t,i denote the i -th coordinate of ¯ g t .
Revised: Let ¯ g t,j denote the j -th coordinate of ¯ g t .

Original: By the updating rule of C OMP -AMS
Revised: By the updating rule of C OMP -AMS, we have

Original: [Equation]
Revised: [Equation]

Revised: t

Revised: [Equation]

Revised: [Equation]

Original: where (a) is due to Cauchy-Schwartz inequality.
Revised: where (a) is due to Cauchy-Schwartz inequality.

Original: Summing over t = 1 , ..., T , we obtain
Revised: Summing over t = 1 , ..., T , we obtain

Original: T (cid:88)
Revised: T (cid:88)

Original: [Equation]
Revised: [Equation]

Original: This completes the proof.
Revised: This completes the proof.

Original: Lemma 2.
Revised: Lemma 2.

Original: Under Assumption 4, we have for ∀ t and each local worker ∀ i ∈ [ n ] ,
Revised: Under Assumption 4, we have for ∀ t and each local worker ∀ i ∈ [ n ] ,

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: We start by using Assumption 1 and Young’s inequality to get
Revised: We start by using Assumption 1 and Young’s inequality to get

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original:  by choosing ρ = 1 − q 2 2 q 2  .
Revised: where (9) is derived by choosing ρ = 1 − q 2 2 q 2 and the fact that q < 1 .

Original: Now by recursion and the initialization e 1 ,i = 0 , we have
Revised: Now by recursion and the initialization e 1 ,i = 0 , we have

Original: [Equation]
Revised: [Equation]

Original: which proves the second argument.
Revised: which proves the second argument.

Original: Meanwhile, the absolute bound (cid:107) e t,i (cid:107) 2 ≤
Revised: Meanwhile, the absolute bound (cid:107) e t,i (cid:107) 2 ≤

Original: [Equation]
Revised: [Equation]

Original: G 2 follows directly from (9).
Revised: 2 follows directly from (9).

Original: Lemma 3.
Revised: Lemma 3.

Original: For the moving average error sequence E t , it holds that
Revised: For the moving average error sequence E t , it holds that

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Let ¯ e t,i be the j -th coordinate of ¯ e t .

Original: Denote K t,i :=(cid:80) tτ =1 ( 1+ q 2 2 ) t − τ E [ (cid:107)∇ f i ( θ τ ) (cid:107) 2 ] and K t,i = 0 , ∀ i ∈ [ n ] .
Revised: Denote K t,i :=

Original: Using the same technique as in the proof of Lemma 1, we have

Original: E [

Original: [Equation]
Revised: [Equation]

Original: τ
Revised: tτ =

Original: [Equation]
Revised: [Equation]

Original: ¯ e

Original: [Equation]

Original: [Equation]

Original: j =

Original: E [( t (cid:88) τ

Original: = β t − τ¯ e τ,j ) 2 ] ( a )

Original: [Equation]

Original: [Equation]

Original: 1 ) 2 d (cid:88) j =

Original: E [( t (cid:88) τ

Original: = β t − τ)( t (cid:88) τ

Original: = β t − τ¯ e 2 τ,j )] ≤ (

Original: [Equation]
Revised: [Equation]

Original: 1 ) t (cid:88) τ = β t − τ E [ (cid:107) ¯ e τ (cid:107) 2 ] [Equation]
Revised: q 2 2 )

Original: [Equation]
Revised: [Equation]

Original: 1 ) t (cid:88) τ

Original: = β t − τ

Original: E [ 1 n n (cid:88) i = (cid:107) e τ,i (cid:107) 2 ] ( [Equation]
Revised: E [ (cid:107)∇ f i ( θ τ ) (cid:107) 2 ] .

Original: [Equation] t (cid:88) τ = [Equation] t − τ(n n (cid:88) i =
Revised: Using the same technique as in the proof of Lemma 1, denoting ¯ e t,j as the j -th coordinate of ¯ e t , it follows that

Original: K

Original: [Equation]
Revised: [Equation]

Original: i ) , where (a) is due to Cauchy-Schwartz and (b) is a result of Lemma 2.
Revised: where (a) is due to Cauchy-Schwartz and (b) is a result of Lemma 2.

Original: Summing over t = 1 , ..., T and using the technique of geometric series summation leads to
Revised: Summing over t = 1 , ..., T and using the technique of geometric series summation leads to

Original: T (cid:88)
Revised: T (cid:88)

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where (a) is derived by the variance decomposition and the last inequality holds due to Assumption 4.
Revised: where (a) is derived by the variance decomposition and the last inequality holds due to Assumption 4.

Original: The desired result is obtained.
Revised: The desired result is obtained.

Original: Lemma 4.
Revised: Lemma 4.

Original: It holds that ∀ t ∈ [ T ] , ∀i ∈
Revised: It holds that ∀ t ∈ [ T ] , ∀

Original: [ d ] , ˆ v t,i ≤ 4(1+ q 2 ) 3 (1 − q 2 ) 2 G 2 .
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: For any t , by Lemma 2 and Assumption 3 we have
Revised: For any t , by Lemma 2 and Assumption 3 we have

Original: [Equation]
Revised: [Equation]

Original: It’s then easy to show by the updating rule of ˆ v t , 
Revised: It’s then easy to show by the updating rule of ˆ v t , there exists a j ∈ [ t ] such that ˆ v t = v j . Then

Original: [Equation]
Revised: [Equation]

Revised: which concludes the claim.

Original: Lemma 5.
Revised: Lemma 5.

Original: Let D t :=
Revised: Let D t :=

Original: [Equation]

Original: √

Original: [Equation]

Original: −

Original: [Equation]

Original: +

Original: [Equation]
Revised: [Equation]

Original: be deﬁned as above.
Revised: be deﬁned as above.

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: By the updating rule of C OMP -AMS, ˆ v t − 1 ≤ ˆ v t for ∀ t .
Revised: By the updating rule of C OMP -AMS, ˆ v t − 1 ≤ ˆ v t for ∀ t .

Original: Therefore, by the initialization ˆ v 0 = 0 , we have
Revised: Therefore, by the initialization ˆ v 0 = 0 , we have

Original: T (cid:88)

Original: [Equation]
Revised: [Equation]

Original: For the sum of squared l 2 norm, note the fact that for a ≥ b > 0 , it holds that 
Revised: For the sum of squared l 2 norm, note the fact that for a ≥ b > 0 , it holds that [Equation] a

Original: [Equation]
Revised: [Equation]

Revised: b )

Revised: [Equation]

Revised: ≤

Revised: [Equation]

Revised: a

Revised: [Equation]

Revised: b )

Revised: [Equation]

Revised: a + b ) = a 2 − b 2 .

Original: Thus,
Revised: Thus,

Original: [Equation]
Revised: [Equation]

Original: which gives the desired result.
Revised: which gives the desired result.