Original: Lower Bounds and Nearly Optimal Algorithms in Distributed Learning with Communication Compression
Revised: Lower Bounds and Nearly Optimal Algorithms in Distributed Learning with Communication Compression

Original: Abstract
Revised: Abstract

Original: Recent advances in distributed optimization and learning have shown that communication compression is one of the most effective means of reducing communication.
Revised: Recent advances in distributed optimization and learning have shown that communication compression is one of the most effective means of reducing communication.

Original: While there have been many results for convergence rates with compressed communication, a lower bound is still missing.
Revised: While there have been many results for convergence rates with compressed communication, a lower bound is still missing.

Original: Analyses of algorithms with communication compression have identified two abstract properties that guarantee convergence: the unbiased property or the contractive property.
Revised: Analyses of algorithms with communication compression have identified two abstract properties that guarantee convergence: the unbiased property or the contractive property.

Original: They can be applied either unidirectionally (compressing messages from worker to server) or bidirectionally.
Revised: They can be applied either unidirectionally (compressing messages from worker to server) or bidirectionally.

Original: In the smooth and non-convex stochastic regime, this paper establishes a lower bound for distributed algorithms whether using unbiased or contractive compressors in unidirection or bidirection.
Revised: In the smooth and non-convex stochastic regime, this paper establishes a lower bound for distributed algorithms whether using unbiased or contractive compressors in unidirection or bidirection.

Original: To close the gap between this lower bound and the best existing upper bound, we further propose an algorithm, NEOLITHIC, that almost reaches our lower bound (except for a logarithm factor) under mild conditions.
Revised: To close the gap between this lower bound and the best existing upper bound, we further propose an algorithm, NEOLITHIC, that almost reaches our lower bound (except for a logarithm factor) under mild conditions.

Original: Our results also show that using contractive compressors in bidirection can yield iterative methods that converge as fast as those using unbiased compressors unidirectionally.
Revised: Our results also show that using contractive compressors in bidirection can yield iterative methods that converge as fast as those using unbiased compressors unidirectionally.

Original: We report experimental results that validate our findings.
Revised: We report experimental results that validate our findings.

Original:  Introduction
Revised:  Introduction

Original: Large-scale optimization is a critical step in many machine learning applications.
Revised: Large-scale optimization is a critical step in many machine learning applications.

Original: Millions or even billions of data samples contribute to the excellent performance in tasks such as robotics, computer vision, natural language processing, healthcare, and so on.
Revised: Millions or even billions of data samples contribute to the excellent performance in tasks such as robotics, computer vision, natural language processing, healthcare, and so on.

Original: However, such a scale of data samples and model parameters leads to enormous communication that hampers the scalability of distributed machine-learning training systems.
Revised: However, such a scale of data samples and model parameters leads to enormous communication that hampers the scalability of distributed machine-learning training systems.

Original: We urgently need communication-reduction strategies.
Revised: We urgently need communication-reduction strategies.

Original: State-ofthe-art strategies include model and gradient compression [4, 12, 54], decentralized communication [37, 9, 19], lazy communication [71, 55, 31, 18] and beyond.
Revised: State-ofthe-art strategies include model and gradient compression [4, 12, 55], decentralized communication [38, 9, 19], lazy communication [72, 56, 32, 18] and beyond.

Original: This article will focus on the former.
Revised: This article will focus on the former.

Original: The most common method of distributed training is Parallel SGD (P-SGD) [15].
Revised: The most common method of distributed training is Parallel SGD (P-SGD) [15].

Original: In P-SGD, the stochastic gradients that workers transmit to a server cause significant communication overhead in large-scale machine learning.
Revised: In P-SGD, the stochastic gradients that workers transmit to a server cause significant communication overhead in large-scale machine learning.

Original: To reduce this overhead, many recent works propose to compress the messages sent unidirectionally from worker to server [5, 27, 4] or compress the messages between them bidirectionally [60, 68].
Revised: To reduce this overhead, many recent works propose to compress the messages sent unidirectionally from worker to server [5, 27, 4] or compress the messages between them bidirectionally [61, 69].

Original: The method of compression is either sparsification or quantization [5, 27, 4] or their combination [27, 14].
Revised: The method of compression is either sparsification or quantization [5, 27, 4] or their combination [27, 14].

Original: The literature [56, 53, 60, 68] reveals that bidirectional compression can save more communication, but leads to slower convergence rates.
Revised: The literature [57, 54, 61, 69] reveals that bidirectional compression can save more communication, but leads to slower convergence rates.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]

Original: [Table]
Revised: [Table]

Original: Although there are many specific compression methods
Revised: Although there are many specific compression methods

Original: [Table]
Revised: [Table]

Original: their convergence analyses are mainly built on unbiased compressibility or contractive compressibility.
Revised: their convergence analyses are mainly built on unbiased compressibility or contractive compressibility.

Original: The literature [
Revised: The literature [

Original: [Table]
Revised: [Table]

Original: 52, 69] summarizes these two properties and how they appear in the analyses.
Revised: 53, 70] summarizes these two properties and how they appear in the analyses.

Original: An unbiased compressor compresses a long vector x into a short vector C ( x ) and satisfies E
Revised: An unbiased compressor compresses a long vector x into a short vector C ( x ) and satisfies E

Original: [ C ( x )]
Revised: [ C ( x )]

Original: = x , i
Revised: = x , i

Original: [Table]
Revised: [Table]

Original: e., no bias is introduced.
Revised: e., no bias is introduced.

Original: The contractive compressor may introduce bias
Revised: The contractive compressor may introduce bias

Original: [Table]
Revised: [Table]

Original: but its compression introduces much less variance.
Revised: but its compression introduces much less variance.

Original: We give their definitions below.
Revised: We give their definitions below.

Original: Although the contractive compressor can empirically work better, the analysis of the unbiased compressor yields faster convergence due to unbiasedness [14, 42, 28, 25].
Revised: Although the contractive compressor can empirically work better, the analysis of the unbiased compressor yields faster convergence due to unbiasedness [14, 43, 28, 25].

Original: Despite the quick progress made in compression techniques and their convergence, we do not yet understand the limits of algorithms with communication compression.
Revised: Despite the quick progress made in compression techniques and their convergence, we do not yet understand the limits of algorithms with communication compression.

Original: Since unbiased and contractive compressibilities are the two representative characteristics of various compressors, we use them to theorize two types of compression methods.
Revised: Since unbiased and contractive compressibilities are the two representative characteristics of various compressors, we use them to theorize two types of compression methods.

Original: For each type, we intend to answer:
Revised: For each type, we intend to answer:

Original: What is the optimal convergence rate that a distributed algorithm can achieve when using any of the compression methods of this type?
Revised: What is the optimal convergence rate that a distributed algorithm can achieve when using any of the compression methods of this type?

Original: Here, we assume that only unbiased compressibility or contractive compressibility can be used, not considering any additional special compressor design; after all, any special design in the literature has been heuristic, whose effectiveness can be explained at best and not proved or quantified.
Revised: Here, we assume that only unbiased compressibility or contractive compressibility can be used, not considering any additional special compressor design; after all, any special design in the literature has been heuristic, whose effectiveness can be explained at best and not proved or quantified.

Original: So, we further clarify our question: Given a class of optimization problems (specified below) and a class of compression methods of the same type, if we choose the worst combination of them to defeat an algorithm, what will the convergence rate that the best-defending algorithm can reach?
Revised: So, we further clarify our question: Given a class of optimization problems (specified below) and a class of compression methods of the same type, if we choose the worst combination of them to defeat an algorithm, what will the convergence rate that the best-defending algorithm can reach?

Original: To our knowledge, they are fundamental questions not addressed yet.
Revised: To our knowledge, they are fundamental questions not addressed yet.

Original: 1.1 Main Results
Revised: 1.1 Main Results

Original: This paper clarifies these open questions by providing lower bounds under the non-convex smooth stochastic optimization setting, and developing effective algorithms that match the lower bounds up to logarithm factors.
Revised: This paper clarifies these open questions by providing lower bounds under the non-convex smooth stochastic optimization setting, and developing effective algorithms that match the lower bounds up to logarithm factors.

Original: In particular, our contributions are:
Revised: In particular, our contributions are:

Original: • We establish convergence lower bounds for distributed algorithms with communication compression in the stochastic non-convex regime.
Revised: • We establish convergence lower bounds for distributed algorithms with communication compression in the stochastic non-convex regime.

Original: Our lower bounds apply to any algorithm conducting unidirectional or bidirectional compression and using unbiased or contractive compressors.
Revised: Our lower bounds apply to any algorithm conducting unidirectional or bidirectional compression and using unbiased or contractive compressors.

Original: We find a clear gap between the established lower bounds and the existing convergence rates.
Revised: We find a clear gap between the established lower bounds and the existing convergence rates.

Original: • We propose a novel ne arly o ptimal a l gor ith m w i
Revised: • We propose a novel ne arly o ptimal a l gor ith m w i

Original: th c ompression (NEOLITHIC) to fill in this gap.
Revised: th c ompression (NEOLITHIC) to fill in this gap.

Original: NEOLITHIC can adopt either unidirectional or bidirectional compression, and is compatible with both unbiased and contractive compressors.
Revised: NEOLITHIC can adopt either unidirectional or bidirectional compression, and is compatible with both unbiased and contractive compressors.

Original: Using any combination, NEOLITHIC provably matches the above lower bound, under an additional mild assumption and up to logarithmic factors.
Revised: Using any combination, NEOLITHIC provably matches the above lower bound, under an additional mild assumption and up to logarithmic factors.

Original: • The convergence results of NEOLITHIC also imply that, for non-convex problems, algorithms using biased contractive compressors bidirectionally can theoretically converge as fast as those with unbiased compressors used unidirectionally.
Revised: • The convergence results of NEOLITHIC also imply that, for non-convex problems, algorithms using biased contractive compressors bidirectionally can theoretically converge as fast as those with unbiased compressors used unidirectionally.

Original: Before out work, it is only established in [46] that, for convex problems and unbiased compressors, algorithms with bidirectional compression can match with the counterparts with unidirectional compression.
Revised: Before out work, it is only established in [47] that, for convex problems and unbiased compressors, algorithms with bidirectional compression can match with the counterparts with unidirectional compression.

Original: • We provide extensive experimental results to validate our theories.
Revised: • We provide extensive experimental results to validate our theories.

Original: All established results in this paper as well as convergence rates of existing state-of-the-art distributed algorithms with communication compression are listed in Table 1.
Revised: All established results in this paper as well as convergence rates of existing state-of-the-art distributed algorithms with communication compression are listed in Table 1.

Original: The transient complexity, which measures how sensitive the algorithm is to the compression strategy (see Section 3), is also listed in the table.
Revised: The transient complexity, which measures how sensitive the algorithm is to the compression strategy (see Section 3), is also listed in the table.

Original: The smaller the transient complexity is, the faster the algorithm converges.
Revised: The smaller the transient complexity is, the faster the algorithm converges.

Original: 1.2 Related Works
Revised: 1.2 Related Works

Original: Distributed learning.
Revised: Distributed learning.

Original: Distributed learning has been increasingly useful in training large-scale machine learning models [20].
Revised: Distributed learning has been increasingly useful in training large-scale machine learning models [20].

Original: It typically follows a centralized or decentralized setup.
Revised: It typically follows a centralized or decentralized setup.

Original: Centralized approaches [2, 36], with P-SGD as the representative, require all workers to synchronize with a central server per iteration.
Revised: Centralized approaches [2, 37], with P-SGD as the representative, require all workers to synchronize with a central server per iteration.

Original: Decentralized approaches, however, are based on partial averaging in which each worker only needs to synchronize with its immediate neighbors.
Revised: Decentralized approaches, however, are based on partial averaging in which each worker only needs to synchronize with its immediate neighbors.

Original: Well-known decentralized algorithms include decentralized SGD [43, 75, 74, 37], D 2 [59, 72], stochastic gradient tracking [67, 34, 3], and their momentum variants [38, 73].
Revised: Well-known decentralized algorithms include decentralized SGD [44, 76, 75, 38], D 2 [60, 73], stochastic gradient tracking [68, 35, 3, 29], and their momentum variants [39, 74].

Original: The lazy communication [71, 55, 18] is also utilized to reduce communication overheads in which workers conducts multiple local updates before sending messages.
Revised: The lazy communication [72, 56, 18] is also utilized to reduce communication overheads in which workers conducts multiple local updates before sending messages.

Original: Communication compression.
Revised: Communication compression.

Original: To alleviate the communication overhead in distributed learning, researchers have proposed two mainstream communication compression methods: quantization and sparsification.
Revised: To alleviate the communication overhead in distributed learning, researchers have proposed two mainstream communication compression methods: quantization and sparsification.

Original: Quantization [30, 58, 78] is essentially an unbiased operator with random noise.
Revised: Quantization [31, 59, 79] is essentially an unbiased operator with random noise.

Original: For example, [54] develops Sign-SGD by using only 1 bit for each entry whose convergence is studied in [12, 13, 64].
Revised: For example, [55] develops Sign-SGD by using only 1 bit for each entry whose convergence is studied in [12, 13, 65].

Original: Q-SGD [4] compresses each entry with more flexible bits and enables a trade-off between convergence rates and communication costs.
Revised: Q-SGD [4] compresses each entry with more flexible bits and enables a trade-off between convergence rates and communication costs.

Original: Sparsification, on the other hand, amounts to a biased but contractive operator.
Revised: Sparsification, on the other hand, amounts to a biased but contractive operator.

Original: [63, 56] propose to transmit a small number of entries, randomly or by taking the largest ones, to achieve sparsity in communication.
Revised: [64, 57] propose to transmit a small number of entries, randomly or by taking the largest ones, to achieve sparsity in communication.

Original: The theories behind contractive compressors are limited to those in [62, 39, 57] due to analysis challenges, and they are established with assumptions such as bounded gradients [80, 32] or quadratic loss functions [65].
Revised: The theories behind contractive compressors are limited to those in [63, 40, 58] due to analysis challenges, and they are established with assumptions such as bounded gradients [81, 33] or quadratic loss functions [66].

Original: More discussions on unbiased and biased compressors can be found in [14, 52, 51].
Revised: More discussions on unbiased and biased compressors can be found in [14, 53, 52].

Original: Communication compression can also be combined with other communication-saving techniques such as decentralization [40, 79].
Revised: Communication compression can also be combined with other communication-saving techniques such as decentralization [41, 80].

Original: Error compensation.
Revised: Error compensation.

Original: The error compensation (feedback) mechanism is introduced by [54] to mitigate the error caused by 1-bit quantization.
Revised: The error compensation (feedback) mechanism is introduced by [55] to mitigate the error caused by 1-bit quantization.

Original: [65] studies SGD with error-compensated quantization for quadratic functions with convergence guarantees.
Revised: [66] studies SGD with error-compensated quantization for quadratic functions with convergence guarantees.

Original: [56] shows that error compensation can reduce quantization-incurred errors for strongly convex loss functions in the single-node setting.
Revised: [57] shows that error compensation can reduce quantization-incurred errors for strongly convex loss functions in the single-node setting.

Original: Errorcompensated SGD is studied in [5] for non-convex loss functions with no establishment of an improved convergence rate.
Revised: Errorcompensated SGD is studied in [5] for non-convex loss functions with no establishment of an improved convergence rate.

Original: The algorithm EF21 [50] applies to the deterministic setting, using contractive compressors unidirectionally.
Revised: The algorithm EF21 [51] applies to the deterministic setting, using contractive compressors unidirectionally.

Original: It can converge under very mild assumptions.
Revised: It can converge under very mild assumptions.

Original: It has been recently extended to the stochastic setting [23] yet no linear-speedup rate is achieved.
Revised: It has been recently extended to the stochastic setting [23] yet no linear-speedup rate is achieved.

Original: Lower bounds in optimization.
Revised: Lower bounds in optimization.

Original: Lower bounds are well studied in convex optimization [22, 1, 21] especially when there is no gradient noise [8, 76, 10, 6, 24].
Revised: Lower bounds are well studied in convex optimization [22, 1, 21] especially when there is no gradient noise [8, 77, 10, 6, 24].

Original: In non-convex optimization, [16, 17] propose a zero-chain model and show a tight bound for first-order methods.
Revised: In non-convex optimization, [16, 17] propose a zero-chain model and show a tight bound for first-order methods.

Original: By splitting the zero-chain model into multiple components, the authors of [81, 7] extend the analysis technique to finite sum and stochastic problems.
Revised: By splitting the zero-chain model into multiple components, the authors of [82, 7] extend the analysis technique to finite sum and stochastic problems.

Original: Recently, [41, 74] also show the lower bounds in the decentralized stochastic setting based on a similar approach.
Revised: Recently, [42, 75] also show the lower bounds in the decentralized stochastic setting based on a similar approach.

Original: For distributed learning with communication compression, a useful work [47] establishes a lower bound for unidirectional/bidirectional compression for strongly convex problems with fixed learning rates and unbiased compressors.
Revised: For distributed learning with communication compression, a useful work [48] establishes a lower bound for unidirectional/bidirectional compression for strongly convex problems with fixed learning rates and unbiased compressors.

Original: There are limited studies on lower bounds for non-convex distributed learning with unbiased/contractive compressions.
Revised: There are limited studies on lower bounds for non-convex distributed learning with unbiased/contractive compressions.

Original:  Problem Setup
Revised:  Problem Setup

Original: We consider standard distributed learning with n parallel workers.
Revised: We consider standard distributed learning with n parallel workers.

Original: The data samples in each worker i follow a local distribution D i , which can be heterogeneous among all workers.
Revised: The data samples in each worker i follow a local distribution D i , which can be heterogeneous among all workers.

Original: These workers, together with a central parameter server, collaborate to train a model by solving
Revised: These workers, together with a central parameter server, collaborate to train a model by solving

Original: [Equation]
Revised: [Equation]

Original: where F ( x ; ξ ) is the loss function evaluated at parameter x with sample ξ .
Revised: where F ( x ; ξ ) is the loss function evaluated at parameter x with sample ξ .

Original: Since the objective f can be non-convex, finding a global minimum of (1) is generally intractable.
Revised: Since the objective f can be non-convex, finding a global minimum of (1) is generally intractable.

Original: Therefore, we turn to seeking a model ˆ x with a small gradient magnitude in expectation, i.e. , E [ ∥∇ f
Revised: Therefore, we turn to seeking a model ˆ x with a small gradient magnitude in expectation, i.e. , E [ ∥∇ f

Original: (ˆ x ) ∥ 2 ] .
Revised: (ˆ x ) ∥ 2 ] .

Original: Next we introduce the setup under which we perform a convergence analysis.
Revised: Next we introduce the setup under which we perform a convergence analysis.

Original: 2.1 Function Class
Revised: 2.1 Function Class

Original: We let the function class F ∆ ,L denote the set of all functions satisfying Assumption 1 for any underlying dimension d ∈ N + and a given initialization point x (0) ∈ R d .
Revised: We let the function class F ∆ ,L denote the set of all functions satisfying Assumption 1 for any underlying dimension d ∈ N + and a given initialization point x (0) ∈ R d .

Original: Assumption 1 ( Smoothness).
Revised: Assumption 1 ( Smoothness).

Original: We assume each local objective f
Revised: We assume each local objective f

Original: i has L -Lipschitz gradient, i.e. ,
Revised: i has L -Lipschitz gradient, i.e. ,

Original: [Equation]
Revised: [Equation]

Original: ∥ for any i ∈ { 1 , . .
Revised: ∥ for any i ∈ { 1 , . .

Original: . , n } , x, y ∈ R d , and f ( x (0) )
Revised: . , n } , x, y ∈ R d , and f ( x (0) )

Original: − inf x ∈ R d f ( x ) ≤ ∆ with f = 1 n (cid:80) ni
Revised: − inf x ∈ R d f ( x ) ≤ ∆ with f = 1 n (cid:80) ni

Original: =1 f i .
Revised: =1 f i .

Original: 2.2 Gradient Oracle Class
Revised: 2.2 Gradient Oracle Class

Original: We assume each worker i has access to its local gradient ∇ f i ( x ) via a stochastic gradient oracle
Revised: We assume each worker i has access to its local gradient ∇ f i ( x ) via a stochastic gradient oracle

Original: O i ( x ; ζ i ) subject to randomness ζ i , e.g. , the mini-batch sampling ζ i ≜ ξ i ∼ D i .
Revised: O i ( x ; ζ i ) subject to randomness ζ i , e.g. , the mini-batch sampling ζ i ≜ ξ i ∼ D i .

Original: We further assume that the output O i ( x, ζ i ) is a time-independent and unbiased estimator of the full-batch gradient ∇ f i ( x ) with a bounded variance.
Revised: We further assume that the output O i ( x, ζ i ) is a time-independent and unbiased estimator of the full-batch gradient ∇ f i ( x ) with a bounded variance.

Original: Formally, we let the stochastic gradient oracle class O σ 2 denote the set of all oracles
Revised: Formally, we let the stochastic gradient oracle class O σ 2 denote the set of all oracles

Original: O i satisfying Assumption 2.
Revised: O i satisfying Assumption 2.

Original: Assumption 2 ( Gradient noise).
Revised: Assumption 2 ( Gradient noise).

Original: For any x ∈ R d and i ∈ { 1 , . . . , n } , the oracle O i satisfies
Revised: For any x ∈ R d and i ∈ { 1 , . . . , n } , the oracle O i satisfies

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: 2.3 Compressor Class
Revised: 2.3 Compressor Class

Original: The two classes of widely-used compressors are i) the ω - unbiased compressor, described by Assumption 3, e.g. , the stochastic quantization operator [4], and ii) the δ - contractive compressor, described by Assumption 4, e.g. , the rand- K and top- K operators [56, 49].
Revised: The two classes of widely-used compressors are i) the ω - unbiased compressor, described by Assumption 3, e.g. , the stochastic quantization operator [4], and ii) the δ - contractive compressor, described by Assumption 4, e.g. , the rand- K and top- K operators [57, 50].

Original: Assumption 3 ( Unbiased compressor).
Revised: Assumption 3 ( Unbiased compressor).

Original: For a (possibly random) compression operator C : R d → R d , we assume there exists a constant ω ≥ 0 such that
Revised: For a (possibly random) compression operator C : R d → R d , we assume there exists a constant ω ≥ 0 such that

Original: [Equation]
Revised: [Equation]

Original: where the expectation is taken over the randomness of the compression operator C .
Revised: where the expectation is taken over the randomness of the compression operator C .

Original: Assumption 4 ( Contractive compressor).
Revised: Assumption 4 ( Contractive compressor).

Original: For a (possibly random) compression operator C : R d → R d , we assume there exists a constant δ ∈ (0 , 1] such that
Revised: For a (possibly random) compression operator C : R d → R d , we assume there exists a constant δ ∈ (0 , 1] such that

Original: [Equation]
Revised: [Equation]

Original: where the expectation is over the randomness of the compression operator C .
Revised: where the expectation is over the randomness of the compression operator C .

Original: Formally, we let the compressor classes U ω and C δ denote the set of all ω -unbiased compressors and δ -contractive compressors satisfying Assumptions 3 and 4, respectively.
Revised: Formally, we let the compressor classes U ω and C δ denote the set of all ω -unbiased compressors and δ -contractive compressors satisfying Assumptions 3 and 4, respectively.

Original: Note that the identity operator I satisfies I ∈
Revised: Note that the identity operator I satisfies I ∈

Original: U ω for any ω ≥ 

Revised: U ω for any ω ≥

Original: and I ∈ C δ for any δ ∈ (0 , 1] .
Revised: and I ∈ C δ for any δ ∈ (0 , 1] .

Original: Generally, an ω -unbiased compressor is not necessarily contractive when ω is larger than 1 .
Revised: Generally, an ω -unbiased compressor is not necessarily contractive when ω is larger than 1 .

Original: However, since C ∈ U ω implies (1 + ω ) − 1 C ∈ C (1+ ω ) − 1 , the scaled unbiased compressor is also contractive though the converse does not hold.
Revised: However, since C ∈ U ω implies (1 + ω ) − 1 C ∈ C (1+ ω ) − 1 , the scaled unbiased compressor is also contractive though the converse does not hold.

Original: Hence, the class of contractive compressors is strictly more general since it contains all unbiased compressors through scaling.
Revised: Hence, the class of contractive compressors is strictly more general since it contains all unbiased compressors through scaling.

Original: 2.4 Algorithm Class
Revised: 2.4 Algorithm Class

Original: We consider a centralized and synchronous algorithm A in which i) workers are allowed to communicate only directly with the central server but not between one another; ii) all iterations are synchronized so that all workers start each of their iterations simultaneously.
Revised: We consider a centralized and synchronous algorithm A in which i) workers are allowed to communicate only directly with the central server but not between one another; ii) all iterations are synchronized so that all workers start each of their iterations simultaneously.

Original: Each worker i holds a local copy of the model, denoted by x ( t ) i , at iteration t .
Revised: Each worker i holds a local copy of the model, denoted by x ( t ) i , at iteration t .

Original: The output ˆ x ( t ) of A after t iterations can be any linear combination of all previous local models, namely,
Revised: The output ˆ x ( t ) of A after t iterations can be any linear combination of all previous local models, namely,

Original: [Equation]
Revised: [Equation]

Original: We further require algorithms A to satisfy the so-called “zero-respecting” property, which appears in [16, 17, 41].
Revised: We further require algorithms A to satisfy the so-called “zero-respecting” property, which appears in [16, 17, 42].

Original: Intuitively, this property implies that the number of non-zero entries of the local model of a worker can be increased only by conducting local stochastic gradient descent with its own samples or synchronizing with the server.
Revised: Intuitively, this property implies that the number of non-zero entries of the local model of a worker can be increased only by conducting local stochastic gradient descent with its own samples or synchronizing with the server.

Original: The zero-respecting property holds in all algorithms in
Revised: The zero-respecting property holds in all algorithms in

Original: Table 1 and most first-order methods based on SGD [44, 33, 29, 77].
Revised: Table 1 and most first-order methods based on SGD [45, 34, 30, 78].

Original: In addition to these properties, the algorithm A has to admit communication compression.
Revised: In addition to these properties, the algorithm A has to admit communication compression.

Original: Specifically, we endow the server with a compressor C 0 and each worker i ∈ { 1 , · · · , n } with a compressor C i .
Revised: Specifically, we endow the server with a compressor C 0 and each worker i ∈ { 1 , · · · , n } with a compressor C i .

Original: If C i
Revised: If C i

Original: =
Revised: =

Original: I for some i ∈ { 0 , · · · , n } , then worker i (or the server if i = 0 ) conducts lossless communication.
Revised: I for some i ∈ { 0 , · · · , n } , then worker i (or the server if i = 0 ) conducts lossless communication.

Original: When
Revised: When

Original: C i ̸
Revised: C i ̸

Original: = I for any i ∈ { 0 , · · · , n } , algorithm A conducts bidirectional compression.
Revised: = I for any i ∈ { 0 , · · · , n } , algorithm A conducts bidirectional compression.

Original: When C 

Revised: When C

Original: =
Revised: =

Original: I , algorithm A conducts unidirectional compression on messages from workers to server.
Revised: I , algorithm A conducts unidirectional compression on messages from workers to server.

Original: The formal definition of the algorithm class with bidirectional/unidirectional compression is as follows.
Revised: The formal definition of the algorithm class with bidirectional/unidirectional compression is as follows.

Original: Definition 1 ( Algorithm class).
Revised: Definition 1 ( Algorithm class).

Original: Given compressors { C 0 , C 1 , . . .
Revised: Given compressors { C 0 , C 1 , . . .

Original: , C n } , write A B { C i } ni =0 for the set of all centralized, synchronous, zero-respecting algorithms admitting bidirectional compression in which i) compressor C i , ∀ 1 ≤ i ≤ n , is applied to messages from worker i to the server, and ii) compressor
Revised: , C n } , write A B { C i } ni =0 for the set of all centralized, synchronous, zero-respecting algorithms admitting bidirectional compression in which i) compressor C i , ∀ 1 ≤ i ≤ n , is applied to messages from worker i to the server, and ii) compressor

Original: C 0 is applied to messages from the server to all workers.
Revised: C 0 is applied to messages from the server to all workers.

Original: When C 

Revised: When C

Original: =
Revised: =

Original: I , we write A B { C 0 =
Revised: I , we write A B { C 0 =

Original: I }∪{ C i } ni =1 , or A U { C i } ni =1 for short, for the set of algorithms admitting unidirectional compression.
Revised: I }∪{ C i } ni =1 , or A U { C i } ni =1 for short, for the set of algorithms admitting unidirectional compression.

Original: The superscript
Revised: The superscript

Original: B or U indicates “bidirectional” or “unidirectional”, respectively.
Revised: B or U indicates “bidirectional” or “unidirectional”, respectively.

Original:  Lower Bounds
Revised:  Lower Bounds

Original: With all interested classes introduced above, we now define the lower bound measure.
Revised: With all interested classes introduced above, we now define the lower bound measure.

Original: Given local loss functions { f i } n i =1 ⊆
Revised: Given local loss functions { f i } n i =1 ⊆

Original: F ∆ ,L , stochastic gradient oracles { O i } n i =1 ⊆
Revised: F ∆ ,L , stochastic gradient oracles { O i } n i =1 ⊆

Original: O σ 2 (with O i for worker i ), compressors {C i } ni
Revised: O σ 2 (with O i for worker i ), compressors {C i } ni

Original: =0 ⊆ C ( C can be U ω or C δ ), and
Revised: =0 ⊆ C ( C can be U ω or C δ ), and

Original: an algorithm A ∈ A to solve problem (1) ( A can be A B { C i } ni =0 or A U { C i } ni =1 ), we let ˆ x A, { f i } ni =1 , { O i } ni =1 , { C i } ni =0 ,T denote the output of algorithm
Revised: an algorithm A ∈ A to solve problem (1) ( A can be A B { C i } ni =0 or A U { C i } ni =1 ), we let ˆ x A, { f i } ni =1 , { O i } ni =1 , { C i } ni =0 ,T denote the output of algorithm

Original: A using no more than T gradient queries and rounds of communication by each worker node.
Revised: A using no more than T gradient queries and rounds of communication by each worker node.

Original: We define the minimax measure
Revised: We define the minimax measure

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: In (2), we do not require the compressors { C i } ni =

Revised: In (2), we do not require the compressors { C i } ni =

Original: to be distinct or independent.
Revised: to be distinct or independent.

Original: When C is U ω or C δ , we allow the compressor parameter ω or δ to be accessible by algorithm A .
Revised: When C is U ω or C δ , we allow the compressor parameter ω or δ to be accessible by algorithm A .

Original: 3.1 Unidirectional Unbiased Compression
Revised: 3.1 Unidirectional Unbiased Compression

Original: Our first result is for algorithms that admit unidirectional compression and ω -unbiased compressors.
Revised: Our first result is for algorithms that admit unidirectional compression and ω -unbiased compressors.

Original: Theorem 1 ( Unidirectional unbiased compression).
Revised: Theorem 1 ( Unidirectional unbiased compression).

Original: For every ∆ , L > 0 , n ≥ 2 , ω ≥ 0 , σ > 0 ,
Revised: For every ∆ , L > 0 , n ≥ 2 , ω ≥ 0 , σ > 0 ,

Original: T ≥ (1 + ω ) 2 , there exists a set of local loss functions { f i } ni =1 ⊆ F ∆ ,L , stochastic gradient oracles
Revised: T ≥ (1 + ω ) 2 , there exists a set of local loss functions { f i } ni =1 ⊆ F ∆ ,L , stochastic gradient oracles

Original: { O i } ni =1 ⊆
Revised: { O i } ni =1 ⊆

Original: O σ 2 , ω -unbiased compressors { C i } ni =0 ⊆
Revised: O σ 2 , ω -unbiased compressors { C i } ni =0 ⊆

Original: U ω with C 

Revised: U ω with C

Original: = I , such that for any algorithm
Revised: = I , such that for any algorithm

Original: A ∈
Revised: A ∈

Original: A U { C i } ni =1 starting from a given constant x (0) , it holds that
Revised: A U { C i } ni =1 starting from a given constant x (0) , it holds that

Original: [Equation]
Revised: [Equation]

Original: Consistency with previous works.
Revised: Consistency with previous works.

Original: The bound in (3) is consistent with best-known lower bounds in different settings.
Revised: The bound in (3) is consistent with best-known lower bounds in different settings.

Original: When ω = 0 , our result reduces to the tight bounds for distributed training without compression [7].
Revised: When ω = 0 , our result reduces to the tight bounds for distributed training without compression [7].

Original: When n = 1 and ω = 0 , our result reduces to the lower bound established in [7] under the single-node non-convex stochastic setting.
Revised: When n = 1 and ω = 0 , our result reduces to the lower bound established in [7] under the single-node non-convex stochastic setting.

Original: When n = 1 , ω
Revised: When n = 1 , ω

Original: = 0 and σ 2 = 

Revised: = 0 and σ 2 =

Original: , our result recovers the tight bound for deterministic non-convex optimization [16].
Revised: , our result recovers the tight bound for deterministic non-convex optimization [16].

Original: Linear-speedup.
Revised: Linear-speedup.

Original: When T is sufficiently large, the first term 1 / √ nT dominates the lower bound (3).
Revised: When T is sufficiently large, the first term 1 / √ nT dominates the lower bound (3).

Original: If an algorithm achieves an O (1 / √ nT ) rate, it will require T
Revised: If an algorithm achieves an O (1 / √ nT ) rate, it will require T

Original: = O
Revised: = O

Original: (1 / ( nϵ 2 )) gradient queries to reach a desired accuracy ϵ , which is inversely proportional to n .
Revised: (1 / ( nϵ 2 )) gradient queries to reach a desired accuracy ϵ , which is inversely proportional to n .

Original: Therefore, an algorithm achieves linear-speedup at T th iteration if, for this T , the term involving nT is dominating the rate.
Revised: Therefore, an algorithm achieves linear-speedup at T th iteration if, for this T , the term involving nT is dominating the rate.

Original: Transient complexity.
Revised: Transient complexity.

Original: Due to the compression-incurred overhead in convergence rate, a distributed stochastic algorithm with communication compression has to experience a transient stage to achieve its linear-speedup stage.
Revised: Due to the compression-incurred overhead in convergence rate, a distributed stochastic algorithm with communication compression has to experience a transient stage to achieve its linear-speedup stage.

Original: Transient complexity are referred to the number of gradient queries (or communication rounds) when T is relatively small so non-
Revised: Transient complexity are referred to the number of gradient queries (or communication rounds) when T is relatively small so non-

Original: nT terms still dominate the rate.
Revised: nT terms still dominate the rate.

Original: The smaller the transient complexity is, the less gradient queries or communication rounds the algorithm requires to achieve the linear-speedup stage.
Revised: The smaller the transient complexity is, the less gradient queries or communication rounds the algorithm requires to achieve the linear-speedup stage.

Original: For example, if an algorithm can achieve the lower bound established in (3), it requires (
Revised: For example, if an algorithm can achieve the lower bound established in (3), it requires (

Original: [Equation]
Revised: [Equation]

Original: ∆ L T , i.e., T = Ω( n (1+ ω ) 2 ) transient gradient queries (or communication rounds) to achieve linear-speedup, which is proportional to the compressionrelated terms (1 + ω ) 2 .
Revised: ∆ L T , i.e., T = Ω( n (1+ ω ) 2 ) transient gradient queries (or communication rounds) to achieve linear-speedup, which is proportional to the compressionrelated terms (1 + ω ) 2 .

Original: Transient complexity is an important metric to evaluate how sensitive the algorithm is to compression errors.
Revised: Transient complexity is an important metric to evaluate how sensitive the algorithm is to compression errors.

Original: It was widely used in decentralized learning [48, 70] to gauge how the network topology can influence the convergence rate.
Revised: It was widely used in decentralized learning [49, 71] to gauge how the network topology can influence the convergence rate.

Original: 3.2 Bidirectional Unbiased Compression
Revised: 3.2 Bidirectional Unbiased Compression

Original: Theorem 1 applies to unidirectional compression where C 

Revised: Theorem 1 applies to unidirectional compression where C

Original: = I .
Revised: = I .

Original: We next consider the bidirectional compression with C 0 ∈ U ω .
Revised: We next consider the bidirectional compression with C 0 ∈ U ω .

Original: Since { C i } ni =1 ⊆
Revised: Since { C i } ni =1 ⊆

Original: U ω with C 

Revised: U ω with C

Original: = I is a special case of { C i } ni =0 ⊆
Revised: = I is a special case of { C i } ni =0 ⊆

Original: U ω , the lower bound for algorithms that admit bidirectional compression is greater than or equal to that with unidirectional compression due to definition in (2).
Revised: U ω , the lower bound for algorithms that admit bidirectional compression is greater than or equal to that with unidirectional compression due to definition in (2).

Original: Corollary 1 ( Bidirectional unbiased compression).
Revised: Corollary 1 ( Bidirectional unbiased compression).

Original: Under the same setting as in Theorem 1, there exists a set of local objectives { f i } ni =1 ⊆ F ∆ ,L , stochastic gradient oracles { O i } ni =1 ⊆
Revised: Under the same setting as in Theorem 1, there exists a set of local objectives { f i } ni =1 ⊆ F ∆ ,L , stochastic gradient oracles { O i } ni =1 ⊆

Original: O σ 2 , ω -unbiased compressors { C i } ni =0 ⊆
Revised: O σ 2 , ω -unbiased compressors { C i } ni =0 ⊆

Original: U ω such that for any algorithm
Revised: U ω such that for any algorithm

Original: A ∈
Revised: A ∈

Original: A B { C i } ni =0 starting from x (0) , the lower bound in (3) is also valid.
Revised: A B { C i } ni =0 starting from x (0) , the lower bound in (3) is also valid.

Original: Theorem 1 and Corollary 1 indicate that distributed learning with both unidirectional and bidirectional communication compression share the same lower bound.
Revised: Theorem 1 and Corollary 1 indicate that distributed learning with both unidirectional and bidirectional communication compression share the same lower bound.

Original: It is intuitive since unidirectional compression is just a special case of bidirectional compression by letting C 

Revised: It is intuitive since unidirectional compression is just a special case of bidirectional compression by letting C

Original: = I .
Revised: = I .

Original: 3.3 Unidirectional Contractive Compression
Revised: 3.3 Unidirectional Contractive Compression

Original: To obtain the lower bounds for communication compression with contractive compressors, we need the following lemma [52, Lemma 1].
Revised: To obtain the lower bounds for communication compression with contractive compressors, we need the following lemma [53, Lemma 1].

Original: Lemma 1 ( Compressor relation).
Revised: Lemma 1 ( Compressor relation).

Original: It holds that δ U δ − 1 − 1 ≜ { δ C : C ∈ U
Revised: It holds that δ U δ − 1 − 1 ≜ { δ C : C ∈ U

Original: δ − 1 − 1 } ⊆
Revised: δ − 1 − 1 } ⊆

Original: C δ .
Revised: C δ .

Original: The above Lemma reveals that any ( δ − 1 − 1) -unbiased compressor is δ -contractive when scaled by δ .
Revised: The above Lemma reveals that any ( δ − 1 − 1) -unbiased compressor is δ -contractive when scaled by δ .

Original: Therefore, if an algorithm A admits all δ -contractive compressors, it will also admits all compressors in δ U δ − 1 − 1 due to Lemma 1.
Revised: Therefore, if an algorithm A admits all δ -contractive compressors, it will also admits all compressors in δ U δ − 1 − 1 due to Lemma 1.

Original: This relation, together with Theorem 1, helps us achieve the following lower bound with respect to δ -contractive compressors.
Revised: This relation, together with Theorem 1, helps us achieve the following lower bound with respect to δ -contractive compressors.

Original: Theorem 2 ( Unidirectional contractive compression).
Revised: Theorem 2 ( Unidirectional contractive compression).

Original: For every ∆ , L > 0 , n ≥ 2 , 0 < δ ≤ 1 , σ > 0 , T ≥ δ − 2 , there exists a set of loss objectives { f i } ni =1 ⊆ F ∆ ,L , a set of stochastic gradient oracles { O i } ni =1 ⊆
Revised: For every ∆ , L > 0 , n ≥ 2 , 0 < δ ≤ 1 , σ > 0 , T ≥ δ − 2 , there exists a set of loss objectives { f i } ni =1 ⊆ F ∆ ,L , a set of stochastic gradient oracles { O i } ni =1 ⊆

Original: O σ 2 , a set of δ -contractive compressors { C i } ni =1 ⊆
Revised: O σ 2 , a set of δ -contractive compressors { C i } ni =1 ⊆

Original: C δ with C 

Revised: C δ with C

Original: = I , such that
Revised: = I , such that

Original: Update v
Revised: Update v

Original: + v
Revised: + v

Original: + c
Revised: + c

Original: ,r ) end for ▷
Revised: ,r ) end for ▷

Original: } R − 1 r =0 will loop return Variable
Revised: } R − 1 r =0 will loop return Variable

Original: R ) .
Revised: R ) .

Original: ▷ v ( k,R ) =
Revised: ▷ v ( k,R ) =

Original: (cid:80) R − 1 r =0 c ( k,r ) for any algorithm
Revised: (cid:80) R − 1 r =0 c ( k,r ) for any algorithm

Original: A ∈
Revised: A ∈

Original: A U { C i } ni =0 starting from x (0) , it holds that
Revised: A U { C i } ni =0 starting from x (0) , it holds that

Original: [Equation]
Revised: [Equation]

Original: nT (cid:19)
Revised: nT (cid:19)

Original: [Equation]
Revised: [Equation]

Original: δT
Revised: δT

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: Transient complexity.
Revised: Transient complexity.

Original: With the discussion on transient complexity in Section 3.1, it is easy to derive the transient iteration complexity as Ω( n/δ 2 ) for the lower bound with δ -contractive compressor.
Revised: With the discussion on transient complexity in Section 3.1, it is easy to derive the transient iteration complexity as Ω( n/δ 2 ) for the lower bound with δ -contractive compressor.

Original: 3.4 Bidirectional Contractive Compression.
Revised: 3.4 Bidirectional Contractive Compression.

Original: Noting that { C i } ni =1 ⊆ C δ with C 

Revised: Noting that { C i } ni =1 ⊆ C δ with C

Original: = I is a special case of { C i } ni =0 ⊆
Revised: = I is a special case of { C i } ni =0 ⊆

Original: C δ , we can also establish the lower bound for algorithms that admit bidirectional compression and contractive compressions.
Revised: C δ , we can also establish the lower bound for algorithms that admit bidirectional compression and contractive compressions.

Original: Corollary 2 ( Bidirectional contractive compression).
Revised: Corollary 2 ( Bidirectional contractive compression).

Original: Under the same settings as in Theorem 2, there exists a set of loss objectives { f i } ni =1 ⊆ F ∆ ,L , a set of stochastic gradient oracles { O i } ni =1 ⊆
Revised: Under the same settings as in Theorem 2, there exists a set of loss objectives { f i } ni =1 ⊆ F ∆ ,L , a set of stochastic gradient oracles { O i } ni =1 ⊆

Original: O σ 2 , a set of δ -contractive compressors { C i } ni =1 ⊆
Revised: O σ 2 , a set of δ -contractive compressors { C i } ni =1 ⊆

Original: C δ , such that for any algorithm
Revised: C δ , such that for any algorithm

Original: A ∈
Revised: A ∈

Original: A B { C i } ni
Revised: A B { C i } ni

Original: = starting form x (0) , the lower bound (4) is also valid.
Revised: = starting form x (0) , the lower bound (4) is also valid.

Original:  NEOLITHIC: A nearly optimal algorithm
Revised:  NEOLITHIC: A nearly optimal algorithm

Original: Comparing the best-known upper bounds listed in Table 1 with the established lower bounds in (3) and (4), we find existing algorithms may not be optimal.
Revised: Comparing the best-known upper bounds listed in Table 1 with the established lower bounds in (3) and (4), we find existing algorithms may not be optimal.

Original: There exists a clear gap between their convergence rates and our established lower bounds.
Revised: There exists a clear gap between their convergence rates and our established lower bounds.

Original: In this section, we propose NEOLITHIC to fill in this gap.
Revised: In this section, we propose NEOLITHIC to fill in this gap.

Original: Its rate will match the lower bounds established in (3) and (4) up to logarithm factors.
Revised: Its rate will match the lower bounds established in (3) and (4) up to logarithm factors.

Original: NEOLITHIC can work with both unidirectional and bidirectional compressions, and it is compatible with both unbiased and contractive compressors.
Revised: NEOLITHIC can work with both unidirectional and bidirectional compression, and it is compatible with both unbiased and contractive compressors.

Original: NEOLITHIC will be discussed in detail with bidirectional contractive compression in this section.
Revised: NEOLITHIC will be discussed in detail with bidirectional contractive compression in this section.

Original: It is easy to be adapted to other settings by simply removing the server-to-worker compression or utilizing the scaled unbiased compressor.
Revised: It is easy to be adapted to other settings by simply removing the server-to-worker compression or utilizing the scaled unbiased compressor.

Original: 4.1 Fast Compressed Communication
Revised: 4.1 Fast Compressed Communication

Original: NEOLITHIC is built on a compression communication protocol listed in Algorithm 1, which we call fast compressed communication (FCC).
Revised: NEOLITHIC is built on a compression communication protocol listed in Algorithm 1, which we call fast compressed communication (FCC).

Original: Given an input vector v ( k,⋆ ) to communicate in the k -th iteration, FCC will first initialize v ( k, 0) = 

Revised: Given an input vector v ( k,⋆ ) to communicate in the k -th iteration, FCC will first initialize v ( k, 0) =

Original: and then recursively compresses the residual with c ( k,r ) ≜
Revised: and then recursively compresses the residual with c ( k,r ) ≜

Original: C ( v ( k,⋆ ) − v ( k,r ) ) and sends it to the receiver for R consecutive rounds, see the main recursion in Algorithm 1.
Revised: C ( v ( k,⋆ ) − v ( k,r ) ) and sends it to the receiver for R consecutive rounds, see the main recursion in Algorithm 1.

Original: When FCC operation finishes, the sender will transmit a set of compressed variables { c ( k,r ) } R − 1 r =0 to the receiver, and return v ( k,R )
Revised: When FCC operation finishes, the sender will transmit a set of compressed variables { c ( k,r ) } R − 1 r =0 to the receiver, and return v ( k,R )

Original: = (cid:80) R − 1 r =0 c ( k,r ) to itself.
Revised: = (cid:80) R − 1 r =0 c ( k,r ) to itself.

Original: Quantity v ( k,R ) can be regarded as a compressed vector of the original input v ( k,⋆ ) after the FCC protocol.
Revised: Quantity v ( k,R ) can be regarded as a compressed vector of the original input v ( k,⋆ ) after the FCC protocol.

Original: The FCC protocol can be conducted by the server (for which all variables in FCC are without any subscripts, e.g., c ( k,r ) and v ( k,r ) ) or by any worker i (for which all variables are with a subscript i , e.g., c ( k,r ) i and v ( k,r ) i ).
Revised: The FCC protocol can be conducted by the server (for which all variables in FCC are without any subscripts, e.g., c ( k,r ) and v ( k,r ) ) or by any worker i (for which all variables are with a subscript i , e.g., c ( k,r ) i and v ( k,r ) i ).

Original: When R = 1 , FCC reduces to the standard compression utilized in existing literature [60, 56, 23, 66, 30].
Revised: When R = 1 , FCC reduces to the standard compression utilized in existing literature [61, 57, 23, 67, 31].

Original: While FCC requires R rounds of communication per iteration, the following lemma establishes that the compression error will be exponentially decreased with the number of communication rounds R .
Revised: While FCC requires R rounds of communication per iteration, the following lemma establishes that the compression error will be exponentially decreased with the number of communication rounds R .

Original: Algorithm 2 NEOLITHIC
Revised: Algorithm 2 NEOLITHIC

Original: Input: Initialize x (0) ; learning rate γ ; compression round R ; δ ( − 1) =
Revised: Input: Initialize x (0) ; learning rate γ ; compression round R ; δ ( − 1) =

Original: δ ( − 1) i = 0 , ∀ i ∈ [ n ] for k = 0 , 1 , · · · , K do
Revised: δ ( − 1) i = 0 , ∀ i ∈ [ n ] for k = 0 , 1 , · · · , K do

Original: On all workers in parallel:
Revised: On all workers in parallel:

Original: Query stochastic gradients ˆ g ( k ) i = 1 R (cid:80) R − 1 r =

Revised: Query stochastic gradients ˆ g ( k ) i = 1 R (cid:80) R − 1 r =

Original: O i ( x ( k ) ; ζ ( k,r ) i )
Revised: O i ( x ( k ) ; ζ ( k,r ) i )

Original: ▷ Gradient accumulation
Revised: ▷ Gradient accumulation

Original: Error compensate ˜ g ( k ) i
Revised: Error compensate ˜ g ( k ) i

Original: =
Revised: =

Original: ˆ g ( k ) i + δ ( k − 1)
Revised: ˆ g ( k ) i + δ ( k − 1)

Original: i
Revised: i

Original: Update error δ ( k ) i
Revised: Update error δ ( k ) i

Original: = ˜
Revised: = ˜

Original: g ( k ) i − FCC
Revised: g ( k ) i − FCC

Original: (˜ g ( k ) i , C i , R, server )
Revised: (˜ g ( k ) i , C i , R, server )

Original: ▷ Worker sends { c ( k,r ) i } to server
Revised: ▷ Worker sends { c ( k,r ) i } to server

Original: On server:
Revised: On server:

Original: Error compensate ˜ g ( k ) =
Revised: Error compensate ˜ g ( k ) =

Original: δ ( k − 1) + 1 n (cid:80) ni =1 (cid:80) R − 1 r =0 c ( k,r )
Revised: δ ( k − 1) + 1 n (cid:80) ni =1 (cid:80) R − 1 r =0 c ( k,r )

Original: i ▷ { c ( k,r ) i } received from workers
Revised: i ▷ { c ( k,r ) i } received from workers

Original: Update error δ ( k ) =
Revised: Update error δ ( k ) =

Original: ˜ g ( k ) − FCC (˜ g ( k ) , C 0 , R, all workers )
Revised: ˜ g ( k ) − FCC (˜ g ( k ) , C 0 , R, all workers )

Original: ▷ Server sends { c ( k,r ) } to workers
Revised: ▷ Server sends { c ( k,r ) } to workers

Original: On all workers in parallel:
Revised: On all workers in parallel:

Original: Update model parameter x ( k +1)
Revised: Update model parameter x ( k +1)

Original: = x ( k ) − γ (cid:80) R − 1 r =0 c ( k,r ) ▷ { c ( k,r ) } received from server end for
Revised: = x ( k ) − γ (cid:80) R − 1 r =0 c ( k,r ) ▷ { c ( k,r ) } received from server end for

Original: Lemma 2 ( FCC property).
Revised: Lemma 2 ( FCC property).

Original: Let C be a δ -contractive compressor and v ( k,R ) =
Revised: Let C be a δ -contractive compressor and v ( k,R ) =

Original: FCC( v ( k,⋆ ) , C, R ) .
Revised: FCC( v ( k,⋆ ) , C, R ) .

Original: It holds for any R ≥ 1 and v ( k,⋆ ) ∈
Revised: It holds for any R ≥ 1 and v ( k,⋆ ) ∈

Original: R d that (proof is in Appendix B)
Revised: R d that (proof is in Appendix B)

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: When R = 1 , the above FCC property (5) reduces to Assumption 4 for standard contractive compressors.
Revised: When R = 1 , the above FCC property (5) reduces to Assumption 4 for standard contractive compressors.

Original: When R is large, FCC can output v ( k,R ) endowed with very small compression errors.
Revised: When R is large, FCC can output v ( k,R ) endowed with very small compression errors.

Original: The
Revised: The

Original: FCC protocol is also closely related to EF21 compression strategy [50] 2 .
Revised: FCC protocol is also closely related to EF21 compression strategy [51] 2 .

Original: However, the major novelty of FCC is to utilize multiple such compression rounds to help develop algorithms that can nearly match the established lower bounds.
Revised: However, the major novelty of FCC is to utilize multiple such compression rounds to help develop algorithms that can nearly match the established lower bounds.

Original: 4.2 NEOLITHIC Algorithm
Revised: 4.2 NEOLITHIC Algorithm

Original: NEOLITHIC is described in Algorithm 2.
Revised: NEOLITHIC is described in Algorithm 2.

Original: The FCC protocol in NEOLITHIC communicates R rounds per iteration.
Revised: The FCC protocol in NEOLITHIC communicates R rounds per iteration.

Original: To balance the gradient queries and communication rounds, NEOLITHIC will query R stochastic gradients per iteration, see the gradient accumulation step in Algorithm 2.
Revised: To balance the gradient queries and communication rounds, NEOLITHIC will query R stochastic gradients per iteration, see the gradient accumulation step in Algorithm 2.

Original: Compared to other algorithms listed in Table 1, the proposed NEOLITHIC takes R times more gradient queries and communication rounds than them per iteration.
Revised: Compared to other algorithms listed in Table 1, the proposed NEOLITHIC takes R times more gradient queries and communication rounds than them per iteration.

Original: Given the same budgets to query stochastic gradients and conduct communications as the other standard compression algorithms, we shall consider K = T/R iterations in NEOLITHIC for fair comparison.
Revised: Given the same budgets to query stochastic gradients and conduct communications as the other standard compression algorithms, we shall consider K = T/R iterations in NEOLITHIC for fair comparison.

Original: We introduce the following assumption to establish the convergence rates of NEOLITHIC.
Revised: We introduce the following assumption to establish the convergence rates of NEOLITHIC.

Original: Assumption 5 ( Gradient dissimilarity).
Revised: Assumption 5 ( Gradient dissimilarity).

Original: There exists some b 2 ≥ 

Revised: There exists some b 2 ≥

Original: such that
Revised: such that

Original: [Equation]
Revised: [Equation]

Original: When local distributions D i are equivalent across all nodes i , we have f i ( x ) =
Revised: When local distributions D i are equivalent across all nodes i , we have f i ( x ) =

Original: f
Revised: f

Original: ( x ) and the above assumption will always hold.
Revised: ( x ) and the above assumption will always hold.

Original: We first establish the convergence rate of NEOLITHIC with bidirectional and contractive compressors.
Revised: We first establish the convergence rate of NEOLITHIC with bidirectional and contractive compressors.

Original: Theorem 3 ( NEOLITHIC with contractive compressors).
Revised: Theorem 3 ( NEOLITHIC with contractive compressors).

Original: Given constants n ≥ 1 , δ ∈ (0 , 1] and Assumption 5, and let x ( k ) be generated by Algorithm 2.
Revised: Given constants n ≥ 1 , δ ∈ (0 , 1] and Assumption 5, and let x ( k ) be generated by Algorithm 2.

Original: If R = ⌈ max { ln (cid:0) δT max { b 2 , σ 2 δ } / ∆ L (cid:1) , ln(8) } /δ ⌉ and learning rate is set as in Appendix B, it holds for any K ≥ 0 and compressors { C i } ni =0 ⊆
Revised: If R = ⌈ max { ln (cid:0) δT max { b 2 , σ 2 δ } / ∆ L (cid:1) , ln(8) } /δ ⌉ and learning rate is set as in Appendix B, it holds for any K ≥ 0 and compressors { C i } ni =0 ⊆

Original: C δ that
Revised: C δ that

Original: [Equation]
Revised: [Equation]

Original: where T = KR is the total number of gradient query/communication rounds on each worker and notation ˜ O ( · ) hides logarithmic factors.
Revised: where T = KR is the total number of gradient query/communication rounds on each worker and notation ˜ O ( · ) hides logarithmic factors.

Original: The above rate implies a transient complexity of ˜ O ( nδ − 2 ) .
Revised: The above rate implies a transient complexity of ˜ O ( nδ − 2 ) .

Original: [Figure]
Revised: [Figure]

Original: When the compressor C is replaced with ω -unbiased compressors, we utilize the fact that (1+ ω ) − 1 C is (1 + ω ) − 1 -contractive to derive that
Revised: When the compressor C is replaced with ω -unbiased compressors, we utilize the fact that (1+ ω ) − 1 C is (1 + ω ) − 1 -contractive to derive that

Original: Corollary 3 ( NEOLITHIC with unbiased compressors).
Revised: Corollary 3 ( NEOLITHIC with unbiased compressors).

Original: Under the same assumptions as in
Revised: Under the same assumptions as in

Original: Theorem 3, it holds for any K ≥ 0 and compressors { C i } ni =0 ⊆
Revised: Theorem 3, it holds for any K ≥ 0 and compressors { C i } ni =0 ⊆

Original: U ω that
Revised: U ω that

Original: [Equation]
Revised: [Equation]

Original: This further leads to a transient complexity of ˜ O (cid:0) n (1 + ω ) 2 (cid:1) .
Revised: This further leads to a transient complexity of ˜ O (cid:0) n (1 + ω ) 2 (cid:1) .

Original: Remark 1.
Revised: Remark 1.

Original: The convergence rates established in Theorem 3 and Corollary 3 are also valid for unidirectional compression when C 

Revised: The convergence rates established in Theorem 3 and Corollary 3 are also valid for unidirectional compression when C

Original: = I .
Revised: = I .

Original: They can match the lower bounds established in Sectionup to logarithm factors.
Revised: They can match the lower bounds established in Sectionup to logarithm factors.

Original: Moreover, these rates are faster than other algorithms listed in Table 1.
Revised: Moreover, these rates are faster than other algorithms listed in Table 1.

Original: Remark 2.
Revised: Remark 2.

Original: The results in Theorem 3 and Corollary 3 also imply that NEOLITHIC with bidirectional compression can perform as fast as its counterpart with unidirectional compression.
Revised: The results in Theorem 3 and Corollary 3 also imply that NEOLITHIC with bidirectional compression can perform as fast as its counterpart with unidirectional compression.

Original: In other words, imposing bidirectional compression can save communications in NEOLITHIC without hurting convergence rates.
Revised: In other words, imposing bidirectional compression can save communications in NEOLITHIC without hurting convergence rates.

Original: Before our work, it is established in literature [56, 53, 60, 68, 14] that bidirectional compression leads to slower convergence than unidirectional compression for non-convex problems.
Revised: Before our work, it is established in literature [57, 54, 61, 69, 14] that bidirectional compression leads to slower convergence than unidirectional compression for non-convex problems.

Original: Remark 3.
Revised: Remark 3.

Original: We remark that Assumption 5 is not required to obtain the lower bounds in Section 3.
Revised: We remark that Assumption 5 is not required to obtain the lower bounds in Section 3.

Original: It is not known whether the lower bounds established in Section 3 can be achieved by NEOLITHIC when Assumption 5 does not hold.
Revised: It is not known whether the lower bounds established in Section 3 can be achieved by NEOLITHIC when Assumption 5 does not hold.

Original: However, it is worth noting that Assumption 5 is milder than those made in most works [80, 32, 56, 66, 60, 11] such as bounded gradients.
Revised: However, it is worth noting that Assumption 5 is milder than those made in most works [81, 33, 57, 67, 61, 11] such as bounded gradients.

Original: To our best knowledge, only
Revised: To our best knowledge, only

Original: EF21-SGD [23] is guaranteed to converge without Assumption 5, which, however, leads to a fairly loose convergence rate (see Table 1) that cannot show
Revised: EF21-SGD [23] is guaranteed to converge without Assumption 5, which, however, leads to a fairly loose convergence rate (see Table 1) that cannot show

Original: linear-speedup
Revised: linear-speedup

Original: O
Revised: O

Original: (1 / √ nT ) .
Revised: (1 / √ nT ) .

Original: Remark 4.
Revised: Remark 4.

Original: In the deterministic scenario, i.e., σ 2 = 0 , NEOLITHIC produces rate ˜ O
Revised: In the deterministic scenario, i.e., σ 2 = 0 , NEOLITHIC produces rate ˜ O

Original: ((1+ ω )∆ L/T ) .
Revised: ((1+ ω )∆ L/T ) .

Original: There exist literature (e.g., [25, 61]) that can achieve rate ˜ O
Revised: There exist literature (e.g., [25, 62]) that can achieve rate ˜ O

Original: ((1 + ω/n )∆ L/T ) which outperforms
Revised: ((1 + ω/n )∆ L/T ) which outperforms

Original: NEOLITHIC especially when ω and n are sufficiently large.
Revised: NEOLITHIC especially when ω and n are sufficiently large.

Original: However, this improvement is built upon an additional assumption that all local compressors { C i } ni =0 are independent of each other and cannot share the same randomness.
Revised: However, this improvement is built upon an additional assumption that all local compressors { C i } ni =0 are independent of each other and cannot share the same randomness.

Original: On the contrary, NEOLITHIC does not impose such an assumption and can be applied to both dependent and independent compressors.
Revised: On the contrary, NEOLITHIC does not impose such an assumption and can be applied to both dependent and independent compressors.

Original:  Experiments
Revised:  Experiments

Original: This section empirically investigates the performance of different compression algorithms with both synthetic simulation and deep learning tasks.
Revised: This section empirically investigates the performance of different compression algorithms with both synthetic simulation and deep learning tasks.

Original: We compare NEOLITHIC with PSGD and its variants with communication compression: MEM-SGD, Double-Squeeze, and EF21-SGD.
Revised: We compare NEOLITHIC with PSGD and its variants with communication compression: MEM-SGD, Double-Squeeze, and EF21-SGD.

Original: Implementation details and more experiments are provided in Appendix C.
Revised: Implementation details and more experiments are provided in Appendix C.

Original: Linear regression.
Revised: Linear regression.

Original: We solve a synthetic least-square problem with all aforementioned algorithms.
Revised: We solve a synthetic least-square problem with all aforementioned algorithms.

Original: We set d = 30 , n = 32 , R = 4 (for NEOLITHIC) and utilize the rand-1 compressor.
Revised: We set d = 30 , n = 32 , R = 4 (for NEOLITHIC) and utilize the rand-1 compressor.

Original: It is observed in Figure 1 (left) that NEOLITHIC beats MEM-SGD and Double-Squeeze with performance close to P-SGD.
Revised: It is observed in Figure 1 (left) that NEOLITHIC beats MEM-SGD and Double-Squeeze with performance close to P-SGD.

Original: It implies that algorithms with bidirectional compression can converge as fast as the counterparts with unidirectional compression, which is consistent with results in Remark 1.
Revised: It implies that algorithms with bidirectional compression can converge as fast as the counterparts with unidirectional compression, which is consistent with results in Remark 1.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Image classification.
Revised: Image classification.

Original: We investigate the performance on two common ResNet models [26] with
Revised: We investigate the performance on two common ResNet models [26] with

Original: CIFAR-10 [35] dataset.
Revised: CIFAR-10 [36] dataset.

Original: We train total 300 epochs and set the batch size to 128 on every worker.
Revised: We train total 300 epochs and set the batch size to 128 on every worker.

Original: All experiments were repeated three times with different seeds.
Revised: All experiments were repeated three times with different seeds.

Original: For NEOLITHIC, we set R = 2 .
Revised: For NEOLITHIC, we set R = 2 .

Original: We utilize top-k compressor [60] with different compression ratios.
Revised: We utilize top-k compressor [61] with different compression ratios.

Original: As shown in Figure 1 (right) and
Revised: As shown in Figure 1 (right) and

Original: Table 2, NEOLITHIC consistently outperforms other compression methods and reach the similar performance to PSGD.
Revised: Table 2, NEOLITHIC consistently outperforms other compression methods and reach the similar performance to PSGD.

Original: The effect of compression ratio.
Revised: The effect of compression ratio.

Original: We also investigate the influence of different compression ratios.
Revised: We also investigate the influence of different compression ratios.

Original: Table 3 used a compression ratio of 1%, which indicates a more harsh setting for compression methods.
Revised: Table 3 used a compression ratio of 1%, which indicates a more harsh setting for compression methods.

Original: NEOLITHIC still outperforms other compression methods.
Revised: NEOLITHIC still outperforms other compression methods.

Original:  Conclusion
Revised:  Conclusion

Original: This paper provides lower bounds for distributed algorithms with communication compression, whether the compression is unidirectional or bidirectional and unbiased or contractive.
Revised: This paper provides lower bounds for distributed algorithms with communication compression, whether the compression is unidirectional or bidirectional and unbiased or contractive.

Original: An algorithm called NEOLITHIC is introduced to match the lower bounds under the assumption of bounded gradient dissimilarity.
Revised: An algorithm called NEOLITHIC is introduced to match the lower bounds under the assumption of bounded gradient dissimilarity.

Original: Future directions include developing optimal algorithms without the assumption, as well as discovering additional compression properties that might produce a better lower bound
Revised: Future directions include developing optimal algorithms without the assumption, as well as discovering additional compression properties that might produce a better lower bound.

Original: Acknowledgements
Revised: Acknowledgements

Original: The authors are grateful to Professor Peter Richtarik from KAUST for the helpful discussions in the relation between FCC and EF21 and various other useful suggestions.
Revised: The authors are grateful to Professor Peter Richtarik from KAUST for the helpful discussions in the relation between FCC and EF21 and various other useful suggestions.

Original: A Lower Bounds
Revised: A Lower Bounds

Original: In this section, we provide the proofs for Theorem 1 and 2.
Revised: In this section, we provide the proofs for Theorem 1 and 2.

Original: A.1 Proof of Theorem
Revised: A.1 Proof of Theorem

Original: Without loss of generality, we assume algorithms to start from x (0)
Revised: Without loss of generality, we assume algorithms to start from x (0)

Original: = 0 .
Revised: = 0 .

Original: We denote the j -th coordinate of a vector x ∈ R d by [ x ] j for j = 1 , . . .
Revised: We denote the j -th coordinate of a vector x ∈ R d by [ x ] j for j = 1 , . . .

Original: , d , and let prog( x ) be prog
Revised: , d , and let prog( x ) be prog

Original: [Equation]
Revised: [Equation]

Original: if x
Revised: if x

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: otherwise .
Revised: otherwise .

Original: Similarly, for a set of points X = { x 1 , x 2 , . . .
Revised: Similarly, for a set of points X = { x 1 , x 2 , . . .

Original: } , we define prog( X ) :
Revised: } , we define prog( X ) :

Original: = max x ∈X prog( x ) .
Revised: = max x ∈X prog( x ) .

Original: As described in [16, 17], a zero chain function f satisfies
Revised: As described in [16, 17], a zero chain function f satisfies

Original: [Equation]
Revised: [Equation]

Original: which implies that, starting from x = 0 , a single gradient evaluation can only make at most one more coordinate for the model parameter x be non-zero.
Revised: which implies that, starting from x = 0 , a single gradient evaluation can only make at most one more coordinate for the model parameter x be non-zero.

Original: We prove the two terms of the lower bound in Theorem 1 separately by constructing two hard-tooptimize examples.
Revised: We prove the two terms of the lower bound in Theorem 1 separately by constructing two hard-tooptimize examples.

Original: The construction of the example, for each term in (3), can be conducted in three steps: 1) constructing local functions { f i } ni =1 by following the zero-chain function model proposed by [16]; 2) constructing compressors { C i } ni =1 ∈ U ω and oracles { O i } ni =1 ⊆
Revised: The construction of the example, for each term in (3), can be conducted in three steps: 1) constructing local functions { f i } ni =1 by following the zero-chain function model proposed by [16]; 2) constructing compressors { C i } ni =1 ∈ U ω and oracles { O i } ni =1 ⊆

Original: O σ 2 that hamper algorithms to increase the non-zero coordinates of the model parameters; 3) show a limitation in terms of the non-zero coordinates of model parameters, for algorithms obeying the desired protocol with T gradient queries and compressed communications on each worker, and translate the limitation into the lower bound of convergence rate.
Revised: O σ 2 that hamper algorithms to increase the non-zero coordinates of the model parameters; 3) show a limitation in terms of the non-zero coordinates of model parameters, for algorithms obeying the desired protocol with T gradient queries and compressed communications on each worker, and translate the limitation into the lower bound of convergence rate.

Original: We first state some key zero-chain functions that will be used to facilitate the analysis.
Revised: We first state some key zero-chain functions that will be used to facilitate the analysis.

Original: Lemma 3 (Lemma 2 of [7]) .
Revised: Lemma 3 (Lemma 2 of [7]) .

Original: Let [ x ] j denote the j -th coordinate of a vector x ∈ R d , and define function
Revised: Let [ x ] j denote the j -th coordinate of a vector x ∈ R d , and define function

Original: [Equation]
Revised: [Equation]

Original: where for ∀ z ∈ R ,
Revised: where for ∀ z ∈ R ,

Original: [Equation]
Revised: [Equation]

Original: Then h satisfy the following properties:
Revised: Then h satisfy the following properties:

Original: h ( x ) − inf x h ( x ) ≤ δ 0 d , ∀ x ∈ R d with δ 0 = 12 . 2.
Revised: h ( x ) − inf x h ( x ) ≤ δ 0 d , ∀ x ∈ R d with δ 0 = 12 . 2.

Original: h is ℓ 0 -smooth with ℓ 0 = 152 .
Revised: h is ℓ 0 -smooth with ℓ 0 = 152 .

Original: ∥∇ h ( x ) ∥ ∞ ≤ g ∞ , ∀ x ∈ R
Revised: ∥∇ h ( x ) ∥ ∞ ≤ g ∞ , ∀ x ∈ R

Original: d with g ∞ = 23 .
Revised: d with g ∞ = 23 .

Original: ∥∇ h ( x ) ∥ ∞ ≥ 1 for any x ∈ R d with [ x ] d = 0 .
Revised: ∥∇ h ( x ) ∥ ∞ ≥ 1 for any x ∈ R d with [ x ] d = 0 .

Original: Lemma 4.
Revised: Lemma 4.

Original: Let functions
Revised: Let functions

Original: [Equation]
Revised: [Equation]

Original: [ x
Revised: [ x

Original: [Equation]
Revised: [Equation]

Original: [ x ]
Revised: [ x ]

Original: [Equation]
Revised: [Equation]

Original: +
Revised: +

Original: [Equation]
Revised: [Equation]

Original: and h 2 ( x ) :=(cid:88)
Revised: and h 2 ( x ) :=(cid:88)

Original: j odd
Revised: j odd

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: j<d
Revised: j<d

Original: [Equation]
Revised: [Equation]

Original: (cid:17) .
Revised: (cid:17) .

Original: Then h 1 and h 2 satisfy the following properties:
Revised: Then h 1 and h 2 satisfy the following properties:

Original: 12 ( h 1 + h 2 )
Revised: 12 ( h 1 + h 2 )

Original: = h , where h is defined in Lemma 3.
Revised: = h , where h is defined in Lemma 3.

Original: For any x ∈ R d , if prog( x ) is odd, then prog( ∇ h 1 ( x ))
Revised: For any x ∈ R d , if prog( x ) is odd, then prog( ∇ h 1 ( x ))

Original: ≤ prog( x ) ; if prog( x ) is even, then prog( ∇ h 2 ( x ))
Revised: ≤ prog( x ) ; if prog( x ) is even, then prog( ∇ h 2 ( x ))

Original: ≤ prog( x ) .
Revised: ≤ prog( x ) .

Original: h 1 and h 2 are also ℓ 0 -smooth with ℓ 0 = 152 .
Revised: h 1 and h 2 are also ℓ 0 -smooth with ℓ 0 = 152 .

Original: Proof of Lemma 4.
Revised: Proof of Lemma 4.

Original: The first property follows the definition of h 1 , h 2 , and h .
Revised: The first property follows the definition of h 1 , h 2 , and h .

Original: The second property follows Lemma 1 of [16] that ψ ( m ) (0) = 0 for any m ∈ N .
Revised: The second property follows Lemma 1 of [16] that ψ ( m ) (0) = 0 for any m ∈ N .

Original: Now we prove the third property.
Revised: Now we prove the third property.

Original: Note that the Hessian of h k for k = 1 , 2 is tridiagonal and symmetric.
Revised: Note that the Hessian of h k for k = 1 , 2 is tridiagonal and symmetric.

Original: Consequently, by the Schur test, we have for any x ∈ R d and k = 1 ,
Revised: Consequently, by the Schur test, we have for any x ∈ R d and k = 1 ,

Original: [Equation]
Revised: [Equation]

Original: Furthermore, it is easy to verify that
Revised: Furthermore, it is easy to verify that

Original: [Equation]
Revised: [Equation]

Original: where the last inequality follows Observation 2 of [7] that
Revised: where the last inequality follows Observation 2 of [7] that

Original: [Equation]
Revised: [Equation]

Original: Combining (7) with (6), we know h k is ℓ 0 -smooth for k = 1 , 2 .
Revised: Combining (7) with (6), we know h k is ℓ 0 -smooth for k = 1 , 2 .

Original: Given Lemmas 3 and 4, we now construct two examples that lead to the two terms in lower bound (3), respectively.
Revised: Given Lemmas 3 and 4, we now construct two examples that lead to the two terms in lower bound (3), respectively.

Original: Example 1.
Revised: Example 1.

Original: The proof of the first term Ω(( ∆ Lσ 2 nT ) 12 ) essentially follows the first example in proving Theorem 1 of [41].
Revised: The proof of the first term Ω(( ∆ Lσ 2 nT ) 12 ) essentially follows the first example in proving Theorem 1 of [42].

Original: We provide the proof for the sake of being self-contained.
Revised: We provide the proof for the sake of being self-contained.

Original: (Step 1.)
Revised: (Step 1.)

Original: Let f i
Revised: Let f i

Original: =
Revised: =

Original: Lλ 2 h ( x/λ ) /ℓ 0 , ∀ i = 1 , . . .
Revised: Lλ 2 h ( x/λ ) /ℓ 0 , ∀ i = 1 , . . .

Original: , n be homogeneous and hence f
Revised: , n be homogeneous and hence f

Original: = Lλ 2 h
Revised: = Lλ 2 h

Original: ( x/λ ) /ℓwhere
Revised: ( x/λ ) /ℓwhere

Original: h is defined in Lemma 3 and λ > 0 is to be specified.
Revised: h is defined in Lemma 3 and λ > 0 is to be specified.

Original: Since ∇ 2 f i
Revised: Since ∇ 2 f i

Original: =
Revised: =

Original: L ∇ 2 h/ℓ 0 and h is ℓ 0 -smooth
Revised: L ∇ 2 h/ℓ 0 and h is ℓ 0 -smooth

Original: (Lemma 3), we know f
Revised: (Lemma 3), we know f

Original: i is L -smooth for any λ > 0 .
Revised: i is L -smooth for any λ > 0 .

Original: By Lemma 3, we have
Revised: By Lemma 3, we have

Original: [Equation]
Revised: [Equation]

Original: Therefore, to ensure f i ∈ F ∆ ,L , it suffices to let
Revised: Therefore, to ensure f i ∈ F ∆ ,L , it suffices to let

Original: [Equation]
Revised: [Equation]

Original: (Step 2.)
Revised: (Step 2.)

Original: We consider all compressors { C i } ni =1 are identity, meaning that there is no compression error in the optimization procedure.
Revised: We consider all compressors { C i } ni =1 are identity, meaning that there is no compression error in the optimization procedure.

Original: Then we naturally have { C i } ni
Revised: Then we naturally have { C i } ni

Original: =1 ⊆
Revised: =1 ⊆

Original: U ω for any ω ≥ 0 .
Revised: U ω for any ω ≥ 0 .

Original: We construct the stochastic gradient oracle O i on worker i , ∀ i = 1 , . . .
Revised: We construct the stochastic gradient oracle O i on worker i , ∀ i = 1 , . . .

Original: , n as the following:
Revised: , n as the following:

Original: [Equation]
Revised: [Equation]

Original: with random variable Z ∼ Bernoulii ( p ) independent of x and f i , and p ∈ (0 , 1) to be specified.
Revised: with random variable Z ∼ Bernoulii ( p ) independent of x and f i , and p ∈ (0 , 1) to be specified.

Original: The oracle O i has probability to make the entry [ ∇ f i ( x )] prog( x )+1 zero.
Revised: The oracle O i has probability to make the entry [ ∇ f i ( x )] prog( x )+1 zero.

Original: Therefore algorithms are hampered by the gradient oracle to achieve more non-zero coordinates for the model parameters.
Revised: Therefore algorithms are hampered by the gradient oracle to achieve more non-zero coordinates for the model parameters.

Original: It is easy to see O i is an unbiased stochastic gradient oracle.
Revised: It is easy to see O i is an unbiased stochastic gradient oracle.

Original: Moreover, since f i is zero-chain, we have prog( O i ( x ; Z )) ≤ prog( ∇ f i ( x ))
Revised: Moreover, since f i is zero-chain, we have prog( O i ( x ; Z )) ≤ prog( ∇ f i ( x ))

Original: ≤ prog( x ) + 1 and hence
Revised: ≤ prog( x ) + 1 and hence

Original: E [ ∥
Revised: E [ ∥

Original: [ O i ( x ; Z )] − ∇ f i ( x ) ∥ 2 ] = | [ ∇ f i ( x )] prog( x )
Revised: [ O i ( x ; Z )] − ∇ f i ( x ) ∥ 2 ] = | [ ∇ f i ( x )] prog( x )

Original: [Equation]
Revised: [Equation]

Original: Therefore, to ensure O i ∈ O σ 2 , it suffices to let
Revised: Therefore, to ensure O i ∈ O σ 2 , it suffices to let

Original: [Equation]
Revised: [Equation]

Original: (Step 3.)
Revised: (Step 3.)

Original: Let x ( t ) i , ∀ t = 0 , . . .
Revised: Let x ( t ) i , ∀ t = 0 , . . .

Original: and 1 ≤ i ≤ n , be the t -th query point of worker i .
Revised: and 1 ≤ i ≤ n , be the t -th query point of worker i .

Original: Since algorithms satisfy the zero-respecting property, as discussed in [16, 17, 41], within T gradient queries on each worker, algorithms can only return model ˆ x such that
Revised: Since algorithms satisfy the zero-respecting property, as discussed in [16, 17, 42], within T gradient queries on each worker, algorithms can only return model ˆ x such that

Original: [Equation]
Revised: [Equation]

Original: which implies
Revised: which implies

Original: [Equation]
Revised: [Equation]

Original: By Lemma 2 of [41]
Revised: By Lemma 2 of [42]

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: On the other hand
Revised: On the other hand

Original: [Equation]
Revised: [Equation]

Original: when prog(ˆ x ) <
Revised: when prog(ˆ x ) <

Original: d , by the fourth point in Lemma
Revised: d , by the fourth point in Lemma

Original: [Equation]
Revised: [Equation]

Original: it holds that
Revised: it holds that

Original: [Equation]
Revised: [Equation]

Original: Therefore, by combining (11) and (12), we have
Revised: Therefore, by combining (11) and (12), we have

Original: [Equation]
Revised: [Equation]

Original: Let
Revised: Let

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Then (8) naturally holds and p = min { g 2 ∞ σ 2 (cid:16)
Revised: Then (8) naturally holds and p = min { g 2 ∞ σ 2 (cid:16)

Original: [Equation]
Revised: [Equation]

Original: g 2 ∞ (cid:17) 12 , 1 } by (9).
Revised: g 2 ∞ (cid:17) 12 , 1 } by (9).

Original: Without loss of generality, we assume d ≥ 2 , which is guaranteed when T = Ω( σ 2 nL ∆ ) .
Revised: Without loss of generality, we assume d ≥ 2 , which is guaranteed when T = Ω( σ 2 nL ∆ ) .

Original: Then, using the definition of p , we have that
Revised: Then, using the definition of p , we have that

Original: [Equation]
Revised: [Equation]

Original: which, combined with (13), further implies
Revised: which, combined with (13), further implies

Original: [Equation]
Revised: [Equation]

Original: Example 2.
Revised: Example 2.

Original: Without loss of generality, we assume n is even, otherwise we can consider the lower bound for the case of n − 1 .
Revised: Without loss of generality, we assume n is even, otherwise we can consider the lower bound for the case of n − 1 .

Original: (Step 1.)
Revised: (Step 1.)

Original: Similar to the construction of Example 1, we let f i
Revised: Similar to the construction of Example 1, we let f i

Original: =
Revised: =

Original: Lλ 2 h 1 ( x/λ ) /ℓ 0 , ∀ 1 ≤ i ≤ n/ 2 and f i
Revised: Lλ 2 h 1 ( x/λ ) /ℓ 0 , ∀ 1 ≤ i ≤ n/ 2 and f i

Original: = Lλ 2 h 2 ( x/λ ) /ℓ 0 , ∀ n/ 2 < i ≤ n , where h 1 and h 2 are defined in Lemma 4, and λ > 0 will be specified later.
Revised: = Lλ 2 h 2 ( x/λ ) /ℓ 0 , ∀ n/ 2 < i ≤ n , where h 1 and h 2 are defined in Lemma 4, and λ > 0 will be specified later.

Original: By the definitions of h 1 and h 2 , we have that f i , ∀ 1 ≤ i ≤ n is zero-chain and f ( x ) = 1 n (cid:80) n i =1 f i ( x ) =
Revised: By the definitions of h 1 and h 2 , we have that f i , ∀ 1 ≤ i ≤ n is zero-chain and f ( x ) = 1 n (cid:80) n i =1 f i ( x ) =

Original: Lλ 2 /ℓ 0 h ( x/λ ) .
Revised: Lλ 2 /ℓ 0 h ( x/λ ) .

Original: Since h 1 and h 2 are also ℓ 0 -smooth, to let f ∈ F ∆ ,L , it suffices to make (8) hold.
Revised: Since h 1 and h 2 are also ℓ 0 -smooth, to let f ∈ F ∆ ,L , it suffices to make (8) hold.

Original: In the above construction, we essentially split a zero-chain function, i.e., h , into two different components: the even component of the chain, i.e., h 1 and the odd component of the chain, i.e., h 2 .
Revised: In the above construction, we essentially split a zero-chain function, i.e., h , into two different components: the even component of the chain, i.e., h 1 and the odd component of the chain, i.e., h 2 .

Original: Recall the second property in Lemma 4 that for any x ∈ R d , if prog( x ) is odd, then prog( ∇ h 1 ( x ))
Revised: Recall the second property in Lemma 4 that for any x ∈ R d , if prog( x ) is odd, then prog( ∇ h 1 ( x ))

Original: ≤ prog( x ) ; if prog( x ) is even, then prog( ∇ h 2 ( x ))
Revised: ≤ prog( x ) ; if prog( x ) is even, then prog( ∇ h 2 ( x ))

Original: ≤ prog( x ) .
Revised: ≤ prog( x ) .

Original: Therefore, the workers, starting from any point with any algorithm A ∈
Revised: Therefore, the workers, starting from any point with any algorithm A ∈

Original: A U { C i } ni =1 , can only earn one more non-zero coordinate if they do not synchronize (communicate) with the server; after that, the number of non-zero coordinates of local models will not increase any more.
Revised: A U { C i } ni =1 , can only earn one more non-zero coordinate if they do not synchronize (communicate) with the server; after that, the number of non-zero coordinates of local models will not increase any more.

Original: That is to say, in order to proceed (i.e.,, achieve more non-zero coordinates), the worker must synchronize the gradient information, via the server, between the odd and even components.
Revised: That is to say, in order to proceed (i.e.,, achieve more non-zero coordinates), the worker must synchronize the gradient information, via the server, between the odd and even components.

Original: (Step 2.)
Revised: (Step 2.)

Original: We consider a gradient oracle that can return the lossless full-batch gradient, i.e., O i ( x ) = ∇ f i ( x ) , ∀ x ∈ R , 1 ≤ i ≤ n .
Revised: We consider a gradient oracle that can return the lossless full-batch gradient, i.e., O i ( x ) = ∇ f i ( x ) , ∀ x ∈ R , 1 ≤ i ≤ n .

Original: For the construction of ω -unbiased compressors, we consider { C i } ni = to be the ds × rand- s operators with shared randomness and s = ⌈ d/ (1 + ω ) ⌉ , where the ds -scaling procedure is to ensure unbiasedness.
Revised: For the construction of ω -unbiased compressors, we consider { C i } ni = to be the ds × rand- s operators with shared randomness and s = ⌈ d/ (1 + ω ) ⌉ , where the ds -scaling procedure is to ensure unbiasedness.

Original: Specifically, during a round of communication, all workers randomly choose s coordinates from the full vector to be communicated, and then transmit the ds -scaled values at chosen coordinates.
Revised: Specifically, during a round of communication, all workers randomly choose s coordinates from the full vector to be communicated, and then transmit the ds -scaled values at chosen coordinates.

Original: The chosen coordinate indexes are identical across all workers due to the shared randomness and are sampled uniformly randomly per communication.
Revised: The chosen coordinate indexes are identical across all workers due to the shared randomness and are sampled uniformly randomly per communication.

Original: Since for any 1 ≤ k ≤ d , the index k has probability s/d to be chosen, we have for any x ∈ R d
Revised: Since for any 1 ≤ k ≤ d , the index k has probability s/d to be chosen, we have for any x ∈ R d

Original: [Equation]
Revised: [Equation]

Original: chosen }
Revised: chosen }

Original: [Equation]
Revised: [Equation]

Original: chosen
Revised: chosen

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: E [ ∥ C
Revised: E [ ∥ C

Original: [Equation]
Revised: [Equation]

Original: ( x )
Revised: ( x )

Original: [Equation]
Revised: [Equation]

Original: x ∥
Revised: x ∥

Original: [Equation]
Revised: [Equation]

Original: ] =
Revised: ] =

Original: [Equation]
Revised: [Equation]

Original: chosen
Revised: chosen

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: k = x
Revised: k = x

Original: [Equation]
Revised: [Equation]

Original: k (cid:32)(cid:18) d
Revised: k (cid:32)(cid:18) d

Original: [Equation]
Revised: [Equation]

Original: (cid:19)
Revised: (cid:19)

Original: [Equation]
Revised: [Equation]

Original: P (
Revised: P (

Original: [Equation]
Revised: [Equation]

Original: where the last inequality follows the definition of s .
Revised: where the last inequality follows the definition of s .

Original: Therefore, the above construction ensures { C i } ni =1 ⊆
Revised: Therefore, the above construction ensures { C i } ni =1 ⊆

Original: U ω .
Revised: U ω .

Original: For any t = 0 , 1 , . . .
Revised: For any t = 0 , 1 , . . .

Original: and 1 ≤ i ≤ n , let v ( t,⋆ ) i be the synchronized point that worker i wants to send to the server at the t -th communication.
Revised: and 1 ≤ i ≤ n , let v ( t,⋆ ) i be the synchronized point that worker i wants to send to the server at the t -th communication.

Original: Due to communication compression, the server can only receive a compressed version C i ( v ( t,⋆ ) i ) instead, which we denote by v ( t ) i ≜
Revised: Due to communication compression, the server can only receive a compressed version C i ( v ( t,⋆ ) i ) instead, which we denote by v ( t ) i ≜

Original: C i ( v ( t,⋆ ) i ) .
Revised: C i ( v ( t,⋆ ) i ) .

Original: It is easy to see that
Revised: It is easy to see that

Original: [Equation]
Revised: [Equation]

Original: Furthermore, since the compressor C i only takes s coordinates, v ( t ) i has probability 1 − s/d ≈ ω 1+ ω in making the coordinate with index prog( v ( t,⋆ ) i ) zero, which implies P (prog( v ( t ) ) < prog( v ( t,⋆ ) ))
Revised: Furthermore, since the compressor C i only takes s coordinates, v ( t ) i has probability 1 − s/d ≈ ω 1+ ω in making the coordinate with index prog( v ( t,⋆ ) i ) zero, which implies P (prog( v ( t ) ) < prog( v ( t,⋆ ) ))

Original: ≈ ω
Revised: ≈ ω

Original: 1+ ω .
Revised: 1+ ω .

Original: This is to say, each worker i has only probability ≈ (1 + ω ) − 1 to transmit its last non-zero entry.
Revised: This is to say, each worker i has only probability ≈ (1 + ω ) − 1 to transmit its last non-zero entry.

Original: Therefore, algorithms are hampered by the compressors to achieve more non-zero coordinates for model parameters in synchronizing the gradient information from all workers.
Revised: Therefore, algorithms are hampered by the compressors to achieve more non-zero coordinates for model parameters in synchronizing the gradient information from all workers.

Original: (Step 3.)
Revised: (Step 3.)

Original: Since we only consider algorithms satisfying the zero-respecting property, as discussed in [16, 17, 41], worker i , within T rounds of communication, can only return model ˆ x such that
Revised: Since we only consider algorithms satisfying the zero-respecting property, as discussed in [16, 17, 42], worker i , within T rounds of communication, can only return model ˆ x such that

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Since f i s are zero-chain functions, ˆ x i satisfies
Revised: Since f i s are zero-chain functions, ˆ x i satisfies

Original: [Equation]
Revised: [Equation]

Original: By Lemma 5, we have
Revised: By Lemma 5, we have

Original: [Equation]
Revised: [Equation]

Original: Combining (18) with (17) and (12), we have that
Revised: Combining (18) with (17) and (12), we have that

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Let
Revised: Let

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Then (8) naturally holds.
Revised: Then (8) naturally holds.

Original: Since T is assumed to be no less than (1+ ω ) 2 , we have d = ⌊ 5 T/ (1+ ω ) ⌋ ≥ 5 T/ (1 + ω ) − 1 ≥ 

Revised: Since T is assumed to be no less than (1+ ω ) 2 , we have d = ⌊ 5 T/ (1+ ω ) ⌋ ≥ 5 T/ (1 + ω ) − 1 ≥

Original: T/
Revised: T/

Original: (1 + ω ) ≥ 4(1 + ω ) ≥ 4 .
Revised: (1 + ω ) ≥ 4(1 + ω ) ≥ 4 .

Original: Then it is easy to verify
Revised: Then it is easy to verify

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: which, combined with (19), further implies
Revised: which, combined with (19), further implies

Original: [Equation]
Revised: [Equation]

Original: Lemma 5.
Revised: Lemma 5.

Original: In Example 2 in the proof of Theorem 1, it holds that
Revised: In Example 2 in the proof of Theorem 1, it holds that

Original: [Equation]
Revised: [Equation]

Original: for any T ≥ 0 .
Revised: for any T ≥ 0 .

Original: Proof.
Revised: Proof.

Original: Note that for any 0 ≤ t < T , at the ( t + 1) -th round of communication, worker i can only transmit vector v ( t,⋆ ) i that is aggregated from local gradient descent and the vectors in the past communication.
Revised: Note that for any 0 ≤ t < T , at the ( t + 1) -th round of communication, worker i can only transmit vector v ( t,⋆ ) i that is aggregated from local gradient descent and the vectors in the past communication.

Original: Therefore, v ( t,⋆ )
Revised: Therefore, v ( t,⋆ )

Original: [Equation]
Revised: [Equation]

Original: satisfies
Revised: satisfies

Original: [Equation]
Revised: [Equation]

Original: We define B (0) = 1 additionally.
Revised: We define B (0) = 1 additionally.

Original: By the definition of B ( t ) and that prog( v ( t,⋆ ) ) ≤ prog( v ( t,⋆ ) i ) , it naturally holds that
Revised: By the definition of B ( t ) and that prog( v ( t,⋆ ) ) ≤ prog( v ( t,⋆ ) i ) , it naturally holds that

Original: [Equation]
Revised: [Equation]

Original: Therefore, one round of communication can increase B ( t ) at most by 1 .
Revised: Therefore, one round of communication can increase B ( t ) at most by 1 .

Original: From (22), we see that B ( t +1) =
Revised: From (22), we see that B ( t +1) =

Original: B ( t ) + 1 only when max 1 ≤ i ≤ n prog( v ( t ) i ) =
Revised: B ( t ) + 1 only when max 1 ≤ i ≤ n prog( v ( t ) i ) =

Original: max 1 ≤ i ≤ n prog( v ( t,⋆ ) i ) .
Revised: max 1 ≤ i ≤ n prog( v ( t,⋆ ) i ) .

Original: Let k = max 1 ≤ i ≤ n prog( v ( t,⋆ ) i ) .
Revised: Let k = max 1 ≤ i ≤ n prog( v ( t,⋆ ) i ) .

Original: Recall that the compressors constructed in Example 2 are built on the shared randomness, therefore max 1 ≤ i ≤ n prog( v ( t ) i ) =
Revised: Recall that the compressors constructed in Example 2 are built on the shared randomness, therefore max 1 ≤ i ≤ n prog( v ( t ) i ) =

Original: max 1 ≤ i ≤ n prog( v ( t,⋆ ) i ) =
Revised: max 1 ≤ i ≤ n prog( v ( t,⋆ ) i ) =

Original: k is equivalent to that the coordinate index k is chosen during the ( t + 1) - round of communication compression, which is of probability sd .
Revised: k is equivalent to that the coordinate index k is chosen during the ( t + 1) - round of communication compression, which is of probability sd .

Original: Therefore, we have
Revised: Therefore, we have

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: chosen
Revised: chosen

Original: [Equation]
Revised: [Equation]

Original: Let E ( t +1) be the event { the coordinate index max 1 ≤ i ≤ n prog( v ( t,⋆ ) i ) is chosen during the ( t + 1) -round of compression } .
Revised: Let E ( t +1) be the event { the coordinate index max 1 ≤ i ≤ n prog( v ( t,⋆ ) i ) is chosen during the ( t + 1) -round of compression } .

Original: Since the compression is uniformly random, we have
Revised: Since the compression is uniformly random, we have

Original: 1 ( E (1) ) , . . .
Revised: 1 ( E (1) ) , . . .

Original: , 1 ( E ( T ) ) i.i.d.
Revised: , 1 ( E ( T ) ) i.i.d.

Original: ∼ Bernoulli( sd ) where 1 ( · ) is the indicator function.
Revised: ∼ Bernoulli( sd ) where 1 ( · ) is the indicator function.

Original: By the above argument, we also have B ( t +1) − B ( t ) ≤ 1 ( E ( t +1) ) for any 0 ≤ t < T .
Revised: By the above argument, we also have B ( t +1) − B ( t ) ≤ 1 ( E ( t +1) ) for any 0 ≤ t < T .

Original: Therefore, we have
Revised: Therefore, we have

Original: P ( B ( T )
Revised: P ( B ( T )

Original: [Equation]
Revised: [Equation]

Original: which directly leads to the conclusion.
Revised: which directly leads to the conclusion.

Original: A.2 Proof of Theorem
Revised: A.2 Proof of Theorem

Original: Theorem 2 essentially follows the same analysis as in Theorem 1.
Revised: Theorem 2 essentially follows the same analysis as in Theorem 1.

Original: The only difference is that we shall construct compressors in proving Ω( ∆ L δT ) by using rand- s operators with shared randomness and s = ⌈ δd ⌉ .
Revised: The only difference is that we shall construct compressors in proving Ω( ∆ L δT ) by using rand- s operators with shared randomness and s = ⌈ δd ⌉ .

Original: There is no scaling procedure in compression.
Revised: There is no scaling procedure in compression.

Original: We can easily check that
Revised: We can easily check that

Original: [Equation]
Revised: [Equation]

Original: k is chosen }
Revised: k is chosen }

Original: [Equation]
Revised: [Equation]

Original: not
Revised: not

Original: [Equation]
Revised: [Equation]

Original: where the last inequality follows the definition of s .
Revised: where the last inequality follows the definition of s .

Original: Therefore, we have { C i } ni =1 ⊆ C δ .
Revised: Therefore, we have { C i } ni =1 ⊆ C δ .

Original: The scaling procedure does not change prog , and thus makes no effect on the argument in terms of non-zero coordinates.
Revised: The scaling procedure does not change prog , and thus makes no effect on the argument in terms of non-zero coordinates.

Original: By considering ω = δ − 1 − 1 , i.e., 1 + ω
Revised: By considering ω = δ − 1 − 1 , i.e., 1 + ω

Original: =
Revised: =

Original: δ , we can easily adapt the proof of Theorem 1 to reach Theorem 2.
Revised: δ , we can easily adapt the proof of Theorem 1 to reach Theorem 2.

Original: B Convergence of NEOLITHIC B.1 Proof of Lemma
Revised: B Convergence of NEOLITHIC B.1 Proof of Lemma

Original: Let v ( k,r ) be the r -th intermediate point generated in FCC for any 0 ≤ r ≤ R .
Revised: Let v ( k,r ) be the r -th intermediate point generated in FCC for any 0 ≤ r ≤ R .

Original: Since C is δcontractive, we have
Revised: Since C is δcontractive, we have

Original: [Equation]
Revised: [Equation]

Original: B.2 Proof of Theorem
Revised: B.2 Proof of Theorem

Original: In this subsection, we provide the convergence proof for NEOLITHIC with bidirectional compression and contractive compressors.
Revised: In this subsection, we provide the convergence proof for NEOLITHIC with bidirectional compression and contractive compressors.

Original: We first introduce some notations: Ω ( k ) :
Revised: We first introduce some notations: Ω ( k ) :

Original: = δ ( k ) + 1 n (cid:80) ni =1 δ ( k )
Revised: = δ ( k ) + 1 n (cid:80) ni =1 δ ( k )

Original: i , ∀ k ≥ − 1 ; y ( k ) :
Revised: i , ∀ k ≥ − 1 ; y ( k ) :

Original: = x ( k ) − γ Ω ( k − 1) , ∀ k ≥ 0 ; Ψ ( k ) :
Revised: = x ( k ) − γ Ω ( k − 1) , ∀ k ≥ 0 ; Ψ ( k ) :

Original: = ∥
Revised: = ∥

Original: δ ( k ) ∥ 2 + 3 n (cid:80) n i =1 ∥ δ ( k ) i ∥ 2 , ∀ k ≥ − 1 .
Revised: δ ( k ) ∥ 2 + 3 n (cid:80) n i =1 ∥ δ ( k ) i ∥ 2 , ∀ k ≥ − 1 .

Original: We will use the following lemmas.
Revised: We will use the following lemmas.

Original: Lemma 6 (R ECURSION F ORMULA ) .
Revised: Lemma 6 (R ECURSION F ORMULA ) .

Original: For any k ≥ 0 , it holds that
Revised: For any k ≥ 0 , it holds that

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: It is observed from Algorithm 1 that the vector v ( k,R ) returned by the FCC operator satisfies
Revised: It is observed from Algorithm 1 that the vector v ( k,R ) returned by the FCC operator satisfies

Original: [Equation]
Revised: [Equation]

Original: FCC is
Revised: FCC is

Original: [Equation]
Revised: [Equation]

Original: node i
Revised: node i

Original: [Equation]
Revised: [Equation]

Original: With (24), the relation between ˜ g ( k ) and δ ( k ) (or between ˜ g ( k ) i and δ ( k ) i ) in Algorithm 2 satisfies
Revised: With (24), the relation between ˜ g ( k ) and δ ( k ) (or between ˜ g ( k ) i and δ ( k ) i ) in Algorithm 2 satisfies

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: Therefore, we have for any k ≥
Revised: Therefore, we have for any k ≥

Original: [Equation]
Revised: [Equation]

Original: 28) where (26) and (27) follow the implementation of Algorithm 2, and we use the notation of Ω ( k ) in (28).
Revised: 28) where (26) and (27) follow the implementation of Algorithm 2, and we use the notation of Ω ( k ) in (28).

Original: Lemma 7 (D ESCENT L EMMA ) .
Revised: Lemma 7 (D ESCENT L EMMA ) .

Original: Let the auxiliary sequence be y ( k ) :
Revised: Let the auxiliary sequence be y ( k ) :

Original: = x ( k ) − γ Ω ( k − 1) , ∀ k ≥ 0 .
Revised: = x ( k ) − γ Ω ( k − 1) , ∀ k ≥ 0 .

Original: Under Assumption 1, if learning rate 0 < γ ≤ 12 L , it holds that for any k ≥ 0 ,
Revised: Under Assumption 1, if learning rate 0 < γ ≤ 12 L , it holds that for any k ≥ 0 ,

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: By Lemma 6 and the definition of y ( k ) for k ≥ 0 , we directly have that
Revised: By Lemma 6 and the definition of y ( k ) for k ≥ 0 , we directly have that

Original: [Equation]
Revised: [Equation]

Original: Since f is L -smooth, we have
Revised: Since f is L -smooth, we have

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Since ˆ g ( k ) i = 1 R (cid:80)
Revised: Since ˆ g ( k ) i = 1 R (cid:80)

Original: Rr =1 O i ( x ( k ) ; ζ ( k,r ) i ) is a unbiased estimator of ∇ f i ( x ( k ) ) , and by Assumption 2, we have
Revised: Rr =1 O i ( x ( k ) ; ζ ( k,r ) i ) is a unbiased estimator of ∇ f i ( x ( k ) ) , and by Assumption 2, we have

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: E
Revised: E

Original: [Equation]
Revised: [Equation]

Original: Taking global expectation over (30)
Revised: Taking global expectation over (30)

Original: [Equation]
Revised: [Equation]

Original: and plugging (31) and (32) into it
Revised: and plugging (31) and (32) into it

Original: [Equation]
Revised: [Equation]

Original: we reach
Revised: we reach

Original: E [ f
Revised: E [ f

Original: [Equation]
Revised: [Equation]

Original: y ( k +1) )]
Revised: y ( k +1) )]

Original: [Equation]
Revised: [Equation]

Original: E [ f ( y ( k ) )]
Revised: E [ f ( y ( k ) )]

Original: [Equation]
Revised: [Equation]

Original: where we use Young’s inequality in (33), and (34) holds by Assumption 1.
Revised: where we use Young’s inequality in (33), and (34) holds by Assumption 1.

Original: Using y ( k ) − x ( k )
Revised: Using y ( k ) − x ( k )

Original: =
Revised: =

Original: − γ Ω
Revised: − γ Ω

Original: ( k − 1) and that
Revised: ( k − 1) and that

Original: [Equation]
Revised: [Equation]

Original: in (34), we reach the conclusion in this lemma.
Revised: in (34), we reach the conclusion in this lemma.

Original: Lemma 8 (V ANISHING E RROR ) .
Revised: Lemma 8 (V ANISHING E RROR ) .

Original: Assume R is sufficiently large such that (1 − δ ) R < 14 and let Ψ ( k ) :
Revised: Assume R is sufficiently large such that (1 − δ ) R < 14 and let Ψ ( k ) :

Original: = ∥
Revised: = ∥

Original: δ ( k ) ∥ 2 + 3 n (cid:80) ni
Revised: δ ( k ) ∥ 2 + 3 n (cid:80) ni

Original: =1 ∥ δ ( k ) i ∥ 2 for k ≥ − 1 .
Revised: =1 ∥ δ ( k ) i ∥ 2 for k ≥ − 1 .

Original: Under Assumptions 2, 4, 5 It holds that
Revised: Under Assumptions 2, 4, 5 It holds that

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: By Lemma 2, it holds that
Revised: By Lemma 2, it holds that

Original: [Equation]
Revised: [Equation]

Original: Note that
Revised: Note that

Original: [Equation]
Revised: [Equation]

Original: n
Revised: n

Original: [Equation]
Revised: [Equation]

Original: Therefore, by using Young’s inequality and (32), we have
Revised: Therefore, by using Young’s inequality and (32), we have

Original: [Equation]
Revised: [Equation]

Original: For any 1 ≤ i ≤ n , using the similar argument
Revised: For any 1 ≤ i ≤ n , using the similar argument

Original: [Equation]
Revised: [Equation]

Original: we have that
Revised: we have that

Original: [Equation]
Revised: [Equation]

Original: Note that for any 1 ≤ i ≤ n ,
Revised: Note that for any 1 ≤ i ≤ n ,

Original: [Equation]
Revised: [Equation]

Original: Taking the average of (36) over all 1 ≤ i ≤ n
Revised: Taking the average of (36) over all 1 ≤ i ≤ n

Original: [Equation]
Revised: [Equation]

Original: and using (37) and Assumption
Revised: and using (37) and Assumption

Original: [Equation]
Revised: [Equation]

Original: we further have that
Revised: we further have that

Original: [Equation]
Revised: [Equation]

Original: Summing up (35) and plugging (38) into it, we obtain
Revised: Summing up (35) and plugging (38) into it, we obtain

Original: [Equation]
Revised: [Equation]

Original: We thus have
Revised: We thus have

Original: [Equation]
Revised: [Equation]

Original: Since R is sufficiently large such that
Revised: Since R is sufficiently large such that

Original: [Equation]
Revised: [Equation]

Original: we have that 4(1 − (1 − δ ) R ) ≥ 3 and hence
Revised: we have that 4(1 − (1 − δ ) R ) ≥ 3 and hence

Original: [Equation]
Revised: [Equation]

Original: With Lemma
Revised: With Lemma

Original: [Equation]
Revised: [Equation]

Original: we easily reach its ergodic version:
Revised: we easily reach its ergodic version:

Original: Lemma 9 (E RGODIC V ANISHING E RROR ) .
Revised: Lemma 9 (E RGODIC V ANISHING E RROR ) .

Original: Suppose θ ≜ 4(1 − δ ) R < 1 .
Revised: Suppose θ ≜ 4(1 − δ ) R < 1 .

Original: Then it holds that for any K ≥ 0 ,
Revised: Then it holds that for any K ≥ 0 ,

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Let θ = 4(1 − δ ) R
Revised: Let θ = 4(1 − δ ) R

Original: [Equation]
Revised: [Equation]

Original: then by Lemma 8 and noting Ψ ( − 1) = 0 , we have
Revised: then by Lemma 8 and noting Ψ ( − 1) = 0 , we have

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Therefore, by taking the summation of (40) over k = 0 , . . .
Revised: Therefore, by taking the summation of (40) over k = 0 , . . .

Original: [Equation]
Revised: [Equation]

Original: K − 1 and using Ψ ( − 1) = 0 again
Revised: K − 1 and using Ψ ( − 1) = 0 again

Original: [Equation]
Revised: [Equation]

Original: we further have
Revised: we further have

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: where we change the summation order of indexes k and ℓ in the last identity.
Revised: where we change the summation order of indexes k and ℓ in the last identity.

Original: Since (cid:80) K − 1 k = ℓ θ k +1 − ℓ = θ
Revised: Since (cid:80) K − 1 k = ℓ θ k +1 − ℓ = θ

Original: [Equation]
Revised: [Equation]

Original: 1 − θ K
Revised: 1 − θ K

Original: [Equation]
Revised: [Equation]

Original: ℓ ) 1 − θ ≤ θ 1 − θ , we thus have
Revised: ℓ ) 1 − θ ≤ θ 1 − θ , we thus have

Original: [Equation]
Revised: [Equation]

Original: Given the above lemmas, now we prove the convergence rate of NEOLITHIC.
Revised: Given the above lemmas, now we prove the convergence rate of NEOLITHIC.

Original: Theorem 4.
Revised: Theorem 4.

Original: Let the communication round be R = (cid:108) ln( n/δ )+ln(4max
Revised: Let the communication round be R = (cid:108) ln( n/δ )+ln(4max

Original: [Equation]
Revised: [Equation]

Original: δ (cid:109) and learning rate be as in (46) .
Revised: δ (cid:109) and learning rate be as in (46) .

Original: Then it holds that for any K ≥ 0 ,
Revised: Then it holds that for any K ≥ 0 ,

Original: [Equation]
Revised: [Equation]

Original: where T = KR is the total number of gradient queries (compressed communications) on each worker.
Revised: where T = KR is the total number of gradient queries (compressed communications) on each worker.

Original: Proof.
Revised: Proof.

Original: Averaging (29) over k = 0 , . . .
Revised: Averaging (29) over k = 0 , . . .

Original: , K , and using the fact that y (0) =
Revised: , K , and using the fact that y (0) =

Original: x (0) and f ( y ( K +1) ) ≥ f ⋆ , we have
Revised: x (0) and f ( y ( K +1) ) ≥ f ⋆ , we have

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: By the definition of Ω ( k − 1) and using the Cauhy-Schwarz inequality, it holds that
Revised: By the definition of Ω ( k − 1) and using the Cauhy-Schwarz inequality, it holds that

Original: [Equation]
Revised: [Equation]

Original: E
Revised: E

Original: [Equation]
Revised: [Equation]

Original: Therefore, by Lemma
Revised: Therefore, by Lemma

Original: [Equation]
Revised: [Equation]

Original: we reach
Revised: we reach

Original: K +
Revised: K +

Original: [Equation]
Revised: [Equation]

Original: ≤ 20 θ
Revised: ≤ 20 θ

Original: [Equation]
Revised: [Equation]

Original: Plugging (42) into (41), we reach that
Revised: Plugging (42) into (41), we reach that

Original: [Equation]
Revised: [Equation]

Original: Assume the learning rate γ is sufficiently small such that
Revised: Assume the learning rate γ is sufficiently small such that

Original: [Equation]
Revised: [Equation]

Original: then we can further bound (43) as
Revised: then we can further bound (43) as

Original: [Equation]
Revised: [Equation]

Original: where we use the definition ∆ :
Revised: where we use the definition ∆ :

Original: = f ( x (0) )
Revised: = f ( x (0) )

Original: − f ⋆ .
Revised: − f ⋆ .

Original: By choosing γ =
Revised: By choosing γ =

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: where we bound f ( x (0) )
Revised: where we bound f ( x (0) )

Original: − f ⋆ by ∆ in the last equation.
Revised: − f ⋆ by ∆ in the last equation.

Original: Let
Revised: Let

Original: [Equation]
Revised: [Equation]

Original: then it holds that
Revised: then it holds that

Original: [Equation]
Revised: [Equation]

Original: and hence 1 − θ = Ω(1) and γ satisfies (44).
Revised: and hence 1 − θ = Ω(1) and γ satisfies (44).

Original: Plugging (48) and (49) into (47)
Revised: Plugging (48) and (49) into (47)

Original: [Equation]
Revised: [Equation]

Original: and applying the notation T = KR
Revised: and applying the notation T = KR

Original: [Equation]
Revised: [Equation]

Original: we reach
Revised: we reach

Original: [Equation]
Revised: [Equation]

Original: C Experiment
Revised: C Experiment

Original: C.1 Synthetic Dataset
Revised: C.1 Synthetic Dataset

Original: Linear regression.
Revised: Linear regression.

Original: We consider the following least-square problem:
Revised: We consider the following least-square problem:

Original: [Equation]
Revised: [Equation]

Original: Coefficient matrix A i and measurement b i are associated with node i , and M is the size of local data.
Revised: Coefficient matrix A i and measurement b i are associated with node i , and M is the size of local data.

Original: We set d = 30 , n = 32 and M = 1000 , and generate data by letting each node i be associated with a local solution x ⋆i randomly generated by N (0 , I d ) .
Revised: We set d = 30 , n = 32 and M = 1000 , and generate data by letting each node i be associated with a local solution x ⋆i randomly generated by N (0 , I d ) .

Original: Then we generate each element in A i following standard normal distribution, and measurement b i is generated by b i
Revised: Then we generate each element in A i following standard normal distribution, and measurement b i is generated by b i

Original: =
Revised: =

Original: A i x ⋆i
Revised: A i x ⋆i

Original: + s i with white noise s i ∼ N (0 , 0 . 01) .
Revised: + s i with white noise s i ∼ N (0 , 0 . 01) .

Original: At each query, every node will randomly sample a row in A i and the corresponding element in b i to evaluate the stochastic gradient.
Revised: At each query, every node will randomly sample a row in A i and the corresponding element in b i to evaluate the stochastic gradient.

Original: We adopt the rand-1 compressor and set the number of rounds R = 4 for NEOLITHIC.
Revised: We adopt the rand-1 compressor and set the number of rounds R = 4 for NEOLITHIC.

Original: We use stair-wise decaying learning rates in which the learning rates are divided by every 2 , 500 communication rounds.
Revised: We use stair-wise decaying learning rates in which the learning rates are divided by every 2 , 500 communication rounds.

Original: Each algorithm is averaged with 20 trials.
Revised: Each algorithm is averaged with 20 trials.

Original: The result is shown in Figure 1 (left).
Revised: The result is shown in Figure 1 (left).

Original: It is observed that NEOLITHIC outperforms MEM-SGD and
Revised: It is observed that NEOLITHIC outperforms MEM-SGD and

Original: Double-Squeeze in convergence rate, and it performs closely to P-SGD.
Revised: Double-Squeeze in convergence rate, and it performs closely to P-SGD.

Original: Logistic regression.
Revised: Logistic regression.

Original: We consider the following logistic regression problem:
Revised: We consider the following logistic regression problem:

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: where { h i,m , y i,m } Mm =1 is the training dateset held by node i in which h i,m ∈ R d with d = 30 is a feature vector while y i,m ∈ {− 1 , +1 } is the corresponding label.
Revised: where { h i,m , y i,m } Mm =1 is the training dateset held by node i in which h i,m ∈ R d with d = 30 is a feature vector while y i,m ∈ {− 1 , +1 } is the corresponding label.

Original: Similar to the least square, each node i is associated with a local solution x ⋆i .
Revised: Similar to the least square, each node i is associated with a local solution x ⋆i .

Original: We generate each feature vector h i,m ∼ N (0 , I d ) , and label y i,m = 1 with probability 1 / (1 + exp( − y i,m h ⊤ i,m x ⋆i )) ; otherwise y i,m = − 1 .
Revised: We generate each feature vector h i,m ∼ N (0 , I d ) , and label y i,m = 1 with probability 1 / (1 + exp( − y i,m h ⊤ i,m x ⋆i )) ; otherwise y i,m = − 1 .

Original: We adopt the rand-1 compressor and set the number of rounds R = 4 for NEOLITHIC.
Revised: We adopt the rand-1 compressor and set the number of rounds R = 4 for NEOLITHIC.

Original: We use stair-wise decaying learning rates in which the learning rates are divided by every 800 communication rounds.
Revised: We use stair-wise decaying learning rates in which the learning rates are divided by every 800 communication rounds.

Original: Each algorithm is averaged with 20 trials.
Revised: Each algorithm is averaged with 20 trials.

Original: The result is shown in Figure 2.
Revised: The result is shown in Figure 2.

Original: It is observed that NEOLITHIC outperforms MEM-SGD and Double-Squeeze in convergence rate, and it performs closely to P-SGD.
Revised: It is observed that NEOLITHIC outperforms MEM-SGD and Double-Squeeze in convergence rate, and it performs closely to P-SGD.

Original: [Figure]
Revised: [Figure]

Original: C.2 Deep Learning Tasks
Revised: C.2 Deep Learning Tasks

Original: Implementation details.
Revised: Implementation details.

Original: We implement all compression algorithms with PyTorch[45] 1.8.2 using
Revised: We implement all compression algorithms with PyTorch[46] 1.8.2 using

Original: NCCL 2.8.3 (CUDA 10.1) as the communication backend.
Revised: NCCL 2.8.3 (CUDA 10.1) as the communication backend.

Original: For PSGD, we used PyTorch’s native
Revised: For PSGD, we used PyTorch’s native

Original: Distributed Data Parallel (DDP) module.
Revised: Distributed Data Parallel (DDP) module.

Original: All deep learning training scripts in this section run on a server with 8 NVIDIA V100 GPUs in our cluster and each GPU is treated as one worker.
Revised: All deep learning training scripts in this section run on a server with 8 NVIDIA V100 GPUs in our cluster and each GPU is treated as one worker.

Original: Image classification.
Revised: Image classification.

Original: We investigate the performance of the aforementioned methods with CIFAR- [35] dataset.
Revised: We investigate the performance of the aforementioned methods with CIFAR- [36] dataset.

Original: For CIFAR-10 dataset, it consists of 50,000 training images and 10,000 validation images categorized in 10 classes.
Revised: For CIFAR-10 dataset, it consists of 50,000 training images and 10,000 validation images categorized in 10 classes.

Original: We utilize two common variants of ResNet [26] model on CIFAR10 (ResNet-20 with roughly 0.27M parameters and ResNet-18 with 11.17M parameters).
Revised: We utilize two common variants of ResNet [26] model on CIFAR10 (ResNet-20 with roughly 0.27M parameters and ResNet-18 with 11.17M parameters).

Original: We train
Revised: We train

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: total 300 epochs and set the batch size to 128 on every worker.
Revised: total 300 epochs and set the batch size to 128 on every worker.

Original: The learning rate is set to 5e-3 for single worker and warmed up in the first 5 epochs and decayed by a factor of 10 at 150 and 250-th epoch.
Revised: The learning rate is set to 5e-3 for single worker and warmed up in the first 5 epochs and decayed by a factor of 10 at 150 and 250-th epoch.

Original: All experiments were repeated three times with different seeds.
Revised: All experiments were repeated three times with different seeds.

Original: For NEOLITHIC, we set
Revised: For NEOLITHIC, we set

Original: R = 2 .
Revised: R = 2 .

Original: Following previous works [60], we use top-k compressor with different compression ratio to evaluate the performance of the aforementioned methods.
Revised: Following previous works [61], we use top-k compressor with different compression ratio to evaluate the performance of the aforementioned methods.

Original: As shown in Figure 1 (left), Figure 3, and
Revised: As shown in Figure 1 (left), Figure 3, and

Original: Table 2, NEOLITHIC consistently outperforms other compression methods and reach the similar performance to PSGD.
Revised: Table 2, NEOLITHIC consistently outperforms other compression methods and reach the similar performance to PSGD.

Original: It is worth noting that EF21-SGD, while guaranteed to converge with milder assumptions than ours, does not provide competitive performance in deep learning tasks listed in
Revised: It is worth noting that EF21-SGD, while guaranteed to converge with milder assumptions than ours, does not provide competitive performance in deep learning tasks listed in

Original: Tables 2,
Revised: Tables 2,

Original: [Table]
Revised: [Table]

Original: and 4.
Revised: and 4.

Original: We find our reported result for EF21-SGD is consistent with Fig.
Revised: We find our reported result for EF21-SGD is consistent with Fig.

Original: 7 (the left plot) in [23].
Revised: 7 (the left plot) in [23].

Original: Performance with heterogeneous data.
Revised: Performance with heterogeneous data.

Original: We simulate data heterogeneity among nodes via a Dirichlet distribution-based partitioning with parameter α controlling the data heterogeneity.
Revised: We simulate data heterogeneity among nodes via a Dirichlet distribution-based partitioning with parameter α controlling the data heterogeneity.

Original: The training data for a particular class tends to concentrate in a single node as α → 0 , i.e. becoming more heterogeneous, while the homogeneous data distribution is achieved as α → ∞ .
Revised: The training data for a particular class tends to concentrate in a single node as α → 0 , i.e. becoming more heterogeneous, while the homogeneous data distribution is achieved as α → ∞ .

Original: We test α = 1 and 10 in Table 4 for all compared methods as corresponding to a setting with high/low heterogeneity.
Revised: We test α = 1 and 10 in Table 4 for all compared methods as corresponding to a setting with high/low heterogeneity.

Original: Effects of accumulation rounds.
Revised: Effects of accumulation rounds.

Original: We also empirically evaluate the performance of NEOLITHIC with different choice of parameter R in deep learning tasks.
Revised: We also empirically evaluate the performance of NEOLITHIC with different choice of parameter R in deep learning tasks.

Original: NEOLITHIC have slightly performance degradation in both compression scenarios as R scales up.
Revised: NEOLITHIC have slightly performance degradation in both compression scenarios as R scales up.

Original: We conjecture that the gradient accumulation step, which amounts to using large-batch samples in gradient evaluation, can help in the optimization and training stage as proved in this paper, but it may hurt the generalization performance.
Revised: We conjecture that the gradient accumulation step, which amounts to using large-batch samples in gradient evaluation, can help in the optimization and training stage as proved in this paper, but it may hurt the generalization performance.

Original: We recommend using NEOLITHIC in applications that are friendly to large-batch training.
Revised: We recommend using NEOLITHIC in applications that are friendly to large-batch training.

Original: [Table]
Revised: [Table]

Original: D More Details of Table
Revised: D More Details of Table

Original: There exist mismatches between some rates listed in Table 1 and those established in literature.
Revised: There exist mismatches between some rates listed in Table 1 and those established in literature.

Original: The mismatches exist because
Revised: The mismatches exist because

Original: we have strengthened the vanilla rates by relaxing their restrictive assumptions (say, DoubleSqueeze) or uncovering the hidden terms (say, CSER);
Revised: we have strengthened the vanilla rates by relaxing their restrictive assumptions (say, DoubleSqueeze) or uncovering the hidden terms (say, CSER);

Original: we have extended
Revised: we have extended

Original: the
Revised: the

Original: vanilla rates 
Revised: vanilla rates to

Original: to the same
Revised: the same

Original: setting
Revised: setting

Original: as NEOLITHIC (say, extend MEM-
Revised: as NEOLITHIC (say, extend MEM-

Original: SGD
Revised: SGD

Original: to non-
Revised: to non-

Original: convex and smooth setting, or transform QSGD to the distributed setting).
Revised: convex and smooth setting, or transform QSGD to the distributed setting).

Original: With these modifications, these baseline algorithms can be compared with NEOLITHIC in a fair manner.
Revised: With these modifications, these baseline algorithms can be compared with NEOLITHIC in a fair manner.

Original: Next we clarify each modification one by one.
Revised: Next we clarify each modification one by one.

Original: D.1 Q-SGD
Revised: D.1 Q-SGD

Original: There is a slight inconsistency between the original rate in [30] and the one listed Table 1 due to the following reasons:
Revised: There is a slight inconsistency between the original rate in [31] and the one listed Table 1 due to the following reasons:

Original: We noticed that the rate stated in [30, Corollary 3] is not optimal
Revised: We noticed that the rate stated in [31, Corollary 3] is not optimal

Original: following
Revised: following

Original: [30, Theorem 2] since the authors set the learning rate as θ (cid:112) M/T where θ is a universal constant.
Revised: [31, Theorem 2] since the authors set the learning rate as θ (cid:112) M/T where θ is a universal constant.

Original: The rate in [30, Corollary 3] can be slightly improved in terms of σ 2 , b 2 , ∆ = f
Revised: The rate in [31, Corollary 3] can be slightly improved in terms of σ 2 , b 2 , ∆ = f

Original: ( x (0) ) − min x f ( x ) by involving them into the learning rate.
Revised: ( x (0) ) − min x f ( x ) by involving them into the learning rate.

Original: We manually optimize the learning rate by choosing
Revised: We manually optimize the learning rate by choosing

Original: [Equation]
Revised: [Equation]

Original: M is the total mini-batch size on all workers in Q-SGD (see the paragraph of contribution, page 2, [30]).
Revised: M is the total mini-batch size on all workers in Q-SGD (see the paragraph of contribution, page 2, [31]).

Original: For a fair comparison with other methods in Table 1, We set M = n to make the number of total gradient queries per iteration equivalent in all algorithms.
Revised: For a fair comparison with other methods in Table 1, We set M = n to make the number of total gradient queries per iteration equivalent in all algorithms.

Original: D.2 CSER
Revised: D.2 CSER

Original: While [66, Corollary 1] does not have a O (1 /T ) term in the convergence rate, it does impose another condition T ≫ n to the convergence statement.
Revised: While [67, Corollary 1] does not have a O (1 /T ) term in the convergence rate, it does impose another condition T ≫ n to the convergence statement.

Original: If T ≫ n holds, then 1 / √ nT ≫ 1 /T and hence the 1 /T term is dominated by 1 / √ nT and thus hidden in the notation O ( · ) .
Revised: If T ≫ n holds, then 1 / √ nT ≫ 1 /T and hence the 1 /T term is dominated by 1 / √ nT and thus hidden in the notation O ( · ) .

Original: However, the condition does not appear in the convergence theorem of NEOLITHIC.
Revised: However, the condition does not appear in the convergence theorem of NEOLITHIC.

Original: To conduct a fair comparison, we have to remove condition T ≫ n from its convergence theorem, which thus incurs additional O (1 /T ) term in the rate accordingly.
Revised: To conduct a fair comparison, we have to remove condition T ≫ n from its convergence theorem, which thus incurs additional O (1 /T ) term in the rate accordingly.

Original: The rate we provided in Table 1 is hence more precise than that in [66].
Revised: The rate we provided in Table 1 is hence more precise than that in [67].

Original: D.3 Double-Squeeze
Revised: D.3 Double-Squeeze

Original: The original rate of Double-Squeeze established in [60] is based on an unrealistic assumption that accumulated compression errors are bounded by unknown ϵ (see [60, Assumption 1.3]).
Revised: The original rate of Double-Squeeze established in [61] is based on an unrealistic assumption that accumulated compression errors are bounded by unknown ϵ (see [61, Assumption 1.3]).

Original: This makes its rate incomparable with other methods.
Revised: This makes its rate incomparable with other methods.

Original: However, we can remove the unrealistic assumption and easily derive a comparable bound where ϵ can be explicitly replaced with O ( G/δ 2 ) .
Revised: However, we can remove the unrealistic assumption and easily derive a comparable bound where ϵ can be explicitly replaced with O ( G/δ 2 ) .

Original: We plug
Revised: We plug

Original: O ( G/δ 2 ) into [60, Corollary 2] to get the rate listed in our Table 1.
Revised: O ( G/δ 2 ) into [61, Corollary 2] to get the rate listed in our Table 1.

Original: To this end, we follow the notations of [60] to derive explicit upper bounds for server/worker compression errors E [ ∥ δ t ∥ ] and E [ ∥ δ ( i ) t ∥ ] which, combined with [60, Corollary 2], leads to the rate in our Table 1.
Revised: To this end, we follow the notations of [61] to derive explicit upper bounds for server/worker compression errors E [ ∥ δ t ∥ ] and E [ ∥ δ ( i ) t ∥ ] which, combined with [61, Corollary 2], leads to the rate in our Table 1.

Original: We consider compressors Q (in server) and Q i (in worker i ) utilized are δ -contractive.
Revised: We consider compressors Q (in server) and Q i (in worker i ) utilized are δ -contractive.

Original: We use g ( i ) t to indicate local (stochastic) gradient.
Revised: We use g ( i ) t to indicate local (stochastic) gradient.

Original: Bound of local compression error: E [ ∥ δ ( i ) t ∥ ]
Revised: Bound of local compression error: E [ ∥ δ ( i ) t ∥ ]

Original: = O ( G/δ ) .
Revised: = O ( G/δ ) .

Original: By δ -contraction and Young’s inequality, we have for any ρ > 0 that
Revised: By δ -contraction and Young’s inequality, we have for any ρ > 0 that

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Iterating (51) for t, t − 1 , . .
Revised: Iterating (51) for t, t − 1 , . .

Original: and noting δ ( i ) 

Revised: and noting δ ( i )

Original: = 0 , we reach
Revised: = 0 , we reach

Original: [Equation]
Revised: [Equation]

Original: where the last inequality holds because E [ ∥ g ( i ) s ∥ 2 ] ≤ G 2 for all 1 ≤ s ≤ t .
Revised: where the last inequality holds because E [ ∥ g ( i ) s ∥ 2 ] ≤ G 2 for all 1 ≤ s ≤ t .

Original: Here one must choose ρ = Ω(1 /δ ) to avoid the explosion of the upper bound, which leads to E [ ∥ δ ( i ) t ∥ 2 ]
Revised: Here one must choose ρ = Ω(1 /δ ) to avoid the explosion of the upper bound, which leads to E [ ∥ δ ( i ) t ∥ 2 ]

Original: =
Revised: =

Original: O ( G 2 /δ 2 ) .
Revised: O ( G 2 /δ 2 ) .

Original: Therefore, by Jessen’s inequality, we have E [ ∥ δ ( i ) t ∥ ] ≤ (cid:113) E [ ∥ δ ( i ) t ∥ 2 ]
Revised: Therefore, by Jessen’s inequality, we have E [ ∥ δ ( i ) t ∥ ] ≤ (cid:113) E [ ∥ δ ( i ) t ∥ 2 ]

Original: =
Revised: =

Original: O ( G/δ ) .
Revised: O ( G/δ ) .

Original: Bound of global compression error: E [ ∥ δ t ∥ ]
Revised: Bound of global compression error: E [ ∥ δ t ∥ ]

Original: = O ( G/δ 2 ) .
Revised: = O ( G/δ 2 ) .

Original: By δ -contraction and Young’s inequality, we have for any ρ > 0 that
Revised: By δ -contraction and Young’s inequality, we have for any ρ > 0 that

Original: [Equation]
Revised: [Equation]

Original: Again by Young’s inequality and δ -contraction, we have
Revised: Again by Young’s inequality and δ -contraction, we have

Original: E
Revised: E

Original: [Equation]
Revised: [Equation]

Original: (cid:13)(cid:13)(cid:13)(cid:13)
Revised: (cid:13)(cid:13)(cid:13)(cid:13)

Original: [Equation]
Revised: [Equation]

Original: (cid:13) (cid:13)(cid:13)(cid:13)(cid:13) 1 n n (cid:88)
Revised: (cid:13) (cid:13)(cid:13)(cid:13)(cid:13) 1 n n (cid:88)

Original: i =
Revised: i =

Original: Q i ( v ( i ) t ) − v ( i ) t (cid:13) (cid:13)(cid:13)(cid:13)(cid:13) 2   + 2 E   (cid:13) (cid:13)(cid:13)(cid:13)(cid:13) 1 n n (cid:88) i
Revised: Q i ( v ( i ) t ) − v ( i ) t (cid:13) (cid:13)(cid:13)(cid:13)(cid:13) 2   + 2 E   (cid:13) (cid:13)(cid:13)(cid:13)(cid:13) 1 n n (cid:88) i

Original: = v ( i ) t (cid:13) (cid:13)(cid:13)(cid:13)(cid:13) 2  
Revised: = v ( i ) t (cid:13) (cid:13)(cid:13)(cid:13)(cid:13) 2  

Original: [Equation]
Revised: [Equation]

Original: n (cid:88) i =
Revised: n (cid:88) i =

Original: [Equation]
Revised: [Equation]

Original: (54) where the last identity is because the upper bound of E [ ∥ v ( i ) t ∥ 2 ] can be obtained by following the derivation in (51) and (52).
Revised: (54) where the last identity is because the upper bound of E [ ∥ v ( i ) t ∥ 2 ] can be obtained by following the derivation in (51) and (52).

Original: Taking ρ = 2(1 − δ ) /δ
Revised: Taking ρ = 2(1 − δ ) /δ

Original: = O
Revised: = O

Original: (1 /δ ) in (53) and using (54), we reach an inequality taking a form like
Revised: (1 /δ ) in (53) and using (54), we reach an inequality taking a form like

Original: [Equation]
Revised: [Equation]

Original: Iterating (55) similarly, we easily reach E [ ∥ δ t ∥ 2 ]
Revised: Iterating (55) similarly, we easily reach E [ ∥ δ t ∥ 2 ]

Original: =
Revised: =

Original: O ( G 2 /δ 4 ) and thus E [ ∥ δ t ∥ ]
Revised: O ( G 2 /δ 4 ) and thus E [ ∥ δ t ∥ ]

Original: = O ( G/δ 2 ) .
Revised: = O ( G/δ 2 ) .

Original: D.4 MEM-SGD
Revised: D.4 MEM-SGD

Original: We notice that only the rate for strongly convex problems is established in the original MEM-SGD paper [56].
Revised: We notice that only the rate for strongly convex problems is established in the original MEM-SGD paper [57].

Original: To compare it fairly with NEOLITHIC, we derive its convergence rate in the non-convex setting by ourselves.
Revised: To compare it fairly with NEOLITHIC, we derive its convergence rate in the non-convex setting by ourselves.

Original: The main recursion of MEM-SGD is
Revised: The main recursion of MEM-SGD is

Original: [Equation]
Revised: [Equation]

Original: Q i ( p it ) , e it +

Revised: Q i ( p it ) , e it +

Original: = p
Revised: = p

Original: it − Q i ( p it ) .
Revised: it − Q i ( p it ) .

Original: The key steps of our derivation are listed as follows.
Revised: The key steps of our derivation are listed as follows.

Original: Following the similar argument to (51) and (52), we can bound the compression error as E [ ∥ e it ∥ 2 ]
Revised: Following the similar argument to (51) and (52), we can bound the compression error as E [ ∥ e it ∥ 2 ]

Original: =
Revised: =

Original: O ( η 2 G 2 /δ 2 ) .
Revised: O ( η 2 G 2 /δ 2 ) .

Original: Note that here η 2 appears since compression is conducted after multiplying the learning rate η . 2.
Revised: Note that here η 2 appears since compression is conducted after multiplying the learning rate η . 2.

Original: The recursion formula of MEM-SGD is y t +

Revised: The recursion formula of MEM-SGD is y t +

Original: =
Revised: =

Original: y t − ηn (cid:80) ni
Revised: y t − ηn (cid:80) ni

Original: =1 ∇ f i ( x t , ξ it ) with y t ≜ x t − 1 n (cid:80) ni
Revised: =1 ∇ f i ( x t , ξ it ) with y t ≜ x t − 1 n (cid:80) ni

Original: =1 e it .
Revised: =1 e it .

Original: In fact, one can easily check that
Revised: In fact, one can easily check that

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: Following the derivation of (34), one can obtain
Revised: Following the derivation of (34), one can obtain

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Setting η ≤ 12 L such that η (1 − ηL )≥ η 4 and rearranging (56), we have
Revised: Setting η ≤ 12 L such that η (1 − ηL )≥ η 4 and rearranging (56), we have

Original: [Equation]
Revised: [Equation]

Original: By the definition of y
Revised: By the definition of y

Original: t
Revised: t

Original: and
Revised: and

Original: step
Revised: step

Original: , we have
Revised: , we have

Original: [Equation]
Revised: [Equation]

Original: Averaging (57)
Revised: Averaging (57)

Original: with
Revised: with

Original: 58) plugged into
Revised: 58) plugged into

Original: we reach
Revised: we reach

Original: [Equation]
Revised: [Equation]

Original: Setting the learning rate η = (2 L + ( Lσ 2 n ∆
Revised: Setting the learning rate η = (2 L + ( Lσ 2 n ∆

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: 2 + ( L
Revised: 2 + ( L

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: 2 ∆ )
Revised: 2 ∆ )

Original: [Equation]
Revised: [Equation]

Original: 3 ) − 1 in (59) leads to the rate we listed in Table 1.
Revised: 3 ) − 1 in (59) leads to the rate we listed in Table 1.

Original: D.5 Comparison with More Algorithms
Revised: D.5 Comparison with More Algorithms

Original: We supplement the comparison between Q-SGD [30], VR-MARINA [25], and DASHA-MVR [61] in Table 6.
Revised: We supplement the comparison between Q-SGD [31], VR-MARINA [25], and DASHA-MVR [62] in Table 6.

Original: The results are compared in terms of the communication/query complexity to reach E [ ∥∇ f ( x ) ∥ 2 ] ≤ ϵ for a sufficiently small ϵ .
Revised: The results are compared in terms of the communication/query complexity to reach E [ ∥∇ f ( x ) ∥ 2 ] ≤ ϵ for a sufficiently small ϵ .

Original: Several additional comments are as follows:
Revised: Several additional comments are as follows:

Original: All three algorithms utilize unidirectional, unbiased, and independent compressors.
Revised: All three algorithms utilize unidirectional, unbiased, and independent compressors.

Original: All algorithms conduct an imbalanced number of compressed communications and of gradient queries.
Revised: All algorithms conduct an imbalanced number of compressed communications and of gradient queries.

Original: We therefore list the communication and gradient query complexity separately in Table 6.
Revised: We therefore list the communication and gradient query complexity separately in Table 6.

Original: The result of Q-SGD is slightly tuned by us, see the argument in Appendix D.1.
Revised: The result of Q-SGD is slightly tuned by us, see the argument in Appendix D.1.

Original: MARINA (with O (1 /ϵ 2 ) ) and SASHA (with O (1 /ϵ 3 ) ) have better communication complexity than QSGD (with O (1 /ϵ 4 ) ) when ϵ is sufficiently small.
Revised: MARINA (with O (1 /ϵ 2 ) ) and SASHA (with O (1 /ϵ 3 ) ) have better communication complexity than QSGD (with O (1 /ϵ 4 ) ) when ϵ is sufficiently small.