Theoretically Better and Numerically Faster Distributed Optimization with Smoothness-Aware Quantization Techniques
Abstract
To address the high communication costs of distributed machine learning, a large body of work has been devoted in recent years to designing various compression strategies, such as sparsiﬁcation and quantization, and optimization algorithms capable of using them.
Recently, Safaryan et al.
[2021] pioneered a dramatically different compression design approach: they ﬁrst use the local training data to form local smoothness matrices and then propose to design a compressor capable of exploiting the smoothness information contained therein.
While this novel approach leads to substantial savings in communication, it is limited to sparsiﬁcation as it crucially depends on the linearity of the compression operator.
In this work, we generalize their smoothness-aware compression strategy to arbitrary unbiased compression operators, which also include sparsiﬁcation.
Specializing our results to stochastic quantization, we guarantee signiﬁcant savings in communication complexity compared to standard quantization.
In particular, we prove that block quantization with n blocks theoretically outperforms single block quantization, leading to a reduction in communication complexity by an O ( n ) factor, where n is the number of nodes in the distributed system.
Finally, we provide extensive numerical evidence with convex optimization problems that our smoothness-aware quantization strategies outperform existing quantization schemes as well as the aforementioned smoothness-aware sparsiﬁcation strategies with respect to three evaluation metrics: the number of iterations, the total amount of bits communicated, and wall-clock time.
Introduction
Training modern machine learning models is typically cast in terms of (regularized) empirical risk minimization problem and requires increasingly more training data to make empirical risk closer to the true risk [Schmidhuber, 2015, Vaswani et al., 2019].
This natural requirement makes it harder (and in some scenarios impossible) to collect all data in one place and carry out the training using a single data source.
As a result, we reconciled with a ﬂock of datasets disseminated across various compute nodes holding the actual training data [Bekkerman et al., 2011, Vogels et al., 2019].
However, such divide-and-conquer approach of handling vast amount of data means that local updates need to be communicated among the nodes (or through some central server orchestrating the process), which often forms the main bottleneck in modern distributed systems [Zhang et al., 2017, Lin et al., 2018].
This issue is further exacerbated by the fact that modern highly performing models are typically overparameterized [Brown et al., 2020, Narayanan et al., 2021].
Distributed training.
In this paper, we consider distributed training formalized as the following optimization problem
[Equation]
and where d is the number of parameters of model x ∈ R d to be trained, n is the number of machines/nodes participating in the training, f i ( x ) is the loss/risk associated with the data stored on machine i ∈ [ n ] def = { 1 , 2 , . . . , n } , f ( x ) is the empirical loss/risk, and R ( x ) is a regularizer.
Because of the communication constraints, large body of work has been devoted in recent years to the design of various compression strategies, such as sparsiﬁcation [Koneˇcný and Richtárik, 2018,
Wangni et al., 2018, Alistarh et al., 2018], quantization [Goodall, 1951, Roberts, 1962, Alistarh et al., 2017], low-rank approximation [Vogels et al., 2019], three point compressor [Richtárik et al., 2022], and optimization algorithms capable of using them, such as Distributed Compressed Gradient
Descent (DCGD)
[Khirirat et al., 2018], QSGD
[Alistarh et al., 2017, Faghri et al., 2020], NUQSGD
[Ramezani-Kebrya et al., 2021], DIANA [Mishchenko et al., 2019, Horváth et al., 2019], PowerSGD [Vogels et al., 2019], signSGD
[Bernstein et al., 2018, Safaryan and Richtárik, 2021], intSGD [Mishchenko et al., 2021], ADIANA [Li et al., 2020], MARINA [Gorbunov et al., 2021].
From scalar smoothness to matrix smoothness.
Typically, distributed optimization algorithms in the literature that employ compressed communication, including all methods from the aforementioned works, use only shallow smoothness information of the loss function such as scalar
L -smoothness [Nesterov, 2004].
Deﬁnition 1 (Scalar Smoothness) .
Differentiable function φ : R d → R is called L -smooth if there exists a non-negative scalar value L ≥
such that
[Equation]
As pointed out by Safaryan et al.
[2021], smoothness constant L reﬂects small part of the rich smoothness information often easily available through the training data.
In their recent work,
Safaryan et al.
[2021] pioneered a dramatically different compression design approach.
First, they propose to use the local training data to form local smoothness matrices , which they claim contain much more useful information than standard smoothness constants.
Deﬁnition 2 (Matrix Smoothness) .
Differentiable function φ : R d → R is called L -smooth if there exists a symmetric positive semideﬁnite matrix L (cid:23) 0 such that
[Equation]
Intuitively, the usefulness of L -smoothness over the standard L -smoothness is the tighter upper bound for functional growth.
In other words, if for a function φ we have the tightest scalar smoothness L and the tightest matrix smoothness L parameters, then L = λ max ( L ) and, hence, upper bound (3) is better than (2).
To understand the relationship deeper, consider the functional growth of φ along the direction
e ∈ R d (without loss of generality assume (cid:107) e (cid:107) = 1 ).
Let x = y + te , where t > 0 is a positive scaling parameter, and consider the quadratic terms t 2 2 L of (2) and t 2 2 (cid:104) e, L e (cid:105) of (3) bounded the functional growth.
Obviously, depending on the direction e , the quadratic form (cid:104) e, L e (cid:105) can be much smaller than L = λ max ( L )
= sup {(cid:104) e, L e (cid:105) : (cid:107) e
(cid:107) = 1 } .
Non-uniform functional growths over different directions hints to design optimization algorithms that are aware of such properties of the objective.
Several works on randomized coordinate descent have successfully exploited this approach [Richtárik and Takáˇc, 2016, Qu and Richtárik, 2016a,b,
Hanzely and Richtárik, 2019, Hanzely and Richtárik, 2019].
For example, the ‘NSync algorithm of Richtárik and Takáˇc [2016] uses the smoothness matrix to estimate smaller, so-called ESO (Expected
Separable Overapproximation) parameters for each coordinate, leading to larger stepsizes for the update rule and improved complexity for the algorithm.
Note that randomized coordinate descent can be viewed as compressed gradient descent with random sparsiﬁcation ( n = 1 number of workers).
In the context of distributed optimization, using smoothness matrices L i of all local loss functions f i ( x ) , i ∈ [ n ] , Safaryan et al.
[2021] design a compressor capable of exploiting the smoothness
[Table]
information contained within the smoothness matrices.
In particular, under certain heterogeneity conditions on the smoothness matrices L i , their new compressor reduces total communication cost by a factor of O (min( n, d ))
[Table]
While this novel approach leads to substantial savings in communication
[Table]
it is limited to random sparsiﬁcation as it crucially depends on the linearity of the compression operator.
It is not clear whether this approach can be useful in the design of other smoothness-aware compression techniques.
Summary of Contributions
Motivated by the above mentioned development, in this work, we made the following contributions.
Extending matrix-smoothness-aware sparsiﬁcation to general compression schemes.
First, we generalize the smoothness-aware sparsiﬁcation strategy [Safaryan et al., 2021] to arbitrary unbiased compressors.
Instead of sparsiﬁcation operator, we consider the generic class B d ( ω ) of (possibly randomized) unbiased compression operators C : R d → R d with bounded variance ω ≥ 0 , i.e.,
[Equation]
This class is quite broad including random sparsiﬁcation and various quantization schemes.
To beneﬁt from the matrix smoothness information with general compressor C , we propose the following modiﬁcation in the communication protocol.
If x ∈ R d is the vector to be communicated, instead of applying compressor C directly to x and sending C ( x ) , we compress it by C
( L † 1 / 2 x ) and decompress it by multiplying L 1 / 2 .
Overall, the receiver estimates the original x by L 1 / 2 C ( L † 1 / 2 x ) .
Distributed compressed methods with improved communication complexity.
To highlight the appropriateness of our generalization, we redesign two distributed compressed methods—DCGD [Khirirat et al., 2018] and DIANA [Mishchenko et al.
, 2019]—to effectively utilize both matrix smoothness information and general compression operators leading to new methods, which we call
DCGD+ (Algorithm 1) and DIANA+ (Algorithm 2).
The key notion we introduce that enables the technical analysis is the following quantity describing interaction between compression operator
C ∈ B d ( ω ) and smoothness matrix L (cid:23) 0 :
[Equation]
This quantity generalizes the one deﬁned in Safaryan et al.
[2021] for sparsiﬁcation, and provides means for tighter theoretical guarantees (Theorems 1 and 2) and better compression design.
Block quantization.
As we are no longer constrained to sparsiﬁcation to exploit matrix smoothness, we consider more aggressive quantization schemes to further reduce the communication cost.
Our ﬁrst extension of standard quantization [Alistarh et al., 2017] is block quantization , where each block is allowed to have a separate quantization parameter.
Notably, we show theoretically that our block quantization with n blocks outperforms single block quantization and saves in communication by a factor of O ( n ) for both DCGD+ (Theorem 3) and DIANA+ (Theorem 4) when n = O ( √ d ) .
Quantization with varying steps.
In our second extension of standard quantization, we go even further and allow all coordinates to have their own quantization steps.
This extension turns out to be more efﬁcient in practice than block quantization and provides savings in communication cost by a factor of O (min( n, d )) for both DCGD+ (Theorem 5) and DIANA+ (Theorem 6).
Experiments.
Finally, we perform extensive numerical experiments using LibSVM data [Chang and Lin, 2011] and provide clear numerical evidence that the proposed smoothness-aware quantization strategies outperform existing quantization schemes as well the aforementioned smoothness-aware sparsiﬁcation strategies with respect to three evaluation metrics: the number of iterations, the total amount of bits communicated, and wall-clock time (see Section 6 and the Appendix).
Smoothness-Aware Distributed Methods with General Compressors
In this section we extend methods DCGD+ and DIANA+ of Safaryan et al.
[2021] to handle arbitrary unbiased compression operators.
We consider the problem (1) with matrix smoothness assumption for all local losses f i ( x ) and with strong convexity of loss function f ( x ) .
Assumption 1 (Matrix smoothness) .
The functions f i : R d → R are differentiable, convex, lower bounded and L i -smooth.
Besides, f is L -smooth with the scalar smoothness constant L def = λ max ( L ) .
First, note that lower boundedness of f i ( x ) is not needed once L i (cid:31) 0 is invertible.
This part of the assumption is not a restriction in applications as all loss function are lower bounded.
Regarding the relation between L and L i , notice that (1) implies L (cid:22) 1 n (cid:80) ni
=1 L i .
This means that whilen (cid:80) ni
=1 L i can serve as a smoothness matrix for f , there might be a tighter estimate, which we denote by L .
Clearly, matrix smoothness provides much more information about the loss function than scalars smoothness.
However, estimating dense smoothness matrix L could be expensive for problems beyond generalized linear models because of the d 2 number of entries and lack of closed-form expression.
On the other hand, estimating sparse, such as diagonal, smoothness matrix
Diag ( L 1 , L 2 , . . .
, L d ) should be feasible.
Lastly, if L is a smoothness matrix (could be dense, diagonal or any structure) of f , then any matrix (cid:101) L (
cid:22) L is also a smoothness matrix for f .
This implies that our theory would still work if the smoothness matrix is over-approximated.
Assumption 2 ( µ -convexity) .
The function f : R d → R is µ -convex for some µ > 0 , i.e.,
[Equation]
This assumption is rather standard in the literature, sometimes referred to as strong convexity.
DCGD+ with arbitrary unbiased compression.
In our version of DCGD+, each node i ∈ [ n ] is allowed to control its own compression operator C
i ∈ B d ( ω ) independent of other nodes.
Denote
[Equation]
where L i def =
L ( C i , L i ) .
Furthermore, as the compressor C i can be random, denote by C ki a copy of C i generated at iteration k .
1: Input:
Similar to the standard DCGD method, convergence of DCGD+ is linear up to some oscillation neighborhood.
However, for the interpolation regime this neighborhood vanishes and the method converges linearly to the exact solution.
Theorem 1.
Let Assumptions 1 and 2 hold and assume that each node i ∈ [ n ] generates its own copy of compression operator C ki ∈ B d ( ω i ) independently from others.
Then, for the step-size 0 < γ ≤ 1 L + 2 n L max , the iterates { x k } of DCGD+ (Algorithm 1) satisfy
[Equation]
where σ ∗ + def = 1 n (cid:80) ni
=1 L i (cid:107)∇ f i ( x ∗ ) (cid:107) 2 L † i .
In particular, for the interpolation regime (i.e., ∇ f i ( x ∗ ) =
for all i ∈ [ n ] ), then DCGD+ converges linearly with iteration complexity
[Equation]
We show later that the iteration complexity (6) of DCGD+ can be much better than one of
DCGD.
However, the size of the neighborhood of DCGD+ might be bigger than of DCGD.
In case of standard (scalar) smoothness (i.e. L i =
L i I ) the size of the neighborhood would be σ ∗ def = 1 n (cid:80) n i =1 ω i
(cid:107)∇ f i ( x ∗ ) (cid:107) 2 , which might be smaller than σ ∗ + .
Even though we have
L i ≤ ω
i λ max ( L i ) from the deﬁnition of L i , it does not imply L i L † i (cid:22) ω
i I .
Thus, with matrixsmoothness-aware compression we ensure faster linear convergence at the cost of a possibly larger oscillation radius.
This is not an issue for the interpolation regime, which can interpolate the whole training data with zero loss.
Moreover, next we present an algorithmic solution to remove the neighborhood using the DIANA method.
DIANA+ with arbitrary unbiased compression.
The mechanism allowing to remove the neighborhood in DIANA+ is based on the DIANA method, which was initially introduced for ternary quantization by Mishchenko et al.
[2019], and then extended to arbitrary unbiased compression operators by Horváth et al.
The high level idea is to learn the local optimal gradients ∇ f i ( x ∗ ) by estimates u ki for all nodes i ∈
[ n ] in a communication efﬁcient manner.
Nodes use these estimates u ki to progressively construct better local gradient estimates g ki reducing the variance induced from the compression.
end
We prove in the Appendix that both iterates x k and all local gradient estimates u ki converge linearly to the exact solution x ∗ and ∇ f i ( x ∗ ) respectively.
Theorem 2.
Let Assumptions 1 and 2 hold and assume that each node i ∈ [ n ] generates its own copy of compression operator C ki ∈ B d ( ω i ) independently from others.
Then, if ω max = max 1 ≤ i ≤ n ω i and the step-size γ =
[Equation]
, DIANA+ (Algorithm 2) converges linearly with iteration complexity
[Equation]
Notice that the cost of removing the neighborhood is the extra O ( ω max log 1 ε ) iterations, which is negligible in the overall complexity (7) above.
Another interesting observation is the second order ﬂavor of the gradient learning technique employed by DIANA+.
Let, for concreteness, matrices L i be invertible and C ki ( − x ) =
−C ki ( x ) for all x ∈ R d (both random sparsiﬁcation and quantization satisfy this).
Typically, the learning procedure of the original DIANA method, u k +1 i =
u ki −
[Equation]
C
[Equation]
u
[Equation]
∇ f i ( x k )) , can be interpreted as a single step of CGD applied to the problem of minimizing the convex quadratic function ϕ ki ( u ) def = 12 (cid:13)(cid:13)
u − ∇ f i ( x k ) (cid:13)(cid:13) 2 , which changes in each iteration because the gradient changes.
In contrast, we observe that the learning mechanism of DIANA+ can be interpreted as a single step of a (damped) Newton’s method with compressed gradients and with the true Hessian.
Indeed, ﬁx the iteration counter k and denote ϕ ki ( u ) def = 12 (cid:13)(cid:13)
u − ∇ f i ( x k ) (cid:13)(cid:13) 2 L − 1 / 2 i .
Then, the update rule of shifts u ki in DIANA+ can be rewritten as u k +
i
[Equation]
∇ f i ( x k )))
=
u ki − α (cid:2) ∇ 2 ϕ
i ( u ki ) (cid:3) − 1 C ki ( ∇ ϕ
i ( u ki )) .
This might serve as an extra explanation on why incorporating smoothness matrices properly can improve the performance of ﬁrst order methods with communication compression.
Baselines for the original methods.
To make the theoretical comparison against DCGD and
DIANA more transparent, we ﬁx the following baselines using the standard quantization scheme.
• Baseline for DIANA.
The iteration complexity of DIANA is T = (cid:101)
O ( ω + L max µ + ωL max nµ ) .
When applying standard quantization from [Alistarh et al., 2017], the amount of bits each node communicates is b
= O ( s 2 + s √ d ) =
max( s 2 , s √ d )
=
O ( dω ) since ω = min( ds 2 , √ ds ) =
d max( s 2 ,
s √ d
) (Lemma 3.1., Alistarh et al.
Thus, the total communication complexity of DIANA is n · T · b = (cid:101) O ( nd + ndL max ωµ + dL max
µ
Thus, the optimal total communication complexity of DIANA is (cid:101) O ( nd + dL max µ ) , which is attained when ω
=
O ( n ) .
• Baseline for DCGD.
Based on the iteration complexity 2 (cid:101) O
( Lµ + ωL max nµ ) of DCGD (in case ∇ f i ( x ∗ ) =
for all i ∈ [ n ] ), we ﬁx the same level of compression ω
= O ( n ) , which results in (cid:101)
O
( L max µ ) iterations complexity.
From the estimate of quantization variance ω = min (cid:16) ds 2 , √ ds (cid:17) we conclude that s
=
O ( √ dn ) should be used.
Finally, with this choice of s , each node communicates O ( s 2 + s √ d ) =
O ( dn ) amount of bits.
Thus, total communication complexity (i.e. how many bits ﬂows through the central server) of DCGD is (cid:101) O ( dL max µ ) .
To compare the proposed methods with these baselines and highlight improvement factors, deﬁne parameters ν
and ν 1 describing local smoothness matrices
L i as follows
[Equation]
where L i = λ max ( L i ) , L max def = max 1 ≤ i ≤ n L
i
and
L i ; j is the j th diagonal element of matrix
L i .
Parameters ν ∈ [1 , n ] and ν 1 ∈ [1 , d ] describe the level of heterogeneity over the nodes and coordinates respectively.
If L i matrices coincide, then ν = n and ν 1 = d .
On the other extreme, when the values of L i are extremely non-uniform, we have ν (cid:28) n and ν 1 (cid:28) d .
Notice that the quantity L max µn in (6) and the quantity ω max + L max µn in (7) depend on compression operators C ki applied by the nodes.
For the rest of the paper we are going to minimize these quantities with respect to the choice of C ki in such a way to minimize total communication complexity of the proposed distributed methods.
We specialize compressors C i to two different extensions of standard quantization and optimize with respect to compression parameters.
Block Quantization
We now present our ﬁrst extension to standard quantization in order to properly capture the matrix smoothness information.
Instead of having a single quantization parameter (e.g. number of levels) for all coordinates, here we divide the space R d into B ∈ { 1 , 2 , . . .
, d } blocks as R d = R d 1 × R d 2 × · · · × R d B and for each subspace R d l , l ∈
[ B ] we apply standard quantization independently from other blocks with different number of levels s l .
Thus, for any l ∈ [ B ] we allocate one parameter s l for l th block of x ∈ R d .
Hence quantization is applied block-wise: for each block we send the norm (cid:107) x l (cid:107) of the block x l ∈ R d l and all entries within this block are quantized with levels { 0 , 1 s l , 2 s l , . . .
In the special case of B = 1 , we get the standard quantization of Alistarh et al.
To get rid of the constraints on s l to be integers, instead of working with the number of levels s l , we introduce the size of the quantization step h l = 1 s l and allow them to take any positive values (even bigger than 1 ).
Thus, for each block l ∈ [ B ] we quantize with respect to levels { 0 , h l , 2 h l , . . . } .
Deﬁnition 3 (Block Quantization) .
For a given number of blocks B ∈ [ d ] and ﬁxed quantization steps h
= ( h 1 , . .
. , h B ) , deﬁne block-wise quantization operator Q
Bh : R d → R d as follows:
[Equation]
where t =
( l − 1) B + j, x ∈ R d , j ∈
[ d l ] , l ∈ [ B ] and ξ l ( v ) for v ≥ 0 is deﬁned via the quantization levels { 0 , h l , 2 h l , . . .
} as follows: if kh l ≤ v < ( k + 1) h l for some k ∈ { 0 , 1 , 2 , . . .
} , then ξ
[Equation] [Equation]
with
[Equation]
Note that Q Bh is an unbiased compression operator as E [ ξ j ( v )]
= v for any v ≥ 0 .
To communicate a vector of the form Q Bh ( x ) , we encode each block (cid:2)
Q Bh ( x ) (cid:3) l ∈ R d l using Elias ω -coding as in the standard quantization scheme [Alistarh et al., 2017].
Hence, for each block l ∈ [ B ] we need to send (cid:101) O
( 1 h 2 l + √ d l h l ) bits and one ﬂoating point number for (cid:107) x l (cid:107) .
Overall, the number of encoding bits for Q Bh ( x ) (up to constant and log factors) can be given by (cid:80)
Bl =1 ( 1 h 2 l + √ d l h l ) + B .
As for the compression noise, we prove in the Appendix the following upper bound for L ( Q Bh , L ) :
[Equation]
where L ll is the l th diagonal block matrix of L with sizes d l × d l .
Next, we are going to minimize communication complexity of DCGD+ and DIANA+ by optimizing parameters of block quantization.
DCGD+ with block quantization.
We ﬁx the number of blocks B ∈ [ d ] for all nodes i ∈
[ n ] and allow each node to apply different block quantization operator
Q Bh i with quantization steps h
i
=
( h i, 1 , . . .
, h i,B ) .
To minimize communication complexity of DCGD+, we need to minimize
L max subject to the communication constraint mentioned above.
Since L max = max i ∈ [ n ] L ( C i , L i ) , each node i ∈ [ n ] can minimize the impact of its own compression by minimizing L ( C i , L i ) based on local smoothness matrix L i .
This leads to the following optimization problem for ﬁnding optimal values of h i for each node i ∈ [ n ] :
[Equation]
where β is the “budget” of communication: Larger β leads to ﬁner quantization levels.
Note that the constraint in (11) depends monotonically from each h l .
Therefore, the optimum is attained when h l (cid:107) Diag ( L lli ) (cid:107) is uniform over l ∈ [ B ] .
Thus, the solution to this problem is given by h i,l = δ i,B (cid:107) Diag ( L lli ) (cid:107) , where δ i,B ≥ 0 is uniquely determined by the constraint equality of (11) as the only positive solution of δ 2 i,B − δ i,B dT i,B β − B − dT 2 i, 1 β − B = 0 , which implies δ i,B = dT i,
[Equation]
+
[Equation]
T i, 1 , where T i,B def = 1 d (cid:80)
Bl =1 √ d l (cid:107)
Diag ( L lli ) (cid:107) .
If this solution of quantization steps h i is used by all nodes i ∈
[ n ] , then we show reduction in communication complexity by a factor of O ( n ) .
Theorem 3.
Assume n = O ( √ d ) and both ν, ν 1 are O (1) .
Then DCGD+ using block quantization with B = n blocks, d l = O ( d / n ) block sizes for all l ∈ [ n ] and quantization steps h i,l = δ i,B / (cid:107)
Diag ( L lli ) (cid:107) with β =
O ( d / n ) reduces overall communication complexity by a factor of O ( n ) compared to DCGD using B = 1 single block quantization.
Formally, to guarantee ε >accuracy, the communication complexity of DCGD+ is O (cid:16) dn L max µ log 1 ε (cid:17) , which is O ( n ) times smaller over DCGD.
DIANA+ with block quantization.
For the rate (7) of DIANA+, we need to optimize ω max + L max nµ part of the complexity under the same communication constraint used in (11).
Since
[Equation]
(12) we can decompose the problem into subproblems for each node i to optimize ω
i + L i nµ with respect to its own quantization parameters
h i .
Analogously, this leads to the following optimization problem for ﬁnding optimal values of h i for each node i ∈ [ n ] :
[Equation]
which can be solved with a similar argument as done for (11).
Details are deferred to the Appendix.
Theorem 4.
Assume n = O ( √ d ) and both ν, ν 1 are O (1) .
Then DIANA+ using block quantization with B = n blocks, d l = O ( d / n ) block sizes for all l ∈ [ n ] and h i,l quantization steps (solution to (13) ) with β
= O ( d / n ) reduces overall communication complexity by a factor of O ( n ) compared to DIANA using B = 1 single block quantization.
Formally, to guarantee ε > 0 accuracy, the communication complexity of DIANA+ is O (cid:0)(cid:0) nd + (cid:113) dn L max µ (cid:1) log 1 ε (cid:1) , which (ignoring n summand in the complexity) is O ( n ) times smaller over DIANA.
Quantization with Varying Steps
Our second extension of standard quantization scheme is to allow different quantization steps for all coordinates { 1 , 2 , . . .
, d } .
In other words, for each coordinate j ∈ [ d ] we quantize with respect to levels { 0 , h j , 2 h j , . . . } .
The standard quantization [Alistarh et al., 2017] is the special case when h j = 1 s for all j ∈ [ d ] , where s is the number of quantization levels.
Deﬁnition 4 (Quantization with varying steps) .
For ﬁxed quantization steps h
= ( h 1 , . .
. , h d ) (cid:62) ∈ R d , deﬁne quantization operator Q h : R d → R d as follows:
[Equation]
where ξ j is deﬁned via the quantization levels { 0 , h j , 2 h j , . .
as in (9).
Note that compression operator Q h is unbiased as E [ ξ j ( v )]
= v for any v ≥
and is not a special case of block quantization deﬁned earlier.
To understand how the number of encoding bits of Q h ( x ) depends on h exactly seems challenging, since it depends on the actual encoding scheme (i.e. binary representation of compressed information).
Besides, even if we ﬁx binary mapping, the closed form expression of total amount of bits is too complicated to be utilized in the further analysis.
We provide theoretical arguments and clear numerical evidence that (cid:107) h − 1 (cid:107) =
(cid:113)(cid:80) dj
=1 h − 2 j is a reasonable proxy for the number of encoding bits for compressor Q h .
Assumption 3.
For any input vector x ∈ R d and quantization steps h ∈ R d , compressed vector
Q h ( x ) can be encoded with O ( (cid:107) h − 1 (cid:107) ) number of bits.
First, consider the special case when all quantization steps are the same, i.e. h j = 1 s .
Then (cid:107) h − 1 (cid:107) =
s √ d recovers the dominant part (provided s = O ( √ d ) ) in (cid:101)
O ( s 2 + s √ d ) showing total amount of bits for standard quantization scheme.
Second, in the Appendix we present an encoding scheme which (up to constant and log d factors) requires E [ ψ ( (cid:107) ˆ x (cid:107) 0 )]
+ (cid:107) h − 1 (cid:107) number of bits in expectation to communicate ˆ x = Q h ( x ) , where ψ ( τ ) def = dH 2 ( τ / d ) + τ ≤ d log 3 , if τ ∈ [0 , d ] and
H 2 is the binary entropy function.
Note that, based on the deﬁnition (9), increasing quantization steps h j forces more sparsity in ˆ x and hence reduces (cid:107) ˆ x (cid:107) 0 .
Thus, (cid:107) ˆ x (cid:107) 0 and hence ψ ( (cid:107) ˆ x (cid:107) 0 ) (notice that ψ (0) = 0 ) are proportional to (cid:107) h − 1 (cid:107) .
Furthermore, we present a numerical experiment which shows that the number of encoding bits of Q h ( x ) and (cid:107) h − 1 (cid:107) are positively correlated.
Hence, in the further analysis, we ﬁx the number of encoding bits of Q h ( x ) by the constraint (cid:107) h − 1 (cid:107) =
β for some parameter β > 0 .
As for the variance induced by the compression operator Q h , we prove the following upper bound for L ( Q h , L ) :
[Equation]
DCGD+ with varying quantization steps.
Now, we optimize the rate (6) of DCGD+ with respect to quantization steps h i
= ( h i ;1 , h i ;2 , . . .
, h i ; d ) of compressor Q h i controlled by i th node for all i ∈ [ n ] .
The term in (6) affected by the compression is L max = max i ∈ [ n ] L ( C i , L i ) , which implies that each node i ∈ [ n ] can minimize the impact of its own compression by minimizing
L ( C i , L i ) based on local smoothness matrix L i .
Based on the upper bound (14) and communication constraint given by (cid:107) h − 1 (cid:107)
=
β for some β > 0 , we get the following optimization problem to choose the optimal quantization parameters
h i for node i ∈ [ n ] :
[Equation]
This problem has the following closed form solution due to KKT conditions (see Appendix):
[Equation]
With this choice of quantization steps we save O (min( n, d )) times in communication.
Theorem 5.
Assume both ν, ν 1 are O (1) and
β =
O ( d / n ) .
Then DCGD+ using quantization with varying steps (25) for all i ∈ [ n ] reduces overall communication complexity by a factor of
O (min( n, d ))
compared to the baseline of DCGD.
Formally, the iteration complexity (6) can be upper bounded as Lµ + L
[Equation]
nµ ≤
[Equation]
, which is min( n, d ) times smaller than the one for DCGD.
As both methods communicate O ( d / n ) bits per node per iteration, we get min( n, d ) times savings in communication complexity.
DIANA+ with varying quantization steps.
Based on (12), each node i ∈ [ n ] optimizes ω i + L i nµ with respect to its quantization parameters
h i , which is equivalent to the problem
[Equation]
where A ij def = L i ; j / nµ .
Due to the KKT conditions (see Appendix), we get the following solution
[Equation]
With this choice of quantization steps we save O (min( n, d )) times in communication.
Theorem 6.
Assume both ν, ν 1 are O (1) and
β =
O ( d / n ) .
Then DIANA+ using quantization with varying steps (18) for all i ∈ [ n ] reduces overall communication complexity by a factor of
O (min( n, d ))
compared to the baseline of DIANA.
Formally, the iteration complexity (7) can be upper bounded as ω
[Equation]
+
[Equation]
+
[Equation]
which is min( n, d ) times smaller than the one for DIANA (ignoring negligible term n ).
Experiments
Setup.
In this section we present two key experiments.
Additional experiments can be found in the Appendix.
We conduct a range of experiments with several datasets from the LibSVM repository [Chang and Lin, 2011] on the (cid:96) 2 -regularized logistic regression problem (1):
[Equation]
where A i,t are data points sorted based on their norms before allocating to local workers for the heterogeneity.
The experiments are performed on a workstation with Intel(R)
Xeon(R) Gold
CPU @ 3.30GHz cores.
The gather and broadcast operations for the communications between master and workers are implemented based on the MPI4PY library [Dalcín et al., 2005] and each
CPU core is treated as a local worker.
For each dataset, we run each algorithm multiples times withrandom seeds for each worker.
Due to space limitations, we present only two of our experiments here deferring the remaining experiments along with experimental details in the Appendix.
Comparison to standard quantization techniques.
In our ﬁrst experiment, we compare smoothness-aware DCGD+ and DIANA+ methods with our varying-step quantization technique
( quant+ ) to the original DCGD [Khirirat et al., 2018] and DIANA [Mishchenko et al., 2019] methods with the standard quantization technique ( quant ) of Alistarh et al.
Figure 1 demonstrates that DCGD+/DIANA+ with quant+ lead to signiﬁcant improvement in both transmitted megabytes and wall-clock time.
An ablation study to disentangle the contributions of exploiting the smoothness matrix and utilizing varying number of levels can be found in Appendix B.
[Figure]
Comparison to matrix-smoothness-aware sparsiﬁcation.
Second experiment is devoted to the performance of three smoothness-aware compression techniques —block quantization ( block quant+ ) of Section 4, varying-step quantization ( quant+ ) of Section 5 and smoothness-aware sparsiﬁcation strategy ( rand- τ + ) of Safaryan et al.
All three compression techniques are shown to outperform the standard compression strategies by at most O ( n ) times in theory.
For the sparsiﬁcation, we use the optimal probabilities and the sampling size τ = d/n as suggested in
Section 5.3 of [Safaryan et al., 2021].
The empirical results in Figure 2 illustrate that the varying-step quantization technique ( quant+ ) is always better than the smoothness-aware sparsiﬁcation [Safaryan et al., 2021], in terms of both communication cost and wall-clock time.
Our block quantization technique also beats sparsiﬁcation when the dimension of the model is relatively high.
[Figure]
A Conclusions and Limitations
A.1 Generalization and quantization . . . . . . . . . . . . . . . . . . . . . . . . . . .
A.2 Technical contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
A.3 Limitations and possible workarounds . . . . . . . . . . . . . . . . . . . . . . . .
B Additional Experiments
B.1 Setup . .
B.2 Comparison to standard quantization techniques . . . . . . . . . . . . . . . . . . .
B.3 Ablation study of DIANA+ ( block quant+ ) and DIANA+ ( quant+ ) . . . . . . .
B.4 Comparison to matrix-smoothness-aware sparsiﬁcation . . . . . . . . . . . . . . .
B.5 Numerical veriﬁcation of Assumption 3 . . . . . . . . . . . . . . . . . . . . . . .
C Proofs for Section 3: Smoothness-Aware Distributed Methods with General Compressors
C.1 Proof of Theorem 1: DCGD+ with arbitrary unbiased compression . . . . . . . . .
C.2 Proof of Theorem 2: DIANA+ with arbitrary unbiased compression . . . . . . . .
D Proofs for Section 4: Block Quantization
D.1 Proof of the variance bound (10) . . . . . . . . . . . . . . . . . . . . . . . . . . .
D.2 Proof of Theorem 3: DCGD+ with block quantization . . . . . . . . . . . . . . . .
D.
3 Proof of Theorem
DIANA+ with block quantization
E Proofs for Section 5: Quantization with varying steps
E.
1 An encoding scheme for Q h operator
E.
2 Proof of the variance bound
E Proof of Theorem 5: DCGD+ with varying quantization
steps . . . . . . . . . . . .
E.4 Proof of Theorem 6: DIANA+ with varying quantization
steps . . . . . . . . . . .
F Notation Table
Appendix
A Conclusions and Limitations
In this work we extended the matrix-smoothness-aware sparsiﬁcation strategy of Safaryan et al.
[2021] to arbitrary unbiased compression schemes.
This signiﬁcantly broadens the use of smoothness matrices in communication efﬁcient distributed methods.
A.1 Generalization and quantization
It is worth to mention that our results generalize those of Safaryan et al.
[2021] in a tight manner.
That is, we recover the same convergence guarantees as a special case.
Indeed, if compression operators
C i are diagonal sketches
C i generated independently from others and via arbitrary samplings, then
[Equation]
with the same probability matrices P i and (cid:101) P i deﬁned in [Safaryan et al., 2021].
Further, we designed two novel quantization schemes (see Deﬁnitions 3 and 4) capable of properly utilizing matrix smoothness information of local loss functions in distributed optimization.
We showed that the proposed quantization schemes can signiﬁcantly outperform the key baselines both in theory and practice.
A.2 Technical contributions
We make two main technical contributions.
First, we introduce the quantity L ( C , L ) that properly captures non-linear interaction between the compressor C and the smoothness matrix L .
Due to the linearity of the sparsiﬁcation (i.e.,
C ( x ) =
C x ), in previous work (Safaryan et al., 2021) it is easy to separate the sparsiﬁer from the compressed gradient and combine it with the smoothness matrix:
[Equation]
where CLC shows a linear interaction between C and L .
Once we came up with the proper notion of L ( C , L ) (we had other approaches before we found the "right" one), the proofs of Theorems 1 and 2 followed standard steps.
Note that L ( C , L ) recovers the previous quantity when the compressor is specialized to a sparsiﬁer (see Sec A.1).
This contribution may seem simple from hindsight, but it is not.
Our second technical contribution is the introduction of two non-linear compressors that provably beneﬁt from smoothness matrices.
Speciﬁcally, we formulate and analytically solve 4 intermediate optimization problems (11), (14), (16), (18) to ﬁnd out the best parameter setting for each quantization scheme based on the smoothness information.
In fact, sections 4 and 5 outline the technical difﬁculties we managed to overcome in order to get min( n, d ) speedup factors in each case.
Our key contribution is the proposal of these two modiﬁed quantization schemes.
We adapt methods DCGD+ and DIANA+ to showcase the potential of our quantization strategies in reducing communication complexity.
We chose DCGD as it is the simplest gradient type method with communication compression, and
DIANA as it is the variance-reduced version of DCGD.
Of course, one can apply our quantization techniques to other distributed methods and gain similar improvements (see Sec A.2).
However, we are not attempting to (and can’t) be exhaustive in this direction as there are many methods in the literature employing communication compression.
A.3 Limitations and
Next, we discuss main limitations of our work.
• Note while in this paper we redesigned only two methods, DCGD+ and DIANA+, the modiﬁcations we suggest are not limited to these two methods and can be applied to other distributed methods.
In particular, with a similar proof technique, ADIANA+ method of Safaryan et al.
[2021] introduced with sparsiﬁcation can also be extended to arbitrary unbiased compression operator using the new notion of L ( C , L ) .
• The computation or estimation of the smoothness matrix L i requires addiotional preprocessing.
For generalized linear models (GLM) (e.g., linear/logistic regression, SVM with smooth hinge loss) the matrix L i can be written in closed form using the local dataset (see Lemma 1 of [Safaryan et al., 2021]).
For example,
[Equation]
for logistic regression, where { A im : m = 1 , . . .
, m i } is the local data of device i .
Beyond GLMs, L i can be difﬁcult to compute.
Note that we do not claim that the proposed method would be practical for high-dimensional deep learning problems - but perhaps this will be overcome in future research.
One possibility is to treat L i ’s as hyper-parameters and learn some rough approximations of the smoothness matrices from the ﬁrst order information obtained by running a gradient type method .
This can be done initially as a preprocessing step, after which the matrices are considered “learned”, and then our compression can be built and used.
• The server is required to store d × d matrices L 1 / 2 i for all nodes i ∈
[ n ] and multiply them by sparse updates C ki ( L † 1 / 2 i ∇ f i ( x k )) in each iteration.
Moreover, each node i is required to store only its smoothness matrix L † 1 / 2 i and perform multiplication L † 1 / 2 i
[Equation]
in each iterate.
Hence, our methods are practical when either dimension d is not too big or smoothness matrices
L i are of special structure (e.g., diagonal, low-rank).
• We did not analyze the compression of the smoothness matrix before communication as it is transferred only once before the training begins.
Besides, we showed in our experiments that the overhead in communication cost is negligible when the number of iterations is large (the transmitted megabytes do not start from 0 in our plots).
However, in practice, compressing the matrix L is a good idea.
One option for that is to initially estimate a diagonal smoothness matrix that is as easy to communicate (still only once ) as one full precision gradient.
Another option is to directly apply compression to the matrix L so that the compressed matrix is an over-approximation.
For example, let
L = (cid:80) dk
=
λ k u k u (cid:62) k be the eigendecomposition of L , where λ k is the k th largest eigenvalue corresponding to eigenvector u k .
Then
[Equation]
The latter
over
-approximation (which serves as a smoothness matrix for f ) can be transferred with rd + 1 ﬂoats where r can be chosen small.
• For the sake of presentation, we analyzed both DCGD+ and DIANA+ when exact local gradients, ∇ f i , can be computed by all nodes in each iteration.
However, we believe that it is possible to extend the analysis to stochastic local gradient oracles.
Current tools handling stochastic gradients can be easily applied to our matrix-smoothness-aware compression techniques.
In our distributed methods we only compress uplink communication from nodes to the server, which is typically more bandwidth limited than downlink communication from the server to nodes.
We believe that techniques that ensure compressed communication in both directions can be applied in our setting, too.
• We developed all our theory
for strongly convex objectives.
Extending the theory to convex and non-convex problems in a tight manner seems to be more challenging.
B Additional Experiments
In this section we provide additional experiments to highlight effectiveness of our approach.
B.1 Setup
We run the experiments with several datasets listed in Table 2 from the LibSVM repository [Chang and Lin, 2011] on the (cid:96) 2 -regularized logistic regression problem described below:
[Equation]
where f i ( x ) = 1 m
[Equation]
where x ∈ R d , A i,l ∈ R d , b i,l ∈ {− 1 , 1 } are the feature and label of l -th data point on the i -th worker, where the features of each A i,l are rescaled into [ − 1 , 1] .
The data points are sorted based on their norms before allocating to local workers to ensure that the data split is heterogeneous.
The experiments are performed on a workstation with Intel(R)
Xeon(R)
Gold 6246 CPU @ 3.30GHz cores.
The gather and broadcast operations for the communications between master and workers are implemented based on the MPI4PY library [Dalcín et al., 2005] and each CPU core is treated as a local worker.
We set λ = 10 − 3 for all datasets.
For each dataset, we run each algorithm multiples times with 5 random seeds for each worker.
[Table]
To implement Elias encoding and decoding, we utilize the EliasOmega library 3 .
We compare the relative errors of different algorithms with respect to 3 measures: the number of iterations, the transmitted megabytes and wall-clock time.
To be speciﬁc, the measured wall-clock time includes 1) the time of computation on each local worker in one iteration (e.g., local gradient computation, matrix multiplication, etc.); 2) the time of Elias coding and decoding; 3) the time of communication ( gather and broadcast ).
It is worth noting that DCGD+ and DIANA+ require extra cost to transmit
L 1 / 2 i beforehand.
Moreover, when coupled with varying number of quantization levels, they also need to transmit h i ; j before the start of training.
These overheads are taken into consideration in our experimental results.
B.2 Comparison to standard quantization techniques
First, we compare DCGD+/DIANA+ with the block quantization technique ( block quant+ ) described in Section 4 to DCGD [Khirirat et al.,
2018]/DIANA [Mishchenko et al., 2019] with the standard quantization technique ( quant ) in [Alistarh et al., 2017].
As shown in Figure 6, DCGD+ ( block quant+ ) and DIANA+ ( block quant+ ) outperform DCGD ( quant ) and DIANA ( quant ) when d is larger.
This is understandable because the extra cost on communication B norms becomes neglectable when the dimension is relatively high given the number of blocks, where splitting the whole parameters into blocks makes more sense.
[Figure]
e d b i t[-s y =
-]
[Figure]
y =0 : 3 x +
Next, we compare DCGD+/DIANA+ with our second quantization technique ( quant+ ) that has varying number of quantization steps per coordinate to DCGD ( quant ) and DIANA ( quant ).
Figuredemonstrates that DCGD+ ( quant+ ) and DIANA+ ( quant+ ) lead to signiﬁcant improvement.
B.3 Ablation study of DIANA+ ( block quant+ ) and DIANA+ ( quant+ )
As mentioned by Alistarh et al.
[2017], combining DCGD and block quantization can improve its iteration complexity at the cost of transmitting extra 32 B bits per iteration, which might also lead to better total communication complexity.
Thus, the advantage of DIANA+ ( block quant+ ) over
DIANA ( quant ) may come from either splitting the features into blocks or exploiting the smoothness matrix.
To further demistefy the improvement of DIANA+ (block quant+) , we compare the results of DIANA+ ( block quant+ ), DIANA+ ( block quant ), DIANA ( block quant ) and DIANA ( quant ) in Figure 5.
The difference between block quant and block-quant+ is that the former one uses the same number of quantization levels for different blocks while the latter one uses varying numbers.
It can be seen from Figure 5 that DIANA+ ( block-quant+ ) consistently outperforms other methods because it optimally exploits the block structure and the smoothness matrix.
We also demonstrate that how DIANA+ perform with varying or ﬁxed number of levels.
As seen in
Figure 6, the varying number of levels are beneﬁcial on most of the datasets.
B.4 Comparison to matrix-smoothness-aware sparsiﬁcation
Moreover, we also compare the performance of three smoothness-aware compression techniques —block quantization ( block quant+ ) of Section 4, varying-step quantization ( quant+ ) of Sectionand smoothness-aware sparsiﬁcation strategy ( rand- τ + ) of Safaryan et al.
All three compression techniques are shown to outperform the standard compression strategies by at most O ( n ) times in theory.
For the sparsiﬁcation, we use the optimal probabilities and the sampling size τ = d/n as suggested in Section 5.3 of [Safaryan et al., 2021].
The empirical results in Figure 8 illustrate that the varying-step quantization technique ( quant+ ) is always better than the smoothness-aware sparsiﬁcation [Safaryan et al., 2021], in terms of both communication cost and wall-clock time.
Our block quantization technique also beats sparsiﬁcation when the dimension of the model is relatively high.
B.5 Numerical veriﬁcation of Assumption
We provide a numerical experiment to verify Assumption 3 that (cid:13)(cid:13)
h − 1 (cid:13)(cid:13) and the communicated bits are positively correlated.
Figure 3 shows that the communicated bits and (cid:13) (cid:13) h − 1 (cid:13) (cid:13) are indeed positively correlated.
[Figure]
[Figure] [Table]
[Figure]
[Table]
[Figure] [Figure]
[Figure] [Figure]
[Figure]
C Proofs for Section 3: Smoothness-Aware Distributed Methods with General Compressors
Here we provide the proofs of Theorem 1 and Theorem 2.
Both proofs follow similar steps done for sparsiﬁcation in [Safaryan et al., 2021].
C.1 Proof of Theorem 1: DCGD+ with arbitrary unbiased compression
To simplify the notation, let us skip the iteration count k in the derivations.
We are going to estimate the quantity E (cid:2) (cid:107) g
( x ) − ∇ f ( x ∗ ) (cid:107) 2 (cid:3) and establish the following bound for the gradient estimator g ( x )
[Equation]
where D f ( x, x ∗ ) is the Bregman divergence between x and x ∗ with respect to f .
Due to Lemma E.
[Hanzely and Richtárik, 2019], we have ∇ f i ( x ) =
L 1 / 2 i r i for some r i .
Therefore,
E
[Equation]
which implies unbiasedness of the estimator g ( x ) , namely E [ g ( x )]
= ∇ f ( x ) .
Note that:
[Equation]
where ♣ is due to E (cid:104)
L 1 /
[Equation]
i C i
( L
[Equation] [Equation]
i ∇ f i ( x )) (cid:105) =
∇ f i ( x ) and ∇ f i ( x )
=
L 1 / 2 i L † 1 /
i
L 1 / 2 i r i =
L 1 / 2 i L † 1 / 2 i ∇ f i ( x ) based on (19) and ∇ f i ( x ) =
L 1 / 2 i r i for some r i .
♠ can be directly obtained by noticing the deﬁnition of L ( C , L ) in Table 3.
♦ is based on the deﬁnition of L i and the fact (cid:107) x + y (cid:107) 2 ≤ 2 (cid:107) x (cid:107) 2 + 2 (cid:107) y
(cid:107) 2 .
(cid:70) is due to Lemma E.3 of Hanzely and Richtárik [2019] and the deﬁnition of σ ∗ + deﬁned in Theorem 1.
The inequality above together with convexity and L -smoothness of f implies
[Equation]
Applying the result of Gorbunov et al.
[2020] we conclude the proof.
C.2 Proof of Theorem 2: DIANA+ with arbitrary unbiased compression
We start with the unbiasedness of the estimator
[Equation]
In (19), we showed unbiasedness using inclusion ∇ f i ( x k ) ∈ range( L i ) .
Assuming u ki ∈ range( L i ) for all k ≥ 0 , we get ∇ f i ( x k ) − u ki ∈ range( L i ) for all k ≥ 0 .
Hence, in the same way we can show unbiasedness of g k as
[Equation]
The inclusion u ki ∈ range( L i ) directly follows from the initialization u
i ∈ range( L i ) (see line 1 of
Algorithm 2) and linear update rule of u k +1 i =
u
ki + α L 1 / 2 i ∆ ki (see line 5 of Algorithm 2).
As both ∇ f i ( x k ) and u ki belong to range( L i ) , denote ∇ f i ( x k ) − u ki =
L 1 / 2 i r ki .
Next we bound
E (cid:2) (cid:107) g
( x ) − ∇ f ( x ) (cid:107) 2 (cid:3)
=
E 
[Equation]
1 n n
[Equation]
= n
[Equation]
≤
[Equation]
n
[Equation]
where σ k + def = 1 n (cid:80) ni =1 (cid:107) u ki − ∇ f i ( x ∗ ) (cid:107) 2 L † i is the error in the gradient learning process.
To proceed, we need to establish contractive recurrence relation for σ k + .
For each summand, we have
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
where we used bounds α ≤ 1
+ ω i and
(cid:22) L 1 / 2 i
L † i L 1 /
i (cid:22) I .
Thus, with α ≤ 1 1+ ω max , the estimator g k of DIANA+ satisﬁes
[Equation]
Again, we apply the generic result of Gorbunov et al.
[2020] to complete the proof.
D Proofs for Section 4: Block Quantization
Here we provide the missing proofs of Section
[Equation]
D.1 Proof of the variance bound (10)
Using Deﬁnition 3 of compression operator Q Bh
[Equation]
we have
[Equation]
From the deﬁnition of L ( Q Bh , L ) we get
[Equation]
which implies (10) if we ignore the ﬁrst term.
D.2 Proof of Theorem 3: DCGD+ with block quantization
First, recall that quantization steps h i are given by h i,l = δ i,B (cid:107) Diag
[Equation]
∈ [ B ] , where δ i,B ≤ d
[Equation]
B + (cid:115)
d β − B T
i
[Equation]
Then, we have
L max
[Equation]
Set β = d / n + n and B = n .
Since n = O ( √ d ) , we have β =
O ( d / n ) and hence d / n β − B
= 1 .
For the sake of simplicity, assume d l = d / n .
Next
T i, 1 √ n ≤
[Equation]
√ nd
[Equation]
(cid:88)
[Equation] [Equation]
T i
[Equation]
n ≤ d n (cid:88) l
= (cid:112)
d l d l (cid:88) j =
L lljj = max l ∈ [ n ] √ d l d
[Equation]
Regardless of the choice h i , using the following inequalities with respect to matrix order
[Equation]
we bound L as follows
[Equation]
Hence
[Equation]
which guarantees n times fewer communication rounds with the same number of bits per round.
In other words, each node communicates O ( d / n ) bits to the master in each iteration
[Equation]
which gives us
O ( d ) communication per communication round.
Thus
[Equation]
overall communication complexity to achieve ε > 0 accuracy is
[Equation]
D.3 Proof of Theorem 4: DIANA+ with block quantization
As already mentioned
[Equation]
for DIANA+ each node aims to minimize ω i + 1 nµ L ( Q Bh i , L i ) with respect to its quantization steps h i .
Notice that
[Equation]
This leads to the following optimization problem with respect to h :
[Equation]
which is solved similar to (11).
Denote
[Equation]
Analogous to (11)
[Equation]
the solution of (22) has the following form
[Equation]
where (cid:101) δ iB is determined by the constraint equality of (22) as
[Equation]
Let us estimate (cid:101) T i, 1 and (cid:101) T i,n using the assumptions B = n and (for the sake of simplicity)
d l = d / n .
[Equation]
Next, using β = d / n + n and ν 1 =
O (1) , we get ω
i
[Equation] [Equation]
Together with (21), we complete the proof with the following iteration complexity:
[Equation]
E Proofs for Section 5: Quantization with varying steps
In this part of the appendix we provide missing proofs and detailed arguments of Section 5.
E.1 An encoding scheme for Q h operator
To communicate a vector of the form Q h ( x ) , we adapt the encoding scheme of Albasyoni et al.
From the deﬁnition, we have
[Equation]
for all j ∈ [ d ] , where ˆ k j ≥ 0 are non-negative random variables coming from (9).
Thus, we need to encode the magnitude (cid:107) x (cid:107) , signs sign( x j ˆ k j ) and non-negative integers ˆ k j .
For the magnitude (cid:107) x (cid:107) we need just 31 bits.
Let n 0 def = |{ j ∈ [ d ]: ˆ k j = 0 }| be the number of coordinates x j that are compressed to 0.
To communicate signs { sign( x j ˆ k j ): j ∈ [ d ] } , we ﬁrst send the locations of those n 0 coordinates and then d − n 0 bits for the values ± 1 .
Sending n 0 positions can be done by sending log d bits representing the number n 0 , followed by log (cid:0) dn 0 (cid:1) bits for the positions.
For the signs, we need log d + log (cid:0) d ˆ n 0 (cid:1) + d − ˆ n 0 ≤ log d + d log 3 bits at most.
Finally, it remains to encode ˆ k j ’s for which we only need to send nonzero entries since the positions of ˆ k j = 0 are already encoded.
We encode ˆ k j ≥ 1 with ˆ k j bits: ˆ k j − 1 ones followed by 0.
Hence, the expected number of bits to encode ˆ k j ’s is
[Equation]
(cid:88)
[Equation]
(cid:118)(cid:117)(cid:117)(cid:116)
[Equation]
(cid:88)
[Equation]
(cid:118)(cid:117)(cid:117)(cid:116)
[Equation]
(cid:88)
[Equation]
(cid:107)
[Equation]
(cid:107)
[Equation]
where v j = | x j | (cid:107) x (cid:107) .
In total, Q h ( x ) can be encoded by
[Equation]
bits.
Lastly, the log (cid:0) d ˆ n 0 (cid:1) term can be upper bounded by the binary entropy function H 2 ( t ) def = − t log t − (1 − t ) log(1 − t ) (see [Albasyoni et al., 2020] for more details), and the expected number of encoding bits for Q h ( x ) can be upper bounded by
[Equation]
where ˆ x = Q h ( x ) .
E.2 Proof of the variance bound (14)
Let v ∈ R
d be the unit vector with non-negative entries v j = | x j | / (cid:107) x
(cid:107) for j ∈ [ d ] .
Then
[Equation] [Equation]
≤ (cid:107) x (cid:107)d (cid:88)
j =
L jj h
j min (cid:18) 1 , v j h j (cid:19) ≤ min   d (cid:88) j
=
L jj h 2 j , d (cid:88) j
=
L jj h j v j   (cid:107) x (cid:107)≤ min   d (cid:88) j =
L jj
[Equation]
j
[Equation]
(cid:118)(cid:117)(cid:117)(cid:116)
d (cid:88)
j =
[Equation]
2 jj
[Equation]
j
[Equation]
(cid:107)
[Equation] [Equation]
which implies (14).
E.3 Proof of Theorem 5: DCGD+ with varying quantization steps
Based on the upper bound (14) and the communication constraint given by (cid:107) h − 1 i (cid:107) =
β for some β > 0 , we get the optimization problem
[Equation]
subject to
[Equation] [Equation]
for choosing the optimal quantization parameters
h i ; j .
This problem has a closed form solution.
Indeed, due to the KKT conditions, we have
[Equation]
where ζ is the multiplier.
Solving this leads to the solution:
[Equation] [Equation]
For the solution (25) we have
[Equation]
Therefore, if both parameters ν and ν 2 are O (1) , then the rate (6) of DCGD+ becomes O ( L max nµ +
L max βnµ ) .
To make a fair comparison against DCGD, we need to ﬁx O ( dn ) number of bits each node communicates to the master server.
Now, to make DCGD+ communicate the same number of bits, we set β
= O ( dn ) .
Hence we have the following iteration complexity for DCGD+ based on solution (25):
[Equation]
which is min( n, d ) times better than the one of DCGD.
E.4 Proof of Theorem 6: DIANA+ with varying quantization steps
Denote A ij def
=
L i ; jj nµ .
Note that
[Equation]
We solve the optimization problem
[Equation]
subject to
[Equation] [Equation]
which has a closed form solution.
Indeed
[Equation]
due to the KKT conditions
[Equation]
we have:
[Equation] [Equation]
For the solution (28) we have
[Equation]
which further leads to O ( n + 1 n L max µ + 1 d L max µ ) iteration complexity if ν 1 = O (1) and
β = O ( dn ) .
[Table] [Table]
[Table]
positive semideﬁnite matrix L
L †
Moore–Penrose inverse of matrix L
L i smoothness matrix of f i
L i ; j , L i ; jj j th diagonal element of L i
L ( C , L )
[Equation]
L i
[Equation]
L max
[Equation]
ν, ν
[Equation]
[Table] [Table]
[Table]
=
[Table]
=
[Table] [Table]