Theoretically Better and Numerically Faster Distributed Optimization with Smoothness-Aware Quantization Techniques
Abstract
To address the high communication costs of distributed machine learning, a largebody of work has been devoted in recent years to designing various compressionstrategies, such as sparsiﬁcation and quantization, and optimization algorithmscapable of using them.
Recently, Safaryan et al.
[2021] pioneered a dramaticallydifferent compression design approach: they ﬁrst use the local training data to formlocal smoothness matrices and then propose to design a compressor capable ofexploiting the smoothness information contained therein.
While this novel approachleads to substantial savings in communication, it is limited to sparsiﬁcation as itcrucially depends on the linearity of the compression operator.
In this work, wegeneralize their smoothness-aware compression strategy to arbitrary unbiasedcompression operators, which also includes sparsiﬁcation.
Specializing our resultsto stochastic quantization, we guarantee signiﬁcant savings in communicationcomplexity compared to standard quantization.
In particular, we prove that blockquantization with n blocks theoretically outperforms single block quantization,leading to a reduction in communication complexity by an O ( n ) factor, where nis the number of nodes in the distributed system.
Finally, we provide extensivenumerical evidence with convex optimization problems that our smoothness-awarequantization strategies outperform existing quantization schemes as well as theaforementioned smoothness-aware sparsiﬁcation strategies with respect to threeevaluation metrics: the number of iterations, the total amount of bits communicated,and wall-clock time.
Introduction
Training modern machine learning models is typically cast in terms of (regularized) empirical riskminimization problem and requires increasingly more training data to make empirical risk closer tothe true risk [Schmidhuber, 2015, Vaswani et al., 2019].
This natural requirement makes it harder (andin some scenarios impossible) to collect all data in one place and carry out the training using a singledata source.
As a result, we reconciled with a ﬂock of datasets disseminated across various compute nodes holding the actual training data [Bekkerman et al., Vogels et al., 2019].
However, such divide-and-conquer approach of handling vast amount of data means that local updates need to be communicated among the nodes (or through some central server orchestrating the process) which
This issue is further exacerbated by the fact that modern highly performing models are typically overparameterized et al., 2020, Narayanan et al.,
Distributed training.
In this paper, we consider distributed training formalized as the followingoptimization problem
[Equation]
f
[Equation]
x
[Equation]
and where d is the number of parameters of model x ∈ R d to be trained, n is the number ofmachines/nodes participating in the training, f i ( x ) is the loss/risk associated with the data stored onmachine i ∈
[ n ] def = { 1 , 2 , . . . , n } , f ( x ) is the empirical loss/risk, and R ( x ) is a regularizer.
Because of the communication constraints, large body of work has been devoted in recent years tothe design of various compression strategies, such as sparsiﬁcation [Koneˇcný and Richtárik, 2018,
Wangni et al., 2018, Alistarh et al., 2018], quantization [Goodall, 1951, Roberts, 1962, Alistarhet al., 2017], low-rank approximation [Vogels et al., 2019], three point compressor [Richtárik et al.
,2022], and optimization algorithms capable of using them, such as Distributed Compressed Gradient
Descent (DCGD)
[Khirirat et al., 2018], QSGD
[Alistarh et al., 2017, Faghri et al., 2020], NUQSGD[Ramezani-Kebrya et al., 2021], DIANA [Mishchenko et al., 2019, Horváth et al., 2019], PowerSGD[Vogels et al., 2019], signSGD
[Bernstein et al., 2018, Safaryan and Richtárik, 2021], intSGD[Mishchenko et al., 2021], ADIANA [Li et al., 2020], MARINA [Gorbunov et al., 2021].1.2.
From scalar smoothness to matrix smoothness.
Typically, distributed optimization algorithms in the literature that employ compressed communication, including all methods from theaforementioned works, use only shallow smoothness information of the loss function such as scalar
L -smoothness [Nesterov, 2004].
Deﬁnition 1 (Scalar Smoothness) .
Differentiable function φ : R d → R is called L -smooth if thereexists a non-negative scalar value L ≥ 0 such that
[Equation]
As pointed out by Safaryan et al.
[2021], smoothness constant L reﬂects small part of the richsmoothness information often easily available through the training data.
In their recent work,
Safaryan et al.
[2021] pioneered a dramatically different compression design approach.
First, theypropose to use the local training data to form local smoothness matrices , which they claim containmuch more useful information than standard smoothness constants.
Deﬁnition 2 (Matrix Smoothness) .
Differentiable function φ : R d → R is called L -smooth if thereexists a symmetric positive semideﬁnite matrix L (cid:23) 0 such that
[Equation]
Using smoothness matrices L i of all local loss functions
f i ( x ) , i ∈ [ n ] , Safaryan et al.
[2021] designa compressor capable of exploiting the smoothness information contained within the smoothnessmatrices.
In particular, under certain heterogeneity conditions on the smoothness matrices L i , theirnew compressor reduces total communication cost by a factor of O (min( n, d )) .
While this novel approach leads to substantial savings in communication, it is limited to randomsparsiﬁcation as it crucially depends on the linearity of the compression operator.
It is not clearwhether this approach can be useful in the design of other smoothness-aware compression techniques.
Summary of Contributions
Motivated by the above mentioned development, in this work, we made the following contributions.
Extending matrix-smoothness-aware sparsiﬁcation
to general compression schemes.
First,we generalize the smoothness-aware sparsiﬁcation strategy [Safaryan et al., 2021] to arbitrary unbiased compressors.
Instead of sparsiﬁcation operator, we consider the generic class B d ( ω ) of (possiblyrandomized) unbiased compression operators C : R d → R d with bounded variance ω ≥ 0 , i.e.,
[Equation]
This class is quite broad including random sparsiﬁcation and various quantization schemes.
Tobeneﬁt from the matrix smoothness information with general compressor C , we propose the following
[Table]
modiﬁcation in the communication protocol.
If x ∈ R d is the vector to be communicated, instead ofapplying compressor C directly to x and sending C ( x )
[Table] we compress it by C ( L † 1 / 2 x ) and decompressit by multiplying L 1 / 2 .
Overall, the receiver estimates the original x by L 1 / 2 C ( L † 1 / 2 x ) .2.2.
Distributed compressed methods with improved communication complexity.
To highlightthe appropriateness of our generalization, we redesign two distributed compressed methods—DCGD[Khirirat et al., 2018] and DIANA [Mishchenko et al.
, 2019]—to effectively utilize both matrixsmoothness information and general compression operators leading to new methods, which we call
DCGD+ (Algorithm 1) and DIANA+ (Algorithm 2).
The key notion we introduce that enables thetechnical analysis is the following quantity describing interaction between compression operator
C ∈ B d ( ω ) and smoothness matrix L (cid:23) 0 :
[Equation]
This quantity generalizes the one deﬁned in Safaryan et al.
[2021] for sparsiﬁcation, and providesmeans for tighter theoretical guarantees (Theorems 1 and 2) and better compression design.2.3.
Block quantization.
As we are no longer constrained to sparsiﬁcation to exploit matrix smoothness, we consider more aggressive quantization schemes to further reduce the communication cost.
Our ﬁrst extension of standard quantization [Alistarh et al., 2017] is block quantization , where eachblock is allowed to have a separate quantization parameter.
Notably, we show theoretically that ourblock quantization with n blocks outperforms single block quantization and saves in communicationby a factor of O ( n ) for both DCGD+ (Theorem 3) and DIANA+ (Theorem 4) when n = O ( √ d ) .2.3.
Quantization with varying steps.
In our second extension of standard quantization, we go evenfurther and allow all coordinates to have their own quantization steps.
This extension turns out to bemore efﬁcient in practice than block quantization and provides savings in communication cost by afactor of O (min( n, d )) for both DCGD+ (Theorem 5) and DIANA+ (Theorem 6).2.4.
Experiments.
Finally, we perform extensive numerical experiments using LibSVM data [Changand Lin, 2011] and provide clear numerical evidence that the proposed smoothness-aware quantizationstrategies outperform existing quantization schemes as well the aforementioned smoothness-awaresparsiﬁcation strategies with respect to three evaluation metrics: the number of iterations, the totalamount of bits communicated, and wall-clock time (see Section 6 and the Appendix).
Smoothness-Aware Distributed Methods with General Compressors
In this section we extend methods DCGD+ and DIANA+ of Safaryan et al.
[2021] to handle arbitraryunbiased compression operators.
We consider the problem (1) with matrix smoothness assumptionfor all local losses f i ( x ) and with strong convexity of loss function f ( x ) .
Assumption 1 (Matrix smoothness) .
The functions f i : R d → R are differentiable, convex, lowerbounded and L i -smooth.
Besides, f is L -smooth with the scalar smoothness constant L def = λ max ( L ) .
First, notice that we denote the smoothenss matrix of global loss function f to be L .
Clearly, therelation (1) between f i and f implies L (cid:22) 1 n (cid:80) ni
=1 L i .
This means that while 1 n (cid:80) ni
=1 L i can be asmoothness matrix for f , there might be a tighter estimate, which we denote by L .
Note that lowerboundedness of f i ( x ) is not needed once L i (cid:31) 0 is invertible.
This part of the assumption is not arestriction in applications as all loss function are lower bounded.
Assumption 2 ( µ -convexity) .
The function f : R d → R is µ -convex for some µ > 0 , i.e.,
[Equation]
DCGD+ with arbitrary unbiased compression.
In our version of DCGD+, each node i ∈ [ n ]is allowed to control its own compression operator C i ∈ B d ( ω ) independent of other nodes.
Denote
[Equation]
where L i def =
L ( C i , L i ) .
Furthermore, as the compressor C i can be random, denote by C ki a copy of C i generated at iteration k .
Similar to the standard DCGD method, convergence of DCGD+ is linear up to some oscillationneighborhood.
However, for the interpolation regime this neighborhood vanishes and the methodconverges linearly to the exact solution.
Theorem 1.
Let Assumptions 1 and 2 hold and assume that each node i ∈ [ n ] generates its owncopy of compression operator C ki ∈ B d ( ω i ) independently from others.
Then, for the step-size0 < γ ≤ 1 L + 2 n L max , the iterates { x k } of DCGD+ (Algorithm 1) satisfy
[Equation]
(5) where σ ∗ + def = 1 n (cid:80) ni
=1 L i (cid:107)∇ f i ( x ∗ ) (cid:107) 2 L † i .
In particular, for the interpolation regime (i.e., ∇ f i ( x ∗ ) =
[-= 
-]0for all i ∈ [ n ] ), then DCGD+ converges linearly with iteration complexity [Equation] , step size parameters γ > 0 and α > 0 , compression operators {C k 1 , . . .
, C kn } 2: for each node i = 1 , . . . , n in parallel do 3: get x k from the server and compute local gradient ∇ f i ( x k ) 4: send compressed update ∆ ki = C ki ( L 1 / 2 i ( ∇ f i ( x k ) − u ki )) to the server 5: update local gradient and shift ∆ k i
= L 1 / 2 i ∆ k i , g k i = u k i + ∆ k
i , u k +1 i = u k i + α ∆ k i 6: end for 7: on server 8: get all sparse updates ∆ ki , i ∈ [ ∆
= 1 n (cid:80) ni =1 ∆ ki = 1 n (cid:80) ni = L 1 / 2 i ∆ , g
{+= ∆ k + u k 9: update the global model to x + prox γR ( γg k and global shift to u k +
 = u k + α ∆ k+}
We show later that the iteration complexity (6) of DCGD+ can be much better than one of DCGD.
However, the size of the neighborhood of DCGD+ might be bigger than of DCGD.
In case of standard (scalar) smoothness (i.e. L i =
L i I ) the size of the neighborhood wouldbe σ ∗ def = 1 n (cid:80) ni
=1 ω i (cid:107)∇ f i ( x ∗ ) (cid:107) 2 , which might be smaller than σ ∗ + .
Even though we have
L i ≤ ω
i λ max ( L i ) from the deﬁnition of L i , it does not imply L i L † i (cid:22) ω
i I .
Thus, with matrixsmoothness-aware compression we ensure faster linear convergence at the cost of a possibly largeroscillation radius.
This is not an issue for the interpolation regime, which can interpolate the wholetraining data with zero loss.
Moreover, next we present an algorithmic solution to remove theneighborhood using the DIANA method.3.2.
DIANA+ with arbitrary unbiased compression.
The mechanism allowing to remove theneighborhood in DIANA+ is based on the DIANA method, which was initially introduced for ternaryquantization by Mishchenko et al.
[2019], and then extended to arbitrary unbiased compressionoperators by Horváth et al.
The high level idea is to learn the local optimal gradients ∇ f i ( x ∗ )by estimates u ki for all nodes i ∈
[ n ] in a communication efﬁcient manner.
Nodes use these estimatesu ki to progressively construct better local gradient estimates g ki reducing the variance induced fromthe compression.
We prove in the Appendix that both iterates x k and all local gradient estimates u ki converge linearlyto the exact solution x ∗ and ∇ f i ( x ∗ ) respectively.
Theorem 2.
Let Assumptions 1 and 2 hold and assume that each node i ∈ [ n ] generates its own copyof compression operator C ki ∈ B d ( ω i ) independently from others.
Then, if ω max = max 1 ≤ i ≤ n ω iand the step-size γ = 1 L +
[Equation]
n
[Equation]
max , DIANA+ (Algorithm 2) converges linearly with iteration complexity
[Equation]
Notice that the cost of removing the neighborhood is the extra O ( ω max log 1 ε ) iterations, which isnegligible in the overall complexity (7) above.
Another interesting observation is the second orderﬂavor of the gradient learning technique employed by DIANA+.
Let, for concreteness, matrices L i beinvertible and C ki ( − x ) =
−C ki ( x ) for all x ∈ R d (both random sparsiﬁcation and quantization satisfythis).
Typically, the learning procedure of the original DIANA method, u k +1 i =
u ki − α C ki ( u ki −∇ f i ( x k )) , can be interpreted as a single step of CGD applied to the problem of minimizing theconvex quadratic function ϕ ki ( u ) def = 12 (cid:13)(cid:13)
u − ∇ f i ( x k ) (cid:13)(cid:13) 2 , which changes in each iteration becausethe gradient changes.
In contrast, we observe that the learning mechanism of DIANA+ can beinterpreted as a single step of a (damped) Newton’s method with compressed gradients and withthe true Hessian.
Indeed, ﬁx the iteration counter k and denote ϕ ki ( u ) def = 12 (cid:13)(cid:13)
u − ∇ f i ( x k ) (cid:13)(cid:13) 2 L − 1 / 2 i .
Then, the update rule of shifts u ki in DIANA+ can be rewritten as u k +1 i =
u ki − α L [Equation] / 2 i C
( L − [Equation] [Equation] i ( u ki −∇ f i ( x k )))
=
u ki − α (cid:2) ∇ 2 ϕ
i ( u ki ) (cid:3) − 1 C ki ( ∇ ϕ
i ( u ki )) .
This might serve as an extra explanation on whyincorporating smoothness matrices properly can improve the performance of ﬁrst order methods withcommunication compression.3.3.
Baselines for the original methods.
To make the theoretical comparison against DCGD and
DIANA more transparent, we ﬁx the following baselines using the standard quantization scheme.
• Baseline for DCGD.
Based on the iteration complexity (cid:101) O
( Lµ + ωL max nµ ) of DCGD (in case∇ f i ( x ∗ ) =
for all i ∈ [ n ] ) the optimal level of compression variance ω
= O ( n ) results in (cid:101) O
( L max µ )iterations complexity.
From the estimate of quantization variance ω = min (cid:16) ds 2 , √ ds (cid:17) (
Lemma 3.1., Alistarh et al. [2017]) we conclude
that s =
O ( √ dn ) should be used.
Finally, with this choice of s , each node communicates O ( s 2 + s √ d ) =
O ( dn ) amount of bits.
Thus, total communication complexity (i.e. how many bits ﬂows through the central server) of DCGD is (cid:101) O ( dL max µ ) .
• Baseline for DIANA.
Based on the iteration complexity (cid:101) O ( ω + L max µ + ωL max nµ ) of DIANA method,we ﬁx the same level of compression with ω =
O ( n ) .
With a similar argument, this leads us to
(cid:101) O ( n + L max µ ) iteration complexity and O ( dn ) bits of communication per node in each iteration.
Whence, total communication complexity becomes (cid:101) O ( dn + dL max µ ) .
To compare the proposed methods with these baselines and highlight improvement factors, deﬁneparameters ν and ν 1 describing local smoothness matrices L i as follows
[Equation]
where L i = λ max ( L i ) , L max def = max 1 ≤ i ≤ n L
i
and
L i ; j is the j th diagonal element of matrix
L i .
Parameters ν ∈ [1 , n ] and ν 1 ∈ [1 , d ] describe the level of heterogeneity over the nodes andcoordinates respectively.
If L i matrices coincide, then ν = n and ν 1 = d .
On the other extreme,when the values of L i are extremely non-uniform, we have ν (cid:28) n and ν 1 (cid:28) d .
Notice that the quantity L max µn in (6) and the quantity ω max + L max µn in (7) depend on compressionoperators C ki applied by the nodes.
For the rest of the paper we are going to minimize these quantitieswith respect to the choice of C ki in such a way to minimize total communication complexity of theproposed distributed methods.
We specialize compressors C i to two different extensions of standardquantization and optimize with respect to compression parameters.
Block Quantization
We now present our ﬁrst extension to standard quantization in order to properly capture the matrixsmoothness information.
Instead of having a single quantization parameter (e.g. number of levels)for all coordinates, here we divide the space R d into B ∈ { 1 , 2 , . . .
, d } blocks as R d = R d 1 × R d 2 ×· · · × R d B and for each subspace R d l , l ∈
[ B ] we apply standard quantization independently fromother blocks with different number of levels s l .
Thus, for any l ∈ [ B ] we allocate one parameter s l forl th block of x ∈ R d .
Hence quantization is applied block-wise: for each block we send the norm (cid:107) x l (cid:107)of the block x l ∈ R d l and all entries within this block are quantized with levels { 0 , 1 s l , 2 s l , . . .
Inthe special case of B = 1 , we get the standard quantization of Alistarh et al.
To get rid of the constraints on s l to be integers, instead of working with the number of levels s l , weintroduce the size of the quantization step h l = 1 s l and allow them to take any positive values (evenbigger than 1 ).
Thus, for each block l ∈ [ B ] we quantize with respect to levels { 0 , h l , 2 h l , . . . } .
Deﬁnition 3 (Block Quantization) .
For a given number of blocks B ∈ [ d ] and ﬁxed quantizationsteps h = ( h 1 , . . .
, h B ) , deﬁne block-wise quantization operator Q
Bh : R d → R d as follows:
[Equation]
where t =
( l − 1) B + j, x ∈ R d , j ∈
[ d l ] , l ∈ [ B ] and ξ l ( v ) for v ≥ 0 is deﬁned via the quantizationlevels { 0 , h l , 2 h l , . . .
} as follows: if kh l ≤ v < ( k + 1) h l for some k ∈ { 0 , 1 , 2 , . . . } , then
[Equation]
Note that Q Bh is an unbiased compression operator as E [ ξ j ( v )]
= v for any v ≥ 0 .
To communicatea vector of the form Q Bh ( x ) , we encode each block (cid:2)
Q Bh ( x ) (cid:3) l ∈ R d l using Elias ω -coding as inthe standard quantization scheme [Alistarh et al., 2017].
Hence, for each block l ∈ [ B ] we need tosend (cid:101) O
( 1 h 2 l + √ d l h l ) bits and one ﬂoating point number for (cid:107) x l (cid:107) .
Overall, the number of encodingbits for Q Bh ( x ) (up to constant and log factors) can be given by (cid:80)
Bl =1 ( 1 h 2 l + √ d l h l ) + B .
As for thecompression noise, we prove in the Appendix the following upper bound for L ( Q Bh , L ) :
[Equation]
where L ll is the l th diagonal block matrix of L with sizes d l × d l .
Next, we are going to minimizecommunication complexity of DCGD+ and DIANA+ by optimizing parameters of block quantization.4.1.
DCGD+ with block quantization.
We ﬁx the number of blocks B ∈ [ d ] for all nodes i ∈
[ n ]and allow each node to apply different block quantization operator
Q Bh i with quantization stepsh i
=
( h i, 1 , . . .
, h i,B ) .
To minimize communication complexity of DCGD+, we need to minimize
L max subject to the communication constraint mentioned above.
Since L max = max i ∈ [ n ] L ( C i , L i ) , each node i ∈ [ n ] can minimize the impact of its own compression by minimizing L ( C i , L i ) basedon local smoothness matrix L i .
This leads to the following optimization problem for ﬁnding optimalvalues of h
i for each node i ∈ [ n ] :
[Equation]
where β is the “budget” of communication: Larger β leads to ﬁner quantization levels.
The solution tothis problem is given by h i,l =
[Equation]
i,B (cid:107) Diag ( L lli ) (cid:107) , where δ i,B ≥ 0 is uniquely determined by the constraintequality of (11) as the only positive solution of δ 2 i,B − δ i,B dT i,B β − B − dT 2 i, 1 β − B = 0 , which implies δ i,B =
[Equation]
T
[Equation]
where T i
[Equation]
B def [Equation]
[Equation]
L lli [Equation]
If this solution of quantization steps h i is used by all nodes i ∈
[ n ] , then we show reduction incommunication complexity by a factor of O ( n ) .
Theorem 3.
Assume n = O ( √ d ) and both ν, ν 1 are O (1) .
Then DCGD+ using block quantization with B = n blocks, d l = O ( d / n ) block sizes for all l ∈ [ n ] and quantization stepsh i,l = δ i,B / (cid:107)
Diag ( L lli ) (cid:107) with β =
O ( d / n ) reduces overall communication complexity by a factorof O ( n ) compared to DCGD using B = 1 single block quantization.
Formally, to guarantee ε > 0accuracy, the communication complexity of DCGD+ is O (cid:16) dn L max µ log 1 ε (cid:17) , which is O ( n ) timessmaller over DCGD.4.2.
DIANA+ with block quantization.
For the rate (7) of DIANA+, we need to optimize ω max +
L max nµ part of the complexity under the same communication constraint used in (11).
Since
[Equation]
we can decompose the problem into subproblems for each node i to optimize ω
i + L i nµ with respect toits own quantization parameters
h i .
Analogously, this leads to the following optimization problemfor ﬁnding optimal values of h i for each node i ∈ [ n ] :
[Equation]
which can be solved with a similar argument as done for (11).
Details are deferred to the Appendix.
Theorem 4.
Assume n = O ( √ d ) and both ν, ν 1 are O (1) .
Then DIANA+ using block quantizationwith B = n blocks, d l = O ( d / n ) block sizes for all l ∈ [ n ] and h i,l quantization steps (solution to(13) ) with β
= O ( d / n ) reduces overall communication complexity by a factor of O ( n ) comparedto DIANA using B = 1 single block quantization.
Formally, to guarantee ε > 0 accuracy, thecommunication complexity of DIANA+ is O (cid:0)(cid:0) nd + (cid:113) dn L max µ (cid:1) log 1 ε (cid:1) , which (ignoring n summandin the complexity) is O ( n ) times smaller over DIANA.
Quantization with Varying Steps
Our second extension of standard quantization scheme is to allow different quantization steps for allcoordinates { 1 , 2 , . . .
, d } .
In other words, for each coordinate j ∈ [ d ] we quantize with respect tolevels { 0 , h j , 2 h j , . . . } .
The standard quantization [Alistarh et al., 2017] is the special case whenh j = 1 s for all j ∈ [ d ] , where s is the number of quantization levels.
Deﬁnition 4 (Quantization with varying steps) .
For ﬁxed quantization steps h
= ( h 1 , . .
. , h d ) (cid:62) ∈ 244 R d , deﬁne quantization operator Q h : R d → R d as follows:
[Equation]
where ξ j is deﬁned via the quantization levels { 0 , h j , 2 h j , . .
as in (9).
Note that compression operator Q h is unbiased as E [ ξ j ( v )]
= v for any v ≥ {+
+}
and is not a specialcase of block quantization deﬁned earlier.
To understand how the number of encoding bits of Q h ( x ) depends on h exactly seems challenging, since it depends on the actual encoding scheme (i.e. binaryrepresentation of compressed information).
Besides, even if we ﬁx binary mapping, the closed formexpression of total amount of bits is too complicated to be utilized in the further analysis.
We providetheoretical arguments and clear numerical evidence that (cid:107) h − 1 (cid:107) =
(cid:113)(cid:80) dj
=1 h − 2 j is a reasonable proxyfor the number of encoding bits for compressor Q h .
Assumption 3.
For any input vector x ∈ R d and quantization steps h ∈ R d , compressed vector
Q h ( x ) can be encoded with O ( (cid:107) h − 1 (cid:107) ) number of bits.
First, consider the special case when all quantization steps are the same, i.e. h j = 1 s .
Then(cid:107)
h − 1 (cid:107) =
s √ d recovers the dominant part (provided s = O ( √ d ) ) in (cid:101)
O ( s 2 + s √ d ) showing totalamount of bits for standard quantization scheme.
Second, in the Appendix we present an encodingscheme which (up to constant and log d factors) requires E [ ψ ( (cid:107) ˆ x (cid:107) 0 )]
+ (cid:107) h − 1 (cid:107) number of bits inexpectation to communicate ˆ x = Q h ( x ) , where ψ ( τ ) def = dH 2 ( τ / d ) + τ ≤ d log 3 , if τ ∈ [0 , d ] and
H 2 is the binary entropy function.
Note that, based on the deﬁnition (9), increasing quantization stepsh j forces more sparsity in ˆ x and hence reduces (cid:107) ˆ x (cid:107) 0 .
Thus, (cid:107) ˆ x (cid:107) 0 and hence ψ ( (cid:107) ˆ x (cid:107) 0 ) (notice thatψ (0) = 0 ) are proportional to (cid:107) h − 1 (cid:107) .
Furthermore, we present a numerical experiment which showsthat the number of encoding bits of Q h ( x ) and (cid:107) h − 1 (cid:107) are positively correlated.
Hence, in the further analysis, we ﬁx the number of encoding bits of Q h ( x ) by the constraint(cid:107)
h − 1 (cid:107) =
β for some parameter β > 0 .
As for the variance induced by the compression operator Q h ,we prove the following upper bound for L ( Q h , L ) :
[Equation]
DCGD+ with varying quantization steps.
Now, we optimize the rate (6) of DCGD+ withrespect to quantization steps h i
= ( h i ;1 , h i ;2 , . . .
, h i ; d ) of compressor Q h i controlled by i th nodefor all i ∈ [ n ] .
The term in (6) affected by the compression is L max = max i ∈ [ n ] L ( C i , L i ) , whichimplies that each node i ∈ [ n ] can minimize the impact of its own compression by minimizing
L ( C i , L i ) based on local smoothness matrix L i .
Based on the upper bound (14) and communicationconstraint given by (cid:107) h − 1 (cid:107) =
β for some β > 0 , we get the following optimization problem to choosethe optimal quantization parameters
h i for node i ∈ [ n ] :
[Equation]
This problem has the following closed form solution due to KKT conditions (see Appendix):
[Equation]
[Equation]
With this choice of quantization steps we save O (min( n, d )) times in communication.
Theorem 5.
Assume both ν, ν 1 are O (1) and
β =
O ( d / n ) .
Then DCGD+ using quantizationwith varying steps (25) for all i ∈ [ n ] reduces overall communication complexity by a factor of
O (min( n, d ))
compared to the baseline of DCGD.
Formally, the iteration complexity (6) can beupper bounded as Lµ + L max nµ ≤ [Equation] L max µ
[Equation]
µ (cid:17) , which is min( n, d )times smaller than the one for DCGD.
As both methods communicate O ( d / n ) bits per node periteration, we get min( n, d ) times savings in communication complexity.5.2.
DIANA+ with varying quantization steps.
Based on (12), each node i ∈ [ n ] optimizes ω i + L i nµwith respect to its quantization parameters
h i , which is equivalent to the problem
[Equation]
17) where A ij def = L i ; j / nµ .
Due to the KKT conditions (see Appendix), we get the following solution
[Equation]
With this choice of quantization steps we save O (min( n, d )) times in communication.
Theorem 6.
Assume both ν, ν 1 are O (1) and
β =
O ( d / n ) .
Then DIANA+ using quantizationwith varying steps (18) for all i ∈ [ n ] reduces overall communication complexity by a factor of
O (min( n, d ))
compared to the baseline of DIANA.
Formally, the iteration complexity (7) can beupper bounded as ω max + Lµ +
[Equation]
,which is min( n, d ) times smaller than the one for DIANA (ignoring negligible term n ).
Experiments
Setup.
In this section we present two key experiments.
Additional experiments can be found in the Appendix.
We conduct a range of experiments with several datasets from the LibSVM repository [Chang and Lin, 2011] on the (cid:96) 2 -regularized logistic regression problem (1):
[Equation]
where A i,t are data points sorted based on their norms before allocating to local workers for theheterogeneity.
The experiments are performed on a workstation with Intel(R)
Xeon(R)
Gold {+
+} CPU @ 3.30GHz cores.
The gather and broadcast operations for the communications betweenmaster and workers are implemented based on the MPI4PY library [Dalcín et al., 2005] and each CPU core is treated as a local worker.
For each dataset, we run each algorithm multiples times with 5random seeds for each worker.
Due to space limitations, we present only two of our experiments heredeferring the remaining experiments along with experimental details in the Appendix.6.2.
Comparison to standard quantization techniques.
In our ﬁrst experiment, we comparesmoothness-aware DCGD+ and DIANA+ methods with our varying-step quantization technique( quant+ ) to the original DCGD [Khirirat et al., 2018] and DIANA [Mishchenko et al., 2019]
methodswith the standard quantization technique ( quant ) of Alistarh et al.
Figure 1 demonstratesthat DCGD+/DIANA+ with quant+ lead to signiﬁcant improvement in both transmitted megabytesand wall-clock time.
An ablation study to disentangle the contributions of exploiting the smoothnessmatrix and utilizing varying number of levels can be found in Appendix B.
[Figure]
∈ R d × d and quantization steps h i ∈ R
d to the before the training.
This costs communication bits
Comparison to matrix-smoothness-aware sparsiﬁcation.
Second experiment is devoted tothe performance of three smoothness-aware compression techniques —block quantization ( blockquant+ ) of Section 4, varying-step quantization ( quant+ ) of Section 5 and smoothness-awaresparsiﬁcation strategy ( rand- τ + ) of Safaryan et al.
All three compression techniques areshown to outperform the standard compression strategies by at most O ( n ) times in theory.
Forthe sparsiﬁcation, we use the optimal probabilities and the sampling size τ = d/n as suggestedin Section 5.3 of [Safaryan et al., 2021].
The empirical results in Figure 2 (in appendix) illustratethat the varying-step quantization technique ( quant+ ) is always better than the smoothness-awaresparsiﬁcation [Safaryan et al., 2021], in terms of both communication cost and wall-clock time.
Our block quantization technique also beats sparsiﬁcation when the dimension of the model isrelatively high.