P ERMUTATION C OMPRESSORS FOR P ROVABLY F ASTER D ISTRIBUTED N ONCONVEX O PTIMIZATION
A BSTRACT
We study the MARINA method of Gorbunov et al.
(2021) – the current state-of-theart distributed non-convex optimization method in terms of theoretical communication complexity.
Theoretical superiority of this method can be largely attributed to two sources: the use of a carefully engineered biased stochastic gradient estimator, which leads to a reduction in the number of communication rounds, and the reliance on independent stochastic communication compression operators, which leads to a reduction in the number of transmitted bits within each communication round.
In this paper we i) extend the theory of MARINA to support a much wider class of potentially correlated compressors, extending the reach of the method beyond the classical independent compressors setting, ii) show that a new quantity, for which we coin the name Hessian variance , allows us to signiﬁcantly reﬁne the original analysis of MARINA without any additional assumptions, and iii) identify a special class of correlated compressors based on the idea of random permutations , for which we coin the term Perm K .
The use of it leads to O ( √ n ) (resp.
O (1 + d/ √ n ) ) improvement in the theoretical communication complexity of MARINA in the low Hessian variance regime when d ≥ n (resp.
d ≤ n ), where n is the number of workers and d is the number of parameters describing the model we are learning.
We corroborate our theoretical results with carefully engineered synthetic experiments with minimizing the average of nonconvex quadratics, and on autoencoder training with the MNIST dataset.
I NTRODUCTION
The practice of modern supervised learning relies on highly sophisticated, high dimensional and data hungry deep neural network models (Vaswani et al., 2017; Brown et al., 2020) which need to be trained on specialized hardware providing fast distributed and parallel processing.
Training of such models is typically performed using elaborate systems relying on specialized distributed stochastic gradient methods (Gorbunov et al., 2021).
In distributed learning, communication among the compute nodes is typically a key bottleneck of the training system, and for this reason it is necessary to employ strategies alleviating the communication burden.
1. T HE PROBLEM AND ASSUMPTIONS
Motivated by the need to design provably communication efﬁcient distributed stochastic gradient methods in the nonconvex regime, in this paper we consider the optimization problem
[Equation]
where n is the number of workers/machines/nodes/devices working in parallel, and f i : R d → R is a (potentially nonconvex ) function representing the loss of the model parameterized by weights x ∈ R d on training data stored on machine i .
While we do not assume the functions { f i } to be convex, we rely on their differentiability, and on the well-posedness of problem (1):
Assumption 1.
The functions f 1 , . .
, f n : R d → R are differentiable.
Moreover, f is lower bounded, i.e., there exists f inf ∈ R such that f ( x ) ≥ f inf for all x ∈ R d .
We are interested in ﬁnding an approximately stationary point of the nonconvex problem (1).
That is, we wish to identify a (random) vector ˆ x ∈ R d such that
[Equation]
while ensuring that the volume of communication between the n workers and the server is as small as possible.
Without the lower boundedness assumption there might not be a point with a small gradient (e.g., think of f being linear), which would render problem (2) unsolvable.
However, lower boundedness ensures that the problem is well posed.
Besides Assumption 1, we rely on the following smoothness assumption:
Assumption 2.
There exists a constant L + > 0 such that 1 n (cid:80) ni
[Equation]
1 (cid:107)∇
[Equation]
L 2+ (cid:107) x − y (cid:107) 2 for all x, y ∈ R d .
To avoid ambiguity, let L + be the smallest such number.
While this is a somewhat stronger assumption than mere L − -Lipschitz continuity of the gradient of f (the latter follows from the former by Jensen’s inequality and we have L − ≤ L + ), it is weaker than L i -Lipschitz continuity of the gradient of the functions
f i (the former follows from the latter with L 2+ ≤ 1 n (cid:80) i
L 2 i ).
So, this is still a reasonably weak assumption.
1. A BRIEF OVERVIEW OF THE STATE OF THE ART
To the best of our knowledge, the state-of-the-art distributed method for ﬁnding a point ˆ x satisfying (2) for the nonconvex problem (1) in terms of the theoretical communication complexity 1 is the MARINA method of Gorbunov et al.
(2021) (see Algorithm 1).
MARINA relies on worker-to-server communication compression , and its power resides in the construction of a carefully designed sequence of biased gradient estimators which help the method obtain its superior communication complexity.
t + x t − γg
∇ t
=
g t + C
i
∇ f i t
− ∇ f i
The method uses randomized compression operators
C i : R d → R d to compress messages (gradient differences) at the workers i ∈ { 1 , 2 , . . . , n } before they are communicated to the server.
It is assumed that these operators are unbiased, i.e., E [ C i ( a )]
= a for all a ∈ R d , and that their variance is bounded as
[Equation]
for all a ∈ R d and some ω ≥ 0 .
For convenience, let U ( ω ) be the class of such compressors.
A key assumption in the analysis of MARINA is the independence of
In particular, MARINA solves the problem (1)–(2) in
[Equation]
communication rounds 2 , where ∆ 0 :
= f
( x 0 ) − f inf , x 0 ∈ R
d is the initial iterate, p ∈ (0 , 1] is a parameter deﬁning the probability with which full gradients of the local functions { f i } are communicated to the server, L − > 0 is the Lipschitz constant of the gradient of f , and L + ≥ L − is a certain smoothness constant associated with the functions { f i } .
In each iteration of MARINA , all workers send (at most) pd + (1 − p ) ζ ﬂoats to the server in expectation, where ζ := max i sup v ∈ R d size( C i ( v )) , where size( C i ( v )) is the size of the message v compressed by compressor C i .
For an uncompressed vector v we have size( v ) =
d in the worst case, and if C i is the Rand K sparsiﬁer, then size( C i ( v ))
=
K .
Putting the above together, the communication complexity of MARINA is T ( pd + (1 − p ) ζ ) , i.e., the product of the number of communication rounds and the communication cost of each round.
See Section B for more details on the method and its theoretical properties.
An alternative to the application of unbiased compressors is the practice of applying contractive compressors, such as Top K (Alistarh et al., 2018), together with an error feedback mechanism (Seide et al., 2014; Stich et al., 2018; Beznosikov et al., 2020).
However, this approach is not competitive in theoretical communication complexity with MARINA ; see Appendix G for details.
1. S UMMARY OF CONTRIBUTIONS
(a) Correlated and permutation compressors.
We generalize the analysis of MARINA beyond independence by supporting arbitrary unbiased compressors, including compressors that are correlated .
In particular, we construct new compressors based on the idea of a random permutation (we called them Perm K ) which provably reduce the variance caused by compression beyond what independent compressors can achieve.
The properties of our compressors are captured by two quantities,
A ≥ B ≥ 0 , through a new inequality (which we call “AB inequality") bounding the variance of the aggregated (as opposed to individual) compressed message.
(b) Reﬁned analysis through the new notion of Hessian variance.
We reﬁne the analysis of
MARINA by identifying a new quantity, for which we coin the name Hessian variance , which plays an important role in our sharper analysis.
To the best of our knowledge, Hessian variance is a new quantity proposed in this work and not used in optimization before.
This quantity is well deﬁned under the same assumptions as those used in the analysis of MARINA by Gorbunov et al.
(c) Improved communication complexity results.
We prove iteration complexity and communication complexity results for MARINA , for smooth nonconvex (Theorem 4) and smooth PolyakŁojasiewicz 3 (Theorem 5) functions.
Our results hold for all unbiased compression operators, including the standard independent but also all correlated compressors.
Most importantly, we show that in the low Hessian variance regime, and by using our Perm K compressors, we can improve upon the current state-of-the-art communication complexity of MARINA due to Gorbunov et al.
(2021) by up to the factor √ n in the d ≥ n case, and up to the factor 1 + d/ √ n in the d ≤ n case.
The improvement factors degrade gracefully as Hessian variance grows, and in the worst case we recover the same complexity as those established by Gorbunov et al.
(d) Experiments agree with our theory.
Our theoretical results lead to predictions which are corroborated through computational experiments.
In particular, we perform proof-of-concept testing with carefully engineered synthetic experiments with minimizing the average of nonconvex quadratics, and also test on autoencoder training with the MNIST dataset.
B EYOND I NDEPENDENCE : T HE P OWER OF C ORRELATED C OMPRESSORS
As mentioned in the introduction, MARINA was designed and analyzed to be used with compressors
C i ∈ U ( ω ) that are sampled independently by the workers.
For example, if the Rand K sparsiﬁcation operator is used by all workers, then each worker chooses the K random coordinates to be communicated independently from the other workers.
This independence assumption is crucial for MARINA to achieve its superior theoretical properties.
Indeed, without independence, the rate would depend on ω instead 4 of ω / n , which would mean no improvement as the number n of workers grows, which is problematic because ω is typically very large.
For this reason, independence is assumed in the analysis of virtually all distributed methods that use unbiased communication compression, including methods designed for convex or strongly convex problems (Khirirat et al., 2018;
Mishchenko et al., 2019; Li et al., 2020; Philippenko & Dieuleveut, 2020).
In our work we ﬁrst generalize the analysis of MARINA beyond independence , which provably extends its use to a much wider array of (still unbiased) compressors, some of which have interesting theoretical properties and are useful in practice.
2. AB INEQUALITY : A TOOL FOR A MORE PRECISE CONTROL OF COMPRESSION VARIANCE
We assume that all compressors {C i } ni =1 are unbiased, and that there exist constants A, B ≥ 0 for which the compressors satisfy a certain inequality, which we call “AB inequality”, bounding the variance of 1 n (cid:80)
i C i ( a i ) as a stochastic estimator of 1 n (cid:80) i a i .
Assumption 3 (Unbiasedness) .
The random operators C 1 , . .
, C n : R d → R d are unbiased, i.e.,
E [ C i ( a )]
= a for all i ∈ { 1 , 2 , . . . , n } and all a ∈ R d .
Assumption 4 (AB inequality) .
There exist constants A, B ≥ 0 such that the random operators
C 1 , . .
, C n : R d → R d satisfy the inequality
[Equation]
[Table]
(3) for all a 1 , . . . , a n ∈ R d .
If these conditions are satisﬁed, we will write {C i } ni =1 ∈ U ( A, B ) .
It is easy to observe that whenever the AB inequality holds, it must necessarily be the case that
A ≥ B .
Indeed, if we ﬁx nonzero a ∈ R d and choose a i = a for all i , then the right hand side of the AB inequality is equal to A − B while the left hand side is nonnegative.
Our next observation is that whenever C i ∈ U ( ω i ) for all i ∈ { 1 , 2 , . . . , n } , the AB inequality holds without any assumption on the independence of the compressors.
Furthermore, if independence is assumed, the A constant is substantially improved.
Lemma 1.
If C i ∈ U ( ω i ) for i ∈ { 1 , 2 , . . . , n } , then {C i } ni =1 ∈ U (max i ω i , 0) .
If we further assume that the compressors are independent, then {C i } ni =1 ∈ U ( 1 n max i ω i , 0) .
In Table 1 we provide a list of several compressors that belong to the class
U ( A, B ) , and give values of the associated constants A and B .
While in the two examples captured by Lemma 1 we had
B = 0 , intuitively, we should want B to be as large as possible.
2. I NPUT VARIANCE COMPRESSORS
Due to the above considerations, compressors for which
A =
B are special, and their construction and theoretical properties are a key contribution of our work.
Moreover, as we shall see in Section 4, such compressors have favorable communication complexity properties.
This leads to the following deﬁnition:
Deﬁnition 1 (Input variance compressors) .
We say that a collection {C i } ni =1 of unbiased operators form an input variance compressor system if the variance of 1 n (cid:80) i C
i ( a i ) is controlled by a multiple of the variance of the input vectors { a i } ni =1 .
That is, if there exists a constant C ≥
such that
[Equation]
for all a 1 , . . .
, a n ∈ R d .
If these conditions are satisﬁed, we will write {C i } ni =1 ∈ IV ( C ) .
If {C i } ni =1 ∈ U ( A, B ) and A = B , then {C i } ni =1 ∈ IV ( A ) .
[Table]
2. P ERM K : PERMUTATION BASED SPARSIFIERS
We now deﬁne two input variance compressors based on a random permutation construction.
The ﬁrst compressor handles the d ≥ n case, and the second handles the d ≤ n case.
For simplicity of exposition, we assume that d is divisible by n in the ﬁrst case, and that n is divisible by d in the second case.
6 Since both these new compressors are sparsiﬁcation operators, in an analogy with the established notation Rand K and Top K for sparsiﬁcation, we will write Perm K for our permutationbased sparsiﬁers.
To keep the notation simple, we chose to include simple variants which do not offer freedom in choosing K .
Having said that, these simple compressors lead to state-of-the-art communication complexity results for MARINA , and hence not much is lost by focusing on these examples.
Let e i be the i th standard unit basis vector in R d .
That is, for any x = ( x 1 , . . .
, x d ) ∈ R
d we have x = (cid:80) i
x i e i .
Deﬁnition 2 (Perm K for d ≥ n ) .
Assume that d ≥ n and d = qn , where q ≥ 1 is an integer.
Let π =
( π 1 , . . .
, π d ) be a random permutation of { 1 , . . .
, d } .
Then for all x ∈ R d and each i ∈ { 1 , 2 , . . . , n } we deﬁne
[Equation]
Note that C i is a sparsiﬁer: we have ( C i ( x ))
l = nx l if l ∈ { π j : q ( i − 1) + 1 ≤ j ≤ qi } and ( C i ( x ))
l = 0 otherwise.
So, (cid:107)C i ( x ) (cid:107) 0 ≤ q := K , which means that C i offers compression by the factor n .
Note that we do not have ﬂexibility to choose K ; we have K = q = d/n .
See Appendix J for implementation details.
Theorem 1.
The Perm K compressors from Deﬁnition 2 are unbiased and belong to IV (1) .
In contrast with the collection of independent Rand K sparsiﬁers, which satisfy the AB inequality with A = d/K − 1 n and B =
(this follows from Lemma 1 since ω i =
d/K − 1 for all i ),
Perm K satisﬁes the AB inequality with A = B = 1 .
While both are sparsiﬁers, the permutation construction behind Perm K introduces a favorable correlation among the compressors: we have (cid:104)C i ( a i ) , C j ( a j ) (cid:105) =
for all i (cid:54)
=
j .
Deﬁnition 3 (Perm K for n ≥ d ) .
Assume that n ≥ d, n > 1 and n = qd, where q ≥ 1 is an integer.
Deﬁne the multiset S :
= { 1 , . . .
, d, . . .
, d } , where each number occurs precisely q times.
Let π =
( π 1 , . . .
, π n ) be a random permutation of S .
Then for all x ∈ R d and each i ∈ { 1 , 2 , . . . , n } we deﬁne
[Equation]
Note that for each i , C i from Deﬁnition 3 is the Rand 1 sparsiﬁer, offering compression factor d .
However, the sparsiﬁers {C i } ni =1 are not mutually independent.
Note that, again, we do not have a choice of K in Deﬁnition 3: we have K = 1 .
Theorem 2.
The Perm K compressors from Deﬁnition 3 are unbiased and belong to IV ( A ) with
A = 1 − n − d n − 1 .
Combining Perm K with quantization.
It is easy to show that if {C i } n i =1 ∈ U ( A, B ) , and Q i ∈ U ( ω i ) are chosen independently of {C i } ni =1 (we do not require mutual independence of {Q i } ), then {C i ◦ Q i } ni
=1 ∈ U ((max i ω
i + 1) A, B ) (see Lemma 9).
This allows us to combine our compression techniques with quantization (Alistarh et al., 2017; Horváth et al., 2019).
[Table]
H ESSIAN V ARIANCE
Working under the same assumptions on the problem (1)–(2) as Gorbunov et al.
(2021) (i.e., Assumptions 1 and 2), in this paper we study the complexity of MARINA under the inﬂuence of a new quantity, which we call Hessian variance.
Deﬁnition 4 (Hessian variance) .
Let L ± ≥ 0 be the smallest quantity such that
[Equation]
We refer to the quantity L 2 ± by the name Hessian variance.
Recall that in this paper we have so far mentioned four “smoothness” constants: L i (Lipschitz constant of ∇ f i ), L − (Lipschitz constant of ∇ f ), L + (see Assumption 2) and L ± (Deﬁnition 4).
To avoid ambiguity, let all be deﬁned as the smallest constants for which the deﬁning inequalities hold.
In case the deﬁning inequality does not hold, the value is set to + ∞ .
This convention allows us to formulate the following result summarizing the relationships between these quantities.
Lemma 2.
[Equation]
It follows that if L i is ﬁnite for all i , then L − , L + and L ± are all ﬁnite as well.
Similarly, if L + is ﬁnite (i.e., if Assumption 2 holds), then L − and L ± are ﬁnite, and L ± ≤ L + .
We are not aware of any prior use of this quantity in the analysis of any optimization methods.
Importantly, there are situations when L − is large, and yet the Hessian variance L 2 ± is small, or even zero.
This is important as the improvements we obtain in our analysis of MARINA are most pronounced in the regime when the Hessian variance is small.
We also wish to stress that the ﬁniteness of L ± is not an additional assumption – it follows directly from Assumption 2.
3. H ESSIAN VARIANCE CAN BE ZERO
We now illustrate on a few examples that there are situations when the values of L − and L i are large and the Hessian variance is zero.
The simplest such example is the identical functions regime.
Example 1 (Identical functions) .
Assume that f 1 = f 2 = · · · = f n .
Then L ± = 0 .
This follows by observing that the left hand side in (7) is zero.
Note that while L ± = 0 , it is possible for L − and L + to be arbitrarily large!
Note that methods based on the Top K compressor (including all error feedback methods) suffer in this regime.
Indeed, EF21 in this simple scenario is the same method for any value of n , and hence can’t possibly improve as n grows.
This is because when a i = a j for all i, j , 1 n (cid:80) i
Top K ( a i ) =
Top K ( a i ) .
As the next example shows, Hessian variance is zero even if we perturb the local functions via arbitrary linear functions.
Example 2 (Identical functions + arbitrary linear perturbation) .
Assume that f i ( x ) =
φ ( x )+ b (cid:62) i x
+ c i , for some differentiable function φ : R d → R and arbitrary b i ∈ R d and c i ∈ R .
Then L ± = 0 .
This follows by observing that the left hand side in (7) is zero in this case as well.
Note that in this example it is possible for the functions { f i } to have arbitrarily different minimizers .
So, this example does not correspond to the overparameterized machine learning regime, and is in general challenging for standard methods.
3. S ECOND ORDER CHARACTERIZATION
To get an insight into when the Hessian variance may be small but not necessarily zero, we establish a useful second order characterization.
Theorem 3.
Assume that for each i ∈ { 1 , 2 , . . . , n } , the function f
i is twice continuously differentiable.
Fix any x, y ∈ R d and deﬁne H i ( x, y ) :
= (cid:82) 1 0 ∇ 2 f i ( x + t ( y − x )) dt,
H ( x, y ) :
= 1 n n (cid:80) i =1 H i ( x, y ) .
Then the matrices L i ( x, y ) :
=
H 2 i ( x, y ) , L − ( x, y ) :
=
H 2 ( x, y ) , L + ( x, y ) :
= 1 n (cid:80) ni
=1 H 2 i ( x, y ) and L ± ( x, y ) :
=
L + ( x, y ) − L − ( x, y ) are symmetric and positive semideﬁnite.
Moreover,
[Equation]
x,y )( x − y )
(cid:107) x − y (cid:107).
While L 2 ± is obviously well deﬁned through Deﬁnition 4 even when the functions { f i } are not twice differentiable, the term “Hessian variance” comes from the interpretation of L 2 ± in the case of quadratic functions.
Example 3 (Quadratic functions) .
Let f i ( x ) =
12 x (cid:62)
A
i x + b (cid:62) i x + c i , where A i ∈ R
d × d are symmetric.
Then L 2 ± = λ max ( 1 n (cid:80) ni
=
A 2 i − (cid:0) 1 n (cid:80) ni
=1 A i ) 2 (cid:1) , where λ max ( · ) denotes the largest eigenvalue.
Indeed, note that the matrix 1 n (cid:80) ni
=
A 2 i − (cid:0) 1 n (cid:80) ni
=1 A i (cid:1) 2 can be interpreted as a matrix-valued variance of the Hessians A 1 , . . .
, A n , and L 2 ± measures the size of this matrix in terms of its largest eigenvalue.
See Table 2 for a summary of the examples mentioned above.
As we shall explain in Section 4, the data/problem regime when the Hessian variance is small is of key importance to the improvements we obtain in this paper.
I MPROVED I TERATION AND C OMMUNICATION C OMPLEXITY
The key contribution of our paper is a more general and more reﬁned analysis of MARINA .
In particular, we i) extend the reach of MARINA to the general class of unbiased and possibly correlated compressors {C i } n i =1 ∈ U
( A, B )
while ii) providing a more reﬁned analysis in that we take the
Hessian variance L 2 ± into account.
Theorem 4.
Let Assumptions 1, 2, 3 and 4 be satisﬁed.
Let the stepsize in MARINA be chosen as 0 < γ ≤ 1 M , where
M =
L − + (cid:113) 1 − pp (cid:0) ( A − B )
L 2+ + BL 2 ± (cid:1) .
Then after T iterations, MARINA ﬁnds a random point ˆ x T ∈ R d for which
[Equation]
In particular, by choosing the maximum stepsize allowed by Theorem 4, MARINA converges in T communication rounds, where T is shown in the ﬁrst row Table 3.
If in this result we replace L 2 ± by the coarse estimate L 2 ± ≤ L 2+ , and further specialize to independent compressors satisfying
C i ∈ U ( ω ) for all i ∈ { 1 , 2 , . . . , n } , then since {C i } ni =1 ∈ U ( ω/n, 0) (recall Lemma 1), our general rate specializes to the result of Gorbunov et al.
(2021), which we show in the second row of Table 3. [Table] [Table]
However, and this is a key ﬁnding of our work, in the regime when the Hessian variance L 2 ± is very small, the original result of Gorbunov et al.
(2021) can be vastly suboptimal!
To show this, in
Table 4 we compare the communication complexity , i.e., the # of communication rounds multiplied by the maximum # of ﬂoats transmitted by a worker to the sever in a single communication round.
We compare the communication complexity of MARINA with the Rand K and Perm K compressors, and the state-of-the-art error-feedback method EF21 of Richtárik et al.
(2021) with the Top K compressor.
In all cases we do not consider the communication complexity of the initial step equal to
O ( d ) .
In each case we optimized over the parameters of the methods (e.g., p for MARINA and K in all cases; for details see Appendix L).
Our results for MARINA with Perm K are better than the competing methods (recall Lemma 2).
4. I MPROVEMENTS IN THE IDEAL ZERO -H ESSIAN - VARIANCE REGIME
To better understand the improvements our analysis provides, let us consider the ideal regime characterized by zero Hessian variance:
L 2 ± = 0 .
If we now use compressors {C i } ni =1 ∈ U ( A, B ) for which A = B , which is the case for Perm K , then the dependence on the potentially very large quantity L 2+ is eliminated completely.
Big model case ( d ≥ n ).
In this case, and using the Perm K compressor, MARINA has communication complexity
O
( L − ∆ 0 ε − 1 d / n ) , while using the Rand K compressor, the communication complexity of MARINA is no better than O
( L − ∆ 0 ε − 1 d / √ n ) .
Hence, we get an improvement by at least the factor √ n .
Moreover, note that this is an n × improvement over gradient descent ( GD ) (Khaled & Richtárik, 2020) and EF21 , both of which have communication complexity O
( L − ∆ 0 ε − 1 d ) .
In
Appendix M, we discuss how we can get the same theoretical improvement even if L 2 ± > 0 .
Big data case ( d ≤ n ).
In this case, and using the Perm K compressor, MARINA achieves communication complexity O
( L − ∆ 0 ε − 1 ) , while using the Rand K compressor, the communication complexity of MARINA is no better than O
( L − ∆ 0 ε − 1 (1 + d / √ n )) .
Hence, we get an improvement by at least the factor 1 + d/ √ n .
Moreover, note that this is a d × improvement over gradient descent ( GD ) and EF21 , both of which have communication complexity
O
( L − ∆ 0 ε − 1 d ) .
E XPERIMENTS
We compare MARINA using Rand K and Perm K , and EF21 with Top K , in two experiments.
In the ﬁrst experiment, we construct quadratic optimization tasks with different L ± to capture the dependencies that our theory predicts.
In the second experiment, we consider practical machine learning task MNIST (LeCun et al., 2010) to support our assertions.
Each plot represents the dependence between the norm of gradient (or function value) and the total number of transmitted bits by a node.
5. T ESTING THEORETICAL PREDICTIONS ON A SYNTHETIC QUADRATIC PROBLEM
To test the predictive power of our theory in a controlled environment, we ﬁrst consider a synthetic (strongly convex) quadratic function f = 1 n (cid:80)
f i composed of nonconvex quadratics f
i ( x ) := 12 x (cid:62) A i x − x (cid:62) b i , where b i ∈ R d , A i ∈ R d × d , and A i =
A (cid:62) i .
We enforced that f is λ –strongly convex, i.e., 1 n (cid:80) ni
=
A i (cid:60) λ I for λ > 0 .
We ﬁx λ = 1e − 6 , and dimension d = 1000 (see Figure 1).
We then generated optimization tasks with the number of nodes n ∈ { 10 , 1000 , 10000 } and
L ± ∈ { 0 , 0 . 05 , 0 . 1 , 0 . 21 , 0 . 91 } .
We take MARINA ’s and EF21 ’s parameters prescribed by the theory and performed a grid search for the step sizes for each compressor by multiplying the theoretical ones with powers of two.
For simplicity, we provide one plot for each compressor with the best convergence rate.
First, we see that Perm K outperforms Rand K , and their differences in the plots reproduce dependencies from Table 4.
Moreover, when n ∈ { 1000 , 10000 } and L ± ≤ 0 .
21 , EF with Top K has worse performance than MARINA with Perm K , while in heterogeneous regime, when
L ± = 0 .
91 , Top K is superior except when n = 10000 .
See Appendix A for detailed experiments.
[Figure]
5. T RAINING AN AUTOENCODER WITH MNIST
Now we compare compressors from Section 5.1 on the MNIST dataset (LeCun et al., 2010).
Our current goal is to learn the linear autoencoder, f ( D , E ) :
= 1 N N (cid:80) i =1 (cid:107) DE
a i − a i (cid:107) 2 , where D ∈
R d f × d e , E ∈ R
d
e × d f , a i ∈ R d f are MNIST images, d f = 784 is the number of features, d e =is the size of encoding space.
Thus the dimension of the problem d = 25088 , and compressors send at most 26 ﬂoats in each communication round since we take n = 1000 .
We use parameter (cid:98) p to control the homogeneity of MNIST split among n nodes: if (cid:98) p
= 1 , then all nodes store the same data, and if (cid:98) p = 0 , then nodes store different splits (see Appendix A.5).
In Figure 2, one plot for each compressor with the best convergence rate is provided for (cid:98) p ∈ { 0 , 0 . 5 , 0 .
We choose parameters of algorithms prescribed by the theory except for the step sizes, where we performed a grid search as before.
In all experiments, Perm K outperforms Rand K .
Moreover, we see that in the more homogeneous regimes, when (cid:98) p ∈ { 0 . 9 , 1 . 0 } , Perm K converges faster than Top K .
When (cid:98) p = 0 .
75 , both compressors have almost the same performance.
In the heterogenous regime, when (cid:98) p ∈ { 0 , 0 . 5 } , Top K is faster than Perm K , but they both signiﬁcantly outperform Rand K .
[Figure]
C ONTENTS 1 Introduction
The problem and assumptions . . . . . . . . . . . . . . . . . . . . . . . . . . . .
A brief overview of the state of the art . . . . . . . . . . . . . . . . . . . . . . . .
Summary of contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2 Beyond Independence: The Power of Correlated Compressors 2.
AB inequality: a tool for a more precise control of compression variance . . . . . .
Input variance compressors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Perm K : permutation based sparsiﬁers . . . . . . . . . . . . . . . . . . . . . . . .
3 Hessian Variance 3.
Hessian variance can be zero . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Second order characterization . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Improved Iteration and Communication Complexity
Improvements in the ideal zero-Hessian-variance regime . .
5 Experiments 5.
Testing theoretical predictions on a synthetic quadratic problem . .
Training an autoencoder with MNIST . .
A Extra Experiments
A.1 Experiments setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
B MARINA Algorithm
C Missing Proofs
C.1 Proof of Lemma 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
C.2 Proof of Lemma 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
C.3 Proof of Theorem 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
C.4 Proof of Theorem 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
C.5 Proof of Theorem 3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
C.6 Proof of Theorem 4 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
D Polyak-Łojasiewicz Analysis
D.1 Proof of Lemma 6 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
D.2 Proof of Theorem 5 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
E EF21 Analysis
[Table] [Table]
F Communication Model
G On Contractive Compressors and Error Feedback
G.1 On Contractive Compressors . . . . . . .
G.2 On Error Feedback . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
H Composition of Compressors with AB Assumption and Unbiased Compressors
I General Examples of Perm K
I.
Case d ≥ n . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
I.
Case n ≥ d . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
J Implementation Details of Perm K
K More
Examples of Permutation
-Based
Compressors
K.1 Block permutation compressor . .
K.2 Permutation of mappings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
L Analysis of Complexity Bounds
L.1 Nonconvex optimization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
L.1.
Case n ≤ d . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
L.1.
Case n ≥ d . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
L.2 PŁ assumption . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
L.2.
Case n ≤ d . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
L.2.
[Table]
n ≥ d . . .
[Table]
A E XTRA E XPERIMENTS
In this section, we provide more detailed experiments and explanations.
A. E XPERIMENTS SETUP
All methods are implemented in Python 3.6 and run on a machine with 24 Intel(R)
Xeon(R)
Gold 6146 CPU @ 3.20GHz cores with 32-bit precision.
Communication between master and nodes is emulated in one machine.
In all experiments, we compare MARINA algorithm with Rand K compressor and Perm K compressor and EF21 with Top K .
In Rand K and Top K , we take K =
(cid:100)
d/n (cid:101) ; we show in Lemmathat K
=
(cid:100) d/n (cid:101) is optimal for Rand K .
For Top K , the optimal rate predicted by the current stateof-the-art theory is obtained when K = d (however, in practice, Top K works much better when
K (cid:28) d ).
Lastly, we make the pessimistic assumption that L 2 ± and L 2+ are equal to their upper bound 1 n (cid:80) ni
=1 L 2 i .
A. E XPERIMENT WITH QUADRATIC OPTIMIZATION TASKS : FULL DESCRIPTION
First, we present Algorithm 2 which is used in the experiments of Section 5.1.
The algorithm is designed to generate sparse quadratic optimization tasks where we can control L ± using the noise scale.
Furthermore, it can be seen that the procedure generates strongly convex quadratic optimization tasks; thus, all assumptions from this paper are fulﬁlled to use theoretical results.
1: Parameters: number nodes n , dimension d , regularizer λ , and noise scale s .
2: for i = 1 , . . .
, n do 3:
Generate random noises ν si = 1 + sξ si and ν bi = sξ bi , i.i.d.
ξ si , ξ bi ∼ N (0 , 1) 4:
Take vector b
i = ν si 4 ( − 1 + ν bi , 0 , · · · , 0) ∈
R d 5:
Take the initial tridiagonal matrix
A i =
ν
si  − −. . .
− −   ∈ R d × d
11: end
13: Output
Homogeneity of optimizations tasks is controlled by noise scale s ; indeed, with noise scale equal to zero, all matrices are equal, and, by increasing noise scale, functions become less “similar” and L 2 ± grows.
In Section 5.1, we take noise scales s ∈ { 0 , 0 .
In Figure 3, we provide the same experiments as in Section 5.1 but with λ = 0 . 0001 to capture dependencies under PŁ condition.
Here, we also see that Perm K has better performance when the number of nodes n ≥ 1000 and L ± ≤ 0 .
A. C OMPARISON OF MARINA WITH R AND K AND MARINA WITH P ERM K ON QUADRATIC OPTIMIZATION PROBLEMS
In this section, we provide detailed experiments from Section 5.1 and comparisons between Rand K and Perm K using different step sizes (see Figure 4 and Figure 5).
We omitted plots where algorithms
[Table]
[Figure]
diverged.
We can see that in all experiments, Perm K behaves better than Rand K and tolerates larger step sizes.
The improvement becomes more signiﬁcant when n increases.
A. C OMPARISON OF EF21 WITH T OP K AND MARINA WITH P ERM K ON QUADRATIC OPTIMIZATION PROBLEMS
In this section, we provide detailed experiments from Section 5.1 and comparisons of EF21 with
Top K and MARINA with Perm K with different step sizes (see Figure 6 and Figure 7).
We omitted plots where algorithms diverged.
As we can see, when L ± ≤ 0 . 21 and n ≥ 10000 , Perm K converges faster than Top K .
While in heterogeneous regimes, when L ± is large, Top K has better performance except when n = 10000 .
When n > d , we see that Perm K converges faster in all experiments.
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
(5 10 ( 10 ( f ( x k ) ( f ( x * )
PermK: x4 PermK: x8 PermK: x16 Ra dK: x
Ra dK: x
Ra dK: x
[Table]
[Figure] [Figure]
[Table]
0.00 0.25 0.50 0.75 1.00 1.25 1.50 #bit# / n 1e6 10 ( 10 (5 10 ( 10 ( f ( x k ) ( f ( x * )
PermK: x4 PermK: x8 PermK: x16 T pK: x256 T pK: x
T pK: x
A. E XPERIMENT WITH MNIST: FULL DESCRIPTION
We introduce parameter (cid:98) p .
Initially, we randomly split MNIST into n + 1 parts: D 0 , D 1 , · · · , D n , where n = 1000 is the number of nodes.
Then, for all i ∈ { 1 , . . . , n } , the i th node takes split Dwith probability (cid:98) p , or split D i with probability 1 − (cid:98) p .
We deﬁne the chosen split as (cid:99) D i .
Using probability (cid:98) p , we control the homogeneity of our distribution optimization task.
Note that if (cid:98) p = 1 , all nodes store the same data D 0 , and if (cid:98) p = 0 , nodes store different splits D i .
Let us consider the more general optimization problem than in Section 5.2.
We optimize the following non-convex loss with regularization:
[Equation]
where a i ∈ R d f are MNIST images, d f = 784 is the number of features, d e = 16 is the size of encoding space.
regularizer λ ≥ 0 .
Each node stores function
[Equation]
In Figure 8, one plot for each compressor with the best convergence rate is provided for λ = { 0 , 0 .
001 } and (cid:98) p
= { 0 , 0 . 5 , 0 .
We see that in homogeneous regimes, when (cid:98) p ∈ { 0 . 9 , 1 . 0 } , Perm K outperforms other compressors for any λ .
And the larger the regularization parameter λ , the faster Perm K convergences compared to rivals.
[Table]
[Figure]
A. C OMPARISON OF MARINA WITH R AND K AND MARINA WITH P ERM K ON MNIST DATASET
In this section, we provide detailed experiments from Section 5.2 and comparisons of Rand K and
Perm K with different step sizes (see Figure 9).
We omitted plots where algorithms diverged.
We see that in all experiments, Perm K is better than Rand K .
Practical experiments on MNIST fully reproduce dependencies from our theory and experiments with synthetic quadratic optimization tasks from Section 5.1.
[Figure]
A. C OMPARISON OF EF21 WITH T OP K AND MARINA WITH P ERM K ON MNIST DATASET
In this section we provide detailed experiments from Section 5.2 and comparisons of Rand K and
Perm K with different step sizes (see Figure 10).
We omitted plots where algorithms diverged.
We see that, when (cid:98) p ∈ { 0 . 9 , 1 . 0 } , Perm K tolerates larger step sizes and convergences faster than Top K .
When (cid:98) p ∈ { 0 , 0 . 5 } both compressors approximately tolerate the same step sizes but Top K has a better performance when λ ∈ { 0 , 0 . 00001 }
[Figure]
[Table]
A. B EHAVIOR OF THE H ESSIAN V ARIANCE IN E XPERIMENTS
Now, we investigate the behavior of the Hessian Variance L ± in practical machine learning tasks and compare it with L + .
Our goal is to show that L ± can be much smaller than L + in some tasks.
A.8. L INEAR R EGRESSION WITH 2 NODES
Let us consider the linear regression problem distributed between 2 machines:
[Equation]
where ( X 1 , y 1 ) and ( X 2 , y 2 ) refer to a random split of the "mg" dataset from LIBSVM ( 1 ,samples and d = 6 features).
Using the results from Example 3, we get L + = 3609 .
6653 while
L ± = 34 .
Clearly, L ± two orders smaller than L + .
A.8. L OCAL H ESSIAN V ARAINCE IN MNIST EXPERIMENTS
Now, we estimate L ± and L + in the following way.
While MARINA running, we capture points x t and calculate the “adaptive” L t ± and L t + , that satisfy n n (cid:88) i =
||∇ f
i
[Equation]
x t ) − ∇ f
i
[Equation]
x
[Equation]
2 −
[Equation]
∇ f
[Equation]
x t
[Equation]
∇
[Equation]
x
[Equation]
±
[Equation] [Equation]
x
[Equation]
− x
[Equation]
+
||and
[Equation]
In experiments from Section 5.2 (see also Figure 2), we evaluate (cid:0) L t + (cid:1) 2 / (cid:0) L t ± (cid:1) 2 in MARINA with
Perm K .
Results are presented in Table 5.
We see that in the homogeneous case, (cid:0) L t ± (cid:1) 2 is about 10 × times smaller than (cid:0) L t + (cid:1) 2 (on average), and (cid:0) L t + (cid:1) 2 is much larger at the start of the process.
On the other hand, in the heterogeneous case, the difference is smaller.
The above experiments provide further numerical justiﬁcation that L 2 ± can be effectively much smaller than L 2+ , and that this drives the improvement of our new Perm K compressor.
B MARINA A LGORITHM
To the best of our knowledge, the state-of-the-art method for solving the nonconvex problem (1) in terms of the theoretical communication efﬁciency is MARINA (Gorbunov et al., 2021) (see Algorithm 1).
In its simplest variant, MARINA performs iterations of the form
[Equation]
where g ki is a carefully designed biased estimator of the gradient ∇ f i ( x k ) , and γ > 0 is a learning rate.
The gradient estimators used in MARINA are initialized to the full gradients, i.e., g 0 i = ∇ f i ( x 0 ) , for i ∈ { 1 , . . . , n } , and subsequently updated as
[Equation]
where θ k is a Bernoulli random variable sampled at iteration k (equal to 1 with probability p ∈ (0 , 1] , and equal to 0 with probability 1 − p ), and
C i : R d → R d is a randomized compression operator sampled at iteration k on node
i independently from other nodes.
In particular, Gorbunov et al.
(2021) assume that the compression operators C i are unbiased , and that their variance is proportional to squared norm of the input vector:
[Equation]
In each iteration of MARINA , the gradient estimator is reset to the true gradient with (small) probability p .
Otherwise, each worker i compresses the difference of the last two local gradients, and communicates the compressed message
[Equation]
to the server.
These messages are then aggregated by the server to form the new gradient estimator via
[Equation]
Note that i) this preserves the second relation in (8), ii) the server can compute g k +1 since it has access to g k , which is the case (via a recursive argument) if g 0 is known by the server at the start of the iterative process 8 .
Further, note that the expected communication cost in each iteration of MARINA is equal to
[Equation]
where d is the cost of communicating a (possibly dense) vector in R d , and ζ i ≤ d is the expected cost of communicating a vector compressed by C i .
MARINA one of the very few examples in stochastic optimization where the use of a biased estimator leads to a better theoretical complexity than the use of an unbiased estimator, with the other example being optimal SGD methods for single-node problems SARAH (Nguyen et al., 2017), SPIDER (Fang et al., 2018), PAGE (Li et al., 2021).
C M ISSING P ROOFS C. P ROOF OF L EMMA
Lemma 1.
If C i ∈ U ( ω i ) for i ∈ { 1 , 2 , . . . , n } , then {C i } ni =1 ∈ U (max i ω i , 0) .
If we further assume that the compressors are independent, then {C i } ni =1 ∈ U ( 1 n max i ω i , 0) .
Proof.
Let us ﬁrst assume unbiasedness only.
By Jensen’s inequality,
[Equation]
n
[Equation]
It remains to apply expectation on both sides and then use inequality
[Equation]
to conclude that {C i } ni =1 ∈ U (max i ω i , 0) .
Let us now add the assumption of independence.
[Equation]
( C i ( a i ) − a i ) (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) 2   =nn (cid:88)
i =
E (cid:104) (cid:107)C i ( a i ) − a i (cid:107) 2 (cid:105) + 1 n 2 (cid:88) i (cid:54) =
j
E [ (cid:104)C i ( a i ) − a i , C j ( a j ) − a j (cid:105) ] ≤ max i ω i nn (cid:88)
i =
(cid:107) a i (cid:107) 2 , by independence, thus, A = max i ω
i /n , B = 0 .
C
P ROOF OF L EMMA
Lemma 2.
L − ≤ L + , L − ≤ 1 n n (cid:80) i =
1 L i , L 2+ ≤ 1 n n (cid:80) i =
1 L 2 i , and L 2+ − L 2 − ≤ L 2 ± ≤ L 2+ .
Proof.
Let us deﬁne
L − ( x, y ) :
= (cid:107)∇ f ( x ) − ∇ f ( y ) (cid:107) 2 ,
L + ( x, y ) :
= n n (cid:88) i =
(cid:107)∇ f i ( x ) − ∇ f i ( y ) (cid:107) 2 ,
L ± ( x, y ) :
=
L + ( x, y ) − L − ( x, y ) .
The inequalities are now established as follows:
By Jensen’s inequality and the deﬁnition of L + , L − ( x, y ) ≤ L + ( x, y ) ≤ L 2+ (cid:107) x − y (cid:107) 2 , thus, L − is at most L + . 2.
By the triangle inequality, we have
[Equation]
n (cid:88)
[Equation]
From the deﬁnition of L i , we have
[Equation]
and L 2+ is at most 1 n (cid:80) ni
=1 L 2 i . 4.
The right inequality follows from L − ( x, y ) ≥ 0 and
[Equation]
Now, we prove the left inequality.
From the deﬁnition of L ± , we have
[Equation]
and
L
[Equation]
hence,
[Equation]
thus L 2+ ≤ L 2 − + L 2 ± .
C. P ROOF OF T HEOREM
Theorem 1.
The Perm K compressors from Deﬁnition 2 are unbiased and belong to IV (1) .
Proof.
We ﬁx any x ∈ R d and prove unbiasedness:
[Equation]
Next, we ﬁnd the second moment:
[Equation]
For all a 1 , . . . , a n ∈ R d , the following inequality holds:
[Equation]
Hence, Assumption 4 is fulﬁlled with A = B = 1 .
C. P ROOF OF T HEOREM
Theorem 2.
The Perm K compressors from Deﬁnition 3 are unbiased and belong to IV ( A ) with
A = 1 − n − d n − 1 .
Proof.
We ﬁx any x ∈ R d and prove unbiasedness:
[Equation]
Next, we ﬁnd the second moment:
[Equation]
For all i (cid:54) =
j ∈ { 1 , 2 , . . . , n } , x, y ∈ R d , we have
[Equation]
For all a 1 , . . . , a n ∈ R d , the following inequality holds:
[Equation]
2   = nn (cid:88) i =
E (cid:104) (cid:107)C i ( a i ) (cid:107) 2 (cid:105) + 1 n 2 (cid:88) i (cid:54) =
j
E [ (cid:104) C
i ( a i ) , C j ( a j ) (cid:105) ]
= d nn (cid:88)
i =
(cid:107) a i (cid:107) 2 + 1 n 2 (cid:88)
i (cid:54) =
j
E [ (cid:104)C i ( a i ) , C j ( a j ) (cid:105) ]
= d nn (cid:88)
i =
(cid:107) a i (cid:107) 2 + ( q − 1) d n 2 ( n − 1) (cid:88)
i (cid:54) =
j (cid:104) a i , a j (cid:105) =
(cid:18) d n − ( q − 1)
d n ( n − 1) (cid:19)n n (cid:88) i =
(cid:107) a i (cid:107) 2 + ( q − 1)
d n − 1 (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) n n (cid:88) i = a i (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
= (cid:18) 1 − n − d n −(cid:19)n n (cid:88)
i =
(cid:107) a i (cid:107) 2 + n − d n − 1 (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) 1 n n (cid:88)
i = a i (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) .
Hence, Assumption 4 is fulﬁlled with A = B = 1 − n − d n − 1 .
C. P ROOF OF T HEOREM
Theorem 3.
Assume that for each i ∈ { 1 , 2 , . . . , n } , the function f
i is twice continuously differentiable.
Fix any x, y ∈ R d and deﬁne H i ( x, y ) :
= (cid:82) 1 0 ∇ 2 f i ( x + t ( y − x )) dt,
H ( x, y ) :
= 1 n n (cid:80) i =1 H i ( x, y ) .
Then the matrices L i ( x, y ) :
=
H 2 i ( x, y ) , L − ( x, y ) :
=
H 2 ( x, y ) , L + ( x, y ) :
=
1 n
[Equation]
ni =1 H 2 i ( x, y ) and L ± ( x, y ) :
=
L + ( x, y ) − L − ( x, y ) are symmetric and positive semideﬁnite.
Moreover,
[Equation]
Proof.
The fundamental theorem of calculus says that for any continuously differentiable function ψ : R → R we have
[Equation]
Choose i ∈ { 1 , 2 , . .
. , n } , j ∈ { 1 , 2 , . .
, d } , distinct vectors x, y ∈ R d , and let
[Equation]
where
e j ∈ R
d is the j th standard unit basis vector.
Since f i is twice continuously differentiable, ψ ij is continuously differentiable, and by the chain rule,
[Equation]
Applying the fundamental theorem of calculus, we get
[Equation]
Let ψ
i : R → R d be deﬁned by ψ i ( t ) := ∇ f i ( x + t ( y − x ))
= ( ψ i 1 ( t ) , . . .
, ψ id ( t )) .
Combining equations 9 for j = 1 , 2 , . .
, d into a vector form using the fact that
[Equation]
we arrive at the identity
[Equation]
Next, since ∇ 2 f i ( x + t
y − x )) is symmetric for all t , so is H i ( x, y ) , and hence L i ( x, y ) :=
H 2 i ( x, y ) =
H (cid:62) i ( x, y )
H i ( x, y ) , which also means that L i ( x, y ) is symmetric and positive semidefinite.
Combining these observations, we obtain
[Equation]
Clearly,
[Equation]
Using the same reasoning
[Equation]
we have ∇ f ( y ) − ∇ f ( x ) =
H ( x, y )( y − x )
[Equation]
and
[Equation]
− y
[Equation]
, where L − ( x, y ) :
= H 2 ( x, y )
=
H (cid:62) ( x, y ) H ( x, y ) is symmetric and positive semideﬁnite, since
H i ( x, y ) are symmetric and positive semideﬁnite.
Finally,
[Equation]
and
[Equation]
Note, that L + ( x, y ) inherits symmetry and positive semideﬁniteness from H 2 i ( x, y ) .
Symmetry of
L ± ( x, y ) is trivial.
To prove positive semideﬁniteness of L ± ( x, y ) , note that
[Equation]
x
[Equation]
which is positive semideﬁnite.
C. P ROOF OF T HEOREM
Theorem 4.
Let Assumptions 1, 2, 3 and 4 be satisﬁed.
Let the stepsize in MARINA be chosen as 0 < γ ≤ 1 M , where
M =
L − + (cid:113) 1 − pp (cid:0) ( A − B )
L 2+ + BL 2 ± (cid:1) .
Then after T iterations, MARINA ﬁnds a random point ˆ x T ∈ R d for which
[Equation]
Proof.
In the proof, we follow closely the analysis of Gorbunov et al.
and adapt it to utilize the power of Hessian variance (Deﬁnition 4) and AB assumption (Assumption 4).
We bound the term E (cid:104)(cid:13)(cid:13)
g t +1 − ∇ f ( x t +1 )
(cid:13)(cid:13) 2 (cid:105) in a similar fashion to Gorbunov et al.
, but make use of the
AB assumption.
Other steps are essentially identical, but reﬁne the existing analysis through Hessian variance.
First, we recall the following lemmas.
Lemma 3 (Li et al.
Suppose that L − is ﬁnite and let x t +1 = x
t − γg t .
Then for any g t ∈ R d and γ > 0 , we have
[Equation]
Lemma 4 (Richtárik et al.
Let a, b > 0 .
If 0 ≤ γ ≤ 1 √ a + b , then aγ 2 + bγ ≤ 1 .
Moreover, the bound is tight up to the factor of 2 since
[Equation]
Next, we get an upper bound of E (cid:104)(cid:13)(cid:13) g t +1 − ∇ f ( x t +1 )
(cid:13)(cid:13) 2 (cid:12)(cid:12)(cid:12) x
t +1 (cid:105) .
Lemma 5.
Let us consider g t +1 from Line 8 of Algorithm 1 and assume, that Assumptions 2, 3 and 4 hold, then
[Equation]
Proof.
In the view of deﬁnition of g t +
[Equation]
we get
[Equation]
x t +1 )
− ∇ f i ( x t ) (cid:1) − ∇ f ( x t +1 ) + ∇ f ( x t ) (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) 2 (cid:12) (cid:12) (cid:12)(cid:12)(cid:12)(cid:12) x t +   + (1 − p ) (cid:13)(cid:13)
g t − ∇ f ( x t ) (cid:13)(cid:13) 2 .
In the last equality we used unbiasedness of C i .
Next, from AB inequality, we have
E (cid:104)(cid:13)(cid:13) g
[Equation]
+1 − ∇ f ( x t +1 ) (cid:13)(cid:13) 2 (cid:12)(cid:12)(cid:12) x
t +1 (cid:105) ≤ (1 − p )
E  
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) 1 n n (cid:88) i =
C i (cid:0) ∇ f i ( x t +1 )
− ∇ f i ( x t ) (cid:1) − ∇ f ( x t +1 ) + ∇ f ( x t ) (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) x t +   + (1 − p ) (cid:13)(cid:13)
g t − ∇ f ( x t ) (cid:13)(cid:13) 2 .
≤ (1 − p ) (cid:32)
A (cid:32)n n (cid:88) i =
(cid:13)(cid:13) ∇ f i ( x t +1 )
− ∇ f i ( x t ) (cid:13)(cid:13)(cid:33) − B (cid:13)(cid:13) ∇ f ( x t +1 )
− ∇ f ( x t ) (cid:13)(cid:13)(cid:33) + (1 − p ) (cid:13)(cid:13)
g t − ∇ f ( x t ) (cid:13)(cid:13)= (1 − p ) (cid:32) ( A − B ) (cid:32)n n (cid:88) i =
(cid:13)(cid:13) ∇ f i ( x t +1 )
− ∇ f i ( x t ) (cid:13)(cid:13)(cid:33) + B (cid:32)n n (cid:88) i = (cid:13)(cid:13) ∇ f i ( x t +1 )
− ∇ f i ( x t ) (cid:13)(cid:13) 2 − (cid:13)(cid:13) ∇ f ( x t +1 )
− ∇ f ( x t ) (cid:13)(cid:13)(cid:33) (cid:33) + (1 − p ) (cid:13)(cid:13)
g t − ∇ f ( x t ) (cid:13)(cid:13) 2 .
Using Assumption 2 and Deﬁnition 4, we obtain (13).
We are ready to prove Theorem 4.
Deﬁning
[Equation]
and using inequalities (12) and (13), we get
[Equation]
where in the last inequality we use
[Equation]
following from the stepsize choice and Lemma 4.
Summing up inequalities E (cid:2)
Φ t +1 (cid:3) ≤ E [Φ t ] − γ 2 E (cid:104) (cid:107)∇ f ( x t ) (cid:107) 2 (cid:105) for t = 0 , 1 , . . .
, T − 1 and rearranging the terms, we get
[Equation]
since g 0 = ∇ f ( x 0 ) and Φ T ≥ 0 .
Finally, using the tower property and the deﬁnition of ˆ x T (see
Section B), we obtain the desired result.
D P OLYAK -Ł OJASIEWICZ A NALYSIS
In this section, we analyze the algorithm under Polyak-Łojasiewicz (PŁ) condition.
We show that
MARINA algorithm with Assumption 5 enjoys a linear convergence rate.
Now, we state the assumption and the convergence rate theorem.
Assumption 5 (PŁ condition) .
Function f satisﬁes Polyak-Łojasiewicz (PŁ) condition, i.e.,
[Equation] [Equation]
where µ > 0 and f (cid:63) := inf x f ( x ) .
Lemma 6.
For L − > 0 and µ from Assumption 5 holds that L − ≥ µ.
Theorem 5.
Let Assumptions 1, 2, 3, 4 and 5 be satisﬁed and
[Equation]
then for x T from MARINA algorithm the following inequality holds:
[Equation]
We provide the proof to Theorem 5 in Section D.2.
In Table 6, we provide communication complexity of MARINA with Perm K and Rand K , and EF with Top K , optimized w.r.t.
parameters of the methods.
As in Section 4, we see that MARINA with
Perm K is not worse than MARINA with Rand K (recall Lemma 2).
Let us consider zero Hessian variance regime:
L ± = 0 .
When d ≥ n , Perm K compressor has communication complexity O (max { dL − / nµ , d } ) , while Rand K compressor has communication complexity O (max { dL − / √ nµ , d } ) .
And the communication complexity of Perm K is strictly better when dL − / √ nµ > d. Moreover, if d ≤ n and (1 + d / √ n ) L − / µ > d , then we get the strict improvement of the communication complexity from O (max { (1 + d / √ n ) L − / µ , d } ) to O (max { L − /µ, d } ) over MARINA with Rand K .
[Table]
D. P ROOF OF L EMMA
Lemma 6.
For L − > 0 and µ from Assumption 5 holds that L − ≥ µ.
Proof.
We can deﬁne L − using the following inequality:
[Equation]
d .
Let us take x = y − 1 /L − ∇ f ( y ) .
Then,
[Equation]
Rearranging the terms and using Deﬁnition
[Equation]
we have
[Equation]
thus, µ ≤ L − .
D. P ROOF OF T HEOREM
Theorem 5.
Let Assumptions 1, 2, 3, 4 and 5 be satisﬁed and
[Equation]
then for x T from MARINA algorithm the following inequality holds:
[Equation]
Proof.
The analysis is almost the same as in Gorbunov et al.
[Equation]
but we include it for completeness.
Let us deﬁne
[Equation]
As in Appendix C.
[Equation]
we use (12) and (13) to get that
[Equation]
In the last inequality
[Equation]
we used γp (1 − p ) (cid:98) L 2 − 12 γ + L − 2 ≤ 0 and γ 2 + γp (1 − p ) ≤ (1 − γµ ) γp , that follow from (15) and Lemma 4.
Unrolling E (cid:2)
Φ t +1 (cid:3) ≤
[Equation]
1 − γµ )E
[Φ t ] and using g 0 = ∇ f ( x 0 ) , we have
[Equation]
This concludes the proof.
E EF21 A NALYSIS
We provide convergence proofs of EF21 algorithm from Richtárik et al.
(2021) for non-convex and
PŁ regimes.
They will be almost identical to the one by Richtárik et al.
(indeed, the only change is the constant L + instead of (cid:101) L ), but we have decided to include it for the sake of clarity.
E. EF21 RATE IN THE NON - CONVEX REGIME
We will be using the following lemmas, the proofs of which are in their corresponding papers.
Lemma 7 (Richtárik et al.
Let C to be α -contractive for 0 < α ≤ 1 .
Deﬁne G ti := (cid:107) g ti − ∇ f i ( x t ) (cid:107) 2 and W t := { g t 1 , . . .
, g tn , x t , x t +1 } .
For any s > 0 we have
[Equation]
16) where θ ( s
[Equation]
:=
[Equation]
[Equation] [Equation]
and
[Equation] [Equation]
Lemma 8 (Richtárik et al.
Let 0 < α ≤ 1 and for s > 0 let θ ( s ) and β ( s ) be as in equation 17.
Then the solution of the optimization problem
[Equation]
(18) is given by s ∗ = 1 √ 1 − α − 1 .
Furthermore, θ ( s ∗ ) = 1 − √ 1 − α , β ( s ∗ ) = 1 − α 1 −√ 1 − α
and (cid:115)
[Equation]
We are now ready to conduct the proof
[Equation]
Theorem 6.
Let Assumptions 1 and 2 hold, and let the stepsize be set as
[Equation]
Fix T ≥ 1 and let ˆ x T be chosen from the iterates x 0 , x 1 , . .
, x T − 1 uniformly at random.
Then
[Equation]
Proof.
STEP 1.
Recall that Lemma 7 says that
[Equation]
where θ = θ ( s ∗ ) and β = β ( s ∗ ) are given by Lemma 8.
Averaging inequalities equation 22 over i ∈ { 1 , 2 , . . . , n } gives
[Equation]
Using Tower property and L -smoothness in equation
[Equation]
we proceed to
[Equation]
STEP 2.
Next, using Lemma 3 and Jensen’s inequality applied to the function x (cid:55)→
[Equation]
x
[Equation]
2 , we obtain the bound
[Equation]
Subtracting f inf from both sides of equation 25 and taking expectation, we get
[Equation]
COMBINING STEP 1 AND STEP 2.
Let δ t := E (cid:2)
f ( x t ) − f inf (cid:3) , s t :=
E [ G t ] and r t :=
E (cid:104)(cid:13)(cid:13) x t +1 − x t (cid:13)(cid:13) 2 (cid:105) .
Then by adding equation 26 with a γ 2 θ multiple of equation 24 we obtain δ t +1 + γ 2 θs t +
[Equation]
The last inequality follows from the bound γ 2 βL
+ θ + L − γ ≤ 1 , which holds from our assumption on the stepsize and Lemma 4.
By summing up inequalities for t = 0 , . . .
, T − 1 , we get
[Equation]
Multiplying both sides by 2 γT , after rearranging we get
[Equation]
It remains to notice that the left hand side can be interpreted as E (cid:104)(cid:13)(cid:13) ∇ f (ˆ x T ) (cid:13)(cid:13) 2 (cid:105) , where ˆ x T is chosen from x 0 , x 1 , . . .
, x T − 1 uniformly at random.
E. EF21 IN PŁ REGIME
Theorem 7.
Let Assumptions 1, 2 and 5 hold, and let the stepsize in EF21 be set as
[Equation]
Let Ψ t :
= f ( x t ) − f inf + γθ G t .
Then for any T ≥ 0 , we have
[Equation]
Proof.
Again, this follows Richtárik et al.
(2021) almost verbatim.
We proceed as in the previous proof, but use the PŁ inequality and subtract f inf from both sides of equation 25 to get
[Equation]
Let δ t := E (cid:2)
f ( x t ) − f inf (cid:3) , s t :=
E [ G t ] and r t := E (cid:104)(cid:13)(cid:13) x
t +1 − x t (cid:13)(cid:13) 2 (cid:105) .
Then by adding the above inequality with a γθ multiple of equation 24, we obtain
[Equation]
Note that our assumption on the stepsize implies that 1 − θ 2 ≤ 1 − γµ and 12 γ − L − 2 − βL
+ γ θ ≥ 0 .
The last inequality follows from the bound γ 2 2 βL 2+ θ +
[Equation]
≤ 1 , which holds because of Lemmaand our assumption on the stepsize.
Thus,
[Equation]
It remains to unroll the recurrence.
F C OMMUNICATION M ODEL
As mentioned in the introduction, we consider the regime where the worker-to-server communication is the bottleneck of the system so that the server-to-workers communication can be neglected .
While this is a standard model used in many prior works, we include a brief explanation of why and when this regime is useful.
Peer-to-peer communication.
First, this regime makes sense when the server is merely an abstraction, and does not exist physically.
Indeed, from the point of view of each worker, the server may merely represent “all other nodes” combined.
In this model, “a worker sending a message to the server” should be interpreted as this worker sending the message to all other workers.
Clearly, in this model there is no need for the “server” to communicate the aggregated message back to the workers since aggregation is performed on all workers independently, and the aggregated message is immediately available to all workers without the need for any additional communication.
Fast broadcast.
Second, the above regime makes sense in situations where the server exists physically, but is able to broadcast to the workers at a much higher speed compared to the worker-to-server communication.
This happens in several distributed systems, e
g., on certain supercomputers (Mishchenko et al., 2019).
Virtually all theoretical works on communication efﬁcient distributed algorithms assume that the server-to-worker communication is cheap, and in this work we follow in their footsteps.
Having said that, our work can be extended to the more difﬁcult regime where the server-to-worker communication is also costly (Horváth et al., 2019; Tang et al., 2019; Philippenko & Dieuleveut, 2020; Gorbunov et al., 2020).
However, for simplicity, we do not explore this extension in this work.
G O N C ONTRACTIVE C OMPRESSORS AND E RROR F EEDBACK G. O N C ONTRACTIVE C OMPRESSORS
The most successful algorithmic solutions to solving the nonconvex distributed optimization problem (1) in a communication-efﬁcient manner under the communication model described in Appendix F involve stochastic gradient descent ( SGD ) methods with communication compression .
There are two large classes of such methods, depending on the type of compression operator involved: (i) methods that work with contractive (and possibly biased stochastic) compression operators, such as Top K or Rank K , and (ii) methods that work with unbiased and independent (across the workers) stochastic compression operators, such as Rand K .
A (randomized) compression operator C : R d → R d is α -contractive
(we write C ∈ C ( α ) ),
where 0 < α ≤ 1 , if
[Equation]
A canonical example is the (deterministic) Top K compressor, which outputs the K largest (in absolute value) entries of the input vector x , and zeroes out the rest.
Top K is α -contractive with α = K / d .
Another example is the Rank K compressor based on the best rank- K approximation of x represented as an a × b = d matrix.
It can be shown that Rank K is α -contractive with α = K / min { a,b } (Safaryan et al., 2021, Section A.3.2).
We refer to the work of Vogels et al.
for a practical communication-efﬁcient method PowerSGD based on low-rank approximations.
Of special importance are α -compressors arising from unbiased compressors via appropriate scaling.
Let Q : R d → R d be an unbiased operator with variance proportional to the square norm of the input vector.
That is, assume that E [ Q ( x )]
= x for all x ∈ R d and that there exists ω ≥
such that
[Equation]
We will write Q ∈ U ( ω ) for brevity.
It is well known that the operator C
= ( ω + 1) − 1 Q is α -contractive with α = ( ω + 1) − 1 .
An example of a contractive compressor arising this way is
( ω + 1) − 1 Rand K , which keeps a subset of K entries of the input vector x chosen uniformly at random, and zeroes out the rest.
As Top K , ( ω + 1) − 1 Rand K is α -contractive, with α = K / d .
Distributed SGD methods relying on general contractive compressors, i.e., on contractive which do not arise from unbiased compressors from scaling, need to rely on the error-feedback / errorcompensation mechanism to avoid divergence.
G. O N E RROR F EEDBACK
An alternative approach to the one represented by MARINA is to seek more aggressive compression, even at the cost of abandoning unbiasedness, in the hope that this will lead to better communication complexity in practice.
This is the idea behind the class of contractive compressors , deﬁned in (29), which have studied at least since the work of Seide et al.
Example of such compressors are the Top K (Alistarh et al., 2018) and Rank K
(Vogels et al., 2019;
Safaryan et al., 2021) compressors.
While such compressors are indeed often very successful in practice, their theoretical impact on the methods using them is dramatically less understood than is the case with unbiased compressors.
One of the key reasons for this that a naive use of biased compressors may lead to (exponential) divergence, even in simple problems (Beznosikov et al., 2020).
Because of this, Seide et al.
(2014) proposed the error feedback framework for controlling the error introduced by compression, and thus taming the method to convergence.
While it has been successfully used by practitioners for many years, error feedback yielded the ﬁrst convergence results only relatively recently (Stich et al., 2018;
Stich & Karimireddy, 2019; Wu et al., 2018; Koloskova et al., 2019; Tang et al., 2019; Karimireddy et al., 2019; Qian et al., 2020; Beznosikov et al., 2020; Gorbunov et al., 2020).
The current best theoretical communication complexity results for error feedback belong to the EF method of Richtárik et al.
(2021) who achieved their improvements by redesigning the original error feedback mechanism using the construction of a Markov compressor.
However, even EF21 currently enjoys substantially weaker iteration and communication complexity than MARINA .
For instance, we show in Appendix L that EF21 with Top K is only proved to have the communication complexity of the gradient descent without any compression.
H C OMPOSITION OF C OMPRESSORS WITH AB A SSUMPTION AND U NBIASED C OMPRESSORS
Lemma 9.
If {C i } ni =1 ∈ U ( A, B ) and Q i ∈ U ( ω i ) for i ∈ { 1 , 2 , . . . , n } , then {C i ◦ Q i }
ni =1 ∈ U ((max i ω
i + 1) A, B ) .
Proof.
By the tower property, for all a 1 , . . . , a n ∈ R d , we have
E 
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) 1 n n (cid:88) i =
C
[Equation]
a
[Equation] [Equation]
1 n n (cid:88) i =
Q i ( a i ) (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) 2   =
E  
E   (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) 1 n n (cid:88) i =
C i ( Q i ( a i )) − 1 n n (cid:88) i =
Q i ( a i ) (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) Q 1 ( a 1 ) , · · · , Q n ( a n )   
 ≤ E 
[Equation]
A 1 n n (cid:88) i = (cid:107)Q i ( a i ) (cid:107)
[Equation]
B (cid:13)(cid:13)(cid:13) (cid:13) (cid:13) n n (cid:88) i =
Q i ( a i ) (cid:13)(cid:13)(cid:13) (cid:13) (cid:13) 2   .
Since Q i ∈ U ( ω i ) for i ∈ { 1 , 2 , . . . , n } , we get
E  
[Equation]
n n (cid:88) i =
[Equation]
=
[Equation]
(cid:107)
[Equation]
E   (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) 1 n n (cid:88) i =
Q i ( a i ) (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) 2   .
Using Jensen’s inequality, we derive inequalities:
E 
[Equation]
=
[Equation] [Equation]
1 n n (cid:88) i = a i (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
and
E 
[Equation] [Equation]
n
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
The last inequality completes the proof.
I G ENERAL E XAMPLES OF P ERM K
For the sake of clarity, in the main part of our paper, we assumed that n | d or d | n , and provided corresponding examples of Perm K (see Deﬁnition 2 and Deﬁnition 3).
Now, we provide two examples of Perm K that work with any n and d and generalize the previous examples.
I. C ASE d ≥ n
The following example generalizes for the case when n does not divide d .
Let us assume that d = kn + r and 0 ≤ r < n.
As in Deﬁnition 2, we permute coordinates and split them into the blocks of sizes { k, . .
, k, r } .
The ﬁrst n block of size k we assign to nodes.
Next, we take the last block of size r and randomly assign each coordinate from this block to one node.
As the size of the last block of size r is less than n , some nodes will send one coordinate less.
Deﬁnition 5 (Perm K ( d ≥ n )) .
Let us assume that d ≥ n, d = kn + r , 0 ≤ r < n, π d = ( π d 1 , . . .
, π dd ) is a random permutation of { 1 , · · · , d } , and π n = ( π n 1 , . . .
, π nn ) is a random permutation of { 1 , · · · , n } .
We deﬁne the tuple of vectors S ( x ) =
( x π d kn +1 e π d kn +1 , . .
, x π d kn + r e π d kn
[Equation]
r , 0 , . . . , 0) of size n .
Then,
[Equation]
Theorem 8.
Compressors {C i } ni =1 from Deﬁnition 5 belong to IV (1) .
Proof.
We ﬁx any x ∈ R d and prove unbiasedness:
[Equation]
for all i ∈ { 1 , . . . , n } .
Next, we derive the second moment:
[Equation] [Equation]
n
[Equation]
We ﬁx x, y ∈ R d .
For all i (cid:54) =
l ∈ { 1 , . . . , n } , we have
E [ (cid:104)C i ( x ) , C l ( y ) (cid:105) ] = n(cid:42)
[Equation]
due to orthogonality of vectors e p , for all p ∈ { 1 , . . .
, d } , and the fact that i (cid:54) =
l .
Thus, for all a 1 , . . . , a n ∈ R d , the following equality holds:
[Equation]
i
[Equation]
Hence, Assumption 4 is fulﬁlled with A = B = 1 .
I. C ASE n ≥ d
The following deﬁnition generalizes Deﬁnition 3 for the case when d does not divide n .
Let us assume that n = qd + r and 0 ≤ r
< d.
As in Deﬁnition 3, we permute the multiset, where each coordinate occures q times.
Then, we randomly assign each element from the multiset of size qd to one node.
Note that r randomly chosen nodes are idle.
Deﬁnition 6 (Perm K , ( n ≥ d )) .
Let us assume that n ≥ d, n = qd + r , 0 ≤ r < d.
Let us ﬁx point x ∈ R d , that we want to compress.
Deﬁne the tuple of vectors (cid:98) S ( x ) =
( x 1 e 1 , . . .
, x 1 e 1 , x 2 e 2 , . . .
, x 2 e 2 , . .
, x d e d , . . .
, x d e d ) , where each vector occurs q times.
Concat r zero vectors to (cid:98) S ( x ) : S ( x )
=
(cid:98) S ( x ) ⊕ (0 , . . . , 0) .
Let π =
( π 1 , . . .
, π n ) be a random permutation of { 1 , . . . , n } .
Deﬁne
[Equation]
Theorem 9.
Compressors {C i } ni =1 from Deﬁnition 6 belong to IV ( A ) with A = 1 − n ( q − 1) ( n − 1) q .
Proof.
We start with proving the unbiasedness:
[Equation]
for all i ∈ { 1 , . . . , n } , x ∈ R d .
Next, we ﬁnd the second moment:
[Equation]
for all i ∈ { 1 , . . . , n } , x ∈ R d .
For all i (cid:54) =
j ∈ { 1 , . . . , n } and x, y ∈ R d , we have
[Equation]
Thus, for all a 1 , . . . , a n ∈ R d , the following equality holds:
[Equation]
1 n n (cid:88) i =
C i ( a i ) (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) 2   = nn (cid:88) i =
E (cid:104) (cid:107)C i ( a i ) (cid:107) 2 (cid:105) + 1 n 2 (cid:88) i (cid:54) =
j
E [ (cid:104) C
i ( a i ) , C j ( a j ) (cid:105) ] = nq n (cid:88) i
= (cid:107) a i (cid:107) 2 + n ( q − 1) ( n − 1)
q   1 n 2 (cid:88) i (cid:54) =
j (cid:104) a i , a j (cid:105) 
 =
(cid:18)q − ( q − 1) ( n − 1) q
(cid:19)n n (cid:88)
i =
(cid:107) a i (cid:107) 2 + n ( q − 1) ( n − 1) q (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) n n (cid:88)
i = a i (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) .
Hence, Assumption 4 is fulﬁlled with A = B = 1 − n ( q − 1) ( n − 1) q .
J I MPLEMENTATION D ETAILS OF P ERM K
Now, we discuss the implementation details of Perm K from Deﬁnition 2.
Unlike Rand K and Top K compressors, Perm K compressors are statistically dependent.
We provide a simple idea of how to manage dependence between nodes.
First of all, note that the samples of random permutation π are the only source of randomness.
By Deﬁnition 2, they are shared between nodes and generated in each communication round.
However, instead of sharing the samples, we can generate these samples in each node regardless of other nodes.
Almost all random generation libraries and frameworks are deterministic (or pseudorandom) and only depend on the initial random seed.
Thus, at the beginning of the optimization procedure, all nodes should set the same initial random seed and then call the same function that generates samples of a random permutation.
The computation complexity of generating a sample from a random permutation π
=
( π 1 , . . .
, π d ) is O ( d ) using the Fisher-Yates shufﬂe algorithm (Fisher & Yates, 1938; Knuth, 1997).
All other examples of compressors can be implemented in the same fashion.
K M ORE E XAMPLES OF P ERMUTATION -B ASED C OMPRESSORS K. B LOCK PERMUTATION COMPRESSOR
In block permutation compressor, we partition the set { 1 , . . .
, d } into m ≤ n disjoint blocks.
For each block P i , (cid:4) nm (cid:5) devices sparsify their vectors to coordinates with indices in P i only.
Deﬁnition 7.
Let P to be a partition of the set { 1 , . . .
, d } into m ≤ n non-empty subsets, and n = mq + r, where 0 ≤ r < m. Deﬁne matrices A 1 , . . .
, A n as follows: put A i :=
if i > mq .
Denote the subsets in P as P 1 , · · · , P m .
Next, for any P i ∈ P , we set A ( i − 1)
q
+1 ,
A ( i − 1) q +2 , · · · , A iq to nq Diag( P i ) .
Here by Diag( S ) we mean the diagonal matrix where each i th diagonal entry is equal to 1 if i ∈ S and 0 otherwise.
Let π =
( π 1 , . . .
, π n ) be a random permutation of set { 1 , . . . , n } .
We deﬁne C i ( x ) :
= A π i x .
We call the set {C i } ni =1 the block permutation compressor .
Lemma 10.
Compressors {C i } ni =1 belong to IV ( A ) with A = 1 − n ( q − 1) ( n − 1) q .
Proof.
We start with the proof of unbiasedness:
[Equation]
for all i ∈ { 1 , . . . , n } , x ∈ R d .
Next, we establish the second moment:
[Equation]
for all i ∈ { 1 , . . . , n } , x ∈ R d .
The following equality will be useful for the AB assumption:
[Equation]
for all i (cid:54)
=
j ∈ { 1 , . . . , n } , x, y ∈ R d .
Thus,
[Equation] [Equation]
n
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
i = (cid:107) a i (cid:107) 2 + n ( q − 1) ( n − 1) q (cid:13)(cid:13)(cid:13)(cid:13) (cid:13) n n (cid:88)
i = a i (cid:13)(cid:13)(cid:13)(cid:13) (cid:13) =
(cid:18) 1 − n ( q − 1) ( n − 1) q (cid:19)n n (cid:88)
i =
(cid:107) a i (cid:107) 2 + n ( q − 1) ( n − 1) q (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) n n (cid:88)
i = a i (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) , for all a 1 , . .
. , a n ∈ R d .
Hence, Assumption 4 is fulﬁlled with A = B = 1 − n ( q − 1) ( n − 1) q .
Deﬁnition 8.
Let Q 1 , . . .
, Q n : R d → R d be a collection of deterministic mappings R d → R d .
Let π =
( π 1 , . . .
, π n ) be a random permutation of set { 1 , . . . , n } , where n > 1 .
Deﬁne C
i :=
Q π i .
Assume that the following conditions hold:
There exists ω ≥ 0 such that E (cid:104) (cid:107)C i ( x ) (cid:107) 2 (cid:105) ≤ ( ω + 1) (cid:107) x (cid:107) 2 for all i ∈ { 1 , . . . , n } , x, y ∈ R d . 2.
There exists θ ∈ R such that n (cid:80) i =
1 (cid:104) Q i ( x ) , Q i ( y ) (cid:105)
= θ (cid:104) x, y (cid:105)
for
all x, y ∈ R d . 3.
1 n n (cid:80) i =
1 Q i ( x ) = x for all x ∈ R d .
We call the collection {C i } ni =1 the permutation of mappings.
Lemma 11.
Permutation of mappings belongs to U ( A, B ) with A = ω +1 n − 1 n − 1 (cid:0) 1 − θn 2 (cid:1) and
B = 1 − n n − 1 (cid:0) 1 − θ n 2 (cid:1) .
Proof.
Unbiasedness follows directly from Condition 3.
Let us ﬁx a 1 , . .
. , a n ∈ R d .
We shall now establish the AB assumption.
[Equation]
a i , a j (cid:105) 
 =
n  ω n (cid:88) i =
(cid:107) a i (cid:107) 2 − (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) n (cid:88)
i = a i (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) + n (cid:88)
i =
(cid:107) a i (cid:107) 2 + (cid:88)
i (cid:54) =
j
E (cid:2)(cid:10) Q π i ( a i ) , Q π j ( a j ) (cid:11)(cid:3) = ( ω + 1) n n n (cid:88)
i
=
(cid:107) a i (cid:107) 2 − (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) 1 n n (cid:88)
i = a i (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) +n(cid:88) i (cid:54) =
j
E (cid:2)(cid:10) Q π i ( a i ) , Q π j ( a j ) (cid:11)(cid:3)
Let us now compute 1 n 2 (cid:80) i (cid:54) =
j E (cid:2)(cid:10)
Q π i ( a i ) , Q π j ( a j ) (cid:11)(cid:3) .
n(cid:88) i (cid:54) =
j
E (cid:2)(cid:10) Q π i ( a i ) , Q π j ( a j ) (cid:11)(cid:3) =
n(cid:88) i (cid:54) =
j n ( n − 1) (cid:88) u (cid:54) = v (cid:104) Q u ( a i ) , Q v ( a j ) (cid:105) = n ( n − 1) (cid:88)
i (cid:54) =
j n(cid:88) u (cid:54) = v (cid:104) Q u ( a i ) , Q v ( a j ) (cid:105) .
Now, n(cid:88) u (cid:54) = v (cid:104) Q u ( x ) , Q v ( y ) (cid:105) = nn (cid:88) u
= n (cid:88) v = (cid:104) Q u ( x ) , Q v ( y ) (cid:105) − 1 nn (cid:88)
u = (cid:104) Q u ( x ) , Q v ( y ) (cid:105)
Condition 3 = (cid:104) x, y (cid:105) − 1 nn (cid:88) u
= (cid:104) Q u ( x ) , Q u ( y ) (cid:105)
Condition 2 = (cid:18) 1 − θ n(cid:19) (cid:104) x, y (cid:105) , ∀ x, y ∈ R d .
Hence, n(cid:88) i (cid:54) =
j
E (cid:2)(cid:10) Q π i ( a i ) , Q π j ( a j ) (cid:11)(cid:3) = n ( n − 1) (cid:18) 1 − θ n(cid:19) (cid:88) i (cid:54) =
j (cid:104) a i , a j (cid:105) .
Finally,
[Equation]
Hence
[Equation]
Assumption 4 is fulﬁlled with A = ω +1 n − 1 n − 1 (cid:0) 1 − θn 2 (cid:1) , B
[Equation]
1 − nn −
[Equation] [Equation]
θn 2 (cid:1) .
L A NALYSIS OF C OMPLEXITY B OUNDS
In this section, we analyze the complexities bounds of optimization methods, and typically these bounds have a structure of a function that we analyze in the following lemma.
Lemma 12.
Let us consider function
[Equation]
, where x ∈ (0 , 1] , y ∈ [ y min , 1] , y min ∈ (0 , 1 / 2] , a ≥ 0 , and b ≥ 0 , then
[Equation]
Proof.
First, let us assume that x ≥ 1 / 2 .
Then,
[Equation]
Second, let us assume that y ≥ 1 / 2 .
Then,
[Equation] [Equation]
Finally, let us assume, that y ≤ 1 / 2 and x ≤ 1 / 2 .
Then,
[Equation]
≥ a (1 − x ) y + bxy (cid:18)(cid:18)y −(cid:19)
+ (cid:18)x −(cid:19)(cid:19) (cid:115)(cid:18)x −(cid:19) (cid:18)y −(cid:19) ≥ a (1 − x ) y + 2 bxy
cid:18)x −(cid:19) (cid:18)y −(cid:19) ≥ a (1 − x ) y
+ 2 b (1 − x ) (1 − y ) ≥ ay min + b 2 .
L. N ONCONVEX OPTIMIZATION L.1. C ASE n ≤ d
We analyze case, when n ≤ d. For simplicity, we assume that n | d , n > 1 , and d > 1 .
For
Perm K from Deﬁnition 2, constants A = B = 1 in AB inequality (see Lemma 1).
We deﬁne the upper bound for the communication complexity of MARINA with Perm K as N Perm K ( p ) , where p is a parameter of MARINA , and MARINA with Rand K as N Rand K ( p, k ) , where k is a parameter of
Rand K .
From Theorem 4, we have that oracle complexity of MARINA with Perm K is equal to
O (cid:18) ∆ε (cid:18)
L − + (cid:114) 1 − p p
L ± (cid:19)(cid:19) .
During each iteration of MARINA , on average, each node sends the number of bits equal to
O
[Equation]
thus, the communication complexity predicted by theory is
[Equation]
up to a constant factor.
Analogously, for Rand K , the communication complexity predicted by theory is
[Equation]
up to a constant factor.
To the best of our knowledge, this is the state-of-the-art theoretical communication complexity bound for the Rand K compressor in the non-convex regime.
Finally, for Top K , by Theorem 6, the theoretical communication complexity is
[Equation]
up to a constant factor.
We consider the variant of EF21 , where g 0 i are initialized with gradients ∇ f i ( x 0 ) , for all i ∈ { 1 , . . . , n } , thus E (cid:2) G 0 (cid:3) = 0 in Theorem 6.
The following lemma will help us to choose the optimal parameters of N Perm K ( p ) , N Rand K ( p, k ) , and N Top K ( k ) .
We wish to stress that in the following sections the term lower bound should not be understood as the lower bound for the communication complexity.
We merely provide lower bounds for the upper bounds predicted by the current state-of-the-art theory by optimizing their parameters.
Lemma 13.
For communication complexity N Perm K ( p ) of MARINA with Perm K , communication complexity N Rand K ( p, k ) of MARINA with Rand K and communication complexity N Top K ( k ) of EF with Top K deﬁned in (31), (32) and (33) the following inequalities hold:
Lower bounds:
[Equation]
Upper bounds:
[Equation]
Lower bounds:
[Equation]
Upper bounds
For
[Equation] [Equation]
Moreover, for all k ∈ { 1 , . . .
, d } , p = 1 ,
[Equation] [Equation]
Proof.
We start with the ﬁrst inequality:
[Equation]
Using Lemma 12 with a = dL − , b = dL ± √ n , and y min
= 1 /n
[Equation]
we get
[Equation]
Perm
[Equation]
for all p ∈ (0 , 1] .
We can obtain the bound 34 if we take p = 1 /n :
N Perm K
[Equation]
We obtain the equality 35 by taking p = 1 .
[Equation]
Using Lemma 12 with a = dL − , b = dL ± √ n , and y min = 1 /d , we get
[Equation]
for all p ∈ (0 , 1] , k ∈ { 1 , . . .
, d } .
We can obtain the bound 36 if we take k ∈ { 1 , . .
. , d/ √ n } and p = k/d :
[Equation]
The equality 37 is obtained by taking p = 1 .
This part is easily proved, using L − ≤ L + from Lemma 2, and directly minimizing (38).
In Table 4, we summarize bounds (34), (35), (36), (37), and (38).
L.1. C ASE n ≥ d
Now, we analyze case, when n ≥ d. For simplicity, without losing the generality, we assume that d | n, n > 1 , and d > 1 .
Then, Perm K from Deﬁnition 3 satisﬁes the AB inequality with A = B = d −n − 1 .
In each iteration of MARINA , on average, Perm K sends
[Equation]
bits, thus the theoretical communication complexity is
[Equation]
up to a constant factor.
Lemma 14.
For communication complexity N Perm K ( p ) of MARINA with Perm K , communication complexity N Rand K ( p, k ) of MARINA with Rand K and communication complexity N Top K ( k ) of EF with Top K deﬁned in (39), (32) and (33) the following inequalities hold:
Lower bounds:
[Equation]
Upper bounds:
[Equation]
Lower bounds:
[Equation]
Upper bounds:
[Equation]
Moreover, for all k ∈ { 1 , . . .
, d } , p = 1 ,
[Equation] [Equation]
Proof.
[Equation] [Equation]
Using Lemma 12 with a = dL − , b = dL ± √ n , and y min = 1 /d, we get
[Equation]
for all p ∈ (0 , 1] .
We can show the bound 40 if we take p = 1 /d :
N Perm K
[Equation]
The bound 41 is obtained by taking p = 1 .
As in Lemma 13 we can get, that
[Equation]
for all p ∈ (0 , 1] , k ∈ { 1 , . . .
, d } .
Moreover, if we take p = 1 /d and k = 1 , we have
[Equation]
The bound 43 is obtained by taking p = 1 .
For Top K , the reasoning the same as in Lemma 13.
In Table 4, we summarize bounds (40), (41), (42), (43), and (44).
L. PŁ ASSUMPTION L.2. C ASE n ≤ d
Using the same reasoning as in Appendix L.1, Theorem 5 and Theorem 7, we can show that communication complexities predicted by theory are equal to
[Equation]
up to a constant factor.
Lemma 15.
For communication complexity N Perm K ( p ) of MARINA with Perm K , communication complexity N Rand K ( p, k ) of MARINA with Rand K and communication complexity N Top K ( k ) of EF with Top K deﬁned in (45), (46) and (47) the following inequalities hold 9 :
[Equation]
Proof.
Rearranging (45), (46) and (47), we get
[Equation]
Note, that
[Equation]
thus in all complexities, the second terms inside the max brackets are at least d .
Analysis of ﬁrst terms inside the max brackets is the same as in Lemma 13.
In Table 6, we provide complexity bounds with optimal parameters of algorithms.
L.2. C ASE n ≥ d
The only difference here is that the communication complexity of Perm K predicted by our theory is the following:
[Equation]
Lemma 16.
For communication complexity N Perm K ( p ) of MARINA with Perm K , communication complexity N Rand K ( p, k ) of MARINA with Rand K and communication complexity N Top K ( k ) of EF with Top K deﬁned in (48), (46) and (47) the following inequalities hold:
[Equation]
[Equation]
p ∈
[Equation]
0 , 1] ,k ∈
[Equation]
1 ,...,d }
[Equation]
Rand
[Equation]
[Table]
The proof of Lemma 16 is the same as in Lemma 15.
Using the same reasoning as before, we provide complexity bounds in Table 6.
M G ROUP H ESSIAN V ARIANCE
We showed the communication complexity improvement of MARINA algorithm with Perm K under the assumption that L ± (cid:28) L − .
In general, L ± can be large; however, we can still use the notion of
L ± but in a different way, by splitting the functions into several groups where L ± is small.
We split a set { 1 , · · · , n } into nonempty sets {G k } gk =1 , (cid:83) gk =1 G k = { 1 , · · · , n } , G i (cid:84)
G j = ∅ , for all i (cid:54) =
j ∈ { 1 , · · · , g } , and |G k | > 0 , for all k ∈ { 1 , · · · , g } .
Let us ﬁx some set G k and deﬁne functions
[Equation]
and the smallest constants L G k − , L G k + , L G k ± for functions L G k − ( x, y ) , L G k + ( x, y ) , and L G k ± ( x, y ) , such that
[Equation]
y ) ≤ (cid:16) L G k + (cid:17) 2 (cid:107) x − y (cid:107) 2 , L G k ± ( x, y ) ≤ (cid:16)
L G k ± (cid:17) 2 (cid:107) x − y (cid:107) 2 , for all k ∈ { 1 , · · · , g } , x, y ∈ R d .
In this section, we have the following assumption about groups.
Assumption 6.
Compressors between groups are independent, i.e. C i and C j are independent, for all i ∈ G k , j ∈ G p , k (cid:54) =
p.
And Assumption 4 is satisﬁed with constants A G k and B G k inside each group G k , for k ∈ { 1 , · · · , g } .
Now, we prove group AB inequality.
Lemma 17 (Group AB inequality) .
Let us assume that Assumptions 3 and 6 hold, then
E
[Equation]
(cid:13)
[Equation]
(cid:13)(cid:13)(cid:13)
[Equation]
Proof.
[Equation]
1 n n (cid:88) i =
C i ( a i ) − 1 n n (cid:88) i = a i (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) 2   =
ng (cid:88) k
=
E   (cid:13) (cid:13)(cid:13)(cid:13)(cid:13)(cid:88) i ∈G k
C i ( a i ) − (cid:88) i ∈G k
a i (cid:13) (cid:13)(cid:13)(cid:13)(cid:13) 2   +n(cid:88) k (cid:54) = p
E   (cid:42)(cid:88) i ∈G k
C i ( a i ) − (cid:88) i ∈G k
a i , (cid:88) i ∈G p
C i ( a i ) − (cid:88) i ∈G p
a i (cid:43)  .
Due to independence and unbiasedness, the last term vanishes, and, using AB inequality, we get
E 
[Equation]

[Equation]
n
[Equation]
=
[Equation]
E
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:88)
[Equation]
k a
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]

[Equation]
=
[Equation]
=
[Equation] [Equation]
G
[Equation]
(cid:88)
[Equation]
k
[Equation]
(cid:88)
[Equation]
∈G k
a
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) 2  
[Equation]
(cid:88)
[Equation]
= |G
[Equation]
|
[Equation]
n
[Equation]
(cid:88)
[Equation]
∈G k (cid:107)
[Equation]
(cid:107)
[Equation]
(cid:33)
[Equation]
k (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) 1 |G k
[Equation]
(cid:88)
i ∈G
k
a i (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) 2   .
From this we can get the result.
Next, we prove analogous lemma to Lemma 5.
Lemma 18.
Let us consider g t +1 from Line 8 of Algorithm 1 and assume, that Assumptions 3 andhold.
Moreover, if Assumption 2 holds for every group G k , for k ∈ { 1 , · · · , g } , then
[Equation] [Equation]
Proof.
In the view of deﬁnition of g t +
[Equation]
we get
[Equation]
(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) x t +   =
(1 − p )
E   (cid:13) (cid:13)(cid:13)(cid:13)(cid:13) 1 n n (cid:88) i =
C i (cid:0) ∇ f i ( x t +1 )
− ∇ f i ( x t ) (cid:1) − ∇ f ( x t +1 )
+
[Equation]
f ( x t ) (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) 2 (cid:12) (cid:12) (cid:12)(cid:12)(cid:12)(cid:12) x t +   + (1 − p ) (cid:13)(cid:13)
g t − ∇ f ( x t ) (cid:13)(cid:13) 2 .
[Equation]
[Table]
[Equation]
[Table]
[Equation]
1 − p ) (cid:13)(cid:13)
g t − ∇ f ( x t ) (cid:13)(cid:13)= (1 − p ) (cid:32) g (cid:88) k
= ( A G k − B G k ) |G
k |n
L G k + ( x t +1 , x t ) + g (cid:88) k
=
B G k |G k |n
L G k ± ( x t +1 , x t ) (cid:33) + (1 − p ) (cid:13)(cid:13)
g t − ∇ f ( x t ) (cid:13)(cid:13)≤ (1 − p ) (cid:32) g (cid:88) k
= ( A G k − B G k )
|G
k |n(cid:16)
L G k + (cid:17) 2 + g (cid:88) k
=
B G k |G k |n(cid:16)
L G k ± (cid:17) 2 (cid:33) (cid:13)(cid:13) x
t +1 − x t (cid:13)(cid:13)+ (1 − p ) (cid:13)(cid:13)
g t − ∇ f ( x t ) (cid:13)(cid:13) 2 .
Let us deﬁne (cid:98) L 2 G :=
(cid:32) g (cid:88) k
= ( A G k − B G k )
|G
k |n(cid:16)
L G k + (cid:17) 2 + g (cid:88) k
=
B G k |G k |n(cid:16)
L G k ± (cid:17) 2 (cid:33) .
Theorem 10.
Let Assumptions 1, 2, 3 and 6 be satisﬁed.
Let the stepsize in MARINA be chosen as γ ≤ (cid:18)
L − + (cid:114) 1 − p p (cid:98) L 2 G (cid:19) −, then after T iterations, MARINA ﬁnds point ˆ x T for which E (cid:104)(cid:13)(cid:13) ∇ f
(ˆ x T ) (cid:13)(cid:13) 2 (cid:105) ≤ 2∆ 0 γT .
Theorem 11.
Let Assumptions 1, 2, 3, 5 and 6 be satisﬁed and
[Equation]
then for x T from MARINA algorithm the following inequality holds:
[Equation]
We omit proofs of this theorems as they repeat proofs from Appendix C.6 and D.2; the only difference is that we have to take (cid:98) L 2 = (cid:98)
L 2 G .
Let us assume that n ≤ d , all groups have equal sizes |G
k | =
G and constants L G k ± =
L G ± , for all k ∈ { 1 , . .
. , g } , and in each group we use Perm K compressor from Deﬁnition 2, thus communication complexity predicted by our theory is the following:
[Equation]
Using the same reasoning as in Lemma 13, we can take p = 1 or p = 1 /G to get that
[Equation]
For the case when we have one group, we restore the communication complexity from Lemma 13.
Comparing (34) with (51), we see that dL − /n from (34) is always better than dL − /G from (51); however; if dL ± / √ n is a bottleneck and L G ± is small, then communication complexity (51) can be better.
Let us consider an example of a quadratic optimization task with two groups, wherein one group, all matrices are equal to A , and in another one, all matrices are equal to B , A (cid:54)
=
B , A
= A (cid:62) (cid:60) 0 and
B = B (cid:62) (cid:60) 0 , then G = n/ 2 , L G ± = 0 , and L ± > 0 (see Example 3).
Hence, we get that
[Equation]
This bound is better than (34) by at least the factor 1 + √
[Equation]
±
[Equation]