[-and C := max (cid:0)(cid:113) 
-]
we note that by ﬁxing the seed of random number generator, the random matrix B can be generated by each agent locally, without needing to transmit any information.
Decentralized Algorithm.
We propose a fully decentralized algorithm for solving (4) with a communication complexity that approaches the lower bound in Theorem 3.2.
Our idea is similar to
Alg. 1 that adopts a dimension reduction step for compression, yet we shall encapsulate the Quantize ,
Communicate , Aggregation , Broadcast steps through a decentralized average consensus protocol with support for quantized communication.
In particular, we consider a setting where K agents are represented as nodes on a connected graph ( V, E ) with a doubly stochastic mixing matrix W 2 R K ⇥ K + .
Observe the following fact:
Fact 1.
Consider a set of vectors v 1 , . .
. , v K 2 R H held by K agents on ( V, E ) .
For any ✏ > 0 , the
CHOCO-GOSSIP protocol [Koloskova et al., 2019] is described as the map:
[Equation]
such that ¯ v k is the vector available at agent k satisfying k ¯ v k  K
P
Kj =1 v j k 2  ✏ .
Let ⇠ = P Kk =1 k v [Equation]
[-Kk =
-] K
[Equation] P Kj = [Equation]
j k 2 be the initial consensus error, the protocol requires at most [Equation] [Equation] log( ⇠ / ✏ ) rounds of quantized communication , where  = 1  |  2 ( W ) | and ! 2 (0 , 1] is controlled by the number of bits in the quantizer.
Details about the CHOCO-GOSSIP protocol can be found in Appendix F.1.
We replace line 5–line 8 in Alg. 1 by CHOCO-GOSSIP to yield the decentralized algorithm:
Algorithm 2 Decentralized Gradient Descent with Compressed Comm. via Linear Compression
[-1: Input: step size η ; initial iterate θ 0 ∈ R D . 2: Initialize: θ 
 i = ¯ θ 0 , ∀ i ∈ [ K ] 3: for t in 0 , . .-]: Input
: Initialize do 4:
Compute : Each agent compute the compressed local gradient g tk = F ( r f k ( ✓ tk )) .
Consensus : Agents exchange gradients (¯ g t 1 , . .
, ¯ g tK ) =
CHOCO-GOSSIP ( g t 1 , . .
. , g tK ; ¯ ✏ t +1 ) .
Update : Each agent updates ✓ t [-+
-]+1 k
[Equation]
7: end for 8: Output: the last iterate ✓ Ti
Theorem 4.3.
Consider using Alg. 2 to solve the quadratic problem (4) , with A being full row rank.
Choose B such that rank ( BA > ) =
N ; set ⌘ =  min ( Z ) 4  2max ( Z ) , and choose ˜ F ( · ) and F ( · ) as
F ( x ) =
Bx, ˜ F ( y ) =
B > y .
For any T  1 , the following holds true:
[Equation]
where  was deﬁned in Theorem 4.2.
Moreover, suppose that A, B are constructed as in Proposition 1, to compute an ✏ -optimal solution (3) , each agent is required to transmit:
[Equation]
with probability at least 1  ⇣ .
Proposition 1 shows that the condition number  is O (1) .
To satisfy rank ( BA > ) =
N , we need
H  N .
Altogether, this yields the communication complexity
O
( N log( N ) log( D ) log(1 / ✏ ) 2 ) to match Theorem 3.2 up to a log( D ) factor.
The Case of Overparameterized Neural Network.
We now explore the possibility of extending the analysis of quadratic problems to a speciﬁc overparameterized neural network, still belonging to class C op .
We will customize Alg. 1 to partially compress the gradients of the loss function.
Our idea is to design F ( · ) (resp.
˜ F ( · ) )
so that we compress (resp.
decompress)
the gradient with respect to the parameters in the widest layer to O ( N ) bits, while directly quantize the remaining gradients.
Note that for a few popular network architectures such as Neural Tangent Kernel (NTK)
[Huang et al., 2021], the widest layer contains most of the parameters ⌦ ( N 4 ) .
Therefore, compressing these parameters into O ( N ) bits already represents signiﬁcant communication .
We will leave the design and analysis of more aggressive schemes which compress the parameters of all layers for future work.
To proceed, let us ﬁrst state the structure of the network.
Let X 2 R N ⇥ n 0 denote the data sample; let a ( · ) be an activation function; let O l 2 R N ⇥ n l as the output of each layer, which can be expressed as
[Equation]
where W l 2 R n l  1 ⇥ n l is the weight matrix in l -th layer.
Further, denote O L,k as the output of the network for k -th agent.
Let ✓ =
( W l ) Ll =1 collect all the weights.
Then objective function becomes
[Equation]
Further, we deﬁne the vectorized gradient of each layer as u l = vec ( r W l f ( ✓ )) and u l,k for each agent.
Now we describe the assumptions on the network.
Assumption 4.
(Pyramidal network topology) Let n 1  N and n 2  n 3  . . .
n L = 1 .
Assumption 5.
(Activation function)
Fix ⌫ 2 (0 , 1) and ⇢ > 0 .
Let a satisfy that: ( i ) a 0 ( x ) 2 [ ⌫ , 1] , (ii) | a ( x ) | 6 | x | for every x 2 R , and (iii) a 0 is ⇢ -Lipschitz.
As showed in [Nguyen and Mondelli, 2020], with Assumption 4 and 5, objective (6) belongs to C op .
With the above Pyramidal structure, layer l = 2 is the widest layer and contains the largest number of parameters, denoted as u 2 2 R n 1 ⇥ n 2 .
As mentioned above, we will design F and ˜ F to compress and decompress u 2 .
To proceed, Let us choose B = I n 2 ⌦ ˜ B 2 R
Hn 2 ⇥ n 1 n 2 where ˜ B 2 R H ⇥ n 1 , such that N 6 H ⌧ n 1 .
Then F and ˜ F are deﬁned as:
[Equation]
This means that we compress the parameters in the second layer, and perform usual gradient descent for the layers l  3 ; further, the parameters in the ﬁrst layer is left frozen at its initialization.
Note that we freeze the ﬁrst layer for some technical reasons, but layer freezing itself is a useful technique to accelerate the training in practice, which can provide strong generalization ability [Yosinski et al., 2014, Advani et al., 2020].
Let us denote O 01 as the output of the ﬁrst hidden layer at initialization.
We have the following result.
Its proof as well as the expressions for various constants, can be found in Appendix G.
Theorem 4.4.
(Informal) Consider using Alg. 1 to solve the quadratic problem (6) , with X being full row rank.
Suppose ✓ 0 is initialized properly (see Appendix G for details).
Choose ˜ B such as rank ( ˜ BO 01 > )
=
N ; choose ˜ F ( · ) and F ( · ) as in (7) .
Then there exists stepsize ⌘ , constant  , C, b such that to compute an ✏ -optimal solution (6) , each agent is required to transmit:
[Equation]
where ⌘ ,  are related to X, ⌫ , ⇢ and singular values of the weight matrices at initialization.
Theorem 4.4 shows that if we choose H = ⇥ ( N ) , and when n 1 is very large, then the total communication complexity can be reduced by a factor of n 1 /H .
Preliminary Numerical Experiments.
We conclude by presenting a numerical experiment for the
UCI Tom’s Hardware 2 dataset using Alg. 2 where we applied b log( t +100) c rounds of communication at the t -th iteration for the CHOCO-GOSSIP subroutine; see Appendix F.1.
We consider a ring network with K = 5 agents, each one has 500 or 1000 samples (thus making N = 2500 , or N = 5000 ).
We construct D -dimensional features from the dataset as NTK features [Bietti and Mairal, 2019].
In Fig.
1, we train a least square regression model in the overparameterized regime.
We observe that the communication cost needed is proportional to the number of samples, instead of the feature dimension.
Particularly, our results indicate that the communication cost required to reach a target train MSE (of 10  4 ) is proportional to the number of training samples.
Acknowledgments and Disclosure of Funding We sincerely thank the anonymous reviewers for the suggestions.
M. Hong and B. Song are supported by NSF grants CIF-1910385, CMMI-1727757 and
CNS-2003033.
A Additional Related Literature and Comparison
In the ﬁrst category, where the commmunication complexities are measured by rounds of communications (where in each round real-valued vectors get exchanged), recent works [Kovalev et al., 2021, 2022] provides lower bounds for a sum of smooth and strongly convex functions over time-varying networks, and for strongly monotone variational inequality problems in a stochastic (ﬁnite-sum) setting.
Other related works include [Scaman et al., 2017, 2018, Arjevani and Shamir, 2015].
In addition, lower bounds for non-convex problems are considered in [Sun and Hong, 2019, Lu and
De Sa, 2021].
Further, there are many works that derive rounds of communication upper bounds for decentralized and federated learning algorithms, see, e.g.,[Stich and Karimireddy, 2019, Stich et al., 2018, Patel and Dieuleveut, 2019, Yu et al., 2019, Wang and Joshi, 2018, Gorbunov et al., 2021].
A recent work [Gorbunov et al., 2021] has analyzed the communication efﬁcient algorithm to solve functions that satisfy PL conditions.
However, it has been focused on analyzing the number of communication rounds needed to achieve certain ✏ optimal solution, while the current paper is focused on ﬁnding the minimum bits to be communicated.
Therefore, although the two works are both about developing communication efﬁcient algorithms for PL functions, the bounds obtained in these works represent different physical quantities, thus cannot be directly compared.
B Communication Complexity Lower Bounds
Let us introduce the so-called equality problem Vempala et al.
[2020], denoted as EQUAL .
Deﬁnition 5 (Equality Problem) .
Consider a set of K agents, each is given an input c k 2 { 0 , 1 } m ; m is the length of the binary input.
Then, the EQUAL problem is deﬁned as follows:
[Equation]
For any deterministic algorithms, the communication complexity lower bound of
EQUAL m ( c
[Equation]
[Equation]
. , c K ) is ⌦ ( Km )
[Vempala et al., 2020, Thm 3.5].
Further, the intermediate steps required to derive the lower bounds involve packing arguments.
Therefore, we provide below a lower bound for the maximum number of points that we can pack into a compact set [0 , 1] n , such that the distance between each pair of points is at least  .
Deﬁnition 6 (Packing Problem) .
We deﬁne the following:
• For a given  > 0 we deﬁne the set S (  ) ✓ [0 , 1] n such that k x  y k >  , 8 x, y 2 S (  ) .
• Assuming that | S (  ) |  2 m , we deﬁne a function h : { 0 , 1 } m ! S (  ) .
For u, v 2 { 0 , 1 } m it holds that u
[Equation]
Lemma B.1.
([Korhonen and Alistarh, 2021, Lemma 2]) For a set S (  ) ✓ [0 , 1] n deﬁned in Def. 6, it holds that | S (  ) |  [Equation]
Next, we repeat here for completeness the Assumptions that the local functions in the Distributed PL problem class satisfy (Def. 1).
Assumption 6.
The local objective functions satisfy:
[Equation]
where ✓ ⇤ ( k ) is a global minimum of f k ( · ) ; µ k ’s some positive constants.
Assumption 7.
There exists positive constants L k ’s and L such that:
[Equation]
In the Lemma below we show that the function instance we are going to use in Theorem 3.1 satisﬁes the PL condition and both Assumptions 6 and 7.
Lemma B.2.
The function f k ( ✓ ) = 12 k ✓  c
k 22 + sin 2 ( ✓ k  c k ) , c 2 R D (where with c k , ✓ k we denote the k th component of vectors c, ✓ , respectively) satisﬁes the PL condition with µ = 18 .
That is, it holds that f k ( ✓ )  f k ( ✓ ⇤ )  12 µ kr f k ( ✓ ) k 2 , 8 ✓ 2 R D , where ✓ ⇤ is the global minimum of f k ( · ) .
Remark 1.
The constructed function f k ( ✓ ) and the respective sum across nodes P Kk =1 f k ( ✓ ) satisfy
Assumptions 6, 7.
Speciﬁcally, Assumption 6 follows trivially from Lemma B.2.
In addition, it can be shown that f k ( ✓ ) and
P
Kk =1 f k ( ✓ ) have bounded Hessians, which implies the Lipschitz gradient property of Assumption 7.
Proof of Lemma B.2.
To begin with we are going to show that the (one dimensional) function
[Equation]
where ✓ k , c k 2 R , satisﬁes the PL condition with µ = 18 ; notice that e f k ( ✓ k ) has a unique minimum with value 0 attained at ✓ k = c k , and we have that
[Equation]
Then, we are going to use this result to prove the PL condition for the function f k ( ✓ ) .
So, let us deﬁne the function
[Equation]
The gradient of the above function is given by
[Equation]
In order to show that e f k ( ✓ k ) satisﬁes the PL property it sufﬁces to prove that g ( ✓ k )  0 , 8 ✓ k 2 R .
Then, it will hold that
[Equation]
sin [Equation] that is the PL condition of e f k ( ✓ k ) will be satisﬁed.
First, notice that
[Equation]
It is clear from the above expression that the term 72 ( ✓ k  c k ) 2 dominates the value of the objective for large values of | ✓ k  c k | .
Therefore, the objective does not become unbounded below.
Secondly, consider the stationary points of g ( ✓ k ) .
Since the objective does not become unbounded below the only possible global minima of g ( ✓ k ) are its stationary points.
We are going to show that the values of the objective at those points is non-negative, effectively proving that g ( ✓ k )  0 , 8 ✓ k 2 R .
The stationary points of g are deﬁned by the following expressions:
[Equation]
Notice that in the interval ✓ k 2 ( c k , ⇡ 2 + c k ⇤ it holds that sin [2( ✓ k  c k )]  0 but  ( ✓ k  c k ) < 0 .
Similarly, for ✓ k⇥  ⇡ 2 + c k , c k ) it holds that sin [2( ✓ k  c k )]
but  ( ✓ k  c k ) > 0 .
Also, for ✓ k / [-∈ (cid:2) − π 2 + c k , c k ) ∪ ( c k , π 
-][Equation] [Equation]
k } it holds that | ✓ k  c k |  ⇡ 2 > 1 .
Therefore, in all the above cases the equation of the stationary points does not have a solution (and thus there are no stationary points) Finally, note that the
equation has a trivial solution at ✓ k = c k , which corresponds to the only stationary point we can get from this equation.
For that point it holds that g ( c k ) =
• cos (2( ✓ k  c k ))
=
From the above equations, and by using the proper trigonometric identities it follows that sin
[Equation] [Equation]
✓
[Equation]
[Equation]
and ✓ [Equation]
⇡ ⇡ ± 2 . 022 .
Then if we plug the above values into g ( ✓ k ) we get
[Equation] [Equation]
It can be easily veriﬁed that ( ⇡ ± 1 . 01) [Equation] for all  2 Z .
In conclusion for all the stationary points ˆ ✓ k it holds that g (ˆ ✓ k )  0 .
As a result, we can claim that g ( ✓ k )  0 , 8 ✓ k 2 R , and the function e f ( ✓ k ) satisﬁes the PL property.
Next, consider the PL property (with µ = 18 ) for the function f k ( ✓ ) , that is
[Equation]
where e k is a vector of all zeros except at index k .
We have shown above that
[Equation]
Also, it trivially holds that
[Equation]
Adding inequalities (10) and (11) we obtain condition (9), which ensures the PL property for the objective f k ( ✓ ) , with µ = 18 .
This completes the proof.
Proof of Theorem 3.1.
To begin with, let ⇡ 2 ⇧ ( ✏ ) be an arbitrary protocol that solves the problems in C pl .
Then, for any input (i.e., for any speciﬁc problem within the class C pl ) the protocol returns an ✏ -approximate minimum ˜ ✓ as given in Def. 3.
Moreover, consider the set S (  ) ✓ [0 , 1] D introduced in Def. 6 (where in place of n we have D ) with  = 2 p 2 ✏ .
Then, Lemma B.1 implies that
[Equation]
Then, we set m = ⇥ (log 2 | S (  ) | ) = ⇥
D log  D ✏  , where the exact value of m is selected such that | S (  ) | > 2 m holds.
As a result, under the assumption that D ✏ = ⌦ (1) , it holds that | S (  ) |  2 , and m  1 .
Next, we will show that protocol ⇡ also solves the EQUAL problem.
That is, we will reduce every instance of EQUAL m ( u 1 , . . .
, u K ) to a problem in C pl .
Towards this end, for an arbitrary input ( u 1 , . . .
, u K ) 2 { 0 , 1 } Km we select the following function from C pl ,
[Equation]
where h ( · ) is introduced in Def. 6, and h ( u k ) 2 S (  ) ✓ [0 , 1] D ; ( h ( u k ))
k , ✓ k denote the k th component of h ( u k ) and ✓ , respectively.
It is shown in Lemma B.2 that the functions f k ’s that correspond to each node satisfy the PL condition, with µ k = 18 , for all k .
We can also easily verify that the rest of the conditions in Assumption 6 – 7 are satisﬁed.
Then, we have the following cases:
• Case 1 (equal inputs): It holds that u 1 = . . .
=
u K := u .
Thus, we have that h ( u 1 )
= . . .
=
h ( u K ) =
h ( u ) and the minimum of (12) is 0 , attained at some point ✓ ⇤
such that ✓ ⇤ = h ( u ) .
Also, protocol ⇡ returns an approximate minimum f (˜ ✓ ) of (12), for which it holds that
[Equation]
where in (a) we used (3); and in (b) we exploited the fact that the minimum is 0 .
• Case 2 (inputs not equal): There exists a pair of nodes ( i, j ) such that u i 6 = u j .
As a result, h ( u i ) 6 = h u j ) .
Then, protocol ⇡ returns an approximate minimum f (˜ ✓ ) of (12) for which it holds that
[Equation]
where in (a) we used (3), in (b) we used the characteristic property (i.e., the minimum distance between two points is k x  y k >  , 8 x, y 2 S (  ) ) of set | S (  )
| (from Def. 6)
and in (c) we use the quantity  = 2 p 2 ✏ .
From the above analysis we see that if f (˜ ✓ )  ✏ then EQUAL m ( u 1 , . . .
, u K )
= 1 .
Otherwise, if we assume that EQUAL m ( u 1 , . .
, u K )
= 0 , then the analysis of case 2 implies that f (˜ ✓ ) > ✏ , a contradiction.
Similarly, we can claim that if f (˜ ✓ ) > ✏ , then EQUAL m ( u 1 , . . .
, u K )
= 0 .
In the opposite case (i.e., if f (˜ ✓ ) > ✏ = ) EQUAL m ( u 1 , . . .
, u K ) = 1 ) we see from case 1 that f (˜ ✓ )  ✏ , that is we reach a contradiction.
In summary, we have that
[Equation]
Finally, the fact that the communication complexity of EQUAL m ( u 1 , . .
, u K ) is ⌦ ( Km ) , and the above reduction imply that ⌦ ( Km )
=
⌦ ( KD log  [Equation]  ) is a lower bound for the communication complexity of C pl .
Proof of Theorem 3.2.
To begin with, let ⇡ 2 ⇧ ( ✏ ) be an arbitrary protocol that solves the problems in C op .
Then, for any input (i.e., for any speciﬁc problem within the class C op ) the protocol returns an ✏ -approximate minimum ˜ ✓ , as described in Def. 3.
Moreover, consider the set S (  ) ✓ [0 , 1] N introduced in Def. 6 (where in place of n we have N ) with  = 2 p 2 ✏ .
Then, Lemma B.1 implies that
[Equation]
Then, we set m = ⇥ (log 2 | S (  ) | ) = ⇥
N log  N ✏  , where the exact value of m is selected such that | S (  ) | > 2 m holds.
As a result, under the assumption that N ✏ = ⌦ (1) , it holds that | S (  ) |  2 , and m  1 .
Now, we are going to show that protocol ⇡ also solves the EQUAL problem.
That is, we are going to reduce every instance of EQUAL m ( u 1 , . . .
, u K ) to a problem in C op .
To be more precise, for an arbitrary input ( u 1 , . .
, u K ) 2 { 0 , 1 } Km we select the following instance from C op , f ( ✓ ) = K
[Equation]
f k ( ✓ ) with f k ( ✓ ) = 1 2 k G ( ✓ )  h ( u k ) k 22 , (13) where h ( · ) is introduced in Def. 6; and with h ( u k ) 2 S ✓ [0 , 1] N .
Then, we have the following cases:
• Case 1 (equal inputs): It holds that u 1 = . . .
=
u K := u .
Thus, we have that h ( u 1 )
= . . .
=
h ( u K ) =
h ( u ) and the minimum of (13) is 0 , attained at some point ✓ ⇤ such that G ( ✓ ⇤ )
=
h ( u ) .
Also, protocol ⇡ returns an approximate minimum f (˜ ✓ ) of ( for which it holds that
[Equation]
where in (a) we used (3), and (b) follows from the fact that the minimum of (13) is 0 in this case (i.
e ., equal inputs)
• Case 2 (inputs not equal): There exists a pair of nodes ( i, j ) such that u i 6 = u j .
As a result, h ( u i ) 6 = h u j ) .
Then, protocol ⇡ returns an approximate minimum f (˜ ✓ ) of ( for which it holds that:
[Equation]
where in (a) expression (3) is used, in (b) we used the characteristic property (i.e., the minimum distance between two points is k x  y k >  , 8 x, y 2 S (  ) ) of set | S (  ) | (from
Def. 6), and in (c) we use the quantity  = 2 p 2 ✏ .
From the above analysis we see that if f (˜ ✓ )  ✏ then EQUAL m ( u 1 , . . .
, u K )
= 1 .
Otherwise, if
EQUAL m ( u 1 , . .
, u K )
= 0 , then the analysis of case 2 implies that f (˜ ✓ ) > ✏ , a contradiction.
Similarly, we can claim that if f ( ˜ ✓ ) > ✏ , then EQUAL m ( u 1 , . . .
, u K )
= 0 .
In the opposite case (i.e., if f (˜ ✓ ) > ✏ = ) EQUAL m ( u 1 , . . .
, u K ) = 1 ) we see from case 1 that f (˜ ✓ )  ✏ , that is we reach a contradiction.
In summary, we have that
[Equation]
Finally, the fact that the communication complexity of EQUAL m ( u 1 , . .
, u K ) is ⌦ ( Km ) , and the above reduction imply that ⌦ ( Km )
=
⌦ ( KN log  N ✏  ) is a lower bound for the communication complexity of C op .
C Proof for Theorem 4. C.1 Proof of Lemma 4.
Let us consider the function
[Equation]
Since r f ( ✓ ⇤ ) = 0 , it is easy to see  ( ✓ ) > 0 .
It can be derived directly that  ( ✓ ⇤ ) = 0 , which means  ( · ) can achieve the minimum at ✓ ⇤ .
So we know ✓ ⇤ 2 argmin  ( ✓ ) .
Then we have the following inequality holds:
[Equation]
where (i) performed one step of gradient decent; (ii) uses the fact
that r f ( ✓ ⇤ ) = 0 ; (iii) is because the Lipshcitz gradient assumption.
Then it follows directly
[Equation]
C.2 Proof for Theorem 4.
First, let us state the sketch of the proof.
Denote g t := K
[Equation]
Step1: We show that the loss function decreases linearly if all the agents update the averaged gradient in one step.
That is, the following holds true.
[Equation]
Step2: We show by induction that for t = 1 , · · · , the following inequalities hold true:
[Equation]
where {  t } 1 t =1 is a sequence of positive numbers.
The proof of Step 1 is straightforward:
[Equation]
where ( i ) is by Decent Lemma; (ii) uses the deﬁnition of g t ; (iii) uses the choice of ⌘ ; (iv) is because
L > 1 ; (v) uses Assumption 1.
Now we prove Step 2 .
To begin with, let us set ⌘ = 1 2 L 32 , and set: [Equation] First
[Equation]
let us verify that (14) holds true when t = 1 .
We have the following series of inequalities:
[Equation]
where (i) comes from the Decent Lemma; (ii) uses the Young’s inequality with constant  ; (iii) uses
Assumption 1; (iv) is from (16); (v) uses the relationship between ` 2 and ` 1 norm; (vi) uses the triangle inequality; (vii) uses the Cauchy-Schwartz inequality; (viii) uses the initialization condition k
[Equation]
(ix) uses the choice of  .
Now let us deﬁne the quantization at initialization.
For each entry in q 0 k , denoted as ( g 0 k ) j , we consider the interval ⇥ [Equation]
⇤ , which is constructed by the closest integers.
We use b 0 bits to make the grid, and quantize each element of the vector by the closest point on the grid.
It is clear that the quantization error for each element is at most
[Equation]
So we set
[Equation]
or equivalently, b 0 = log
[Equation]
Now since we have chosen C > 16 Lµ 2 + L 32 µ , we can bound the coefﬁcient in front of f ( ✓ 0 ) as:
[Equation]
18) where in (i) we plugged in the choice of ⌘ and D > 1 ; (ii) is because we have assumed that L >and D > 1 ; (iii) uses the choice of C in (17).
It follows f ( ✓ 1 ) {+
+} (1  µ 8 L 32 ) f ( ✓ 0 ) .
Thus, (14) holds for t = 1 .
Second, let us analyze (15) for t = 1 .
Observe that:
[Equation]
where (i) uses Assumption 2; (ii) is from triangle inequality; (iii) uses the relationship between ` 2 and ` 1 norm; (iv) uses the condition k q 0 k  g 0 k k 1 6 p f k ( ✓ 0 ) CD 3 p K ; (v) uses Assumption 1 and the
Cauchy-Schwartz inequality; (vi) plugs in the choice of stepsize and the fact that C > 100 ; (v) compares the left side with deﬁnition of ↵ in (17).
By Lemma 4.1, it follows that:
[Equation]
Next, let us use induction to prove (14) and (15).
First, we analyze the decent of f ( ✓ t ) :
[Equation]
where (i) comes from the Decent Lemma [Equation] (ii) uses the Young’s inequality with constant  ; (iii) uses
Assumption 1; (iv) is from (16) and induction assumption (15); (v) uses the induction assumption (14); (vi) uses the deﬁnition of  t  1 .
Finally, set  = 2(1  ⌘ µ ) L µ , we will get the last equality.
Next, let us analyze the coefﬁcient in the expression (19):
[Equation]
where in (i) we plugged in the choice of ⌘ ; (ii) is because we have assumed that L > 1 ; (iii) plugged in the choice of stepsize; (iv) uses the deﬁnition of ⌧ in (17); and the last equality comes from the deﬁnition of ↵ in (17).
Plugging (20) to (19), we obtain f ( ✓ t +1 ) 6 ( ↵ ) t +1 f
( ✓ 0 ) .
Second, we show that (15) holds for t
+ 1 .
We have:
[Equation]
where (i) uses Assumption 2 and the induction assumption; (ii) uses the triangle inequality to decompose k q t k and uses the relationship between ` 1 and ` 2 ; (iii) is from Assumption 1 and induction assumption (15); (iv) uses the induction assumption (14); (v) uses the deﬁnition of  t ; (vi) is because µ < 1 ; (vii) comes from the fact that ⌧ < 110 D in (17); (viii) compares the choice of ↵ in (17).
Thus, we obtain
[Equation]
t .
Then by Lemma 4.1, we have (15) holds for t + 1 .
Now we have proved by induction that (14) and (15) hold.
So for t > 0 , there is [Equation]
, where ↵ = 1  µ 8 L 32 .
Thus, to compute an ✏ -optimal solution, the total number of iterations required is log( f ( ✓ 0 ) / ✏ ) /
log(1  µ/ 8 L 32 ) .
Since in each iteration, each agent k transmits a length- D vector q tk , it follows that the total number of bits each agent needs to communicate is D log( f ( ✓ 0 ) / ✏ ) / log(1 / (1  µ/ 8 L 32 )) bits.
Notice that log(1 / (1  µ/ 8 L 32 ) =
log(1  µ/ 8 L 32 ) ⇠ 8 L 32 /µ , so we can derive the simpliﬁed total number of bits as bD · 8 L 32 µ log  [Equation]
D The Proof of Theorem 4.
First, let us provide the sketch of the proof.
Denote
[Equation]
Using the above notation, the agents’ local update step (i.e., the ‘Update’ step in Alg. 1) can be expressed as:
[Equation]
Step 1: We show that the loss function decreases linearly if all the agents update parameters using the direction ˜
g t , as follows:
[Equation]
Step 2: Let ⌧ = 1 2 b  1 , we show by induction that for t = 1 , · · · , the following inequalities hold true:
(1) f ( ✓ t ) 6 ( ↵ ) t f ( ✓ 0 ) , for some 0 < ↵ < 1 .
[Equation]
where {  t } 1 t =1 is a sequence of positive numbers.
We prove Step 1 ﬁrst.
At t -the iteration, Let us ﬁrst expand the objective function as:
[Equation]
To proceed
[Equation]
let us provide explicit expressions for g t and ˜ g t :
[Equation]
By using the above
[Equation]
the inner product in (24) can be bounded as follows
[Equation]
Using the above relations
[Equation]
we can further bound the descent of the objective function as
[Equation]
where (i) comes from plugging (27) and (26) into (24); (ii) extracts the largest eigenvalue of Z ; (iii) is due to the deﬁnition of the objective function; the last two equalities hold due to the deﬁnition ⌘ =  min ( Z )  2max ( Z ) .
Next
[Equation]
we prove Step 2 .
To begin with
[Equation]
let us deﬁne:
[Equation]
Further let us set
[Equation] [Equation]
⌧ given by
[Equation]
[Equation]
First
[Equation]
we analyze (22) for t = 1 .
We have the following relations:
[Equation]
[Equation]
where (i) explicitly expands the f ( · ) function
[Equation]
and uses (28); (ii) applies the Young inequality with constant  ; (iii) extracts the largest singular value of A , and uses (28); (iv) set  = 1  min ( Z ) ; (v) uses the relationship between ` 2 and ` 1 norm; (vi) uses the fact that k B > q 0  ˜ g
[Equation]
(vii) uses the Cauchy-Schwartz inequality; (viii) uses the condition  q 0 k
[Equation]
g
[Equation]
1p f[-( · ) function, and uses (28); (ii) applies the Young inequality with constant β ; (iii) extracts the largest singular value of A , and uses (28); (iv) set β = 1 σ min ( Z ) ; (v) uses the relationship between ℓ 2 and ℓ ∞ norm; (vi) uses the fact that ∥ B ⊤ q 0 − ˜ g 
 ∥ ⩽ H 2 ∥ ˜ B ∥ ∞ ∥ q 0 − g 
-]
( ✓ 0 )
CD p K in
Algorithm 1; (vii) uses the induction assumption (22) and the deﬁnition of  t  1 in (29); (viii) plug in the choice of stepsize; (ix) comes from the choice of  and ⌧ .
So we have showed (22) holds for t = 1 ; ( ix ) plug in the choice of stepsize and constant C > 1  2 ⌧ 2 ; (x) uses the choice of  and ⌧ .
Next, let us analyze (23) for t = 1 .
The idea is that, if we can show that k g 1  q [-
 ∥ ∞ ⩽ γ-]0 k 1 6  0 , then we will be able to use Lemma 4.1 to show (23).
More speciﬁcally, we have:
[Equation]
where ( i ) and ( iii ) come from the ‘Quantize’ step in Algorithm 1; ( ii ) and ( iv ) are from the two inequalities in Lemma 4.1 (assuming that k g 1  q [-
 ∥ ∞ ⩽ γ-]0 k 1 6  0 holds).
Next, we show k g 1  q 0 k 1   0 .
We observe that:
[Equation]
+ k g
[Equation]
q
[Equation]
k( v ) 6 ⌘ max ( Z )
[Equation] k BA > ( A ✓ 0  b ) k + H k q [Equation]
g 0 k 1 ) + k g
[Equation]
q 0 k( vi ) 6 ⌘ 32 max ( Z ) k
A ✓ 0
[Equation]
k
[Equation] (1 + ⌘ H  max ( Z )) k g 0  q 0 k ( vii ) 6 ⌘ 32 max ( Z )
[Equation]
A
[Equation]
max
[Equation]
Z ))
[Equation]
viii ) 6 ⌘
[Equation]
Z ) p
[Equation]
[Equation]
ix ) 6 ⌘ 32 max
[Equation]
Z ) p
[Equation]
⌘
[Equation]
Z )
[Equation] CD [Equation] [Equation]
where (i) is due to the triangle inequality; (ii) expands the expression of g 1 and g 0 in (25), uses the relation between ` 2 norm and ` 1 norm and uses the update rule (21); (iii) uses the fact that non-zero eigen values of BA > AB > and Z are the same and extracts the largest eigen value of Z ; (iv) uses triangle inequality ; (v) uses the relationship between ` 2 and ` 1 norm; (vi) is because s 2max ( BA > ) =
max ( Z ) and extract the largest singular value of BA > ; (vii) uses triangle inequality; (viii) uses the initial condition k q 0 k
g {+
 k+} k 1 6 p f k ( ✓ 0 ) C
· D p K in Algorithm 1; (ix) uses Cauchy-Schwartz inequality; (x) plug in the choice of stepsize; (xi) is because  > 1 , H 6 D, D > 1 and the choice of  ; (xii) is from C > 12  ; the last inequality comes from p ↵ > p 22 > 13 since  > 1 .
So we can show  g 1  q 1   0 .
Next, we will show (22) holds for t + 1 by induction, based on the base assumption that (22) and (23) holds for t .
We have the following series of relations:
[Equation]
where (i) we have explicitly expands the f ( · ) function, and have used (28); (ii) applies the Young inequality with constant  ; (iii) extracts the largest singular value of A , and uses (28); (iv) set  = 1  min ( Z ) ; (v) uses the relationship between ` 2 and ` 1 norm; (vi) uses the second inequality in induction assumption (23); (vii) uses the induction assumption (22) and the deﬁnition of  t  1 in (29); (viii) plug in the choice of stepsize; (ix) comes from the choice of  and ⌧ .
Finally, we will show (23) holds for t + 1 by induction, again base assumptions (22) holds for t and (23) holds for t .
We have:
[Equation] [Equation]
where (i) uses the triangle inequality; (ii) expands the ﬁrst term by (25), uses the relationship between ` 2 and ` 1 norm, and and uses the ﬁrst inequality in induction assumption (23); (iii) plug in the update of parameter: ✓ t [-+
-]+1 =
✓ t  ⌘ B > q t ; (iv) comes from the fact that non-zero singular values of Z and
BA > AB > are the same and extracts the largest singular value of Z ; (v) uses the triangle inequality; (vi) is due to the relationship between ` 2 and ` 1 norm; (vii) uses the ﬁrst inequality in induction assumption (23); (viii) uses the fact that s 2max ( BA > )
=
max ( Z ) and extracts s max ( BA > ) ; (ix) plug in the choice of stepsize; (x) uses the deﬁnition of  t  1 in (29) and induction assumption in (22); (xi) is because the choice of ⌧ and  ; (xii) is because p ↵ = q 1  12  2 > p 22 > 13 since  > 1 .
Thus, we obtain
[Equation] ↵
[Equation]
( ✓ 0 ) =
t .
Then by Lemma 4.1, with the correspondence that c = g t +1 , p = q t , r =  t , we can obtain
[Equation]
where ( i ) and ( iii ) come from the ‘Quantize’ step in Algorithm 1; ( ii ) and ( iv ) are from the two relations in Lemma 4.1 (since we have proved that k g t +1  q t k 1 6  t holds).
Now we have proved that (14) and (15) hold.
So for t > 0 , there is f
[Equation]
[Equation] f ( [Equation] 0 ) , where ↵ = 1
[Equation]
2  [Equation] Thus, if we want the objective function to compute an ✏ -optimal solution, the total number of iterations is log( f ( ✓ 0 ) / ✏ ) / log(1 / (1  12  4 )) .
Since in each iteration, each agent k transmits a length- H vector q tk , so we conclude that the total number of bits each node needs to communicate is log( f ( ✓ 0 ) / ✏ ) / log(1 / (1  12  4 )) bits .
Notice that log(1 / (1  12 
[Equation]
we can derive the simpliﬁed total number of bits as 2  4 · log( f ( ✓ 0 ) / ✏ ) .
E Proof for Proposition
Now we prove Proposition 1.
We ﬁrst state two lemmas that will be used.
Lemma E.1.
Rudelson and Vershynin [2010] Let X be a H ⇥ N matrix whose entries are independent standard normal random variables.
Then
[Equation]
Lemma E.2.
Suppose A > i 2 R D follows N ( µ, ⌃ ) .
If ⌃ is a diagonal matrix, then each element of
A i is independent.
We ﬁrst write down the SVD decomposition of A > :
A > = V ⌃ W, where V 2 R D ⇥ N , ⌃ 2 R N ⇥ N , W 2 R N ⇥ N .
Since A is full rank, denote ˆ  := s max ( A > ) s min ( A > ) > 0 .
Consider each entry in BA > = BV ⌃ W .
It is clear that each entry of BV follows normal distribution because ( BV ) hi is linear combination of variables that are standard normal.
Then we aim to show that each entry in BV follows standard normal distribution.
Notice that vec ( BV ) =
( B 1 V, B 2 V, · · · , B H V ) > 2 R HN , where B h is each row of B .
Denote U ij =
Cov ( B i V, B j V )
R N ⇥ N , we can obtain that [Equation] [Equation] N
[Equation]
N [Equation] U NN CCA
Notice that U ij =
Cov ( X > B > i , X > B > j ) = X > Cov ( B > i , B > j ) X .
Since B > i and B > j are independent, so V ij = 0 , i 6 = j .
Since X > X =
I N , we have U ij =
Cov ( B > i , B > i )
=
I N for i = 1 , 2 , · · · , N. Then we obtain Cov ( vec ( BV ))
=
I HN .
From Lemma E.2, each entry in BV are independent.
By Lemma E.1, we know the condition number of BV , which is  ( BV ) is independent of D , and we have
[Equation]
Now we consider Z = AB > BA > .
Notice the non-zero eigen values of Z are the same as BA > AB > , then we consider BA > AB > ,
[Equation]
Notice the non-zero eigen values of BV ⌃ 2 V > B > are the same as ⌃ 2 BV V > B > , we can derive
[Equation]
where ˆ  is condition number of A and the inequality is because the property of square and invertible matrix.
So we have with probability at least 1  2 e  t 2 / 2 ,
[Equation]
F.1 The CHOCO-GOSSIP Protocol
For completeness, we describe the CHOCO-GOSSIP protocol [Koloskova et al., 2019] for decentralized average consensus with compressed communication as follows.
Notice the number of gossip rounds
T g is dependent to the error ✏ as discussed in Fact 1.
Note that the protocol uses the compressor Q ( · ) for compressed communication, for example, this can be the randomized quantizer, see [Koloskova et al., 2019] for other examples.
The protocol is summarized below:
Input: step size  ; initial vectors g 01 , . .
, g 0 K ; gossip rounds T g ; compressor Q ( . ) ; mixing matrix
W ; neighbor sets N 1 , . . .
, N K .
Initialize: ˆ g 0 i = 0 , 8 i 2 [ K ] .
for t in 0 , . . .
, T g  1 do
Compress : Each agent i compress the difference, q ti = Q ( g ti  ˆ g ti )
Communicate : Each agent i receives q tj from neighbor j 2 N i and update ˆ g t +1 j
= ˆ g tj + q tj
Aggregation : Each agent i combines received vectors, i.e., g t +1 i
[Equation] ) end for
Output: g T g i at each agent i .
To derive the number of bits required in Theorem 4.3, we focus on using the random quantizer for
Q ( . ) , i.e., for x 2 R d , s 2 N + and ⌧ = (1 + min { d/s 2 , p d/s } ) , we have
[Equation]
where ⇠ s Uniform [0 , 1] d and sending Q ( x ) across the network requires d log( s + 1) + d + 64 bits of communication.
F.2 The Proof of Theorem 4.
Since F ( x ) =
Bx , ˜ F ( y ) =
B > y , the consensus error can be expressed as E ti :
= ¯ g ti n
P nj =1 B r f j ( ✓ tj ) .
We also denote the deviation of locally computed gradient as  G ti :=n P nj =1 B ( r f j ( ✓ tj )  r f j ( ✓ ti )) .
Note that k E ti k  ¯ ✏ / ( t + 1) .
We ﬁrst observe the updated it- erate,
[Equation]
t +1 i
[Equation]  [Equation] ⇤ Next, we observe that the objective function value evolves as,
[Equation]
↵
[Equation]
where the last inequality requires the observations
[Equation]
It remains to deal with the error terms separately.
Observe that
[Equation]
and n
[Equation]
Putting together into (31) and setting ⌘ =  min ( Z ) 4  2max ( Z ) yields
[Equation]
To bound k  G ti k
[Equation]
we observe that by the algorithm and the initial condition ✓ 0 i = ✓ 0 j , for any t  0 ,
[Equation]
where the last inequality applied
P 1 s
[Equation]
1 s
[Equation]
2 = ⇡ 2 / 6 , together with Fact 1 and the assumption ✏ t = ¯ ✏ / ( t + 1) .
Then, the error  G ti can be bounded by
[Equation]
With the notation  BA > A := 1 n
P nj =1 k BA > j A j k 2 , (32) gives us
[Equation]
where we have simpliﬁed notations by setting  =  max ( Z )  min ( Z ) .
By adjusting ¯ ✏ , we can achieve dimension-independent linear convergence.
Communication Complexity Let ⇠ t =
[-=
-]P
[Equation] P nj [Equation] k 2 .
We recall from Fact 1 that achieving a consensus error of ¯ ✏ / ( t + 1) at iteration t via CHOCO-GOSSIP requires
[Equation]
rounds of communication .
Applying b -bits random quantization to our projected H -dimensional vector, we have
[Equation]
Fix ✏ > 0 , to ﬁnd an ✏ -optimal solution, we set
[Equation]
> A ◆
[Equation]
and t  T ✏ := 8  2 log(2 f ( ✓ 0 i ) / ✏ ) .
Altogether, the number of communication rounds for achieving an ✏ optimal solution is bounded by:
[Equation]
where we have assumed that max t 2 [ T ✏ ] log( ⇠ t ) is dominated by max { log(1 / ¯ ✏ ) , log( T ✏ ) } .
Each communication round requires to send O (( b + 1) H ) bits per agent.
We can optimize the choice of b by
[Equation]
[-⇒ b ⋆ Hω − 
-]
O
[Equation] H [Equation]
H
[Equation]
with b ? = log( H
+ 1)
Under the properties of A, B described in Proposition 1, by Lemma E.1, simplifying k BA > j A j k shows that with probability 1  ⇣ ,
[Equation] ◆ [Equation]
As such, we have  BA >
A  ⇣ p H + p D + q 2 log 2 ⇣ ⌘ 6 .
Putting together gives the communication complexity upper bound of C op and the proof is completed.
D
H
Compression Ratio  1.
F.3 Additional Numerical Result
Fig.
2 shows the test MSE against communication cost for the cases of N = 2500 and N = 5000 .
The models are evaluated on the testing dataset of 23175 samples.
Our result indicates that increasing the dimension of kernel features decreases the testing MSE.
It suggests that overparameterization improves generalization on unseen data.
The communication budget limits the number of training samples, thus bottlenecks the generalization accuracy.
In the scenario where both computation and communication budgets are limited, Fig. 2 shows that the generalization power of Alg.
2 beneﬁts more from spending into communication budget (thus more training samples) than from overparametrizing, under the situation where H ⌧ D and more training samples are available with no cost.
Tableshows the compression ratio for the schemes used in our experiments.
Lastly, Fig. 3 (left) provides additional results for the large-scale overparameterized scenario, where
DN is as large as 19 which is close to the degree of over-parameterization in practice.
Comparing the two cases with D = 24 , 849 , D = 97 , 111 where the latter case uses 4 times more parameters, we observe that the bit complexity to reach an MSE of 10  5 only increases by 1 . 4 times.
Fig.
3 (right) examines empirically the effects of running L local = 2 local
G.1 Additional Notations
Let us further deﬁne some notations before we go into the details of the proof.
Denote the parameter in t -th iteration as ✓ t =
( W tl ) Ll =1 ; P l = vec ( O l ) as the vectorized output from layer l ; the pre- activation output from layer l as Q l .
Similarly, we can deﬁne P l,k , Q l,k for each agent.
Speciﬁcally, O
L = P L = Q L .
Further, let us deﬁne some notations related to the singular values of the weight matrices.
[Equation]
Speciﬁcally, ¯  l represents the quantity related to the largest singular value of weight matrices for different layers;  l represents smallest singular value of weight matrices for different layers.
Further, denote  O = s min [Equation] as the smallest singular value of the output from ﬁrst hidden layer at initialization multiplied by ˜ B .
Let us deﬁne some notations related to gradient.
Recall we have deﬁned u l = vec( r W l f ( ✓ )) .
Let us deﬁne the per iteration gradient as: g tk = ( Bu t 2 ,k , u t 3 ,k ; · · · ; u tL,k ) that collects all the (compressed) gradient of each layer.
Further, let us denote ˜ u t 2 ,k
[Equation]
B >
[Equation]
t 2 ,
[Equation] [Equation]
˜ q tk = ( B > z t 2 ,k ; z t 3 ,k · · · ; z tL,k ) , where z t 2 ,k quantizes Bu t 2 ,k and z tl,k quantizes u tl,k for l > 3 ; denote ˜ z t 2 , k
= B > z t 2 ,k .
Note, that in the above notation, both quantities with ˜ · has the original dimension
D .
Further, ˜ q tk is the actual vector that gets used at k -th agent, while ˜ g tk represents a “virtual" vector which has not been quantized.
Now let us deﬁne:
[Equation]
Further, we deﬁne  W tl , ˜  W tl , which are the unvectorized ˜ u
t 2 and ˜ q t 2 for l = 2 and unvectorized ˜ u tl and ˜ q tl for l > 3 :
[Equation]
That is,  W [Equation] [Equation]
Using the above notation, the ‘Update’ step in Algorithm 1, it can be expressed as
[Equation]
Denote ⌃ tl = diag [vec ( a 0 ( Q tl ))] 2 R Nn l ⇥ Nn l , which is a diagonal matrix, whose diagonal entries are the vectorized gradient of the activation function in each layer.
Recall that we have deﬁned
B = I n 2 ⌦ ˜ B in Section 4.2, where ˜ B is a Gaussian random matrix of size H ⇥ n 1 .
For convenience, let us assume s max ( B )
=
s max ( ˜ B ) > 1 .
G.2 Useful Lemma
Now we ﬁrst state a collection of results from [Nguyen and Mondelli, 2020] that will be used in our proof.
Notice that in the aforementioned work, the same pyramidal neural network structure and the l 2 loss function are used, so the loss function is the same as (6).
It follows that all the properties of the loss function can be reused.
Lemma G.1.
Suppose Assumption 4 and 5 hold, for each ✓ t , we have the following relations:
[Equation] [Equation]
Furthermore, given with ✓ a and ✓ b , if ¯ ⇤ l  max
[Equation]
W
[Equation]
, s max
[Equation]
bl  for some scalars ¯ ⇤ l .
Let ˜ R :
= L
[Equation]
, ¯ ⇤
[Equation]
.
Then, for l 2 [ L ] , 6 .
[Equation]
[Equation]
vec
[Equation]
Now let us discuss the properties above one by one.
The relations (38) and (39) show how the vectorized gradients of each layer, as well as the vectorized gradients of the output over each layer are computed, which are true regardless of the algorithm; see the ﬁrst and the second equalities in [Nguyen and Mondelli, 2020, Lemma 4.1], respectively; (40) is the lower bound of the norm of the vectorized gradient over W 2 , which is true as long as the network has the pyramidal structure (and regardless of the algorithm); see the third relation in [Nguyen and Mondelli, 2020, Lemma 4.1]; (41) is the upper bound of the norm of vectorized gradient over W l , which is true regardless of the algorithm; see the third relation in [Nguyen and Mondelli, 2020, Lemma 4.2]; (42) can be derived by summing over l in (41).
To be speciﬁc, we have
[Equation] [Equation]
The relation in (43) gives the upper bound of the gap between output layer with two sets of parameters ✓ a and ✓ b , and it is regardless of the algorithm; see [Nguyen and Mondelli, 2020, Eq.
(19)]; Similarly, (44) states the gap between the vectorized Jacobian matrix over W l ; see [Nguyen and Mondelli, 2020,
Eq.
(20)]; Finally, (45) computes the Lipschitz constants for the gradient, it is independent of the algorithm, and comes from plugging in (42), (43) and (44) into [Nguyen and Mondelli, 2020, Eq.
G.3 Initialization Strategy
As stated in Theorem 4.4, special initialization strategy is required.
Now let us describe the initialization in detail.
Recall the deﬁnition of ¯  l and  l in (35) and  O = s min [Equation]
Initialize ✓ [Equation] such that the following holds:
[Equation]
To satisfy the above relation, it requires that k O {+
+} L  y k cannot be large, which means the ✓ 0 should not be far from the optimal solution so that the initial loss is small.
Further, it requires the gap between ¯  l and  l is small.
From (45), we know that small ¯  l induces small Lipschitz constant for gradient, so the condition requires that the Lipschitz constant is not very large, which further implies that the optimization landscape is smooth.
On the other hand, from (40), we know the lower bound of the norm of u t 2 is related to  l .
So the initialization condition guarantees the lower bound of k u t 2 k is not too small, which avoids vanished gradient.
The initialization can be realized by using the procedure suggested [Nguyen and Mondelli, 2020,
Sec. 3.1].
The idea is that we can scale up W 01 to make  O not too small, and then randomly choose
W 02 with small entries.
Then for all l > 3 , set W 0 l as scaled identity matrices (top block as scaled identities) with large entries.
G.4 Formal Statement and Proof of Theorem 4.
First, let us state the formal Theorem 4.4:
Theorem 4.4 Consider using Alg. 1 to solve the problem (6) , with X being full row rank.
Suppose ✓ 0 is initialized as (46) .
Choose ˜ B such as rank ( ˜ BO 01 > )
=
N ; choose ˜ F ( · ) and F ( · ) as in (7) .
Set stepsize ⌘ and bits number b as following:
[Equation]
where  is deﬁned in (47) , ⌧ is deﬁned in (50) , ⇤ is deﬁned in (48) , Q 0 is deﬁned in (49) .
The following hold true:
[Equation]
To compute an ✏ -optimal solution (6) , each agent is required to transmit:
[Equation]
Proof.
To begin with, let us set the following constants:
[Equation] [Equation]
where
[Equation]
Deﬁne
[Equation]
Further
[Equation]
let us deﬁne
[Equation]
For convenience, let us assume Q 0 > 1 .
Let us explain the above constants: ↵ is the constants based on which the objective function contracts; ⌘ is the stepsize;  is related to the lower bound of k u t 2 k ; ⇤ is related to the upper bound of k ˜ g t k ; Q 0 is the Lipschitz constant for the full gradient; ⌧ is the parameter related to quantization; ⇤ l is related to the upper bound for k u tl k .
The majority of the proof consists of showing the following relations by induction:
[Equation]
Let us explain the meanings of the above relations.
Relations (i) and (ii) provide the upper and lower bounds of the singular values of weight matrices in each iteration; (iii) and (iv) give the upper and lower bound of the singular values of weight matrices after one step update without quantization; (v) shows the decrease of linear decrease of loss function in each iteration; (vii) and (viii) provide the error bound of gradient after quantization.
Compared to the induction proof in [Nguyen and Mondelli, 2020], the key challenges are: (1) Our analysis includes the quantization of gradient; (2) The update of parameter is not the simple gradient but a function of gradient.
Our proof consists of two steps:
Step 1: We show the above relations hold for t =
Step 2:
We show that
if the above relations hold for t , then they hold for t +
We show Step 1 ﬁrst.
For simplicity, we set g {+
+} = q 0 , it is easy to verify the relations hold if we choose q 0 in Algorithm 1 with large enough C. Step 1 will be shown in the following ﬁve substeps: (a) Show (i) and (ii) in (52); (b) Show (iii) and (iv) in (52); (c) Show (v) in (52); (d) Show (vi) in (52); (e) Show (vii) and (viii) in (52).
(Step 1.a)
We will show
[Equation]
We will use the fact that, according to the update rule in (37), we have q 0 = g 0 .
For l = 2 , we have:
[Equation]
where (i) is from the update rule in (37); (ii) extracts the largest singular value of B ; (iii) uses the upper bound of the gradient norm in (41) and the deﬁnition of ⇤ 2 in (51) ; (iv) uses the fact that strategy
Next, we show the case where l > 3 .
We have
[Equation]
where (i) is because the update of parameter in (37); (ii) uses the upper bound of the gradient norm in (41) and deﬁnition of ⇤ l in (51) ; (iii) uses the fact that P
{+[Equation] =
 p ↵+} > 1 ; (iv) plugs in the choice of ↵ and 1 + ↵ 12 < 2 ; (v) comes from the initialization strategy in (46).
Applying Weyl’ inequality to the matrices W 0 l and ( W 1 l  W 0 l ) , we have
[Equation]
This concludes the proof of this substep.
(Step 1.b)
We will show
[Equation]
Since we have ˜ q 0 = ˜ g 0 , it is easy to derive that
[Equation]
Thus we can conclude ⇢
[Equation]
This concludes the proof for this substep.
(Step 1.c)
We will show
[Equation]
From (Step 1.a) and (Step 1.b) we know
[Equation]
Using the above relation, we can upper bound the differences of the gradients by using (45).
More speciﬁcally, using the the deﬁnition of Q 0 in (49), we have
[Equation]
where we repeat the deﬁnition of Q 0 below (where R is deﬁned in (49)):
[Equation]
Further, it is easy to verify that for any ˆ ✓ 0 between ✓ 0 and ✓ 1 , we still have s max ( W l (ˆ ✓ 0 ))
6 32 ¯  l .
So we can apply the same argument leading to (53), and obtain:
[Equation]
[Equation]
As long as for any ˆ ✓ 0 (54) holds, we can apply Decent Lemma.
More speciﬁcally, we have
[Equation]
[Equation]
where (i) uses Decent Lemma, (ii) uses the fact that u 0 is the stacked version of { u 0 i } i  2 , and ˜ g 0 are stacked versions of { u 0 i } i  3 and ˜ u 02 .
Next, we will bound the inner product on the right hand side of the above relation:
[Equation]
where (i) uses the expression of u 02 by (38) and ˜ u
02 = B > Bu 02 ; (ii) and (iii) are from the property of
Kronecker product; (iv) is from Assumption 5 and fact that s max ( W 0 l ) > 12  l .
Next, we upper bound the term k ˜ g [-
 ∥-]0 k 2 in (55).
We have
[Equation]
where (i) uses the deﬁnition of ˜ g 0 ; (ii) extracts the largest singular value of B ; (iii) is because we assume s max ( ˜ B ) > 1 ; (iv) uses the upper bound of gradient in (42), (v) comes from the fact that s max ( W 0 l ) > 12 ¯  l ; (iii) is because the deﬁnition of ⇤ in (48).
Recall that we have deﬁned
[Equation]
If we choose ⌘ =  Q 0 ⇤ 2 , then plug (56) and (57) into (55), we obtain f
[Equation]
58) (Step 1.d)
We will show
[Equation]
Since ˜ g 0 = ˜ q 0 , we have f ( ✓ 1 ) =
f ( ✓ 0  ⌘ ˜ g 0 )
( i ) 6 (1  ⌘ ) f
( ✓ 0 ) <
↵ f ( ✓ 0 ) , where (i) is because (58).
This completes the proof of this step.
(Step 1.e)
We will show that:
[Equation]
Notice that for l = 2 , we have
[Equation]
where (i) uses triangle inequality; (ii) extracts the largest singular value of B and uses g 0 = q 0 ; (iii) is because k u 12  u 02 k 6 k u 1  u 0 k ; (iv) is from (53), which uses (45) and deﬁnition of Q 0 in (49); (v) uses the update rule in (37); (vi) is because g 0 = q 0 ; (vii) uses the deﬁnition of ⇤ in (48); (viii) uses the deﬁnition of  in (47) and the fact ↵ = 1
[Equation]
Then by Lemma 4.1, with the correspondence that c
=
Bu 12 , p = z 02 , r =  0 , we can obtain
[Equation]
where ( i ) comes from the ‘Quantize’ step in Algorithm 1; ( ii ) is from the ﬁrst relation in Lemma 4.1.
For l > 3 , we have
[Equation]
where (i) uses triangle inequality; (ii) is because g 0 = q 0 and uses the fact k u 0 l  z 0 l k 6 k g 0  q 0 k ; (iii) is from (53); (iv) uses the update rule in (37); (v) is because ˜ g
= ˜ q 0 ; (vi) uses the upper bound of k ˜ g 0 k in (42) and the deﬁnition of ⇤ in (48); (vii) comes from the choice of  in (47) and the fact ↵ = 1
[Equation]
Then by Lemma 4.1, with the correspondence that c = u 1 l , p = z 0 l , r =  0 , we can obtain
[Equation]
where ( i ) comes from the ‘Quantize’ step in Algorithm 1; ( ii ) is from the ﬁrst relation in Lemma 4.1.
Now we have showed all the induction assumptions hold for t = 1 .
Next, we show Step 2.
Given these inequalities in (52) hold for iteration t , we aim to show that they hold for t
+ 1 .
We prove Step 2 in ﬁve substeps similarly as in the proof of Step 1.
(Step 2.a)
We will show that
[Equation]
For l = 2 , we have:
[Equation]
[Equation]
where (i) uses triangle inequality; (ii) uses the update rule in (37); (iii) uses triangle inequality; (iv) extracts the largest singular value of B > B , the upper bound of k u t 2 k in (41), deﬁnition of ⇤in (51) and the relationship between l 2 and l 1 norm; (v) uses the fact that k B > ( Bu r 2  z r 2 ) {+k 
+}
Hn 2 k ˜ B
k 1 k Bu r 2  z r 2 k 1 ; (vi) uses the induction assumption
k Bu r 2  z r 2 k 1 6 ⌧
r  1 and deﬁnition of  r  1 ; (vii) reorganizes the terms; (viii) is because 0 < ↵ < 1 ; (ix) uses 1 + ↵ 12 < 2 and plugs in the choice of ↵ ; (x) is because the initialization strategy in (46) and choice of ⌧ in (50).
So Applying Weyl’ inequality to the matrices W t 2 and ( W t +  W t 2 ) and (59), we have
[Equation]
For l > 3 , we have the following relation:
[Equation]
[Equation] where (i) uses triangle inequality; (ii) uses the update rule in (37); (iii) uses triangle inequality; (iv) uses the upper bound of k u tl k in (41), the deﬁnition of ⇤ l in (51) and the relationship between l 2 and l 1 norm; (v) uses the induction assumption k u rl  z rl k 6 ⌧ r  1 ; (vi) uses reorganizes the terms; (vii) is because 0 < ↵ < 1 ; (viii) uses the fact 1 + ↵ 12 < 2 and plugs in the choice of ↵ ; (ix) is from the initialization strategy in (46) and the choice of ⌧ in (50).
Applying Weyl’ inequality to the matrices W t +1 l and ( W t +1 l  [Equation] ) , we have [Equation]
t
[Equation]
(Step 2.b)
We will show that
[Equation]
For l = 2 , we have
[Equation]
where (i) uses Weyl’s inequality and the relationship between the Frobenius norm and l 2 norm of matrix; (ii) uses the deﬁnition in (36); (iii) uses the relationship between l 2 and l 1 norm and the deﬁnition of ˜
u t 2 and ˜
z t 2 ; (iv) uses the fact that k B > ( Bu r 2  z r 2 )
{+k 1 
+}
Hn 2 k ˜ B k 1 k Bu r 2  z r 2 k 1 and the deﬁnition of  r  1 ; (v) is from the induction assumption
k Bu r 2  z r 2 k 1 6 ⌧ r  1 ; (vi) is from the fact that ↵ 6 1 ; (v) is from the (59) and choice of ⌧ and  in (50) and (47).
For l > 3 , by Weyl’s inequality, we have
[Equation]
where (i) uses the Weyl’s inequality on W t +1 l and ⌘ ( ˜  W tl   W tl ) ; (ii) uses the update rule in (37); (iii) uses the relationship between l 2 and l 1 norm and the induction assumption; (iv) is because ↵ < 1 ; (v) uses (60) and choice of ⌧ and  in (50) and (47).
Similarly, s min
[Equation]
W t +1 l [Equation] where (i) uses Weyl’s inequality on W t +1 l and ⌘ ( ˜  W t l   W t l ) ; (ii) plugs in the update rule in (37); (iii) uses the relationship between l 2 and l 1 norm and the induction assumption; (iv) is because ↵ < 1 ; (v) comes from (60) and choice of ⌧ and  in (50) and (47).
(Step 2.c)
We will show that:
[Equation]
From (1) and (2) we know
[Equation]
Using the above relation, we can upper bound the differences of the gradients by using (45).
More speciﬁcally, using the the deﬁnition of Q 0 in (49), we have
[Equation]
where Q 0 is deﬁned in (49).
Further, it is easy to verify that for any ˆ ✓ t between ✓ t and ✓ t +1 , we still have s max ( W l (ˆ ✓ t ))
6 32 ¯  l .
So we can apply the same argument leading to (53), and obtain:
[Equation]
So we have
[Equation]
where (i) uses Decent Lemma; (ii) expands h u t ,
˜ g t i by utilizing the stacked structure.
Similar to (56), with the induction assumption s min ( W tl ) > 12  l , l > 3 , we have the following relation:
[Equation]
Similar to (57), with the induction assumption s max ( W tl ) 6 32 ¯  l , l > 2 , we have
[Equation]
If we choose ⌘ Q 0 ⇤
[Equation]
we obtain
[Equation]
(Step 2.d)
We will show that: f
[Equation]
We have the following relation:
[Equation]
where (i) expands and reorganizes the loss function; (ii) uses Young’s inequality with constant  ; (iii) uses the induction assumption f ( ✓ t +1 ) [Table] 1  ⌘ ) f ( ✓ t )
(iv uses the mean value Theorem where ˆ ✓ t
Now we discuss ˆ ✓ t and derive a bound for k vec ( r G (ˆ ✓ t )) k .
By the Mean Value Theorem we have
[Equation]
for some  2 [0 , 1] .
By (61) and (62), we know that
[Equation]
So it is easy to conclude that for l > 3 ,
[Equation]
where (i) uses Wely’s inequality on W t +1 l and ⌘  ˜  W tl   W tl  ; (ii) is because  2 [0 , 1] ; (iii) uses (62).
We can derive the similar result for l = 2 .
By (42) without k O tL  y k , it is clear that
[Equation]
So we have
[Equation]
(i) uses (42) with the fact that s max ( W l (ˆ ✓ t ))
6 32 ¯  l , and expands k ˜ q t  ˜ g t k 2 ; (ii) uses the relationship between l 2 and l 1 norm; (iii) uses the fact that k B > ( Bu r 2  z r 2 )
{+k 1 
+}
Hn 2 k ˜ B k 1 k Bu r 2  z r 2 k 1 ; (iv) uses the induction assumption
[Equation] f ( ✓ 0 ) and the deﬁnition of  t  1 .
Let  = 12 ⌘ 1  ⌘ , we have
[Equation]
where (i) plugs in the choice of  ; (ii) is because the choice of ⌧ and  in (50) and (47).
(Step 2.e)
We will show that:
[Equation]
For l = 2 , we have
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table] [Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
where (i) uses triangle inequality; (ii) extracts the largest singular value of B , (iii) uses the fact k u t +1  u t k > k u t +  u t 2 k ; (iv) uses the deﬁnition of u t ; (v) upper bounds the differences of the gradients by using (45) (vi) uses the update rule in (37); (vii) uses triangle inequality; (viii) uses the upper bound of gradient norm in (42), and it extracts the largest singular value of B and deﬁnition of ⇤ in (48); (ix) uses the stacked structure of g t and q t to expand k g t  q t k ; (x) uses the relationship between l 2 and l 1 norm; (xi) reorganizes the terms and uses induction assumption; (xii) uses the deﬁnition of  t  1 ; (xiii) is from the choice of ⌧ and  in (50) and (47).
Then by Lemma 4.1, with the correspondence that c =
Bu t +12 , p =
z t 2 , r =  t , we can obtain
[Equation]
where ( i ) comes from the ‘Quantize’ step in Algorithm 1; ( ii ) is from the ﬁrst relation in Lemma 4.1.
For l > 3 , we have
[Equation]
[Equation]
where (i) uses triangle inequality; (ii) uses the fact k u t +1  u t k > k u t +  u t 2 k ; (iii) uses the deﬁnition of u t ;(iv) uses the update rule in (37); (v) uses triangle inequality; (vi) uses the upper bound of gradient norm in (42), extracts the largest singular value of B and uses the deﬁnition of ⇤ in (48); (vii) uses the stacked structure of g t and q t to expand k g t  q t k ; (viii) uses the relationship between l 2 and l 1 norm; (ix) uses the induction assumption; (x) uses the deﬁnition of  t  1 ; (xi) is because the choice of ⌧ and  in (50) and (47).
Then by Lemma 4.1, with the correspondence that c =
Bu t +1 2 , p =
z t 2 , r =  t , we can obtain
[Equation]
6 ⌧ t , where ( i ) comes from the ‘Quantize’ step in Algorithm 1; ( ii ) is from the ﬁrst relation in Lemma 4.1.
Now we have proved that (52) holds.
So for t > 0 , there is [Equation]
where
[Equation]
Thus, if we want the objective function to compute an ✏ -optimal solution, the total number of iterations is log( f ( ✓ 0 ) / ✏ ) / log(1 / (1   2 4 Q 0 ⇤ 2 )) .
Since in each iteration, each agent k transmits a length- H vector q tk , so we conclude that the total number of bits each node needs to communicate is log( f ( ✓ 0 ) / ✏ ) / log(1 / (1
[Equation]
2 )) bits .
Notice that log(1 / (1
[Equation]
[Equation]
4 Q
[Equation]
⇤ 2  2 , so we can derive the simpliﬁed total number of bits as b ·
[Equation]
· log(
[Equation]