Distributed Optimization for Overparameterized Problems: Achieving Optimal Dimension Independent Communication Complexity
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Decentralized optimization are playing an important role in applications suchas training large machine learning models, among others.
Despite its superiorpractical performance, there has been some lack of fundamental understandingabout its theoretical properties.
In this work, we address the following open researchquestion: To train an overparameterized model over a set of distributed nodes, whatis the minimum communication overhead (in terms of the bits got exchanged) thatthe system needs to sustain, while still achieving (near) zero training loss?
Weshow that for a class of overparameterized models where the number of parameters
D is much larger than the total data samples N , the best possible communicationcomplexity is ⌦ ( N ) , which is independent of the problem dimension D .
Further,for a few speciﬁc overparameterized models (i.e., the linear regression, and certainmulti-layer neural network with one wide layer), we develop a set of algorithmswhich uses certain linear compression followed by adaptive quantization, and showthat they achieve dimension independent, near-optimal communication complexity.
To our knowledge, this is the ﬁrst time that dimension independent communicationcomplexity has been shown for distributed optimization.
Introduction
The research of decentralized/distributed optimization has recently gained tremendous momentum,partly due to the fact that a well-designed decentralized/distributed algorithm is capble of traininglarge-scale machine learning models, by utilizing massively distributed computation and data resources.
In a typical decentralized optimization setting, multiple local agents hold small to moderatelysized datasets and they collaborate by iteratively solving their local problems while sharing someinformation with other agents.
The majority of the state-of-the-art distributed/decentralized optimization algorithms are deeply rooted in classical consensus-based approaches [Tsitsiklis, 1984, Tsitsiklisand Luo, 1987] where the agents repetitively share the local optimization variables, or the gradientsof local functions, so to reach an optimal consensual solution.
It is important to note that in those classical settings, the agents typically solve small-scale problemswhere the problem dimension (i.e., the dimension of the optimization variable, and the gradients)is relatively small (e.g., in the order of hundreds or thousands).
Therefore, it is reasonable thatthe agents should iteratively share their local parameters/gradients to improve the quality of themodels.
However, many modern learning models are in the overparameterized regime with veryhigh-dimensional parameters [He et al., 2016, Vaswani et al., 2017], such as deep and/or wideneural networks.
This practice poses some signiﬁcant challenges to the classical consensus-basedapproaches, because by directly sharing the model parameters and/or gradients, the communicationburden increases as problem size increases.
A line of recent works have been developed, which adopt strategies such as gradient/model compression, lazy communication, etc, and they are able to signiﬁcantly reduce the overall communicationcomplexities [Xu et al., 2020, Sun et al., 2020, Li et al., 2019, Karimireddy et al., 2019]. A fewworks focus on designing compressors [Safaryan et al., 2020, Horváth et al., 2019, Beznosikov et al. ,2020, Koneˇcn`y et al., 2016, Safaryan et al., 2021].
Although the majority of these works are ableto reduce per round communication burden, it is by no means clear if, to achieve certain solutionquality, the total amount of communication can be saved.
In fact, it has been shown in [Arjevaniand Shamir, 2015], that the total communication achievable is dependent on the problem dimension,which could be huge.
More importantly, there is a general lack of fundamental understanding aboutthe performance limits for decentralized and distributed optimization, when the models are ‘large’.
Without such characterization, it is by no means clear if the existing algorithms are optimal inreducing the communication complexities.
In this work, we focus on distributed optimization for ‘large’ machine learning models (i.e., overparameterized problems, to be deﬁned shortly), and we ask the following question:
(Q)
What is the best possible communication complexity (in terms of number of bits needed to obtain certain solution accuracy), achievable by any distributed algorithms for a class of overparameterized problems?
Which class of algorithms achieves such complexity bounds?
Related Works.
A number of recent works are devoted to reducing the communication cost indistributed/decentralized optimization by combining the optimization algorithms with compression ,e.g., sparsiﬁcation, quantization, etc.
For the distributed setting with the server-worker architecture,relevant works include [Alistarh et al., 2017, Wen et al., 2017, Wang et al., 2018, Bernstein et al.
,2018, Haddadpour et al., 2021, Mishchenko et al., 2019, Gorbunov et al., 2021]; see [Richtárik et al. ,2021] for a recent survey.
For the decentralized setting without a central server, recent works such as[Koloskova et al., 2019, Reisizadeh et al., 2019, Tang et al., 2018, Vogels et al., 2020, Magnússonet al., 2020] proposed algorithms that utilize error compensation, low-rank compression, etc.
Whilethey have demonstrated appealing performance in practice, the theoretical guarantees do not scale foroverparameterized problems.
For instance, [Koloskova et al., 2019] showed an O (1 /T + 1 / ( ! T ) 2 )iteration complexity upper bound for their algorithm.
The quantity 1 / ! is the compressor parameterwhich is at least ⌦ ( p D ) , where D is the problem dimension.
Lower communication complexity bounds for distributed optimization problems have been studiedsince the 1980s.
There are two major categories of lower bound results in literature depending on howwe count the “communication” required to reach a solution of given accuracy, i.e., 1) in terms of thenumber of communication rounds, or 2) in terms of the number of bits got communicated.
The ﬁrstcategory is not directly related to the current work; we refer to readers to Appendix A for discussions.
In second category, the ﬁrst lower bound is given in [Tsitsiklis and Luo, 1987] where the authorsconsider a distributed setting with two nodes and provide a ⌦ ( D log ( D/ ✏ )) bound for the class ofquadratic objectives, where ✏ is the solution accuracy, and D is the variable dimension.
In [Zhanget al., 2013] lower communication bounds are derived for certain statistical estimation tasks, suchas linear and probit regression.
Moreover, in [Vempala et al., 2020] the communication complexityof several distributed optimization tasks is studied, including ` p regression, linear, semideﬁnite andconvex optimization.
In [Korhonen and Alistarh, 2021], an ⌦ ( KD log ( D/ ✏ )) bound is derived forquadratic problems on a multi-node setting, for both deterministic and randomized protocols, where
K is the number of clients.
Recently, overparameterized problem has drawn great research interests.
It has been showed that remarkable performance can be achieved with overparameterized networks [Jacot et al., 2018, Du et al.
,2019, Allen-Zhu et al., 2019].
Some recent works study the overparameterization of decentralizedproblems, which guarantee the global convergence [Huang et al., 2021, Deng et al., 2022].
However,the high communication cost caused by the large model is a key problem in decentralized algorithm.
For example, the ReLU fully connected networks designed in the aforementioned two works are ofwidths ⌦ ( N 4 ) and ⌦ ( N 18 ) respectively, where N is the number of data samples available.
Contributions.
This work addresses both questions in (Q) .
The contributions are highlighted below:
(i)
Regarding to the best possible communication complexity (i e., the communication complexity
lower bound), we ﬁrst show that for a class of Polyak-Lojasiewicz (PL) problems, the communicationlower bound is ⌦ ( KD log( D/ ✏ )) , which is similar to the known results for strongly convex problems.
Further, we show that for a class of overparameterized models where the loss function is a composition of a quadratic problem and some nonlinear functions, and that the number of parameters D ismuch larger than the total data samples N , the communication complexity lower bound becomes⌦ ( KN log( K/ ✏ )) , which is independent of the problem dimension D .
(ii) We develop a distributed algorithm which can be used to optimize both classes of problems, in the setting where multiple clients are coordinated by a server.
We show that the lower bound forthe PL problem can be matched; while by using certain linear random compression/decompressionscheme, followed by adaptive quantization, the lower bound for the overparameterized problem can be matched (up to a log( D ) factor) for two of its important special cases, i e., linear regression andcertain multi-layer neural network with one wide layer
Further we show that the proposed algorithmcan be extended to a fully decentralized setting without the
coordination of a server. To our knowledge, these are the ﬁrst set of results that attempt to comprehensively address the issueof optimal communication complexities for overparameterized problems.
We expect that resultsprovided in this work will open doors for understanding more complex communication-efﬁcientdistributed algorithms, as well as larger classes of overparameterized problems.
The System Model
[Figure]
Consider a system having K distributed agents, and a total of N samples { ( X i , y i ) } Ni =1 .
Supposeeach agent has N k data samples, with P
[Equation]
k = N ; let set X k (resp.
Y k ) collect agent k ’s data 107 samples (resp. labels).
They cooperatively solve an optimization problem of the following form:
[Equation]
where f k : ⇥ !
R is the local loss function for agent k ; ✓ is the optimization variable, whose domainis ⇥ ✓ R D .
For notational simplicity, we will ignore the reference for the data samples X , Y and
X k , Y k whenever possible, i.e., we will write f ( ✓ ) and f k ( ✓ ) as the global and local loss functions.
Throughout the paper, we will consider a few special cases of (1), where the local function f k ’s areoverparameterized .
The ﬁrst of such problem class is referred to as the distributed PL problems .
Tointroduce this class, let us provide a few assumptions.
Assumption 1.
The local objective functions satisfy:
[Equation]
where ✓ ⇤ ( k ) is a global minimum of f k ( · ) ; µ k ’s some positive constants.
Assumption 2.
There exists positive constants L k ’s and L  1 such that:
[Equation]
Note that (2) is the so-called PL assumption, which was introduced in [Polyak, 1963].
It is weakerthan strong convexity, while it can still be used to ensure the linear convergence of gradient descent.
It is also less restrictive than several other related conditions under which linear global convergencecan be derived, such as the restricted secant inequality [Karimi et al., 2016].
The second inequality in(2) has also been used in the literature; see, e.g., [Oymak and Soltanolkotabi, 2019], in which sucha bound for overparameterized network has been showed.
Assumption 2 is the standard Lipschitzgradient condition.
Deﬁnition 1 (Distributed PL problems) .
Consider problems of the form (1) .
Deﬁne:
[Equation]
The second class of problems is referred to as the distributed overparameterized problem, wherethe loss function is a composition of least square loss and a non-linear function.
This form of lossfunction often appears in analyzing neural networks, see, e.g., [Oymak and Soltanolkotabi, 2019].
Deﬁnition 2 (Distributed overparameterized problems) .
Consider problems of the form (1) .
Deﬁnefunctions G k ( · ) : ⇥ !
R N k and constants
b k 2 R N k , for all k .
We call G k ( · ) overparameterized iffor every b k 2 R N k , there exists ✓ 2 ⇥ such that G k ( ✓ ) =
b k .
Deﬁne the following class of problems:
[Equation]
Communication
In this section we derive the communication complexity lower bounds for the two classes of problems
C op and C pl .
All the proofs for results presented in this section can be found in Appendix B.
We start with a description of the coordinator model (also see [Braverman et al., 2013, Korhonenand Alistarh, 2021]), which will be used later as the basis of our derivation.
Consider a set of Kdistinct nodes/agents, and a single coordinator node.
Each node and the coordinator have a two-waycommunication channel, over which binary messages are exchanged.
Consider a class of problems Cof the form (1), that is f ( ✓ ) =
P
Kk =1 f k ( ✓ ) , where each node k has access only to objective f k ( ✓ ) .
Moreover, let ⇧ be a set of protocols that are executed over the coordinator model.
Then, given aproblem class C , a protocol ⇡ 2 ⇧ ( ✏ ) takes as input a problem c 2 C , performs a number of messageexchanges (over the coordinator model), and outputs an ✏ -approximate solution ˜ ✓ 2 ⇥ of problem(1).
That is, the output ˜ ✓ satisﬁes (where ✓ ⇤ is a point where the exact minimum is attained):
[Equation]
Next, let us look at ⇡ in more detail.
We assume that for ⇡ 2 ⇧ ( ✏ ) , it performs T ( ✏ ) rounds ofcommunication and computation, where during each round the following steps are carried out:
• At round t = 0 , each agent/node k is given as input the local objective f k (e
g., the data samplesX k , Y k assigned to each node);
• At round 1  t < T ( ✏ ) ,
– The coordinator sends a message m ( t ) c !
k 2 { 0 , 1 } to some node k .
– Node k sends message m ( t ) k !
c 2 { 0 , 1 } to the coordinator.
• At the ﬁnal round t = T ( ✏ ) the coordinator outputs an ✏ -approximate solution.
Next, we deﬁne the communication complexity of the distributed protocol.
Let us denote the totalnumber of bits exchanged under protocol ⇡ , for problem c 2 C , as M ( c, ⇡ , ✏ ) ; Use the followingdeﬁnition, which has been used extensively in related works such as [Tsitsiklis and Luo, 1987].
Deﬁnition 3 (Communication complexity) .
The communication complexity of protocol ⇡ overproblem class C is deﬁned as the maximum number of bits exchanged across the problems in C , i.e.,
M ( C , ⇡ , ✏ ) :=
max c 2 C M ( c, ⇡ , ✏ ) .
Also, the communication complexity of problem class C is theminimum communication complexity across all protocols that solve C , i.e.,
[Equation]
Next, we derive communication complexity lower bounds for certain problem classes C .
Ourconstruction strategies follow those from [Korhonen and Alistarh, 2021], that is, we will reducea problem with known communication complexity, to hard problem instances within classes ofour interest.
The main difference from the aforementioned work (which only deals with quadraticproblems) is that, we need to construct hard PL and overparameterized problems instances.
Below we derive communication complexity lower bounds for the class of distributed problemswhere the local loss functions belong to the class of PL functions deﬁned in Def. 1.
Theorem 3.1.
Consider the problem class C pl with given parameters K, D, ✏ , for which it holds that
D ✏ = ⌦ (1) .
Then we have M ( C pl , ✏ ) =
⌦ ( KD log
D ✏  ) .
Note that the condition D/ ✏ = ⌦ (1) is reasonable since we will mainly focus on the regime where Dis large.
Next we consider C op .
We have the following communication complexity lower bound.
Theorem 3.2.
Consider the problem class C op , with given parameters K, N, D, ✏ , for which it holdsthat N ✏ = ⌦ (1) .
Then we have that M ( C op , ✏
[Equation]
[Equation]
[Equation] Clearly, the lower bound for class C op does not explicitly depend on the problem dimension D , whilethat for the C pl does.
This appears to be reasonable, partly because in the class of C pl the numberof data samples N has not been explicitly modeled.
Nevertheless, at this point it is not clear if sucha dimension-independent lower bound is tight or not, because we are yet to ﬁnd an algorithm tomatch such a lower bound.
To our knowledge, these are the ﬁrst set of results that characterize thecommunication complexity for the classes C pl and C op , and they will serve to guide our subsequent 177 algorithm development and analysis.
Algorithms Achieving the Optimal Communication Complexities
In this section, we design algorithms and analyze the achievable communication complexities, forthe PL and the overparameterized problems in Def. 1 and 2.
We will begin with the simple settingwhere all the agents are connected to a coordinator, and then extend to the case where the agents aredecentralized.
Throughout this section, for simplicity of exposition, we will assume that ⇥ ⌘ R D .
Let us begin with a generic algorithm to deal with the server-agent setting.
The algorithm is anadaptation of the classical quantize-then-communicate strategy [Tsitsiklis and Luo, 1987, Magnússonet al., 2020].
At each iteration, each agent ﬁrst computes its gradient, followed by certain adaptivequantization scheme, before getting transmitted to the collaborating agents.
Speciﬁcally, at t -th iteration, each agent uses a function F : R D !
R H to compress the localgradient; Then the output is quantized as q tk 2 R H (use the method in Def. 4).
Each agent thenshares q tk to the server, and the aggregated q t is broadcast to all agents.
Each agent then uses afunction ˜ F : R H !
R D to process the received vector, and updates the local parameters.
One keydifference compared with [Tsitsiklis and Luo, 1987, Magnússon et al., 2020] is that, the algorithmuses a pair of compression and decompression functions F ( · ) and ˜ F ( · ) to process the gradientstransmitted/received.
We will see shortly that such a pair plays an important role in matching thelower bound.
For the detailed steps, see Algorithm 1 below.
1: Initialize: Fix F ( · ) and ˜ F ( · ) , ⌘ > 0 ,  t 2 R + , b 2 Z + ; set ✓ 0 k = ✓ 0 , choose q 0 k which can be represented by b 0 bits for each entry, such that k q 0 k  F ( r f k ( ✓ 0 ))
k 1 6 p f k ( ✓ 0 ) C
· D 3 p K , where C is some constant independent of D .
[Equation]
[Equation]
In Algorithm 1, we note that as long as the initial solution ✓ 0 is the same among all the agents, then✓ t +1 k computed in the ‘ Update ’ step will always be the same across the agents.
This is the reason thatwe can directly set ✓ t +1 k as ✓ t +1 .
Subsequently we will extend this algorithm to a fully decentralizedsetting, where precise consensus is no longer required.
We choose the quantization method for theﬁrst iteration deﬁned in Appendix C.
In the subsequent iterations, we use the following adaptivequantization scheme; also see [Magnússon et al., 2020].
Deﬁnition 4.
Fix r 2 R + and let b 2 N be an positive integer.
Let quant : R H ⇥ R H ⇥ R + ⇥ N !
R H be the quantization function deﬁned component-wise as follows,
where  ( r, b ) :
= r/  2 b  , [ quant ( c, p, r, b )] j => < > : p j  r if
[Equation]
Essentially, the quantization scheme searches for a closest point on the grid (which has a width 2 rand is centered at p =
( p 1 , . . .
, p H ) .
The distance between the neighbor points on the grid is  ( r, b ) .
Lemma 4.1.
Fix p 2 R H , then for all c 2 R H , Q 2 R D ⇥ H , such that k c  p k 1  r we have
[Equation]
4.1 Communication Complexity Upper Bounds for C pl
We will begin with analyzing the class of problem C pl , as deﬁned in Def. 1.
Before we proceed, letus make an additional assumption.
Assumption 3.
Assume that each f k ( · ) is non-negative in its domain; Further, assume that theoptimal objective value for problem (1) is zero, that is, there exists ✓ ⇤ 2 R D such that f ( ✓ ⇤ )
= 0 .
The above ‘zero loss’ assumption has been used by a number of recent works to characterize problemswith a large number of parameters, e.g., in overparameterized neural network [Liu et al., 2022, Nguyenand Mondelli, 2020]; in optimization literature, this assumption is also used to improve the analysisof gradient-based algorithms; see, e.g. [Razaviyayn et al., 2019]. Lemma 4.2.
Suppose Assumption 2 – 3 are satisﬁed, then we have the following inequality: [Equation]
The above lemma shows that together with the PL condition, Assumptions 2, 3 imply:
[Equation]
For convenience, throughout our analysis we set µ < 1 , L > 1 .
We will refer to the class of problems satisfying Assumption 1 to 3 as e
C pl .
We note that e
C pl is a strict 217 subset of the class
C pl .
Next, let us present our result for C pl .
The proof can be found in Appendix C.
Theorem 4.1.
Suppose Assumptions 2 – 3 are satisﬁed.
Consider using Alg. 1 to solve C pl ; set⌘
[Equation]
[Equation]
[Equation]
=
[Equation]
p
[Equation]
[Equation]
, b = max
[Equation]
[Equation]
x, and C := max
[Equation]
.
Then the following holds true:
[Equation]
Further, to compute an ✏ -optimal solution satisfying (3) , each agent is required to transmit:
[Equation]
The quantization strategy at the ﬁrst iteration (in Appendix C) requires b 0 to be O  log( D )  , so wecan conclude b
= O  log( D )  .
Thus Theorem 4.1 gives the total communication complexity for eachagent as O
D log
[Equation]
D )
· log
[Equation]
1 ✏
[Equation]
, which matches the lower bound in Theorem 3.1, up to a gap log( D ) .
4.2 Communication Complexity Upper Bounds for C op
Next, we develop algorithms for the distributed overparameterized problem deﬁned in Def. 2.
Notethat functions in the class C op can be non-convex, therefore one may not be able to ﬁnd an algorithmthat can solve the entire class of problems in polynomial time.
Our strategy is to identify special casesof C op for which the analysis is possible.
Again, although these problems are restrictions of C op , wedo not expect the lower bound in Theorem 3.2 to be improved in either the dependency on N or on ✏ .
The Case for Quadratic Problems.
Towards this end, let us consider an overparameterized quadratic 231 problem.
To proceed, assume that each agent k has data tuple ( A k , b k ) with N k samples, where A k 2 232 R N k ⇥
D is the feature matrix, and b k 2 R N k ⇥ 1 is the label.
Denote A :=
[ A 1 ; · · · ; A K ] 2 R N ⇥ D ,where N =
P
Kk =1 N k .
Suppose N < D .
Then the generic problem (1) reduces to:
[Equation]
Assuming that each A k has full row rank, then the above problem is a special case of C op .
Next, we specialize Alg. 1.
Recall that the lower bound derived in Theorem 3.2 is independent of D ,so we cannot directly apply the algorithm we used for the class C pl , in which each agent transmits adimension D vector at every iteration.
To proceed, let us consider the following linear compressionscheme, in which we use a matrix B of size H ⇥ D to compress (resp.
decompress)
the gradientsthat get transmitted (resp. received), where H is a ﬁxed positive constant satisfying N 6 H ⌧ D .
Inparticular, let us specialize the two processing operators in Alg. 1 as: F ( x ) =
Bx, ˜ F ( y )
=
B > y .
Further, deﬁne  max ( · ) and  min ( · ) (resp.
s max ( · ) and s min ( · ) ) as the largest and smallest eigenvalues(resp.
singular values) for a matrix.
Then we have the following result; see Appendix D for proof.
Theorem 4.2.
Consider using Alg. 1 to solve the quadratic problem (4) , with A being full row rank.
Choose B such that rank ( BA > ) =
N ; set ⌘ =  min Z )  2max ( Z ) ; set b and C according to (30) , and choose˜ F ( · ) and F ( · ) as F ( x ) =
Bx, ˜ F ( y )
=
B > y .
Then the following holds true:
[Equation]
where  is the condition number of the matrix Z := AB > BA > given by  ( Z ) :
=
max ( Z )  min ( Z ) .
Further,to compute an ✏ -optimal solution (3) , each agent is required to transmit:
[Equation]
In Theorem 4.2, it is required that matrix BA > is full rank, and this can be satisﬁed with probability1 if we choose B to be a random matrix.
Further, the convergence rate depends on the conditionnumber  ( Z ) .
Although we have showed the per iteration communication cost can be reduced to Hby linear compression, it is not clear yet whether the dimension D can affect the value of  ( Z ) .
Inthe following, we show  ( Z ) is independent of D ; The proof can be found in Appendix E.
Proposition 1. Consider the matrices A and B in Theorem 4.2, and deﬁne Z := AB > BA > .
Supposeeach entry in B satisﬁes B hj i.i.d ⇠ N (0 , 1) , 8 h 2 [ H ] , j 2 [ D ] .
Then the following holds:
(1) The condition number  ( Z ) is independent of D.(2)
Let A > = V ⌃ W be the compact form of SVD, where V 2 R D ⇥ N satisﬁes V T V
= I N ;⌃ 2 R N ⇥ N is the diagonal matrix whose diagonal entries are singular values of A ; and W 2 R N ⇥ N 258 is a unitary matrix.
Then for any ﬁxed A , we have
[Equation]
Proposition 1 implies that, as long as the compression matrix is standard normal, then  ( Z ) isindependent of D .
This means that the total bits required for each agent, as provided in (8), willonly be dependent on N, H and log( f ( ✓ 0 ) / ✏ ) , but not on D .
To further understand the dependencyof  ( Z ) with N and H , let us treat the ratio between the largest and smallest singular values asbeing ﬁxed 1 .
Then set 2 e  t 2 / 2 =  > 0 , we have t = q 2 log( 2  ) , so the upper bound for becomes  ( AA > ) · p H + p N + p 2log( 2  )
p H
p N  p 2log( 2  ) .
Assuming that N > 4 t 2 , H = 4 N , then we can concludethat P (  ( Z )  49)  1   .
Applying this result to (8), we can conclude that with probability 1   ,the total number of bits transmitted by each agent is 8 b ·  4 ( A T A ) · N log( f ( ✓ 0 ) / ✏ ) .
Similarly as in
Theorem 4.1, b is in the order of O (log( D ))
(see Appendix C.).
Therefore the total communicationcomplexity of each agent can be reduced to O
[Equation]
.
When H = ⇥ ( N ) , then thecomplexity bound matches the lower bound in Theorem 3.2, up to a gap
O (log( D )) .
Additionally,we note that by ﬁxing the seed of random number generator, the random matrix B can be generatedby each agent locally, without needing to transmit any information.
Decentralized Algorithm.
We propose a fully decentralized algorithm for solving (4) with acommunication complexity that approaches the lower bound in Theorem 3.2.
Our idea is similar to
Alg. 1 that adopts a dimension reduction step for compression, yet we shall encapsulate the Quantize ,
Communicate , Aggregation , Broadcast steps through a decentralized average consensus protocolwith support for quantized communication.
In particular, we consider a setting where K agents are represented as nodes on a connected graph( V, E ) with a doubly stochastic mixing matrix W 2 R K ⇥ K + .
Observe the following fact:
Fact 1.
Consider a set of vectors v 1 , . .
. , v K 2 R H held by K agents on ( V, E ) .
For any ✏ > 0 , the
CHOCO-GOSSIP protocol [Koloskova et al., 2019] is described as the map:
[Equation]
such that ¯ v k is the vector available at agent k satisfying k ¯ v k  K
P
[Equation]
 ✏ .
Let ⇠ =
P
[Equation]
P
[Equation]
k 2 be the initial consensus error, the protocol requires at most(82 /
[Equation]
log( ⇠
[Equation] ✏ [Equation] rounds of quantized communication , where  = 1  |  2 ( W ) | and ! 2 (0 , 1] is controlled by the number of bits in the quantizer.
Detailsabout the CHOCO-GOSSIP protocol can be found in Appendix F.1.
We replace line 5–line 8 in Alg. 1 by CHOCO-GOSSIP to yield the decentralized algorithm:
Algorithm 2 Decentralized Gradient Descent with Compressed Comm. via Linear Compression
Initialize 3: for do 4:
Compute : compute the compressed local gradient g tk = F ( r f k ( ✓ tk )) .
Consensus : Agents exchange gradients g tK
= CHOCO
GOSSIP g tK
+1 ) .
Update : Each agent updates
[Equation]
7: end for 8: Output: the last iterate ✓ Ti 286 Theorem 4.3.
Consider using Alg. 2 to solve the quadratic problem (4) , with A being full rowrank.
Choose B such that rank ( BA > ) =
N ; set ⌘ =  min ( Z ) 4  2max ( Z ) , and choose ˜ F ( · ) and F ( · ) as
F ( x ) =
Bx, ˜ F ( y )
=
B > y .
For any T  1 , the following holds true:
[Equation]
where  was deﬁned in Theorem 4.2.
Moreover, suppose that A, B are constructed as in Proposition 1,to compute an ✏ -optimal solution (3) , each agent is required to transmit:
O
[Equation]
⇣
[Equation]
, with probability at least 1  ⇣ .
Proposition 1 shows that the condition number  is O (1) .
To satisfy rank ( BA > )
=
N , we need
H  N .
Altogether, this yields the communication complexity O
( N log( N ) log( D ) log(1 / ✏ ) 2 ) tomatch Theorem 3.2 up to a log( D ) factor.
The Case of Overparameterized Neural Network.
We now explore the possibility of extendingthe analysis of quadratic problems to a speciﬁc overparameterized neural network, still belongingto class C op .
We will customize Alg. 1 to partially compress the gradients of the loss function.
Ouridea is to design F ( · ) (resp.
˜ F ( · ) )
so that we compress (resp.
decompress)
the gradient with respectto the parameters in the widest layer to O ( N ) bits, while directly quantize the remaining gradients.
Note that for a few popular network architectures such as Neural Tangent Kernel (NTK)
[Huanget al., 2021], the widest layer contains most of the parameters ⌦ ( N 4 ) .
Therefore, compressing theseparameters into O ( N ) bits already represents signiﬁcant communication .
We will leave the designand analysis of more aggressive schemes which compress the parameters of all layers for future work.
To proceed, let us ﬁrst state the structure of the network.
Let X 2 R N ⇥ n 0 denote the data sample; leta ( · ) be an activation function; let O l 2 R N ⇥ n l as the output of each layer, which can be expressed as
[Equation]
where W l 2 R n l  1 ⇥ n l is the weight matrix in l -th layer.
Further, denote O L,k as the output of thenetwork for k -th agent.
Let ✓ =
( W l ) Ll =1 collect all the weights.
Then objective function becomesf
[Equation]
✓ [Equation] Further, we deﬁne the vectorized gradient of each layer as u l = vec ( r W l f ( ✓ )) and u l,k for eachagent.
Now we describe the assumptions on the network.
Assumption 4.
(Pyramidal network topology) Let n 1  N and n 2  n 3  . . .
n L = 1 .
Assumption 5.
(Activation function)
Fix ⌫ 2 (0 , 1) and ⇢ > 0 .
Let a satisfy that: ( i ) a 0 ( x ) 2 [ ⌫ , 1] ,(ii) | a ( x ) | 6 | x | for every x 2 R , and (iii) a 0 is ⇢ -Lipschitz.
As showed in [Nguyen and Mondelli, 2020], with Assumption 4 and 5, objective (6) belongs to C op .
With the above Pyramidal structure, layer l = 2 is the widest layer and contains the largest number ofparameters, denoted as u 2 2 R n 1 ⇥ n 2 .
As mentioned above, we will design F and ˜ F to compress anddecompress u 2 .
To proceed, Let us choose B
= I n 2 ⌦ ˜ B 2 R
Hn 2 ⇥ n 1 n 2 where ˜ B 2 R H ⇥ n 1 , suchthat N 6 H ⌧ n 1 .
Then F and ˜ F are deﬁned as:
[Equation]
This means that we compress the parameters in the second layer, and perform usual gradient descentfor the layers l  3 ; further, the parameters in the ﬁrst layer is left frozen at its initialization.
Notethat we freeze the ﬁrst layer for some technical reasons, but layer freezing itself is a useful techniqueto accelerate the training in practice, which can provide strong generalization ability [Yosinski et al.
,2014, Advani et al., 2020].
Let us denote O 01 as the output of the ﬁrst hidden layer at initialization.
We have the following result.
Its proof as well as the expressions for various constants, can be found in Appendix G.
Theorem 4.4.
(Informal) Consider using Alg. 1 to solve the quadratic problem (6) , with X beingfull row rank.
Suppose ✓ 0 is initialized properly (see Appendix G for details).
Choose ˜ B such asrank ( ˜ BO 01 > ) =
N ; choose ˜ F ( · ) and F ( · ) as in (7) .
Then there exists stepsize ⌘ , constant  , C, b 328 such that to compute an ✏ -optimal solution (6) , each agent is required to transmit:
[Equation]
where ⌘ ,  are related to X, ⌫ , ⇢ and singular values of the weight matrices at initialization.
Theorem 4.4 shows that if we choose H = ⇥ (
N ) , and when n 1 is very large, then the totalcommunication complexity can be reduced by a factor of n 1 /H .
Preliminary Numerical Experiments.
We conclude by presenting a numerical experiment for the
UCI Tom’s Hardware 2 dataset using Alg. 2 where we applied b log( t +100) c rounds of communicationat the t -th iteration for the CHOCO-GOSSIP subroutine; see Appendix F.1.
We consider a ring networkwith K = 5 agents, each one has 500 or 1000 samples (thus making N = 2500 , or N = 5000 ).
We construct D -dimensional features from the dataset as NTK features [Bietti and Mairal, 2019].
In Fig.
1, we train a least square regression model in the overparameterized regime.
We observethat the communication cost needed is proportional to the number of samples, instead of the featuredimension.
Particularly, our results indicate that the communication cost required to reach a targettrain MSE (of 10  4 ) is proportional to the number of training samples.