Distributed Optimization for Overparameterized Problems: Achieving Optimal Dimension Independent Communication Complexity
Abstract
Decentralized optimization are playing an important role in applications such as training large machine learning models, among others.
Despite its superior practical performance, there has been some lack of fundamental understanding about its theoretical properties.
In this work, we address the following open research question: To train an overparameterized model over a set of distributed nodes, what is the minimum communication overhead (in terms of the bits got exchanged) that the system needs to sustain, while still achieving (near) zero training loss?
We show that for a class of overparameterized models where the number of parameters
D is much larger than the total data samples N , the best possible communication complexity is ⌦ ( N ) , which is independent of the problem dimension D .
Further, for a few speciﬁc overparameterized models (i.e., the linear regression, and certain multi-layer neural network with one wide layer), we develop a set of algorithms which uses certain linear compression followed by adaptive quantization, and show that they achieve dimension independent, near-optimal communication complexity.
To our knowledge, this is the ﬁrst time that dimension independent communication complexity has been shown for distributed optimization.
Introduction
The research of decentralized/distributed optimization has recently gained tremendous momentum, partly due to the fact that a well-designed decentralized/distributed algorithm is capble of training large-scale machine learning models, by utilizing massively distributed computation and data resources.
In a typical decentralized optimization setting, multiple local agents hold small to moderately sized datasets and they collaborate by iteratively solving their local problems while sharing some information with other agents.
The majority of the state-of-the-art distributed/decentralized optimization algorithms are deeply rooted in classical consensus-based approaches [Tsitsiklis, 1984, Tsitsiklis and Luo, 1987], where the agents repetitively share the local optimization variables, or the gradients of local functions, so to reach an optimal consensual solution.
It is important to note that in those classical settings, the agents typically solve small-scale problems where the problem dimension (i.e., the dimension of the optimization variable, and the gradients) is relatively small (e.g., in the order of hundreds or thousands).
Therefore, it is reasonable that the agents should iteratively share their local parameters/gradients to improve the quality of the models.
However, many modern learning models are in the overparameterized regime with very high-dimensional parameters [He et al., 2016, Vaswani et al., 2017], such as deep and/or wide neural networks.
This practice poses some signiﬁcant challenges to the classical consensus-based approaches, because by directly sharing the model parameters and/or gradients, the communication burden increases as problem size increases.
A line of recent works have been developed, which adopt strategies such as gradient/model compression, lazy communication, etc, and they are able to signiﬁcantly reduce the overall communication complexities
[Xu et al., 2020, Sun et al., 2020, Li et al., 2019, Karimireddy et al., 2019].
A few works focus on designing compressors [Safaryan et al., 2020, Horváth et al., 2019, Beznosikov et al., 2020, Koneˇcn`y et al., 2016, Safaryan et al., 2021].
Although the majority of these works are able to reduce per round communication burden, it is by no means clear if, to achieve certain solution quality, the total amount of communication can be saved.
In fact, it has been shown in [Arjevani and Shamir, 2015], that the total communication achievable is dependent on the problem dimension, which could be huge.
More importantly, there is a general lack of fundamental understanding about the performance limits for decentralized and distributed optimization, when the models are ‘large’.
Without such characterization, it is by no means clear if the existing algorithms are optimal in reducing the communication complexities.
In this work, we focus on distributed optimization for ‘large’ machine learning models (i.e., overparameterized problems, to be deﬁned shortly), and we ask the following question:[-
-]
(Q)
What is the best possible communication complexity (in terms of number of bits needed to obtain certain solution accuracy), achievable by any distributed algorithms for a class of overparameterized problems?
Which class of algorithms achieves such complexity bounds?
Related Works.
A number of recent works are devoted to reducing the communication cost in distributed/decentralized optimization by combining the optimization algorithms with compression , e.g., sparsiﬁcation, quantization, etc.
For the distributed setting with the server-worker architecture, relevant works include [Alistarh et al., 2017, Wen et al., 2017, Wang et al., 2018, Bernstein et al., 2018, Haddadpour et al., 2021, Mishchenko et al., 2019, Gorbunov et al., 2021]; see [Richtárik et al., 2021] for a recent survey.
For the decentralized setting without a central server, recent works such as [Koloskova et al., 2019, Reisizadeh et al., 2019, Tang et al., 2018, Vogels et al., 2020, Magnússon et al., 2020] proposed algorithms that utilize error compensation, low-rank compression, etc.
While they have demonstrated appealing performance in practice, the theoretical guarantees do not scale for overparameterized problems.
For instance, [Koloskova et al., 2019] showed an O (1 /T
+ 1 / ( ! T ) 2 ) iteration complexity upper bound for their algorithm.
The quantity 1 / !
is the compressor parameter which is at least ⌦ ( p D ) , where D is the problem dimension.
Lower communication complexity bounds for distributed optimization problems have been studied since the 1980s.
There are two major categories of lower bound results in literature depending on how we count the “communication” required to reach a solution of given accuracy, i.e., 1) in terms of the number of communication rounds, or 2) in terms of the number of bits got communicated.
The ﬁrst category is not directly related to the current work; we refer to readers to Appendix A for discussions.
In second category, the ﬁrst lower bound is given in [Tsitsiklis and Luo, 1987] where the authors consider a distributed setting with two nodes and provide a ⌦ ( D log ( D/ ✏ )) bound for the class of quadratic objectives, where ✏ is the solution accuracy, and D is the variable dimension.
In [Zhang et al., 2013] lower communication bounds are derived for certain statistical estimation tasks, such as linear and probit regression.
Moreover, in [Vempala et al., 2020] the communication complexity of several distributed optimization tasks is studied, including ` p regression, linear, semideﬁnite and convex optimization.
In [Korhonen and Alistarh, 2021], an ⌦ ( KD log ( D/ ✏ )) bound is derived for quadratic problems on a multi-node setting, for both deterministic and randomized protocols, where
K is the number of clients.
Recently, overparameterized problem has drawn great research interests.
It has been showed that remarkable performance can be achieved with overparameterized networks [Jacot et al., 2018, Du et al., 2019, Allen-Zhu et al., 2019].
Some recent works study the overparameterization of decentralized problems, which guarantee the global convergence [Huang et al., 2021, Deng et al., 2022].
However, the high communication cost caused by the large model is a key problem in decentralized algorithm.
For example, the ReLU fully connected networks designed in the aforementioned two works are of widths ⌦ ( N 4 ) and ⌦ ( N 18 ) respectively, where N is the number of data samples available.
Contributions.
This work addresses both questions in (Q) .
The contributions are highlighted below:
(i)
Regarding to the best possible communication complexity (i
e the communication complexity lower bound)
we ﬁrst
show that
for a class of Polyak- Lojasiewicz (PL) problems the communication
lower
bound is ⌦ ( KD log
D/ ✏ )) ,
which is
similar
to the known results for strongly convex problems. Further, we show that for a class of overparameterized models where the loss function is a composition of a quadratic problem and some nonlinear functions, and that the number of parameters D is much larger than the total data samples N , the communication complexity lower bound becomes ⌦ ( KN log( K/ ✏ )) , which is independent of the problem dimension D
(ii) We develop a distributed algorithm which can be used to optimize both classes of problems, in the setting where multiple clients are coordinated by a server.
We show that the lower bound for the PL problem can be matched; while by using certain linear random compression/decompression scheme, followed by adaptive quantization, the lower bound for the overparameterized problem can be matched (up to a log( D ) factor) for two of its important special cases, i
e., linear regression
and certain multi-layer neural network with one wide layer.
Further, we show that the proposed algorithm can be extended to a fully decentralized setting without the coordination of a server.
To our knowledge, these are the ﬁrst set of results that attempt to comprehensively address the issue of optimal communication complexities for overparameterized problems.
We expect that results provided in this work will open doors for understanding more complex communication-efﬁcient distributed algorithms, as well as larger classes of overparameterized problems.
The System Model
Consider a system having K distributed agents, and a total of N samples { ( X i , y i ) } Ni =1 .
Suppose each agent has N k data samples, with
[Equation]
=1 N k = N ; let set X k (resp.
Y k ) collect agent k ’s data samples (resp. labels).
They cooperatively solve an optimization problem of the following form:
[Equation]
where f k : ⇥ !
R is the local loss function for agent k ; ✓ is the optimization variable, whose domain is ⇥ ✓ R D .
For notational simplicity, we will ignore the reference for the data samples X , Y and
X k , Y k whenever possible, i.e., we will write f ( ✓ ) and f k ( ✓ ) as the global and local loss functions.
Throughout the paper, we will consider a few special cases of (1), where the local function f k ’s are overparameterized .
The ﬁrst of such problem class is referred to as the distributed PL problems .
To introduce this class, let us provide a few assumptions.
Assumption 1.
The local objective functions satisfy:
[Equation]
where ✓ ⇤ ( k ) is a global minimum of f k ( · ) ; µ k ’s some positive constants.
Assumption 2.
There exists positive constants L k ’s and L  1 such that:
[Equation]
Note that (2) is the so-called PL assumption, which was introduced in [Polyak, 1963].
It is weaker than strong convexity, while it can still be used to ensure the linear convergence of gradient descent.
It is also less restrictive than several other related conditions under which linear global convergence can be derived, such as the restricted secant inequality [Karimi et al., 2016].
The second inequality in (2) has also been used in the literature; see, e.g., [Oymak and Soltanolkotabi, 2019], in which such a bound for overparameterized network has been showed.
Assumption 2 is the standard Lipschitz gradient condition.
Deﬁnition 1 (Distributed PL problems) .
Consider problems of the form (1) .
Deﬁne:
[Equation]
satisﬁes Assumptions
[Equation]
The second class of problems is referred to as the distributed overparameterized problem, where the loss function is a composition of least square loss and a non-linear function.
This form of loss function often appears in analyzing neural networks, see, e.g., [Oymak and Soltanolkotabi, 2019].
Deﬁnition 2 (Distributed overparameterized problems) .
Consider problems of the form (1) .
Deﬁne functions G k ( · ) : ⇥ !
R N k and constants
b k 2 R N k , for all k .
We call G k ( · ) overparameterized if for every b k 2 R N k , there exists ✓ 2 ⇥ such that G k ( ✓ ) =
b k .
Deﬁne the following class of problems:
[Equation]
In this section we derive the communication complexity lower bounds for classes C op and C pl . Throughout this section and the paper, our focus is given to deterministic algorithms where no randomness is involved.
All the proofs for results presented in this section are given in Appendix B.
We start with a description of the coordinator model (also see [Braverman et al., 2013, Korhonen and Alistarh, 2021]), which will be used later as the basis of our derivation.
Consider a set of K distinct nodes/agents, and a single coordinator node.
Each node and the coordinator have a two-way communication channel, over which binary messages are exchanged.
Consider a class of problems C of the form (1), that is f ( ✓ ) =
P
Kk =1 f k ( ✓ ) , where each node k has access only to objective f k ( ✓ ) .
Moreover, let ⇧ be a set of protocols that are executed over the coordinator model.
Then, given a problem class C , a protocol ⇡ 2 ⇧ ( ✏ ) takes as input a problem c 2 C , performs a number of message exchanges (over the coordinator model), and outputs an ✏ -approximate solution ˜ ✓ 2 ⇥ of problem (1).
That is, the output ˜ ✓ satisﬁes (where ✓ ⇤ is a point where the exact minimum is attained):
[Equation]
Next, let us look at ⇡ in more detail.
We assume that for ⇡ 2 ⇧ ( ✏ ) , it performs T ( ✏ ) rounds of communication and computation, where during each round the following steps are carried out:
• At round t = 0 , each agent/node k is given as input the local objective f k (e.g., the data samples X k , Y k assigned to each node); • At round 1  t < T ( ✏ ) , – The coordinator sends a message m ( t ) c !
k 2 { 0 , 1 } to some node k .
– Node k sends message m ( t ) k !
c 2 { 0 , 1 } to the coordinator.
• At the ﬁnal round t = T ( ✏ ) the coordinator outputs an ✏ -approximate solution.
Next, we deﬁne the communication complexity of the distributed protocol.
Let us denote the total number of bits exchanged under protocol ⇡ , for problem c 2 C , as M ( c, ⇡ , ✏ ) ; Use the following deﬁnition, which has been used extensively in related works such as [Tsitsiklis and Luo, 1987].
Deﬁnition 3 (Communication complexity) .
The communication complexity of protocol ⇡ over problem class C is deﬁned as the maximum number of bits exchanged across the problems in C , i.e.,
M ( C , ⇡ , ✏ ) :=
max c 2 C M ( c, ⇡ , ✏ ) .
Also, the communication complexity of problem class C is the minimum communication complexity across all protocols that solve C , i.e.,
[Equation]
Next, we derive communication complexity lower bounds for certain problem classes C .
Our construction strategies follow those from [Korhonen and Alistarh, 2021], that is, we will reduce a problem with known communication complexity, to hard problem instances within classes of our interest.
The main difference from the aforementioned work (which only deals with quadratic problems) is that, we need to construct hard PL and overparameterized problems instances.
Below we derive communication complexity lower bounds for the class of deterministic distributed problems where the local loss functions belong to the class of PL functions deﬁned in Def. 1.
Theorem 3.1.
Consider the problem class C pl with given parameters K, D, ✏ , for which it holds that D ✏ = ⌦ (1) .
Then we have M ( C pl , ✏ ) =
⌦ ( KD log
D ✏  ) .
Note that the condition D/ ✏ = ⌦ (1) is reasonable since we will mainly focus on the regime where D is large.
Next we consider C op .
We have the following communication complexity lower bound.
Theorem 3.2.
Consider the problem class C op , with given parameters K, N, D, ✏ , for which it holds that N ✏ = ⌦ (1) .
Then we have that M ( C op , ✏ ) =
⌦ ( KN log
N ✏  ) .
Clearly, the lower bound for class C op does not explicitly depend on the problem dimension D , while that for the C pl does.
This appears to be reasonable, partly because in the class of C pl the number of data samples N has not been explicitly modeled.
Nevertheless, at this point it is not clear if such a dimension-independent lower bound is tight or not, because we are yet to ﬁnd an algorithm to match such a lower bound.
To our knowledge, these are the ﬁrst set of results that characterize the communication complexity for the classes C pl and C op , and they will serve to guide our subsequent algorithm development and analysis.
Algorithms Achieving the Optimal Communication Complexities
In this section, we design algorithms and analyze the achievable communication complexities, for the PL and the overparameterized problems in Def. 1 and 2.
We will begin with the simple setting where all the agents are connected to a coordinator, and then extend to the case where the agents are decentralized.
Throughout this section, for simplicity of exposition, we will assume that ⇥ ⌘ R D .
Let us begin with a generic algorithm to deal with the server-agent setting.
The algorithm is an adaptation of the classical quantize-then-communicate strategy [Tsitsiklis and Luo, 1987, Magnússon et al., 2020].
At each iteration, each agent ﬁrst computes its gradient, followed by certain adaptive quantization scheme, before getting transmitted to the collaborating agents.
Speciﬁcally, at t -th iteration, each agent uses a function F : R D !
R H to compress the local gradient; Then the output is quantized as q tk 2 R H (use the method in Def. 4).
Each agent then shares q tk to the server, and the aggregated q t is broadcast to all agents.
Each agent then uses a function ˜ F : R H !
R D to process the received vector, and updates the local parameters.
One key difference compared with [Tsitsiklis and Luo, 1987, Magnússon et al., 2020] is that, the algorithm uses a pair of compression and decompression functions F ( · ) and ˜ F ( · ) to process the gradients transmitted/received.
We will see shortly that such a pair plays an important role in matching the lower bound.
For the detailed steps, see Algorithm 1 below.
In Algorithm 1, we note that as long as the initial solution ✓ 0 is the same among all the agents, then ✓ t +1 k computed in the ‘ Update ’ step will always be the same across the agents.
This is the reason that we can directly set ✓ t +1 k as ✓ t +1 .
Subsequently we will extend this algorithm to a fully decentralized setting, where precise consensus is no longer required.
We choose the quantization method for the ﬁrst iteration deﬁned in Appendix C.
In the subsequent iterations, we use the following adaptive quantization scheme; also see [Magnússon et al., 2020].
Deﬁnition 4.
Fix r 2 R + and let b 2 N be an positive integer.
Let quant : R H ⇥ R H ⇥ R + ⇥ N !
R H be the quantization function deﬁned component-wise as follows,
where  ( r, b ) :
= r/
[Equation]
b [Equation] [ quant ( c, p, r, b )] j
[Equation]
Essentially, the quantization scheme searches for a closest point on the grid (which has a width 2 r and is centered at p =
( p 1 , . . .
, p H ) .
The distance between the neighbor points on the grid is  ( r, b ) .
Lemma 4.1.
Fix p 2 R H , then for all c 2 R H , Q 2 R D ⇥ H , such that k c  p k 1  r we have k quant ( c, p, r, b )  c k 1 
[Equation]
4.1 Communication Complexity Upper Bounds for C pl
We will begin with analyzing the class of problem C pl , as deﬁned in Def. 1.
Before we proceed, let us make an additional assumption.
Assumption 3.
Assume that each f k ( · ) is non-negative in its domain; Further, assume that the optimal objective value for problem (1) is zero, that is, there exists ✓ ⇤ 2 R D such that f ( ✓ ⇤ )
= 0 .
The above ‘zero loss’ assumption has been used by a number of recent works to characterize problems with a large number of parameters, e.g., in overparameterized neural network [Liu et al., 2022, Nguyen and Mondelli, 2020]; in optimization literature, this assumption is also used to improve the analysis of gradient-based algorithms; see, e.g. [Razaviyayn et al., 2019].
Lemma 4.2.
Suppose Assumption 2 – 3 are satisﬁed, then we have the following inequality:
[Equation]
The above lemma shows that together with the PL condition, Assumptions 2, 3 imply:
[Equation]
For convenience, throughout our analysis we set µ < 1 , L > 1 .
We will refer to the class of problems satisfying Assumption 1 to 3 as e
C pl .
We note that e
C pl is a strict subset of the class
C pl .
Next, let us present our result for C pl .
The proof can be found in Appendix C.
Theorem 4.1.
Suppose Assumptions 2 – 3 are satisﬁed.
Consider using Alg. 1 to solve C pl ; set
[Equation]
and C := max
[Equation]
Then the following holds true:
[Equation]
Further, to compute an ✏ -optimal solution satisfying (3) , each agent is required to transmit:
[Equation]
The quantization strategy at the ﬁrst iteration (in Appendix C) requires b 0 to be O  log( D )  , so we can conclude b
= O  log( D )  .
Thus Theorem 4.1 gives the total communication complexity for each agent as O
[Equation]
, which matches the lower bound in Theorem 3. 1, up to a gap log( D )
4.2 Communication Complexity Upper Bounds for C op
Next, we develop algorithms for the distributed overparameterized problem deﬁned in Def. 2.
Note that functions in the class C op can be non-convex, therefore one may not be able to ﬁnd an algorithm that can solve the entire class of problems in polynomial time.
Our strategy is to identify special cases of C op for which the analysis is possible.
Again, although these problems are restrictions of C op , we do not expect the lower bound in Theorem 3.2 to be improved in either the dependency on N or on ✏ .
The Case for Quadratic Problems.
Towards this end, let us consider an overparameterized quadratic problem.
To proceed, assume that each agent k has data tuple ( A k , b k ) with N k samples, where A k 2 R N k ⇥ D is the feature matrix, and b k 2 R N k ⇥ 1 is the label.
Denote A :=
[ A 1 ; · · · ; A K ] 2 R N ⇥ D , where N =
P
Kk =1 N k .
Suppose N < D .
Then the generic problem (1) reduces to:
[Equation]
Assuming that each A k has full row rank, then the above problem is a special case of C op .
Next, we specialize Alg. 1.
Recall that the lower bound derived in Theorem 3.2 is independent of D , so we cannot directly apply the algorithm we used for the class C pl , in which each agent transmits a dimension D vector at every iteration.
To proceed, let us consider the following linear compression scheme, in which we use a matrix B of size H ⇥ D to compress (resp.
decompress)
the gradients that get transmitted (resp. received), where H is a ﬁxed positive constant satisfying N 6 H ⌧ D .
In particular, let us specialize the two processing operators in Alg. 1 as: F ( x ) =
Bx, ˜ F ( y )
=
B > y .
Further, deﬁne  max ( · ) and  min ( · ) (resp.
s max ( · ) and s min ( · ) ) as the largest and smallest eigenvalues (resp.
singular values) for a matrix.
Then we have the following result; see Appendix D for proof.
Theorem 4.2.
Consider using Alg. 1 to solve the quadratic problem (4) , with A being full row rank.
Choose B such that rank ( BA > ) =
N ; set ⌘ =  min Z )  2max ( Z ) ; set b and C according to (30) , and choose ˜ F ( · ) and F ( · ) as F ( x ) =
Bx, ˜ F ( y )
=
B > y .
Then the following holds true:
[Equation]
where  is the condition number of the matrix Z := AB > BA > given by  ( Z ) :
=
max ( Z )  min ( Z ) .
Further, to compute an ✏ -optimal solution (3) , each agent is required to transmit:
[Equation]
In Theorem 4.2, it is required that matrix BA > is full rank, and this can be satisﬁed with probability 1 if we choose B to be a random matrix.
Further, the convergence rate depends on the condition number  ( Z ) .
Although we have showed the per iteration communication cost can be reduced to H by linear compression, it is not clear yet whether the dimension D can affect the value of  ( Z ) .
In the following, we show  ( Z ) is independent of D ; The proof can be found in Appendix E.
Proposition 1. Consider the matrices A and B in Theorem 4.2, and deﬁne Z := AB > BA > .
Suppose each entry in B satisﬁes B hj i.i.d ⇠ N (0 , 1) , 8 h 2 [ H ] , j 2 [ D ] .
Then the following holds:
(1) The condition number  ( Z ) is independent of D. (2) Let A > = V ⌃ W be the compact form of SVD, where V 2 R D ⇥ N satisﬁes V T V
= I N ; ⌃ 2 R N ⇥ N is the diagonal matrix whose diagonal entries are singular values of A ; and W 2 R N ⇥ N is a unitary matrix.
Then for any ﬁxed A , we have
[Equation]
Proposition 1 implies that, as long as the compression matrix is standard normal, then  ( Z ) is independent of D .
This means that the total bits required for each agent, as provided in (8), will only be dependent on N, H and log( f ( ✓ 0 ) / ✏ ) , but not on D .
To further understand the dependency of  ( Z ) with N and H , let us treat the ratio between the largest and smallest singular values as being ﬁxed 1 .
Then set 2 e  t 2 / 2 =  > 0 , we have t = q 2 log( 2  ) , so the upper bound for  becomes  ( AA > ) · p H + p N + p 2log( 2  )
p H p N  p 2log( 2  ) .
Assuming that N > 4 t 2 , H = 4 N , then we can conclude that P (  ( Z )  49)  1   .
Applying this result to (8), we can conclude that with probability 1   , the total number of bits transmitted by each agent is 8 b ·  4 ( A T A ) · N log( f ( ✓ 0 ) / ✏ ) .
Similarly as in
Theorem 4.1, b is in the order of O (log( D ))
(see Appendix C.).
Therefore the total communication complexity of each agent can be reduced to O
[Equation]
.
When H = ⇥ ( N ) , then the complexity bound matches the lower bound in Theorem 3.2, up to a gap
O (log( D )) .
Additionally, we note that by ﬁxing the seed of random number generator, the random matrix B can be generated by each agent locally, without needing to transmit any information.
Decentralized Algorithm.
We propose a fully decentralized algorithm for solving (4) with a communication complexity that approaches the lower bound in Theorem 3.2.
Our idea is similar to
Alg. 1 that adopts a dimension reduction step for compression, yet we shall encapsulate the Quantize ,
Communicate , Aggregation , Broadcast steps through a decentralized average consensus protocol with support for quantized communication.
In particular, we consider a setting where K agents are represented as nodes on a connected graph ( V, E ) with a doubly stochastic mixing matrix W 2 R K ⇥ K + .
Observe the following fact:
Fact 1.
Consider a set of vectors v 1 , . .
. , v K 2 R H held by K agents on ( V, E ) .
For any ✏ > 0 , the
CHOCO-GOSSIP protocol [Koloskova et al., 2019] is described as the map:
[Equation]
such that ¯ v k is the vector available at agent k satisfying k ¯ v k  K
P
Kj =1 v j k 2  ✏ .
Let ⇠ =
P
Kk =1 k v
[Equation]
K
[Equation]
P
Kj =
[Equation]
j k 2 be the initial consensus error, the protocol requires at most
[Equation]
[Equation]
log( ⇠ / ✏ ) rounds of quantized communication , where  = 1  |  2 ( W ) | and ! 2 (0 , 1] is controlled by the number of bits in the quantizer.
Details about the CHOCO-GOSSIP protocol can be found in Appendix F.1.
We replace line 5–line 8 in Alg. 1 by CHOCO-GOSSIP to yield the decentralized algorithm:
Algorithm 2 Decentralized Gradient Descent with Compressed Comm. via Linear Compression
: Input : Initialize do 4:
Compute : Each agent compute the compressed local gradient g tk = F ( r f k ( ✓ tk )) .
Consensus : Agents exchange gradients (¯ g t 1 , . . , ¯ g tK ) =
CHOCO-GOSSIP ( g t 1 , . .
. , g tK ; ¯ ✏ t +1 ) .
Update : Each agent updates ✓ t +1 k
[Equation]
7: end for 8: Output: the last iterate ✓ Ti Theorem 4.3.
Consider using Alg. 2 to solve the quadratic problem (4) , with A being full row rank.
Choose B such that rank ( BA > ) =
N ; set ⌘ =  min ( Z ) 4  2max ( Z ) , and choose ˜ F ( · ) and F ( · ) as
F ( x ) =
Bx, ˜ F ( y )
=
B > y .
For any T  1 , the following holds true:
[Equation]
where  was deﬁned in Theorem 4.2.
Moreover, suppose that A, B are constructed as in Proposition 1, to compute an ✏ -optimal solution (3) , each agent is required to transmit:
[Equation]
with probability at least 1  ⇣ .
Proposition 1 shows that the condition number  is O (1) .
To satisfy rank ( BA > )
=
N , we need
H  N .
Altogether, this yields the communication complexity
O
( N log( N ) log( D ) log(1 / ✏ ) 2 ) to match Theorem 3.2 up to a log( D ) factor.
The Case of Overparameterized Neural Network.
We now explore the possibility of extending the analysis of quadratic problems to a speciﬁc overparameterized neural network, still belonging to class C op .
We will customize Alg. 1 to partially compress the gradients of the loss function.
Our idea is to design F ( · ) (resp.
˜ F ( · ) )
so that we compress (resp.
decompress)
the gradient with respect to the parameters in the widest layer to O ( N ) bits, while directly quantize the remaining gradients.
Note that for a few popular network architectures such as Neural Tangent Kernel (NTK)
[Huang et al., 2021], the widest layer contains most of the parameters ⌦ ( N 4 ) .
Therefore, compressing these parameters into O ( N ) bits already represents signiﬁcant communication .
We will leave the design and analysis of more aggressive schemes which compress the parameters of all layers for future work.
To proceed, let us ﬁrst state the structure of the network.
Let X 2 R N ⇥ n 0 denote the data sample; let a ( · ) be an activation function; let O l 2 R N ⇥ n l as the output of each layer, which can be expressed as
[Equation]
where W l 2 R n l  1 ⇥ n l is the weight matrix in l -th layer.
Further, denote O L,k as the output of the network for k -th agent.
Let ✓ =
( W l ) Ll =1 collect all the weights.
Then objective function becomes
[Equation]
Further, we deﬁne the vectorized gradient of each layer as u l = vec ( r W l f ( ✓ )) and u l,k for each agent.
Now we describe the assumptions on the network.
Assumption 4.
(Pyramidal network topology) Let n 1  N and n 2  n 3  . . .
n L = 1 .
Assumption 5.
(Activation function)
Fix ⌫ 2 (0 , 1) and ⇢ > 0 .
Let a satisfy that: ( i ) a 0 ( x ) 2 [ ⌫ , 1] , (ii) | a ( x ) | 6 | x | for every x 2 R , and (iii) a 0 is ⇢ -Lipschitz.
As showed in [Nguyen and Mondelli, 2020], with Assumption 4 and 5, objective (6) belongs to C op .
With the above Pyramidal structure, layer l = 2 is the widest layer and contains the largest number of parameters, denoted as u 2 2 R n 1 ⇥ n 2 .
As mentioned above, we will design F and ˜ F to compress and decompress u 2 .
To proceed, Let us choose B
= I n 2 ⌦ ˜ B 2 R
Hn 2 ⇥ n 1 n 2 where ˜ B 2 R H ⇥ n 1 , such that N 6 H ⌧ n 1 .
Then F and ˜ F are deﬁned as:
[Equation]
This means that we compress the parameters in the second layer, and perform usual gradient descent for the layers l  3 ; further, the parameters in the ﬁrst layer is left frozen at its initialization.
Note that we freeze the ﬁrst layer for some technical reasons, but layer freezing itself is a useful technique to accelerate the training in practice, which can provide strong generalization ability [Yosinski et al., 2014, Advani et al., 2020].
Let us denote O 01 as the output of the ﬁrst hidden layer at initialization.
We have the following result.
Its proof as well as the expressions for various constants, can be found in Appendix G.
Theorem 4.4.
(Informal) Consider using Alg. 1 to solve the quadratic problem (6) , with X being full row rank.
Suppose ✓ 0 is initialized properly (see Appendix G for details).
Choose ˜ B such as rank ( ˜ BO 01 > ) =
N ; choose ˜ F ( · ) and F ( · ) as in (7) .
Then there exists stepsize ⌘ , constant  , C, b such that to compute an ✏ -optimal solution (6) , each agent is required to transmit:
[Equation]
where ⌘ ,  are related to X, ⌫ , ⇢ and singular values of the weight matrices at initialization.
Theorem 4.4 shows that if we choose H = ⇥ ( N ) , and when n 1 is very large, then the total communication complexity can be reduced by a factor of n 1 /H .
Preliminary Numerical Experiments.
We conclude by presenting a numerical experiment for the
UCI Tom’s Hardware 2 dataset using Alg. 2 where we applied b log( t +100) c rounds of communication at the t -th iteration for the CHOCO-GOSSIP subroutine; see Appendix F.1.
We consider a ring network with K = 5 agents, each one has 500 or 1000 samples (thus making N = 2500 , or N = 5000 ).
We construct D -dimensional features from the dataset as NTK features [Bietti and Mairal, 2019].
In Fig.
1, we train a least square regression model in the overparameterized regime.
We observe that the communication cost needed is proportional to the number of samples, instead of the feature dimension.
Particularly, our results indicate that the communication cost required to reach a target train MSE (of 10  4 ) is proportional to the number of training samples.
Acknowledgments and Disclosure of Funding We sincerely thank the anonymous reviewers for the suggestions.
M. Hong and B. Song are supported by NSF grants CIF-1910385, CMMI-1727757 and
CNS-2003033.