L INEAR C ONVERGENT D ECENTRALIZED O PTIMIZATION WITH C OMPRESSION
A BSTRACT
Communication compression has become a key strategy to speed up distributed optimization.
However, existing decentralized algorithms with compression mainly focus on compressing DGD-type algorithms.
They are unsatisfactory in terms of convergence rate, stability, and the capability to handle heterogeneous data.
Motivated by primal-dual algorithms, this paper proposes the ﬁrst LinEAr convergent Decentralized algorithm with compression
LEAD
Our theory de- scribes the coupled dynamics of the inexact primal and dual update as well as compression error, and we provide the ﬁrst consensus error bound in such settings without assuming bounded gradients.
Experiments on convex problems validate our theoretical analysis, and empirical study on deep neural nets shows that LEAD is applicable to non-convex problems
I NTRODUCTION
Distributed optimization solves the following optimization problem
[Equation]
with n computing agents and a communication network.
Each f i ( x ) : R d → R is a local objective function of agent i and typically deﬁned on the data D i settled at that agent.
The data distributions {D i } can be heterogeneous depending on the applications such as in federated learning.
The variable x ∈ R d often represents model parameters in machine learning.
A distributed optimization algorithm seeks an optimal solution that minimizes the overall objective function f ( x ) collectively.
According to the communication topology, existing algorithms can be conceptually categorized into centralized and decentralized ones.
Speciﬁcally, centralized algorithms require global communication between agents (through central agents or parameter servers).
While decentralized algorithms only require local communication between connected agents and are more widely applicable than centralized ones.
In both paradigms, the computation can be relatively fast with powerful computing devices; efﬁcient communication is the key to improve algorithm efﬁciency and system scalability, especially when the network bandwidth is limited.
In recent years, various communication compression techniques, such as quantization and sparsiﬁcation, have been developed to reduce communication costs.
Notably, extensive studies (Seide et al., 2014;
Alistarh et al., 2017; Bernstein et al., 2018; Stich et al., 2018;
Karimireddy et al., 2019;
Mishchenko et al., 2019; Tang et al., 2019b; Liu et al., 2020) have utilized gradient compression to signiﬁcantly boost communication efﬁciency for centralized optimization.
They enable efﬁcient large-scale optimization while maintaining comparable convergence rates and practical performance with their non-compressed counterparts.
This great success has suggested the potential and signiﬁcance of communication compression in decentralized algorithms.
While extensive attention has been paid to centralized optimization, communication compression is relatively less studied in decentralized algorithms because the algorithm design and analysis are more challenging in order to cover general communication topologies.
There are recent efforts trying to push this research direction.
For instance, DCD-SGD and ECD-SGD (Tang et al., 2018a) introduce difference compression and extrapolation compression to reduce model compression error.
(Reisizadeh et al., 2019a;b) introduce QDGD and QuanTimed-DSGD to achieve exact convergence with small stepsize.
DeepSqueeze (Tang et al., 2019a) directly compresses the local model and compensates the compression error in the next iteration.
CHOCO-SGD (Koloskova et al., 2019; 2020) presents a novel quantized gossip algorithm that reduces compression error by difference compression and preserves the model average.
Nevertheless, most existing works focus on the compression of primal-only algorithms, i.e., reduce to DGD (Nedic & Ozdaglar, 2009; Yuan et al., 2016) or P-DSGD (Lian et al., 2017).
They are unsatisfying in terms of convergence rate, stability, and the capability to handle heterogeneous data.
Part of the reason is that they inherit the drawback of DGD-type algorithms, whose convergence rate is slow in heterogeneous data scenarios where the data distributions are signiﬁcantly different from agent to agent.
In the literature of decentralized optimization, it has been proved that primal-dual algorithms can achieve faster converge rates and better support heterogeneous data (Ling et al., 2015; Shi et al., 2015; Li et al., 2019; Yuan et al., 2020).
However, it is unknown whether communication compression is feasible for primal-dual algorithms and how fast the convergence can be with compression.
In this paper, we attempt to bridge this gap by investigating the communication compression for primal-dual decentralized algorithms.
Our major contributions can be summarized as:
• We delineate two key challenges in the algorithm design for communication compression in decentralized optimization, i.e., data heterogeneity and compression error, and motivated by primal-dual algorithms, we propose a novel decentralized algorithm with compression, LEAD.
• We prove that for LEAD, a constant stepsize in the range (0 , 2 / ( µ + L )] is sufﬁcient to ensure linear convergence for strongly convex and smooth objective functions.
To the best of our knowledge, LEAD is the ﬁrst linear convergent decentralized algorithm with compression.
Moreover, LEAD provably works with unbiased compression of arbitrary precision.
• We further prove that if the stochastic gradient is used, LEAD converges linearly to the O ( σ 2 ) neighborhood of the optimum with constant stepsize.
LEAD is also able to achieve exact convergence to the optimum with diminishing stepsize.
• Extensive experiments on convex problems validate our theoretical analyses
and the empirical study on training deep neural nets shows that LEAD is applicable for nonconvex problems.
LEAD achieves state-of-art computation and communication efﬁciency in all experiments and signiﬁcantly outperforms the baselines on heterogeneous data.
Moreover, LEAD is robust to parameter settings and needs minor effort for parameter tuning.
R ELATED W ORKS
Decentralized optimization can be traced back to the work by Tsitsiklis et al.
DGD (Nedic & Ozdaglar, 2009) is the most classical decentralized algorithm.
It is intuitive and simple but converges slowly due to the diminishing stepsize that is needed to obtain the optimal solution (Yuan et al., 2016).
Its stochastic version D-PSGD (Lian et al., 2017) has been shown effective for training nonconvex deep learning models.
Algorithms based on primal-dual formulations or gradient tracking are proposed to eliminate the convergence bias in DGD-type algorithms and improve the convergence rate, such as D-ADMM (Mota et al., 2013), DLM (Ling et al., 2015), EXTRA (Shi et al., 2015), NIDS (Li et al., 2019), D 2 (Tang et al., 2018b), Exact Diffusion (Yuan et al., 2018),
OPTRA(Xu et al., 2020), DIGing (Nedic et al., 2017), GSGT (Pu & Nedi´c, 2020), etc.
Recently, communication compression is applied to decentralized settings by Tang et al.
(2018a).
It proposes two algorithms, i.e., DCD-SGD and ECD-SGD, which require compression of high accuracy and are not stable with aggressive compression.
Reisizadeh et al.
(2019a;b) introduce QDGD and QuanTimed-DSGD to achieve exact convergence with small stepsize and the convergence is slow.
DeepSqueeze (Tang et al., 2019a) compensates the compression error to the compression in the next iteration.
Motivated by the quantized average consensus algorithms, such as (Carli et al., 2010), the quantized gossip algorithm CHOCO-Gossip (Koloskova et al., 2019) converges linearly to the consensual solution.
Combining CHOCO-Gossip and D-PSGD leads to a decentralized algorithm with compression, CHOCO-SGD, which converges sublinearly under the strong convexity and gradient boundedness assumptions.
Its nonconvex variant is further analyzed in (Koloskova et al., 2020).
A new compression scheme using the modulo operation is introduced in (Lu & De Sa, 2020) for decentralized optimization.
A general algorithmic framework aiming to maintain the linear convergence of distributed optimization under compressed communication is considered in (Magn´usson et al., 2020).
It requires a contractive property that is not satisﬁed by many decentralized algorithms including the algorithm in this paper.
A LGORITHM
We ﬁrst introduce notations and deﬁnitions used in this work.
We use bold upper-case letters such as X to deﬁne matrices and bold lower-case letters such as x to deﬁne vectors.
Let 1 and 0 be vectors with all ones and zeros, respectively.
Their dimensions will be provided when necessary.
Given two matrices X , Y ∈ R n × d , we deﬁne their inner product as (cid:104) X , Y (cid:105) =
tr ( X (cid:62) Y ) and the norm as (cid:107) X (cid:107) =
(cid:112) (cid:104) X , X (cid:105) .
We further deﬁne (cid:104) X , Y (cid:105) P = tr ( X (cid:62) PY ) and (cid:107) X (cid:107) P = (cid:112) (cid:104) X , X (cid:105) P for any given symmetric positive semideﬁnite matrix P ∈ R n × n .
For simplicity, we will majorly use the matrix notation in this work.
For instance, each agent i holds an individual estimate x i ∈ R d of the global variable x ∈ R d .
Let X k and ∇ F ( X k ) be the collections of { x k i } n i =1 and {∇ f i ( x k i ) } n i = which are deﬁned below:
[Equation]
( x k 1 ) , . . . , ∇ f n ( x kn ) (cid:3) (cid:62) ∈ R n × d .
[Equation]
We use ∇ F ( X k ; ξ k ) to denote the stochastic approximation of ∇ F ( X k ) .
With these notations, the update X k +1 =
X k − η ∇ F
( X k ; ξ k ) means that x k +1 i
=
x ki − η ∇ f i ( x ki ; ξ ki ) for all i .
In this paper, we need the average of all rows in X k and ∇ F ( X k ) , so we deﬁne X k
= ( 1 (cid:62) X k ) /n
and ∇ F ( X k ) =
( 1 (cid:62) ∇ F ( X k ))
/n .
They are row vectors, and we will take a transpose if we need a column vector.
The pseudoinverse of a matrix M is denoted as M † .
The largest, i th-largest, and smallest nonzero eigenvalues of a symmetric matrix M are λ max ( M ) , λ i ( M ) , and λ min ( M ) .
Assumption 1 (Mixing matrix) .
The connected network G = {V , E} consists of a node set V = { 1 , 2 , . .
. , n } and an undirected edge set E .
The primitive symmetric doubly-stochastic matrix W = [ w ij ] ∈ R n × n encodes the network structure such that w ij = 0 if nodes i and j are not connected and cannot exchange information.
Assumption 1 implies that − 1 < λ n ( W ) ≤ λ n − 1 ( W ) ≤ · · · λ 2 ( W ) < λ 1 ( W )
= 1 and W1 =
1 (Xiao & Boyd, 2004; Shi et al., 2015).
The matrix multiplication X k +1 =
WX k describes that agent i takes a weighted sum from its neighbors and itself, i.e., x k +1 i
= (cid:80) j ∈ N i ∪{ i } w ij x kj , where N i denotes the neighbors of agent i .
3. T HE P ROPOSED A LGORITHM
The proposed algorithm LEAD to solve problem (1) is showed in Alg.
1 with matrix notations for conciseness.
We will refer to the line number in the analysis.
A complete algorithm description from the agent’s perspective can be found in Appendix A.
The motivation behind Alg. 1 is to achieve two goals: (a) consensus ( x ki − ( X k ) (cid:62) → 0 ) and (b) convergence ( ( X k ) (cid:62) → x ∗ ).
We ﬁrst discuss how goal (a) leads to goal (b) and then explain how LEAD fulﬁlls goal (a).
In essence, LEAD runs the approximate SGD globally and reduces to the exact SGD under consensus.
One key property for LEAD is 1 (cid:62) n × 1 D k = 0 , regardless of the compression error in ˆ Y k .
It holds because that for the initialization, we require D
= ( I − W
) Z for some Z ∈ R n × d , e.g., D 1 = 0 n × d , and that the update of D k ensures D k ∈ Range ( I − W ) for all k and 1 (cid:62) n × 1 ( I − W )
=
as we will explain later.
Therefore, multiplying (1 /n ) 1 (cid:62) n × 1 on both sides of
Line 7 leads to a global average view of Alg. 1:
[Equation]
which doesn’t contain the compression error.
Note that this is an approximate SGD step because, as shown in (2), the gradient ∇ F ( X k ; ξ k ) is not evaluated on a global synchronized model
X k .
However, if the solution converges to the consensus solution, i.e., x ki − ( X k ) (cid:62) → 0 , then
E ξ k [ ∇ F ( X k ; ξ k ) − ∇ f
( X k ; ξ k )]
→ 0 and (3) gradually reduces to exact SGD.
Input: Stepsize η , parameter ( α, γ ), X 0 , H 1 , D 1 =
( I − W
) Z for any Z
Output: X K or 1 /n (cid:80) ni
=1 X Ki 1: H 1 w =
WH2: X 1 = X 0 − η ∇ F
( X 0 ; ξ 0 ) 3: for k
= 1 , 2 , · · · , K − 1 do 4:
Y k
= X k − η ∇ F ( X k ; ξ k ) − η D k 5: ˆ Y k , ˆ Y kw , H k +1 , H k +1 w
=
C OMM ( Y k , H k , H kw ) 6:
D k
+
=
D k + γ 2 η
( ˆ Y k − ˆ Y kw ) 7:
X k +
=
X k − η ∇ F
( X k ; ξ k ) − η D k + 8: end for 9: procedure C OMM ( Y , H , H w ) 10:
Q = C
OMPRESS ( Y − H ) 11: ˆ Y = H + Q 12: ˆ Y w =
H w + WQ 13:
H =
(1 − α ) H + α ˆ Y 14:
H w = (1 − α ) H w + α ˆ Y w 15:
Return: ˆ Y , ˆ Y w , H , H w 16: end procedure
With the establishment of how consensus leads to convergence, the obstacle becomes how to achieve consensus under local communication and compression challenges.
It requires addressing two issues, i.e., data heterogeneity and compression error.
To deal with these issues, existing algorithms, such as DCD-SGD, ECD-SGD, QDGD, DeepSqueeze, Moniqua, and CHOCO-SGD, need a diminishing or constant but small stepsize depending on the total number of iterations.
However, these choices unavoidably cause slower convergence and bring in the difﬁculty of parameter tuning.
In contrast, LEAD takes a different way to solve these issues, as explained below.
Data heterogeneity .
It is common in distributed settings that there exists data heterogeneity among agents, especially in real-world applications where different agents collect data from different scenarios.
In other words, we generally have f i ( x ) (cid:54) =
f j ( x ) for i (cid:54) =
j .
The optimality condition of problem (1) gives 1 (cid:62) n × 1 ∇ F ( X ∗ ) = 0 , where X ∗ =
[ x ∗ , · · · , x ∗ ] is a consensual and optimal solution.
The data heterogeneity and optimality condition imply that there exist at least two agents
i and j such that ∇ f i ( x ∗ ) (cid:54) = 0 and ∇ f j ( x ∗ ) (cid:54)
= 0 .
As a result, a simple D-PSGD algorithm cannot converge to the consensual and optimal solution as X ∗ (cid:54) =
WX ∗ − η E ξ ∇ F ( X ∗ ; ξ ) even when the stochastic gradient variance is zero.
Gradient correction .
Primal-dual algorithms or gradient tracking algorithms are able to convergence much faster than DGD-type algorithms by handling the data heterogeneity issue, as introduced in
Section 2.
Speciﬁcally, LEAD is motivated by the design of primal-dual algorithm NIDS
(Li et al., 2019)
and the relation becomes clear if we consider the two-step reformulation of NIDS adopted in (Li & Yan, 2019):
[Equation] [Equation]
where X k and D k represent the primal and dual variables respectively.
The dual variable D k plays the role of gradient correction.
As k → ∞ , we expect D k → −∇ F ( X ∗ ) and X k will converge to X ∗ via the update in (5) since D k +1 corrects the nonzero gradient ∇ F ( X k ) asymptotically.
The key design of Alg. 1 is to provide compression for the auxiliary variable deﬁned as Y k =
X k − η ∇ F ( X k ) − η D k .
Such design ensures that the dual variable D k lies in Range ( I − W ) , which is essential for convergence.
Moreover, it achieves the implicit error compression as we will explain later.
To stabilize the algorithm with inexact dual update, we introduce a parameter γ to control the stepsize in the dual update.
Therefore, if we ignore the details of the compression, Alg.can be concisely written as
[Equation] [Equation]
[Equation]
where ˆ Y k represents the compression of Y k and F ( X k ; ξ k ) denote the stochastic gradients.
Nevertheless, how to compress the communication and how fast the convergence we can attain with compression error are unknown.
In the following, we propose to carefully control the compression error by difference compression and error compensation such that the inexact dual update (Line 6) and primal update (Line 7) can still guarantee the convergence as proved in Section 4.
Compression error .
Different from existing works, which typically compress the primal variable
X k or its difference, LEAD ﬁrst construct an intermediate variable Y k and apply compression to obtain its coarse representation ˆ Y k as shown in the procedure C OMM ( Y , H , H w ) :
• Compress the difference between Y and the state variable H as Q ; • Q is encoded into the low-bit representation, which enables the efﬁcient local communication step ˆ Y w
= H w + WQ .
It is the only communication step in each iteration.
• Each agent recovers its estimate ˆ Y by ˆ Y
= H + Q
and we have ˆ Y w =
W ˆ Y .
• States H and H w are updated based on ˆ Y and ˆ Y w , respectively.
We have H w = WH .
By this procedure, we expect when both Y k and H k converge to X ∗ , the compression error vanishes asymptotically due to the assumption we make for the compression operator in Assumption 2.
Remark 1.
Note that difference compression is also applied in DCD-PSGD
(Tang et al., 2018a) and CHOCO-SGD (Koloskova et al., 2019), but their state update is the simple integration of the compressed difference.
We ﬁnd this update is usually too aggressive and cause instability as showed in our experiments.
Therefore, we adopt a momentum update H =
(1 − α ) H + α ˆ Y motivated from
DIANA (Mishchenko et al., 2019), which reduces the compression error for gradient compression in centralized optimization.
Implicit error compensation.
On the other hand, even if the compression error exists, LEAD essentially compensates for the error in the inexact dual update (Line 6), making the algorithm more stable and robust.
To illustrate how it works, let E k
= ˆ Y k − Y k denote the compression error and e ki be its i -th row.
The update of D k gives
[Equation]
where − WE k indicates that agent i spreads total compression error − (cid:80) j ∈ N i ∪{ i } w ji e ki = − e ki to all agents and E k indicates that each agent compensates this error locally by adding e ki back.
This error compensation also explains why the global view in (3) doesn’t involve compression error.
Remark 2.
Note that in LEAD, the compression error is compensated into the model
X k +1 through
Line 6 and Line 7 such that the gradient computation in the next iteration is aware of the compression error.
This has some subtle but important difference from the error compensation or error feedback in (Seide et al., 2014;
Wu et al., 2018; Stich et al., 2018; Karimireddy et al., 2019; Tang et al., 2019b;
Liu et al., 2020; Tang et al., 2019a), where the error is stored in the memory and only compensated after gradient computation and before the compression.
Remark 3.
The proposed algorithm, LEAD in Alg. 1, recovers NIDS (Li et al., 2019), D 2 (Tang et al., 2018b),
Exact Diffusion (Yuan et al., 2018).
These connections are established in Appendix B.
T HEORETICAL A NALYSIS
In this section, we show the convergence rate for the proposed algorithm LEAD.
Before showing the main theorem, we make some assumptions, which are commonly used for the analysis of decentralized optimization algorithms.
All proofs are provided in Appendix E.
Assumption 2 (Unbiased and C -contracted operator) .
The compression operator Q : R d → R d is unbiased, i.e., E Q ( x ) = x , and there exists C ≥ 0 such that E (cid:107) x − Q ( x ) (cid:107) 22 ≤ C (cid:107) x (cid:107) 22 for all x ∈ R d .
Assumption 3 (Stochastic gradient) .
The stochastic gradient ∇ f i ( x ; ξ ) is unbiased
Assumption 4.
Each f i is L -smooth and µ -strongly convex with L ≥ µ > 0 ,
i
[Equation]
e., for i = 1 , 2 , . . . , n and ∀ x , y ∈ R d , we have
[Equation]
Theorem 1 (Constant stepsize) .
Let { X k , H k , D k } be the sequence generated from Alg.
and X ∗ is the optimal solution with D ∗ = −∇ F ( X ∗ ) .
Under Assumptions 1-4, for any constant stepsize η ∈ (0 , 2 / ( µ + L )] , if the compression parameters α and γ satisfy
[Equation]
with β := λ max ( I − W ) .
Then, in total expectation we have
[Equation]
) where
[Equation]
The result holds for C → 0 .
Corollary 1 (Complexity bounds) .
Deﬁne the condition numbers of the objective function and communication graph as κ f = Lµ and κ g = λ max ( I − W ) λ
+min
( I − W ) , respectively.
Under the same setting in
Theorem 1, we can choose η = 1 L , γ = min { 1 Cβκ f , 1 (1+3 C ) β } , and
α =
O
( 1 (1+ C ) κ f ) such that
[Equation]
With full-gradient (i
[Equation]
e., σ = 0 ), we obtain the following complexity bounds:
• LEAD converges to the (cid:15) -accurate solution with the iteration complexity
[Equation]
• When C =
(i
[Equation]
e., there is no compression), we obtain ρ = max { 1 − O
( 1 κ f ) , 1 − O ( 1 κ g ) } , and the iteration complexity O (cid:16) ( κ f + κ g ) log 1 (cid:15) (cid:17) .
This exactly recovers the convergence rate of NIDS (Li et al., 2019).
• When C ≤ κ
[Equation]
+
[Equation]
κ
[Equation]
κ g , the asymptotical complexity is O (cid:16) ( κ f + κ g ) log
[Equation]
, which also recovers that of NIDS (Li et al., 2019) and indicates that the compression doesn’t harm the convergence in this case.
• With C = 0 (or C ≤ κ f + κ g κ f κ g
+ κ f + κ g ) and fully connected communication graph (i.e., W = 11 (cid:62) n ), we have β = 1 and κ g = 1 .
Therefore, we obtain ρ = 1 − O ( 1 κ f ) and the complexity bound O
( κ f log 1 (cid:15) ) .
This recovers the convergence rate of gradient descent (Nesterov, 2013).
Remark 4.
Under the setting in Theorem 1, LEAD converges linearly to the O ( σ 2 ) neighborhood of the optimum and converges linearly exactly to the optimum if full gradient is used, e.g., σ = 0 .
The linear convergence of LEAD holds when η < 2 /L , but we omit the proof.
Remark 5 (Arbitrary compression precision) .
Pick any η ∈ (0 , 2 / ( µ + L )] , based on the compression-related constant C and the network-related constant β , we can select γ and α in certain ranges to achieve the convergence.
It suggests that LEAD supports unbiased compression with arbitrary precision, i.e., any C > 0 .
Corollary 2 (Consensus error) .
Under the same setting in Theorem 1 , let x k = 1 n (cid:80) ni
=1 x ki be the averaged model and H
= H 1 , then all agents achieve consensus at the rate
[Equation]
where ρ is deﬁned as in Corollary 1 with appropriate parameter settings.
Theorem 2 (Diminishing stepsize) .
Let { X k , H k , D k } be the sequence generated from Alg.
and
X ∗ is the optimal solution with D ∗ = −∇ F ( X ∗ ) .
Under Assumptions 1-4, if η k =
[Table]
5 k +2 and γ k = θ 4 η k , by taking α k = Cβγ k 2(1+ C ) , in total expectation we have
[Equation]
[Equation]
where θ 1 , θ 2 , θ 3 , θ 4 and θ 5 are constants deﬁned in the proof.
The complexity bound for arriving at the (cid:15) -accurate solution is O ( 1 (cid:15) ) .
Remark 6.
Compared with CHOCO-SGD, LEAD requires unbiased compression and the convergence under biased compression is not investigated yet.
The analysis of CHOCO-SGD relies on the bounded gradient assumptions, i.e., (cid:107)∇ f i ( x ) (cid:107) 2 ≤ G , which is restrictive because it conﬂicts with the strong convexity while LEAD doesn’t need this assumption.
Moreover, in the theorem of
CHOCO-SGD, it requires a speciﬁc point set of γ while LEAD only requires γ to be within a rather large range.
This may explain the advantages of LEAD over CHOCO-SGD in terms of robustness to parameter setting.
N UMERICAL E XPERIMENT
We consider three machine learning problems – (cid:96) 2 -regularized linear regression, logistic regression, and deep neural network.
The proposed LEAD is compared with QDGD (Reisizadeh et al., 2019a),
DeepSqueeze (Tang et al., 2019a), CHOCO-SGD (Koloskova et al., 2019), and two non-compressed algorithms DGD (Yuan et al., 2016) and NIDS (Li et al., 2019).
Setup.
We consider eight machines connected in a ring topology network.
Each agent can only exchange information with its two 1-hop neighbors.
The mixing weight is simply set as 1 / 3 .
For compression, we use the unbiased b -bits quantization method with ∞ -norm
[Equation]
where · is the Hadamard product,
| x | is the elementwise absolute value of x , and u is a random vector uniformly distributed in [0 , 1] d .
Only sign ( x ) , norm (cid:107) x (cid:107) ∞ , and integers in the bracket need to be transmitted.
Note that this quantization method is similar to the quantization used in QSGD (Alistarh et al., 2017) and CHOCO-SGD (Koloskova et al., 2019), but we use the ∞ -norm scaling instead of the 2 -norm.
This small change brings signiﬁcant improvement on compression precision as justiﬁed both theoretically and empirically in Appendix C.
In this section, we choose 2 -bit quantization and quantize the data blockwise (block size = 512 ).
For all experiments, we tune the stepsize η from { 0 . 01 , 0 . 05 , 0 . 1 , 0 . 5 } .
For QDGD, CHOCO-SGD and Deepsqueeze, γ is tuned from { 0 . 01 , 0 . 1 , 0 . 2 , 0 . 4 , 0 . 6 , 0 . 8 , 1 . 0 } .
Note that different notations are used in their original papers.
Here we uniformly denote the stepsize as η and the additional parameter in these algorithms as γ for simplicity.
For LEAD, we simply ﬁx α = 0 .
5 and γ = 1 .
for all experiments since we ﬁnd LEAD is robust to parameter settings as we validate in the parameter sensitivity analysis in Appendix D.1.
This indicates the minor effort needed for tuning LEAD.
Detailed parameter settings for all experiments are summarized in Appendix D.3.
Linear regression.
We consider the problem: f ( x ) = (cid:80) ni
=1 ( (cid:107)
A i x − b i (cid:107)
+ λ (cid:107) x (cid:107) 2 ) .
Data matrices
A i ∈ R 200 × 200 and the true solution x (cid:48) is randomly synthesized.
The values b i are generated by adding Gaussian noise to A i x (cid:48) .
We let λ = 0 .
1 and the optimal solution of the linear regression problem be x ∗ .
We use full-batch gradient to exclude the impact of gradient variance.
The performance is showed in Fig.
The distance to x ∗ in Fig.
1a and the consensus error in Fig.
1c verify that LEAD converges exponentially to the optimal consensual solution.
It signiﬁcantly outperforms most baselines and matches NIDS well under the same number of iterations.
Fig.
1b demonstrates the beneﬁt of compression when considering the communication bits.
Fig.
1d shows that the compression error vanishes for both LEAD and CHOCO-SGD while the compression error is pretty large for QDGD and DeepSqueeze because they directly compress the local models
[Figure]
Logistic regression.
We further consider a logistic regression problem on the MNIST dataset.
The regularization parameter is 10 − 4 .
We consider both homogeneous and heterogeneous data settings.
In the homogeneous setting
[Figure]
the data samples are randomly shufﬂed before being uniformly partitioned among all agents such that the data distribution from each agent is very similar.
In the heterogeneous setting, the samples are ﬁrst sorted by their labels and then partitioned among agents.
Due to the space limit
[Figure]
we mainly present the results in heterogeneous setting here and defer the homogeneous setting to Appendix D.2.
The results using full-batch gradient and mini-batch gradient (the mini-batch size is 512 for each agent) are showed in Fig.
2 and Fig.
3 respectively
and both settings shows the faster convergence and higher precision of LEAD
[Figure]
[Figure] [Figure]
Neural network.
We empirically study the performance of LEAD in optimizing deep neural network by training AlexNet ( 240 MB) on CIFAR10 dataset.
The mini-batch size is 64 for each agents.
Both the homogeneous and heterogeneous case are showed in Fig.
In the homogeneous case,
CHOCO-SGD, DeepSqueeze and LEAD perform similarly and outperform the non-compressed variants in terms of communication efﬁciency, but CHOCO-SGD and DeepSqueeze need more efforts for parameter tuning because their convergence is sensitive to the setting of γ .
In the heterogeneous cases, LEAD achieves the fastest and most stable convergence.
Note that in this setting, sufﬁcient information exchange is more important for convergence because models from different agents are moving to signiﬁcantly diverse directions.
In such case, DGD only converges with smaller stepsize and its communication compressed variants, including QDGD, DeepSqueeze and
CHOCO-SGD, diverge in all parameter settings we try.
In summary, our experiments verify our theoretical analysis and show that LEAD is able to handle data heterogeneity very well.
Furthermore, the performance of LEAD is robust to parameter settings and needs less effort for parameter tuning, which is critical in real-world applications.
C ONCLUSION
In this paper, we investigate the communication compression in decentralized optimization.
Motivated by primal-dual algorithms, a novel decentralized algorithm with compression, LEAD, is proposed to achieve faster convergence rate and to better handle heterogeneous data while enjoying the beneﬁt of efﬁcient communication.
The nontrivial analyses on the coupled dynamics of inexact primal and dual updates as well as compression error establish the linear convergence of
LEAD when full gradient is used and the linear convergence to the O ( σ 2 ) neighborhood of the optimum when stochastic gradient is used.
Extensive experiments validate the theoretical analysis and demonstrate the state-of-the-art efﬁciency and robustness of LEAD.
LEAD is also applicable to non-convex problems as empirically veriﬁed in the neural network experiments
but we leave the non-convex analysis as the future work.
A CKNOWLEDGEMENTS
Xiaorui Liu and Dr. Jiliang Tang are supported by the National Science Foundation (NSF) under grant numbers CNS-1815636, IIS-1928278, IIS-1714741, IIS-1845081, IIS- and IIS1955285.
Yao Li and Dr. Ming Yan are supported by NSF grant DMS-2012439 and Facebook
Faculty Research Award (Systems for ML).
Dr. Rongrong Wang is supported by NSF grant CCF1909523.
Contents of Appendix A LEAD in
agent’s
perspective
B Connections with exiting works
C Compression method
C.1 p-norm b-bits quantization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
C.2 Compression error . .
D Experiments
D.1 Parameter sensitivity . .
D.2 Experiments in homogeneous setting . . . . . . . . . . . . . . . . . . . . . . . . .
D.3 Parameter settings . .
E Proofs of the theorems
E.1 Illustrative ﬂow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
E.2 Two central Lemmas . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
E.3 Proof of Lemma 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
E.4 Proof of Lemma 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
E.5 Proof of Theorem 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
E.6 Proof of Theorem 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
A LEAD IN AGENT ’ S PERSPECTIVE
In the main paper, we described the algorithm with matrix notations for concision.
Here we further provide a complete algorithm description from the agents’ perspective.
input: stepsize η , γ
∀
} output: x Ki , ∀ i ∈ { 1 , 2 , . . . , n or (cid:80) ni
=1 x Ki n 1: for each agent i ∈ { 1 , 2 , . . . , n } do 2: d 1 i
=
z i − (cid:80) j ∈ N i ∪{ i } w ij z j 3: ( h w ) 1 i = (cid:80) j ∈ N i ∪{ i } w ij ( h w ) 1 j 4: x 1 i = x
i − η ∇ f i ( x 0 i ; ξ 0 i ) 5: end for 6: for k = 1 , 2 , . . .
, K − 1 do parallel for all agents i ∈ { 1 , 2 , . . . , n } 7: compute ∇ f i ( x k i ; ξ k i ) (cid:66)
Gradient computation 8: y ki
= x ki − η ∇ f i ( x ki ; ξ ki ) − η d ki 9: q ki =
Compress ( y ki − h ki ) (cid:66) Compression 10: ˆ
y ki = h ki + q ki 11: for neighbors j ∈ N i do 12:
Send q ki and receive q kj (cid:66) Communication 13: end for 14: (ˆ y w ) ki
= ( h w ) ki
+ (cid:80) j ∈ N i ∪{ i } w ij q kj 15: h k +1 i
=
(1 − α )
h
ki + α ˆ y ki 16: ( h w ) k +1 i
=
(1 − α )
( h w ) ki + α (ˆ y w ) ki 17:
d k +1 i
=
d ki + γ
[Equation] [Equation]
Model update 19: end for
B C ONNECTIONS WITH EXITING WORKS
The non-compressed variant of LEAD in Alg.
1 recovers NIDS
(Li et al., 2019), D 2 (Tang et al., 2018b) and Exact Diffusion (Yuan et al., 2018) as shown in Proposition 1.
In Corollary 3, we show that the convergence rate of LEAD exactly recovers the rate of NIDS when C = 0 , γ = 1 and σ = 0 .
Proposition 1 (Connection to NIDS, D 2 and Exact Diffusion) .
When there is no communication compression (i.e., ˆ Y k
= Y k ) and γ = 1 , Alg.
1 recovers D 2 :
[Equation] [Equation]
Furthermore, if the stochastic estimator of the gradient ∇ F ( X k ; ξ k ) is replaced by the full gradient, it recovers NIDS and Exact Diffusion with speciﬁc settings.
Corollary 3 (Consistency with NIDS) .
When C =
(no communication compression), γ = 1 and σ = 0 (full gradient), LEAD has the convergence consistent with NIDS with η ∈ (0 , 2 / ( µ + L )] :
[Equation]
See the proof in E.5.
Proof of Proposition 1.
Let γ =
and ˆ Y k
= Y k .
Combing Lines 4 and 6 of Alg.
1 gives
[Equation]
Based on Line 7, we can represent η D k from the previous iteration as
[Equation] [Equation]
Eliminating both D k and D k +1 by substituting (17)-(18) into Line 7, we obtain
[Equation]
which is exactly D 2 .
It also recovers Exact Diffusion with A
= I + Wand M = η I in Eq.
(97) of (Yuan et al., 2018).
C C OMPRESSION METHOD C. P - NORM B - BITS QUANTIZATION
Theorem 3 (p-norm b-bit quantization) .
Let us deﬁne the quantization operator as
[Equation]
where · is the Hadamard product,
| x | is the elementwise absolute value and u is a random dither vector uniformly distributed in [0 , 1] d .
Q p ( x ) is unbiased, i.e., E Q p ( x ) = x , and the compression variance is upper bounded by
[Equation]
[Equation]
which suggests that ∞ -norm provides the smallest upper bound for the compression variance due to (cid:107) x (cid:107) p ≤ (cid:107) x (cid:107) q ,
∀ x if 1 ≤ q ≤ p ≤ ∞ .
Remark 7.
For the compressor deﬁned in (20), we have the following the compression constant
[Equation]
Proof.
Let denote v =
(cid:107) x (cid:107) p sign( x )2 − ( b − 1) , s = 2 b − 1 | x | (cid:107) x
[Equation]
p , s 1 = (cid:106)
b − 1 | x | (cid:107) x (cid:107) p (cid:107) and s 2 = (cid:108) 2 b − 1 | x | (cid:107) x (cid:107) p (cid:109) .
We can rewrite x as x = s · v .
For any coordinate i such that s i =
( s 1 )
i , we have Q p ( x i ) =
( s 1 )
i v i with probability 1 .
Hence E
Q p ( x ) i = s i v i
= x i and
E ( x i − Q p ( x ) i )
= ( x i − s i v i ) 2 = 0 .
For any coordinate i such that s i (cid:54) =
( s 1 )
i , we have ( s 2 )
i − ( s 1 ) i = 1 and Q p ( x ) i satisﬁes
[Equation]
Thus
[Equation]
we derive
[Equation]
and
[Equation]
Considering both cases
[Equation]
we have E Q ( x ) = x and
[Equation]
x − Q p
[Equation]
x )
[Equation]
=
[Equation] [Equation]
C. C OMPRESSION ERROR
[Figure]
[Figure]
To verify Theorem 3, we compare the compression error of the quantization method deﬁned in (20) with different norms ( p = 1 , 2 , 3 , . .
. , 6 , ∞ ).
Speciﬁcally, we uniformly generate 100 random vectors in R 10000 and compute the average compression error.
The result shown in Figure 5 veriﬁes our proof in Theorem 3 that the compression error decreases when p increases.
This suggests that ∞ -norm provides the best compression precision under the same bit constraint.
Under similar setting, we also compare the compression error with other popular compression methods, such as top-k and random-k sparsiﬁcation.
The x-axes represents the average bits needed to represent each element of the vector.
The result is showed in Fig.
Note that intuitively top-k methods should perform better than random-k method, but the top-k method needs extra bits to transmitted the index while random-k method can avoid this by using the same random seed.
Therefore, top-k method doesn’t outperform random-k too much under the same communication budget.
The result in Fig.
6 suggests that ∞ -norm b-bits quantization provides signiﬁcantly better compression precision than others under the same bit constraint.
[Figure]
[Figure]
D E XPERIMENTS D. P ARAMETER SENSITIVITY
In the linear regression problem
[Figure]
the convergence of LEAD under different parameter settings of α and γ are tested.
The result showed in Figure 7 indicates that LEAD performs well in most settings and is robust to the parameter setting.
Therefore, in this paper
[Figure]
we simply set α = 0 .
5 and γ = 1 .for
LEAD in all experiment
[Figure]
which indicates the minor effort needed for parameter tuning
[Figure]
D. E XPERIMENTS IN HOMOGENEOUS SETTING
The experiments on logistic regression problem in homogeneous case are showed in Fig.
8 and
Fig.
It shows that DeepSqueeze, CHOCO-SGD and LEAD converges similarly while Deep-
Squeeze and CHOCO-SGD require to tune a smaller γ for convergence as showed in the parameter setting in Section D.3.
Generally, a smaller γ decreases the model propagation between agents since γ changes the effective mixing matrix and this may cause slower convergence.
However, in the setting where data from different agents are very similar, the models move to close directions such that the convergence is not affected too much
[Figure] [Figure]
D. P ARAMETER SETTINGS
The best parameter settings we search for all algorithms and experiments are summarized in Tables 1– 4.
QDGD and DeepSqueeze are more sensitive to γ and CHOCO-SGD is slight more robust.
LEAD is most robust to parameter settings and it works well for the setting α = 0 .
5 and γ = 1 .
in all experiments in this paper.
[Table] [Table]
η γ α
DGD
[Table]
Heterogeneous
[Table] [Table]
η
[Table] [Table]
E P ROOFS OF THE THEOREMS E. I LLUSTRATIVE FLOW
The following ﬂow graph depicts the relation between iterative variables and clariﬁes the range of conditional expectation.
{G k } ∞ k =0 and {F k } ∞ k =0 are two σ − algebras generated by the gradient sampling and the stochastic compression respectively.
They satisfy
[Equation]
k
F
[Equation]
[Table]
F
[Table]
F
[Table]
The solid and dashed arrows in the top ﬂow illustrate the dynamics of the algorithm, while in the bottom, the arrows stand for the relation between successive F - σ -algebras.
The downward arrows determine the range of F - σ -algebras.
E.g., up to E k , all random variables are in F k − 1 and up to ∇ F ( X k ; ξ k ) , all random variables are in G k − 1 with G k − 1 ⊂ F k − 1 .
Throughout the appendix, without speciﬁcation, E is the expectation conditioned on the corresponding stochastic estimators given the context.
E. T WO CENTRAL L EMMAS
Lemma 1 (Fundamental equality) .
Let X ∗ be the optimal solution, D ∗ := −∇ F ( X ∗ ) and E k denote the compression error in the k th iteration, that is E k
=
Q k − ( Y k − H k )
=
ˆ Y k − Y k .
From Alg. 1, we have
[Equation]
, where M := 2( I − W ) † − γ I and γ < 2 /λ max ( I − W ) ensures the positive deﬁniteness of M over range ( I − W ) .
Lemma 2 (State inequality) .
Let the same assumptions in Lemma 1 hold.
From Alg. 1, if we take the expectation over the compression operator conditioned on the k -th iteration, we have
E (cid:107)
[Equation]
E. P ROOF OF L EMMA
Before proving Lemma
[Equation]
we let E k
= ˆ Y k − Y k and introduce the following three Lemmas.
Lemma 3.
Let X ∗ be the consensus solution.
Then, from Line 4-7 of Alg. 1, we obtain
[Equation]
Proof.
From the iterations in Alg.
[Equation]
we have
[Equation]
from Line
[Equation] [Equation]
where the fourth equality holds due to ( I − W ) X ∗ = 0 and the last equality comes from Line 7 of
Alg. 1.
Rewriting this equality, and we obtain (22).
Lemma 4.
Let D ∗ = −∇ F ( X ∗ ) ∈ span { I − W } , we have
[Equation]
where M = 2( I − W ) † − γ I and γ < 2 /λ max ( I − W ) ensures the positive deﬁniteness of M over span { I − W } .
Proof.
Since D k +1 ∈ span { I − W } for any k , we have
[Equation]
Similarly, we have
[Equation]
To make sure that M is positive deﬁnite over span { I − W } , we need γ < 2 /λ max ( I − W ) .
Lemma 5.
Taking the expectation conditioned on the compression in the k th iteration
[Equation]
we have
[Equation]
Proof.
The proof is straightforward and omitted here
[Equation]
Proof of Lemma 1.
From Alg. 1, we have
[Equation]
Line
[Equation]
( from Line 7 )
[Equation]
∗
[Equation]
Then we consider the terms on the right hand side of (25) separately.
Using 2 (cid:104) A − B , B − C (cid:105)
=
(cid:107)
A − C (cid:107) 2 − (cid:107) B − C
(cid:107) 2 − (cid:107)
A − B (cid:107) 2 , we have 2 (cid:104)
X
k +1 − X ∗ , X k − X k +1 (cid:105)
=2 (cid:104)
X ∗ − X k
+1 ,
X k +1 − X k (cid:105) =
(cid:107)
X k − X ∗ (cid:107) 2 − (cid:107) X
k
+
− X k (cid:107) 2 − (cid:107) X
k +1 − X ∗ (cid:107) 2 .
Using
[Equation]
2 , we have
[Equation] [Equation]
Combining (25), (26), (27)
[Equation]
and (23), we obtain
[Equation]
where the last equality holds because
[Equation]
Thus, we reformulate it as
[Equation]
which completes the proof
[Equation]
E. P ROOF OF L EMMA
Proof of Lemma 2.
From Alg. 1, we take the expectation conditioned on k th compression and obtain
[Equation]
Line
[Equation]
In the second equality
[Equation]
we used the unbiasedness of the compression, i.e., E E k = 0 .
The last equality holds because of
[Equation]
In addition, by taking the conditional expectation on the compression
[Equation]
we have
[Equation]
Combing the above two equations (28) and (29) together
[Equation]
we have
[Equation]
which completes the proof
[Equation]
E. P ROOF OF T HEOREM
Proof of Theorem 1.
Combining Lemmas 1, 2, and 5, we have the expectation conditioned on the compression satisfying
[Equation]
31) where a 1 is a non-negative number to be determined.
Then we deal with the three terms on the right hand side separately.
We want the terms B and C to be nonpositive.
First, we consider B .
Note that
D k ∈ Range ( I − W ) .
If we want B ≤ 0 , then, we need 1 − 2 a 1 α > 0 , i.e., a 1 α < 1 / 2 .
Therefore we have
B = − (1 − 2 a 1 α
[Equation]
where λ n − 1 ( M ) > 0 is the second smallest eigenvalue of M .
It means that we also need
[Equation]
which is equivalent to
[Equation]
Then we look at C .
We have
[Equation]
Because we have 1 − a 1 α > 1 /
[Equation]
so we need
[Equation]
That is
[Equation]
Next, we look at A .
Firstly, by the bounded variance assumption, we have the expectation conditioned on the gradient sampling in k th iteration satisfying
[Equation]
Then with the smoothness and strong convexity from Assumptions
[Equation]
we have the co-coercivity of ∇ g i ( x ) with g i ( x ) :
=
f i ( x ) − u 2 (cid:107) x
(cid:107)
[Equation]
which gives
[Equation]
When η ≤ 2 / ( µ + L )
[Equation]
we have
[Equation]
Therefore
[Equation]
we obtain
[Equation]
Conditioned on the k the iteration, (i
[Equation]
e., conditioned on the gradient sampling in k th iteration), the inequality (31) becomes
[Equation]
if the step size satisﬁes η ≤ 2 µ + L .
Rewriting (37), we have
[Equation]
and thus
[Equation]
With the deﬁnition of L k in (12)
[Equation]
we have
[Equation]
40) with
[Equation]
where
[Equation]
Recall all the conditions on the parameters a 1 , α , and γ to make sure that ρ < 1 :
[Equation]
In the following, we show that there exist parameters that satisfy these conditions.
Since we can choose any a 1 , we let
[Equation]
such that a
[Equation]
Cβγ
[Equation]
Then we have
[Equation] [Equation]
Conditions (43) and (44) show a 1 α ∈
[Equation]
→ [0 , 2] , if C = 0 or γ → 0 .
Hence in order to make (41) and (42) satisﬁed, it’s sufﬁcient to make
[Equation] [Equation]
where we use λ n − 1 ( M )
=
[Equation]
max
[Equation]
When C > 0 , the condition (45) is equivalent to
[Equation]
min
[Equation]
The ﬁrst term can be simpliﬁed using
[Equation]
β due to √ 1 − x ≤ 1 − x 2 when x ∈ (0 , 1) .
Therefore, for a given stepsize η , if we choose γ
[Equation]
0 , min
[Equation]
and
[Equation]
∈
[Equation]
then, all conditions (41)-(44) hold.
Note that γ < 2 (3 C +1) β implies γ < 2 β , which ensures the positive deﬁniteness of M over span { I −
W } in Lemma 4.
Note that η ≤
[Equation]
+ L ensures
[Equation]
So, we can simplify the bound for α as
[Equation]
Lastly, taking the total expectation on both sides of (40) and using tower property, we complete the proof for C > 0 .
Proof of Corollary 1.
Let’s ﬁrst deﬁne κ f =
Lµ
and κ g = λ max ( I − W ) λ
+min
( I − W )
=
λ max ( I − W ) λ max (( I −
W ) † ) .
We can choose the stepsize η = 1 L such that the upper bound of γ is
[Equation]
due to x (2 − x ) 2 − x (2 − x ) ≥ x 2 − x ≥ x when x ∈ (0 , 1) .
Hence we can take γ = min { 1 (3 C +1) β , 1 κ f Cβ } .
The bound of α is
[Equation]
When γ is chosen as
[Equation]
Cβ , pick
[Equation]
When
[Equation]
≤
[Equation]
Cβ , the upper bound of α is
[Equation]
In this case, we pick α =
[Equation]
Note α
= O (cid:16) 1 (1+ C )
κ f (cid:17) since 6 C +1 12 C +3 is lower bounded by 13 .
Hence in both cases (Eq. (48) and
Eq.
α = O (cid:16) 1 (1+ C ) κ f (cid:17) , and the third term of ρ is upper bounded by
[Equation]
−
[Equation]
min
[Equation]
C +
[Equation]
12 C + 3 ,
[Equation]
7 C +
[Equation]
4(1 + C )(3 C + 1) (cid:27)
In two cases of γ , the second term of ρ becomes
[Equation]
Before analysing the ﬁrst term of ρ , we look at a 1 α in two cases of γ .
When γ = 1 κ f Cβ , a
[Equation]
When γ = 1 (3 C +1) β , a 1 α = min (cid:26)
[Equation] [Equation]
) , κ
[Equation]
(cid:27) ≤κ f .
In both cases, a 1 α ≤ 1 κ f .
Therefore, the ﬁrst term of ρ becomes
[Equation]
To summarize, we have ρ ≤ 1 − min (cid:26)
[Equation]
min
[Equation]
C
[Equation]
12 C
[Equation] [Equation]
(cid:27) 7 C +4(
[Equation]
C
[Equation]
(3 C +
[Equation]
(cid:27) and therefore
[Equation]
With full-gradient (i.e., σ = 0 ), we get (cid:15) − accuracy solution with the total number of iterations
[Equation]
When C = 0 , i.e., there is no compression, the iteration complexity recovers that of NIDS, (cid:101) O
( κ f + κ g ) .
When C ≤ κ f + κ g κ f κ g + κ f + κ g , the complexity is improved to that of NIDS, i.e., the compression doesn’t harm the convergence in terms of the order of the coefﬁcients.
Proof of Corollary 2.
Note that ( x k ) (cid:62) =
X k and 1 n × 1 X ∗ =
X ∗ , then n (cid:88)
i
[Equation]
The last inequality holds because we have a 1 α ≤ 1 / 2 .
Proof of Corollary 3.
From the proof of Theorem 1, when C = 0 , we can set γ = 1 , α = 1 , and a 1 = 0 .
Plug those values into ρ , and we obtain the convergence rate for NIDS.
E. P ROOF OF T HEOREM
Proof of Theorem 2.
In order to get exact convergence, we pick diminishing step-size, set α = Cβγ 2(1+ C ) , a 1 α = 2 Cβγ k Cβγ k +2 , θ 1 = 1 2 λ max (( I − W ) † ) and θ 2 =
Cβ 2(1+ C ) , then ρ k = max (cid:26)
[Equation]
(cid:27)
If we further pick diminishing η k and γ k such that µη k (2 − µη k ) − a 1 α ≥ a 1 α, then µη k (2 − µη
[Equation]
−
[Equation]
Notice that Cβγ ≤ 23 since (3 C + 1) − (cid:112) (3 C + 1) 2 − 4 C is increasing in C > 0 with limit 23 at ∞ .
In this case we only need,
[Equation]
And
[Equation]
− θ 1 γ k , 1 − θ 2 γ k } ≤ 1 − θ 3 γ k if θ 3 =
min
θ 1 , θ 2 } and note
that θ 2 ≤ Cβ.
We deﬁne
[Equation]
Hence
[Equation]
From a 1 α ≤ µη k (2 − µη k ), we get
[Equation]
If we pick γ k = θ 4 η k , then it’s sufﬁcient to let
[Equation]
Hence if θ 4 < µ Cβ and let η ∗ = 2( µ − Cβθ 4 ) µ, then η k = γ k θ 4 ∈ (0 , η ∗ ) guarantees the above discussion and
[Equation]
So far all restrictions for η k are η k ≤ min (cid:26)
[Equation]
, η ∗
[Equation]
and
[Equation]
Let θ 5 = min (cid:26)
[Equation] [Equation]
[Equation] [Equation]
[Equation] [Equation]
k =
[Equation]
and D = max (cid:110)
[Equation]
(cid:111)
, we claim that if we pick B = θ 3 θ 4 2 and some A , by setting η k = 2 θ 3 θ 4 k +2 A , we get
E L k ≤
D
Bk + A.
Induction:
When k = 0 , it’s obvious.
Suppose previous k inequalities hold.
Then
[Equation]
θ
[Equation]
θ
[Equation] [Equation]
Multiply M :=
( θ 3 θ 4 k + θ 3 θ 4 + 2 A )( θ 3 θ 4 k + 2 A )(2 D ) − 1 on both sides, we get
[Equation] [Equation]
+
[Equation]
D
[Equation]
≤ θ 3 θ 4 k + 2 A.
Hence
[Equation]
This induction holds for any A such that η k is feasible, i.e.
[Equation]
Here we summarize the deﬁnition of constant numbers:
[Equation]
Therefore, let A = 1 θ 5 and η k =
[Equation]
[Equation]
4 θ 5 k +2 , we get
[Equation]
Since 1 − a 1 α k ≥ 1 / 2 , we complete the proof.