Generalization of Decentralized Gradient Descent with Separable Data
Abstract
Decentralized learning offers privacy and communication efﬁciency when data are naturally distributed among agents communicating over an underlying graph.
Motivated by overparameterized learning settings, in which models are trained to zero training loss, we study algorithmic and generalization properties of decentralized learning with gradient descent on separable data.
Speciﬁcally, for decentralized gradient descent (DGD) and a variety of loss functions that asymptote to zero at inﬁnity (including exponential and logistic losses), we derive novel ﬁnite-time generalization bounds.
This complements a long line of recent work that studies the generalization performance and the implicit bias of gradient descent over separable data, but has thus far been limited to centralized learning scenarios.
Notably, our generalization bounds match in order their centralized counterparts.
Critical behind this, and of independent interest, is establishing novel bounds on the training loss and the rate-of-consensus of DGD for a class of self-bounded losses.
Finally, we conduct numerical experiments which corroborate our theoretical results.
1. Introduction
Machine learning tasks often revolve around inference from data using empirical risk minimization (ERM):
[Equation]
Here f : R d × R d (cid:48)
→ R is a loss function and x i :=
y i a
i , where ( a i , y i ) ni
=1 iid ∼ D represent features and labels, sampled from a distribution D .
In large scale machine learning, due to privacy concerns and communication constraints, data points are often distributed on a set of local computing agents.
Decentralized learning methods aim at minimizing the global loss function (1) while agents communicate their parameters on an underlying connected graph.
The most ubiquitous of these algorithms is Decentralized Gradient Descent (DGD).
Here the (cid:96) th agent runs a step of gradient descent followed by an averaging step in which every agent replaces its parameter with the average of its neighbors [18]:
[Equation]
The superscripts signify the iteration number and A (cid:96)k refers to the averaging weights used by agent (cid:96) for the parameter of agent k .
The global loss ˆ F is the average of local loss functions
ˆ F (cid:96) , (cid:96) ≤ N , where each ˆ F (cid:96) is formed as the average empirical risk evaluated on the local training dataset S (cid:96) of the (cid:96) th agent:
[Equation]
(3) where n (cid:96) denotes the dataset size of agent (cid:96) .
Convergence properties of the train loss ˆ F ( · ) in DGD have been studied extensively in literature, e.g., [13, 16–18, 31].
The bulk of these studies build upon classical optimization theory [20] suited for studying the train loss per iteration.
In particular, it is well-stablished in the literature that DGD converges at the rate 1 T (cid:80)
Tt =
1 ˆ F ( ¯ w ( t ) )
− ˆ F (cid:63) =
O ( 1 √ T ) for smooth convex functions [16].
Here ¯ w ( t ) is the average of local parameters w ( t ) (cid:96) .
Our results in
Sections 2.1-2.2 show a rate of ˆ F ( ¯ w ( T ) )
=
O ( (log T ) 2 T ) and (cid:107) W ( T ) − ¯ W ( T ) (cid:107) 2 F
= O ( (log T ) 4 T) for the training loss and consensus error of DGD over separable data with “exponentially tailed” losses.
The study of generalization performance of DGD algorithms in the literature is mostly limited to empirical observations e.g., [8, 9, 29], making the theory behind test error performance largely unexplored.
Our ﬁrst goal in this paper is to complement prior general results on the convergence of training loss in DGD by considering speciﬁc, but commonly encountered, settings in ERM over separable data.
This includes the analysis of non-smooth objectives such as the exponential loss, analysis of logistic regression in the interpolating regime where the optimum is achieved at inﬁnity, and analysis of objectives satisfying the PL condition.
Our second goal is to study, for the ﬁrst time, convergence rates of the DGD test loss F ( ¯ w ( t ) ) :
=
E x ∼D [ f ( ¯ w ( t ) , x )] .
Finally, we leverage recent advances in the study of centralized learning with separable data to design fast algorithms for decentralized learning.
Contributions.
Our contributions are summarized as follows,
• In Section 2.1 we derive convergence rates for the training and test loss of the decentralized gradient descent algorithm with separable data.
Our results hold for convex losses satisfying realizability and self-boundedness, as well as, convex losses satisfying self-boundedness and the PL condition.
In Section 2.2, we prove that under additional self-boundedness assumptions on the Hessian and gradient, which holds for exponentially tailed losses, the test loss bound can be improved to approximately match the test loss bounds of centralized GD.
In Appendix A, we extend our results to losses satisfying the PL condition.
Our numerical experiments in Appendix H verify our results.
• In Appendix G, we propose two algorithms for speeding up the convergence of decentralized learning with separable data.
Notably, numerical experiments demonstrate that our proposed algorithms signiﬁcantly improve the train error and test error of decentralized logistic regression.
Further related works
Decentralized learning.
Over the last few years there have been numerous research works which consider the convergence of ﬁrst order methods for decentralized learning; an incomplete list includes [2, 8, 10, 11, 13, 16, 18, 22, 27, 28, 30, 31].
The study of generalization of DGD was mostly limited to numerical experiments.
While this paper was being written, we were notiﬁed of the recent work [26] which studies the generalization bounds of decentralized methods for Lipschitz convex losses.
However, we consider the setting of separable data and exponentially tailed losses and show that in our settings the train/test loss of DGD is closely similar to centralized GD.
Compared to this work, we also propose improved algorithms for learning with separable data.
We highlight that our rates on the training loss are comparable to [10, Theorem 2].
While [10] also derives convergence of DGD train loss on separable data, their analysis is valid only for bounded optimizers.
In contrast, we derive training loss bounds which are true for the case of unbounded optimizers as is the case for logistic regression over separable data.
Implicit bias of GD.
For centralized settings, a line of recent works [5, 6, 15, 23–25] studies the parameter convergence, as well as training and test loss convergence, of gradient descent on separable data, showing that for (a class of) monotonic losses the solution to ERM and the maxmargin solution are the same in direction.
, i.e., (cid:107) ˆ w ( t ) − ˆ w MM (cid:107) → 0 .
Here ˆ w ( t ) :
= w ( t ) / (cid:107) w ( t ) (cid:107) and ˆ w MM :=
w MM / (cid:107) w MM (cid:107) , where the vector w MM is the solution to the hard-margin support vector machine problem.
Notably, [5, 25] characterized the rate of convergence for margin gap to be (cid:107) ˆ w ( T ) − ˆ w MM (cid:107) =
O (1 / log( T )) and for the training loss to be ˆ F ( w ( T ) )
=
O ( 1 ηT ) .
Recently,
Shamir [24] and Schliserman and Koren [23] showed that the test loss of GD for logistic regression on linearly separable data satisﬁes F ( w ( T ) ) =
˜ O ( 1 ηT + 1 n ) signifying that overﬁtting does not happen during the iterates of GD.
In Section 2.2 (Remark 7), we show that the test loss of DGD with logistic regression on linearly separable data satisﬁes E [ F ( ¯ w ( T ) )]
=
˜ O ( 1 ηT + 1 n + η 2 ) .
This signiﬁes that, under the data separability assumption, the rate of decay for test loss is essentially identical for centralized and decentralized scenarios.
2. Main results
Throughout the paper we make the following standard assumption on the mixing matrix
A =
[ A ij ] N × N corresponding to the underlying connected network.
Assumption 1 (Mixing matrix)
The mixing matrix A ∈ R N × N is symmetric, doubly stochastic with bounded spectrum i.e., | λ i ( A ) | ∈ (0 , 1] and λ 2 ( A ) < 1 .
First, we state a lemma which relates the generalization loss of DGD at iteration t to its train loss and consensus error up to iteration t .
The lemma is derived based on a stability analysis [3, 4, 12].
Speciﬁcally we use a self-boundedness and a realizability assumption [23] which makes the stability analysis feasible for settings such as logistic regression on separable data.
Additionally, we assume convexity and L -smoothness of the loss function.
For simplicity, we denote f i ( w ) :
= f ( w, x i ) for i ≤ n .
Assumption 2 (Convexity) The loss functions f
i : R d → R are convex and differentiable, satisfying,
[Equation]
Assumption 3 (Smoothness)
The loss functions f
i : R d → R are L -smooth and differentiable:
[Equation]
Assumption 4 (Self-boundedness of the gradient)
The loss functions f
i : R d → R satisfy the self-boundedness property with the parameters ( c, α ) ,
[Equation]
≤
[Equation]
α , for α ∈ [ 12 , 1] .
We note that Assumption 4 is weaker than Assumption 3, since an L -smooth function f satisﬁes (cid:107)∇ f ( w ) (cid:107) 2 ≤ 2 L ( f ( w ) − f (cid:63) ) , where f (cid:63) := inf w f ( w ) .
However, we make use of the smoothness property whenever it suits the analysis, in particular in the training loss analysis.
Throughout the paper, we assume that the training data is separable.
Assumption 5 (Data separability)
The training loss ˆ F in Eq.
(3) satisﬁes the data separability condition i.e., ˆ F (cid:63) = 0 ,
where ˆ F (cid:63) :=
inf w ∈ R d ˆ F ( w ) .
Before stating our key lemma, we introduce a few necessary notations.
We deﬁne the matrix
W ( t ) ∈
R N × d as the concatenation of all agents’ parameters at iteration t , i.e., W = [ w ( t ) 1 , · · · , w ( t ) N ] (cid:62) .
We also denote by ¯ w ( t ) the average of local parameters, i.e., ¯ w ( t ) :
= 1 N (cid:80) Ni =1 w ( t ) i , and denote by ¯ W ( t ) ∈
R N × d its concatenated matrix ¯ W ( t ) =
[ ¯ w ( t ) , · · · , ¯ w ( t ) ] .
Lemma 1 (Key lemma, Informal version)
Consider the iterates of decentralized gradient descent in Eq.
(2) with a ﬁxed positive step-size η ≤ 2 L .
Let Assumptions 1-4 hold.
Then for the test loss F at iteration T ≥ 1 , it holds that
[Equation]
where the expectation is over training samples.
The precise statement and the proof of Lemma 1 are deferred to Appendix B. As a result of
Lemma 1, the test error is related to the test loss to the train loss and the consensus error.
It is worth remarking that Eq.
(4) is in fact valid for both DGD and the Decentralized Gradient Tracking (DGT)
[19] algorithms.
This is merely because the proof of Lemma 1 only relies on the updates of the “averaged” parameter ¯ w ( t ) and moreover the update rule of ¯ w ( t ) for both DGD and DGT is ¯
[Equation]
Thus by Eq.(4) one can also obtain test loss bounds of
DGT after replacing appropriate bounds of DGT for the training loss and consensus error.
We leave this to future work.
In the following sections, we discuss how Lemma 1 yields test loss bounds on DGD under different assumptions on the loss function.
2.1. Convergence with general convex losses
The upper-bound in Eq.(4) shows how the consensus error and train loss of DGD affect the test loss.
The next lemma bounds the training loss and consensus error of DGD for general convex losses.
The proof is deferred to Appendix C.
Lemma 2 (Training bounds for convex losses) Under Assumptions 1-4, for any w ∈ R d and for a ﬁxed step-size η < min( 1 − α 1 L , 1 L (cid:113) 1 − α 1 2 α 2 ) , where α 1 ∈ (0 , 1) , α 2 > 0 are parameters that depend only on the mixing matrix, the following holds for the train loss and consensus error of DGD (2) :
[Equation] [Equation]
To bound the training loss for loss functions f ( · ) where the optimum is attained at inﬁnity we need a realizability assumption.
In particular, we choose w ∈ R d (of Lemma 2) using the following assumption.
Assumption 6 (Realizability)
The loss functions f
i : R d → R satisfy the realizability condition, i.e. there exists a decreasing function ρ : R + → R + such that for every ε > 0 there exists ˆ w ∈ R d with (cid:107) ˆ w (cid:107) ≤ ρ ( ε ) that satisﬁes f i ( ˆ w ) ≤ ε .
The set of assumptions 2-6 covers logistic loss over linearly separable data, in addition to losses with other exponential-type tails exp( − w r ) and polynomial tail w − r , for r > 0 .
Remark 3 (Training loss of DGD on separable data)
The realizability assumption appeared in [12, 23] (and implicitly used in [5, 24]) as the required assumption for bounding the training error under separable data.
It can be checked that for γ denoting the margin of training data, loss functions with an exponential tail such as logistic loss satisfy this assumption with ρ ( ε )
= 1 γ log( 1 ε ) (e.g., see Proposition 28 and [23, Lemma 4]).
Based on Lemma 2, this leads to the following bound for DGD training loss for all ε > 0 ,
[Equation]
In particular, choosing ε = 1 /T , gives a rate of O ( (log T ) 2 ηT ) , surprisingly matching up to the corresponding rate for centralized GD derived in [5, Theorem 1.1].
Theorem 4 (Test loss with convex losses) Under Assumptions 1-6, by choosing η < min( 1 − α 1 L √ T ,
[Equation]
L
[Equation]
T ) where α 1 ∈ (0 , 1) , α 2 > 0 are parameters that depend only on the mixing matrix and assuming ε ≤ ρ ( ε ) ηT the following bound holds for the test error of DGD for iteration T ,
[Equation]
where the expectation is over i.i.d.
training samples.
Remark 5 (DGD with logistic regression never overﬁts)
The proof of Theorem 4 is delayed to
Appendix C.2.
As in Remark 3, we take logistic regression on separable data as our case study.
For logistic regression (as well as other loss functions with an exponential tail), it can be veriﬁed that the self-boundedness assumption holds with α = 1 .
Similar to Remark 3 it holds that ρ ( ε ) =
1 γ log( 1 ε ) , thus choosing ε = 1 / √ T results in a test loss rate ˜ O
( 1 √ T + 1 n ) by Eq.
This indicates that the upper-bound decreases at a rate of ˜ O
( 1 √ T ) until after T = n 2 · (max( 1 Lc , Lc )) 4 iterations where the upper bound essentially reduces to ˜ O
( L 2 c 2 n ) .
Additionally, the fact that the upper-bound is decreasing proves that with appropriate choice of step-size, overﬁtting never happens along the path of DGD at any iteration.
.2. On the convergence of DGD with exponentially-tailed losses
We note that the bounds in Lemma 2 and Theorem 4 hold for the average loss across iterations t ≤ T .
It is straight-forward to see that if DGD is a descent algorithm i.e.,
ˆ F ( ¯ w ( t +1) ) ≤ ˆ F ( ¯ w ( t ) ) for all t ≤ T , then ˆ F ( ¯ w ( T ) ) ≤ 1 T (cid:80)
Tt =
1 ˆ F ( ¯ w ( t ) ) ; thus implying that the upper-bounds on training and test loss hold for the last iterate of DGD.
Here, we show that under certain conditions on the loss and step-size, DGD is a descent algorithm.
Moreover, we show that the consensus error of 2 as well as the test loss bounds of Theorem 4 can be improved for this class of convex loss functions.
In particular, we use the following assumptions together with the self-boundedness gradient assumption (Assumption 4) with α = 1 as well as the convexity assumption.
Assumption 7 (Self-bounded Hessian)
The local losses ˆ F (cid:96) : R d → R satisfy the following for the Hessian matrices ∇ 2 ˆ F (cid:96) and a positive constant h ,
[Equation]
Assumption 8 (Self-lowerbounded gradient)
The global loss satisﬁes for a constant τ that
[Equation]
ˆ F ( w ) (cid:107) ≥ τ ˆ F ( w ) .
Assumptions 2, 4, 7 and 8 considered in this section, include linear classiﬁcation with nonsmooth losses such as the exponential loss, losses with super-exponential tails ( exp( − x r ) , r > 1 ) and the logistic loss; e.g., see Proposition 26 in the appendix.
Theorem 6 (Last iterate convergence of DGD) Consider DGD with the loss functions and mixing matrix satisfying Assumptions 1,2,7,8 and Assumption 4 with α = 1 and c = h .
Assume that the step-size satisﬁes η < γ ˆ F (1) , for a constant γ depending only on the mixing matrix, τ and h .
Then, the train loss and the consensus error of DGD at iteration T satisfy the following for all w ∈ R d ,
[Equation]
Remark 7 (Improved rates)
We prove Theorem 6 in Appendix D.1.
While similar to Lemma 2, for logistic regression we have ˆ F ( ¯ w ( T ) )
=
˜ O ( 1 ηT + 1 T ) , for the consensus error rate we have by applying Theorem 6 and noting that ρ ( ε ) =
log(1 /ε ) /γ ,
[Equation]
After choosing ε = 1 /T , we have the improved rate (cid:107) W ( T ) − ¯ W ( T ) (cid:107) 2 F
=
˜
O
( 1 T 2 ) , which is a signiﬁcant improvement from ˜ O
( 1 T ) for general convex losses with constant η (Lemma 2).
For the test loss, employing Lemma 1 with the new rates for the consensus error leads to the following rate for DGD with logistic regression,
[Equation]
We note that this is a superior rate for the test loss of logistic regression, compared to the rate of
Remark 5.
Notably, this is comparable to the test loss rate for “centralized” logistic regression on linearly separable data, derived as E [ F ( ¯ w ( T ) )]
=
˜ O ( 1 ηT + 1 n ) in [23, 24].
3. Conclusion
We studied the behavior of train loss and test loss of decentralized gradient descent (DGD) methods when training dataset is separable.
Notably, our results are the ﬁrst rigorous guarantees for the generalization error of DGD.
This line of work can be extended in several directions, with the most exciting one being the analysis of non-convex objectives.
Appendix
The appendix is organized as follows:
Convergence and generalization under the PL condition is discussed in Appendix A.
The proofs are included in Appendix E. 2.
The proof of Lemma 1 is included in Appendix B. 3.
The proofs for Section 2.1 are included in Appendix C. 4.
The proofs for Section 2.2 are included in Appendix D. 5.
Auxiliary results on our assumptions are included in Appendix F. 6.
Our proposed algorithms for improving decentralized learning with separable data are presented in Appendix G. 7.
Finally, we conduct numerical experiments in Appendix H.
Notation
Throughout the appendix we use the following notations:
[Equation]
where recall that d is the dimension of ambient space, n is the total sample size, N is the number of agents and each agent has access to n/N samples.
Appendix A. Convergence under the PL condition
In this section, we extend our results to the case where the global loss satisﬁes the µ -PL condition.
Formally, the PL condition [14, 21] is deﬁned as follows,
Assumption 9 (PL condition)
The loss function ˆ F : R d → R satisﬁes the Polyak-Lojasiewic(PL) condition with the parameter µ :
[Equation]
The next lemma shows that DGD enjoys an exponential rate under the PL condition and smoothness.
See Appendix E.1 for a proof.
Lemma 8 (Train loss under the PL condition) Let Assumptions 1,3 and 9 hold and let the stepsize
[Equation] [Equation]
, 1 L ) , where the constants α 1 ∈ (0 , 1) and α 2 > 0 depend only on the mixing matrix.
Deﬁne ζ := 1 − ηµ 2 , then under the data separability assumption, the iterates of DGD satisfy for all t ≥ 1 ,
[Equation]
We use this lemma combined with our key lemma 1 to obtain the test loss bound in the next theorem.
The proof is provided in Appendix E.2.
Theorem 9 (Test loss under the PL condition) Let Assumptions 1-4 and 9 hold, and let η and ζ be as in Lemma 8.
Then the iterates of DGD satisfy for all T ≥ 1 ,
[Equation]
Remark
The bound in (8) is directly related to ( ηT ) 2 − 2 α .
When α < 1 , as in the case of smooth functions such as highly over-parameterized Least-squares f ( w, x ) = (cid:107) 1 − w (cid:62) x
(cid:107) 2 where d >> n , the bound becomes vacuous as it is increasing with T .
This indicates the existence of overﬁtting in
DGD under such scenarios; with the optimal value of T achieved at the very early steps of gradient descent.
Appendix B. Proof of Lemma
Lemma 11 (Formal statement of Lemma 1) Consider the iterates of decentralized gradient descent in Eq.
(2) with a ﬁxed positive step-size η ≤ 2 L .
Let Assumptions 1-4 hold.
Then for the test loss F at iteration T ≥ 1 , it holds that
[Equation]
where the expectation is over training samples and W ( t )
¬ i , ¯ W ( t ) ¬ i denotes the parameter matrix and averaged parameter matrix at iteration t for the DGD algorithm when the i th data sample is left out.
Proof We deﬁne w ( t ) (cid:96), ¬ i as the parameter of agent (cid:96) resulting from decentralized gradient descent at iteration t , when the i th training sample i ≤ n , is left out during training.
We emphasize that the i th sample may or may not belong to the dataset of agent (cid:96) .
We deﬁne
[Equation]
as the average of all agents’ parameters at iteration t , when the i th sample is left of the algorithm.
Thus, the parameter matrices W ( t ) ¬ i , ¯ W ( t ) ¬ i ∈ R N × d are deﬁned as follows,
[Equation]
The ﬁrst step in the proof is to bound the term 1 n (cid:80) ni =1 (cid:107) ¯ w ( t ) − ¯ w ( t )
¬ i (cid:107) 2 .
By deﬁnition of DGD in
Eq.(2), we have the following update rule for the averaged parameter,
[Equation]
Thus by adding and subtracting ¯ ∇ ˆ F ( ¯ W ( t ) ) and ¯ ∇ ˆ F ( ¯ W ( t ) ¬ i ) , we have
[Equation]
i (cid:107)
[Equation]
For the last term, using smoothness, we can write
[Equation]
The second term is upper-bounded similarly.
Using these bounds, splitting the gradient ∇ ˆ F ( ¯ W ( t ) )
=
1 n (cid:80) i (cid:48)
(cid:54) =
i f
( ¯ w ( t ) , x i (cid:48) )
+ 1 n f ( ¯ w ( t ) , x i ) , using smoothness and convexity (cid:107) w + η ∇ f ( w ) − v − η ∇ f ( v ) (cid:107) ≤ (cid:107) w − v (cid:107) for η ≤ 2 /L
[20] and employing Assumption 4 we can write
[Equation]
w
[Equation]
By summing over t ∈ [ T ] ,
[Equation]
We deﬁne for the ease of notation the following two consensus terms,
[Equation]
Thus the bound for the squared term can be written as follows
[Equation]
By averaging over i ∈ [ n ] and noting that α ∈ [1 /
[Equation]
1] so that x α is concave
[Equation]
we conclude that
[Equation]
Thus we have for iteration T :
[Equation]
Next we use [23, lemma 7]
[Equation]
which states that for the L -smooth loss f , the test error for a w ∈ R d as the output of an algorithm with dataset size n , satisﬁes the following,
[Equation]
where expectations are taken over the training set ( x 1 , x 2 , · · · , x n ) .
We replace w with ¯ w ( T ) and by using (10),
[Equation]
i =
[Equation]
i
[Equation]
N
E
[Equation]
T ) ] .
This leads to (9) and completes the proof.
Finally, we note that we can combine the two terms for the the consensus error e ( T ) i , e ( T ) , as the consensus error is not affected (order-wise) by removing a sample from the dataset.
This can be seen by noting that our bounds on the consensus are not affected by the sample-size.
Rather they depend crucially on the smoothness parameter L which upper-bounds the smoothness parameter of leave-one-out loss.
This leads to the informal version of the lemma.
Appendix C. Proofs for Section 2.
Lemma 12 (Recursions for the consensus error) Let the step-size η ≤ (1 − λ ) / 4 L where λ :=
max(( | λ 2 ( A ) | , | λ N ( A ) | )) 2 .
The consensus error of DGD under Assumptions 1,3 satisﬁes the following:
[Equation]
(11) where
α 1 :=
3+ λ 4 , α 2 := 4( 2 1 − λ − 1) .
Proof Denoting A ∞ :=
lim t →∞ A t = 1 N 11 T , it holds by Assumption 1,
[Equation]
where W i is the i th column of W .
By Assumption 1, λ = max(( | λ 2 ( A ) | , | λ N ( A ) | ))
2 < 1 .
For the consensus error, we can write,
[Equation]
, where the second step is due to (cid:107) X − ¯ X (cid:107) F ≤ (cid:107) X (cid:107) F [9, 10].
The last line holds for any β > 0 , due to (cid:107) a + b (cid:107) 2 ≤ (1 + β − 1 ) (cid:107) a (cid:107) 2 + (1 + β − 1 ) (cid:107) b (cid:107) 2 .
Based on this inequality and by noting (12) and using the L − smoothness assumption, we can deduce that,
[Equation]
−
[Equation]
t
[Equation] [Equation]
1 ) η
[Equation]
13) ≤ (1 + β ) λ (cid:107) W ( t − 1)
− ¯ W ( t − 1) (cid:107) 2 F + 2(1 + β − 1 )
η
2 L 2 (cid:107) W ( t − 1)
− ¯ W ( t − 1) (cid:107) 2 F
[Equation]
where the last step is due to L − smoothness and the non-negativity of ˆ F (cid:96) .
[Equation]
Thus,
[Equation]
choose β = (1 − λ ) / (2 λ ) , then by the assumption η ≤ (1 − λ ) /
4 L
[Equation]
we can derive the following two inequalities,
[Equation]
This concludes the lemma.
By telescoping summation over the iterates t = 1 , · · · , T of the consensus error in Eq.(11), we end up with the consensus error at iteration T .
The ﬁnal expression is stated in the next lemma.
Lemma
Under the assumptions of Lemma 12, it holds for T > 1 that,
[Equation]
Lemma
Under the assumptions of Lemma 12 and the zero initialization assumption for all agents, the average consensus error satisﬁes,
[Equation]
Proof By Lemma 13 and the zero initialization assumption
[Equation]
we have
[Equation]
Lemma 15 Under Assumptions 2,3 and for any w ∈ R d
[Equation]
the following recursions hold for the
Decentralized gradient descent updates:
[Equation]
Proof Starting by expanding
[Equation]
we get
[Equation]
For the second term
[Equation]
using the L − smoothness property and non-negativity of loss
[Equation]
we obtain the following:
[Equation]
For the third term by using L − smoothness and convexity properties we can write,
[Equation]
Combining these inequalities we derive the following:
[Equation]
Summing these equations for t = 1 , 2 , ..., T − 1 ,
[Equation]
C.1. Proof of Lemma
Lemma 16 (Restatement of Lemma 2) Under Assumptions 1-4 and zero initialization, for any w ∈ R d and for a ﬁxed step-size η < min( 1 − α 1 L , 1 L (cid:113) 1 − α 1 2 α 2 ) , where α 1 ∈ (0 , 1) , α 2 > 0 are parameters that depend only on the mixing matrix, the following holds for the train loss and consensus error of DGD (2) :
[Equation]
Proof We deduce from Lemma 15 that,
[Equation]
By Lemma 14,
[Equation]
where the condition on η ensures that (2 L 2 η + L ) α 2 η 2 L/ (1 − α 1 ) < 1 / 2 .
This gives the statement of the lemma for the training loss in Eq.(14).
Appealing again to Lemma 14 for the consensus error yields (15)
[Equation]
C.2. Proof of Theorem
We are ready to prove Theorem 4 by combining our results from Lemmas 2 and 1.
We state the proof for general choice of step-size η .
In particular, Theorem 4 follows by the next theorem after choosing η = O (1 / √ T )
[Equation]
Theorem 17 (Theorem 4 for general η ) Consider DGD under Assumptions 1-6, and choose η < min( 1 − α 1 L , 1 L (cid:113) 1 − α 1 2 α 2 ) .
The following bound holds for the averaged test error of DGD with separable data up to iteration T , assuming ε ≤ ρ ( ε ) 2 /ηT ,
[Equation]
Proof By Lemma
[Equation]
Thus, by Lemma 2,
[Equation]
By Assumption 6 and assuming ε ≤ ρ ( ε ) 2 /ηT , the statement of the theorem follows.
Appendix D. Proofs for Section 2.
Lemma 18 (Iterates of consensus error) Consider DGD with the loss functions and mixing matrix satisfying Assumptions 1,2, 7 and Assumption 4 with α = 1 and c = h .
By choosing η t − 1 ≤ 1 − λ 4 hM ( t − 1) the consensus error at iteration t > 1 satisﬁes
[Equation]
where β 1 := (3 + λ ) / 4 , β 2 :
= (4 / (1 − λ ) − 2) ,
λ = max( λ 2 ( A ) 2 , λ N ( A ) 2 ) and M ( t − 1) :
=
max( ˆ F ( W ( t − 1) ) , ˆ F ( ¯ w ( t − 1) )) .
Proof By Lemma 12 and the inequality (13), the consensus error satisﬁes for any β > 0 ,
[Equation] [Equation]
For the second term in (17)
[Equation]
we have the following chain of inequalities,
[Equation]
The Taylor’s remainder theorem gives (18) and v i ∈
[ w ( t − 1) i , ¯ w ( t − 1) ] denotes a point that lies on the line connecting w ( t − 1)
i and ¯ w ( t − 1) .
Also, (19) is valid due to the self-boundedness of the Hessian stated in Assumption 7.
The inequality (20) follows by the assumption of convexity of ˆ F i , due to the fact that for a convex function f : R d → R and any two points w 1 , w 2 ∈ R d , it holds that max v ∈ [ w 1 ,w 2 ] f ( v ) ≤ max( f ( w 1 ) , f ( w 2 )) .
To derive (22), we used max i ≤ N ˆ F i ( w i ) ≤ N ˆ F ( W ) and max i ≤ N ˆ F i ( ¯ w ) ≤ N · ˆ F ( ¯ w ) , which hold since the loss functions are non-negative.
In order to derive an upper-bound on the last term in (17), we use Assumption 4, (cid:107)∇
[Equation]
N (cid:88)
[Equation]
(cid:107)∇
[Equation]
Replacing the upper-bounds back in (17), we conclude
[Equation]
Choose β = 1 − λ 2 λ .
Then by the assumption of lemma η t − 1 ≤
[Equation]
hM
[Equation]
1) , it can be veriﬁed that the following two inequalities hold,
[Equation]
This concludes the proof.
The next lemma bounds the consensus error at iteration T , by recursively evaluating (16).
Lemma 19 (Last iterate consensus error)
Under the assumptions and notations of Lemma 18, the consensus error at iteration T satisﬁes
[Equation]
Lemma
Under the assumptions and notations of Lemma 18, with zero initialization W (1)
=
¯ W (1) = 0 and by choosing η t ≤ (1 − λ ) √ 1 − β 1
h 2 M ( t ) √ β
for t ∈ { 1 , · · · , T − 1 } , it holds at iteration T that
[Equation]
Proof First, we prove ˆ F ( W ( T ) ) ≤ 2 ˆ F ( ¯ w ( T ) ) .
If ˆ F ( W ( T ) ) ≤ ˆ F ( ¯ w ( T ) ) , the desired statement is proved.
Assume ˆ
F ( W ( T ) ) ≥ ˆ F ( ¯ w ( T ) ) , then by applying Taylor’s remainder theorem, the selfboundedness assumption 4 with c
= h, α = 1 , convexity of ˆ F , Lemma 19 and the restriction on the step-size, in respective order, we have the following inequalities,
[Equation]
Thus ˆ F ( W T ) ≤ 2 ˆ F ( ¯ w ( T ) ) .
By exchanging W ( T ) and ¯ w ( T ) and in a similar style we derive ˆ F ( W T ) ≥ 12 ˆ F ( ¯ w ( T ) ) .
This completes the proof of the lemma.
We are ready to prove Theorem 6.
First, we prove that DGD is a descent algorithm in the next lemma.
Lemma 21 (descent lemma) Consider DGD under the assumptions and notations of Lemma 18.Moreover, let Assumption 8 hold, then by choosing η t ≤ γ ˆ F ( ¯ w ( t ) ) for t ≤ T , where γ :=
(max( 4 h 3 τ 2 , h 2 , 6 h 2 β 2 1 − β 1 , 4 h 2 √ β 2 τ (1 − β 1 ) )) − 1 , DGD is a descent algorithm i.e.,
[Equation]
Proof With the self-boundedness assumption on the Hessian (Assumption 7) and applying the
Taylor’s remainder theorem for step T + 1 of DGD, we obtain the following, ˆ F
[Equation]
where for the third step we used
[Equation]
In the next step of the proof
[Equation]
we upper-bound the second and third terms in (23).
For the second term, by noting that 2 (cid:104) a, b (cid:105) = (cid:107) a (cid:107) 2 + (cid:107) b (cid:107) 2 − (cid:107) a − b (cid:107) 2 , we can write (cid:104)∇ ˆ F (
[Equation]
By recalling (22) and Lemma
[Equation]
we ﬁnd an upper-bound the last term in (24) as follows,
[Equation]
Returning back to (23), thus far we have derived the following,
[Equation]
We aim to prove that ˆ F ( ¯ w ( T +1) )
≤ ˆ F ( ¯ w ( T ) ) for all T ≥ 1 .
If ˆ F ( ¯ w ( T +1) ) > ˆ F ( ¯ w ( T ) ) , applying (26) with the assumption η t < γ ˆ F ( ¯ w ( t ) ) yields,
[Equation]
Note that it holds due to (25) that,
[Equation]
Replacing this in (27) and noting that M ( T ) ≤ 2 ˆ F ( ¯ w ( T ) ) by Lemma 20, we can simplify the inequality (27) as follows,
[Equation]
where for the ease of notation we deﬁne C (cid:48) :
= 4 h 2 β 2 (1 − β 1 ) − 1 .
Recalling η T < γ ˆ F ( ¯ w ( T ) ) and noting that by the assumption of the lemma γ ≤ τ 2 4 h 3 and γ < 1 h 2 and γ < τ 2 h √ C (cid:48)
we conclude that,
[Equation]
Dividing both sides by ˆ F ( ¯ w ( T ) ) leads to the contradiction due to the fact that τ ≤ h and thus τ/ 2 h < 1 .
Thus ˆ F ( ¯ w ( T +1) ) ≤ ˆ F ( ¯ w ( T ) ) .
This completes the proof.
D.1. Proof of Theorem
Theorem 22 (Restatement of Theorem 6) Consider DGD with the loss functions and mixing matrix satisfying Assumptions 1, 2, 7, 8 and Assumption 4 with α = 1 and c = h .
Assume that the step-size satisﬁes η <
[Equation]
, for a constant
[Equation]
:=
[Equation]
max
[Equation]
√ βτ (1 − β 1 ) )) − 1 , where β 1 , βare deﬁned same as in Lemma 18.
Then DGD is a descent algorithm i.e, for all T ≥ 1 it holds that
[Equation]
Moreover, the train loss and the consensus error of DGD at iteration T satisfy the following for all w ∈ R d ,
[Equation]
Proof First, we note that by Lemma 21, under the assumption η t ≤ γ/ ˆ F ( ¯ w ( t ) ) for t ≤ T , we have ˆ F ( ¯ w ( T +1) )
≤ ˆ F ( ¯ w ( T ) ) .
Thus ﬁxing η ≤ γ/ ˆ F ( ¯ w (1) ) , ensures that ˆ F ( ¯ w ( T +1) )
≤ ˆ F ( ¯ w ( T ) ) , for all
T .
Next, we derive the train loss and consensus error under the assumptions of the theorem.
[Equation] [Equation]
For the second term, we can write,
[Equation]
For the third term in (30)
[Equation]
we have,
[Equation] [Equation]
[Equation]
h ˆ F (
[Equation]
( t )
[Equation]
(cid:107)
[Equation]
t )
[Equation]
¯
[Equation]
( t ) (cid:107) F + ˆ F ( w ) − ˆ F (
[Equation]
( t ) )
[Equation]
Here (31) follows by convexity of ˆ F i , and (32) follows by the assumption on self-boundedness of the gradient
[Equation]
Thus, the inequality (30) can be written as follows,
[Equation]
Moreover, by Lemma 19 and the assumption on η ,
[Equation]
and
[Equation]
Thus (33) changes into,
[Equation]
Telescoping sum leads to
[Equation]
By Lemma
[Equation]
we have ˆ F ( ¯ w ( t ) ) ≤ 2 ˆ F ( W t ) ) .
Finally, as we proved in the beginning
[Equation]
DGD is a descent algorithm, implying
[Equation]
In view of (34)
[Equation]
this yields the claim of the theorem for the train loss (28).
Appealing to Lemma 19, gives (29).
This completes the proof of the theorem.
Appendix E. Proofs for Section A
Lemma 23 (Train loss under PL condition) Let Assumptions 1,3 and 9 hold, and let η ≤ min(1 /µ, (cid:113) (1 − α 1 ) µ 4 L 4 α 2 , 1 /L ) and ¯ ζ := max( 1+ α 1 2 , 1 − ηµ/ 2) , where α 1 :
= 3+ λ 4 , α 2 := 4( 2 1 − λ − 1) same as in Lemma 12, Then for t ≥
[Equation]
Proof by L − smoothness we have
[Equation]
By µ − PL condition we have,
[Equation]
By Lemma
[Equation]
36) which results in,
[Equation]
By induction assume ˆ F ( ¯ w ( t ) ) ≤ ¯ ζ t − 1 ˆ F ( ¯ w (1) ) then using the assumptions on ¯ ζ and η yield the following inequalities,
[Equation]
This completes the proof of the lemma
[Equation]
E.1. Proof of Lemma
Lemma 24 (Restatement of Lemma 8) Let Assumptions 1,3 and 9 hold and let the step-size η ≤ min( 1 − α 1 µ , 12 L 2 (cid:113) (1 − α 1 ) µ α, 1 L ) , where the constants α 1 ∈ (0 , 1) and α 2 > 0 are deﬁned same as in
Lemma 23.
Deﬁne ζ := 1 − ηµ 2 , then under the data separability assumption, the iterates of DGD satisfy for all t ≥ 1 ,
[Equation]
Proof The bound on the train loss follows directly by Lemma 23, after noting that η ≤ 1 − α 1 µ implies 1+ α 1≤ 1 − ηµ/ 2 .
The consensus error is derived by (36) and using the bound on ˆ F ( ¯ w ( t ) ) .
E.2. Proof of Theorem
Theorem 25 (Restatement of Theorem 9) Let Assumptions 1-4 and 9 hold, and let η and ζ be as in Lemma 23.
Then the iterates of DGD under the data separability assumption satisfy for all
T ≥ 1 ,
[Equation]
Proof By simplifying Lemma 1 using the convergence bounds in Lemma 24, we end up with the following,
[Equation]
Based on the deﬁnition of ζ , we have ( η 1 − ζ ) 2 α = ( 2 µ ) 2 α and
[Equation]
≤ 4 µ 2 .
This proves the statement of the theorem.
Appendix F. Auxiliary Results
Proposition 26 (Bounds on the exponential loss) Consider linear classiﬁcation with the exponential loss f ( w, ( a, y )) =
exp( − y · w (cid:62) a ) over linearly separable dataset ( a i , y i ) ni
=1 with binary labels y i and with max i (cid:107) a i (cid:107) ≤ r for a constant r .
The training loss in this case satisﬁes for all w ∈ R d ,
[Equation]
for constants c, c (cid:48) and h independent of w .
Proof Recall that,
[Equation]
One can deduce that,
[Equation]
Therefore it holds that,
[Equation]
A similar approach for the Hessian of ˆ F results in the following inequality,
[Equation]
Moreover
[Equation]
due to linear separability there exists a w (cid:63) ∈ R d such that,
[Equation]
where γ > 0 denotes the margin.
Therefore
[Equation]
using the supremum deﬁnition of norm we can write,
[Equation]
This completes the proof
[Equation]
Proposition 27 (Bounds on the logistic loss) Consider linear classiﬁcation with the logistic loss f ( w, ( a, y ))
=
log(1 + exp( − y · w (cid:62) a )) over linearly separable dataset ( a i , y i ) ni
=1 with binary labels y i and with max i (cid:107) a i (cid:107) ≤ r for a constant r .
The training loss in this case satisﬁes for all w ∈ R d ,
[Equation]
for Φ( w ) := 1 n (cid:80) ni
[Equation]
i
[Equation]
and constants c, c (cid:48)
and h independent of w .
Proof The training loss in this case is the following:
[Equation]
Thus,
[Equation]
By considering the norm and noting that exp( t ) / (1 + exp( t )) ≤ log(1 + exp( t )) ,
[Equation] [Equation]
) (cid:107)
[Equation]
Likewise, since exp( t ) / (1 + exp( t )) 2 ≤ 2 log(1 + exp( t )) , we can conclude that the operator norm of the Hessian satisﬁes,
[Equation]
This completes the proof of upper-bounds for the gradient and Hessian.
For the lower-bound on gradient note that by using the supremum deﬁnition of norm and recalling the max-margin separator satisﬁes y
[Equation]
≥ γ for the margin γ > 0 and all i ∈ [ n ] , we can write,
[Equation]
This yields the lower bound γ Φ( w ) on the norm of gradient and completes the proof.
Proposition
(Realizability of the exponential and logistic loss [23])
The exponential loss function satisﬁes the realizability assumption (Assumption 6) with ρ ( ε )
=
− 1 γ log( ε ) , where γ denotes the margin.
Moreover, the logistic loss function satisﬁes the realizability assumption with ρ ( ε ) =
− 1 γ log(exp( ε ) − 1) .
Appendix G. Improved algorithms for decentralized learning with separable data
In this section, we consider decentralized learning with exponential tail losses on separable data and propose algorithms for improving the convergence rates based on the normalized GD mechanism.
Our proposal –Fast Distributed Logistic Regression( FDLR )– is summarized in Algorithm 1.
Each agent keeps two local variables w (cid:96) , v (cid:96) ∈ R d which are also communicated to neighbor agents in each round.
In matrix notation, Algorithm 1 has the following two steps:
[Equation]
Here A ∈ R N × N is the mixing matrix corresponding to an undirected network which satisﬁes the regularity conditions in Assumption 1.
Furthermore W ( t ) , V ( t ) , ∇ ˆ F ( W ( t ) )
∈ R N × d are formed by stacking w ( t ) (cid:96) , v ( t ) (cid:96) and local gradients ∇
ˆ F (cid:96) ( w ( t ) (cid:96) ) for all (cid:96) , respectively.
The matrix (cid:101) V ( t ) is formed by concatenation of the vectors v ( t ) (cid:96) / (cid:107) v ( t ) (cid:96) (cid:107) .
In step (2), every agent (cid:96) runs in parallel an update rule which resembles the distributed gradient descent update rule, with the exception that the local gradient ∇
ˆ F (cid:96) ( w ( t ) (cid:96) ) is replaced by v ( t ) (cid:96) / (cid:107) v ( t ) (cid:96) (cid:107) .
In the next step, agents send their local parameters w t(cid:96) , v t(cid:96) to their neighbors.
Step (4) is the consensus step in which agent (cid:96) computes a weighted average of w ( t +1 / 2) k sent from neighbor agent k , in order to update w ( t ) (cid:96) .
Step (5) uses the newly computed local gradient ∇
ˆ F (cid:96) ( w ( t +1) (cid:96) ) and the gradient computed in the previous step to updates the local parameter v ( t ) (cid:96) .
This step is previously used in the gradient tracking algorithm
Input: Initial values w (1) (cid:96) , v (1) (cid:96)
∈ R d for all agents (cid:96) ∈
[ N ] , step size η t and mixing matrix
A =
[ A (cid:96)k ] N × N for t = 1 , . . .
, T all agents (cid:96) ∈
[ N ] in parallel do w ( t + 12 ) (cid:96) =
w ( t ) (cid:96)
[Equation]
( t ) (cid:96) (cid:107)
[Equation]
[Equation]
(cid:96) (cid:107) send and receive local variables w ( t + 12 ) (cid:96) and v ( t ) (cid:96)
[Equation]
end
Input: Initial values w (1) (cid:96) , v (1) (cid:96) , z (1) ∈ R
d for all agents (cid:96) ∈
[ N ] , hyper-parameters η t , γ t and mixing matrix
A =
[ A (cid:96)k ] N × N for t = 1 , . . .
, T all agents i ∈
[ N ] in parallel do z
[Equation]
t +
[Equation]
− η t ( z ( t +1) (cid:96) + v ( t ) (cid:96) (cid:107) v ( t ) (cid:96) (cid:107) ) send and receive local variables w ( t + 12 ) (cid:96) and v ( t ) (cid:96) w ( t +1) (cid:96)
= (cid:80) k ∈N (cid:96) A
(cid:96)k w ( t + 12 ) k v ( t +1) (cid:96) =
(cid:80) k ∈N (cid:96) A
(cid:96)k v ( t ) k
+ ∇ ˆ F (cid:96) ( w ( t +1) (cid:96) )
− ∇
ˆ F (cid:96) ( w ( t ) (cid:96) ) end (e.g. see [19]).
The purpose behind introducing the variable v ( t ) (cid:96) is to estimate the global loss.
In particular, in the inﬁnite time limit we can deduce that,
[Equation]
While each v ( t ) (cid:96) is distinct among agents, each of them is estimating the global loss of distributed optimization i.e., (cid:80) N(cid:96)
=1 ∇ ˆ F (cid:96) ( w ( t ) (cid:96) ) .
With this, step (2) of FDLR is approximately implementing a normalized GD update, which provably speeds up the convergence in centralized logistic regression over separable data[15].
A Nesterov accelerated variant of normalized GD has already been studied by [7] for centralized logistic regression.
In distributed case, the problem is more subtle as each agent has access only to local gradients.
Algorithm 2 uniﬁes FDLR with a Nesterov Momentum.
In particular, in step (3) the local parameter w ( t ) (cid:96) is updated by a weighted average ( z ( t +1) (cid:96) ) of normalized gradients from previous iterations.
While we do not prove convergence rates for the proposed algorithms, we study their asymptotic convergence.
Notably, the next theorem shows that for a time-decaying step-size η = 1 / √ t , the solution of FDRL converges in direction to the solution of centralized gradient descent.
We defer numerical performance of both Algorithms to the next section.
Theorem 29 (Asymptotic convergence of FDLR ) Let the sequence { w ( t ) (cid:96) } be generated by FDRL (Algorithm 1) for a separable dataset and with η t
= O (1 / √ t ) .
Then for all (cid:96) ∈ [ N ] ,
[Equation]
t
[Equation]
(cid:96)
[Equation]
where w MM is the solution to max-margin problem.
Proof Replace v ( t ) (cid:96) (cid:107) v ( t ) (cid:96) (cid:107) in step 2 of Algorithm 1 by arbitrary perturbations ε ( t ) (cid:96) of unit norm.
Then note that the sequence { w ( t ) (cid:96) } generated by step 2 is identical to decentralized GD with η (cid:107) ε ( t ) (cid:96) (cid:107) → 0 .
Thus by [16, Lemma 1], consensus is asymptotically achieved for all (cid:96) ∈
[ N ] , i.e.,
[Equation]
Thus
[Equation]
This implies that for all i ∈ [ N ] we have lim t →∞
[Equation]
) −
[Equation]
F
[Equation]
w
[Equation]
t )
[Equation]
(cid:107) = 0 , thus by appealing again to [16, Lemma 1] and applying it to step (5) of Algorithm 1, we ﬁnd that,
[Equation]
Aggregations of gradients in step (5) imply that ¯ v ( t )
=
1 (cid:62) ∇L ( W ( t ) )
N → ∇ ˆ F ( ¯ w ( t ) ) .
Thus step (2) of
FDLR for every agent i converges to ¯ w ( t ) − η ∇ ˆ F ( ¯ w ( t ) )
(cid:107)∇
[Equation]
, i.e.,
[Equation]
Thus for all (cid:96) , the sequence { w ( t ) (cid:96) } converges to the solution of normalized GD, i.e., the max-margin separator w MM , for linearly separable datasets ([15, Theorem 5]).
This leads to the statement of the theorem.
Appendix H. Numerical Experiments
In this section, we present numerical experiments to verify our theoretical results and demonstrate the beneﬁts of our proposed algorithms.
We begin with a numerical study of the performance of
FDLR .
[Figure]
M a r g i n G a p
[Figure]
H.1. Experiments on FDLR
In Figure 1(Left), we compare the performance of FDLR and its momentum variant to distributed gradient descent(GD) and gradient tracking(GT) for exponential loss with signed measurements (i.e., y = sign ( a (cid:62) w (cid:63) ) for samples a , labels y and the true vector of regressors w (cid:63) ) with n = 100 , d = 25 .
The underlying graph is selected as an Erdos-R`enyi graph with N = 50 agents (each having access to 2 samples from training set) and connectivity probability p c = 0 .
The y − axis,
Margin Gap represents the distance of normalized w ( t ) (cid:96) to the normalized ﬁnal solution for agent (cid:96) = 1 , i.e., (cid:107) w ( t ) 1 (cid:107) w ( t )
[Equation]
Intuitively, Margin Gap can be thought as the difference in direction to the ﬁnal solution.
The hyper-parameters η t , γ t are ﬁne-tuned for each algorithm to represent the best of each algorithm and the ﬁnal values are η t = 0 .
5 and 0 .
2 for Distributed GD,
GT, Algorithm 1 and Algorithm 2, respectively and γ t = 0 .
8 for Algorithm 2.
Our algorithms signiﬁcantly outperform the well-known distributed learning algorithms in directional convergence to the ﬁnal solution.
Regardless, in this case we noticed that the gain obtained by including the momentum is small.
In Figure 1(Right), we consider a binary classiﬁcation task on a real-world dataset (two classes of the UCI WINE dataset [1]) where d = 13 and n = 107 .
We compare the performance of FDLR (blue line) and its momentum variant (red line) with DGD and DGT on an Erdos-R`enyi graph with N = 10 and p c = 0 .
The hyper-parameters are ﬁne-tuned to η t = 12 , 1 , 0 .
9 , 2 for DGD, DGT, Algorithm 1 and Algorithm 2 respectively, and γ t = 0 .
88 for
Algorithm 2.
Notably, while Algorithm 1 signiﬁcantly outperforms both DGD and DGT, the beneﬁts of adding the momentum are also signiﬁcant in this case as Algorithm 2 demonstrates a faster rate of convergence than Algorithm 1.
Figure 2(Top) illustrates the train and test errors based on iteration for the same setting as Figure 1 with n = 800 and d = 50 .
Here the hyper-parameters are ﬁne-tuned to be η t = 0 .
01 , 0 . 01 , 0 . 4 , 0 .for
DGD,DGT, Algorithm 1 and Algorithm 2, respectively and γ t = 0 .
5 for Algorithm 2.
Figure 2(Bottom) represents the train ˆ F ( ¯ w ( t ) ) and test F ( ¯ w ( t ) ) losses.
Here, we use the same dataset with N = 10 , p c = 0 .
4 and an exponential loss.
The hyper-parameters are ﬁne-tuned to η t =
[Figure]
Iteration
[Figure]
E rr
[Figure]
Distributed GD
Distributed GT
Alg Alg
[Figure] [Figure]
Iteration
[Figure]
Distributed GD
Distributed GT
Alg Alg
[Figure]
6 and γ t = 0 .
Both of our algorithms outperform the commonly used methods gradient descent and gradient tracking, in training error and test error performance.
We can see that FDLR and its momentum variant are faster in interpolating the training data.
Also, the gains of adding the momentum are signiﬁcant, since FDLR with Nesterov momentum reaches an approximation of its ﬁnal test accuracy in 50 iterations, while the same happens for FDLR with approximately 300 iterations.
An interesting phenomenon in Figure 2 (see also the bottom plot in Figure 3) is the behavior of “test loss”: while during the starting phase the test loss is monotonically decreasing, after sufﬁcient iterations the test loss starts increasing.
This behavior of test loss is indeed captured by the bounds on the test loss of DGD in Theorems 4-6 and Remarks 5-7.
In particular, the increase in test loss is observed in the bound for the test loss O
( (log T )
[Equation]
+ (log T ) 2 n ) in Remark 5, where the presence of the term (log T ) 2 n suggests that the bound after sufﬁcient iterations starts to slowly increase.
[Figure]
H.2. Experiments on the convergence behavior of DGD
Next, we conduct numerical experiments on the convergence behavior of DGD for the train loss and the consensus error.
We consider the same network topology, mixing matrix and data setup as in the last ﬁgure.
Figure 3 (Left and middle) shows the consensus error 1 N (cid:107) W ( t ) − ¯ W ( t ) (cid:107) 2 F and the train loss ˆ F ( ¯ w ( t ) ) in solid lines, for two over-parameterization ratios d/n .
We recall that d and n represent the dimension of the ambient space and the dataset size, respectively.
The dashed lines help to show for each solid line, the approximate rate of convergence after sufﬁcient number of iterations.
Notably, we observe that the convergence rates on the consensus error ( ˜ O (1 /t 2 ) ) and on the train error ( ˜ O (1 /t ) ) stated in Remark 7 are attained in both cases (recall that ˜ O ( · ) hides logarithmic factors).
Figure 3 (Right) depicts the Test loss of DGD for d/n = 0 .
For comparison, the corresponding curve for centralized GD is also shown.
Here step-sizes are ﬁne-tuned to represent the best of each algorithm.
In agreement with our ﬁndings in Remark 7, we observe the similar behavior between the convergence behavior of two algorithms.
As a ﬁnal remark, the slight increase in the curves of test loss are due to the logarithmic factors in the derived upper-bounds for the test loss.
H.3. Experiments on over-parameterized Least-squares
In Figure 4, we conduct experiments for highly over-parameterized Least-squares ( f ( w, x ) = (cid:107) 1 − w (cid:62) x
(cid:107) 2 ), where d is typically signiﬁcantly larger than n to ensure perfect interpolation of dataset.
Note that, the train loss is not strongly-convex in this case, instead it satisﬁes the PL condition(Assumption 9).
Notably, as predicted by Lemma 8, we notice the linear convergence of the train loss and the consensus error in Figure 4 (Left).
On the other hand, for the test loss, we observe its remarkably fast convergence (after approximately 50 iterations) to the optimal value, which is followed by a sharp increase in the subsequent iterations.
[Figure] [Figure]
H.4. On the update rule of FDLR
In the ﬁnal section of the paper
[Figure]
we state a remark regarding the update rule of FDLR .
Recall the update rule of DGD,
[Equation]
Notably, we expect FDLR to be perhaps the simplest approach for accommodating normalized gradients in decentralized learning setting since in DGD the agents only have access to their local gradients.
In particular consider a Normalized DGD algorithm with the same update as in (38) but with ∇ ˆ F i ( w ( t ) i ) replaced by ∇ ˆ F i ( w ( t ) i ) /
(cid:107)∇ ˆ F i ( w ( t ) i ) (cid:107) , i.e.,
[Equation]
The Normalized DGD algorithm above does not lead to faster convergence.
This is due to the fact that in DGD the local gradient norm (cid:107)∇ ˆ F i ( w ( t ) i ) (cid:107) can be different than the global gradient norm (cid:107)∇ ˆ F ( w ( t ) i ) (cid:107) .
Thus even if with the update rule (39) the local parameters w ( t ) i converge to the global optimal solution, still the update rule for the averaged parameter ¯ w ( t ) is different than the update rule of centralized normalized GD.
Our numerical experiment in Figure 5 demonstrates the incapability of Normalized DGD in speeding up DGD.
In particular, we note that for any choice of step-size Normalized DGD does not lead to acceleration compared to DGD whereas FDLR massively outperforms DGD.