On Generalization of Decentralized Learning with Separable Data
Abstract
Decentralized learning offers privacy and communication efﬁciency when data are naturally distributed among agents communicating over an underlying graph.
Motivated by overparameterized learning settings, in which models are trained to zero training loss, we study algorithmic and generalization properties of decentralized learning with gradient descent on separable data.
Speciﬁcally, for decentralized gradient descent (DGD) and a variety of loss functions that asymptote to zero at inﬁnity (including exponential and logistic losses), we derive novel ﬁnite-time generalization bounds.
This complements a long line of recent work that studies the generalization performance and the implicit bias of gradient descent over separable data, but has thus far been limited to centralized learning scenarios.
Notably, our generalization bounds approximately match in order their centralized counterparts.
Critical behind this, and of independent interest, is establishing novel bounds on the training loss and the rate-of-consensus of DGD for a class of self-bounded losses.
Finally, on the algorithmic front, we design improved gradient-based routines for decentralized learning with separable data and empirically demonstrate orders-of-magnitude of speed-up in terms of both training and generalization performance.
1. Introduction
Machine learning tasks often revolve around inference from data using empirical risk minimization (ERM):
[Equation]
[Equation] Here f : R d ⇥ R d 0 !
R is a loss function and x i :=
y i a
i , where ( a i , y i ) ni
=1 iid ⇠ D represent features and labels, sampled from a distribution D .
In large scale machine learning, due to privacy concerns and communication constraints, data points are often distributed on a set of local computing agents.
Decentralized learning methods aim at minimizing the global loss function (1) while agents communicate their parameters on an underlying connected graph.
The most ubiquitous of these algorithms is Decentralized Gradient Descent (DGD).
Here the ` th agent runs a step of gradient descent followed by an averaging step in which every agent replaces its parameter with the average of its neighbors [20]:
[Equation]
The superscripts signify the iteration number and A ` k refers to the averaging weights used by agent ` for the parameter of agent k .
The global loss ˆ F is the average of local loss functions ˆ F ` , `  N , where each ˆ F ` is formed as the average empirical risk evaluated on the local training dataset S ` of the ` th agent:
[Equation]
where n ` denotes the dataset size of agent ` .
Convergence properties of the train loss ˆ F ( · ) in DGD have been studied extensively in literature, e.g., [15, 18–20, 34].
The bulk of these studies build upon classical optimization theory [22] suited for studying the train loss per iteration.
In particular, it is well-stablished in the literature that DGD converges at the rate 1 T
[Equation]
for smooth convex functions [18].
Here ¯ w ( t ) is the average of local parameters w ( t ) ` .
Our results in
Sections 2.1-2.2 show a rate of ˆ F ( ¯ w ( T ) )
=
O ( (log T ) 2 T ) and k W
[Equation]  [Equation] ) for the training loss and consensus error of DGD over separable data with “exponentially tailed” losses.
The study of generalization performance of DGD algorithms in the literature is mostly limited to empirical observations e.g., [9, 11, 32], making the theory behind test error performance largely unexplored.
Our ﬁrst goal in this paper is to complement prior general results on the convergence of training loss in DGD by considering speciﬁc, but commonly encountered, settings in ERM over separable data.
This includes the analysis of non-smooth objectives such as the exponential loss, analysis of logistic regression in the interpolating regime where the optimum is achieved at inﬁnity, and analysis of objectives satisfying the PL condition.
Our second goal is to study, for the ﬁrst time, convergence rates of the DGD test loss F ( ¯ w ( t ) ) :
= E x ⇠ D [ f ( ¯ w ( t ) , x )] .
Finally, we leverage recent advances in the study of centralized learning with separable data to design fast algorithms for decentralized learning.
Contributions.
Our contributions are summarized as follows,
• In Section 2.1 we derive convergence rates for the training and test loss of the decentralized gradient descent algorithm with separable data.
Our results hold for convex losses satisfying realizability and self-boundedness, as well as, convex losses satisfying self-boundedness and the PL condition.
In Section 2.2, we prove that under additional self-boundedness assumptions on the Hessian and gradient, which holds for exponentially tailed losses, the test loss bound can be improved to approximately match the test loss bounds of centralized GD.
We extend our results to losses satisfying the PL condition (in the long version of this work [29, Sec. 2.3]).
Numerical experiments in verify our theoretical results [29, Sec. 3.2].
• We propose two algorithms (see [29, Sec. 2.4]) for speeding up the convergence of decentralized learning with separable data.
Notably, numerical experiments [29, Sec. 3.1] demonstrate that our proposed algorithms signiﬁcantly improve the train error and test error of decentralized logistic regression.
Further related works Decentralized learning.
Over the last few years there have been numerous research works which consider the convergence of ﬁrst order methods for decentralized learning; an incomplete list includes [3, 9, 12, 13, 15, 18, 20, 24, 30, 31, 33, 34].
The study of generalization of DGD was mostly limited to numerical experiments.
While this paper was being written, we were notiﬁed of the recent work [28] which studies the generalization bounds of decentralized methods for Lipschitz convex losses.
However, we consider the setting of separable data and exponentially tailed losses and show that in our settings the train/test loss of DGD is closely similar to centralized GD.
Compared to this work, we also propose improved algorithms for learning with separable data.
We highlight that our rates on the training loss are comparable to [12, Theorem 2].
While [12] also derives convergence of DGD train loss on separable data, their analysis is valid only for bounded optimizers.
In contrast, we derive training loss bounds which are true for the case of unbounded optimizers as is the case for logistic regression over separable data.
Implicit bias of GD.
For centralized settings, a line of recent works [6, 7, 17, 25–27] studies the parameter convergence, as well as training and test loss convergence, of gradient descent on separable data, showing that for (a class of) monotonic losses the solution to ERM and the maxmargin solution are the same in direction.
, i.e., k ˆ w ( t )  ˆ w MM k ! 0 .
Here ˆ w ( t ) :
= w ( t ) / k w ( t ) k and ˆ w MM :=
w MM / k w MM k , where the vector w MM is the solution to the hard-margin support vector machine problem.
Notably, [6, 27] characterized the rate of convergence for margin gap to be k ˆ w ( T )  ˆ w MM k
= O (1 / log( T )) and for the training loss to be ˆ F ( w ( T ) )
=
O ( 1 ⌘ T ) .
Recently,
Shamir [26] and Schliserman and Koren [25] showed that the test loss of GD for logistic regression on linearly separable data satisﬁes F ( w ( T ) ) =
˜ O ( 1 ⌘ T + 1 n ) signifying that overﬁtting does not happen during the iterates of GD.
In Section 2.2 (Remark 9), we show that the test loss of DGD with logistic regression on linearly separable data satisﬁes E [ F ( ¯ w ( T ) )]
=
˜ O ( 1 ⌘ T + 1 n + ⌘ 2 ) .
This signiﬁes that, under the data separability assumption, the rate of decay for test loss is essentially identical for centralized and decentralized scenarios.
2. Main results
Throughout the paper we make the following standard assumption on the mixing matrix
A =
[ A ij ] N ⇥ N corresponding to the underlying connected network.
Assumption 1 (Mixing matrix)
The mixing matrix A 2 R N ⇥ N is symmetric, doubly stochastic with bounded spectrum i.e., |  i ( A ) | 2 (0 , 1] and  2 ( A ) < 1 .
First, we state a lemma which relates the generalization loss of DGD at iteration t to its train loss and consensus error up to iteration t .
The lemma is derived based on a stability analysis [4, 5, 14].
Speciﬁcally we use a self-boundedness and a realizability assumption [25] which makes the stability analysis feasible for settings such as logistic regression on separable data.
Additionally, we assume convexity and L -smoothness of the loss function.
For simplicity, we denote f i ( w ) :
= f ( w, x i ) for i  n .
Assumption 2 (Convexity) The loss functions f
i : R d ! R are convex and differentiable, i.e,
[Equation]
Assumption 3 (Smoothness)
The loss functions f
i : R d ! R are L -smooth and differentiable:
[Equation]
Assumption 4 (Self-boundedness of the gradient)
The loss functions f
i : R d ! R satisfy the self-boundedness property with the parameters ( c, ↵ ) , for ↵ 2 [ 12 , 1] .
[Equation]
We note that Assumption 4 is weaker than Assumption 3, since an L -smooth function f satisﬁes kr f ( w ) k 2  2 L ( f ( w )  f ? ) , where f ? : = inf w f ( w ) .
However, we make use of the smoothness property whenever it suits the analysis, in particular in the training loss analysis.
Throughout the paper, we assume that the training data is separable.
Assumption 5 (Data separability)
The training loss ˆ F in Eq.
(3) satisﬁes the data separability condition i.e., ˆ F ?
= 0 , where ˆ F ? :
= inf w 2 R d ˆ F ( w ) .
Before stating our key lemma, we introduce a few necessary notations.
We deﬁne the matrix
W ( t ) 2 R N ⇥ d as the concatenation of all agents’ parameters at iteration t , i.e., [Equation]
[Equation] We also denote by ¯ w ( t ) the average of local parameters, i.e., ¯ w ( t ) :
= 1 N [Equation] i , and denote by ¯ W
[Equation] t ) [Equation] its concatenated matrix
[Equation]
Lemma 1 (Key lemma, Informal version)
Consider the iterates of decentralized gradient descent in Eq.
(2) with a ﬁxed positive step-size ⌘  2 L .
Let Assumptions 1-4 hold.
Then for the test loss F at iteration T  1 , it holds that
[Equation]
where the expectation is over training samples.
Lemma 1 bounds the test loss with respect to the train loss and the consensus error.
In the following sections, we show how Lemma 1 yields test loss bounds on DGD by establishing bounds on the train loss and consensus errors under different assumptions on the loss function.
It is worth remarking that Eq.
(4) is in fact valid not only for DGD, but also for Decentralized Gradient Tracking (DGT).
DGT is another popular algorithm for distributed learning that can accelerate train error convergence over DGD by modifying the update in Eq.(2) such that each agent keeps a running estimate of the global gradient [21]. The reason why Eq.(4) continues to hold for DGD is that the proof of Lemma 1 only relies on the updates of the “averaged” parameter ¯ w ( t ) : = 1 N P
N ` =1 w ` and that the update rule of ¯ w ( t ) for both DGD and DGT is derived as
[Equation]
Thus, starting with Eq.(4) one can also obtain test loss bounds of DGT after replacing appropriate bounds of DGT for the training loss and consensus error.
We leave this to future work.
In the following sections, we discuss how Lemma 1 yields test loss bounds on DGD under different assumptions on the loss function.
2.1. Convergence with general convex losses
The upper-bound in Eq.(4) shows how the consensus error and train loss of DGD affect the test loss.
The next lemma bounds the training loss and consensus error of DGD for general convex losses.
Lemma 2 (Training bounds for convex losses) Under Assumptions 1-4, for any w 2 R d and for a ﬁxed step-size ⌘ < min( 1  ↵ 1 L , 1 L q 1  ↵ 1 2 ↵ 2 ) , where ↵ 1 2 (0 , 1) , ↵ 2 > 0 are parameters that depend only on the mixing matrix, the following holds for the train loss and consensus error of DGD (2) :
[Equation]
To bound the training loss for loss functions f ( · ) where the optimum is attained at inﬁnity we need a realizability assumption.
In particular, we choose w 2 R d (of Lemma 2) using the following assumption.
Assumption 6 (Realizability)
The loss functions f
i : R d ! R satisfy the realizability condition, i.e. there exists a decreasing function ⇢ : R + ! R + such that for every " > 0 there exists ˆ w 2 R d with k ˆ w k  ⇢ ( " ) that satisﬁes f i ( ˆ w )  " .
The set of assumptions 2-6 covers logistic loss over linearly separable data, in addition to losses with other exponential-type tails exp(  w r ) and polynomial tail w  r , for r > 0 ; See [29, Prop. 24].
Remark 3 (Training loss of DGD on separable data)
The realizability assumption appeared in [14, 25] (and implicitly used in [6, 26]) as the required assumption for bounding the training error under separable data.
It can be checked that for  denoting the margin of training data, loss functions with an exponential tail such as logistic loss satisfy this assumption with ⇢ ( " )
= 1  log( 1 " ) (e.g., see Proposition 30 and [25, Lemma 4]).
Based on Lemma 2, this leads to the following bound for DGD training loss for all " > 0 ,
[Equation]
In particular, choosing " = 1 /T , gives a rate of O ( (log T ) 2 ⌘ T ) , surprisingly matching up to the corresponding rate for centralized GD derived in [6, Theorem 1.1].
Remark 4 The bounds of Lemma 2 are true for any dataset { x i } i 2 [ n ] provided that Assumptionsand 3 hold for all f x = f x i = f ( w, x i ) : = f i ( w ) , i 2 [ n ] . Similarly, (6) holds provided Assumption 6 is true over the training set (i.e. provided the training dataset is separable).
However, bounding the test loss in Lemma 1, requires bounding the expectation over all datasets of the train/consensus errors. This is guaranteed by Assumptions 2-6 as they hold for any point x in the distribution. Theorem 5 (Test loss with convex losses) Under Assumptions 1-6, by choosing ⌘ < min( 1  ↵ 1 L p T , 1 L q 1  ↵ 1 2 ↵ 2 T ) where ↵ 1 2 (0 , 1) , ↵ 2 > 0 are parameters that depend only on the mixing matrix and assuming "  ⇢ ( " ) ⌘ T the following bound holds for the test error of DGD for iteration T , T
T
X t = E [ F ( ¯ w ( t ) )] = O ⇣ ⇢ ( " )p T + L 2 c 2 ⇢ ( " ) 4 ↵ n 3  2 ↵
T 1  ↵ + L 4 ⇢ ( " )p T ⌘ , (7) where the expectation is over i.i.d.
training samples.
Remark 6 (DGD with logistic regression never overﬁts)
As in Remark 3, we take logistic regression on separable data with margin  > 0 as our case study.
For logistic regression (as well as other loss functions with an exponential tail), it can be veriﬁed that the self-boundedness assumption holds with ↵ = 1 .
Similar to Remark 3 it holds that ⇢ ( " )
= 1  log( 1 " ) , thus choosing " = 1 / p T results in a test loss rate ˜ O
( 1 p T + 1 n ) by Eq.
This indicates that the upper-bound decreases at a rate of ˜ O
( 1 p T ) until after T = n 2 · (max( 1 Lc , Lc )) 4 iterations where the upper bound essentially reduces to ˜ O
( L 2 c 2 n ) .
Additionally, the fact that the upper-bound is decreasing proves that with appropriate choice of step-size, overﬁtting never happens along the path of DGD at any iteration. Remark 7 (Log factors) The attentive reader will have recognized in Remarks 3 and 6 that due to the “ ⇢ ( " ) =
O (log( T )) ” factor, the upper bound on the test loss in Eq.
(7) increases (very) slowly with log 4 ( T ) .
Note that this term becomes dominant only when T is exponentially large with respect to the sample size n and the margin  . Our experiments (in the long version of this work [29]) conﬁrm this slow logarithmic increase late in the training phase. Analogous behavior, but for centralized GD training, are discussed in [25, 27]. 2.2. On the convergence of DGD with
exponentially-tailed losses We note that the bounds in Lemma 2 and Theorem 5 hold for the average loss across iterations t  T .
It is straight-forward to see that if DGD is a descent algorithm i.e., ˆ F ( ¯ w ( t +1) )  ˆ F ( ¯ w ( t ) )
for all t  T , then ˆ F ( ¯ w ( T ) )  1 T
P
Tt =1 ˆ F ( ¯ w ( t ) ) ; thus implying that the upper-bounds on training and test loss hold for the last iterate of DGD.
Here, we show that under certain conditions on the loss and step-size, DGD is a descent algorithm.
Moreover, we show that the consensus error of 2 as well as the test loss bounds of Theorem 5 can be improved for this class of convex loss functions.
In particular, we use the following assumptions together with the self-boundedness gradient assumption (Assumption 4) with ↵ = 1 as well as the convexity assumption.
Assumption 7 (Self-bounded Hessian)
The local losses ˆ F ` : R d ! R satisfy the following for the Hessian matrices r 2 ˆ F ` and a positive constant h ,
[Equation]
Assumption 8 (Self-lowerbounded gradient)
The global loss satisﬁes for a constant ⌧ that
[Equation]
Assumptions 2, 4, 7 and 8 considered in this section, include linear classiﬁcation with nonsmooth losses such as the exponential loss, losses with super-exponential tails ( exp(  x r ) , r > 1 ) and the logistic loss.
Theorem 8 (Last iterate convergence of DGD) Consider DGD with the loss functions and mixing matrix satisfying Assumptions 1,2,7,8 and Assumption 4 with ↵ = 1 and c = h .
Assume that the step-size satisﬁes ⌘ <  ˆ F (1) , for a constant  depending only on the mixing matrix, ⌧ and h .
Then, the train loss and the consensus error of DGD at iteration T satisfy the following for all w 2 R d ,
[Equation]
Remark 9 (Improved rates) While similar to Lemma 2, for logistic regression we have ˆ F ( ¯ w ( T ) )
=
˜ O ( 1 ⌘ T + 1 T ) , for the consensus error rate we have by applying Theorem 8 and noting that ⇢ ( " )
= log(1 / " ) /  ,
[Equation]
After choosing " = 1 /T , we have the improved rate k W ( T )  ¯ W ( T ) k 2 F = ˜
O
( 1 T 2 ) , which is a signiﬁcant improvement from ˜ O
( 1 T ) for general convex losses with constant ⌘ (Lemma 2).
For the test loss, employing Lemma 1 with the new rates for the consensus error leads to the following rate for DGD with logistic regression,
[Equation]
We note that this is a superior rate for the test loss of logistic regression, compared to the rate of
Remark 6.
Notably, this is comparable to the test loss rate for “centralized” logistic regression on linearly separable data, derived as E [ F ( ¯ w ( T ) )]
=
˜ O ( 1 ⌘ T + 1 n ) in [25, 26].
3. Conclusion
We studied the behavior of train loss and test loss of decentralized gradient descent (DGD) methods when training dataset is separable.
[-Remark 
-]
To the best of our knowledge, this yields the ﬁrst rigorous guarantees for the generalization error of DGD in such a setting.
[-Lemma 
-]
[-Lemma 
-]
[-Lemma 
-]
[-¯ W (1) = 0 and by choosing η t ≤ (1 − λ ) √ 1 − β 1 
-]
[-h 2 M ( t ) √ β 
-]
For the same setting, we also proposed fast algorithms and empirically veriﬁed that they accelarate both training and test accuracy.
[-Proposition 
-]
We believe our work opens several directions, with perhaps the most exciting one being the analysis of non-convex objectives.
Generalization analysis of quantized DGD methods over time-varying and directed topologies [2] is another future direction.
We are also interested in extending our results to other distributed settings such as federated learning [10] and Gradient Tracking [21]