Thinking Outside the Ball: Optimal Learning with Gradient Descent for Generalized Linear Stochastic Convex Optimization
Abstract
We consider linear prediction with a convex Lipschitz loss, or more generally, stochastic convex optimization problems of generalized linear form, i.e. where each instantaneous loss is a scalar convex function of a linear function.
We show that in this setting, early stopped Gradient Descent (GD), without any explicit regularization or projection, ensures excess error at most ε (compared to the best possible with unit Euclidean norm) with an optimal, up to logarithmic factors, sample complexity of ˜ 𝑂 ( 1 / ε 2 ) and
only ˜ 𝑂 ( 1 / ε 2 ) iterations.
This contrasts with general stochastic convex optimization, where Ω ( 1 / ε 4 ) iterations are needed Amir et al.
The lower iteration complexity is ensured by leveraging uniform convergence rather than stability.
But instead of uniform convergence in a norm ball, which we show can guarantee suboptimal learning using Θ ( 1 / ε 4 ) samples, we rely on uniform convergence in a distribution-dependent ball.
Introduction
The recent success of learning using deep networks, with many more parameters than training points, and even without any explicit regularization, has brought back interest in how biases and “implicit regularization” due to the optimization method used, can ensure good generalization, even in situations where minimizing the optimization objective itself cannot [28].
Alongside interest in understanding the algorithmic biases of optimization in non-convex, deep models and how they can yield good generalization [e g. 3, 6, 8, 13–15, 21, 23–25, 29, 30, 39], there has also been renewed interest in understanding the fundamentals of algorithmic regularization in convex models [1, 2, 9, 10, 31, 33], both as an interesting and important setting in its own right and even more so as a basis for understanding the situation in more complex models (how can we hope to understand phenomena in deep, non-convex models, if we can’t even understand them in convex models?).
In particular, these fundamental questions include the relationship between algorithmic regularization, explicit regularization, uniform convergence and stability; and the importance of stochasticity and early stopping.
In this paper we focus on the algorithmic bias of deterministic (full batch) optimization methods, and in particular of full-batch gradient descent (i.e. gradient descent on the empirical risk, or “training error”).
Even when gradient descent (GD) is run to convergence, it affords some bias that can ensure generalization.
E.g. in linear regression (and even slightly more general settings) where interpolation is possible, it can be shown to converge to the minimum norm interpolating solution.
This can be sufficient for generalization even in underdetermined situations where other interpolators (i.e. other minimizers of the optimization objective) would not generalize well.
Indeed, the generalization ability of the minimum norm interpolating solution, and hence of GD, even in noisy situations, is the subject of much study.
And in parallel, there has also been work going beyond GD on linear regression, characterizing the limit point of GD, or of other optimization methods such as Mirror
Descent, steepest descent w.r.t.
a norm, coordinate descent and AdaBoost, for different types of loss functions [12, 17–19, 22, 34, 35, 37, 38].
But what about situations where interpolation, or completely minimizing the empirical risk, is not desirable, and generalization requires compromising on the empirical risk?
In this cases one can consider early stopping, i.e. running GD only for some specific number of iterations.
Indeed, early stopped GD is a common regularization approach in practice, and other learning approaches, most prominently Boosting, can also be viewed as early stopping of an optimization procedure (coordinate descent in the case of Boosting).
When and how can such early stopped GD allow for generalization?
How does this compare to Stochastic Gradient Descent (SGD), or to using explicit regularization, both in terms of generalization ability, and the number of required optimization iterations?
And what tools, such as uniform convergence, distribution-dependent uniform convergence, and stability, are appropriate for studying the generalization ability of GD?
Our main result We will show that when training a linear predictor with a convex Lipschitz loss (or more generally, for stochastic convex optimization with a generalized linear instantaneous objective),
GD with early stopping can generalize optimally, up to logarithmic factors, with optimal sample complexity ˜ 𝑂 ( 1 / ε 2 ) , and with an optimal number ˜ 𝑂 ( 1 / ε 2 ) of iterations (the same as stochastic gradient descent), even without any explicit regularization, and in particular without projections into a norm ball—just unconstrained GD on the training error.
This contrasts to previous results regarding early stopped GD for arbitrary stochastic convex optimization (not necessarily generalized linear), for which Amir et al.
[2] showed Θ ( 1 / ε 4 ) GD iterations are needed (even if projections or explicit regularization is used).
Stability vs Uniform Convergence for SGD and GD
The important difference here, and the only property of generalized linear models (GLMs) that we rely on, is that GLMs satisfy uniform convergence Bartlett and Mendelson [4]: empirical losses converge to their expectations uniformly over all predictors in Euclidean balls.
This is in contrast to general Stochastic Convex Optimization (SCO), for which no such uniform convergence is possible [33].
Consequently, rather then uniform convergence, the analysis for SCO is based on stability arguments Bassily et al.
For SGD, stability at each step can be used in an online analysis, combined with an online-to-batch conversion, which is sufficient for ensuring optimal generalization with an optimal 𝑂 ( 1 / ε 2 ) number of iterations.
But for GD, we must consider the stability of the method as a whole, which leads to optimal rates in the case of smooth losses Hardt et al.
but otherwise is much worse, necessitating using a smaller stepsize, and hence quadratically more iterations—[2] showed that this is not just an analysis artifact, but a real limitation of GD for general SCO.
In this paper, we show that once generalization can be ensured via uniform convergence, e.g. for GLMs, then GD does not have to worry about stability, can take much larger step sizes, and generalize optimally after only ˜ 𝑂 ( 1 / ε 2 ) iterations.
But we must be careful with how we ensure uniform convergence!
To rely on uniform convergence, we need to ensure the output of GD lies in some ball, and the generalization error would then scale with the radius of the ball.
A naïve approach would be to ensure that output of GD lies in a norm ball around the origin, and rely on uniform convergence in this ball.
In Section 4 we show that this approach can ensure generalization, but only with suboptimal sample complexity of 𝑂 ( 1 / ε 4 ) —that is, worse than with the stability-based approach.
Instead, we show that, with high probability, the output of GD lies in a small (constant radius) distribution dependent ball, centered not at the origin, but around the (distribution dependent) output of GD on the population objective.
Even though this ball is unknown to the algorithm, this is still sufficient for generalization.
The situation here is similar to the notion of algorithm dependent uniform convergence introduced by [26], though we should emphasize that here we show that algorithm-dependent uniform convergence is sufficient for optimal generalization.
Context and Insights Our results complement recent results exposing gaps between generalization using stochastic optimization versus explicit regularization or deterministic optimization in stochastic convex optimization.
[31] showed that for a setting that is slightly weaker (where only the population loss is convex, but instantaneous losses can be non-convex), there can be large gaps between SGD versus GD or explicit regularization: even though SGD can learn with the optimal sample complexity of 𝑂 ( 1 / ε 2 ) , explicit regularization, in the form of regularized empirical risk minimization cannot learn at all, even with arbitrarily many samples, and GD requires at least a suboptimal Ω ( 1 / ε 2 . 4 ) number of samples (it is not clear if this sample complexity is sufficient for GD, or even if it can at all ensure learning).
This highlights that the generalization ability of SGD cannot be understood in terms of mimicking some regularizer, as well as gaps between stochastic and deterministic optimization.
Returning to the strict SCO setting, where the instantaneous losses are convex, still, uniform convergence does not hold, and generalization can only be ensured via algorithmic-dependent bounds.
Nevertheless, optimal generalization can be ensured either thorough explicit regularization, SGD or
GD—all three approaches can ensure learning with Θ ( 1 / ε 2 ) samples [5, 27, 33].
Even so, differences between deterministic and stochastic methods still exist.
[9] shows that in SCO, the output of
SGD cannot even be guaranteed to lie in some “small" distribution-dependent set of (approximate) empirical risk minimizers 1 , and so the generalization ability of SGD in this settings cannot be attributed to any “regularizer” being small.
Dauber et al.
also shows that GD does not follow some distribution-independent regularization path.
In contrast, here we show that if we do take the distribution into account, then GD is constrained to follow (at least approximately) a predetermined path (i.e. deterministic path, that is independent of the sample, but does depend only on the distribution).
Finally, there is also a gap between SGD and GD, in this case, in the required number of iterations [2].
Surprisingly, this gap cannot be fixed by adding regularization to the objective [1].
Importantly, for either weak or strict SCO, regularization in the form of constraining the norm of the predictor (i.e. empirical risk minimization inside the hypothesis class we are competing with), is not sufficient for learning [10, 33].
The failure of constrained ERM is critical here for the constructions of Amir et al.
, Dauber et al.
and Sekhari et al.
establishing the gaps above: since projected gradient descent would quickly converge to the constrained ERM, it would also generalize just as well, and the gaps and constructions are valid also for projected gradient descent.
In this paper we turn to the GLM setting, which is perhaps one of the most well studied frameworks in the learning theory literature, as it captures fundamental problems such as logistic regression,
SVM and many more.
Uniform convergence for GLMs [4, 20, 32] ensures that constrained ERM learns with optimal sample complexity, and hence so would projected GD with 𝑂 ( 1 / ε 2 ) iterations.
Interestingly, GD on the Tikhonov-type regularized objective similarly yields optimal learning with only 𝑂 ( 1 / ε 2 ) iterations [36], indicating the lower bounds of Amir et al.
on the iteration complexity do not hold in this setting.
But both projected GD and GD on the Tikhonov regularized objective are forms of explicit regularization, and so we study unregularized, unprojected , GD.
Our results show that (a) when uniform convergence holds, both sample complexity and iteration complexity gaps between stochastic and deterministic optimization disappear (though stochastic optimization is of course still much more computationally efficient), and (b) perhaps surprisingly, distributiondependent uniform convergence is not only able to explain generalization of the output of GD, but it can do so better than stability; (c) though it appears to be suboptimal, distribution independent uniform convergence can explain generalization even though the norm could become very large (by making explanations based on uniform convergence inside a fixed distribution independent class).
Problem Setup and Background
We study the problem of stochastic optimization from i.i.d.
data samples.
For that purpose, we consider the standard setting of stochastic convex optimization.
A learning problem consists of a family of loss functions 𝑓 : W × Z → ℝ defined over a fixed domain W ⊆ ℝ 𝑑 , parameterized by the parameter space Z .
For each 𝐵 we denote the domain
[Equation]
Our underlying assumption is that for each 𝑧 ∈ Z the function 𝑓 ( 𝑤 ; 𝑧 ) is convex and 𝐿 -Lipschitz with respect to its first argument 𝑤 .
In this setting, a learner is provided with a sample 𝑆 = { 𝑧 1 , . . .
, 𝑧 𝑛 } of i.i.d.
examples drawn from an unknown distribution 𝐷 .
The goal of the learner is to optimize the population risk defined as follows:
[Equation]
More formally, an algorithm A is said to learn the class W (in expectation), with sample complexity 𝑚 ( ε ) if given i.i.d.
sample 𝑆 = { 𝑧 1 , . .
, 𝑧 𝑛 } such that 𝑛 ≥ 𝑚 ( ε ) , the learner returns a solution
A ( 𝑆 ) that holds
[Equation]
A prevalent approach in stochastic optimization, is to consider, and optimize, the empirical risk over a sample 𝑆 :
[Equation]
without projections)
[Equation]
A concrete and prominent way in minimizing the empirical risk in Eq.
(1) is with Gradient Descent (GD).
GD is an iterative algorithm that runs for 𝑇 iterations, and has the following update rule that depends on a learning-rate η :
[Equation]
where 𝑤 𝑆 0 = 0, and ∇ 𝐹 𝑆 ( 𝑤 𝑆𝑡 ) is the (sub)gradient of 𝐹 𝑆 at point 𝑤 𝑆𝑡 .
The output of GD is normally taken to be the average iterate, ¯ 𝑤 𝑆
[Equation]
Throughout the paper we consider the aforementioned output in Eq.
(3), though we remark that our results also extend to any reasonable averaging scheme (including prevalent schemes such as tail-averaging or random choice of 𝑤 𝑡 ).
It is well known (e.g. [32]) that the output of GD with standard initialization at the origin, enjoys the following guarantee over the empirical risk , for every 𝐵 :
[Equation]
In contrast, as far as the population risk, it was recently shown in [2] that, for sufficiently large 𝑑 , GD suffers from the following sub optimal rate
[Equation]
In particular, setting for concreteness 𝐵 = 1, a choice of η = 1 /√ 𝑇 and 𝑇 = 𝑂 ( 1 / ε 2 ) may lead to ε -error over the empirical risk, but is susceptible to overfitting.
In fact, it turns out that 𝑇 = 𝑂 ( 1 / ε 4 ) iterations are necessary and sufficient [5] for GD (with or without projections) to obtain 𝑂 ( ε ) population error.
2.2 Generalized Linear Models.
One, important, class of convex learning problems is the class of generalized linear models (GLMs).
In a GLM problem, 𝑓 ( 𝑤 ; 𝑧 ) takes the following generalized linear form:
[Equation]
where Z = X
× Y , ℓ ( 𝑎, 𝑦 ) is convex and 𝐿 -Lipschitz w.r.t.
𝑎 , and, φ : X → 𝐻 is an embedding of the domain X into some norm bounded set in a linear space (potentially infinite).
As we mostly care about norm bounded solutions, we will assume for concretness that ∥ φ ( 𝑥 )∥ ≤ 1.
We also treat the value of the function at zero as constant, namely | ℓ ( 0 , 𝑦 )| ≤ 𝑂 ( 1 ) .
In turn, the function 𝑓 is 𝐿 -Lipschitz as well, and we obtain by Lipschitness that:
[Equation]
Uniform Convergence of GLMs.
One desirable property of GLMs is that, in contrast with general convex functions, they enjoy dimension-independent uniform convergence bounds .
This, potentially, allow us to circumvent the bound in Eq.
In more detail, a seminal result due to Bartlett and
Mendelson [4] shows that, under our restrictions on ℓ and φ , the empirical risk 𝐹 𝑆 ( 𝑤 ) converges uniformly to the population loss 𝐹 𝐷 ( 𝑤 ) as follows for any 𝐵 :
[Equation]
By a similar reasoning, one can show a stronger property for GLMs, which we will need: uniform convergence on any ball, not necessarily centered around zero.
More formally, for every 𝑢 and 𝐾 let us notate
[Equation]
We can then bound the expected generalization error of 𝑤 ∈ W 𝐾𝑢 as follows:
Lemma 2.1.
Suppose ℓ ( 𝑎, 𝑦 ) is 𝐿 -Lipschitz w.r.t.
𝑎 and φ : X → 𝐻 is an embedding in a linear space 𝐻 such that ∥ φ ( 𝑥 )∥ ≤ 1 .
Then, for 𝑓 as in Eq.
We have for any 𝑢 and 𝐾
[Equation]
The proof is essentially the same as the proof of Eq.
(8), and exploit the Rademacher complexity of the class.
For the sake of completeness we repeat it in the supplementary material.
Eq.
(8), and more generally Lemma 2.1, imply that any algorithm, A whose range is restricted to a 𝐵 -bounded ball, that has an optimization guarantee of,
[Equation]
will also obtain an 𝑂 ( 𝐿𝐵 /√ 𝑛 ) convergence rate of the excess population risk.
For example, projectedGD is an algorithm that enjoys the aforementioned generalization bound.
Main Results
Theorem 3.1.
Suppose 𝐷 is a distribution over Z = X
× Y .
Let 𝑓 : ℝ 𝑑 × Z → ℝ be a loss function of the form given in Eq.
(6) such that for all 𝑦 ∈ Y , ℓ (· , 𝑦 ) is convex and 𝐿 -Lipschitz with respect to its first argument and such that for all 𝑥 ∈ X : ∥ φ ( 𝑥 )∥ ≤ 1 .
Then, for the gradient descent solution in Eq.
(3) and for any 𝐵 ∈ ℝ we have:
[Equation]
In particular, setting η = 𝑂 ( 1 /( 𝐿 √ 𝑇 )) and 𝑇
=
𝑛 we get,
[Equation]
Thus, Theorem 3.1 shows that using GD, in the case of GLMs, we can learn the class W 𝐵 , with sample complexity 𝑚 ( ε ) =
˜ 𝑂
( 1 / ε 2 ) and number of iterations 𝑇 =
˜ 𝑂
( 1 / ε 2 ) .
Note that, an improved rate of ˜ 𝑂 ( 𝐿 ∥
𝑤 ∥/√ 𝑛 ) can be obtained, if we choose η = 𝑂 (∥ 𝑤 ∥/( 𝐿 √ 𝑇 )) , but this requires prior knowledge of the bounded domain radius.
The bound we formulate above is for a learning rate that is oblivious to the norm of the optimal choice 𝑤 .
The key technical tool in proving Theorem 3.1 is the following structural result that characterizes the output of GD over the empirical risk, and may be of independent interest:
Theorem 3.2.
Let 𝑆 = { 𝑧 1 , . .
, 𝑧 𝑛 } be an i.i.d sequence drawn from some unknown distribution 𝐷 .
Assume that 𝑓 : ℝ 𝑑 × Z → ℝ is a convex and 𝐿 -Lipschitz function, for all 𝑧 ∈ Z , with respect to its first argument.
Then, given the distribution 𝐷 there exists a sequence 𝑢 1 , . .
, 𝑢 𝑇 that depends on 𝐷 (but independent of the sample 𝑆 ), such that, for any δ ∈ ( 0 , 1 ) with probability of at least 1 − δ over 𝑆 , the iterates of gradient descent, as depicted in Eq.
(2) , satisfy:
[Equation]
For example, we can set η = ˜ 𝑂 ( 1 /√ 𝑇 ) and 𝑇 =
𝑂 ( 𝑛 ) , then with probability 𝑂 ( 1 / 𝑛 ) :
[Equation]
As such, for the natural choice of learning rate and iteration complexity, we obtain that the iterates of GD remain at the vicinity of a deterministic trajectory, predetermined by the distribution to be learned.
Our following result shows that this bound is tight up to some logarithmic factor.
We refer the reader to the supplementary material for the full proof.
Theorem 3.3.
Fix η , 𝐿,𝑇 and 𝑛 .
For any sequence 𝑢 1 , . . .
, 𝑢 𝑇 , independent of the sample 𝑆 , there exists a convex and 𝐿 -Lipschitz function 𝑓 : W × Z
→ ℝ and a distribution 𝐷 over Z , such that, with probability at least 1 / 20 over 𝑆 , the iterates of gradient descent, as depicted in Eq.
(2) , satisfy:
[Equation]
3.1 High Probability Rates
We next move to discuss results for learnability with high probability.
We first remark that using
Markov’s inequality and, standard techniques to boost the confidence, One can achieve high proability rates at a logarithmic computational and sample cost in the confidence.
If we want, though, to achieve high probability rates for the algorithm without alterations, the standard approach requires concentration bounds which normally rely on boundness of the predictor.
This, unfortunately is not true when we run GD without projection, as the prediction can be potentially unbounded.
However, under natural structural assumptions that are often met for the types of losses we are usually interested in, we can achieve such boundness by clipping procedure which we next describe.
For this section we consider a loss function ℓ ( 𝑎, 𝑦 ) , such that 𝑦 ∈ [− 𝑏, 𝑏 ] and, we assume that:
[Equation]
Note that this is a very natural assumption to have in the case of convex surrogates for prediction tasks.
For example, this holds for the widely used hinge loss ℓ ( 𝑎, 𝑦 )
=
max { 0 , 1 − 𝑦𝑎 } in binary classification.
Observe that under this assumption, if we consider 𝑤 · φ ( 𝑥 ) as a predictor of 𝑦 , the learner has no incentive to return a prediction that is outside of the interval [− 𝑏, 𝑏 ] .
Thus, we define the following mapping 𝑔 ( 𝑎 )
[Equation]
and consider the clipped solution 𝑔 ( ¯ 𝑤 𝑆 · φ ( 𝑥 )) where ¯ 𝑤 𝑆 is the original output of GD defined in
Eq.
We can now present our high probability result.
Theorem 3.4.
Suppose 𝐷 is a distribution over Z = X
× Y and let W 𝐵 = (cid:8) 𝑤 ∈ ℝ 𝑑 : ∥ 𝑤 ∥ ≤ 𝐵 (cid:9) .
Let ℓ ( 𝑎, 𝑦 ) be a generalized linear loss function of the form given in Eqs.
(6) and (9) such that for all 𝑦 ∈ Y , ℓ (· , 𝑦 ) is convex and 𝐿 -Lipschitz with respect to its first argument.
Then, for the gradient descent solution in Eq.
(3) and the function 𝑔 ( 𝑎 ) in Eq.
(10) we have with probability at least 1 − δ 𝔼 ( 𝑥,𝑦 )∼ 𝐷
[Equation]
In particular
[Equation]
setting η = 1 /( 𝐿 √ 𝑇 ) and 𝑇
=
𝑛 we obtain that with probability 1 − δ :
[Equation]
Comparison with non-distribution-dependent uniform convergence
In this section, we contrast our approach with what might be possible with a more traditional, non-distribution-dependent, uniform convergence.
In particular, we consider an argument based on ensuring that the output of GD, with some stepsize η and number of iterations 𝑇 , is always (or with high probability) in a ball of radius 𝐵 ( η ,𝑇 ) around the origin, namely, ∥ ¯ 𝑤 𝑆 ∥ ≤ 𝐵 ( η ,𝑇 ) .
Hence, using
Rademacher complexity bounds for GLMs, we can say that its population risk is within 𝑂 ( 𝐵𝐿 /√ 𝑛 ) of its empirical risk.
By selecting η and 𝑇 to be very small, one can always ensure that the output of GD is in a small ball, but we also need to balance that with the empirical suboptimality of the output.
That is, traditional bounds require us to find η and 𝑇 that balance between the guarantee on the empirical suboptimality and the guarantee on the norm of the output.
Such an approach would then result in population suboptimality that is governed by these two terms:
[Equation]
where 𝐷 and 𝑓 are taken over all valid distributions such that 𝑓 is convex and Lipschitz.
In particular, we would get,
[Equation]
Claim 4.1.
For GLMs it holds that 𝐺 ( 𝑛 ) =
Θ ( 𝐿 / 𝑛 1 / 4 ) .
Proof.
The upper bound follows by taking η = 1 /( 𝐿 √ 𝑇 ) and 𝑇 = √ 𝑛 , bounding the first term using the standard GD guarantee in Eq.
(4), and the second term by noting that each step increases the gradient by at most η 𝐿 .
Therefore, ∥ ¯ 𝑤 𝑆 ∥ ≤ η 𝐿𝑇
and we obtain,
[Equation]
For the lower bound we consider two deterministic functions in one dimension.
When η 𝑇 ≥ 𝑛 1 / 4 / 𝐿 we consider the deterministic objective 𝑓 ( 𝑤 ; 𝑧 )
=
𝐿𝑤 .
Clearly, the norm of the solution is ∥ ¯ 𝑤 𝑆 ∥ =
𝑇 (cid:80)
𝑇𝑡 = 1 η 𝐿𝑡 ≥ η 𝐿𝑇 / 2, thus:
[Equation]
When η 𝑇 < 𝑛 1 / 4 / 𝐿 we consider the objective 𝑓 ( 𝑤 ; 𝑧 )
=
𝐿𝑤 / 𝑛 1 / 4 .
Similarly, we have that ¯ 𝑤 𝑆 = − η 𝐿 ( 𝑇 + 1 )/( 2 𝑛 1 / 4 ) .
Thus, the empirical risk is:
[Equation]
■
The claim shows that uniform convergence to a fixed ball around the origin can ensure learning, but with a suboptimal sample complexity of 𝑂 ( 1 / ε 4 ) .
To obtain better bounds using this approach, additional structural assumption on the loss or the distribution are required.
For example, the work of [34] assumes max-margin and smoothness(specifically the logistic loss) to obtain bounds on the norm of the solution, while our result applies to general Lipschitz GLMs without any distribution assumptions.
It is insightful to directly contrast Eq.
(11) to the approach of Section 3, which essentially entails looking at:
[Equation]
for some deterministic 𝑢 .
We can then apply a uniform concentration guarantee for a ball around it, and so ˜ 𝐺 is also sufficient to ensure:
[Equation]
In Section 3 we show that ˜ 𝐺 ( 𝑛 ) =
˜ 𝑂
( 𝐿 /√ 𝑛 ) , improving on 𝐺 ( 𝑛 ) and yielding optimal learning, up to logarithmic factors.
Technical Overview
Our main result, Theorem 3.1, establishes a generalization bound for GD without projection, and it builds upon a structural result presented in Theorem 3.2.
We next, then, outline the derivation of both these results.
Because the most interesting implication of our results are derived when we choose η = ˜ 𝑂 ( 1 /√ 𝑇 ) and 𝑇 = 𝑂 ( 𝑛 ) , we will focus here in the exposition on this regime.
This is mainly to avoid clutter notation.
Hence, unless stated otherwise we assume that 𝑇 and η are fixed appropriately.
Our proof of Theorem 3.1 relies on two steps.
First, through Theorem 3.2 we argue that that output of GD will be (w.h.p) in a fixed ball that depends solely on the distribution (but not on the sample).
Then, as a second step, we can apply standard uniform convergence for a bounded-norm balls (see
Lemma 2.1) to reason about generalization.
The second step, builds on a standard generalization bound that is derived through Rademacher complexity.
Also notice, that the first step follows immediately from Theorem 3.2.
Indeed, Theorem 3. argues that there exist a sequence 𝑢 1 , . . .
, 𝑢 𝑇 such that if 𝑤 𝑆𝑡 is the trajectory of GD over the empirical risk, we will have (w.h.p):
[Equation]
The output of GD is the averaged iterate, hence we obtain that indeed ¯ 𝑤 𝑆 is restricted to a ball around 𝑢 = 1 𝑇 (cid:80)
𝑇𝑡 = 1 𝑢 𝑡 , as required.
Therefore, we are left with proving our main structural result: That the iterates of GD are in a proximity of a trajectory 𝑢 1 , . . .
, 𝑢 𝑇 that depends solely on the distribution (i.e. Theorem 3.2).
For the end of proving Theorem 3.2, we introduce the following GD trajectory:
Gradient Descent on the population loss.
We consider an alternative sequence of gradient descent that operates on the population loss 𝐹 𝐷 ( 𝑤 ) rather than the empirical risk 𝐹 𝑆 ( 𝑤 ) .
The update rule is then,
[Equation]
The sequence 𝑤 𝐷 1 , . . .
, 𝑤 𝐷𝑡 will serve as the sequence 𝑢 1 , . .
𝑢 𝑇 .
In the proof of Theorem 3.2 we require high probability rates.
However, for the simplicity of the exposition, we will prove here a slightly weaker, in expectation, result: (the proof is defered to the supplementary material).
Lemma 5.1.
Let 𝑆 = { 𝑧 1 , . .
, 𝑧 𝑛 } be an i.i.d sequence drawn from some unknown distribution 𝐷 .
Assume that 𝑓 : ℝ 𝑑 × Z → ℝ is a convex and 𝐿 -Lipschitz function, for all 𝑧 ∈ Z , with respect to its first argument.
Then, the iterates of gradient descent, as depicted in Eqs.
(2) and (12) , satisfy:
[Equation]
This lemma bounds the distance, in expectation, between the GD trajectory over the empirical risk and the GD trajectory over the population risk.
Again, we remark that to obtain the final result in Theorem 3.2, a stronger high probability version of Lemma 5.1 is required.
The proof of Theorem 3.2 follows similar lines to that of Lemma 5.1 with modifications concerning specific concentration inequalities.
One crucial challenge in proving Lemma 5.1 stems from the adaptivity of the gradient sequence.
In particular, notice that the sequences 𝑤 𝑆𝑡 , 𝑤 𝐷𝑡 are governed, respectively, by the dynamics
[Equation]
At a first glance, since ∇ 𝐹 𝑆 ( 𝑤 ) is an estimate of ∇ 𝐹 𝐷 ( 𝑤 ) , it might seem that the result can be obtained by standard concentration bounds and application of a union bound along the trajectory.
Unfortunately, such a naive argument cannot work.
Indeed, since 𝑤 𝑡 , for 𝑡 ≥ 2, depends on the sequence 𝑆 , ∇ 𝐹 𝑆 ( 𝑤 𝑡 ) is not necessarily an unbiased estimate of ∇ 𝐹 𝐷 ( 𝑤 𝑡 ) .
This is not just seemingly, and a construction in [2] demonstrates how, even after only two iterates the gradient ∇ 𝐹 𝑆 ( 𝑤 𝑡 ) can diverge, significantly, from ∇ 𝐹 𝐷 ( 𝑤 𝑡 ) .
While these constructions are outside of the scope of GLMs, we remark that Theorem 3.2 holds for any convex and Lipschitz function.
Also, it was in fact shown that even for GLMs [11], the estimate of the gradients don’t have any dimension-independent uniform convergence bound, making it a challenge to pursue such a proof direction.
To summarize, we need to prove that the two sequences remain in the vicinity, even though, apriori, the update steps of the two functions may be different at each iteration.
Towards this goal, we follow an analysis, reminiscent of the uniform argument stability analysis of [5] for non-smooth convex losses.
In a nutshell, both arguments compare the trajectory to a reference trajectory, and bound the incremental difference between the two trajectories while exploiting monotonicity of the convex function.
In the case of stability, the reference trajectory is the trajectory 𝑤 𝑆 ′ 𝑡 induced by a sample 𝑆 ′ =
( 𝑧 1 , . . .
, 𝑧 ′ 𝑖 , . .
, 𝑧 𝑛 ) that differ on a single example from the sample 𝑆 = ( 𝑧 1 , . . .
, 𝑧 𝑖 , . . .
, 𝑧 𝑛 ) .
Bassily et al.
showed that if 𝑆 and 𝑆 ′ are two sequences that differ on a single example, then:
[Equation]
In comparison, we show:
[Equation]
We note, that both results are optimal.
Namely, the stability bound of [5] is the optimal stability rate, and the proximity bound we provide is the best possible bound against a fixed point, independent of the sample.
Note that both bounds yield 𝑂 ( 1 ) difference for the natural choice of η and 𝑇 .
However, 𝑂 ( 1 ) - stability guarantees are vacuous in the sense that they do not provide any interesting implication over the generalization of the algorithm.
Whereas, we provide 𝑂 ( 1 ) -proximity guarantee to some fixed point, completely independent of the sample 𝑆 .
This does imply generalization in our setup.
One might also suggest that our result of 𝑂 ( 1 ) -proximity can be derived from 𝑂 ( 1 ) -stability.
However, it turns out this is not the case.
For the same instance we use to lower bound the proximity by Ω ( η 𝑇 /√ 𝑛 ) in Theorem 3.3, it can be easily shown that GD will be 𝑂 ( η 𝑇 / 𝑛 ) -stable.
Setting η = 𝑂 ( 1 ) and 𝑇 = 𝑂 ( 𝑛 ) will then imply 𝑂 ( 1 ) -stability
but with Ω (√ 𝑛 ) -proximity.
This asserts that our result is not a direct consequence of standard stability arguments.
Acknowledgements RL received support from the Israeli Science Foundation (ISF) (grant no 2188/20) and from a Google research scholar award.